In this blog, I'd like to show you how you can not only monitor real time HTTP requests but also transactions running in the background.

Short introduction - Glowroot

Since 2023.q4, Glowroot has been bundled with Liferay.

Enabling Glowroot requires to add an environment variable at startup in your development environment in order to get a local Glowroot to run on your workstation. This local Glowroot will save monitoring information in an HSQL Database local to your workstation.

In a production setup, you are most likely going to rely on a more robust setup where the Glowroot agent is connected to a Cassandra database. This will allow you to have all metrics forwarded to a single place, which is particularly useful in case you operate a Liferay cluster, with several nodes or if you have multiple applications to monitor.

If you'd like to watch a 30 minutes introduction to Glowroot and Liferay, how about you check this presentation I made during Liferay DevCon 2024 in Budapest? :)

https://www.youtube.com/watch?v=ghGQKfP_thw

What does Glowroot monitor, by default?

If you have ever enabled Glowroot in one of your environments, you may have noticed that it automatically logs a transaction for every inbound HTTP request the server handles.

However, not everything in Liferay is handled synchroneously within the boundaries of an HTTP request.

Let me give you a few examples:

• Liferay upgrade steps: those operations are triggered automatically at startup when Liferay detects that the database is behind the current version of Liferay.
• Search Index reindexing: this operation is triggered by the administrator and will register a background task which is going to execute the individual indexers.
• Publications / staging publications: those operations are either scheduled or executed immediately, however in both cases, they will be processed through a background task.
• Batch API calls (eg. export / import): when a client hits one of those endpoints, the actual processing of the request also happens in the background.

Instrumentation of background tasks

Even though those transactions are not recorded by default, getting them to be monitored is just one instrumentation away.

The reason why your inbound HTTP requests are monitored is because Glowroot comes with some predefined instrumentation plugins. And one of them records a transaction for every HTTP request processed by the server.

Now, if you'd like to monitor more stuff, like the aforementioned operations, then all you need is to configure additional instrumentations in Glowroot.

An instrumentation requires that you identify the Fully Qualified Class Name (FQCN) of the class and its method which identifies as the entry point of the operation you'd like to monitor.

Upgrade step instrumentation

The following instrumentation records upgrade processes, individually.

{
  "className": "com.liferay.portal.kernel.upgrade.UpgradeProcess",
  "methodName": "upgrade",
  "methodParameterTypes": [],
  "captureKind": "transaction",
  "transactionType": "Upgrade",
  "transactionNameTemplate": "Upgrade {{this.class.name}}",
  "transactionSlowThresholdMillis": 0,
  "alreadyInTransactionBehavior": "capture-new-transaction",
  "traceEntryMessageTemplate": "Upgrade {{this.class.name}}",
  "timerName": "Upgrade process"
}

You may observe that the instrumentation has been made against the interface but that we include {{this.class.name}} in the trace headline in order to identify which specific upgrade step has been executed.

Search index reindexing

The following instrumentation records a transaction for each indexer, individually.

{
  "className": "com.liferay.portal.kernel.search.BaseIndexer",
  "methodName": "doReindex",
  "methodParameterTypes": [
    ".."
  ],
  "captureKind": "transaction",
  "transactionType": "Indexing",
  "transactionNameTemplate": "Index {{this.class.name}}",
  "transactionSlowThresholdMillis": 0,
  "alreadyInTransactionBehavior": "capture-trace-entry",
  "traceEntryMessageTemplate": "Index {{this.class.name}}",
  "timerName": "Index"
}

As you may know, documents & medias have their indexer, so do pages, users, custom objects... In case you witness performance issues with indexing, this individual monitoring will help you identify which specific Liferay entities are causing long indexing times.

Background tasks

Staging or publications execute background tasks. There are some other background task use cases. You can track them all using the following instrumentation:

{
  "className": "com.liferay.portal.kernel.backgroundtask.BackgroundTaskExecutor",
  "methodName": "execute",
  "methodParameterTypes": [
    ".."
  ],
  "captureKind": "transaction",
  "transactionType": "Background Task",
  "transactionNameTemplate": "Execute background task {{this.class.name}}",
  "transactionSlowThresholdMillis": 1000,
  "alreadyInTransactionBehavior": "capture-new-transaction",
  "traceEntryMessageTemplate": "Execute background task {{this.class.name}}",
  "timerName": "Execute background task"
}

Batch API calls

The following instrumentation will isolate the monitoring of the execution of batch engine import tasks within their own Glowroot transactions:

{
  "className": "com.liferay.batch.engine.BatchEngineImportTaskExecutor",
  "methodName": "execute",
  "methodParameterTypes": [
    ".."
  ],
  "captureKind": "transaction",
  "transactionType": "Background Task",
  "transactionNameTemplate": "execute batch engine import {{this.class.name}}",
  "alreadyInTransactionBehavior": "capture-new-transaction",
  "traceEntryMessageTemplate": "execute batch engine import {{this.class.name}}",
  "timerName": "executeBatchEngineImport"
}

One word about nested transactions

Sometimes, Glowroot monitored transactions might be nested. This means that there is a transaction instrumentation which got triggered within the boundaries of another one.

You can decide whether you want to track those nested transactions as individual transactions or have Glowroot trace them as trace entries within their parent transactions.