Blogs

I've been frequently involved in projects where I had to deal with performance issues. I knew there were many commercial APM solutions but I wanted to check whether there was some free alternative that I could use both on my developer workstation and on the project's infrastructure.

This is how I came across https://glowroot.org.

Glowroot is a Java Application Performance Monitoring application. It integrates with the target application as a Java agent and has 2 operation modes:

Agent + GUI: In that setup, Glowroot collects data at JVM runtime and will persist it in a local H2 store. In addition, the agent exposes a web GUI on a dedicated port (default 4000) where you can consult the collected data. In a clustered environment, you'd have one Glowroot GUI per Liferay node, each having its own collected data.
Collector Agent + Glowroot central: In that setup, the agent just collects the data and shares it to Glowroot central where all collected data is aggregated. You'll have to add Cassandra to the infrastructure to manage the collected information.

I've personnally only used the first basic setup of the two. This is mainly due to the nature of my missions. I come and go from one project to the other, often to deal with some specific issue and I'm not involved in everyday operation.

As you will see, that default setup can be rolled out in a couple of hours on a project infrastructure. When I know a mission will involve some performance monitoring, I usually share the Glowroot installation steps to my contact so that we can use Glowroot the day or the week after because I know it should be simple enough to be dealt with it even with short notice.

If I was responsible for the operations of a Liferay infrastructure, I'd likely implement the second setup.

I've recently made a presentation during Liferay Dev24 showcasing how you can use Glowroot in a Liferay context: https://liferay.dev/twentyfour#Glowroot%20%20-%20Let's%20monitor%20Liferay%20performance%20with%20an%20APM

This first blog post is about setting up Glowroot. I'll write some additional posts later on in order to share some tips like the ones I have talked about during Dev24.

Setting up Glowroot

The Glowroot homepage claims it is very easy to use: https://glowroot.org

And it's almost true. There's just one extra step for Liferay because of its OSGi architecture. As a Java Agent, Glowroot hooks itself all over the JVM. As a consequence, its Java classes have to be loaded whereever it is hooked.

But beware, every OSGi bundle inside of Liferay has its own classloader. For that reason, we need to make sure that the Glowroot java package are visible from them and there is a specific portal property you need to adjust so that those packages are added to the existing list of packages we want to be imported inside of all OSGi bundles.

Dave has written a detailed article about this topic that you can check here: https://liferay.dev/blogs/-/blogs/liferay-ce-7-x-liferay-dxp-7-x-java-agents

That specific portal property is called module.framework.properties.org.osgi.framework.bootdelegation.

The list of packages imported into all OSGi bundles varies from one Liferay version to the other. So, be careful when overriding it and I encourage you to keep an eye on the Liferay default values in the documentation (eg. https://docs.liferay.com/portal/7.4-latest/propertiesdoc/portal.properties.html for DXP 7.4).

In the case of Liferay DXP 7.4, this is what I'm adding to my portal-ext.properties file:

module.framework.properties.org.osgi.framework.bootdelegation=\
    __redirected,\
    com.liferay.expando.kernel.model,\
    com.liferay.portal.servlet.delegate,\
    com.liferay.portal.servlet.delegate*,\
    com.sun.ccpp,\
    com.sun.ccpp.*,\
    com.sun.crypto.*,\
    com.sun.image.*,\
    com.sun.imageio.plugins.*,\
    com.sun.jmx.*,\
    com.sun.jna,\
    com.sun.jndi.*,\
    com.sun.mail.*,\
    com.sun.management.*,\
    com.sun.media.*,\
    com.sun.msv.*,\
    com.sun.org.*,\
    com.sun.tools.*,\
    com.sun.xml.*,\
    com.yourkit.*,\
    jdk.*,\
    sun.*,\
    weblogic.jndi,\
    weblogic.jndi.*,\
    org.glowroot.agent,\
    org.glowroot.agent.*

You now may also want to update the Glowroot admin.json file if you want to adjust some network settings, in particular this section:

  "web": {
    "port": 4000,
    "bindAddress": "127.0.0.1",
    "contextPath": "/",
    "sessionTimeoutMinutes": 30,
    "sessionCookieName": "GLOWROOT_SESSION_ID"
  },

Obviously, port defines the port where the Glowroot GUI will listen.
bindAddress is the network interface where the Glowroot GUI will listen from.
- With the 127.0.0.1 loopback default value, you'll only be able to access the GUI from the local workstation, which is fine in a dev environment.
- You can also set it up to 0.0.0.0 so that it listens from all network interfaces or the IP of the server.
contextPath gives you the ability to make the GUI available behind a specific context path, which is quite useful in reverse proxying scenarios. For example, you might have some nginx or apache server in front of Liferay: you could use it to do some HTTP port forwarding and make the Glowroot GUI available behind some specific context so that you can implement a rule based on a URL pattern. In that scenario, you could even add some IP or client certificate white listing mechanism.

Since we're talking about security, the usual next step (you can usually skip that on the local workstation) is to configure roles, users and passwords.

Here's an example of such configuration:

  "users": [
    {
      "username": "admin",
      "passwordHash": "5NUeWXBUk/ISErr952zZsQ:IG4QENmGup4WNmdaBQkeHg:500000",
      "roles": [
        "Administrator"
      ]
    },
    {
      "username": "glowroot",
      "passwordHash": "mOIl2J0caY2xcWthotYxcg:p0bTZQE3/Rvj51zPuaIBHw:500000",
      "roles": [
        "Glowroot"
      ]
    }
  ],
  "roles": [
    {
      "name": "Administrator",
      "permissions": [
        "agent:transaction",
        "agent:error",
        "agent:jvm",
        "agent:incident",
        "agent:config",
        "admin"
      ]
    },
    {
      "name": "Glowroot",
      "permissions": [
        "agent:transaction",
        "agent:error",
        "agent:jvm",
        "agent:incident"
      ]
    }
  ],

Because hashing those passwords is a bit tedious (you have to download glowroot-central.jar from here https://github.com/glowroot/glowroot/wiki/Central-Collector-Installation and run java -jar glowroot-central.jar hash-password <plain password> to create a password hash), you may just decide to create those roles and users from the GUI.