Setting up Glowroot as Central Collector - Liferay

Shubhankar Soni
Published Date1 Year Ago - 6305 Views

It is critical to have the correct monitoring tools in the right places while debugging Performance Bottlenecks in Production/Non-Production Environments. Glowroot is a strong open-source APM solution that is simple to set up and operate. There are different modes of setting up Glowroot i.e.

  1. Glowroot as Agent
  2. Glowroot as Central Collector

In this blog, we'll go through how to configure Glowroot as a Central Collector and use Collector agents deployed on Liferay to collect data and deliver it to Glowroot central.

Pre-Requisites:

  1. Separate VM/Server dedicated to glowroot.

  2. Regular OS User preferably glowroot.

  3. JAVA_HOME set for OS user created above (glowroot). In this blog Azul JDK11 is used.

  4. Cassandra 3.1 or later will be used as Database for Glowroot Central.

  5. Glowroot Central and Cassandra DB will be installed on Same server.

H/w Recommendations:

The following suggestions are not official, but rather based on experience.

  1. CPU: 8 Cores
  2. Memory/RAM: 8/16GB
  3. Storage: 100GB

Architecture Diagram:


 

Some Advantages:

  1. Does not use Liferay's JVM to run, hence it does not interfere with Liferay's JVM operation and places little to no impact on Liferay's JVM.
  2. Because Glowroot central is installed on a different VM/server, it does not consume and hinder the Liferay server's OS resources, which is not the case when Glowroot as Agent is installed on the Liferay server.

Download and Install JDK:

  1. Login via glowroot user in Glowroot server/VM.

  2. Download any jdk11 ( here Azul jdk11 is used)

     

  3. Go to /home/glowroot/ directory and open .bash_profile or .bashrc and add the jvm path like below 

         export JAVA_HOME=/home/glowroot/zulu11.66.15-ca-jdk11.0.20-linux_x644
         export PATH=$JAVA_HOME/bin:$PATH
         

     4. Save and Exit the above file.

     5. Now run source .bashrc command to reflect the changes.
         

     6. Once done check Java as per the below snapshot.

         

Download & Install Cassandra 

  1. Go to https://dlcdn.apache.org/cassandra/4.0.11/apache-cassandra-4.0.11-bin.tar.gz and download Cassandra binary file and paste it in the /home/glowroot/ directory.
     

  2. Unzip the binary file downloaded above in the same directory /home/glowroot/
    (use tar -xvzf apache-cassandra-4.0.11-bin.tar.gz to unzip)
     

  3. Go to /home/glowroot/apache-cassandra-4.0.11/bin and execute below command to start cassandra db

    nohup ./cassandra &

     

  4. Cassandra will start running in some time.

Download & Install Glowroot Central:

  1. Go to https://github.com/glowroot/glowroot/wiki/Central-Collector-Installation and download the available glowroot central zip file.


     

  2. Copy the downloaded zip file to the server in directory /home/glowroot/ and unzip.

  3. After unzipping go inside the glowroot-central/ directory.


     

  4. Now, start Glowroot central by running below command
    nohup java -jar glowroot-central.jar &


     

  5. Now open the browser and hit http://GlowrootIP:4000 (replace GlowrootIP with the IP address of the Glowroot server)

Configure Glowroot Collector Agent:

(Follow the below steps in all Liferay App Nodes)

  1. Go to the Liferay server and go to liferay_Home/
     

  2. Download the glowroot-agent zip from the link
    https://github.com/glowroot/glowroot/wiki/Agent-Installation-(for-Central-Collector)


     

  3. Copy the zip downloaded above to liferay_home/ folder and extract it.


     

  4. Go inside glowroot/ directory and create a file with the name glowroot.properties


     

  5. Open the glowroot.properties file and add below parameters
    agent.id=LiferayNodeHostname
    collector.address=         http://GlowrootServerIP:8181


     

  6. Save and exit after making the above changes.
     

  7. Make sure to replace LiferayNodeHostname with the original hostname of liferay app server.
     

  8. Go to Liferay_home/tomcat/bin/ directory and open setenv.sh file.
     

  9. Add the Java agent entry in setenv.sh file.


     

  10. Start/Restart liferay servers to load the javaagent .

  11. Below logs will appear in catalina.out log file which means Liferay’s glowroot agent is successfully able to communicate with Glowroot central server.


     

  12. Once the Liferay app is UP and running, go to Glowroot Central in Browser. In the Java agent drop-down, the Liferay node hostname will start coming.

  13. Voilà, you're done! 

References

#

Software/Tool

Reference Links

1.

Azul JDK 11

https://www.azul.com/downloads/?version=java-11-lts&os=rhel&architecture=x86-64-bit&package=jdk#zulu

2.

Cassandra DB

https://dlcdn.apache.org/cassandra/4.0.11/apache-cassandra-4.0.11-bin.tar.gz

3.

Glowroot Central

https://github.com/glowroot/glowroot/wiki/Central-Collector-Installation

4.

Glowroot Collector Agent

https://github.com/glowroot/glowroot/wiki/Agent-Installation-(for-Central-Collector)

 

MUST READ:

https://liferay.dev/blogs/-/blogs/setting-up-glowroot-apm

https://glowroot.org/

 

Comment (2)

More Blog Entries

thumbnail

Want to manage custom fields (Service builder entity) from the Liferay Control Panel? Here you go.

thumbnail
Bhargav R Vaghasiya
11 Aug
0
thumbnail

Moving from AMD to Browser modules in Liferay DXP

thumbnail
Iván Zaera Avellón
11 Aug
0
Blogs
ABHISHEK KUMAR
3 Months Ago - Edited

Hello,

We are facing  below issue on agent level and agent not able to connect to collector.

We checked there is connectivity avaible from agent to collector via telnet.

 

On agent logs we are getting below error:

"2024-08-16 13:13:18.880 INFO  org.glowroot - Glowroot version: 0.13.6, built 2020-03-01 01:25:31 +0000 2024-08-16 13:13:18.881 INFO  org.glowroot - Java version: 1.8.0_371 (IBM Corporation / Linux) 2024-08-16 13:13:18.882 INFO  org.glowroot - Java args: -Xoptionsfile=/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/options.default -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/PrimitiveHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+LegacyXlogOption -XX:+EnsureHashed:java/lang/Class,java/lang/Thread -Xjcl:jclse29 -Xshareclasses:name=webspherev85_1.8_64_bundled_%g,nonFatal -Xcheck:dump -Xbootclasspath/p:/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmorb.jar -verbose:gc -Xms12288m -Xmx18432m -Xcompressedrefs -Xscmaxaot16M -Xscmx120M -Xgcpolicy:balanced -Xgcthreads16 -javaagent:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/liferay/glowroot_agent/glowroot/glowroot.jar 2024-08-16 13:13:20.239 WARN  org.glowroot - one or more important classes were loaded before Glowroot instrumentation could be applied to them: java.util.concurrent.ThreadPoolExecutor$Worker, java.util.concurrent.Executors$RunnableAdapter, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask, java.util.concurrent.FutureTask 2024-08-16 13:13:20.395 INFO  org.glowroot - agent id: ITSP4SBICOINAPP3UAT 2024-08-16 13:13:26.893 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://XXXXXX:XXXX(will keep trying...): org.glowroot.agent.shaded.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f 2024-08-16 13:14:20.375 ERROR o.g.a.util.LazyPlatformMBeanServer - platform mbean server was never created by container"

 

On central collector logs we are getting below error:

"2024-08-16 18:09:31.518 INFO  org.glowroot - gRPC listening on 0.0.0.0:8181 2024-08-16 18:09:32.373 INFO  org.glowroot - UI listening on 0.0.0.0:9084 2024-08-16 18:09:32.374 INFO  org.glowroot - startup complete 2024-08-16 18:15:02.130 ERROR c.d.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 1000 milliseconds 2024-08-16 18:15:03.131 ERROR c.d.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 1000 milliseconds 2024-08-16 18:15:04.132 ERROR c.d.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 1000 milliseconds 2024-08-16 18:15:05.004 ERROR o.g.central.SyntheticMonitorService - com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)     at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:553)     at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:514)     at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:91)     at org.glowroot.central.util.Session.read(Session.java:145)     at org.glowroot.central.repo.ActiveAgentDao.readActiveTopLevelAgentRollups(ActiveAgentDao.java:127)     at org.glowroot.central.repo.ActiveAgentDao.readActiveAgentRollups(ActiveAgentDao.java:161)     at org.glowroot.central.repo.ActiveAgentDao.readRecentlyActiveAgentRollups(ActiveAgentDao.java:156)     at org.glowroot.central.SyntheticMonitorService.runInternal(SyntheticMonitorService.java:274)     at org.glowroot.central.SyntheticMonitorService.run(SyntheticMonitorService.java:233)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)     at java.lang.Thread.run(Thread.java:750) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)     at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:284)     at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)     at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:376)     at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:140)     at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:122)     at com.datastax.driver.core.SessionManager.execute(SessionManager.java:697)     at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:143)     at org.glowroot.central.util.Session.lambda$readAsync$0(Session.java:211)     at org.glowroot.central.util.Session.throttle(Session.java:409)     at org.glowroot.central.util.Session.throttleRead(Session.java:390)     at org.glowroot.central.util.Session.readAsync(Session.java:211)     ... 9 common frames omitted"

 

 

We are using glowroot-0.13.6 version of agent and collector and apache-cassandra-4.0.13 version. All are running on java-1.8.0-openjdk-1.8.0.412.b08-2.el8.x86_64 version.

I request you to please guide us here.

Thanks and regards

 

Please sign in to reply.
Shubhankar Soni ABHISHEK KUMAR
2 Months Ago

Hello Abhishek Kumar, 

After checking the logs I see that there is no communication from Application(where Glowroot agent exists) and Glowroot Central server. 

"WARN o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://XXXXXX:XXXX(will keep trying...)"

Make sure Application is able to communicate to the Central Collector URL http://XXXXXX:XXXX

Please sign in to reply.