Blogs

Blogs

How to monitor Liferay (Tomcat) remotely through firewalls using VisualVM

 

  I'm on site these last days and one of my tasks was to execute some load testing and monitor a Liferay instance hosted in a network accessed only via SSH port. No other port except 22 and 80 was open. So after almost 2 days struggling with several different configurations I was able to monitor a remote instance of Liferay through JMX+RMI over SSH tunneling. I had to put together several pieces of information to get this working, let's see the steps.
  I will split this post in two configuration sections for JMX server (Tomcat) and JMX client (VisualVM).
 

  JMX server

  First of all you need to drop in the catalina-jmx-remote.jar to TOMCAT_HOME/lib directory of Tomcat. This will add to the classpath the listener org.apache.catalina.mbeans.JmxRemoteLifecycleListener responsible for enable the JMX+RMI over SSH tunneling. More info about this listener can be found at JMX Remote Lifecycle Listener. This jar may be found under extras directory of download binary area of Tomcat. Only versions newer or equal to 6.0.24 has it available. Your second step is configure the listener on server.xml add the following snippet under the <Server> tag (around line 35).
  
    <Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" useLocalPorts="true" />
 
  The third and last step on the server side is set new parameters to JVM as following to enable JMX remote. Set them on the setenv.(sh|bat) of your Tomcat installation
 
    JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
 

  JMX client

  Again your first step is drop in the jar catalina-jmx-remote.jar into the VISUALVM_HOME/platform/lib. This is a required step otherwise you gonna get a ClassNotFoundException for JmxRemoteLifecycleListener. After that you must start a SSH connection to your server with two tunnel enabled, you can use the command below

 
  ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com
 
  Then you can fire a VisualVM and add a reference to your remote server under the Remote section. Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi and voilà! We are done and ready to start the real work!
 
  

  Other considerations

 
  If you are trying to monitor a cluster of machines be aware that the property  java.rmi.server.hostname=localhost will break the cache replication.
  Maybe your are wondering about the reference of localhost in almost all commands instead of the server's names, right? Don't worry this is the right way beacuse we are using SSH tunnelling.
  
  Some folks would like to work with jconsole. Server configuration are identical. On the client side you gonna fire jconsole with the following parameter -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar to be able to connect. 
  If you have connection problems I would suggest you to use jconsole to debug. You can use the follwoing command line and logging.properties file to enable debugging
 
jconsole -J-Djava.util.logging.config.file=logging.properties -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar
 
  

 

// logging.properties file to activate JMX traces.
// Use this file on the java command line:
//     java -Djava.util.logging.conig.file=<logging.properties>
//
handlers= java.util.logging.ConsoleHandler
.level=INFO
 
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
 
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
 
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
//
javax.management.level=FINEST
javax.management.remote.level=FINER
 
  Abraço!
Awesome! Thanks Thiago!
If you don't like fancy GUI you can still use the good and old CLI to monitor your Liferay. Check it out how at http://www.wolfe.id.au/2011/10/16/monitoring-the-openjdk-from-the-cli/
Nice! Thanks Thiago!
Good and useful information. Thanks.
Great piece of work Thiago. Thanks.
nice
Hi. Thanks for that info.
I just want to ask where in server.xml specifically you insert the JmxRemoteLifecycleListener listener? There are many xml tags in server.xml and I'm not sure where to insert it.

Thanks for the help.

-Warner
Hey Warner,

Around the line 35, there is a list of listeners already declared.
Thanks Thiago. How about connecting using jconsole? I've tried jconsole but it didn't worked. But visualvm did.
Hi Thiago,

I am trying to set profiler for a remote server ,Now in our scenerio we have
My local windows machine for profiling --> cloud ( gateway firewall --->apache----> Tomcat) .

1) After doing all configuration it says "start a SSH connection to your server with two tunnel enabled,
you can use the command below
ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com"
where should I use this command ? Does in your scenerio your local machine is linux and you are using this command there
,or you are using this command at final linux server having tomcat.

2) "Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi " I hope this will be automatically open when we fire remote
or do you mean to add these lines somewhere

Regards
Manish
Unfortunatly I have problems implementing this solution.

Listener is set on the remote tomcat...

Tunneling to my remote server is also successfull...

But If I want to open the service in VisualVM i get following error:

"Cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi"

The catalina-jmx-remote.jar is placed on the remote tomcat and in my platform/lib folder of VisualVm.

There is no firewall set up.

Some tips would be very great!

regards
thank you very much...it works with JConsole using

jconsole service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi -J-Djava.util.logging.config.file=/usr/lib/jvm/jdk1.6.0_35/lib/logging.properties -J-Djava.class.path=/usr/lib/jvm/jdk1.6.0_35/lib/jconsole.jar:/usr/lib/jvm/jdk1.6.0_35ib/tools.jar:/usr/lib/jvm/jdk1.6.0_35/lib/catalina-jmx-remote.jar;
It's just awesome!!
Everything start to work from first try!)))
I register on this site just for say thanks to you.
Thank you man, i appreciate you very much!
Hi,
I am struggling with trying to connect to a remove liferay 6.2 instance the way you have provided.
I have followed step by step the guide but I am only getting a "cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi" . I tried jconsole too. It first tells me that it cannot connect securely, asks to fallback in a non secure connection and then fails.
I have tried adding the -Djava.rmi.server.hostname=127.0.0.1 parameter too but no luck.
On the tomcat logs I can clearly see that the ports are configured :
Nov 05, 2015 10:02:47 AM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServer
INFO: The JMX Remote Listener has configured the registry on port 10001 and the server on port 10002 for the Platform server

On my local machine I am opening a tunnel as follows :
ssh marco@<myserver> -L10001:127.0.0.1:10001 -L10002:127.0.0.1:10002 -vN

on the ssh logs I can see the connection going TO the remote server :
debug1: Connection to port 10001 forwarding to 127.0.0.1 port 10001 requested.
debug1: channel 5: new [direct-tcpip]

but I can't see the other port being connected.

Can you please help ?
Hey Marco, were you able to use jconsole with logging configured? It is the recommended way to debug problems.