How to monitor Liferay (Tomcat) remotely through firewalls using VisualVM

I'm on site these last days and one of my tasks was to execute some load testing and monitor a Liferay instance hosted in a network accessed only via SSH port. No other port except 22 and 80 was open. So after almost 2 days struggling with several different configurations I was able to monitor a remote instance of Liferay through JMX+RMI over SSH tunneling. I had to put together several pieces of information to get this working, let's see the steps.

I will split this post in two configuration sections for JMX server (Tomcat) and JMX client (VisualVM).

JMX server

First of all you need to drop in the catalina-jmx-remote.jar to TOMCAT_HOME/lib directory of Tomcat. This will add to the classpath the listener org.apache.catalina.mbeans.JmxRemoteLifecycleListener responsible for enable the JMX+RMI over SSH tunneling. More info about this listener can be found at JMX Remote Lifecycle Listener. This jar may be found under extras directory of download binary area of Tomcat. Only versions newer or equal to 6.0.24 has it available. Your second step is configure the listener on server.xml add the following snippet under the <Server> tag (around line 35).

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" useLocalPorts="true" />

The third and last step on the server side is set new parameters to JVM as following to enable JMX remote. Set them on the setenv.(sh|bat) of your Tomcat installation

JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

JMX client

Again your first step is drop in the jar catalina-jmx-remote.jar into the VISUALVM_HOME/platform/lib. This is a required step otherwise you gonna get a ClassNotFoundException for JmxRemoteLifecycleListener. After that you must start a SSH connection to your server with two tunnel enabled, you can use the command below

ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com

Then you can fire a VisualVM and add a reference to your remote server under the Remote section. Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi and voilà! We are done and ready to start the real work!

Other considerations

If you are trying to monitor a cluster of machines be aware that the property java.rmi.server.hostname=localhost will break the cache replication.

Maybe your are wondering about the reference of localhost in almost all commands instead of the server's names, right? Don't worry this is the right way beacuse we are using SSH tunnelling.

Some folks would like to work with jconsole. Server configuration are identical. On the client side you gonna fire jconsole with the following parameter -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar to be able to connect.

If you have connection problems I would suggest you to use jconsole to debug. You can use the follwoing command line and logging.properties file to enable debugging

jconsole -J-Djava.util.logging.config.file=logging.properties -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar

// logging.properties file to activate JMX traces.

// Use this file on the java command line:

// java -Djava.util.logging.conig.file=<logging.properties>

handlers= java.util.logging.ConsoleHandler

.level=INFO

java.util.logging.FileHandler.pattern = %h/java%u.log

java.util.logging.FileHandler.limit = 50000

java.util.logging.FileHandler.count = 1

java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter

java.util.logging.ConsoleHandler.level = FINEST

java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

// Use FINER or FINEST for javax.management.remote.level - FINEST is

// very verbose...

javax.management.level=FINEST

javax.management.remote.level=FINER

Abraço!

22 Comments

Please sign in to comment.

Awesome! Thanks Thiago!

Please sign in to reply.

If you don't like fancy GUI you can still use the good and old CLI to monitor your Liferay. Check it out how at http://www.wolfe.id.au/2011/10/16/monitoring-the-openjdk-from-the-cli/

Please sign in to reply.

Nice post baron!

Please sign in to reply.

Nice! Thanks Thiago!

Please sign in to reply.

Good and useful information. Thanks.

Please sign in to reply.

Great piece of work Thiago. Thanks.

Please sign in to reply.

Thaks Thiago. Nice post!

Please sign in to reply.

nice

Please sign in to reply.

Hi. Thanks for that info.
I just want to ask where in server.xml specifically you insert the JmxRemoteLifecycleListener listener? There are many xml tags in server.xml and I'm not sure where to insert it.

Thanks for the help.

-Warner

Please sign in to reply.

Hey Warner,

Around the line 35, there is a list of listeners already declared.

Please sign in to reply.

Thanks Thiago. How about connecting using jconsole? I've tried jconsole but it didn't worked. But visualvm did.

Please sign in to reply.

Have you tried the configuration at the end of my post to jconsole? It works fine!

Please sign in to reply.

Hi Thiago,

I am trying to set profiler for a remote server ,Now in our scenerio we have
My local windows machine for profiling --> cloud ( gateway firewall --->apache----> Tomcat) .

1) After doing all configuration it says "start a SSH connection to your server with two tunnel enabled,
you can use the command below
ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com"
where should I use this command ? Does in your scenerio your local machine is linux and you are using this command there
,or you are using this command at final linux server having tomcat.

2) "Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi " I hope this will be automatically open when we fire remote
or do you mean to add these lines somewhere

Regards
Manish

Please sign in to reply.

Thanks Thiago ,
I did it finally ..But I have a question ..This has connected to the server for monitoring purpose can we also profile this way cause my purpose is to do some memory profiling ..if yes how to enable it.

Thanks
manish

Please sign in to reply.

Nope! It's just to monitoring.

Please sign in to reply.

great post! thanks.

Please sign in to reply.

Unfortunatly I have problems implementing this solution.

Listener is set on the remote tomcat...

Tunneling to my remote server is also successfull...

But If I want to open the service in VisualVM i get following error:

"Cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi"

The catalina-jmx-remote.jar is placed on the remote tomcat and in my platform/lib folder of VisualVm.

There is no firewall set up.

Some tips would be very great!

regards

Please sign in to reply.

Hey Johann, try to use the jconsole, It is better to debug with it.

Please sign in to reply.

thank you very much...it works with JConsole using

jconsole service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi -J-Djava.util.logging.config.file=/usr/lib/jvm/jdk1.6.0_35/lib/logging.properties -J-Djava.class.path=/usr/lib/jvm/jdk1.6.0_35/lib/jconsole.jar:/usr/lib/jvm/jdk1.6.0_35ib/tools.jar:/usr/lib/jvm/jdk1.6.0_35/lib/catalina-jmx-remote.jar;

Please sign in to reply.

It's just awesome!!
Everything start to work from first try!)))
I register on this site just for say thanks to you.
Thank you man, i appreciate you very much!

Please sign in to reply.

Hi,
I am struggling with trying to connect to a remove liferay 6.2 instance the way you have provided.
I have followed step by step the guide but I am only getting a "cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi" . I tried jconsole too. It first tells me that it cannot connect securely, asks to fallback in a non secure connection and then fails.
I have tried adding the -Djava.rmi.server.hostname=127.0.0.1 parameter too but no luck.
On the tomcat logs I can clearly see that the ports are configured :
Nov 05, 2015 10:02:47 AM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServer
INFO: The JMX Remote Listener has configured the registry on port 10001 and the server on port 10002 for the Platform server

On my local machine I am opening a tunnel as follows :
ssh marco@<myserver> -L10001:127.0.0.1:10001 -L10002:127.0.0.1:10002 -vN

on the ssh logs I can see the connection going TO the remote server :
debug1: Connection to port 10001 forwarding to 127.0.0.1 port 10001 requested.
debug1: channel 5: new [direct-tcpip]

but I can't see the other port being connected.

Can you please help ?

Please sign in to reply.

Hey Marco, were you able to use jconsole with logging configured? It is the recommended way to debug problems.

Please sign in to reply.

Menu Display

Blogs

Category Filter

JMX server

JMX client

Other considerations

More Blog Entries

Radio Liferay Episode 6: Aaron Delani

Radio Liferay Episode 7: Julio Camarero