How to monitor Liferay (Tomcat) remotely through firewalls using VisualVM

Thiago Leão Moreira
Thiago Leão Moreira
A Minute Read

 

  I'm on site these last days and one of my tasks was to execute some load testing and monitor a Liferay instance hosted in a network accessed only via SSH port. No other port except 22 and 80 was open. So after almost 2 days struggling with several different configurations I was able to monitor a remote instance of Liferay through JMX+RMI over SSH tunneling. I had to put together several pieces of information to get this working, let's see the steps.
  I will split this post in two configuration sections for JMX server (Tomcat) and JMX client (VisualVM).
 

  JMX server

  First of all you need to drop in the catalina-jmx-remote.jar to TOMCAT_HOME/lib directory of Tomcat. This will add to the classpath the listener org.apache.catalina.mbeans.JmxRemoteLifecycleListener responsible for enable the JMX+RMI over SSH tunneling. More info about this listener can be found at JMX Remote Lifecycle Listener. This jar may be found under extras directory of download binary area of Tomcat. Only versions newer or equal to 6.0.24 has it available. Your second step is configure the listener on server.xml add the following snippet under the <Server> tag (around line 35).
  
    <Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" useLocalPorts="true" />
 
  The third and last step on the server side is set new parameters to JVM as following to enable JMX remote. Set them on the setenv.(sh|bat) of your Tomcat installation
 
    JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
 

  JMX client

  Again your first step is drop in the jar catalina-jmx-remote.jar into the VISUALVM_HOME/platform/lib. This is a required step otherwise you gonna get a ClassNotFoundException for JmxRemoteLifecycleListener. After that you must start a SSH connection to your server with two tunnel enabled, you can use the command below

 
  ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com
 
  Then you can fire a VisualVM and add a reference to your remote server under the Remote section. Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi and voilà! We are done and ready to start the real work!
 
  

  Other considerations

 
  If you are trying to monitor a cluster of machines be aware that the property  java.rmi.server.hostname=localhost will break the cache replication.
  Maybe your are wondering about the reference of localhost in almost all commands instead of the server's names, right? Don't worry this is the right way beacuse we are using SSH tunnelling.
  
  Some folks would like to work with jconsole. Server configuration are identical. On the client side you gonna fire jconsole with the following parameter -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar to be able to connect. 
  If you have connection problems I would suggest you to use jconsole to debug. You can use the follwoing command line and logging.properties file to enable debugging
 
jconsole -J-Djava.util.logging.config.file=logging.properties -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar 

         
 

         

          // logging.properties file to activate JMX traces.
         

         

          // Use this file on the java command line:
         

         

          //     java -Djava.util.logging.conig.file=<logging.properties>
         

         

          //
         

         

          handlers= java.util.logging.ConsoleHandler
         

         

          .level=INFO
         

         

           
         

         

          java.util.logging.FileHandler.pattern = %h/java%u.log
         

         

          java.util.logging.FileHandler.limit = 50000
         

         

          java.util.logging.FileHandler.count = 1
         

         

          java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
         

         

           
         

         

          java.util.logging.ConsoleHandler.level = FINEST
         

         

          java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
         

         

           
         

         

          // Use FINER or FINEST for javax.management.remote.level - FINEST is
         

         

          // very verbose...
         

         

          //
         

         

          javax.management.level=FINEST
         

         

          javax.management.remote.level=FINER
         

         

           
         

         

            Abraço!
         

        

       

      

     

     

      

       

        

         

          

           

            

             

             

              
              
               
 Page Comments 
                
               
 
               

                
                

                 

                  

                   
thumbnail
                   

                   

                    

                     

                      
Thiago Leão Moreira
                      

                      
14 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     If you don't like fancy GUI you can still use the good and old CLI to monitor your Liferay. Check it out how at http://www.wolfe.id.au/2011/10/16/monitoring-the-openjdk-from-the-cli/
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                 
                

                
                
                
                
                

                 

                  

                   

                      
                       
                      
                   

                   

                    

                     

                      
Warner Pinero
                      

                      
13 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     Hi. Thanks for that info.
I just want to ask where in server.xml specifically you insert the JmxRemoteLifecycleListener listener? There are many xml tags in server.xml and I'm not sure where to insert it.

Thanks for the help.

-Warner
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                 

                  

                   

                    
thumbnail
                    

                    

                     

                      

                       
Thiago Leão Moreira  
                          
                           
                          Warner Pinero
                         
 
                       

                       
13 Years Ago
                       

                      

                     

                    

                    

                    

                   

                   

                    
  
                    

                    

                     

                      Hey Warner,

Around the line 35, there is a list of listeners already declared.
                     

                    

                    

                     
 Please sign in to reply. 
                     

                     

                      

                       
 
                       

                       

                      

                     

                    

                   

                  

                  

                   

                    

                     

                        
                         
                        
                     

                     

                      

                      
 
                      

                     

                    

                   

                  

                  
                 

                

                

                 

                  

                   
thumbnail
                   

                   

                    

                     

                      
Manish Kumar Jaiswal
                      

                      
13 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     Hi Thiago,

I am trying to set profiler for a remote server ,Now in our scenerio we have 
My local windows machine for profiling --> cloud ( gateway firewall --->apache----> Tomcat) . 

 1) After doing all configuration it says "start a SSH connection to your server with two tunnel enabled, 
 you can use the command below 
ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 myuser@www.myliferayremoteserver.com" 
where should I use this command ? Does in your scenerio your local machine is linux and you are using this command there
 ,or you are using this command at final linux server having tomcat.
 
2) "Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi " I hope this will be automatically open when we fire remote
or do you mean to add these lines somewhere

Regards
Manish
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                 

                  

                   

                    
thumbnail
                    

                    

                     

                      

                       
Manish Kumar Jaiswal  
                          
                           
                          Manish Kumar Jaiswal
                         
 
                       

                       
13 Years Ago
                       

                      

                     

                    

                    

                    

                   

                   

                    
  
                    

                    

                     

                      Thanks Thiago , 
I did it finally ..But I have a question ..This has connected to the server for monitoring purpose can we also profile this way cause my purpose is to do some memory profiling ..if yes how to enable it.

Thanks
manish
                     

                    

                    

                     
 Please sign in to reply. 
                     

                     

                      

                       
 
                       

                       

                      

                     

                    

                   

                  

                  

                   

                    

                     

                        
                         
                        
                     

                     

                      

                      
 
                      

                     

                    

                   

                  

                  
                 

                

                

                 

                  

                   
thumbnail
                   

                   

                    

                     

                      
Johann Kneringer
                      

                      
12 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     Unfortunatly I have problems implementing this solution.

Listener is set on the remote tomcat...

Tunneling to my remote server is also successfull...

But If I want to open the service in VisualVM i get following error:

"Cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi"

The catalina-jmx-remote.jar is placed on the remote tomcat and in my platform/lib folder of VisualVm.

There is no firewall set up.

Some tips would be very great!

regards
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                 
                

                

                 

                  

                   
thumbnail
                   

                   

                    

                     

                      
Johann Kneringer
                      

                      
12 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     thank you very much...it works with JConsole using

jconsole service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi -J-Djava.util.logging.config.file=/usr/lib/jvm/jdk1.6.0_35/lib/logging.properties -J-Djava.class.path=/usr/lib/jvm/jdk1.6.0_35/lib/jconsole.jar:/usr/lib/jvm/jdk1.6.0_35ib/tools.jar:/usr/lib/jvm/jdk1.6.0_35/lib/catalina-jmx-remote.jar;
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                

                

                 

                  

                   

                      
                       
                      
                   

                   

                    

                     

                      
Ilya Ilya
                      

                      
10 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     It's just awesome!!
Everything start to work from first try!))) 
I register on this site just for say thanks to you.
Thank you man, i appreciate you very much!
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                

                

                 

                  

                   
thumbnail
                   

                   

                    

                     

                      
Marco Ferretti
                      

                      
10 Years Ago
                      

                     

                    

                   

                   

                   

                  

                  

                   
  
                   

                   

                    

                     Hi, 
I am struggling with trying to connect to a remove liferay 6.2 instance the way you have provided.
I have followed step by step the guide but I am only getting a "cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi" . I tried jconsole too. It first tells me that it cannot connect securely, asks to fallback in a non secure connection and then fails.
I have tried adding the -Djava.rmi.server.hostname=127.0.0.1 parameter too but no luck. 
On the tomcat logs I can clearly see that the ports are configured : 
Nov 05, 2015 10:02:47 AM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServer
INFO: The JMX Remote Listener has configured the registry on port 10001 and the server on port 10002 for the Platform server

On my local machine I am opening a tunnel as follows : 
ssh marco@<myserver> -L10001:127.0.0.1:10001 -L10002:127.0.0.1:10002 -vN

on the ssh logs I can see the connection going TO the remote server : 
debug1: Connection to port 10001 forwarding to 127.0.0.1 port 10001 requested.
debug1: channel 5: new [direct-tcpip]

but I can't see the other port being connected.

Can you please help ?
                    

                   

                   

                    
 Please sign in to reply. 
                    

                    

                     

                      
 
                      

                      

                     

                    

                   

                  

                 

                 

                  

                   

                    

                       
                        
                       
                    

                    

                     

                     
 
                     

                    

                   

                  

                 

                 

                  

                   

                    
thumbnail
                    

                    

                     

                      

                       
Thiago Leão Moreira  
                          
                           
                          Marco Ferretti
                         
 
                       

                       
10 Years Ago
                       

                      

                     

                    

                    

                    

                   

                   

                    
  
                    

                    

                     

                      Hey Marco, were you able to use jconsole with logging configured? It is the recommended way to debug problems.
                     

                    

                    

                     
 Please sign in to reply. 
                     

                     

                      

                       
 
                       

                       

                      

                     

                    

                   

                  

                  

                   

                    

                     

                        
                         
                        
                     

                     

                      

                      
 
                      

                     

                    

                   

                  

                 

                

               

              

             

            

           

          

         

        

       

      

     

     

      

       

        

         

          

           

           

          

         

        

       

      

     

     

      

       
Related Assets...

      

     

     

      

       

        No Results Found
       

      

     

     

      

       
More Blog Entries...