Here is a quick tip to optimize the elasticsearch installation that you might be using with your Liferay Portal or DXP 7.x (7.0, 7.1 or 7.2) and Oracle JDK 8.

The optimization, if you are affected by this bug, will contribute to a lower CPU use and a lower latency in the requests coming to elasticsearch through Liferay Portal/DXP.

Root cause

If you’re using JDK 1.8 (Oracle Hotspot and maybe others in version 8), this bug might be affecting your elasticsearch’s performance.

This bug causes - if your are using CMS garbage collector, which is the GC set in elasticsearch - the memory in young generation to be much lower than expected by default (which it should be ⅓), making the old generation very big and the promotion from young generation to old very frequent and thus results in garbage collection operations heavier.

Solution

To solve this behaviour, you just need to set the ratio between young and old generations in the JVM Heap. To do that, edit jvm.settings and include the following flag:

-XX:NewRatio=2

Monitoring before and after the change

To check the effectiveness of the change before and after with the jstat command  (jstat -gccapacity <pid>). 

Before the change, the size of the young generation (in the output check the NGCMN column – Minimum new generation capacity), possibly will be around 300 MB (which it might be low)

After the change, it will be much bigger.

You can also monitor GC operations enabling the gc.log through the jvm.settings file and process it with tools such as GCViewer or GCEasy (this one it’s an online tool)

To enable gc.log file, you just need to include in jvm.settings:

-Xloggc:/opt/elasticsearch/logs/gc.log

-XX:+UseGCLogFileRotation

-XX:NumberOfGCLogFiles=32

-XX:GCLogFileSize=128M

Another way to check if you were affected is by checking your elasticsearch log files where you might see lines such as the following (showing that every second, much time is used for GC):

[gc][78417] overhead, spent [493ms] collecting in the last [1s]

[2020-09-29T04:13:46,570][INFO ][o.e.m.j.JvmGcMonitorService] [node1] [gc][78418] overhead, spent [492ms] collecting in the last [1s]

[2020-09-29T04:13:47,583][WARN ][o.e.m.j.JvmGcMonitorService] [node1] [gc][78419] overhead, spent [545ms] collecting in the last [1s]

[2020-09-29T04:13:48,599][INFO ][o.e.m.j.JvmGcMonitorService] [node1] [gc][78420] overhead, spent [507ms] collecting in the last [1s]

I hope this was useful to optimize the search and filter operations in your Liferay Portal or DXP.