Blogs

Overview

A problem we someetimes face with our Liferay Portal enterprise (EE) customers is configuring and resolving data sync issues across nodes in a cluster.

Thsi article provides some tips and configuration for tracing Liferay Portal data sync issues in a clustered environment.

NOTE: There is no "one size fits all" solution to resolving data sync issues.

Scenario

You are using Liferay Portal 6.2 EE and have installed the Liferay Ehcache Cluster EE plugin from the Liferay Marketplace.

You have configured the Liferay Portal clustering and caching portal properties, but are not seeing data replicated across nodes in the cluster.

In some environments, multicast simply does not work. Packets are lost. Gone, gone, gone. A workaround is to use point-to-point unicast to ensure packets are delivered to each node in the network.

Tips

Cluster Tracing Tips

• Review Liferay Portal messages on Liferay channel as mbeans via JMX
• Review ehcache cluster config
• Review cluster event send/receive messages in portal log(s). Refer to Configuration (below).
• Review cluster event send/receive messages using IEEE Cluster Utility.

Configuration

Portal Configuration

Enable cluster link and replication in portal properties

Update file "LIFERAY_HOME/portal-ext.properties" with the following portal properties.

Sample Configuration

e.g.

##
## Cluster Link
##

    #
    # Set this to true to enable the cluster link. This is required if you want
    # to cluster indexing and other features that depend on the cluster link.
    #
    #cluster.link.enabled=false
    cluster.link.enabled=true

    # https://www.liferay.com/group/customer/knowledge/kb/-/knowledge_base/article/37840259
    #ehcache.cluster.link.replication.enabled=false
    ehcache.cluster.link.replication.enabled=true


##
## Cluster Executor
##

    # Set this to true to enable the cluster executor debugging. This will
    # attach a debugging listener which will log every cluster event it
    #cluster.executor.debug.enabled=false
    cluster.executor.debug.enabled=true

Logging Configuration

The following logging configuration will enable logging of cache events and cache replication.

NOTE: Also define portal property "cluster.executor.debug.enabled=true".

Define some, or all, of the following logging categories to assist with monitoring cache events and cache replication

Logging Configuration via Liferay Portal Control Panel

• Sign into Liferay Portal as admin user.
• Navigate to "Control Panel > Server Administration > Log Levels".
• Add/update the folllowing logging entries.

Sample Configuration

e.g.

com.liferay.portal.cache.cluster = ALL

com.liferay.portal.cache.cluster.clusterlink = ALL

com.liferay.portal.cache.cluster.clusterlink.ClusterLinkPortalCacheClusterChannelFactory = ALL

com.liferay.portal.cache.cluster.clusterlink.messaging.ClusterLinkPortalCacheClusterListener = ALL

com.liferay.portal.cache.ehcache = ALL

com.liferay.portal.cluster = ALL

com.liferay.portal.cluster.DebuggingClusterEventListenerImpl = ALL

com.liferay.portal.deploy.hot.PluginPackageHotDeployListener = ALL

com.liferay.portal.kernel.cache = ALL

com.liferay.portal.kernel.cache.cluster = ALL

com.liferay.portal.kernel.cache.cluster.PortalCacheClusterLink = ALL

com.liferay.portal.kernel.cluster = ALL

com.liferay.portal.kernel.cluster.messaging = ALL

net.sf.ehcache = ALL

org.jgroups.protocols.pbcast = ALL

Logging Configuration via portal-log4j-ext.xml file

FILE: LIFERAY_HOME/tomcat/lib/ext/META-INF/portal-log4j-ext.xml

Sample Configuration

e.g.

<category name="com.liferay.portal.cache.cluster">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.cache.cluster.clusterlink">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.cache.cluster.clusterlink.ClusterLinkPortalCacheClusterChannelFactory">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.cache.cluster.clusterlink.messaging.ClusterLinkPortalCacheClusterListener">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.cache.ehcache">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.cluster">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.cluster.DebuggingClusterEventListenerImpl">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.deploy.hot.PluginPackageHotDeployListener">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.kernel.cache">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.kernel.cache.cluster">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.kernel.cache.cluster.PortalCacheClusterLink">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.kernel.cluster">
   <priority value="ALL" />
</category>

<category name="com.liferay.portal.kernel.cluster.messaging">
   <priority value="ALL" />
</category>

<category name="net.sf.ehcache">
   <priority value="ALL" />
</category>

<category name="org.jgroups.protocols.pbcast">
   <priority value="ALL" />
</category>

References

Liferay

• Liferay Portal 6.2 User Guide - Using Liferay Portal 6.2 - Liferay Clustering
• Liferay Marketplace - Ehcache Cluster EE
• Liferay Developer Network - Liferay Clustering
• Liferay Marketplace - Liferay Ehcache Cluster EE plugin
• Liferay Marketplace - IEEE Cluster Utility
• LPS-50401 - Add property "ehcache.cluster.link.replication.enabled" to portal properties

Misc

• randombugs - Liferay Support KB - How to install and configure a Liferay cluster
• freshblurbs - Explaining java.lang.OutOfMemoryError: PermGen space