Solving Performance Issue in Liferay 6.1 Staging

Issue

In Liferay 6.1, the staging site is extremely slow if versioning is enabled. It can take 30 seconds to 10 minutes to refresh a page on the staging site (depends on how many contents/portlets on your page).

Cause

If you enable hibernate show SQL by setting the log level to DEBUG for "org.hibernate.SQL" category in "Server Administration" or use JAVA profiler, you will find the following SQL is executed hundreds of times which slows down the staging site a lot:

The AOP class com.liferay.portal.service.impl.LayoutSetLocalServiceStagingAdvice has a pointcut on LayoutSetLocalService. The LayoutSetLocalService can be called hundreds of times in one page refresh. Each time the LayoutSetLocalService is getting called, the LayoutSetLocalServiceStagingAdvice.wrapLayoutSet() will be invoked.

If we look at the code: com.liferay.portal.service.impl.LayoutSetLocalServiceStagingAdvice.wrapLayoutSet()

It will first calls LayoutSetStagingHandler layoutSetStagingHandler = LayoutStagingUtil.getLayoutSetStagingHandler(layoutSet);

Check the code com.liferay.portal.staging.LayoutStagingImpl.getLayoutSetStagingHandler(LayoutSet layoutSet):

The above method will always return null because layoutSet is not a proxy class.

It will return null back to the LayoutSetLocalServiceStagingAdvice.wrapLayoutSet() method. Then the (LayoutSet)ProxyUtil.newProxyInstance( PACLClassLoaderUtil.getPortalClassLoader(), new Class[] {LayoutSet.class}, new LayoutSetStagingHandler(layoutSet)) will be to create a new proxy class to warp the LayoutSet.

Here comes important part: When creating a new LayoutSetStagingHandler(layoutSet), it will call the com.liferay.portal.service.persistence.LayoutSetBranchFinderImpl.findByMaster(long groupId, boolean privateLayout) method. But the LayoutSetBranchFinderImpl finder method does not have the Liferay finder cache enabled, so Liferay will query the database every time it is getting called.

So, each time LayoutSetLocalService is getting called, there will be at least one database query to the LayoutSetBranch table, which makes the staging site very slow.

Solution

The best solution in my mind is to change the way how Liferay handles the branching/versioning in Liferay staging or to reduce the call to LayoutSetLocalService class. But it requires a lot of work. I am still working on it and will send a pull request if I can finish it.

Another solution is to use ext plugin to overwrite the LayoutSetBranchFinderImpl class to add the Liferay finder cache and overwrite LayoutSetBranchPersistenceImpl class to clear the Liferay finder cache upon create/update/delete.

If you do not want to use ext environment, here is a workaround using hook plugin:

Override the com.liferay.portal.service.LayoutSetBranchLocalService in the hook plugin:

Implement the com.kzhang.liferay.portal.service.LayoutSetBranchLocalServiceWrapper:

The main idea is to leverage getLayoutSetBranches(groupId, privateLayout) method to get the master LayoutSetBranch instead of using the finder method, because the getLayoutSetBranches() method has Liferay entity cache enabled. But as you can see, it has a silly for loop in the code which is not cool.

After deployed the hook, the page loading time is reduced from 3 minutes to 2 seconds.

Or, what you can do is: you can use getLayoutSetBranch(groupId, privateLayout, "main-variation") instead of getLayoutSetBranches(groupId, privateLayout) to get better performance if you never change the name of the master branch (the default name of the master branch is "main-variation").

15
Blogs
Very nice work Kan!!!

How close are you to implementing a full solution? Can you send me the pull request? (https://github.com/rotty3000)
See http://issues.liferay.com/browse/LPS-35856

I referenced your blog post emoticon

See https://github.com/brianchandotcom/liferay-portal/pull/11487
Cool emoticon
The pull request is to add the cache layer which works well.
I was also working on to reduce the following call
--------
(LayoutSet)ProxyUtil.newProxyInstance(
PACLClassLoaderUtil.getPortalClassLoader(),
new Class[] {LayoutSet.class},
new LayoutSetStagingHandler(layoutSet));
--------
in LayoutSetLocalServiceStagingAdvice.wrapLayoutSet() because I don't like the fact the LayoutSetStagingHandler is being newed hundreds to thousands times in one single request.

The LayoutStagingUtil.getLayoutSetStagingHandler(layoutSet) acts as a cache layer which works well when updating the layout because the layoutSet is already wrapped. But it does not work when reading the layoutSet from database.

I was thinking to cache the LayoutSetStagingHandler to prevent it to be newed again and again. But I think it is not critical right now.

Same thing happens in LayoutLocalServiceStagingAdvice for layouts too.

One more thing, the LayoutSetLocalService.createLayoutSet(long layoutSetId) and the LayoutLocalService.createLayout(long plid) does not work because LayoutSetLocalServiceStagingAdvice/LayoutLocalServiceStagingAdvice has pointcut on it -- it tries to fetch the group object from the newly created Layout/LayoutSet (with groupId=0) which throws the NoSuchGroupException.
Hi Kan,
Do you think same issue exist for com.liferay.portal.service.LayoutLocalService ?

We have 2000 private pages and clicking Private pages tab in control panel or clicking select pages on publish window invokes following query 2000 times.

select layoutimpl0_.plid as plid14_, layoutimpl0_.uuid_ as uuid2_14_, layoutimpl0_.groupId as groupId14_, layoutimpl0_.companyId as companyId14_, layoutimpl0_.createDate as createDate14_, layoutimpl0_.modifiedDate as modified6_14_, layoutimpl0_.privateLayout as privateL7_14_, layoutimpl0_.layoutId as layoutId14_, layoutimpl0_.parentLayoutId as parentLa9_14_, layoutimpl0_.name as name14_, layoutimpl0_.title as title14_, layoutimpl0_.description as descrip12_14_, layoutimpl0_.keywords as keywords14_, layoutimpl0_.robots as robots14_, layoutimpl0_.type_ as type15_14_, layoutimpl0_.typeSettings as typeSet16_14_, layoutimpl0_.hidden_ as hidden17_14_, layoutimpl0_.friendlyURL as friendl18_14_, layoutimpl0_.iconImage as iconImage14_, layoutimpl0_.iconImageId as iconIma20_14_, layoutimpl0_.themeId as themeId14_, layoutimpl0_.colorSchemeId as colorSc22_14_, layoutimpl0_.wapThemeId as wapThemeId14_, layoutimpl0_.wapColorSchemeId as wapColo24_14_, layoutimpl0_.css as css14_, layoutimpl0_.priority as priority14_, layoutimpl0_.layoutPrototypeUuid as layoutP27_14_, layoutimpl0_.layoutPrototypeLinkEnabled as layoutP28_14_, layoutimpl0_.sourcePrototypeLayoutUuid as sourceP29_14_ from Layout layoutimpl0_ where (layoutimpl0_.groupId=? )AND(layoutimpl0_.privateLayout=? )AND(layoutimpl0_.parentLayoutId=? ) order by layoutimpl0_.parentLayoutId ASC , layoutimpl0_.priority ASC

Not sure why service class is not retrieving all layouts in one query and firing query for each ;ayout.
Hi Mahesh,

I double checked the source code and the layout has the cache enabled.

Regarding to your issues, I think there are two possible causes:
1. The max number of elements in cache is set too low in in your liferay ehcache xml. (By default the max number of elements in cache is 10000 for each entity ).
2. The cache is cleared upon the layout(page) update. If you changed anything on the page, the cache will be cleared. Liferay will invokes the query first time when you go to the "Manage pages". But why it has so many queries (2000 time in your case)? Here is the reason:
The "Manage pages" displays a tree view for the dropdown box for "copy to page" function and "link to page" page type. To generate the tree view, Liferay calls com.liferay.portal.util.LayoutLister.getLayoutView() to create the tree.
In LayoutLister.getLayoutView(), it has a recursive method called _createList(long parentLayoutId, int parentId, int depth) which iterates thru the each of the layout to get the child layouts of each layout. If you have 2000 pages, it will call _createList_createList() 2000 times and you will have 2000 queries in the database.
I am working to improve the above logic too, but recently I was overwhelmed by my work. Trying hard to find time...
Hi Kan,
Thanks for your explanation.
Slowness of Manage Page seems to be due to LayoutLister.getLayoutView() and now we noticed that even clicking page from site navigation is causing 2000 DB calls. These calls seems to be for LayoutRevision and noticed that multiple calls are made for same layoutId.
It looks like that LayoutRevision also need to be revisited emoticon

Regarding ehcache point, I see that 10000 is max no of entries then 2000 layout should be OK, isn't it ?
Hi Mahesh,

10000 is max no of entries should be OK for 2000 layouts. So it is not the cache issue.

Regarding the LayoutRevision concerns, it is caused by LayoutLocalServiceStagingAdvice.wrapReturnValue() - same reason as I mentioned in the blog post - but is for LayoutRevision insetad of LayoutSetBranch. But the LayoutRevision has the finder cache enabled, it only happens first time after you update a layout.
Is this issue solved in EE? This is a no-go for our production environment.
It would be really nice if the outstanding issues had tickets created with all these details attached and we can make sure that they get fixed and patches are created for EE.

@Kan, Mahnesh, can you guys do that, then link them here so we can work on them please?
Hi Ray,
It seems that LS has created ticket for this issue - http://issues.liferay.com/browse/LPS-36049.
Hi Mahesh,
I want to know if you were able to solve the "Manage page" slowness on 6.1.1 CE GA2
I'm working with almost 3000 public pages and, as you know, it turns very very slow when it comes to load/update layouts in the manage page section.
I've read the ticket (LPS-36049) and it says that the problem is solved in 6.2.0 CE M6 but I'm looking for the solution for the 6.1.1 CE version.
Do you know anything else about this problem and how to solve it in the latest stable community edition?
Thank you very much Mahesh, and of course, thanks a lot to Kan who wrote this blog entry.
Hi Matias,
We received patch from support for 6.1.20 EE version.
Thanks Mahesh for your response.
My co-worker found the solution last friday while I was out.
For those who want to know, we have replaced the LayoutLister class with an improvement made by Raymond Auge in http://issues.liferay.com/browse/LPS-36049 (the same issue I've mentioned before).
I think the next time I'll have to read more carefully.
Thanks Ray.
Kan, thank you for this post and all your other contributions. I really like the bootstrap-theme too. I am learning alot from you.

thanks again