RE: Control panel users list disappears but was able to login.

Joey Wong, modified 7 Years ago. Junior Member Posts: 34 Join Date: 4/4/16 Recent Posts

I'm using 6.2 GA6 for more than a year. The portal lives on AWS EC2 connecting to its DB on RDS.

This couple of months, we've been noticing a strange behavior where the list of users disappears. Even the list of installed apps was gone.

Search yields no result. We know for a fact there are a few thousand users in the system, but there were just gone. Oddly enough, login authentication works. 

Suspecting JDBC connectivity with DB may have sever or staled but I can remotely connect to that same db via Workbench tool.

 

Our current remedy is restarting Tomcat but there are no traces of log or trail for us to troubleshoot.

Question.

1) Has anyone face similar issue before?

2) If yes,  what was the cause?

3) Is there a specific log I should enable to trace this issue?

Any clue will be helpful for me to find out and share the findings as I embark to gather more info.

 

Thanks.

thumbnail
David H Nebinger, modified 7 Years ago. Liferay Legend Posts: 14933 Join Date: 9/2/06 Recent Posts

This can point to issues with your index.  Are you using lucene or did you externalize to solr?

Normally reindexing the data in the system admin area of the control panel is all that is necessary to restore the listing.

thumbnail
Fernando Fernandez, modified 7 Years ago. Expert Posts: 401 Join Date: 8/22/07 Recent Posts
David H Nebinger:

This can point to issues with your index.  (...) Normally reindexing the data in the system admin area of the control panel is all that is necessary to restore the listing.

 

This problem is so frequent that I would suggest someone should add a small assertion after an empty result from the index, comparing with the row count from the DB and automatically start a reindex if necessary.

Just my $0.02  :-)

 

Fernando

 

 

 

 

thumbnail
David H Nebinger, modified 7 Years ago. Liferay Legend Posts: 14933 Join Date: 9/2/06 Recent Posts

The problem with that, Fernando, is that the system cannot determine if an off count is on purpose or not.

There can be numerous reasons why you might want to exclude an entry from the index (hide from search) or include additional entries (include search from a larger result set).

Joey Wong, modified 7 Years ago. Junior Member Posts: 34 Join Date: 4/4/16 Recent Posts

Am not very sure but likely is lucene. I will check to confirm.

How do I externalize to Solr?

What's the difference?

Oddly a restart of tomcat, resumes everything. 

Do you mean, the re-indexing happens and connectivity to data model is offline?

thumbnail
David H Nebinger, modified 7 Years ago. Liferay Legend Posts: 14933 Join Date: 9/2/06 Recent Posts

When using lucene, the index is stored locally in the data directory.  For you I'm guessing that's an S3 bucket maybe?

https://dev.liferay.com/en/discover/deployment/-/knowledge_base/6-2/liferay-clustering#clustering-search

Solr is usually resilient to individual node index corruption issues.

On restart, it is likely finding the index is not there and is reindexing automagically.

No, I mean that Liferay goes to the index for many things first, database second.  List of users is that way, so if the index is missing, corrupted or empty, the user list will be empty even though the DB is populated.

Joey Wong, modified 7 Years ago. Junior Member Posts: 34 Join Date: 4/4/16 Recent Posts

The Lucene data directory is located locally not S3.  (Re-indexing-lucene1.JPG)

Does that mean everytime this occurs I'll just need to invoke the re-indexing for both these to correct the issue? (Re-indexing-lucene.JPG)

 

Joey Wong, modified 7 Years ago. Junior Member Posts: 34 Join Date: 4/4/16 Recent Posts

This problem still persists.

No even just the user list, my custom portlets connection to database also distrupted.

I can login still but nothing displayed.

Is so frustrating. Hard to find a solution.

 

Update: It was not Lucene. I had to restart tomcat + apache. Then all back to normal. Suspect it has something to do with AJP but I maybe wrong.