1. Home
  2. Development
  3. RE: Schedulers stop triggering after some time

RE: Schedulers stop triggering after some time

Simone de Blasiis, modified 5 Years ago.
Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
Hello,
we currently have several schedulers on our Liferay 7.1.3 CE GA4 instance that we developed using this guide.
They've been working fine for months until a week ago when they started behaving in a strange way:  after a while that the server has been started, all the schedulers stop triggering.

All the schedulers have a StorageType.PERSISTED and as far as we understand there's no other configuration needed that could change how the schedulers work.

We tried reading the catalina.out but we can't find any error that could tell us what the problem might be, we always make our schedulers print something in the log and at some point the just stop printing and running without any error. 

Can somebody help us?
Thanks in advance
thumbnail
Andrew Jardine, modified 5 Years ago.
RE: Schedulers stop triggering after some time
Liferay Legend Posts: 2416 Join Date: 12/22/10 Recent Posts
Hi Simone,Is this server part of a cluster? I ask because scheduled jobs only run on one node in the cluster. Could it be that the server where it is "not running" simply is not the node in the cluster that has been elected to run the jobs?
Simone de Blasiis, modified 5 Years ago.
RE: Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
Hi Andrew,no the server is not part of a cluster and the schedulers have been working every single day without a problem since May.
thumbnail
Andrew Jardine, modified 5 Years ago.
RE: Schedulers stop triggering after some time
Liferay Legend Posts: 2416 Join Date: 12/22/10 Recent Posts
Hey Simone,That is weird indeed then. I have a couple more questions. 

All the schedulers have a StorageType.PERSISTED and as far as we understand there's no other configuration needed that could change how the schedulers work.


If you change them to be MEMORY, do they work the way you expect them to? 


We tried reading the catalina.out but we can't find any error that could tell us what the problem might be, we always make our schedulers print something in the log and at some point the just stop printing and running without any error. 

I wonder if maybe there is nothing in the logs because an exception is been silently gobbled up, or because the logging level is not set to the right level to reveal the message. I'm sure you are aware that, under the hood, Liferay is using Quartz .. have you tried increasing the logging level for quartz? or maybe try bumping the com.liferay.portal.kernel.scheduler level to DEBUG or TRACE to see if something finds it's way into the logs?
Simone de Blasiis, modified 5 Years ago.
RE: Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
If you change them to be MEMORY, do they work the way you expect them to?


Before they stopped working, all the schedulers were registered as MEMORY_CLUSTERED and the problem made us realize that it wasn't the correct way to configure them for our situation so we change it to PERSISTED. This didn't change anything as no matter how we registered them they might've worked for a day, a week or even just a few hours before stopping so we think that shouldn't be the culprit.


I wonder if maybe there is nothing in the logs because an exception is been silently gobbled up, or because the logging level is not set to the right level to reveal the message. I'm sure you are aware that, under the hood, Liferay is using Quartz .. have you tried increasing the logging level for quartz? or maybe try bumping the com.liferay.portal.kernel.scheduler level to DEBUG or TRACE to see if something finds it's way into the logs?


We thought of this but I can't seem to find a way on how to set a finer debug level of Quartz. Is it something to do when declaring every single scheduler or is it a global Liferay configuration (for instance in the portal.ext.properties)? I tried searching online but I couldn't find anything on the matter.

Thanks for trying to help us as we really don't know what to do
thumbnail
Andrew Jardine, modified 5 Years ago.
RE: Schedulers stop triggering after some time
Liferay Legend Posts: 2416 Join Date: 12/22/10 Recent Posts
We thought of this but I can't seem to find a way on how to set a finer debug level of Quartz. Is it something to do when declaring every single scheduler or is it a global Liferay configuration (for instance in the portal.ext.properties)? I tried searching online but I couldn't find anything on the matter.


If you mean you couldn't figure out how to change the logging level, then it's easy peasy. Log in as your administrator and go to Control Panel > Configuration > Server Administration. Across the top there is a tab called Log Levels. In there you can change the logging levels for existing categories (those found in the portal-log4j.properties) of you can, as I think you would need to do, Add Category. In the add category section you can specify org.quartz and then DEBUG, or the package I referenced above with DEBUG and that should do it. NOTE: That this change is NOT persisted across restarts of the server though so if you need to restart for whatever reason you will need to add them again -- UNLESS! you create a portal-log4j-ext.xml file with the settings you need. If you are not sure how to do that, then let me know. I actually posted a video on both of these topics at www.masteringliferay.com -- in case that helps. 
Simone de Blasiis, modified 5 Years ago.
RE: Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
Thanks for the really fast reply, I just changed Quartz's log level to debug.Now we'll have to wait for the schedulers to hang again and considering that yesterday night it happened two times in two hours I'd say that we won't have to wait for long.
thumbnail
Andrew Jardine, modified 5 Years ago.
RE: Schedulers stop triggering after some time
Liferay Legend Posts: 2416 Join Date: 12/22/10 Recent Posts
No problem. If something does turn up, or you figure out the issue, do me a favour and let me know. I'm curious as to what the problem might be -- plus good to know a solution if you find one, in case I (or anyone else who wanders along here) ever have the same issue in the future. 

I am also curious, you mentioned that the MEMORY case did't work for "your situation". When you get a chance, can you explain what your situation is and why it doesn't work for you?
Simone de Blasiis, modified 5 Years ago.
RE: Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
As soon as the schedulers hang or I manage to find a solution I'll update the thread for sure.
Our situation is that we currently have only one Liferay instance on a single server and that instance is running some mission critical schedulers that need to run in order for the portal that we developed to work as intended. To make it simple, it need to periodically download some data from an endpoint to keep our database updated and if one of those updates doesn't trigger it could break some portlets entirely and cripple others.  From what I understood this is the perfect case scenario to use PERSISTED as a StorageType for ours schedulers.
thumbnail
Andrew Jardine, modified 5 Years ago.
RE: Schedulers stop triggering after some time
Liferay Legend Posts: 2416 Join Date: 12/22/10 Recent Posts
Hi Simone,Yes, as far as I know PERSISTED would be a good option for you because it would mean that any missed jobs would be detected and then run when the server was back up and running. This would be especially critical is you were using a CRON trigger because with the MEMORY or MEMORY_CLUSTERED options, it wouldn't run again until the cron expression evaluated to true of course.

Still, very strange that it just isn't running. When it does run, how long does it take to pull the data down from the remote system? And how often does the job run? Is it possible that the jobs are overlapping in some way? 
Simone de Blasiis, modified 5 Years ago.
RE: Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
Sorry for the late reply but I was out of office.We have 7 different schedulers that take different times: some of them take a few seconds each and some of them might even take tens of minutes but we scheduled them in a way that they shouldn't overlap.

Two nights ago it happened again and this time we had the debug level for the logs but the issue it's even stranger:
when one of the schedulers is being executed it prints on the log that it started running but the other night, even though Quartz was correctly triggering and calling execute on the jobs, they weren't printing anything in the logs.
The respective portlets that these schedulers are part of were still correctly running so we really can't understand what is going on.
thumbnail
Andrew Jardine, modified 5 Years ago.
RE: Schedulers stop triggering after some time
Liferay Legend Posts: 2416 Join Date: 12/22/10 Recent Posts
Hi Simone,

Yes, that does sound very strange. But restarting the server, I think you said, always seems to set everything right once more, and the tasks start running again for a time? Is there any change you can share the code for one of these schedulers with us?
Simone de Blasiis, modified 5 Years ago.
RE: Schedulers stop triggering after some time
New Member Posts: 12 Join Date: 5/14/19 Recent Posts
Hi Andrew,
yes when we restart Tomcat the schedulers start working again for some time.

Yesterday night we observed an entirely new behaviour where the schedulers stopped for some time but then started back up on their own without the need for us to restart Tomcat or the server itself. To me this is even stranger than the schedulers stopping outright.

Sadly we can't share the entire code of the schedulers but they are identical to the ones in the guide that I linked in the OP apart from the fact that we call some of our functions that fetch some data, iterate on it and confront it with our database to see if there were any changes that we need to replicate on our database and send email notifications about. Here's an example of one of the schedulers.