RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-05-08T13:20:57Z

Hello Guys,

I noticed that something good happened.
After a few restarts from each production server (I dont know the order of who was restarted first).
It comes back to the original state, starting the jobs by the main url.
If I try to deploy the job scheduler again, deleting the existed one. The problem happens again but for now I don't need to worry about this problem anymore.
Thanks a million for all your help.
God bless the Liferay's heros hehehe'

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-11T20:03:10Z

I don't see any issue with your cluster config, assuming the tcp.xml is configured correctly. Easiest way (at least that was taught to me and I still use) to make sure you cluster (replication) is working is to bring up the same page on both servers in two different browsers. Add a new portlet to the first page and then simply refresh the second page. If your cluster is working, then the change (to the page) will be replicated and you can see the results correctly on both servers.

NOTE though, that since your scheduled tasks don't actually do anything with clustering, clustering is not really required for this. All you need is more than one Liferay server all pointing to the same database.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-11T19:46:20Z

Andrew JardineHi Alex,

I don't think your test is valid. Let me try to explain once more --

1. Your servers are stopped.
2. Your code is NOT deployed on either Prod1 or Prod2
3. You start Prod1, and then you start Prod2
4. The first (Liferay) scheduled task fires on Prod1 -- Prod1 gets the Lock for scheduled tasks, and runs the task.
5. A second (Liferay) scheduled task fires on Prod1 -- Prod1 has the Lock, so the task runs.
6. The first (Liferay) scheduled task fires on Prod2 -- Prod2 tries but FAILS to get the Lock, so the task can't run.

.. this continues this way where all scheduled tasks (Liferay) will run only on Prod1.

7. You deploy your code on both nodes.
8. Your scheduled task runs on Prod1 -- the task executes because Prod1 has the Lock.
9. You scheduled task runs on Prod2 -- the task FAILS (as in it simply won't start in the first place) because Prod2 doesn't have the Lock

... so even with your services deployed, they will still only run on the node with the lock. This is nothing to do with load balancing, there is no request routing here as these threads spin up and run independent of a proxied request -- which is kind of the point of a scheduled task

10. You UNDEPLOY your scheduled task from Prod1.
11. Eventually, your scheduled task tried to run on Prod2 -- but again, Prod2 still doesn't have the lock do you can't run the task. So it just won't run on any node now

.. remember that your task is probably not the only scheduled task in the system so your task probably doesn't control WHEN and WHICH server get the lock. If you set your task to run at 7pm, but start your server earlier, then almost surely one of the Liferay tasks will fire first (like the JournalCheckInterval) and the server it runs on will obtain the lock.

12. You put your code back on BOTH servers - so that your cluster deployments are properly sync'ed now.

.. now HERE is the proper test to see if there is a problem.

13. Assuming that Prod1 has the Lock still, SHUT THE SERVER DOWN.

.. this will cause the server node to be removed from the cluster leaving just Prod2 in the pool. Now when the next scheduled task fires, Prod2 will be able to obtain the Lock and will now be the node running the tasks.

14. Start Prod1 back up

... and at this point you will be in the inverse scenario. Prod1, when it tries to run tasks, will no longer have or be able to obtain the Lock so the scheduled tasks will never run on prod1 and instead run on prod2.

Now, you might say "well that's no good! I want to blance the task execution amongst all my nodes!" -- and that's a fair point. But it's the limitation. The advantage though is the automatic failover to the other node when one goes down. The failover at least maintains business continuity. It does mean however a couple of things --

1. Only ONE node in your cluster will ever run the scheduled tasks (all of them) until that node goes down
2. You cannot (without a lot of effort) designate the node that will run scheduled tasks
3. You should make sure that your scheduled task has what it needs to run on ANY node in your cluster because of #2 and to support the failover scenario.

Make sense?

Totally. I'm gonna do this tomorrow.
About the cluster configuration that I posted yesterday, in your point of view is that ok or do I need something more?
Thanks a million.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-11T13:36:34Z

Hi Alex,

I don't think your test is valid. Let me try to explain once more --

1. Your servers are stopped.
2. Your code is NOT deployed on either Prod1 or Prod2
3. You start Prod1, and then you start Prod2
4. The first (Liferay) scheduled task fires on Prod1 -- Prod1 gets the Lock for scheduled tasks, and runs the task.
5. A second (Liferay) scheduled task fires on Prod1 -- Prod1 has the Lock, so the task runs.
6. The first (Liferay) scheduled task fires on Prod2 -- Prod2 tries but FAILS to get the Lock, so the task can't run.

.. this continues this way where all scheduled tasks (Liferay) will run only on Prod1.

7. You deploy your code on both nodes.
8. Your scheduled task runs on Prod1 -- the task executes because Prod1 has the Lock.
9. You scheduled task runs on Prod2 -- the task FAILS (as in it simply won't start in the first place) because Prod2 doesn't have the Lock

... so even with your services deployed, they will still only run on the node with the lock. This is nothing to do with load balancing, there is no request routing here as these threads spin up and run independent of a proxied request -- which is kind of the point of a scheduled task

10. You UNDEPLOY your scheduled task from Prod1.
11. Eventually, your scheduled task tried to run on Prod2 -- but again, Prod2 still doesn't have the lock do you can't run the task. So it just won't run on any node now

.. remember that your task is probably not the only scheduled task in the system so your task probably doesn't control WHEN and WHICH server get the lock. If you set your task to run at 7pm, but start your server earlier, then almost surely one of the Liferay tasks will fire first (like the JournalCheckInterval) and the server it runs on will obtain the lock.

12. You put your code back on BOTH servers - so that your cluster deployments are properly sync'ed now.

.. now HERE is the proper test to see if there is a problem.

13. Assuming that Prod1 has the Lock still, SHUT THE SERVER DOWN.

.. this will cause the server node to be removed from the cluster leaving just Prod2 in the pool. Now when the next scheduled task fires, Prod2 will be able to obtain the Lock and will now be the node running the tasks.

14. Start Prod1 back up

... and at this point you will be in the inverse scenario. Prod1, when it tries to run tasks, will no longer have or be able to obtain the Lock so the scheduled tasks will never run on prod1 and instead run on prod2.

Now, you might say "well that's no good! I want to blance the task execution amongst all my nodes!" -- and that's a fair point. But it's the limitation. The advantage though is the automatic failover to the other node when one goes down. The failover at least maintains business continuity. It does mean however a couple of things --

1. Only ONE node in your cluster will ever run the scheduled tasks (all of them) until that node goes down
2. You cannot (without a lot of effort) designate the node that will run scheduled tasks
3. You should make sure that your scheduled task has what it needs to run on ANY node in your cluster because of #2 and to support the failover scenario.

Make sense?

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-10T19:34:48Z

I'm gonna test everything in the end of the day.
Just fixing what I've said before: The job works fine on the main node (prd1). The second node (prd2) is the real problem.
The last test, I deleted the jobs of my osgi/modules from prd1 and I kept them on prd2.
Result: Doens't work.
So, I did the opposite: delete from prd2 and kept them on prd1.
Result: Worked fine using a specific ip server url.

But talking about cluster configuration, I have this on my portal-ext.properties:
This config is on both servers.

##
## Cluster Link
##

&nbsp; &nbsp; #
&nbsp; &nbsp; # Set the cluster node bootup response timeout in milliseconds.
&nbsp; &nbsp; #
&nbsp; &nbsp; cluster.link.node.bootup.response.timeout=10000

&nbsp; &nbsp; #
&nbsp; &nbsp; # Set this to true to enable the cluster link. This is required if you want
&nbsp; &nbsp; # to cluster indexing and other features that depend on the cluster link.
&nbsp; &nbsp; #
&nbsp; &nbsp; cluster.link.enabled=true
&nbsp;&nbsp; &nbsp;
&nbsp; &nbsp; #
&nbsp; &nbsp; # Set the JGroups properties for each channel, we support up to 10 transport
&nbsp; &nbsp; # channels and 1 single required control channel. Use as few transport
&nbsp; &nbsp; # channels as possible for best performance. By default, only one UDP
&nbsp; &nbsp; # control channel and one UDP transport channel are enabled. Channels can be
&nbsp; &nbsp; # configured by XML files that are located in the class path or by inline
&nbsp; &nbsp; # properties.
&nbsp; &nbsp; #
&nbsp; &nbsp; cluster.link.channel.properties.control=tcp.xml
&nbsp; &nbsp; cluster.link.channel.properties.transport.0=tcp.xml
&nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; # Set this property to autodetect the default outgoing IP address so that
&nbsp; &nbsp; # JGroups can bind to it. The property must point to an address that is
&nbsp; &nbsp; # accessible to the portal server, www.google.com, or your local gateway.
&nbsp; &nbsp; #cluster.link.autodetect.address=www.google.com:80

Is this config ok or do I need to implement something more?

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-10T07:37:38Z

Alex Camaroti
I referenced the primary node just because the job was deployed on both nodes, and it doenst work in one of them. before restart it was working fine.

Here lies your actual problem! Except for some really special scenarios, you should not care, on which node the job is executed. It simply should not matter. If one machine is down, the other one should run the job.

Alex Camaroti

The closest you could get to making sure Node 1 and not Node 2 runs the task would be to restart your cluster, 
but only bring up one node for a time until you are sure that at least one task has been run and that Node 1 (your primary node) ran it. 
Then you could start Node 2.

I think that I'm going to try this one first.

Please note that this is highly unreliable. One restart in the wrong order -> Problem.

May I suggest a few other options:
-) Create an extra service just for the job and deploy it only on one server
-) Add a portal-ext property "enable_my_service". Set it true on server one, false on server two and just don't run the job on server two.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-10T07:29:55Z

Alex Camaroti
I also made it. It helps a lot but nobody wants to wake up at midnight everyday to start the job manually hehe'
Also, something strange happened. I made it for two jobs, one of them works fine creating the content requested, but the another one just write the log information but doens't create new contents. I decided just to focus on the main problem, If I solve the job scheduler not running on the main node, the rest service triggering the job is not necessary anymore.

I am not sure if your main problem is actually the job scheduler. You have the service deployed on both nodes, but it behaves correctly only on one of them? I'd say, there is something fishy going on.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T20:06:42Z

Andrew JardineHey Alex,

You keep referring to the job running on the main node -- I just want to make sure that you understand that you can't control what node the job will run on. It's basically a race. Whichever node in your cluster runs a task first will get the lock. And to be clear, it doens't have to be YOUR task that runs first. Liferay has several scheduled tasks that run as well -- for example the JournalCheckInterval that runs (by default) every 15 minutes to see if there is content that should be published/unpublished.

The closest you could get to making sure Node 1 and not Node 2 runs the task would be to restart your cluster, but only bring up one node for a time until you are sure that at least one task has been run and that Node 1 (your primary node) ran it. Then you could start Node 2.

But I would say that the design of this solution (from Liferay) is such that it expects the job to be able to run on any node -- hence the fail over we talked about yesterday.

You keep referencing the Primary Node. Does it HAVE to run on the primary node? and if yes, why?

Hmm, I learned the idea of load-balance some days ago and I think that this is what you are talking about.
I referenced the primary node just because the job was deployed on both nodes, and it doenst work in one of them. before restart it was working fine.

The closest you could get to making sure Node 1 and not Node 2 runs the task would be to restart your cluster, 
but only bring up one node for a time until you are sure that at least one task has been run and that Node 1 (your primary node) ran it. 
Then you could start Node 2.

I think that I'm going to try this one first.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T19:11:43Z

Hey Alex,

You keep referring to the job running on the main node -- I just want to make sure that you understand that you can't control what node the job will run on. It's basically a race. Whichever node in your cluster runs a task first will get the lock. And to be clear, it doens't have to be YOUR task that runs first. Liferay has several scheduled tasks that run as well -- for example the JournalCheckInterval that runs (by default) every 15 minutes to see if there is content that should be published/unpublished.

The closest you could get to making sure Node 1 and not Node 2 runs the task would be to restart your cluster, but only bring up one node for a time until you are sure that at least one task has been run and that Node 1 (your primary node) ran it. Then you could start Node 2.

But I would say that the design of this solution (from Liferay) is such that it expects the job to be able to run on any node -- hence the fail over we talked about yesterday.

You keep referencing the Primary Node. Does it HAVE to run on the primary node? and if yes, why?

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T19:01:22Z

Christoph RabelNot sure, if it helps in any way, but maybe you can use the idea somehow:
I had a related problem, that I needed to control from the outside, when a job was run. (Usually scheduled, but sometimes it needed to be executed immediately).

I created a rest service, that triggers the execution of the job. A cronjob starts the job every night but it can also be manually triggered and triggered by external services.

I also made it. It helps a lot but nobody wants to wake up at midnight everyday to start the job manually hehe'
Also, something strange happened. I made it for two jobs, one of them works fine creating the content requested, but the another one just write the log information but doens't create new contents. I decided just to focus on the main problem, If I solve the job scheduler not running on the main node, the rest service triggering the job is not necessary anymore.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T18:50:57Z

Hahaha -- hey, sometimes the best solution to the problem is to appease the Gods right?

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T17:31:01Z

Not sure, if it helps in any way, but maybe you can use the idea somehow:
I had a related problem, that I needed to control from the outside, when a job was run. (Usually scheduled, but sometimes it needed to be executed immediately).

I created a rest service, that triggers the execution of the job. A cronjob starts the job every night but it can also be manually triggered and triggered by external services.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T17:19:53Z

Christoph RabelI didn't really follow the thread, but I had a superweird problem with the scheduler once too. After a few hours, it simply stopped working till I restarted the server. I really fiddled a while with the problem. Then I deleted the osgi/state folder, restarted and the problem was gone. I know, it's a wild guess and kinda a "sacrifice a lamb and dance around the fire" solution, but deleting the state folder resolved weird issues for me a couple of times now.

It's also a great tip Christoph.
I'll keep this option as a resource if I take too long to find a solution. I want to run the scheduled job from my primary node (prd1), instead of every time I access the second node (prd2) by ip.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T17:16:18Z

Andrew JardineWell, it's not really a hole. The Quartz Scheduler is built so that it only runs on one node. So if you have a cluster of 15 nodes, even if you deploy the module to all 15 nodes (which you should) it will only ever run on one node. My point is that if you have that setup, 15 nodes all with your module, and you stop/start the module on a node that isn't the one with the quartz lock, then perhaps that is why you see that message -- but if the message is reported on a node that is NOT controlling the lock, then maybe you can ignore it. So that is why I was strying to figure out if the error shows up on ALL your nodes? or on your Number of Nodes - 1 ... if it is the Number of Nodes - 1 then I would suspect that you are fine.

... does the job still run in the end? on Any of the nodes?

You are right, it happens just on one node that is the main node (prd1). The second node (prd2) that I need to access via ip the job runs.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T17:00:45Z

I didn't really follow the thread, but I had a superweird problem with the scheduler once too. After a few hours, it simply stopped working till I restarted the server. I really fiddled a while with the problem. Then I deleted the osgi/state folder, restarted and the problem was gone. I know, it's a wild guess and kinda a "sacrifice a lamb and dance around the fire" solution, but deleting the state folder resolved weird issues for me a couple of times now.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T16:52:23Z

Well, it's not really a hole. The Quartz Scheduler is built so that it only runs on one node. So if you have a cluster of 15 nodes, even if you deploy the module to all 15 nodes (which you should) it will only ever run on one node. My point is that if you have that setup, 15 nodes all with your module, and you stop/start the module on a node that isn't the one with the quartz lock, then perhaps that is why you see that message -- but if the message is reported on a node that is NOT controlling the lock, then maybe you can ignore it. So that is why I was strying to figure out if the error shows up on ALL your nodes? or on your Number of Nodes - 1 ... if it is the Number of Nodes - 1 then I would suspect that you are fine.

... does the job still run in the end? on Any of the nodes?

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T16:46:30Z

Yep, when I tried to run alone in a specific ip of prd2, it worked. but just accesing prd2. I saw that my job is deployed in both servers but I don't remember the order right now.

When I runned the job on prd2 it also gaves me some information on prd2:2019-04-04 07:55:05.025 INFO [default-6194][ClusterSchedulerEngine:745] Receive notification from master, add memory clustered job {groupName=com.admin.cronjob.job.CronJob, jobName=com.admin.cronjob.job.CronJob, storageType=MEMORY_CLUSTERED}.

Ok I am wondering now if maybe you are trying to initialize the service on the node that isn't the one managing the jobs. For example, let's say your Prod2 is the one running the tasks, but you are in the Control Panel on Prod1.

We deployed the same application on both servers, but both nodes should be sincronized, communicating with each other, right? As I'm a beginner entering in the middle of the project, I don't know how exactly is this communication between servers.
Hmm, I got your point. I just can not see a way out of this hole hehehe'

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T15:20:44Z

Ok I am wondering now if maybe you are trying to initialize the service on the node that isn't the one managing the jobs. For example, let's say your Prod2 is the one running the tasks, but you are in the Control Panel on Prod1.

Is there a way for you to open two browers where one is on Prod1 and the other (incognito, or a different browser) is on Prod2? And then try the same seteps on each to see if the erro shows up in one log but not the other?

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-09T14:48:36Z

Sure.

To reply the case, I access:

1. Control Panel > Applications > Applications Management
2. I choose a specific job, and I stop/start with the time changed on my ddl.
Note: Remembering that the same code woks fine on development server and test server (because it has just one node). Just production server that has 2 nodes (prd1 and prd2) that is not working by some clustering configuration that maybe was lost.

On the main server it doens't give to me too much information:

2019-04-04 07:45:50.206 INFO [http-nio-8080-exec-257][CronJob:93] Deactivate : Thu Apr 04 07:45:50 BRT 
20192019-04-04 07:45:50.280 INFO [http-nio-8080-exec-257][BundleStartStopLogger:38] STOPPED com.admin.cronjob_1.0.0 [1168]
2019-04-04 07:45:58.061 INFO [http-nio-8080-exec-290][BundleStartStopLogger:35] STARTED com.admin.cronjob_1.0.0 [1168]
2019-04-04 07:45:58.072 INFO [http-nio-8080-exec-290][CronJob:63] Activate : Thu Apr 04 07:45:58 BRT 
20192019-04-04 07:45:58.074 INFO [http-nio-8080-exec-290][CronJob:178] Minutes : 1440
2019-04-04 07:45:58.074 INFO [http-nio-8080-exec-290][CronJob:79] The job is going to be executed&nbsp;on: Thu Apr 04 07:50:00 BRT 
20192019-04-04 07:45:58.090 INFO [http-nio-8080-exec-290][ClusterSchedulerEngine:358] Memory clustered job is not yet deployed on master
2019-04-04 07:45:58.095 INFO [http-nio-8080-exec-290][CronJob:68] CronJobINFO :Job Registered.

RE: Job doenst trigger. Msg: Liferay job node not deployed on master

2019-04-08T22:15:51Z

Hi Alex,

I understand -- certainly not the first time I have tried to provide advice that assumes an ideal scenario of full control over the environment and the time it takes to figure it out and build it right

Coming back to your actual problem, I did some more digging and I found the class ClusterSchedulerEngine seems to be the class that reports the error you are experiencing --

...

try {
   SchedulerResponse schedulerResponse = future.get(
      _callMasterTimeout, TimeUnit.SECONDS);

   if ((schedulerResponse == null) ||
      (schedulerResponse.getTrigger() == null)) {

      if (_log.isInfoEnabled()) {
         _log.info(
            StringBundler.concat(
               "Memory clustered job ",
               getFullName(jobName, groupName),
               " is not yet deployed on master"));
      }
   }
   else {
      addMemoryClusteredJob(schedulerResponse);
   }
}
...

.. can you share a full stack trace so I can follow the rabbit down the hole?

Also, at the very least can you remove the call to setup the scheduler in the doReceive? just to make sure that it's not part of the issue? I say that because normally when you register a task with a simple trigger, when it is deployed it is immediately run so I am wondering if when you deloy, you register the trask, but since the listener runs right away it is trying to register the task right away (a second time). Even if that is not the case, you shouldn't need that code all the same.