Hi Alex,
Ok -- phew! this is a lot of code just to setup a trigger. I'm going to give you a little unsolicited advice if you don't mind -- mostly because I am not sure I entirely agree with your solution
First off, this causes me some concern --
The next thing I would say is that you don't need this code --
Now, for your settings. I think what you are trying to do is provide two definitions. One for a Simple Trigger and one for a CRON possibly? Actually it looks like two Simple Triggers but if you want one to run "daily" at a specific time, you're better to use a CRON Expression. My guess is that you want this becuase in DEV you want to run the task more frequently than you do in production.
First off, that is a great idea and I love the fact that you have the foresight to make it a configuration that can be changed at runtime. The downside here is two fold.
1. Changes you make (the way you have it now) won't Deactivate/Activate the component (automatically) so it won't be automatically picked up -- so if you have it set (in your DDL) to 5 minutes and then you change it to 10, it won't change unless you stop and start
2. It doesn't follow the best practices for configuring a component.
SOOOOOO! .. what i would suggest is this.
Start simple. Add two new settings to the portal-ext.properties and then in your Activate method, instead of reading a DDL setting you can use the PropsUtil class from kernel to get your setting (and the GetterUtil to convert it to the correct type). Then test with that.
If that works and you want to have a runtime setting, then you should use the Configuration API to create a configuration in the Control Panel. Then you can reference it using the configurationPid attribute on your @Component reference which will be a much simpler way (and more correct) way to handle the setting.
-- BUT, bottom line, I would start with the properties, or even something hard coded to make sure that it works ok, and then build from there.
Let me know if any of this doesn't make sense, or if you have more questions.
PS> One last thing, I noticed you are extending the BaseSchedulerEntryMessageListener class which is actually depracated. You should instead switch that to BaseMessageListener.
Ok -- phew! this is a lot of code just to setup a trigger. I'm going to give you a little unsolicited advice if you don't mind -- mostly because I am not sure I entirely agree with your solution
First off, this causes me some concern --
@Override
protected void doReceive(Message message) throws Exception {
try {
//Calling my implementation over here.
} catch (Exception e) {
e.printStackTrace();
}
registerScheduler();
}
The next thing I would say is that you don't need this code --
/**
* getStorageType: Utility method to get the storage type from the scheduler entry wrapper.
* @return StorageType The storage type to use.
*/
protected StorageType getStorageType() {
if (schedulerEntryImpl instanceof StorageTypeAware) {
return ((StorageTypeAware) schedulerEntryImpl).getStorageType();
}
return StorageType.MEMORY_CLUSTERED;
}
Now, for your settings. I think what you are trying to do is provide two definitions. One for a Simple Trigger and one for a CRON possibly? Actually it looks like two Simple Triggers but if you want one to run "daily" at a specific time, you're better to use a CRON Expression. My guess is that you want this becuase in DEV you want to run the task more frequently than you do in production.
First off, that is a great idea and I love the fact that you have the foresight to make it a configuration that can be changed at runtime. The downside here is two fold.
1. Changes you make (the way you have it now) won't Deactivate/Activate the component (automatically) so it won't be automatically picked up -- so if you have it set (in your DDL) to 5 minutes and then you change it to 10, it won't change unless you stop and start
2. It doesn't follow the best practices for configuring a component.
SOOOOOO! .. what i would suggest is this.
Start simple. Add two new settings to the portal-ext.properties and then in your Activate method, instead of reading a DDL setting you can use the PropsUtil class from kernel to get your setting (and the GetterUtil to convert it to the correct type). Then test with that.
If that works and you want to have a runtime setting, then you should use the Configuration API to create a configuration in the Control Panel. Then you can reference it using the configurationPid attribute on your @Component reference which will be a much simpler way (and more correct) way to handle the setting.
-- BUT, bottom line, I would start with the properties, or even something hard coded to make sure that it works ok, and then build from there.
Let me know if any of this doesn't make sense, or if you have more questions.
PS> One last thing, I noticed you are extending the BaseSchedulerEntryMessageListener class which is actually depracated. You should instead switch that to BaseMessageListener.
Hi Alex,
I understand -- certainly not the first time I have tried to provide advice that assumes an ideal scenario of full control over the environment and the time it takes to figure it out and build it right
Coming back to your actual problem, I did some more digging and I found the class ClusterSchedulerEngine seems to be the class that reports the error you are experiencing --
.. can you share a full stack trace so I can follow the rabbit down the hole?
Also, at the very least can you remove the call to setup the scheduler in the doReceive? just to make sure that it's not part of the issue? I say that because normally when you register a task with a simple trigger, when it is deployed it is immediately run so I am wondering if when you deloy, you register the trask, but since the listener runs right away it is trying to register the task right away (a second time). Even if that is not the case, you shouldn't need that code all the same.
I understand -- certainly not the first time I have tried to provide advice that assumes an ideal scenario of full control over the environment and the time it takes to figure it out and build it right
Coming back to your actual problem, I did some more digging and I found the class ClusterSchedulerEngine seems to be the class that reports the error you are experiencing --
...
try {
SchedulerResponse schedulerResponse = future.get(
_callMasterTimeout, TimeUnit.SECONDS);
if ((schedulerResponse == null) ||
(schedulerResponse.getTrigger() == null)) {
if (_log.isInfoEnabled()) {
_log.info(
StringBundler.concat(
"Memory clustered job ",
getFullName(jobName, groupName),
" is not yet deployed on master"));
}
}
else {
addMemoryClusteredJob(schedulerResponse);
}
}
...
.. can you share a full stack trace so I can follow the rabbit down the hole?
Also, at the very least can you remove the call to setup the scheduler in the doReceive? just to make sure that it's not part of the issue? I say that because normally when you register a task with a simple trigger, when it is deployed it is immediately run so I am wondering if when you deloy, you register the trask, but since the listener runs right away it is trying to register the task right away (a second time). Even if that is not the case, you shouldn't need that code all the same.
Hi Alex,
I don't think your test is valid. Let me try to explain once more --
1. Your servers are stopped.
2. Your code is NOT deployed on either Prod1 or Prod2
3. You start Prod1, and then you start Prod2
4. The first (Liferay) scheduled task fires on Prod1 -- Prod1 gets the Lock for scheduled tasks, and runs the task.
5. A second (Liferay) scheduled task fires on Prod1 -- Prod1 has the Lock, so the task runs.
6. The first (Liferay) scheduled task fires on Prod2 -- Prod2 tries but FAILS to get the Lock, so the task can't run.
.. this continues this way where all scheduled tasks (Liferay) will run only on Prod1.
7. You deploy your code on both nodes.
8. Your scheduled task runs on Prod1 -- the task executes because Prod1 has the Lock.
9. You scheduled task runs on Prod2 -- the task FAILS (as in it simply won't start in the first place) because Prod2 doesn't have the Lock
... so even with your services deployed, they will still only run on the node with the lock. This is nothing to do with load balancing, there is no request routing here as these threads spin up and run independent of a proxied request -- which is kind of the point of a scheduled task
10. You UNDEPLOY your scheduled task from Prod1.
11. Eventually, your scheduled task tried to run on Prod2 -- but again, Prod2 still doesn't have the lock do you can't run the task. So it just won't run on any node now
.. remember that your task is probably not the only scheduled task in the system so your task probably doesn't control WHEN and WHICH server get the lock. If you set your task to run at 7pm, but start your server earlier, then almost surely one of the Liferay tasks will fire first (like the JournalCheckInterval) and the server it runs on will obtain the lock.
12. You put your code back on BOTH servers - so that your cluster deployments are properly sync'ed now.
.. now HERE is the proper test to see if there is a problem.
13. Assuming that Prod1 has the Lock still, SHUT THE SERVER DOWN.
.. this will cause the server node to be removed from the cluster leaving just Prod2 in the pool. Now when the next scheduled task fires, Prod2 will be able to obtain the Lock and will now be the node running the tasks.
14. Start Prod1 back up
... and at this point you will be in the inverse scenario. Prod1, when it tries to run tasks, will no longer have or be able to obtain the Lock so the scheduled tasks will never run on prod1 and instead run on prod2.
Now, you might say "well that's no good! I want to blance the task execution amongst all my nodes!" -- and that's a fair point. But it's the limitation. The advantage though is the automatic failover to the other node when one goes down. The failover at least maintains business continuity. It does mean however a couple of things --
1. Only ONE node in your cluster will ever run the scheduled tasks (all of them) until that node goes down
2. You cannot (without a lot of effort) designate the node that will run scheduled tasks
3. You should make sure that your scheduled task has what it needs to run on ANY node in your cluster because of #2 and to support the failover scenario.
Make sense?
I don't think your test is valid. Let me try to explain once more --
1. Your servers are stopped.
2. Your code is NOT deployed on either Prod1 or Prod2
3. You start Prod1, and then you start Prod2
4. The first (Liferay) scheduled task fires on Prod1 -- Prod1 gets the Lock for scheduled tasks, and runs the task.
5. A second (Liferay) scheduled task fires on Prod1 -- Prod1 has the Lock, so the task runs.
6. The first (Liferay) scheduled task fires on Prod2 -- Prod2 tries but FAILS to get the Lock, so the task can't run.
.. this continues this way where all scheduled tasks (Liferay) will run only on Prod1.
7. You deploy your code on both nodes.
8. Your scheduled task runs on Prod1 -- the task executes because Prod1 has the Lock.
9. You scheduled task runs on Prod2 -- the task FAILS (as in it simply won't start in the first place) because Prod2 doesn't have the Lock
... so even with your services deployed, they will still only run on the node with the lock. This is nothing to do with load balancing, there is no request routing here as these threads spin up and run independent of a proxied request -- which is kind of the point of a scheduled task
10. You UNDEPLOY your scheduled task from Prod1.
11. Eventually, your scheduled task tried to run on Prod2 -- but again, Prod2 still doesn't have the lock do you can't run the task. So it just won't run on any node now
.. remember that your task is probably not the only scheduled task in the system so your task probably doesn't control WHEN and WHICH server get the lock. If you set your task to run at 7pm, but start your server earlier, then almost surely one of the Liferay tasks will fire first (like the JournalCheckInterval) and the server it runs on will obtain the lock.
12. You put your code back on BOTH servers - so that your cluster deployments are properly sync'ed now.
.. now HERE is the proper test to see if there is a problem.
13. Assuming that Prod1 has the Lock still, SHUT THE SERVER DOWN.
.. this will cause the server node to be removed from the cluster leaving just Prod2 in the pool. Now when the next scheduled task fires, Prod2 will be able to obtain the Lock and will now be the node running the tasks.
14. Start Prod1 back up
... and at this point you will be in the inverse scenario. Prod1, when it tries to run tasks, will no longer have or be able to obtain the Lock so the scheduled tasks will never run on prod1 and instead run on prod2.
Now, you might say "well that's no good! I want to blance the task execution amongst all my nodes!" -- and that's a fair point. But it's the limitation. The advantage though is the automatic failover to the other node when one goes down. The failover at least maintains business continuity. It does mean however a couple of things --
1. Only ONE node in your cluster will ever run the scheduled tasks (all of them) until that node goes down
2. You cannot (without a lot of effort) designate the node that will run scheduled tasks
3. You should make sure that your scheduled task has what it needs to run on ANY node in your cluster because of #2 and to support the failover scenario.
Make sense?
Andrew JardineHi Alex,
I don't think your test is valid. Let me try to explain once more --
1. Your servers are stopped.
2. Your code is NOT deployed on either Prod1 or Prod2
3. You start Prod1, and then you start Prod2
4. The first (Liferay) scheduled task fires on Prod1 -- Prod1 gets the Lock for scheduled tasks, and runs the task.
5. A second (Liferay) scheduled task fires on Prod1 -- Prod1 has the Lock, so the task runs.
6. The first (Liferay) scheduled task fires on Prod2 -- Prod2 tries but FAILS to get the Lock, so the task can't run.
.. this continues this way where all scheduled tasks (Liferay) will run only on Prod1.
7. You deploy your code on both nodes.
8. Your scheduled task runs on Prod1 -- the task executes because Prod1 has the Lock.
9. You scheduled task runs on Prod2 -- the task FAILS (as in it simply won't start in the first place) because Prod2 doesn't have the Lock
... so even with your services deployed, they will still only run on the node with the lock. This is nothing to do with load balancing, there is no request routing here as these threads spin up and run independent of a proxied request -- which is kind of the point of a scheduled task
10. You UNDEPLOY your scheduled task from Prod1.
11. Eventually, your scheduled task tried to run on Prod2 -- but again, Prod2 still doesn't have the lock do you can't run the task. So it just won't run on any node now
.. remember that your task is probably not the only scheduled task in the system so your task probably doesn't control WHEN and WHICH server get the lock. If you set your task to run at 7pm, but start your server earlier, then almost surely one of the Liferay tasks will fire first (like the JournalCheckInterval) and the server it runs on will obtain the lock.
12. You put your code back on BOTH servers - so that your cluster deployments are properly sync'ed now.
.. now HERE is the proper test to see if there is a problem.
13. Assuming that Prod1 has the Lock still, SHUT THE SERVER DOWN.
.. this will cause the server node to be removed from the cluster leaving just Prod2 in the pool. Now when the next scheduled task fires, Prod2 will be able to obtain the Lock and will now be the node running the tasks.
14. Start Prod1 back up
... and at this point you will be in the inverse scenario. Prod1, when it tries to run tasks, will no longer have or be able to obtain the Lock so the scheduled tasks will never run on prod1 and instead run on prod2.
Now, you might say "well that's no good! I want to blance the task execution amongst all my nodes!" -- and that's a fair point. But it's the limitation. The advantage though is the automatic failover to the other node when one goes down. The failover at least maintains business continuity. It does mean however a couple of things --
1. Only ONE node in your cluster will ever run the scheduled tasks (all of them) until that node goes down
2. You cannot (without a lot of effort) designate the node that will run scheduled tasks
3. You should make sure that your scheduled task has what it needs to run on ANY node in your cluster because of #2 and to support the failover scenario.
Make sense?
Totally. I'm gonna do this tomorrow.
About the cluster configuration that I posted yesterday, in your point of view is that ok or do I need something more?
Thanks a million.
Community
Company
Feedback