1. Home
  2. Commerce
  3. RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch

RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch

Dominik Ratzow, modified 6 Years ago.
Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
New Member Posts: 10 Join Date: 3/11/14 Recent Posts
Dear community,

I updated the Liferay Commerce 1.1.2 to 1.1.3. After that my Elasticsearch is not working anymore. I use the embedded one.
I deleted the /data/elasticsearch6 folder - still not working.

Please find attached my complete startup of the server. There are many many errors with elasticsearch.

I use the normale installation of DXP 7.1 SP1 and updated to fix pack 8. No other portlets or custom things are installed (only Commerce).


Thank you for any advice.

Best regards,
Dominik

Attachments:

Mark Clarke, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
Regular Member Posts: 107 Join Date: 3/3/12 Recent Posts
Hi Dominik,

I have tried upgrading in my test environment and also getting  errors about  hibernate mappings and elastic search. I tcould be that a reindex of the elastic search db would solve the problem with elastic search but I can't even log in at this stage. I suppose there is some upgrade script for the db that needs to be run.

I would suggest others hold off on trying the upgrade.
thumbnail
Ethan Bustad, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
Junior Member Posts: 26 Join Date: 8/23/12 Recent Posts
Hi Dominik (and Mark),

Sorry to hear about the issues you're experiencing. That's obviously not intended.

2 things:
1. Could you try shutting down your server, clearing out the osgi/state folder, and starting back up? This may fix your issue.
2. Could you let me know how you deployed 1.1.3? I.e. are you running off of the LPKG or building from source, and how did you actually deploy those files to the bundle?

You aren't the only ones experiencing these issues, but at the moment they seem to be related to the deployment process rather than the 1.1.3 codebase itself. If we can get your errors resolved, then hopefully we can document the steps that others will have to take to avoid problems like this.

Ethan
Mark Clarke, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
Regular Member Posts: 107 Join Date: 3/3/12 Recent Posts
Ethan Bustad1. Could you try shutting down your server, clearing out the osgi/state folder, and starting back up? This may fix your issue.
2. Could you let me know how you deployed 1.1.3? I.e. are you running off of the LPKG or building from source, and how did you actually deploy those files to the bundle?

Hi Ethan,

Thanks for the reply.

1, I have cleared the osgi state folder as suggested and restarted all looks good now thanks. I had to sigkill some prcesses to release the elasticsearch process after the first restart threw lots of errors. This is an embedded version of  elasticsearch used in the test environment.
​​​​​​​
2. I simply copied the Liferay Commerce 1.1.3.lpkg into the deploy folder. There was already a 1.1.2 installation.

If I wanted to avoid these issues and deploy to production what process should I follow?
thumbnail
Ethan Bustad, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
Junior Member Posts: 26 Join Date: 8/23/12 Recent Posts
Hi Mark,

Glad to hear things are resolved!

In the future, a safe bet is to always clear osgi/state during an upgrade (before startup, that is).

Another good step would be to remove the old LPKGs before startup, to ensure Liferay doesn't try to register the old version before deploying the new version (see my reply to Dominik for more details). So the deployment steps would be:
  1. Clear osgi/state
  2. Remove osgi/marketplace/*Commerce*
  3. Copy the new version to the deploy folder
If you're interested in more detail as to why this is the case:
  • When a file is placed in the deploy folder, it's picked up by Liferay once the AutoDeployListeners have started up (which, on portal startup, comes after the big LIFERAY is printed -- you should be able to see it picking up the files in the logs).
  • When AutoDeployListeners pick up files in the deploy folder, it moves them to the appropriate osgi/ subfolder:
    • Licenses are moved to osgi/modules
    • Jars are moved to osgi/modules
    • Wars are moved to osgi/war
    • LPKGs are moved to osgi/marketplace
  • When files are located in the appropriate osgi/ subfolder, they're picked up more quickly by portal. On startup, this comes before the big LIFERAY is printed (except for wars, which come later -- you should be able to enable a more sensitive log level to see these things getting picked up and registered)
  • So when 1.1.2 is in osgi/marketpace and 1.1.3 is in the deploy folder, 1.1.2 is registered before 1.1.3, which causes issues since clearing osgi/state is required for a clean upgrade.
  • As you might guess, you can work around this ordering issue by copying 1.1.3 directly to osgi/marketplace. If you do this, then there's no need to remove the 1.1.2 LPKGs, as Liferay will overwrite the old one automatically.

Let me know if you have any other questions on this emoticon

Ethan
Dominik Ratzow, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
New Member Posts: 10 Join Date: 3/11/14 Recent Posts
Hi Ethan, hi Mark,

the first time I installed 1.1.2 I got also errors with the Elasticsearch. A redindex solved this problems. But not at this time for Commerce 1.1.3.

If you check my uploaded catalina.out around line 54+55 there is something wrong with the bundles but I do not know why:

2019-03-14 09:37:39.174 INFO  [ModuleFramework-Static-Bundles-2][LPKGBundleTrackerCustomizer:491] Skipping installation of stax2-api with version 3.1.4 in Liferay Connector to Elasticsearch 6 - API because an identical bundle exists
2019-03-14 09:37:40.464 INFO  [Refresh Thread: Equinox Container: aa21ec56-e67c-4744-aac8-f0af1c450b90][LPKGBundleTrackerCustomizer:491] Skipping installation of stax2-api with version 3.1.4 in Liferay Connector to Elasticsearch 6 - API because an identical bundle exists

I think this resulted to the class not found errors:
2019-03-14 09:38:35.080 WARN  [liferay/search_writer/SYSTEM_ENGINE-2][ProxyMessageListener:88] java.lang.Exception: java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.common.xcontent.XContentBuilder
java.lang.Exception: java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.common.xcontent.XContentBuilder
    at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:91)
    at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
    at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
    at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
    at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
    at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.common.xcontent.XContentBuilder
    at org.elasticsearch.common.xcontent.json.JsonXContent.contentBuilder(JsonXContent.java:47)
    at org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:117)
    at org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder(XContentFactory.java:44)
    at com.liferay.portal.search.elasticsearch6.internal.document.DefaultElasticsearchDocumentFactory.getElasticsearchDocument(DefaultElasticsearchDocumentFactory.java:59)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.ElasticsearchBulkableDocumentRequestTranslator.translate(ElasticsearchBulkableDocumentRequestTranslator.java:110)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.ElasticsearchBulkableDocumentRequestTranslator.translate(ElasticsearchBulkableDocumentRequestTranslator.java:46)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.BulkDocumentRequestExecutorImpl.lambda$createBulkRequestBuilder$0(BulkDocumentRequestExecutorImpl.java:121)
    at com.liferay.portal.search.engine.adapter.document.IndexDocumentRequest.accept(IndexDocumentRequest.java:38)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.BulkDocumentRequestExecutorImpl.createBulkRequestBuilder(BulkDocumentRequestExecutorImpl.java:114)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.BulkDocumentRequestExecutorImpl.execute(BulkDocumentRequestExecutorImpl.java:53)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.ElasticsearchDocumentRequestExecutor.executeBulkDocumentRequest(ElasticsearchDocumentRequestExecutor.java:48)
    at com.liferay.portal.search.engine.adapter.document.BulkDocumentRequest.accept(BulkDocumentRequest.java:30)
    at com.liferay.portal.search.engine.adapter.document.BulkDocumentRequest.accept(BulkDocumentRequest.java:23)
    at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:62)
    at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.updateDocument(ElasticsearchIndexWriter.java:275)
    at jdk.internal.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
    ... 6 more

What I already did:

- Installing DXP 7.1 SP1 fix pack 8
- Installing Commerce 1.1.2 (copying LPKG to the deploy folder)
- Shutting down and clear osgi/state - tomcat/temp - tomcat/work
- Startup with errors
- reindex Elasticsearch  from the control panel- Everything works now!
- One week later the same process with Commerce 1.1.3
- Installing Commerce 1.1.3 (copying LPKG to the deploy folder)
- Shutting down and clear osgi/state - tomcat/temp - tomcat/work
- Startup with errors (NoClassDefFoundError, java.lang.NullPointerException...)
- I cannot click the reindex at the control panel because the same errors will appear in the log

So to be honest I do not think my actual problem is related to any index problems. I think there is something wrong with my bundles.
But I do not deploy any other portlets or custom things.

Liferay does not have this problem only in Commerce. Our whole portal has no problems with this error. No user or organizations are shown emoticon
Everything that requires Elasticsearch is broken.

-------------------------------
EDIT: I forgot to mentioned that I deleted the /data/elasticsearch6 folder also and let it recreate. Still not working
-------------------------------


Thank you and best regards,
Dominik

Best regards,
Dominik
Mark Clarke, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
Regular Member Posts: 107 Join Date: 3/3/12 Recent Posts
Hi Dominik,

I get the line about stax2-api in my log file too but I don't have any issues with commerce-1.1.3 now.
  
2019-03-15 09:24:33.769 INFO  [ModuleFramework-Static-Bundles-2][LPKGBundleTrackerCustomizer:479] Skipping installation of stax2-api with version 3.1.4 in Liferay CE Connector to Elasticsearch 6 - API because an identical bundle exists

I am not sure about your subsequent errors.

HTH
thumbnail
Ethan Bustad, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
Junior Member Posts: 26 Join Date: 8/23/12 Recent Posts
Hi Dominik,

The stax2-api conflict is a known issue and we'll be working on solving it shortly -- but it shouldn't be causing the other errors you're experiencing.

One other thing that would be worth a try is:
  1. Clear osgi/state
  2. Remove osgi/marketplace/*Commerce* (this gets rid of the Commerce 1.1.2 packages)
  3. Copy Commerce 1.1.3 LPKG to the deploy folder

I did some testing yesterday to try to narrow down the issue that we're seeing with the 1.1.2 -> 1.1.3 upgrade, and I think the issue is caused just by osgi/state. The problem is, if you skip step #2 above, then as Liferay starts up it registers 1.1.2 before deploying 1.1.3. Because of this, even though you cleared osgi/state, the state is repopulated with 1.1.2 before the 1.1.3 upgrade, so you still encounter the issue. And issues with the osgi framework sometimes sabotage the whole system, so I'm not surprised other parts of portal are broken :/.

If you're able to try this out, let me know whether it solves your issue! If it doesn't, I'd love to troubleshoot this more in depth.

Ethan
Amos Fong, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
New Member Posts: 10 Join Date: 3/11/14 Recent Posts
Hi Mark,

thank you for this information.
I submitted a ticket to Liferay.

I will keep this thread up to date if I learn more.

Best regards,
​​​​​​​Dominik
Dominik Ratzow, modified 6 Years ago.
RE: Liferay Commerce 1.1.3 not working? FrameworkEvent ERROR Elasticsearch
New Member Posts: 10 Join Date: 3/11/14 Recent Posts
Hi guys,

I find a way to reproduce it posted on my ticket ... the problem is the osgi/state folder itsself emoticon
​​​​​​​
Hi Jorge,

after a hard day of try to reproduce it I finally got it.
The problem for me is the osgi/state folder. After clearing it the problem were starting.

My way to reproduce it:

- Extract liferay-dxp-tomcat-7.1.10.1-sp1-20190110085705206.zip (no change to any file, no fixpack, HSQL Demoticon
- Create deploy folder
- Copy my licence file for commerce and dxp to the deploy folder
- Copy Commerce 1.1.3 to the deploy folder
- starting portal

The portal ask me to restart because of first usage of Commerce. 
So I restarted. Multiple times ... doing the basic configuration of the portal.
It works like a charm.

I tested many things to reproduce my error. Finally I moved the osgi/state folder to osgi/state-bak and restarted.

The error appeared! I shut down the portal and deleted the new osgi/state folder and moved my old osgi/state-bak to osgi/state back. Startup the portal and voila ... works again.

I did the same procedure with Commerce 1.1.2 and it is working even if I clear the state folder!


Best regards,
Dominik