Ask Questions and Find Answers

Initial cause was Connection refused to host: 10.203.23.6

Is 10.203.23.6 one of your cluster machines? It might be that they find each other through Multicast, but the connection after exploring the other machine is denied, e.g. by Firewall rules.

Just remember that Liferay clustering is limited to Cache and Search (unless you have configured an external search server -- which I believe you said above you have). 

So in your case when you make a change on Node A, it will alert the other nodes in the cluster about the change so that the caches stay in sync. Database changes (like adding new records) are not affected by the cluster replication. 

One simple test is this. Bring up two crowsers each on the same page. Add a portlet to browser A and then refresh browser B. If the replication is working correctly, you should see the portlet on Browser B as well as Browser A. 

​​​​​​​

Admin CAUCE
File portal-setup.propierties

...
liferay.home = / opt / liferay-ce-portal-7.1.0-ga1
...

I guess the filename is a typo, as it gets generated by Liferay as portal-setup-wizard.properties.
With regards to the directory name though: I didn't find it in the documentation (it might be missing there), but in the announcement: If you're really running GA1: Please upgrade to the latest GA. Clustering is available out-of-the-box from GA3 on. For prior versions, you'd need to deploy extra code to enable clustering (e.g. build the plugin yourself). I'd say it's not worth to build and ignore all of the fixes that were introduced since GA1.

You probably don't need to mess with that part but if you are dying to get under the hood -- In your portal.properties there is a configuration

##
## Ehcache
##

    #
    # Set the classpath to the location of the Ehcache config file for internal
    # caches. Edit the file specified in the property
    # "ehcache.multi-vm.config.location" to enable clustered cache.
    #
    # Env: LIFERAY_EHCACHE_PERIOD_MULTI_PERIOD_VM_PERIOD_CONFIG_PERIOD_LOCATION
    # Env: LIFERAY_EHCACHE_PERIOD_SINGLE_PERIOD_VM_PERIOD_CONFIG_PERIOD_LOCATION
    #
    ehcache.single.vm.config.location=/ehcache/liferay-single-vm.xml
    ehcache.multi.vm.config.location=/ehcache/liferay-multi-vm-clustered.xml

Here you can see the two files used for ehcache, one for single nodes, and the multi for cluster. Both of these files are in the portal-impl, but you can grab them out of the source to see their content. If you want to change them, or add your own containers, you can make the changes and then change the location based on the properties above. The few times I have had to do this I have placed the files in /webapps/ROOT/WEB-INF/classes/META-INF/ehcache and then update the properties to include a /META-INF at the front.

Most of the time though, I think the defaults that Liferay provides suffice.

Hi Admin Cauce,


You configuration Liferay side was not correctly setup, as Liferay is trying to connect to 127.0.0.1 machine:

2019-04-22 15:15:20.677 ERROR [Framework Event Dispatcher: Equinox Container: 887023c8-5deb-4b2c-9b45-ad16f32b264a][com_liferay_portal_search:97] FrameworkEvent ERROR

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{GOxE7E5DR4CeDBst6Re0yA}{localhost}{127.0.0.1:9300}]]

        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:349)

Try do the following:

1. Start Liferay
2. After starting Liferay go to Control Panel => Configuration => System Settings => Platform => Search => Elasticsearch and setup here:
   • Operation Mode  => REMOTE
   • Transport Addresses  => your elasticsearch hostname and port
3. Go to Control Panel => Configuration => Search
4. Execute a full reindex (click in Reindex all search indexes)
5. Check log file
6. If everything goes fine, try restarting

After doing all that configuration, if you want a *.config file, you can go to Control Panel => Configuration => System Settings => Platform => Search => Elasticsearch click in the "kebab menú" (three dots menu) and select "Export"

You will export current configuration to a file that can be deployed to osgi/config.

Hi Admin CAUCE,

Admin CAUCE
And on the other hand, I already got the Liferay service up correctly without the problems I had before. In the file catalina.out I do not see any message of ERROR. Now I just have to add the second node to the cluster. Is there a procedure to add a second LR node to the cluster? I mean, not the configuration in the portal.propierties file, but the way to add it as a cluster.

All configuration related to System Settings are stored in database (inside configuration_ table), so it is not neccessary to do additional tasks in case of adding a new node to cluster.

Note: in case you have already setup more than one Liferay node and you do any change to Elasticsearch configuration in Liferay, after doing the change in first node, you will have to reboot rest of the nodes.

Admin CAUCE
Now I have this other problem in one of the nodes when I try tu start,

​​​​​​​2019-04-24 09:10:51.025 ERROR [Start Level: Equinox Container: 7100e0e6-fb71-4e0f-9a56-d3d491cc3a6e][Cache:224] Unable to set localhost. This prevents creation of a GUID. Cause was: JRDLRC10: JRDLRC10: Nombre o servicio desconocido
java.net.UnknownHostException: JRDLRC10: JRDLRC10: Nombre o servicio desconocido
        at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
&nbsp; &nbsp; &nbsp; &nbsp; at net.sf.ehcache.Cache.<clinit>(Cache.java:222)
</clinit>

Did you google the error message (no pun intended ;) )? It is actually pretty good, and the first hit on stackoverflow even has to do with ehcache.

Maybe a hostname typo somewhere?

Admin CAUCE

1 - Currently, when I launch the Liferay service, I pick it up with the "root" user, since within the Liferay path, it is configured with owner and group to "root". Could it be changed for LR to lift the service with the user, for example, "tomcat" created manually for this service? If so, how should I do it?

No root. Period. Check this blog article series (there are linked articles on the next chapters)

2 - You could mount a service to stop and / or start the LR service using the systemctl, since to stop the service, I do it with a kill -9 <id process>, and to boot it sh [path_lr] / tomcat /bin/catalina.sh start, so that it is more comfortable for the administrator.

Don't do kill -9 unless you've absolutely exhausted all of your other options. Check the blog series linked above or just go with systemctl.

3 - I'm thinking about doing a process to deploy the war files that you want to deploy to all the nodes of the cluster simultaneously, for example, create a folder where files are deposited, and launch a script that copies those files in the path deploy of each of the nodes, so that both of them have the same version of the file deployed, and since it has the property of hot deploy, this is transparent and easier for the developer. It's a good idea?

If this is a good idea depends on your other deployment strategies. My main concern for deploying new software is always: Can I rebuild the same server if the current one fails hard? Just operating a single server without taking care of the occasional server outage and recovery: You're fine any way.
Otherwise, you should always automate the deployment. How exactly you do that largely depends on your other infrastructure and tools that you're using for deployment.

Notice that maintenance of a running server sometimes is also the undeployment of existing  components, not only the deployment of additional ones.

I know of people who'll never hotdeploy. Others do this for a limited amount of times before restarting the server. And yet again others will just hotdeploy as they like. Whatever you do: automate the (un)deployment as well as you can. Optimize for recovery, not for the individual update of the system.

Admin CAUCE

I was looking at documentation to look at the Liferay version update process. I understand that when I release a higher version than I have, version 7.1.2, and that it is stable, it is convenient to perform an update. My question is, what process must be followed to perform a correct version update in Liferay?

As I'm typically on DXP, rather than CE, I can't tell you from experience if you need to run the upgrade tool for minor upgrades as well, or just for major ones. Sorry, you'd need to find that out for yourself, or through somebody else chiming in to this thread.

For major upgrades, you definitely need the tool.

Hi Admin CAUCE and everyone else,

Good job on configuring the cluster. Now i may need some assistance.
Considering this is my very first time setting a cluster environment, i may have missed pivotal configuration(s) even after reading the documentation for 100th time at https://portal.liferay.dev/docs/7-1/deploy/-/knowledge_base/d/liferay-clustering .

Before going forward let me introduce you with my settings.
I am using Liferay CE liferay-portal-7.1.2-ga3(tomcat) on 2 linux machines fronted by a nginx on both server each.
Both pointing to a single database server. And also on top there is a load balancer.

My previous setup (being the smartass i was) , both liferay connected to the same database, and using dbstore for data files.
And i did a rsync for whichever directories that i think that mattered. Of course this setting failed almost miserably. 
The issue was (as Olaf had mentioned) change to server A doesn't get affected on server B, for at least not immediately.
It only does after almost an hour or so, or as soon as I clear the database cache on server B. I did some more tweaking and of course it didnt work.

So now im back at the documentation that I had failed to understand the first time around. Still I am not getting it right.
Sorry for being long winded, but here is the start of my question:

1. All nodes should point to the same database or database cluster. - DONE
2. Documents and Media repositories must have the same configuration and be accessible to all nodes of the cluster. - DONE DBSTORE
3. Search should be on a separate search server that is optionally clustered. - DON'T HAVE EXTRA SERVER FOR THIS
4. Cluster Link must be enabled so the cache replicates across all nodes of the cluster. - PROBABLY WHERE MY ISSUE IS
5. Applications must be auto-deployed to each node individually. - HAVEN'T REACH HERE YET

After setting 

cluster.link.enabled=true 

in portal-ext.properties, i started liferay on server A.
This is my error log:

....
INFO  [main][ModuleFrameworkImpl:1636] Starting dynamic bundles
INFO  [Start Level: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][JGroupsClusterChannelFactory:158] Autodetecting JGroups outgoing IP address and interface for www.google.com:80
INFO  [Start Level: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][JGroupsClusterChannelFactory:197] Setting JGroups outgoing IP address to 11.11.1.10 and interface to ens192
INFO  [Start Level: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][JGroupsReceiver:91] Accepted view [DEV1-15243|0] (1) [DEV1-15243]
INFO  [Start Level: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][JGroupsClusterChannel:105] Create a new JGroups channel {channelName: liferay-channel-control, localAddress: DEV1-15243, properties: UDP(discard_incompatible_packets=true;internal_thread_pool_min_threads=2;internal_thread_pool_keep_alive_time=30000;time_service_interval=500;thread_pool_max_threads=10;internal_thread_pool_queue_enabled=true;mcast_group_addr=239.255.0.1;ergonomics=true;enable_unicast_bundling=true;port_range=50;loopback_copy=false;thread_naming_pattern=cl;suppress_time_out_of_buffer_space=60000;internal_thread_pool_rejection_policy=discard;internal_thread_pool_enabled=true;stats=true;oob_thread_pool_enabled=true;oob_thread_pool_rejection_policy=discard;suppress_time_different_version_warnings=60000;mcast_send_buf_size=100000;id=21;thread_pool_rejection_policy=Discard;logical_addr_cache_max_size=2000;suppress_time_different_cluster_warnings=60000;loopback=true;timer_rejection_policy=abort;oob_thread_pool_min_threads=2;max_bundle_timeout=20;enable_diagnostics=true;mcast_recv_buf_size=500000;disable_loopback=false;internal_thread_pool_max_threads=4;external_port=0;oob_thread_pool_max_threads=10;log_discard_msgs=true;name=UDP;oob_thread_pool_keep_alive_time=30000;bind_addr=10.88.2.20;wheel_size=200;bundler_capacity=20000;log_discard_msgs_version=true;enable_batching=true;tick_time=50;timer_max_threads=4;ucast_send_buf_size=100000;thread_pool_queue_enabled=true;enable_bundling=true;ucast_recv_buf_size=64000;oob_thread_pool_queue_enabled=false;thread_pool_keep_alive_time=30000;bind_port=0;thread_pool_min_threads=2;ignore_dont_bundle=true;ip_ttl=8;bind_interface_str=;diagnostics_ttl=8;tos=8;loopback_separate_thread=true;logical_addr_cache_expiration=120000;oob_thread_pool_queue_max_size=500;diagnostics_addr=224.0.75.75;receive_on_all_interfaces=false;mcast_port=23301;internal_thread_pool_queue_max_size=500;timer_queue_max_size=500;thread_pool_queue_max_size=10000;max_bundle_size=64000;physical_addr_max_fetch_attempts=1;ip_mcast=true;timer_min_threads=2;thread_pool_enabled=true;bundler_type=transfer-queue;timer_keep_alive_time=5000;logical_addr_cache_reaper_interval=60000;timer_type=new3;diagnostics_port=7500;who_has_cache_timeout=2000):PING(async_discovery_use_separate_thread_per_request=false;ergonomics=true;stagger_timeout=0;force_sending_discovery_rsps=true;async_discovery=false;timeout=3000;always_send_physical_addr_with_discovery_request=true;max_members_in_discovery_request=500;send_cache_on_join=false;num_initial_srv_members=0;break_on_coord_rsp=true;stats=true;use_disk_cache=false;num_initial_members=10;name=PING;discovery_rsp_expiry_time=60000;id=6;return_entire_cache=false):MERGE3(check_interval=48000;stats=true;min_interval=10000;ergonomics=true;name=MERGE3;id=54;max_participants_in_merge=100;max_interval=30000):FD_SOCK(get_cache_timeout=1000;sock_conn_timeout=1000;client_bind_port=0;ergonomics=true;start_port=0;port_range=50;suspect_msg_interval=5000;num_tries=3;bind_interface_str=;stats=true;external_port=0;name=FD_SOCK;bind_addr=127.0.0.1;keep_alive=true;id=3):FD_ALL(use_time_service=true;stats=true;timeout_check_interval=2000;ergonomics=true;name=FD_ALL;interval=8000;id=29;timeout=40000;msg_counts_as_heartbeat=false):VERIFY_SUSPECT(num_msgs=1;use_mcast_rsps=false;bind_interface_str=;stats=true;ergonomics=true;name=VERIFY_SUSPECT;bind_addr=127.0.0.1;id=13;timeout=1500;use_icmp=false):NAKACK2(resend_last_seqno_max_times=3;use_mcast_xmit=false;ergonomics=true;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=30000;become_server_queue_size=50;xmit_interval=500;print_stability_history_on_failed_xmit=false;resend_last_seqno=true;max_xmit_req_size=511600;discard_delivered_msgs=true;suppress_time_non_member_warnings=60000;max_msg_batch_size=500;xmit_table_num_rows=100;stats=true;xmit_from_random_member=false;log_discard_msgs=true;log_not_found_msgs=true;xmit_table_resize_factor=1.2;name=NAKACK2;id=57;max_rebroadcast_timeout=2000;use_mcast_xmit_req=false):UNICAST3(ergonomics=true;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=60000;ack_threshold=5;sync_min_interval=2000;max_retransmit_time=60000;xmit_interval=500;max_xmit_req_size=511600;conn_close_timeout=10000;max_msg_batch_size=500;conn_expiry_timeout=0;ack_batches_immediately=true;xmit_table_num_rows=100;stats=true;xmit_table_resize_factor=1.2;log_not_found_msgs=true;name=UNICAST3;id=64):STABLE(cap=0.1;stability_delay=0;stats=true;ergonomics=true;name=STABLE;desired_avg_gossip=50000;max_bytes=4000000;id=16;send_stable_msgs_to_coord_only=true):GMS(max_join_attempts=10;print_local_addr=true;handle_concurrent_startup=true;view_bundling=true;leave_timeout=1000;log_view_warnings=true;install_view_locally_first=false;ergonomics=true;use_delta_views=true;resume_task_timeout=20000;use_flush_if_present=true;use_merger2=true;print_physical_addrs=true;join_timeout=2000;view_ack_collection_timeout=2000;stats=true;num_prev_views=10;merge_timeout=5000;max_bundling_time=50;name=GMS;num_prev_mbrs=50;id=14;log_collect_msgs=false;membership_change_policy=org.jgroups.protocols.pbcast.GMS$DefaultMembershipPolicy@2ec4eccd):UFC(min_threshold=0.4;stats=true;ergonomics=true;max_credits=2000000;name=UFC;min_credits=800000;id=45;max_block_time=5000;ignore_synchronous_response=false):MFC(min_threshold=0.4;stats=true;ergonomics=true;max_credits=2000000;name=MFC;min_credits=800000;id=44;max_block_time=5000;ignore_synchronous_response=false):FRAG2(frag_size=60000;stats=true;ergonomics=true;name=FRAG2;id=5):RSVP(ack_on_delivery=true;stats=true;ergonomics=true;name=RSVP;resend_interval=2000;id=55;throw_exception_on_timeout=true;timeout=10000)}
INFO  [Start Level: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][JGroupsReceiver:91] Accepted view [DEV1-34758|0] (1) [DEV1-34758]
INFO  [Start Level: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][JGroupsClusterChannel:105] Create a new JGroups channel {channelName: liferay-channel-transport-0, localAddress: DEV1-34758, properties: UDP(discard_incompatible_packets=true;internal_thread_pool_min_threads=2;internal_thread_pool_keep_alive_time=30000;time_service_interval=500;thread_pool_max_threads=10;internal_thread_pool_queue_enabled=true;mcast_group_addr=239.255.0.2;ergonomics=true;enable_unicast_bundling=true;port_range=50;loopback_copy=false;thread_naming_pattern=cl;suppress_time_out_of_buffer_space=60000;internal_thread_pool_rejection_policy=discard;internal_thread_pool_enabled=true;stats=true;oob_thread_pool_enabled=true;oob_thread_pool_rejection_policy=discard;suppress_time_different_version_warnings=60000;mcast_send_buf_size=100000;id=21;thread_pool_rejection_policy=Discard;logical_addr_cache_max_size=2000;suppress_time_different_cluster_warnings=60000;loopback=true;timer_rejection_policy=abort;oob_thread_pool_min_threads=2;max_bundle_timeout=20;enable_diagnostics=true;mcast_recv_buf_size=500000;disable_loopback=false;internal_thread_pool_max_threads=4;external_port=0;oob_thread_pool_max_threads=10;log_discard_msgs=true;name=UDP;oob_thread_pool_keep_alive_time=30000;bind_addr=10.88.2.20;wheel_size=200;bundler_capacity=20000;log_discard_msgs_version=true;enable_batching=true;tick_time=50;timer_max_threads=4;ucast_send_buf_size=100000;thread_pool_queue_enabled=true;enable_bundling=true;ucast_recv_buf_size=64000;oob_thread_pool_queue_enabled=false;thread_pool_keep_alive_time=30000;bind_port=0;thread_pool_min_threads=2;ignore_dont_bundle=true;ip_ttl=8;bind_interface_str=;diagnostics_ttl=8;tos=8;loopback_separate_thread=true;logical_addr_cache_expiration=120000;oob_thread_pool_queue_max_size=500;diagnostics_addr=224.0.75.75;receive_on_all_interfaces=false;mcast_port=23302;internal_thread_pool_queue_max_size=500;timer_queue_max_size=500;thread_pool_queue_max_size=10000;max_bundle_size=64000;physical_addr_max_fetch_attempts=1;ip_mcast=true;timer_min_threads=2;thread_pool_enabled=true;bundler_type=transfer-queue;timer_keep_alive_time=5000;logical_addr_cache_reaper_interval=60000;timer_type=new3;diagnostics_port=7500;who_has_cache_timeout=2000):PING(async_discovery_use_separate_thread_per_request=false;ergonomics=true;stagger_timeout=0;force_sending_discovery_rsps=true;async_discovery=false;timeout=3000;always_send_physical_addr_with_discovery_request=true;max_members_in_discovery_request=500;send_cache_on_join=false;num_initial_srv_members=0;break_on_coord_rsp=true;stats=true;use_disk_cache=false;num_initial_members=10;name=PING;discovery_rsp_expiry_time=60000;id=6;return_entire_cache=false):MERGE3(check_interval=48000;stats=true;min_interval=10000;ergonomics=true;name=MERGE3;id=54;max_participants_in_merge=100;max_interval=30000):FD_SOCK(get_cache_timeout=1000;sock_conn_timeout=1000;client_bind_port=0;ergonomics=true;start_port=0;port_range=50;suspect_msg_interval=5000;num_tries=3;bind_interface_str=;stats=true;external_port=0;name=FD_SOCK;bind_addr=127.0.0.1;keep_alive=true;id=3):FD_ALL(use_time_service=true;stats=true;timeout_check_interval=2000;ergonomics=true;name=FD_ALL;interval=8000;id=29;timeout=40000;msg_counts_as_heartbeat=false):VERIFY_SUSPECT(num_msgs=1;use_mcast_rsps=false;bind_interface_str=;stats=true;ergonomics=true;name=VERIFY_SUSPECT;bind_addr=127.0.0.1;id=13;timeout=1500;use_icmp=false):NAKACK2(resend_last_seqno_max_times=3;use_mcast_xmit=false;ergonomics=true;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=30000;become_server_queue_size=50;xmit_interval=500;print_stability_history_on_failed_xmit=false;resend_last_seqno=true;max_xmit_req_size=511600;discard_delivered_msgs=true;suppress_time_non_member_warnings=60000;max_msg_batch_size=500;xmit_table_num_rows=100;stats=true;xmit_from_random_member=false;log_discard_msgs=true;log_not_found_msgs=true;xmit_table_resize_factor=1.2;name=NAKACK2;id=57;max_rebroadcast_timeout=2000;use_mcast_xmit_req=false):UNICAST3(ergonomics=true;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=60000;ack_threshold=5;sync_min_interval=2000;max_retransmit_time=60000;xmit_interval=500;max_xmit_req_size=511600;conn_close_timeout=10000;max_msg_batch_size=500;conn_expiry_timeout=0;ack_batches_immediately=true;xmit_table_num_rows=100;stats=true;xmit_table_resize_factor=1.2;log_not_found_msgs=true;name=UNICAST3;id=64):STABLE(cap=0.1;stability_delay=0;stats=true;ergonomics=true;name=STABLE;desired_avg_gossip=50000;max_bytes=4000000;id=16;send_stable_msgs_to_coord_only=true):GMS(max_join_attempts=10;print_local_addr=true;handle_concurrent_startup=true;view_bundling=true;leave_timeout=1000;log_view_warnings=true;install_view_locally_first=false;ergonomics=true;use_delta_views=true;resume_task_timeout=20000;use_flush_if_present=true;use_merger2=true;print_physical_addrs=true;join_timeout=2000;view_ack_collection_timeout=2000;stats=true;num_prev_views=10;merge_timeout=5000;max_bundling_time=50;name=GMS;num_prev_mbrs=50;id=14;log_collect_msgs=false;membership_change_policy=org.jgroups.protocols.pbcast.GMS$DefaultMembershipPolicy@1197b2de):UFC(min_threshold=0.4;stats=true;ergonomics=true;max_credits=2000000;name=UFC;min_credits=800000;id=45;max_block_time=5000;ignore_synchronous_response=false):MFC(min_threshold=0.4;stats=true;ergonomics=true;max_credits=2000000;name=MFC;min_credits=800000;id=44;max_block_time=5000;ignore_synchronous_response=false):FRAG2(frag_size=60000;stats=true;ergonomics=true;name=FRAG2;id=5):RSVP(ack_on_delivery=true;stats=true;ergonomics=true;name=RSVP;resend_interval=2000;id=55;throw_exception_on_timeout=true;timeout=10000)}
INFO  [main][ModuleFrameworkImpl:1725] Started dynamic bundles
INFO  [main][ModuleFrameworkImpl:413] Navigate to Control Panel > Configuration > Gogo Shell and enter "lb" to see all bundles
WARN  [Elasticsearch initialization thread][EmbeddedElasticsearchConnection:288] Liferay is configured to use embedded Elasticsearch as its search engine. Do NOT use embedded Elasticsearch in production. Embedded Elasticsearch is useful for development and demonstration purposes. Refer to the documentation for details on the limitations of embedded Elasticsearch. Remote Elasticsearch connections can be configured in the Control Panel.
ERROR [Framework Event Dispatcher: Equinox Container: e9a28482-d79f-413e-904c-6xxxxxxxxxxx][com_liferay_portal_search:97] FrameworkEvent ERROR
java.lang.IllegalStateException: Unable to initialize Elasticsearch cluster: {........

First question before i go to the next, am i supposed to set else where like in nginx or tomcat within liferay to allow the cluster to work?
I even tried purging the elasticsearch directory but error persists. Am i supposed to start server A and server B together to make this work?

Thanks in advance everyone.