Liferay & Cloudflare

How one little setting can break your site...

Another quick one this week...

A community member reached out and asked for help with a problem on their DXP Cloud site. They were already working with DXP Cloud support, but they had been struggling with the issue for over a week and were trying to get help from all corners...

When they accessed their site using the direct URL (an internal URL that all DXP Cloud admins have access to), the site rendered fine.

But when they accessed their site through their custom domain name, the site was broken and the javascript console had errors like:



Starting from the top error, it looked like jQuery wasn't being loaded. There were a bunch of errors around the $ being undefined and that cascaded into other errors.

We checked the network tab and all of the responses were 200, so there didn't appear to be any issues accessing the necessary resources.

The next step was to look at the responses for each request to see if there was any difference.

On the third request down, we noticed a difference... On the DXP Cloud URL for the combo servlet, jQuery appeared right at the top of the file and the page rendered fine. On the domain name, jQuery was not at the top, it actually ended up being mixed in the response somewhere in the middle, but not at the top where it was expected to be.

I checked the two URLs, shown below but with the hostnames masked off:

https://www.example.com/combo?browserId=other&minifierType=js&languageId=en_US&b=7310&
  t=1646057119839&/o/frontend-js-jquery-web/jquery/jquery.min.js&
  /o/frontend-js-jquery-web/jquery/init.js&/o/frontend-js-jquery-web/jquery/ajax.js&
  /o/frontend-js-jquery-web/jquery/bootstrap.bundle.min.js&
  /o/frontend-js-jquery-web/jquery/collapsible_search.js&
  /o/frontend-js-jquery-web/jquery/fm.js&/o/frontend-js-jquery-web/jquery/form.js&
  /o/frontend-js-jquery-web/jquery/popper.min.js&
  /o/frontend-js-jquery-web/jquery/side_navigation.js

https://internal-url/combo?browserId=chrome&minifierType=js&languageId=en_US&b=7310&
  t=1646057119839&/o/frontend-js-jquery-web/jquery/jquery.min.js&
  /o/frontend-js-jquery-web/jquery/init.js&/o/frontend-js-jquery-web/jquery/ajax.js&
  /o/frontend-js-jquery-web/jquery/bootstrap.bundle.min.js&
  /o/frontend-js-jquery-web/jquery/collapsible_search.js&
  /o/frontend-js-jquery-web/jquery/fm.js&/o/frontend-js-jquery-web/jquery/form.js&
  /o/frontend-js-jquery-web/jquery/popper.min.js&
  /o/frontend-js-jquery-web/jquery/side_navigation.js

Other than the hostnames, the URLs are the same, but the responses were different. And not different in that maybe minification was happening in one but not the other, complete sections were out of order in the response for the domain name one.

My friend Andrew Betts on the DXP Cloud support team noticed something important in the response: It looked like the response sections were actually in alphabetical order from the filename. So instead of jQuery, init, ajax, bootstrap, collapsible_search, fm, form, popper and side_navigation, the order in the response was ajax, bootstrap, collapsible_search, fm, form, init, jQuery, popper and side_navigation.

Another clue, by happenstance there was a warning in the logs from an invalid version URL:

[dxp] WARN  [http-nio2-8080-exec-21][HttpImpl:453] Invalid url 
  https://www.example.com/combo/?&b=7010&browserId=other&languageId=en_US&
  minifierType=&t=1638373748280

The one thing I know about Liferay URLs, the parameters are never provided in alphabetical order.

This realization was key to understanding the combo servlet response order... The combo servlet will include in the response the same order as it receives in the URL parameter. So the response was sorted alphabetically because the parameter list had been sorted alphabetically.

But what was sorting the URL that we were submitting to Liferay?

Then it hit me - it had to be something that was caching responses from Liferay...

If I were an aggressive caching system, I would want to take urls like https://www.example.com?c=1&a=2&b=3 and turn them into https://www.example.com?a=2&b=3&c=1 so that way it wouldn't matter what order the parameters were handed to me, by reordering them I could maximize my cache hits.

This had to be what was going on, it was the only way to explain that combo servlet would generate the response in the order it was using.

We asked the client what CDN or caching layer they were using and they indicated they were using Cloudflare. This was the final nail in the coffin.

We pulled up the handy Cloudflare documentation at https://support.cloudflare.com/hc/en-us/articles/206776797-Understanding-Query-String-Sort and found that yes, Cloudflare can sort the query string in order to increase the effectiveness of the cache and yes, client did have this setting enabled.

After disabling this setting, the domain name-based URLs started working fine.

Moral of the story:

When using Cloudflare, disable the Query String Sort.

Now you might be asking "Won't this negatively impact the cache performance?"

No it won't. Liferay generates URLs in the same order every time. So the combo servlet call that we have above, it will always be in that order, the order will not vary in any way because it is not generated by a developer, it is generated by the Liferay code and it follows an algorithm.

So the query sort is not needed because Liferay URLs will always match the previous one, unless they don't because it is actually a different request.