Securing Liferay Chapter 3: Port issues and HTTP/HTTPS

You probably know the basic installation instructions for Liferay Bundles: „unzip and run startup.sh“ - with this you get to a working Liferay installation in a minute. It will run with all defaults - which might not be what you want in production.

This is part 3 of a series. Start with part 1 for "Introduction, Basics and Operating System Level", continue with part 2, "Liferay's configuration" and come back here. You might also want to check if more chapters are already available.

8080? I want 80!

In Chapter 1 we kept tomcat running on port 8080 and I promised that this will be mitigated later. Now is the time. Apart from port 80 we'll also cover port 443 for https access, but let's go step by step:

In order to bind to a port below 1024, an application on Unix must run as root or gain those privileges in some other way. I've already commented that this is a very bad idea for a process that is connected to the internet. In case there's any security issue that can be exploited remotely, you're toast as it's trivial to gain root access on your computer.

For this reason (and some others) I like to run a proper webserver in front of tomcat. Let's take Apache httpd for this chapter. Substitute with the one you are most familiar with. I'll abbreviate it as "Apache" for the rest of this chapter.

Apache drops the root permissions after binding to ports 80 and 443, so effectively it will not run as root. This is a trick that is easy if you run native on the operating system, but hard for a JVM process. Win: We're answering requests on port 80 without running as root. Fail: Now Apache serves our content, not tomcat - they'll need to be connected. Several options are available for this purpose

HTTP vs AJP

Apache offers mod_proxy and mod_jk (among others). They differ in the protocol that is spoken between it and tomcat. mod_proxy (to be exact: mod_proxy_http) communicates through http, while mod_jk (also to be complete: and mod_proxy_ajp) communicate with a binary protocol, named AJP.

I'm a big proponent of AJP, as it covers all of the default expectations that you have for this purpose. Assuming that you're using your distribution's Apache and you've installed mod_jk, here's what you do:

Configure some workers.properties that are pointing to your tomcat's AJP-connector. Where's that? Check your conf/server.xml file in tomcat. The default is port 8009. For the purpose of this documentation, I'm assuming that Apache is running on www.example.com, while tomcat is running on tomcat.example.com.

workers.properties:

for me, this file is /etc/apache2/workers.properties, as the next snippet refers to it.

ps=/ 
worker.list=tomcat1 
worker.tomcat1.port=8009 
worker.tomcat1.host=tomcat.example.com 
worker.tomcat1.type=ajp13 
worker.tomcat1.lbfactor=1

Now, how does this get into Apache?

You'll most likely have some VirtualHost configuration in Apache anyway for the server that you're building. Here's some pseudocode for general Apache configuration, as well as for the virtual hosts. On Ubuntu the next snippet might go into /etc/apache2/conf/liferay-settings.

ServerSignature Off ServerTokens ProductOnly TraceEnable Off FileETag None Options -Indexes
JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel error
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories +ForwardURICompatUnparsed
NameVirtualHost your-ip-address:80 NameVirtualHost *:80 NameVirtualHost your-ip-address:443 NameVirtualHost *:443

and a snippet from /etc/apache2/sites-available/default

<VirtualHost _default_:80>
        ServerAdmin webmaster@example.com
        ServerName www.example.com
        DocumentRoot /srv/www/
        ErrorLog /var/log/apache2/error.log
        CustomLog /var/log/apache2/access.log combined
        Options +MultiViews
        JkMount /* tomcat1
        JkMount /  tomcat1
	JkUnmount /static/*
</VirtualHost>

What does this do? Every request that gets to Apache's default virtual host will be forwarded to tomcat. The only exception is that requests to www.example.com/static/* will still be handled by Apache (see the JkUnmount line).

Achievement unlocked: We're answering on port 80 but still run as the unprivileged user that we've been used in chapter 1.

https anyone?

What about https? Well, not much to change. Configure Apache like you would for https anyway, add the same JkMount instructions to the virtual host. With AJP you're set: Tomcat/Liferay knows that you're communicating on https, knows the ports, host names etc.

I don't go too much into the setup of a proper https server - a lot of recommendations have changed with the issues that surfaced lately. Just so much: You might want to check your setup for the recent issues. ssllabs.com is one of the sites that offer free instant testing.

Keep your private key under tight control, get a certificate for your key, set up the virtual host and you're set: https is ready.

Should I force https?

If your site contains data (or uses passwords) that should be protected, and you offer https anyway, I believe that it's a good idea to force https on anybody. Won't this generate significant overhead on the webserver? Measure!

With the setup that we have so far, you could easily add a https-terminator into the game, or have https completely handled by your Apache. You'll need to figure out by yourself what fits your environment and load profile.

If you want to force https, just implement unconditional redirect on the VirtualHost for port 80 to the https VirtualHost, like this:

<VirtualHost _default_:80>
        ServerAdmin webmaster@example.com
        ServerName www.example.com
        DocumentRoot /srv/www/
        ErrorLog /var/log/apache2/error.log
        CustomLog /var/log/apache2/access.log combined
        RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

And if you know HSTS, you might want to add one line to your VirtualHost on port 443:

<VirtualHost _default_:443>
        ServerAdmin webmaster@example.com
        ServerName www.example.com
        SSLEngine On
        # further https options omitted
        ErrorLog /var/log/apache2/error.log
        CustomLog /var/log/apache2/access.log combined
        JkMount /* tomcat1
        JkMount /  tomcat1
	JkUnmount /static/*
        Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</VirtualHost>

Naturally, this requires the required modules to be installed: Header and Rewrite* are not in the Apache core, but readily available.

What about http/https mixed mode?

This is quite a popular question. Why not use http for users that are not logged in, but use https as soon as they log in. Until recently I publicly stated that This! Does! Not! Work!. The main reason is that you'll definitely miss some setup and, sooner or later, leak some data, cookie or other information.

HSTS app iconRecently I found a neat workaround that limits the amount of configuration errors. As soon as the next Internet Explorer is available and adopted, it might even be a viable option (all other browsers support it). You can conditionally enable HSTS just when a user logs in to Liferay. More information in the Liferay HSTS app that yours truly has published on marketplace. With this, the case for mixed mode turns a bit towards a mixed mode that I don't totally reject.

Check the description in the app for the options that it opens. Note that you'll still make your life easier with the single line I give above. But if you drive up the download numbers and give reviews for that plugin, that is very welcome ;)

Other options: mod_proxy_http

Another quite popular configuration is to communicate http to tomcat. This has some drawback, e.g. all requests to tomcat will originate on Apache, tomcat will have no idea where in the world they came from. Also, tomcat will believe that its hostname is tomcat.example.com - this is true, but in a properly firewalled network, this address will not be available from the outside. We'll need to hack this with a few more options:

If you prefer proxying through http, look up ProxyPreserveHost On, which will make the original hostname, www.example.com, available to tomcat. Also, you want to configure Liferay's portal-ext.properties to have the proper ports. Check this in the original portal.properties that you already read during the previous chapter:

#
# Set the HTTP and HTTPs ports when running the portal in a J2EE server that
# is sitting behind another web server like Apache. Set the values to -1 if
# the portal is not running behind another web server like Apache.
#
web.server.http.port=-1
web.server.https.port=-1

(you probably want to set these ports to 80 and 443)

All of this is not necessary with AJP - everything is readily communicated to tomcat.

https and mod_proxy_http

With mod_proxy_http you'll need more work to let tomcat know that you're communicating https. You'll typically terminate the https connection on Apache and just forward to tomcat through http. For this reason tomcat doesn't know about the encryption - it never sees any encrypted connection.

A neat hack that you can use here is: Introduce another HTTP connector on tomcat that you'll purely use for proxy requests from your https virtual host. Add the secure="true" attribute to let tomcat know that the original requests on this connector have been encrypted. The relevant part of your server.xml might look like this:

    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" URIEncoding="UTF-8" />

    <Connector executor="tomcatThreadPool"
               port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" URIEncoding="UTF-8"
               secure="true"/>

Now you only need to make sure that nobody but the encrypted VirtualHost on Apache does connect to 8081 and tomcat assumes that requests coming in on 8081 have indeed been encrypted - but doesn't need to handle any encryption itself.

Future chapters

...coming soon...

Remember: This is not the only - and not the complete - truth. Please add your experience (and disagreements) in the comments

More Blog Entries

Blogs
Nice article.
I personally prefer to use mod_ajp, the configuration is a lot simpler and it can easily be switched from ajp to http by changing the protocol in the url, if need arises (e.g. for debugging purposes).

<VirtualHost *:80>
ServerName www.example.com
ServerAlias liferay.example.com
ProxyPreserveHost On
ProxyPass / ajp://localhost:8009
</VirtualHost>

And that should be it, with a default Liferay installation (ajp on port 8009)

For me, mod_jk has just one big advantage: NTLM won't work with mod_ajp, only with mod_jk.

About https:
This tutorial is really useful to configure https (as of November 2014 at least, the web changes so fast nowadays ...)
https://raymii.org/s/tutorials/Strong_SSL_Security_On_Apache2.html

It is really difficult to get the cipher string "right".

Performance:
I usually disable the Zip-Filter in Liferay and compress it in Apache instead.

Also it's a good idea to cache static files (Theme images, icons, css) in the reverse proxy. (Or even better: Use a CDN).
To find out, what to cache, measure it! e.g. just use http://www.webpagetest.org to get some hints.
Thanks for the feedback. As of mod_proxy, ajp, jk, I'm agnostic - it's good to use the one you're most familiar with. The most important point is to not run as root in order to bind port 80. And especially thanks for the SSL pointer - it's really hard to find accurate information (or to know if information is accurate or not)
Proxy, cache etc. are rather related to performance than to security - but that's another series, currently in planning phase. I'll also be discuss it with Brett Swaim on Radio Liferay.
Thanks for this article. What would be the best configuration if we want to use 2 Apache HTTP Servers in Front of a cluster of 2 Tomcat Servers running Liferay ?
This is rather related to availability/scalability than to security. I wouldn't expect much difference in configuration, but the one correct answer is "it depends". You can have a separate load balancer on top of Apache. You can configure DNS round-robin. Or anything else...
Well that's actually complicated.
I would recommend to use just one Apache and keep a second one as backup on standby. Performance wise one apache should be able to easily handle all requests. At least more, than most backend servers. Configuration is much easier.

In any case: You need to stay sticky on one Liferay server. It's really bad to switch servers all the time, performance wise.

Apache can handle stickyness quite well, the second example should be helpful:
https://httpd.apache.org/docs/current/mod/mod_proxy_balancer.html#example

If you really want two Apache servers as reverse proxies, I'd still recommend a load balancer in front of them.

Using DNS round robin is possible, but be careful. Some clients don't do failover. e.g. Java. Also, at least IE 7 doesn't do failover at all, 50% of all requests will just fail if one proxy is down. I think IE8 (maybe Win 7 only) will do it right. Sadly, these dinosaurs are still out there ...
[...] Are you tasked with setting up a webserver that must be accessible in both http and https? When somebody demands that this mixed mode must be possible, here are some arguments why you shouldn't give in (because in general it does not work - at ... [...] Weiterlesen
Hi Olaf,

I followed your recvommendations to set up Apache inf Front of tomcat.
In the middle of your article to talk about basic Apache configuration in /etc/apache2/conf/liferay-settings
We are running apache under debian 4.4.5, so filestructure might be differend. I am not sure, if I have to crate a new file, and if so where do I have to store it (in tomact or apache dir).

In fact I did not implement these lines - but nevertheless it seems to work.
So my question is, what the excat meaning and purpose of this lines?

Thomas
Hi Thomas,
some distributions include everything they find in /etc/apache2/conf.d into a virtual httpd.conf (which is the general configuration file for Apache. This is so that every single file has a single purpose and you can track what the changes are there for. I've checked back: Indeed it might have been rather /etc/apache2/conf.d or, through having a configuration in /etc/apache2/conf-available and linking it in /etc/apache2/conf-enabled to be closer to standard ubuntu - however, the description in this article should be understood as pseudocode.
Maintaining a web-facing server is not an easy thing - that's why I don't like to give copy&paste advice and rather accept (or intentionally make) unexact instructions in situations like this blog article. There's a lot of copy-pasteable documentation available on the web.
Regarding your question as of the meaning of those lines: The gray ones are boilerplate, just giving some context from one of the random servers that I've grabbed content of. The "Jk" lines are the ones that configure mod_jk - If you decide to use mod_jk you can easily look up the directives if they don't speak for themselves. The google hits that I get on page one for just the directive name are all good.
If you decide to go with another option (mod_proxy, mod_proxy_ajp or others) your instructions will vary anyways. When you say that your installation already works without these instructions, odds are that you don't use mod_jk but one of the other modules.
It would require deeper insight into your Apache configuration to judge. I'd recommend to ask someone with Apache experience to inspect and bulletproof your installation
Hi Olaf,

Thanks for response. I like the idea to ask an Apache Expert. Do you know one ? Maybe you could give me some of your time. Currently I am yust playing around in test-envirement. But before deploying on production I definitely want advice from expert.
But please allow one further question:
Which changes are necesarry on tomcat site, especially in server.xml. By default ther is an entry for AJP Connector on Port 8009. What about the entries for port 80 and 443. Are they still necessary?
Hi Thomas,
reviewing your installation would be a consulting gig - public forum- and blogposts wouldn't show enough of the environment. This is possible, but has to go through sales-de (at liferay com)
Tomcat's server.xml can go completely unchanged: Apache httpd is covering port 80 by forwarding it to 8009 (through ajp/mod_jk) or 8080 (mod_proxy). There are reasons to change it, but no necessity. I'd hope that - in case you have port 80 and 443 in there, that you don't run tomcat as root in order to get access to those ports. That would be something that needs to be changed asap - see chapter 1 of this blog article series (as well as this one). In case your server gets exploited through any means, you don't want the attacker to have root privileges.
Debian 4? I hope that's a typo. Debian 4 doesn't receive security updates for several years anymore. You really should upgrade. There were lots of important bug and security fixes since it was discontinued.

About your question:
I am not sure how it can possibly work, if you don't create said file. Or add the config to some other file.

I believe:
Apache is not working at all and you have configured tomcat to serve ports 80 and port 443. Please review the apache logfiles, they are usually located in /var/log/

You can check on the commandline using "netstat -tulpn " which service "owns" which ports. You should see lines like the following

tcp6 0 0 :::80 :::* LISTEN 10690/apache2
tcp6 0 0 :::443 :::* LISTEN 10690/apache2

Please note the ports 80 & 443 and apache2 (maybe http2).
If you see instead something like this, 80 or 443 and java:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 927/java

Well, in that case you are not using apache at all.
Hi Christoph,

It is Debian 7.11 (so it is a typo). After hours of reading web-articles I got it running and you are right with your assumptions.

So what was the problem:
(1) With the Debian Distribution, when installung mode_jk, a separat directory is created. In my case it was /etc/libapache2-mod-jk. In that directory is a configuration file named httpd-jk.conf. All mentiond configuration was already there, so no change is needed !

(2) Olaf mentioned, that normally no changes to server.xml on tomcat site is necassary. That is right, when you setup a fresh liferay package. Then http connectors point to ports 8080 or 8443. In our case, they pointed to port 80 and 443 (where 80 was redirected to 443) - and that didnt work- as you assumpted. So our final configration for server.xml has only 2 connectors one for ajp on port 8009 which redirects to port 8080. And port 8080 without any redirection. No connector for port 443 or 8443!

(3) mode_jk is not mode_proxy_jk (there are some instructions in the net for mode_proxy_jk, but this article here is about mode_jk)

(4) workers.properties needs path to tomcat and jvm. In my case I only needed to overwrite standard entries in preconfigured workers.properties. workers.tomcat_home=/var/lib/liferay/product/tomcat and workers.java_home=/home/liferay/local/jdk . I left worker.list to the standard ajp13_worker. So I did not have to change it on other preconfigured places (you only have to give attetion to it, when configuring the virtual hosts).

(5) SSL: We already used SSL on tomcat (port 80 was redirected to 443 by server.xml). To terminate SSL at Apache, we only had to add the path to key an cert in Virtual Host and put SSL on (as mentioned in (2) connector for 443 was commented out), so that Apache works on 443 with encryption and tomact on port 8080 without encryption) :
SSLEngine On
SSLCertificateFile /home/liferay/ssl-keys/www.outsourcing.de.crt
SSLCertificateKeyFile /home/liferay/ssl-keys/www.outsourcing.de.key
SSLCACertificateFile /home/liferay/ssl-keys/intermediate.crt
To get that working it is necssary to enable SSL on Apache. With our distribution it is just entering a2enmod ssl on console.

(6) And the last point. To enforce SSL as done by rewrite rule in the virtualhost example, it is also necessary to enable rewrite on apache as well and that is just by entering a2enmod rewrite.

The reason, why we first thought it is working was:
When running Apache and tomcat on the same ports (80 and 443)- tomcat throws an exeption at the beginning of startup (what I did not see). After tomcat startup, pages are delivered by tomcat! When I switched APACHE off- no pages were delivered. So Apache was communcating with tomcat, but not in the right way :-)
Hmm, not sure about this sentence:
"So our final configration for server.xml has only 2 connectors one for ajp on port 8009 which redirects to port 8080. And port 8080 without any redirection. No connector for port 443 or 8443!"

ajp port 8009 shouldn't redirect to port 8080. Actually, in my productive setups I usually remove even the listener on port 8080 (or make it localhost only) and just use the ajp listener on 8009.

Each service should expose the minimal amount of ports possible.

Also, not sure what " mod_proxy_jk" is, I guess you mean mod_proxy_ajp.

Well, glad you got it running!
...and in case you're stuck with http on the layer between Apache httpd and your app server, David Nebinger has documented an alternative way to forward information like the original protocol (http vs https) here: https://web.liferay.com/en/web/user.26526/blog/-/blogs/revisiting-ssl-termination-at-apache-httpd