Tomcat 9.0.31, Ghostcat and AJP

My friend, Olaf Kock, recently shared with me that he had struggled with and resolved an issue after moving to Tomcat 9.0.31 when using AJP.

Update: What maybe we didn't know, Tomcat 9.0.31 (and other versions of Tomcat 6, 7, 8 and 8.5) were all being fixed to address a newly identified attack vector against Tomcat nicknamed Ghostcat: https://www.zdnet.com/article/ghostcat-bug-impacts-all-apache-tomcat-versions-released-in-the-last-13-years/

Since I'm a big fan of using AJP to connect Apache HTTPd and Tomcat, I thought I'd share what he found with you. I must say, though, that this is just a blog about Olaf's efforts, I'm now just playing Watson to his Holmes...

Long story short, in Tomcat 9.0.31 (and onward), the AJP connector is not going to be enabled by default. It was removed to prevent exposure as a security attack vector. There were also some changes to configuring AJP correctly.

I'll present these shortly, but wanted to summarize first...

Why AJP?

I think typically you'll find most often that sys admins front tomcat with Apache HTTPd using mod_proxy, basically a mechanism where httpd accepts the incoming HTTP connection on port 80 and will proxy an HTTP request to tomcat running typically on port 8080. Port 8080 is open and the default Tomcat port and it's why we all just spin up a Tomcat instance and will use 8080 in our browser URLs for testing.

Because this is just an OOTB configuration for both HTTPd and Tomcat, it is really an easy way to connect the two. Oh, and Apache doesn't normally ship the AJP connector with HTTPd, so there's extra effort to get AJP connection going in your environment.

So why does it make sense to try and pursue the AJP option?

First, it's a connection leveraging a binary protocol. The headers, cookies, parameters and other values in an HttpServletRequest? Those get parsed by Apache HTTPd already. Using AJP, the pre-parsed form is passed on to Tomcat for processing, unlike with the proxy method where Apache HTTPd is parsing, but then Tomcat also has to do the parsing again.

Second, Tomcat sees the request as though it came straight through to Tomcat, without Apache HTTPd in the middle. So the source address, port, and other information are given to Tomcat as-is.  With the Proxy option, the request that Tomcat gets originates from the HTTPd server, not the remote client. So you have to go through additional configuration to expose things like the incoming URL, the source IP address and port, etc. to Tomcat.

To me, these two things alone make AJP a clear win, even though it has the extra deployment and setup concerns.

What Changed?

Rather than rewording myself, I'm just going to quote Olaf since he did all of the work:

As soon as our bundles are updated to Tomcat 9.0.31, note that there were some changes in the AJP Connectors that might disrupt people's expectation: They're disabled by default, and enabling them requires setting an interface explicitly, as well as a secret that must be shared with the reverse proxy.


Those who don't use AJP don't need to do anything, those that use AJP should notice this "secure by default" configuration and make sure that they don't open a security hole by re-enabling. Relevant sections from https://tomcat.apache.org/tomcat-9.0-doc/changelog.html

  • Disable (comment out in server.xml) the AJP/1.3 connector by default. (markt)
  • Change the default bind address for the AJP/1.3 connector to be the loopback address. (markt)
  • Rename the requiredSecret attribute of the AJP/1.3 Connector to secret and add a new attribute secretRequired that defaults to true. When secretRequired is true the AJP/1.3 Connector will not start unless the secret attribute is configured to a non-null, non-zero length String. (markt)
  • Add a new attribute, allowedRequestAttributesPattern to the AJP/1.3 Connector. Requests with unrecognised attributes will be blocked with a 403. (markt)

Caused a few confusions that were discussed on the tomcat mailing list.

So basically to continue to use AJP, you're going to be editing the default server.xml that Tomcat ships with to enable the connector, possibly bind to a network address (if HTTPd is on another server), plus include these attribute changes highlighted above.

So, two important points here:

1. If you update your Tomcat server to 9.0.31 (or later), you are going to need to make some changes to the configuration for these new AJP updates.

2. If you see Olaf out and about, you should thank him (as I do) for gathering this information for us...

Update

Since posting the blog, news of Ghostcat has been spreading: https://www.zdnet.com/article/ghostcat-bug-impacts-all-apache-tomcat-versions-released-in-the-last-13-years/

Every system admin, whether using AJP or not, needs to make changes in their environment:

  • If not using AJP, disable the AJP connection on port 8009 in server.xml. It's also a good idea to block external access to the port in case the AJP gets re-enabled accidentally in the future.
  • If using AJP, update your Tomcat version to the newly updated ones with the patched AJP code.
  • Establish firewall rules that only allow AJP connections from trusted hosts.

 

 

Blogs

Hi David, I'm a fan of AJP too. Do you know if there is a way to encrypt the communication between Apache and Tomacat using AJP? In most cases I have to use mod_proxy to take advantage of SSL.

I think I could get used to mentioning something to David. This saves a lot of work writing an article myself - Big Win! :) One additional note, that I didn't mention in the snippet: AJP is thought of as dangerous, thus it needs to be enabled explicitly. It's dangerous if your AJP port is open to the world - e.g. if third-party webservers could connect to it: They'll have quite some power of tricking tomcat into believing in the request that's coming. This is also the reason why "just enabling it" won't be enough, but you'll need to configure the mentioned "secret" as non-null, non-blank string. Notice also that AJP is totally and utterly unencrypted - cleartext (well clearbinary) by design. If you require encryption on that connection, consider going https - or establish a tunnel or VPN between the servers. I love AJP, due to the features that David already mentioned. It'll just be even more work once tomcat comes "secure by design". By the way, this change is also incorporated into tomcat 8.5.51 and 7.0.100

Was also looking at using the secret option and noticed that HTTPd supports this under version 2.4.42 which is yet to be released.  Any recommendations on what versions to use to support this?

Despite the secret's name, it travels across an unencrypted network connection, and you can continue to operate through AJP if you know what you're doing: The AJP connection between httpd and tomcat is unencrypted, so you'll need to trust the connection. If you limit access to your AJP connector to trusted sources (e.g. your httpd), that'll go almost all the way. The secret will only provide additional protection if you accidentally make your AJP port available to non-trusted traffic sources. They'd need to know this secret.

 

In a related note: I've mentioned above that AJP is a binary protocol and got corrected in a private conversation - it contains a lot of textual stuff - largely what http contains anyway. When I labelled it as binary, I meant to describe it as "not what you'd typically want to construct manually" (as some simple HTTP/1 requests that can be issued through telnet). And no, I've never tried constructing AJP requests manually - not interested to go in that deep.

 

Again: Keep your network under control, under no circumstance open your AJP port to anyone other than your reverse proxy, and you should be largely set.

The explanation was awesome, do we have a working example what to set in tomcat and what need to add-in httpd, doest it look correct?

 

    <Connector protocol="AJP/1.3"                address="::1"                port="8029"                redirectPort="8443"                 allowTrace="true"                secret="12345"

               allowedRequestAttributesPattern="7080"/> #               allowedRequestAttributesPattern="AJP_REMOTE_PORT"

 

Does it look correct and what should we add in the in httpd.

 

Tomcat documentation is very confusing maybe for experts it is good but it need add more examples.