Maven Madness in Liferay Land

In this post I want to share how you can use Maven to do 2 important things in a Liferay project: patching and creating a deploy package.
 
While this way of patching is a very powerful way to modify Liferay in an automated way (no manual WAR/JAR patching) you should use it as sparingly as possible. If possible you should always try to use more conventional ways of modding Liferay, for example using a hook, when possible. But whenever that's not possible, or like me you don't really like EXT or extlets, feel free to use the methods described in this post.
 
While this post will be focused on using Maven as a build tool to achieve the desired result, the concepts and techniques should also apply to other build tools like Ant, Gradle or Buildr (and if you're not already using a build tool... this might be the moment to start using one).
 

Patching

While it can produce similar results as the Liferay EE patching tool in some respects, the patching described in this section is a slightly different beast. They even can, as you'll see in the deploy package section, be used together. There are a number of different reasons to use this additional way of patching Liferay:
  • you can't use the Liferay patching tool because you're using Liferay CE
  • you can't wait for an official patch/fix and need a temporary workaround
  • you need to add or change Liferay files that you can't (or don't want to) change using a hook
  • ...
There are a lot of parts of Liferay that you might want/need to patch:
  • the portal WAR
  • the portal-service.jar
  • one of the other Liferay JARs that are contained in the portal WAR: util-java.jar, util-bridges.jar, ...
  • the most special case of all: a JAR that's included in another JAR that itself is included in the portal WAR aka 'overlay-ception'

The portal WAR

Let's start with the easiest case: patching the Liferay WAR file. What makes this the easiest one is that you can simply use the well known Maven WAR plugin's overlay ability to achieve the desired result. For most file types you just need to put them in the correct directory of your module for them to be correctly overlayed:
  • src/main/java: Java classes you want to override. They'll be put in the WEB-INF/classes directory and they'll be loaded before any classes from the WEB-INF/lib directory.
  • src/main/resources: non-Java source files that you want to end up in WEB-INF/classes like log4j configuration or a JGroups configuration file to configure your cluster to use unicast instead of multicast.
  • src/main/webapp: all files here will override their counterparts their corresponding directory in the WAR (useful for overriding JSPs) or you can even use it to add new files (JS libs, images, ...). You can also add a WEB-INF subdirectory and add modified versions of web/Liferay descriptors like web.xml or liferay-portlet.xml.
The Maven XML configuration that is needed for this is pretty simple: you just reference the Liferay WAR as a dependency with a *runtime* scope. During the build Maven will unpack the WAR and then copy over (aka overlay) everything from src/main and repackage it again.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   ...
   <build>
        <finalName>portal-web</finalName>
      <plugins>
         <plugin>
            <artifactId>maven-war-plugin</artifactId>
            <configuration>
               <warName>${project.build.finalName}</warName>
               <outputFileNameMapping>@{artifactId}@.@{extension}@</outputFileNameMapping>
               <overlays>
                  <overlay>
                     <groupId>com.liferay.portal</groupId>
                     <artifactId>portal-web</artifactId>
                  </overlay>
               </overlays>
            </configuration>
         </plugin>
      </plugins>
    </build>

   <dependencies>
      <dependency>
            <groupId>com.liferay.portal</groupId>
            <artifactId>portal-web</artifactId>
            <!-- Include a specific version of Liferay with a release -->
            <version>${liferay.version}</version>
            <type>war</type>
            <scope>runtime</scope>
        </dependency>
   </dependencies>
</project>
You can even exclude some of the JARs that are already in the WEB-INF/lib directory, such as the JAI JARs, if your OS already provides them or for some other reason. Excluding a JAR dependency in the configuration section of the Maven WAR plugin and then adding a dependency to your overridden version in pom.xml will make sure your custom overlayed JAR will get added instead of the original one. If you just want to add an additional JAR then you only need to add it as a dependency, e.g. a platform specific Xuggler JAR.
 

The portal service JAR

The next type of overlay is a JAR overlay. This is slightly more complicated, but can still be accomplished by using a pretty standard Maven plugin: the Maven Dependency plugin. The unpack goal during the prepare-package phase allows us to unpack a JAR to the target directory of our module. We then need to provide the pluging with a list of files we want to exclude so that we can override them with our own. We'll calculate this list using the Maven Groovy pluginbased on the contents of the  src/main/java directory .
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   ...
   <build>
      <finalName>portal-service</finalName>
      <plugins>
         <plugin>
            <groupId>org.codehaus.gmaven</groupId>
            <artifactId>gmaven-plugin</artifactId>
            <executions>
               <execution>
                  <id>generate-patched-file-list</id>
                  <phase>compile</phase>
                  <goals>
                     <goal>execute</goal>
                  </goals>
                  <configuration>
                     <source>${basedir}/src/main/groovy/listfiles.groovy</source>
                  </configuration>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
               <execution>
                  <id>unpack</id>
                  <phase>prepare-package</phase>
                  <goals>
                     <goal>unpack</goal>
                  </goals>
                  <configuration>
                     <artifactItems>
                        <artifactItem>
                           <groupId>com.liferay.portal</groupId>
                           <artifactId>portal-service</artifactId>
                           <version>${liferay.version}</version>
                           <type>jar</type>
                           <overWrite>true</overWrite>
                           <outputDirectory>${project.build.directory}/classes</outputDirectory>
                           <excludes>${patched.files}</excludes>
                        </artifactItem>
                     </artifactItems>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
   ...
</project>
The actual magic that calculates the list of exclusions happens in the listfiles.groovy file:
def files = []
new File(project.build.directory).eachFileRecurse() { file ->
   def s = file.getPath()
   if (s.endsWith(".class")) {
      files << "**" + s.substring(s.lastIndexOf('/'))
   }
}

project.properties['patched.files'] = files.join(",")
The other Liferay JAR files, like util-java.jar, util-bridges.jar, ... or basically any other JAR (e.g.: I patched the PDFBox library in Liferay like this) can be overlayed in a similar way.
 

Overlay-ception

The third and most peculiar overlay we might need to do is related to something special Liferay does while hot deploying a WAR. Depending on the contents of the  liferay-plugin-package.properties file of your portlet the Liferay deployer will copy one or more JARs ( util-bridges.jar, util-java.jar, util-sl4j.jar and/or util-taglib.jar), that are embedded in the /com/liferay/portal/deploy/dependencies folder of portal-impl.jar over to the WEB-INF/lib directory of your deployed module. There are also a bunch of other files, like DTDs, in there, that you might want/need to override, that are also copied over in some cases depending on your configuration.
 
If you for example had to override util-taglib.jar to fix something, you will not only need to make sure your customized version gets included in the overlayed WAR file, but you'll also need to put it in the correct location in your overlayed portal-impl.jar that also needs to be included in the overlayed WAR file. For this we'll again use the Maven Dependency plugin, but instead of just 1 execution that unpacks the JAR we need to add a 2nd execution that uses the copy goal to copy all provided artifactItems (basically dependencies) to the correct location inside of the target directory. So first it will unpack the original contents of the JAR, then overlays it with any custom classes/resources and lastly it will copy over any overlayed libraries.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   ...
   <build>
      <finalName>portal-impl</finalName>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
               ...
               <execution>
                  <id>copy-overlay</id>
                  <phase>generate-sources</phase>
                  <goals>
                     <goal>copy</goal>
                  </goals>
                  <configuration>
                     <artifactItems>
                        <artifactItem>
                           <groupId>be.planetsizebrain.liferay</groupId>
                           <artifactId>util-taglib-overlay</artifactId>
                           <outputDirectory>${project.build.directory}/classes/com/liferay/portal/deploy/dependencies</outputDirectory>
                           <destFileName>util-taglib.jar</destFileName>
                        </artifactItem>
                     </artifactItems>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>

   <dependencies>
      <dependency>
         <groupId>com.liferay.portal</groupId>
         <artifactId>portal-impl</artifactId>
      </dependency>
      <dependency>
         <groupId>be.planetsizebrain.liferay</groupId>
         <artifactId>util-taglib-overlay</artifactId>
      </dependency>
      ...
   </dependencies>
</project>
When I finally figured out how to get the above to work I was like: 
 

Deploy package

One last special thing we'll use Maven for is to create what we call a deploy package. A deploy package is basically a ZIP file that contains everything that is needed to do a deploy on a server or even locally on a developer's machine. While currently there are alternative options for this like Docker we find that at a lot of customers this isn't always an option. Our deploy package solution will work on just about any Unix based OS as it only uses (Bash) scripting. Some nice additional functionality is possible if your servers have access to some sort of Maven repository like a Nexus or Artifactory, but it isn't needed per se. If you have access to a repository you only need to put the deploy script on the server and it will download the deploy package for you and install it. If you don't have access to a repository you'll need to first download the package yourself and transfer it to the server and then run the script.
 

Building the package

The packaging of the ZIP is done using the Maven Assembly plugin in the deploy module. The deploy module needs to be last module in your list of modules of your root pom.xml for it to be able to correctly reference all the build artifacts. Because we want to be able to use this package, together with the script described in the next section, to be able to produce a reproduceable deploy, it really needs to contain just about everything and the kitchen sink:
  • a base Liferay Tomcat bundle
  • any hotfixes/patches/security fixes
  • the patching tool itself (if you need a newer version than available in the bundle)
  • a database driver (unless already present in the bundle above)
  • your overlayed Liferay WAR or JARs
  • all your custom portlets, hooks, themes & layouts
  • any Marketplace portlets you use (possibly overlayed using the techniques described above)
  • all the configuration files, like the portal-ext.properties, Tomcat's setenv.sh, license files, etc..., for all environments (so 1 bundle can be used)
Basically anything you can put in a Maven repository and can reference in your deploy module's pom.xml can be packaged into the deploy package. If your server environment has access to a Maven repository, some things, like the base Liferay Tomcat bundle and the patching tool, can be left out of the package as they can be downloaded just in time by the deploy script described in the next section.
 
This deploy module is also an excellent place to create an escrow package which is all the source code, resources and libraries that are needed to recreate a specific version of your portal. How you can do this can be found in an older blog post by me: Creating an escrow package with Maven.
 
How the actual package is made is defined in the assembly.xml file that is used to configure the Maven Assembly plugin. In this file we define the format of the artifact, ZIP, and where all the dependencies, defined in the module's pom.xml, need to be put in this ZIP file. In the example you'll see that:
  • using a fileSet all the configuration files, found in src/main/resources are put in the /liferay subdirectory
  • using a dependencySet all the WAR files, defined as dependencies in pom.xml, are put in the /war subdirectory
  • using a dependencySet all the ZIP files, defines as dependencies in pom/xml, are put in the /patches subdirectory 
<assembly>
   <id>portal-release-bundle</id>
   <formats>
      <format>zip</format>
   </formats>
   <includeBaseDirectory>false</includeBaseDirectory>
   <fileSets>
      <fileSet>
         <directory>${basedir}/src/main/resources</directory>
         <outputDirectory>liferay</outputDirectory>
         <includes>
            <include>**/*</include>
         </includes>
         <useDefaultExcludes>true</useDefaultExcludes>
         <fileMode>0644</fileMode>
         <directoryMode>0755</directoryMode>
      </fileSet>
   </fileSets>
   <dependencySets>
      <dependencySet>
         <includes>
            <include>*:war</include>
         </includes>
         <outputDirectory>/wars</outputDirectory>
         <outputFileNameMapping>${artifact.artifactId}.war</outputFileNameMapping>
         <useProjectArtifact>false</useProjectArtifact>
      </dependencySet>
      <dependencySet>
         <includes>
            <include>*:zip</include>
         </includes>
         <outputDirectory>/patches</outputDirectory>
         <outputFileNameMapping>${artifact.artifactId}.zip</outputFileNameMapping>
         <useProjectArtifact>false</useProjectArtifact>
      </dependencySet>
   </dependencySets>
</assembly>
You can see that with some simple changes to the assembly.xml you could create additional/other subdirectories containing other sets of configuration files/dependencies to suit your needs.
 

The deploy script

As this really is just a script it can basically do anything you like and can be easily modified to fit your specific needs, the server environment's constaints, etc... . The example script you can find on Github does the following actions (in order):
  • Ask for some values (that can also be provided on the commandline):
    • Which repository you want to download the package from (snapshots, releases, local)
    • Which version of the package you want to deploy (dynamically build list if not provided on command line)
    • Which environment you'll be deploying on
    • Ask which node you're deploying on in case it is a clustered environment
  • Stop the Tomcat server (and kill the process if it doesn't stop nicely)
  • Backup some files like the Lucene index or the document library
  • Remove the whole Liferay bundle
  • Download and unzip the base Liferay bundle in the correct location
  • Download and unzip the requested deploy package in a temporary location
  • Overlays our Liferay WAR overlay from the temporary location over the unzipped Liferay bundle
  • Download, unzip & configure the latest Liferay patching tool
  • Restores the backed up files
  • Cleans some Tomcat directories
  • Overlays our environment specific files over Tomcat/Liferay
  • Installs Liferay patches/hotfixes/...
  • Copies all our custom WARs to the deploy directory
  • Removes the temporary directory where we unzipped the deploy package to
  • Starts the Tomcat server again 
Depending on the actual project we've added things like Liquibase upgrades, NewRelic installation & configuration, copying additional libraries to Tomcat's lib/ext, ... or just about anything we needed because the scripting system is so flexible. While it might not be as fancy as Docker and consorts it still accomplishes (except for the OS part) about the same result: a perfectly reproduceable deploy.
 

Conclusion

While working all this out in Maven I might have had the occasional, but typical, f*ck Maven (╯°□°)╯︵ ┻━┻ moment
 
 
but after figuring out all of this  JAR & WAR  monkeypatching combined with the deploy package system/script it ended up being quite flexible & powerful and made me one happy code monkey in the end!
 
All the example code from this post and more can be found on Github: https://github.com/planetsizebrain/monkeypatching.