Dockerizing a Liferay Bundle, Part 2

Converting an Existing Bundle

In Part 1, I presented a brief overview of how the local file system can interact with the official Liferay Docker images. In this part, I will cover how to migrate or convert an existing bundle that is running on a local machine into a Docker image that is easily maintainable and distributable, with an eye towards DXP Cloud.

This is not a full tutorial on Docker, and as such, I will only be focusing on the parts relevant to Liferay Portal/DXP. In short, the Liferay images will recognize certain directories that can be mapped to the local file system and copy them into specific directories in the container. If we have an understanding of how the directories work, we can import content and migrate our local bundle into a Docker image/container.

Migration Steps

Before we start on the process, let's define some terms:
- ${LIFERAY_HOME} = the location of your Liferay bundle that you want to migrate
- ${TOMCAT_HOME} = the location of the Tomcat app server. Usually inside ${LIFERAY_HOME}
- ${DOCKER_PROJECT} = the location where you want to place your Docker files

Prerequisites

  • Docker installed
  • Basic working knowledge of Docker
  • Local Liferay bundle

1. Create your folder structure

${Docker Project}
    |---elasticsearch
    |    |---config
    |    |    |---docker-entrypoint-es-plugins.sh
    |    |    |---elasticsearch.yml
    |---mount
    |    |---deploy
    |    |---files
    |    |---patching
    |    |---scripts
    |---mysql
    |    |---config
    |    |    |---my.cnf
    |    |    |---lportal.sql
    |---docker-compose.yml
    

The mount directory is the one described in Part 1, and where the Liferay Docker image resides.

The Elasticsearch configuration files are a basic Elasticsearch installation and are not covered here. In short, they are what allow for the installation of the necessary plugins to use it with Liferay, and the configuration to connect into Liferay.

The MySQL configuration files are the database import and the configuration. the my.cnf contains my configuration of MySQL, namely the lower_case_table_names=1 setting. Obviously the lportal.sql file is the database I want to import. The file name must match the database specified in the portal-ext.properties file.

Sidebar

Why lower_case_table_name=1? For some reason, MySQL by default is case sensitive when installed on OSX or Linux, but not case sensitive when installed on Windows. I happen to run a Windows machine, but the Docker images are all Linux based. In Linux, Liferay will create a table like AnnouncementsEntry, but in Windows, it will be announcementsentry. Without the setting, if exporting from Windows into Linux, they will NOT be the same, and Liferay will make blank tables!

Why not just rename all the tables to be camel cased? NO. What's Rule #1 of Liferay development? You do not directly modify the database.

2. Setup the docker-compose.yml

Docker-compose is useful if you need to persist data and have multiple containers that need to work together. My default docker-compos.yml users Liferay DXP 7.2, MySQL 5.7, and Elasticsearch 6.8

Liferay (Sample)

  tomcat:
    image: liferay/dxp:7.2.10-sp2     
    container_name: tomcat
    depends_on:     
      - elasticsearch
      - mysql
    environment:
      LIFERAY_RETRY_PERIOD_JDBC_PERIOD_ON_PERIOD_STARTUP_PERIOD_DELAY: 10
      LIFERAY_RETRY_PERIOD_JDBC_PERIOD_ON_PERIOD_STARTUP_PERIOD_MAX_PERIOD_RETRIES: 10
      LIFERAY_JVM_OPTS: "-Xms2560m -Xmx4096m"
    networks:
      - fs72sp1
    ports:
      - 8080:8080
      - 127.0.0.1:11311:11311
    volumes:
      - ./mount:/mnt/liferay
      - liferay-data:/opt/liferay/data
      - liferay-osgi-configs:/opt/liferay/osgi/configs
      - liferay-osgi-marketplace:/opt/liferay/osgi/marketplace
      - liferay-osgi-modules:/opt/liferay/osgi/modules
      - liferay-osgi-war:/opt/liferay/osgi/war

This is the Liferay DXP section of the docker-compose, which I've called tomcat. The volumes section is important because we are mapping directories in our bundle that hold data and need them to persist across restarts. In the MySQL and Elasticsearch sections, we will also need volumes mapped as well. We want to persist the database store and index for those containers too.

In Part 1, I gave an example of Environment Variables, and in this snippet, we can see where they go and how they will be used.

I've included my full docker-compose.yml as a gist here for reference: https://gist.github.com/jonaschoi/200759a986baaa3affe37197ab949d56

Note that in the "volumes" section of the docker-compose.yml, I have listed the volumes from each of the images.

3. Identify what needs to be migrated

    - Database drivers
    - plugins
    - configs
    - themes
    - custom code
    - license file
    - document library store

    Let's start with the Liferay data directory. If you're using the default mappings for the data store, then the files will be located at ${LIFERAY_HOME}/data/
    There are some other directories in ${LIFERAY_HOME}/data that might be of use to transfer over:
        - document_library
       
    Next, let's look at the osgi directory. We'll move over our configs, themes, Marketplace plugins, custom plugins/modules, and legacy wars
        - configs (elasticsearch configuration)
        - marketplace (be careful here, see note below)
        - modules
        - war
        
    The osgi/marketplace directory contains lots of Liferay default modules, but also contains the deployed plugins that originate from the Liferay Marketplace. DO NOT just copy the whole directory. You need to identify the LPKG files that are from plugins downloaded, and place those in the new osgi/marketplace directory. I have the Fjord theme in my bundle, and so I’ve copied over the “Liferay Fjord Theme - Impl.lpkg” file. I will do the same with the rest of my other Marketplace plugins.
    
    Why not just deploy Marketplace LPKGs each time? You could. Some Marketplace plugins require you to restart each time you deploy it, which means due to impermanence of hot deployed plugins, will be gone after the restart, and you’ll need to deploy it again, which will ask you to restart again, etc... Additionally, if you're going to use something like DXP Cloud, you might run into an issue with file size on GitHub. The max file size on GitHub is 100 MB, and it's possible an LPKG is greater than that, so deploying it as is doesn't work.

The solution to both issues is to unzip the LPKG; you'll find that it's split into an API and an Impl LKPG file. If you place those into the mount/files/osgi/marketplace directory, you'll bypass the restart, and the GitHub file size issue. If you're deploying Commerce manually on DXP Cloud, you'll need to do this.

Once we have identified what needs to go over, we should replicate the file structure in ${DOCKER_PROJECT}/mount/files. As mentioned in Part 1, this is where we put everything we're migrating over. This is the equivalent of the Tomcat directory in a Liferay Tomcat bundle.
Inside the ${DOCKER_PROJECT}/mount/files, we should have:

mount
    |---files
        |---data
        |---osgi
        |---tomcat
        |---portal.ext.properties

And inside the osgi directory, we should have:

osgi
    |---configs
    |---marketplace
    |---modules
    |---war

This should look familiar, as it reflects the anatomy of a Liferay Tomcat bundle! We can simply copy over what we need from the respective local bundle directory to the Docker file system.

Checkpoint

Our directory tree should look something like this:

${Docker Project}
    |---elasticsearch
    |    |    config
    |    |    |---docker-entrypoint-es-plugins.sh
    |    |    |---elasticsearch.yml
    |---mount
    |    |---deploy
    |    |---files
    |    |    |---data
    |    |    |    |---document_library
    |    |    |    |    |---(All the doc lib store files)
    |    |    |---osgi
    |    |    |    |---configs
    |    |    |    |    |---com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config
    |    |    |    |---marketplace
    |    |    |    |    |---Liferay Fjord Theme - Impl.lpkg
    |    |    |    |---modules
    |    |    |    |    |---(DXP license key and custom jars)
    |    |    |    |---war
    |    |    |    |    |---(legacy WARs and themes)
    |    |    |---portal-ext.properties
    |    |---patching
    |    |---scripts
    |---mysql
    |    |---config
    |    |    |---my.cnf
    |    |    |---lportal.sql
    |---docker-compose.yml

The files I've listed here are examples of what I've moved over, but not the whole bundle.

4. Library and drivers

If you're using a database that's not Hypersonic (HSQL), then you need to move over your database driver JAR. You shouldn't be using HSQL for anything other than testing the setup or demoing. I'm using MySQL so I need to make sure my MySQL JAR gets copied into the Docker image.

If you've been paying attention to the directory structure, you might be able to figure out how the driver JAR is placed. Create the directory structure of files/tomcat/lib/ext and place the JAR in there.

Additionally, my bundle uses Xuggler, and I have to place the JAR in there to make it work. If I install Xuggler from the Control Panel, it will ask me to restart the bundle but the directory that the xuggler JAR is placed into is not one that is persisted across restarts. Which means I have to manually place the JAR. Since we're already in the tomcat dir, we'll make the following directory structure files/tomcat/webapps/ROOT/WEB-INF/lib.

Note, if you are hosting the local bundle on a Windows machine, and you have the Xuggler JAR, you need to get the Linux version for Docker. Xuggler has some code that runs outside of the JVM.

5. Optional directories

If you go to mount/deploy, that's the hot deploy directory. If you deploy something there, every time you restart the bundle, you'll have to redeploy the file. What if I want to deploy something every time I restart, but I don't want to do it manually?

You can put a deploy directory in the mount/files directory and that will copy over when you restart an image. In my instance. I placed my Liferay DXP activation/license keys in there, along with the Commerce key. They will be deployed/copied over each time. However, since you have 2 deploy directories, don't get confused about the two.

Checkpoint 2

Our directory tree now looks like this:

${Docker Project}
    |---elasticsearch
    |    |---config
    |    |    |---docker-entrypoint-es-plugins.sh
    |    |    |---elasticsearch.yml
    |---mount
    |    |---deploy
    |    |---files
    |    |    |---data
    |    |    |    |---document_library
    |    |    |    |    |---(All the doc lib store files)
    |    |    |---deploy
    |    |    |---osgi
    |    |    |    |---configs
    |    |    |    |    |---com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config
    |    |    |    |---marketplace
    |    |    |    |    |---Liferay Fjord Theme - Impl.lpkg
    |    |    |    |---modules
    |    |    |    |    |---(DXP license key and custom jars)
    |    |    |    |---war
    |    |    |    |    |---(legacy WARs and themes)
    |    |    |---tomcat
    |    |    |    |---lib
    |    |    |    |    |---ext
    |    |    |    |    |    |---mysql.jar
    |    |    |    |---webapps
    |    |    |    |    |---ROOT
    |    |    |    |    |    |---WEB-INF
    |    |    |    |    |    |    |---lib
    |    |    |    |    |    |    |    |---xuggler.jar
    |    |    |---portal-ext.properties
    |    |---patching
    |    |---scripts
    |---mysql
    |    |---config
    |    |    |---my.cnf
    |    |    |---lportal.sql
    |---docker-compose.yml

6. Importing the Database

We’ve managed to take care of the bundle, now we need to deal with the database. Let’s take a look at the docker-compose for MySQL

  mysql:
    image: mysql:5.7.29
    container_name: mysql
    networks:
      - fs72sp1
    ports:
      - 3307:3306
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: lportal
    volumes:
      - ./mysql/config/lportal.sql:/docker-entrypoint-initdb.d/lportal.sql
      - ./mysql/config/my.cnf:/etc/mysql/conf.d/my.cnf
      - mysql:/var/lib/mysql

Nothing special here, we’ve specified the same network as the Tomcat image, and in the environment variables, we’re specifying the database name and the root password. In the volumes section, we’re mapping the directories that MySQL uses to persist their data, and where to put our SQL file for import. I have a my.cnf in there as well, because in my instance, I’m importing a MySQL 5.6 database into a 5.7 instance, and that requires a special flag, and this bundle originated in a Windows instance, which requires a lower case table name flag.*

If you look at the file structure trees from the Checkpoints, you’ll see that the SQL file has gone into ${Docker Project}/mysql/config, and then we’re taking that file and copying it into the docker-entrypoint for the MySQL instance. This will have the MySQL container import the SQL file when it first starts, and since we’ve mapped the volume, it will persist the info across restarts, unless we delete the volumes.

That should be it! Our formerly on-prem bundle's data is in the lportal.sql ready to be imported into the MySQL image. Our document library files are mapped to be copied into the container when it starts. Our plugins, configs, modules, themes, drivers, etc... are also ready to be copied in.

As mentioned before, the beauty of a docker-compose is you can set multiple images to work together, and set some to be dependent on others. In my docker-compose, I've set the tomcat to be dependent on Elasticsearch and MySQL to be up and running before it starts. That way I'll have my data imported and my index is ready to go before DXP starts.

To start the system, go to ${DOCKER_PROJECT} in the terminal or command line and use the command docker-compose up with all the logs or docker-compose up -d to have it run detached.

* See Sidebar up above

Some management notes

Shutdown and resets

With this setup, data and changes made to the bundle will persist across restarts of the container images. However, it can be reset to the base data that was imported. This is good if you have a demo that you want to reset after each time you use it, and don’t want to manually revert changes.

The docker-compose command to stop the containers now has importance.
docker-compose down means the containers will stop and starting them again will have no impact on the data.
docker-compose down -v means the containers will stop and the volumes will be deleted. When the containers are restarted, it will go through the import process again

Because of this distinction, the docker-compose down -v command essentially becomes our reset button. If you reset, there is no undoing the reset.

Patching

My Docker setup is for Liferay DXP, so I get access to hotfixes and fixpacks from the Liferay Support team. To apply either on a local bundle, I would use the included patching-tool utility to have it apply my hotfixes or fixpacks. That’s not feasible with a Docker container, so there has to be another way.

Remember the “patching” directory in the file tree? Place the hotfix zip files there and the image will do the rest. Note that it will do this every time the container is restarted, so don’t have too many.

For fixpacks, you could put hotfixes into the patching directory, but you’ll need to have the “patching-files.zip” which is an additional 1 GB to patch along with it, which is not ideal.

If you want to apply a fixpack, the best way is to change the underlying image to be that fixpack! Because the way everything is imported, it takes the baseline image, and then adds all the stuff on top of it, we can just change the base fixpack and then apply all our stuff on top of it. Of course, this doesn’t work for upgrades, but it’s a quick way to move between Service Packs.

To change the image, we look at the docker-compose.yml file and change the “image” tag. Please refer to the snippet above. To know what tags are valid, look at the Docker Hub page and the tags.

Example

This process was used to create the bundles here: https://github.com/jonaschoi/se-projects/tree/master/docker-demos

This is my GitHub repository where I have loaded all the necessary files for these demo bundles. I am providing them as reference and example. Since I used DXP bundles, a license key will be required to use these bundles. I chose to use GitHub as a distribution method because it was a way that I could guarantee whomever had access to the repository would always have the latest build.

Onward to DXP Cloud!

So why did I go through all this trouble to write this, migrate this bundle and blather on about the whole process? Liferay DXP Cloud. If you're not interested in DXP Cloud, that's fine, and I hope this has proven useful to you in understanding migrating an on-premise bundle into a Docker image that you can run on your own cloud servers or infrastructure.

When you are provisioned with the GitHub repos on DXP Cloud, you'll get a file structure like this:

lctmyrepos
    liferay
    |---configs
    |    |---common
    |    |---dev
    |    |---dr
    |    |---infra
    |    |---local
    |    |---prd
    |    |---uat

These are the various environments that DXP Cloud provides to you to use. In this case, the common, dev, prd, and uat directories are the equivalent to the "mount" directory for the Docker image. Think of all of these as the "mount" dir for the specific environments!

In what is technically Part 3, I’ll go over the process I used to move this bundle into DXP Cloud.