Migrating Your Bundle to DXP Cloud (Part 3?)

Going from Docker to DXP CLoud

This could be considered Part 3 of "Dockerizing a Liferay Bundle" but has very little to do with Docker, with the exception that DXP Cloud uses Docker images. I split this in to a separate section because not everyone is interested in DXP Cloud.

This will focus more on converting a Liferay DXP instance using a Docker image to DXP Cloud. These steps will work as well for taking a Tomcat (on-premise type)  bundle and moving it to DXP Cloud as well.

Prerequisites

  • Liferay DXP Tomcat bundle, or Docker image
  • tar utility (or git bash in Windows)
  • DXP Cloud account/access
  • cURL utility
  • git utility

DXP Cloud Preparation

  1. Provision the DXP Cloud environment
  2. Read the Welcome e-mail
    1. This contains all the necessary information required to access the DXP Cloud Console and provisioned services
  3. Clone the provided GitHub repository.
    1. Clone, not fork. The provided repos is deleted after 10 days, and forking it will mean when the original repos is deleted, the fork will be deleted as well.
    2. Connect the cloned repository to DXP Cloud.
      1. Directions on connecting the provided Jenkins service with the GitHub repository can be found here.
  4. Check it out to the local file system.

Why Docker Bundles? What about from a regular bundle?

As we will see in the process of migrating, having migrated to using an official Liferay Docker image means most of the work has been done for us, in terms of the migration. It collects all the files we need in one location. If coming from a regular on-premise type bundle, we will essentially need to duplicate the work to migrate to a Docker image. Whether the intermediate step of using a Docker image is done or not, following the process in the Docker migration blog entry will provide the preparation to migrate into DXP Cloud.

To find out more information about using the official Liferay Docker images, please refer to my previous 2 blog entries, Part 1 and Part 2. You can also look at the whitepaper about Liferay and Docker here. Most of my work on Docker was done via self-exploration.

Migrating to DXP Cloud

If the original bundle is a Docker demo, then all of these files are already gathered in one location and ready to migrate.

The cloned repository will look something like this

lctmyrepos
    |---backup
    |---ci
    |---database
    |---liferay
    |---search
    |---webserver

For the migration operation, we will concern ourselves with the Liferay section

lctmyrepos
|---liferay
|    |---configs
|    |    |---common
|    |    |---dev
|    |    |---local
|    |    |---prd
|    |    |---uat

Also present in the liferay directory is the gradle directory. This is also a Liferay Workspace, which means we can directly put code that can be built in this location, but that’s a different blog entry all together.

Within this directory, we have common, dev, local, prd, and uat directories. Common is for items that will be deployed to all environments. Dev, prd, and uat are for specific deployments into those environments. Local is for the creation of a local Docker image.

For this example, we will use the dev environment.

Copy the files from a Docker image over

If you have followed the prior blog entry about converting an existing bundle to a Docker image, it’s just copy/paste.

  1. Copy the contents of ${DOCKER_PROJECT}/files/mount to lctmyrepos/liferay/configs/dev
    1. If going to uat or prd, use the corresponding directory.
  2. There are some things you can exclude
    1. data directory, as that will be handled later
    2. osgi/configs/elasticsearch config, as that is already done
  3. Add everything to the git commit, write a message, and push it to GitHub.
  4. Deploy your build to the environment once Jenkins has finished testing it and it's available.
    1. For how to deploy a build, please refer to DXP Cloud docs here.

That takes care of all our portal-ext.properties file, plugins, deployments, license keys, themes, etc… Yes, it’s just a simple copy/paste. We just need to upload out data (document library store and database).

This is just for the plugins that you are not building from source. If you are building from source, put that code into the Liferay Workspace! Simply go to the liferay directory, and then create a modules and war and themes directory. When all the files are committed to GitHub, the DXP Cloud Jenkins services will build everything and deploy it.

Why not copy in the data directory with GitHub?

You could. However if you have an uploaded document that’s over 100 MB, then you’re out of luck. GitHub has a 100 MB file size limit. Essentially, you’d be restoring the backup every single time you restart the Liferay service, and potentially overriding any changes that have been made in the meantime. The DXP Cloud backup restore method shown below works for all situations and will persist across restarts.

Accepted File Formats

DXP Cloud wants the archives to be in *.tgz. It MUST be *.tgz. It does not recognize *.tar.gz as an acceptable file format, and while it says it is OK with *.zip, it’s not. It threw an error trying to work with it as a *.zip file.

Archive the document library store

  1. In ${DOCKER_PROJECT}/mount/files/data, take the document_library directory and tar it, then gzip it to make document_library.tar.gz
  2. Change the file extension to *.tgz

Archive the Database schema file

  1. The database export MUST be a MySQL database export. If you are coming from a different database vendor, there are conversion tools from different vendors.
    1. I strongly suggest the schema be called something other than “lportal” If the existing database is not called “lportal” then it’s OK. If it is, then it needs to be exported as something else, and must set that name automatically when importing.
    2. If the database schema is called “lportal” there may be issues when it tries to import into DXP Cloud’s MySQL service.
  2. Export the database as a file
    1. Command to export mysqldump -u root -p --databases --add-drop-database lportal | gzip -c | cat > database.gz
      1. For the import/restore to succeed, it requires a specific set of commands in the SQL file, hence the flags above.
      2. Replace "lportal" with your schema name.
  3. Tar the *.gz file into a *.tgz file
    1. Command tar zcvf database.tgz database.gz

For Windows Users

Windows does not have tar as a native command, but can be done via 7-Zip, or using Linux commands via the Git Bash interface.

Since the default DXP database name is “lportal”, there may be issues with importing a new database to overwrite the old one. Having it as not “lportal” is beneficial. In my testing, I used a different file name than “lportal”

Set Database Flag for Windows Schema (Optional)

Recall in Part 2 we had to set a database flag to set things as lower case table names in the Docker image. We have to do the same thing here, except we won't be doing it via a config file. The reason for doing so is the same. If your SQL file originated from a Windows machine, you need to do this.

  1. Go to lctmyrepos environment -> Services -> Database - Environment Variables
  2. Add a new Environment Variable called LCP_GCP_DATABASE_FLAG_LOWER_CASE_TABLE_NAMES
    1. All database flags that we can set will start with LCP_GCP_DATABASE_FLAG_

    2. For valid flags, please refer to Google Cloud SQL docs here.

  3. Set the value to 1


 

Upload the document repos and database to DXP Cloud

  1. Place the two tgz files in the same directory.
  2. Use the following command
Linux/OSX
curl -X POST \
https://backup-${REPO_NAME}-prd.lfr.cloud/backup/upload \
 -H 'Content-Type: multipart/form-data' \
 -F 'database=@database.tgz' \
 -F 'volume=@document_library.tgz' \
-u email@address.com:password
Windows
curl -X POST ^
 https://backup-myinstance-prd.lfr.cloud/backup/upload ^
 -H "Content-Type: multipart/form-data" ^
 -F "database=@database.zip" ^
 -F "volume=@document_library.zip" ^
 -u email@address.com:password

In both instances, the \ or ^ can be dispensed with and everything written in one line. Note that in Linux/OSX, it uses single quotes, whilst in Windows, it uses double quotes.

Note that only the PRD environment of the DXP Cloud instance will have the backup capability, so the POST URL will always be to the PRD environment.

In the examples of the cURL command, I am using email address and password login to DXP Cloud. However, DXP Cloud offers the ability to use some sort of token authentication via SSO. If you are logging in to DXP Cloud via SSO or some kind of token authentication method, then you will need to supply the token instead of the “-u email@address.com:password”

3. Wait
The backup should appear in the console momentarily. If not check the Backup service logs.

Deploy the backup

  1. Go to the DXP Cloud PRD environment and the Backups section
  2. The new backup should be present.
    1. Look for timestamps and file size
  3. Use the 3-dot menu to restore the backup to an environment
    1. This will remove ALL data
    2. You can deploy to any of the environments, not just PRD.

Change the Database Name Secret

  1. Once the services have restarted, change the database used. (Must be Admin to do this)
    1. Go to the restored environment’s Liferay Service
    2. Click on “Environment Variables”
    3. Click on “lcp-secret-database-name”

  1. Click on “Edit” in the upper right
  2. Show, then change the database name secret to what the database dump was exported as.

  1. Restart the Liferay service.

As suggested earlier, the database was exported as not “lportal” and would be imported as such. Therefore, the Liferay service must be configured to use the new database name.

Troubleshooting the Restore Process

If the backup doesn’t appear in the Backup page, then look in PRD -> Services -> Backup and examine the log. If there are any error messages, examine them and then correct it. Most likely, it is something to do with the files that were uploaded. Fix it and re-upload.

If the Backup Service log doesn’t show anything, then the problem may lie in the database schema. If you see a message that says “Unable to delete lportal” or something like that, then you didn’t heed my warning.

Backing Up a DXP Cloud deployment

DXP Cloud’s PRD environment has an automated backup process and by default, will take daily backups of PRD.

Once we have something in DXPC, we may want to pull a local backup of some kind or even just back it up for posterity (like a demo).

DXP Cloud offers numerous mechanisms to let us accomplish this. First, there is “local” directory in the GitHub repository, which has a way of making it run as a local Docker image. I haven’t been able to figure that out yet, so when I do, that can be a part 4.

Much like pushing a backup into DXP Cloud, when pulling a backup, we only need the database and the document library. However, we will need to grab the backup name from the DXP Cloud Backup Service.

Option 1: Downloading the backup

  1. Go to PRD environment -> Backup
  2. Find the desired backup
  3. Use the 3-dot menu to download the backup.

Option 2: Using cURL to pull the backup

  1. Go to Services -> Backup -> Logs
  2. Find a line of the log that mentions the Backup ID and copy it
    1. It will look like dxpcloud-wwwrylxxjsngqicrva-202011181908
  3. Find the appropriate timestamp in the backup section and replace the string above with the timestamp number
    1. dxpcloud-aoeusnthaoeusnth-${TIMESTAMP}
      1. This string is an internal project ID and doesn’t change for the DXP Cloud instance.
      2. Timestamp values can be found in the Backup page and is the string of numbers.
  4. Use the following cURL commands, with ${BACKUP_ID} as the adjusted string.
    1. Document library volume: curl -X GET https://backup-{$REPO_NAME}-prd.lfr.cloud/backup/download/volume/${BACKUP_ID} --output volume.tgz -u email@address.com:password
    2. Database backup: curl -X GET https://backup-{$REPO_NAME}-prd.lfr.cloud/backup/download/database/${BACKUP_ID} --output database.tgz -u email@address.com:password

So far, this is all for the PRD environment in DXP Cloud. What if we want to pull a backup of the other environments? Currently, there is no official way to pull backups of those environments. Numerous people have found some creative ways of pulling backups of DEV or UAT by using the console to create the backup files and then have Tomcat or nginx serve them for download. However, they aren’t officially supported methods!

In Conclusion

What I've outlined here may change in the future. DXP Cloud is changing and evolving, but the migration steps on what needs to be put where and what should be the same. The process may change in the future to make the restore process easier or simpler than it is now. Rumor has it that there will be an official migration tool, to automate everything I just wrote. If that's the case, then great, but until then, this is what we have to do.

Looking Forward

I mentioned the fact that there is a local directory in the repository, and it's for creating a local Docker image of the DXP Cloud build. Maybe this will be a Part 4, but it could be off in the future.