Introduction

Let me start by saying that I'm a huge fan of Upgrade Processes. I've actually written many blogs about Upgrade Processes or using Upgrade Processes:

• https://liferay.dev/blogs/-/blogs/old-versus-new
• https://liferay.dev/blogs/-/blogs/thinking-outside-of-the-box-resources-importer
• https://liferay.dev/blogs/-/blogs/servicebuilder-and-upgrade-processes
• https://liferay.dev/blogs/-/blogs/building-in-upgrade-support

In my regular consulting activities, I also recommend and often use Upgrade Processes to set up environments in a consistent fashion, populate data, etc.

Liferay 7.3 (+) changed the default handling of upgrade processes to not auto-run on deployment which, to me, is a blow to how I use them. But I think it's important to share the reasoning why the change was made as well as to cover what options we have.

Upgrade Challenges

Liferay actually started this process of disabling auto upgrade process execution some time ago, and it was first introduced for ServiceBuilder. In 7.0, updates to the service.xml to add entities, update entities, etc. were not updated automatically as they used to in prior versions of Liferay.

This, of course, was challenging to developers because historically SB changes were automagically applied and the developer didn't need to do anything to get the service changes applied.

This process of auto-applying ServiceBuilder DDL changes was actually quite problematic. You can actually get yourself into trouble making changes to the database while the server (or nodes, if you're in a cluster) is processing incoming requests.

If I want to add a new column to an existing table, this seems pretty easy, all I need to do is execute the ALTER TABLE ADD COLUMN ... command. But how are your nodes supposed to react? Will this new column have additional support in the Liferay SB layer to set initial or default values? Can a non-updated node insert a record into this table? Would that record be valid after the upgrade was applied?

And what if we're trying to change column types or remove a column? Incoming requests being processed by the cluster could fail drastically only because they weren't quite updated.

So originally the idea was that the Upgrade Process would be used to apply these kinds of DML changes rather than auto-changing from the SB code. The Upgrade Process could handle not just making the DML changes, but also apply logic such as initializing a new column or handling the column datatype changes through a conversion, ...

This change worked mostly in single-server environments, but in a cluster where you might be doing rolling deployments to keep serving traffic, this solution would fail. I might have updated nodes 1 and 2 in the cluster, but if nodes 3 and 4 are up serving traffic and try to insert, fetch or update rows in the table via the older SB code, you still end up with a failure at least and possibly data corruption at worst.

Yes, data corruption. Imagine if we are adding a column to a table. We create the upgrade process to add the column and then perhaps it does some logic to come up with the value for the new column based off of data in the other columns. The last thing that happens when this Upgrade Process runs, the version in the Release table is updated so this Upgrade Process never runs again. In our 4 node cluster example, however, imagine 3 and 4 are busy inserting new records into the database, but they still haven't been updated so they are not populating this new column. They eventually will get updated, but what about those rows they were able to insert? Since the Upgrade Process completed and updated the Release table, there is no reason for it to run again and fix those rows that would have a null value instead of the computed value the Upgrade Process assigned to existing rows. So yeah, you now have a data corruption in your environment just because you were trying to do a rolling deployment.

And this kind of data corruption, well it's not easy to identify or solve. You'll see no errors in the log, because there aren't any errors (until you try to use that new column value). It will be a sporadic issue, since most of the records were updated and it is only the ones added by nodes 3 and 4 that would not have the value populated, so it may be hard to consistently reproduce. It can be hard to resolve, the code to determine the value is in the Upgrade Process so maybe it requires a new Upgrade Process to apply the same logic to rows with a null in the new column. And ultimately there's the question of who is responsible, is it a Liferay product issue or was it a deployment issue or a developer issue?

To prevent these kinds of issues occurring, as of 7.3+, Liferay will not auto-run Upgrade Processes during deployment.

Running Upgrade Processes

Since the auto-run of Upgrade Processes was disabled, what are our options to get them to run?

Well, for developers, we can add the upgrade.database.auto.run=true to portal-ext.properties or just include-and-override=portal-developer.properties will get the same value, but this is not recommended in production.

For non-development environments, you have two choices:

1. Use the db-upgrade-client tool to apply the upgrade processes.
2. Use Gogo to execute the upgrade processes.

Both of these cases require you to deploy your modules first. The db-upgrade-client tool would then be used in an offline mode, and Gogo would be used in an online mode.

A Blow to How I Use Upgrade Processes

As I've blogged about in the past, I use Upgrade Processes for all kinds of things, not just for actual upgrade work. I've shared how I use them to create an initial set of roles, for example, and for Resources Importer how I could load specific content resources via an upgrade process. Lately I've been using an Upgrade Process to create custom fragments (so I can ensure that each environment will have the same, consistent fragment set), clean up data (using an ActionableDynamicQuery to select multiple matching rows and apply a specific transform to them unrelated to an actual DB upgrade), and even to update configuration to auto-change some values.

So for how I use upgrade processes, disabling the auto-run is really a setback. I can still use the Upgrade Processes in the same way as I did, but I now have to either use the db-upgrade-client offline or use the Gogo commands to apply the upgrades.

Programmatic Invocation

So I don't really want to do all of this manual nonsense to run my upgrades. I don't want to just enable everything because, well, I don't have any control over what Upgrade Processes Liferay might have provided or that someone else is working on...

In reviewing the Gogo commands, I see there is one that looks like what I need, upgrade:execute [module_name] which takes the bundle symbolic name as the argument.

This is kind of awesome, because I know that Gogo commands are just Java classes that have annotations on them and they're also OSGi components. With either of these, it could be a way for me to get a reference to a service and invoke the execute method.

I tracked down the Gogo command, it is implemented in the com.liferay.portal.upgrade.internal.release.osgi.commands.ReleaseManagerOSGiCommands class.

I'm immediately disheartened. See that "internal" there in the package? That means Liferay didn't export any of these packages, so I can't use them in OSGi. I even checked the command implementation to see if I could export the logic and use externally, but no joy. It's an internal command that uses internal services to internally execute Upgrade Processes.

Now I could use my Fragment Bundle Package Exporting Trick to export the packages so I can use them, but this is kind of tedious.

So my next thought is that since they are Gogo commands, I just need a CommandProcessor instance and I can invoke the commands through there.

First I needed the right OSGi dependency. Liferay uses a variation on the Felix Gogo Runtime, so you can either use the official version or Liferay's patched version:

compileOnly group: "com.liferay", name: "org.apache.felix.gogo.runtime", 
  version: "1.1.0.LIFERAY-PATCHED-2"

In my class, I needed the imports for the Gogo CommandProcessor class:

import org.apache.felix.service.command.CommandProcessor;
import org.apache.felix.service.command.CommandSession;
import org.apache.felix.service.command.Converter;

import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;

I needed a CommandProcessor reference, so I created a Component and added the reference dependency:

@Reference
private CommandProcessor _commandProcessor;

In my component, I planned to add an @Activate method that would merely be a call to upgradeBundle("my.bundle.symbolic.name") (or maybe I include the Bundle parameter so I'm given the BSN) so I needed to define that function basically as:

private void upgradeBundle(final String bundleSymbolicName) throws PortalException {
  CommandSession commandSession = null;
	
  InputStream emptyInputStream = null;
  UnsyncByteArrayOutputStream outputUnsyncByteArrayOutputStream = null;
  UnsyncByteArrayOutputStream errorUnsyncByteArrayOutputStream = null;
  PrintStream outputPrintStream = null;
  PrintStream errorPrintStream = null;
	
  try {
    emptyInputStream = new UnsyncByteArrayInputStream(new byte[0]);
		
    outputUnsyncByteArrayOutputStream = new UnsyncByteArrayOutputStream();
    errorUnsyncByteArrayOutputStream = new UnsyncByteArrayOutputStream();
		
    outputPrintStream = new PrintStream(outputUnsyncByteArrayOutputStream);
    errorPrintStream = new PrintStream(errorUnsyncByteArrayOutputStream);
		
    commandSession = _commandProcessor.createSession(emptyInputStream, 
      outputPrintStream, errorPrintStream);
		
    // invoke the gogo upgrade:execute command and provide the BSN
    Object result = commandSession.execute("upgrade:execute " + bundleSymbolicName);
		
    if (result != null) {
      outputPrintStream.print(commandSession.format(result, Converter.INSPECT));
    }
		
    errorPrintStream.flush();
    outputPrintStream.flush();
		
    String errorContent = errorUnsyncByteArrayOutputStream.toString();
		
    if (Validator.isNotNull(errorContent)) {
      throw new Exception(errorContent);
    }
		
    String outputContent = outputUnsyncByteArrayOutputStream.toString();
		
    if (Validator.isNotNull(outputContent)) {
      _log.info(outputContent);
    }
  } catch (Exception e) {
    _log.error("Error upgrading bundle: " + e.getMessage(), e);
    throw new PortalException(e);
  } finally {
    // close all of the resources...
  }
}

So now, when my bundle is deployed, the @Activate annotated method will get invoked, and in this method I use the same Gogo command I'd need to invoke the Upgrade Process, but I do it in code, automatically. I'm only invoking my own bundle's upgrades, so other bundles won't be touched.

The beauty of this kind of technique is that we get the benefit of triggering the real Liferay upgrade process handling without having to do any of the heavy lifting ourselves, and since it uses the real Liferay code, it would be the same as if I executed the Gogo command manually.

Conclusion

So the most important takeaway from this blog post should be the following:

We learned why, in fact, Liferay ended at this position.

We also reviewed the methods that Liferay provides for executing the Upgrade Processes.

Finally, we closed with a code-based solution which will allow our modules to basically auto-run their own Upgrade Processes themselves.

And who knows, maybe this is how we should process our own upgrades. Liferay's choices with respect to auto-run is based upon how Liferay uses Upgrade Process, maybe not how we might use them ourselves.

Ah, well. Enjoy! Leave me a note below, let me know how this works out for you, which side of the "should Liferay auto-run upgrade processes" you're on, or even just to say Hi!