Why we built preupgrade verifiers

If you have ever run a Liferay upgrade on a large database, you know the worst-case scenario: the upgrade starts, runs for a long time, modifies a large amount of data, and then fails somewhere in the middle because of a misconfiguration that was already there before you pressed Start. At that point, you are not just fixing the original issue — you are also restoring the database (and sometimes the document library) to its pre-upgrade state so you can try again.

Preupgrade verifiers are a guard that runs before any database modification happens. If something about the environment will make the upgrade fail, the suite detects it up front and blocks the upgrade until the problem is fixed. Nothing has been touched yet, so there is nothing to restore.

The feature was introduced in Liferay DXP 2025.Q3. It was added to shift upgrade failures from late and destructive (halfway through the upgrade, with a partial schema left behind) to early and fixable (before the first DDL statement, with a clean database).

How the suite runs

Seven verifiers are executed in sequence before the real upgrade starts:

1. Company Users — every virtual instance has an administrator and a guest user.
2. Database Character Set — the database uses a supported character set, and on MariaDB/MySQL, all tables match the database configuration.
3. Database Privileges — the JDBC user can actually create, alter, and drop tables.
4. Database State — no tables are missing, no stale tables from a previous failed attempt, no missing columns.
5. Properties — portal and system properties have not been left behind in obsolete or migrated-away locations.
6. Store Access — the Document Library storage backend is reachable and writable.
7. Store Filesystem Structure — for filesystem stores, the on-disk layout matches the configured store implementation.

An important detail: all seven verifiers run regardless of earlier failures. When the suite finishes, every error message is collected and reported together, so a single run tells you everything to fix. You should not need to re-run the upgrade seven times to learn about seven different problems.

The rest of this post walks through every verifier in detail: what it checks, every error it can emit, and the exact steps to resolve each one. Use it as a reference while preparing your upgrade — when an error shows up, jump to the matching verifier, apply the fix, and re-run the suite until it passes clean. At the end of the post you will also find a short configuration reference for the properties that control the suite.

---

Verifier 1: Company Users

Classname: com.liferay.portal.verify.PreupgradeVerifyCompanyUsers

Verifies that every virtual instance (company) has both an administrator user and a guest (default) user. Both are essential for Liferay to function.

Error: No admin user was found for company {companyId}

What it means: The specified virtual instance has no user with the Administrator role. This can happen if the admin user was manually deleted from the database, or if the User_, Users_Roles, or Role_ tables contain inconsistent data.

How to fix:

1. Identify the affected company by the companyId in the error message.

2. Confirm the absence of an administrator by running the same query the verifier uses. On older Liferay versions (no type_ column), use:

   SELECT COUNT(*)
   FROM User_ u
   INNER JOIN Users_Roles ur ON u.userId = ur.userId
   INNER JOIN Role_ r ON ur.roleId = r.roleId
   WHERE r.name = 'Administrator'
     AND u.companyId = {companyId}
     AND r.companyId = {companyId};

   On newer Liferay versions (with the type_ column), add AND u.type_ = 1 (regular user) to the query. If the result is 0, the verifier will fail.

3. Restore an administrator user from a pre-upgrade database backup. This is the safest approach, since recreating role assignments manually is error-prone.
4. If no backup is available, manually assign the Administrator role to an existing regular user in the affected company by inserting a row into Users_Roles.
5. If the virtual instance is no longer needed, consider removing it from the Company table before retrying the upgrade.

Error: No guest user was found for company {companyId}

What it means: The specified virtual instance has no default / guest user. The guest user is used internally by Liferay for unauthenticated access and is required for the platform to operate.

How to fix:

1. Confirm the absence of a guest user by running the same query the verifier uses. The verifier adapts to the schema: if the defaultUser column exists on User_, it filters by defaultUser = TRUE; otherwise it filters by type_ = 0 (guest).

   -- Newer Liferay versions (defaultUser column exists):
   SELECT COUNT(*) FROM User_
   WHERE companyId = {companyId} AND defaultUser = TRUE;
   
   -- Older Liferay versions:
   SELECT COUNT(*) FROM User_
   WHERE companyId = {companyId} AND type_ = 0;

   If the count is 0, the verifier will fail.

2. Restore the guest user from a pre-upgrade database backup. Guest users are created automatically when companies are provisioned — a missing guest user usually indicates database corruption or partial data loss, and a backup is the safest recovery.
3. If the virtual instance is no longer needed, consider removing it from the Company table before retrying the upgrade.

---

Verifier 2: Database Character Set

Classname: com.liferay.portal.verify.PreupgradeVerifyDatabaseCharacterSet

Validates that the database uses a supported character set. On MariaDB/MySQL, it additionally checks for mixed character sets across tables.

Error: Unsupported database character set: {charset}

What it means: The database is using a character set that Liferay does not support. Liferay requires a Unicode-compatible character set (such as utf8 or utf8mb4) to store multilingual content correctly.

How to fix:

1. Check the current database character set:

   • DB2: SELECT value FROM sysibmadm.dbcfg WHERE name = 'codeset'
   • MySQL/MariaDB: SELECT @@character_set_database, @@collation_database;
   • PostgreSQL: SELECT pg_encoding_to_char(encoding) FROM pg_database WHERE datname = current_database();
   • Oracle: SELECT value$ FROM sys.props$ WHERE name = 'NLS_CHARACTERSET';
   • SQL Server: SELECT DATABASEPROPERTYEX(DB_NAME(), 'Collation');

2. Convert the database to a supported character set. For MySQL/MariaDB, the recommended character set is utf8mb4 with collation utf8mb4_unicode_ci:

   ALTER DATABASE your_database CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

3. After changing the database default, you may also need to convert existing tables:

   ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

4. Restart the upgrade process.

Warning: Mixed character set and collation (log only, does not block the upgrade)

What it means: Some Liferay tables in a MariaDB/MySQL database have a different character set or collation than the database default. For example, the database uses utf8mb4 but some tables still use latin1. This can cause sorting inconsistencies and join errors.

Log message format:

Mixed character set and collation: {tableName} has {charsetName} character set
and {collation} collation, but database has {defaultCharset} character set and
{defaultCollation} collation. Recommended character set is utf8mb4 and
recommended collation is utf8mb4_unicode_ci.

How to fix:

1. Identify affected tables from the log output.

2. Convert each table to match the database default:

   ALTER TABLE {tableName} CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

3. While this warning does not block the upgrade, it is strongly recommended to fix it before upgrading to avoid data issues.

---

Verifier 3: Database Privileges

Classname: com.liferay.portal.verify.PreupgradeVerifyDatabasePrivileges

Verifies that the JDBC database user has all required SQL privileges by performing a full DDL (Data Definition Language, e.g. CREATE/ALTER/DROP) and DML (Data Manipulation Language, e.g. INSERT/UPDATE/DELETE) lifecycle on a temporary table: CREATE, INDEX, ALTER, INSERT, UPDATE, SELECT, DELETE, and DROP.

Error: Database user is missing privileges: {sqlExceptionMessage}

What it means: The database user configured in portal-ext.properties / portal-upgrade-ext.properties (jdbc.default.username) does not have sufficient privileges to perform all operations required during the upgrade. The SQL exception message indicates which specific operation failed.

How to fix:

1. Identify the missing privilege from the SQL exception message in the error. Common failures include:

   • CREATE TABLE / DROP TABLE: the user lacks DDL privileges.
   • CREATE INDEX: the user lacks index creation privileges.
   • ALTER TABLE: the user lacks table modification privileges.
   • INSERT / UPDATE / DELETE: the user lacks DML privileges.

2. Grant the necessary privileges to the database user:

   MySQL/MariaDB:

   GRANT ALL PRIVILEGES ON your_database.* TO 'your_user'@'your_host';
   FLUSH PRIVILEGES;

   PostgreSQL:

   GRANT ALL PRIVILEGES ON DATABASE your_database TO your_user;
   GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO your_user;

   Oracle:

   GRANT CONNECT, RESOURCE, DBA TO your_user;

   SQL Server:

   ALTER ROLE db_owner ADD MEMBER your_user;

3. After granting the privileges, restart the upgrade process.

---

Verifier 4: Database State

Classname: com.liferay.portal.verify.PreupgradeVerifyDatabaseState

The most comprehensive verifier. It checks for missing tables, stale tables from a previous failed upgrade, and missing columns. This verifier is skipped if the database is new, already at the latest schema version, or if the current release state is not STATE_GOOD.

Error: Missing tables were detected: {tableNames}

What it means: Tables that should exist in the database (based on registered service components) are missing. This indicates that tables were manually dropped, a previous upgrade partially failed, or the database was restored from an incomplete backup.

How to fix:

1. Review the list of missing tables in the error message.

2. Determine the cause:

   • Manual deletion: if tables were intentionally removed, this indicates a data integrity issue. Restore them from a backup.
   • Partial upgrade failure: if a previous upgrade attempt was interrupted, some tables may not have been created. Restore the database to the pre-upgrade backup and retry.
   • Incomplete backup: ensure the backup contains all Liferay tables.
3. For DB Partition (multi-tenant database partitioning) environments, the error will also indicate the affected company: Missing tables were detected for company {companyId}: {tableNames}.

Error: Missing views were detected for company {companyId}: {viewNames}

What it means: In a DB Partition environment, non-default virtual instances access shared (control) tables through database views. This error means expected views are missing for the specified company partition.

How to fix:

1. This typically occurs when a virtual instance's database partition was not properly set up or was corrupted.
2. Verify the DB Partition configuration and re-create the missing views for the affected company.
3. If the views cannot be re-created from the partition configuration, restore them from a pre-upgrade backup.

Error: Stale tables were detected: {tableNames}

What it means: Tables exist in the database that belong to the target upgrade version, not the current version. This indicates a previous upgrade attempt partially ran and created some tables before failing. The database is in an inconsistent state.

How to fix:

1. Recommended approach: restore the database from the pre-upgrade backup and retry the upgrade from a clean state.

2. Alternative (advanced): if no backup is available, you can manually drop the stale tables listed in the error. Only do this if you are certain these tables were created by a failed upgrade and do not contain data from a successful prior upgrade:

   DROP TABLE {tableName};

3. After resolving, restart the upgrade process.

Error: Column {columnName} is missing for {tableName}

What it means: A column that should exist in the specified table is absent. This indicates the table schema does not match the expected schema for the current version. Possible causes include manual DDL modifications, partial migrations, or an incomplete restore.

How to fix:

1. Verify the expected column definition by checking the Liferay source code or service builder XML for the affected module.

2. Add the missing column manually:

   ALTER TABLE {tableName} ADD COLUMN {columnName} {columnType};

3. If multiple columns are missing, consider restoring the database from a known-good backup instead of manually adding columns.

Warning: Column {columnName} is not defined as {columnType} for {tableName} (log only)

What it means: A column exists but its data type does not match the expected definition. For example, a column might be VARCHAR(75) when VARCHAR(200) is expected. This is logged as a warning and does not block the upgrade, but it may cause issues during or after the upgrade.

How to fix:

1. Review the column definition in the database:

   -- MySQL/MariaDB:
   DESCRIBE {tableName};
   
   -- PostgreSQL:
   SELECT column_name, data_type, character_maximum_length
   FROM information_schema.columns
   WHERE table_name = '{tableName}' AND column_name = '{columnName}';

2. If the mismatch is significant, alter the column before upgrading:

   ALTER TABLE {tableName} MODIFY COLUMN {columnName} {expectedType};

---

Verifier 5: Properties

Classname: com.liferay.portal.verify.PreupgradeVerifyProperties

Checks for portal properties and system properties that have been migrated, renamed, made obsolete, or modularized between Liferay versions.

Error: Incorrect use of migrated portal properties keys: {keys}

What it means: Certain portal properties (configured in portal-ext.properties) have been migrated to system properties (configured in system-ext.properties or passed as JVM arguments). The upgrade cannot proceed until these are moved. The affected keys are typically module.framework.* and HTTP security-related properties.

How to fix:

1. Open your portal-ext.properties file.
2. For each key listed in the error, remove it from portal-ext.properties.
3. Add the equivalent property to system-ext.properties (placed in {LIFERAY_HOME}) or pass it as a JVM argument (-Dkey=value).

4. The migrated keys and their new locations are listed in the log. Common ones include:

   • module.framework.beginning.start.level
   • module.framework.runtime.start.level
   • module.framework.system.packages.extra
   • module.framework.web.start.level
   • module.framework.concurrent.startup.enabled
   • http.header.secure.x.content.type.options
   • http.header.secure.x.frame.options
   • cookie.http.only.names.excludes
5. Restart the upgrade process.

Warning: Renamed, obsolete, and modularized properties (log only, does not block the upgrade)

The verifier also logs warnings for other property issues that do not block the upgrade:

• Renamed portal properties: Portal property "{oldKey}" was renamed to "{newKey}" — update your portal-ext.properties to use the new key name.
• Obsolete portal properties: Portal property "{key}" is obsolete — remove the property from your configuration. It no longer has any effect.
• Modularized portal properties: Portal property "{oldKey}" was modularized to {moduleName} as "{newKey}" — the property is now configured through the module's OSGi (Open Service Gateway initiative) configuration (System Settings in the Control Panel) rather than portal-ext.properties.
• Migrated system properties: System property "{oldKey}" was migrated to the portal property "{newKey}" — move the property from JVM arguments / system-ext.properties to portal-ext.properties.
• Renamed system properties: System property "{oldKey}" was renamed to "{newKey}" — update the system property name.
• Obsolete system properties: System property "{key}" is obsolete — remove the property.
• Modularized system properties: System property "{oldKey}" was modularized to {moduleName} as "{newKey}" — configure it through the module's OSGi configuration.

While these warnings do not block the upgrade, it is recommended to clean up your configuration to avoid unexpected behavior after upgrading.

---

Verifier 6: Store Access

Classname: com.liferay.portal.verify.PreupgradeVerifyStoreAccess

Verifies that the Document Library storage backend is available and functional by creating, reading, and deleting a test file.

This verifier is skipped when:

• upgrade.database.dl.storage.check.disabled=true
• The database is new.
• The store implementation is DBStore (files are stored in the database, so there is no external storage to check).

Error: {DL_STORE_IMPL} is not available

What it means: The Document Library store OSGi service (configured by dl.store.impl in portal-ext.properties) could not be found. The storage backend is not running, or is not properly configured.

How to fix:

1. Check the dl.store.impl value in portal-ext.properties. Common values:

   • com.liferay.portal.store.file.system.FileSystemStore
   • com.liferay.portal.store.file.system.AdvancedFileSystemStore
   • com.liferay.portal.store.s3.S3Store
   • com.liferay.portal.store.db.DBStore

2. Verify that the corresponding store module is deployed and active. Check it from the OSGi console:

   lb | grep "store"

3. Ensure the store's configuration is correct (for example: S3 credentials, filesystem paths).

4. If the store is not needed during the upgrade, you can skip this check:

   upgrade.database.dl.storage.check.disabled=true

Error: Unable to create temporary file in store

What it means: The store service is available but cannot perform basic file operations (create and read). This typically indicates a permissions issue, a full disk, or an unreachable storage backend (for example: an S3 bucket with wrong credentials).

How to fix:

1. Filesystem stores: check that the Liferay process has read/write permissions to the rootDir directory configured for the store. Check available disk space.
2. S3 Store: verify AWS credentials, bucket permissions, and network connectivity.
3. Other stores: check the store-specific configuration and ensure the backend is accessible.
4. Review the server logs for additional details about the I/O failure.

---

Verifier 7: Store Filesystem Structure

Classname: com.liferay.portal.verify.PreupgradeVerifyStoreFileSystemStructure

Validates that the Document Library filesystem directory structure matches the expected layout for the configured store implementation (FileSystemStore or AdvancedFileSystemStore). This catches filesystem corruption or manual modifications that could cause the upgrade to fail.

This verifier is skipped when:

• upgrade.database.dl.storage.check.disabled=true
• The database is new.
• The store implementation is not FileSystemStore or AdvancedFileSystemStore.

Before troubleshooting: verify the correct store is configured

The most common root cause of "invalid structure" errors is an incorrect dl.store.impl value. The two filesystem stores use completely different directory layouts, and switching between them (or being misconfigured) causes the verifier to compare the filesystem against the wrong expected structure.

Before investigating individual errors, confirm which store is configured and ensure it matches the actual filesystem layout:

1. Check the current value:

   # In portal-ext.properties, or the module's System Settings:
   dl.store.impl=com.liferay.portal.store.file.system.FileSystemStore
   # OR
   dl.store.impl=com.liferay.portal.store.file.system.AdvancedFileSystemStore

2. Inspect the actual filesystem under the configured rootDir. If the structure does not match the configured store (see "Expected directory structures" below), the store configuration is wrong — not the filesystem.
3. Do NOT attempt to "migrate" files between the two layouts manually. If the store was changed at some point by mistake, revert dl.store.impl to match the existing filesystem layout.

Expected directory structures

FileSystemStore layout:

{rootDir}/
  {companyId}/              e.g. 20097/
    {repositoryId}/         e.g. 20201/
      {fileName}/           directory — MUST NOT contain "." (no extension)
        {versionLabel}      file — matches pattern "\d+\.\d+.*" (e.g., "1.0", "1.1.pdf")
                            OR starts with PRIVATE_WORKING_COPY_VERSION

AdvancedFileSystemStore layout (hash-based):

{rootDir}/
  {companyId}/              e.g. 20097/
    {repositoryId}/         e.g. 20201/
      {hash-dirs}/          directory names ≤ 2 characters (hash segments), nested recursively
        {fileName.ext}      file with an extension
      DLFE/                 special subdirectory (recursed into like a normal dir)

The key visible difference: FileSystemStore has directories named after file names (no extensions), while AdvancedFileSystemStore has directories with 2-character hash names and files with extensions.

Error: Unable to get root directory

What it means: The store OSGi service could not be located, or its rootDir property could not be read.

How to fix:

1. Confirm dl.store.impl matches a store module that is actually deployed. Run lb | grep store in the Gogo shell to list active store bundles.
2. Verify that the rootDir is configured in the store's OSGi configuration (System Settings > File Storage) or in portal-ext.properties.
3. Check the server logs for OSGi service registration errors.

Error: Root directory does not exist: {rootDir}

What it means: The configured root directory path does not exist on the filesystem.

How to fix:

1. Check the path reported in the error.
2. If the path is incorrect (typo, moved, wrong host), update the store configuration.
3. If the directory was accidentally deleted, restore it from a backup.
4. Ensure the directory is accessible from the server (check mount points, network drives, permissions).

Error: {companyIdPath} is not a directory

What it means: An entry directly under the root directory, named with a valid company ID, is a regular file instead of a directory. Company IDs must always be directories.

How to fix:

1. Navigate to the root directory and inspect the offending entry.
2. If it is a stray file accidentally placed there, move it elsewhere.
3. If a document library directory was mistakenly replaced by a file, restore the directory from a backup.

Error: Missing directories in {rootDirPath} for companies: {companyIds}

What it means: Every active virtual instance (company) registered in the Company table should have a corresponding directory under the document library root, named with its numeric company ID. This error lists company IDs that exist in the database but have no matching directory on disk.

Common causes:

• The document library directory was restored from a backup taken from a different environment (missing some companies).
• A virtual instance was created in the database but the document library directory was never provisioned (this can happen if the company was added while the filesystem was unreachable).
• The document library root is mounted from a different server or path than expected.
• Company directories were accidentally deleted or moved.

How to fix:

1. For each missing company ID, check whether the company is actually in use:

   SELECT companyId, webId, name FROM Company WHERE companyId IN ({companyIds});

2. If the company is still active, create the missing directory (it can be empty — Liferay will populate it as files are added):

   mkdir -p {rootDir}/{companyId}

   Ensure the Liferay process has read/write permissions to the new directory.

3. If the company is no longer needed, remove it from the database before retrying the upgrade. Deleting a company removes all of its data — make sure the virtual instance really is unused before removing it.
4. If the directory was accidentally deleted, restore it from a backup.

Errors: Invalid directory structure

Two "invalid structure" errors share the same root cause and fix — the filesystem does not match the layout expected by the configured store:

• Advanced file system store directory structure {path} is invalid
• File system store directory structure {path} is invalid

Each one is triggered by one or more of the following specific issues found during the scan (also shown in the log):

For AdvancedFileSystemStore:

• Unexpected file {path} in advanced file system structure — a regular file was found at a level where only directories are expected (repository level or hash-directory level).
• File {path} name has more than 2 characters and no extension in advanced file system structure — a subdirectory name inside the hash structure is longer than 2 characters and has no file extension. This typically means filename-style directories from a FileSystemStore layout exist where hash directories are expected.

For FileSystemStore:

• Unexpected file {path} in file system structure — a regular file was found where only directories are expected (repository or file-name level).
• Unexpected file name directory {path} with extension in file system structure — a directory name contains a period (e.g., mydoc.pdf/). File-name directories must not have extensions in this layout. This typically means file-with-extension entries from an AdvancedFileSystemStore layout exist where plain file-name directories are expected.
• Unexpected file {path} not matching version label pattern — a file inside a file-name directory does not match the pattern \d+\.\d+.* (e.g., 1.0, 1.1.pdf) and does not start with PRIVATE_WORKING_COPY_VERSION.

How to fix:

1. First, verify the store configuration is correct (see "Before troubleshooting" above). Many of these errors disappear when dl.store.impl is corrected to match the actual filesystem layout. Switching dl.store.impl between the two filesystem stores is NOT supported — if the filesystem was built for one, the other will not recognize it.

2. If the store configuration is correct, the filesystem has been modified in ways that do not match the expected layout. Review each reported path:

   • Files manually dropped into the document library root should be moved out.
   • Stray files left by a failed backup/restore, or by a test/debug session, should be removed.
   • If the structure is severely inconsistent, restore the document library directory from a known-good backup.
3. Do not attempt to manually reorganize files into the expected layout. Liferay stores filesystem paths and metadata in the database; moving files by hand will break file references.

Error: Unable to verify {advanced} system store directory structure {rootDirPath}

What it means: An I/O exception was thrown while walking the filesystem. The verifier could not complete the structure check.

How to fix:

1. Verify that the Liferay process user has read permissions on the entire document library directory tree.
2. Check for filesystem errors (run fsck or the equivalent for your platform).
3. If the document library lives on a network share, ensure it is mounted and reachable.
4. Review the stack trace in the server logs for the underlying I/O error.

---

Configuration reference

Property	Default	Description
upgrade.database.preupgrade.verify.enabled	true	Enables or disables the entire preupgrade verification suite. When set to false, all 7 verifiers are skipped.
upgrade.database.dl.storage.check.disabled	false	Disables the Store Access and Store Filesystem Structure verifiers.

Disabling preupgrade verification

If you fully understand the tradeoff and still need to bypass the suite, the properties below are the mechanism.

To bypass the entire preupgrade verification suite:

upgrade.database.preupgrade.verify.enabled=false

To bypass only the Store Access and Store Filesystem Structure verifiers:

upgrade.database.dl.storage.check.disabled=true

---

Wrapping up

Preupgrade verifiers exist to turn a painful, late-stage upgrade failure into an up-front, fixable checklist. Every error they emit points at something specific — a missing user, a privilege gap, a misaligned filesystem — that you can fix without ever having to restore from a backup.

The right way to use them is to treat every error the suite reports as a task to complete before the real upgrade runs. Disabling the suite, or the Document Library checks, is always an option, but it does not remove the underlying problem — it only hides it until the upgrade is halfway through your data and much harder to recover from.

If you are planning an upgrade, run the verifiers as early in the process as you can. The time you spend resolving their errors is time you will not spend recovering from a failed upgrade.

— Jorge

​​​​​​​