Leveraging Maven Proxies

Taking a short break from the Vue.js portlet because I had to implement a repository proxy. Since I was implementing it, I wanted to blog about it and give you the option of implementing one too. Next blog post will be back to Vue.js, however...

Introduction

I love love love Maven repositories. I can distribute a small project w/o all of the dependencies included and trust that the receiver will be able to pull the dependencies when/if they do a build.

And I like downloading small project files such as the Liferay source separately from the multitude of dependencies that would otherwise make the download take forever.

But you know what I hate? Maven repositories.  Well, downloading from remote Maven repositories. Especially if I have to pull the same file over and over. Or when the internet is slow or out and development grinds to a halt. Or maintenance on the remote Maven repositories makes them unavailable for some unknown period of time.

Now, of course the Maven (and Gradle and Ivy) tools know about remote repository download issues and will actually leverage a local repository cache so you really only have to download once. Except that they made this support optional, and frankly it gets overlooked a lot. The default Liferay Gradle repository references don't list mavenLocal() in the repo list and it is not used by default.

Enterprise or multi-system users (like myself) have additional remote repository issues.  First, a local repo on one user's system is not shared w/ a team or multiple systems, so you end up having to pull the file to your internal network multiple times, even if it has already been pulled before.

A complete list of use cases would include:

  • Anyone developing behind an HTTP proxy (typical corporate users). Using a repository proxy removes all of the necessary proxy configuration from your development tools, the repository proxy needs to know how to connect through the HTTP proxy, but your dev tools don't.
  • Anyone developing in teams. Using a repository proxy, you will only pay for downloading to your network once, then all developers will be able to access the artifact from the proxy and not have to download on their own.
  • Anyone using an approved artifact list. Using a repository proxy populated with approved artifacts and versions, you have automatic control over the development environment to ensure that an unapproved version does not get through.
  • Anyone developing in multiple workspaces, projects, or machines. Shares the benefits from the team development where you only download once, then share the artifact.
  • Anyone who suffers network outages. Using a repository proxy, you can pull previously loaded artifacts even when the external network is down. You can't pull new artifacts, but you should have most of the ones you regularly use.
  • Anyone who needs to publish team artifacts. A repository proxy can also hold your own published artifacts, so it is easy to share those with the team and use each others artifacts as dependencies.
  • Anyone using a secured continuous integration server. CI servers should not have access to the interweb, but they still need to pull dependencies for builds. The repository proxy gives your CI server a repository to pull from without having direct internet access.
  • Anyone behind a slow internet link. Your internal network is typically always faster than your uplink, so having a local proxy cache of artifacts reduces or eliminates lag issues from your uplink.
  • Anyone interested in simplifying repository references. Liferay uses two repositories, a public mavenCentral mirror and a release repository. Then there's mavenCentral, of course, and there are other repositories out there too. By using a proxy, you can have a single repository reference in your development tools, but that reference could actually represent all of these separate repositories as a single entity.

So I'm finally giving up the ghost, I'm moving to a repository proxy.

What Is A Repository Proxy?

For those that don't know, a repository proxy is basically a local service that you're going to run that proxies artifact requests to remote Maven repositories and caches the artifacts to serve back later on. It's not a full mirror of the external repositories because it doesn't pull and cache everything from the repository, only the artifacts and versions you use.

So for most of us, our internal network is usually significantly faster than the interweb link, so once the proxy cache is fully populated, you'll notice a big jump when building new projects. And for team development, the entire team can benefit from leveraging the cache.

Setting Up A Repository Proxy

There are actually a bunch of freely-available repository proxies you can use. One popular option is Artifactory.  But for my purposes, I'm going to set up Apache Archiva. It's a little lighter than Artifactory and can be easily deployed into an existing Tomcat container (Artifactory used to support that, but they've since buried or deprecated using a war deployment).

The proxy you choose is not so important, it will just impact how you go about configuring it for the various Liferay remote repositories.

Follow the instructions from the Archiva site to deploy and start your Archiva instance. In the example I'm showing below, I have a local Tomcat 7 instance running on port 888 and have deployed the Archiva war file per the site instructions. After starting it up, I created a new administrator account and was ready to move on to the next steps.

Once you're in, add two remote repositories:

These two repositories are used by Liferay for public and private artifacts.  Order the liferay-public-releases guy first, then the public one second.

In Archiva, you also need to define proxy connectors:

Once these are set, you can then define a local repository:

Initially if you browse your artifacts, your list will be empty. Later, after using your repo proxy, when you browse you should see the artifacts.

And finally you may want to use a repository group so you only need a single URL to access all of your individual repositories.

Configuring Liferay Dev Tools For The Repository Proxy

So this is really the meat of this post, how to configure all of the various Liferay development tools to leverage the repository proxy.

In all examples below, I'm just pointing at a service running on localhost port 888; for your own environment, you'll just make necessary changes to the URL for host/port details, but otherwise the changes will be similar.

Liferay Gradle Workspace

This is handled by changing the root settings.gradle file.  You'll take out references to cdn.liferay.com and instead just point at your local proxy as such:

buildscript {
  dependencies {
    classpath group: "com.liferay", name: "com.liferay.gradle.plugins.workspace", version: "1.0.40"
  }

  repositories {
    maven {
      url "http://localhost:888/archiva/repository/liferay/"
    }
  }
}

apply plugin: "com.liferay.workspace"

Now if you happen to have other repositories listed, you may want to make sure that they too are pushed up to your repository proxy. No reason to not do so. And by using the repository group, we can simplify the repository list in the settings.gradle file.

This is the only file that typically has repositories listed, but if you have an existing workspace you might have added some references to the root build.gradle file or a build.gradle file in one of the subdirectories.

Liferay Gradle Projects

For standalone Liferay Gradle projects, your repositories are listed in the build.gradle file, these will also change to point at the repository proxy:

buildscript {
  dependencies {
    classpath group: "com.liferay", name: "com.liferay.gradle.plugins", version: "3.1.3"
  }

  repositories {
    mavenLocal()

    maven {
      url "http://localhost:888/archiva/repository/liferay/"
    }
  }
}

apply plugin: "com.liferay.plugin"

dependencies {
  compileOnly group: "org.osgi", name: "org.osgi.core", version: "6.0.0"
  compileOnly group: "org.osgi", name: "org.osgi.service.component.annotations", version: "1.3.0"
}

repositories {
  mavenLocal()

  maven {
    url "http://localhost:888/archiva/repository/liferay/"
  }
}

Global Gradle Mirrors

Gradle supports the concept of "init" scripts.  These are global scripts that are executed before tasks and can tweak the build process that the build.gradle or settings.gradle might otherwise define. Create a file in your ~/.gradle directory called init.gradle and set it to the following:

allprojects {
  buildscript {
    repositories {
      mavenLocal()
      maven { url "http://localhost:888/archiva/repository/liferay" }
    }
  }
  repositories {
    mavenLocal()
    maven { url "http://localhost:888/archiva/repository/liferay" }
  }
}

This should normally have all Gradle projects use your local Maven repository first and your new proxy repository second.  Any repositories listed in the build.gradle or settings.gradle file will come after these. This also sets repository settings for both Gradle plugin lookups as well as general build dependency resolution.

Liferay SDK

The Liferay SDK leverages Ant and Ivy for remote repository access. Our change here is to point Ivy at our repository proxy.

Edit the main ivy-settings.xml file to point at the repository proxy:

<ivysettings>
  <settings defaultResolver="default" />

  <resolvers>
    <ibiblio m2compatible="true" name="liferay" root="http://localhost:888/archiva/repository/liferay/" />
    <ibiblio m2compatible="true" name="local-m2" root="file://${user.home}/.m2/repository" />

    <chain dual="true" name="default">
      <resolver ref="local-m2" />

      <resolver ref="liferay" />
    </chain>
  </resolvers>
</ivysettings>

Liferay Maven Projects

For simple Liferay Maven projects, we just have to update the repository like we would for any normal pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  ...

  <repositories>
    <repository>
      <id>liferay</id>
      <url>http://localhost:888/archiva/repository/liferay/</url>
    </repository>
  </repositories>

Liferay Maven Workspace

The Liferay Maven Workspace is a new workspace based on Maven instead of Gradle. You can learn more about it here.

In the root pom.xml file, we're going to add our repository entry but we also want to disable using the Liferay CDN as the default repository:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  ...

  <repositories>
    <repository>
      <id>liferay</id>
      <url>http://localhost:888/archiva/repository/liferay/</url>
    </repository>
  </repositories>

  <properties>
    <liferay.workspace.default.repository.enabled>false</liferay.workspace.default.repository.enabled>
    <liferay.modules.default.repository.enabled>false</liferay.workspace.modules.repository.enabled>
  </properties>

Global Maven Mirrors

Maven supports the concept of mirrors, these can be local repositories that should be used in place of other commonly named repositories.  Create (or update) your ~/.m2/settings.xml file and make sure you have the following:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">

  <mirrors>
    <mirror>
      <id>repoproxy</id>
      <name>Repo Proxy</name>
      <url>http://localhost:888/archiva/repository/liferay</url>
      <mirrorOf>*</mirrorOf>
    </mirror>
  </mirrors>
</settings>

The <mirrorOf /> tag using the wildcard means that this repository is a mirror for all repositories and Maven builds will go to this repository proxy for all dependencies, build or otherwise.

Configuring Liferay Source For The Repository Proxy

We took care of all of your individual project builds, but what about if you have the source locally and want to build it using the repository proxy?

I actually combined a bunch of the previous listed techniques. For the maven portion of the build (if there is one), my settings.xml mirror declaration ensures that my repo proxy will be used. For the Gradle portion of the build, I used the init.gradle script (although I copied my ~/.gradle/init.gradle to the .gradle directory created inside of the folder as the ~/.gradle/init.gradle script was ignored).

In addition, in my build.<username>.properties file, I set basejar.url=http://localhost:888/archiva/repository/liferay.

And I also had to set my ANT_OPTS environment variable, so I used "-Xmx8192m -Drepository.url=http://localhost:888/archiva/repository/liferay".

With all of these changes, I was able to build the liferay-portal from master with all dependencies coming through my repository proxy.

You might be wondering why you would want to go through this exercise. For me, it seemed like an ideal way to pre-populate my proxy with most of the necessary public dependencies and Liferay modules. Sure this isn't necessary because, since we're using a proxy, if we need an updated version later on the proxy will happily fetch it for us. It's just a way to pre-populate with a lot of the artifacts that you'll be needing.

Publishing Artifacts

The other big reason to use a local repository service is the ability to publish your own artifacts into the repository. When you're working in teams, being able to publish artifacts into the repository so teammates can use the artifacts without building themselves can be quite valuable. This will also force you to address your versioning strategy since you need to bump versions when you publish; I see this as a good thing because it will force you to have a strategy going into a project rather than trying to create one in the middle or at the end of the project.

So next we'll look into the changes necessary to support publishing from each of the Liferay development environments to your new repository service.

Note that we're going to assume that you've set up users in the repository service to ensure you don't get uploads from unauthorized users, so these instructions will include the details for setting up the repository with authenticated access for publishing.

Finally, during the research for this blog post I have quickly come to find that there different ways to publish artifacts, sometimes even based on the repository proxy you're using. For example, the fine folks at JFrog actually have their own Gradle plugin to support Artifactory publication. I didn't try it since I'm currently targeting Archiva, but you might want to look it over if you are targeting Artifactory.

These instructions are staying with the generic plugins so they might work across repositories w/o much change, but obviously you should test them out for yourself.

Liferay Gradle Workspace

The Liferay workspace was actually the most challenging configuration out of all of these different methods if only because tweaking the various build.gradle files to support the subproject publication can take a while.

In fact, Gradle has an older uploadArchives() method that has since been replaced by a newer maven-publish plugin, but for the life of me I couldn't get the submodules to work right with it. I could get submodules to use maven-publish if each submodule build.gradle file had the full stanzas for individual publication, but then I couldn't get the gradlew publish command to work in the root project.

So the instructions here purposefully leverage the older uploadArchives() method because I could get it working with the bulk of the configuration and setup in the root build.gradle and minor updates to the submodule build.gradle files.

Add Properties

The first thing we will do is add properties to the root gradle.properties file. This will isolate the URL and publication credentials and keep them out of the build.gradle files. For SCM purposes, you would not want to check this file into revision control as it would expose the publishers credentials to those who have access to the revision control system.

# Define the URL we'll be publishing to
publishUrl=http://localhost:888/archiva/repository/liferay

# Define the username and password for authenticated publish
publishUsername=dnebing
publishPassword=dnebing1

Root build.gradle Changes

The root build.gradle file is where the bulk of the changes go. By adding the following content to this file, we are adding support for publishing to every submodule that might be added to the Liferay Workspace.

// define the publish for all subprojects
allprojects {

  // all of our artifacts in this workspace publish to same group id
  // set this to your own group or override in submodule build.gradle files
  // if they need to change in the submodules.
  group = 'com.dnebinger.gradle'
  
  apply plugin: 'java'
  apply plugin: 'maven'
  
  // define a task for the javadocs
  task javadocJar(type: Jar, dependsOn: javadoc) {
    classifier = 'javadoc'
    from 'build/docs/javadoc'
  }
  
  // define a task for the sources
  task sourcesJar(type: Jar) {
    classifier = 'sources'
    from sourceSets.main.allSource
  }
  
  // list all of the artifacts that will be created and published
  artifacts {
    archives jar
    archives javadocJar
    archives sourcesJar
  }
  
  // configure the publication stuff
  uploadArchives {
    // disable upload, force each submodule to enable
    enabled false
    
    repositories.mavenDeployer {
      repository(url: project.publishUrl) {
        authentication(userName: project.publishUsername, password: project.publishPassword)
      }
    }
  }
}

The instructions in this file will have you building a source jar and a javadocs jar. These and the build artifacts will all be published to the repository from the gradle.properties file using the credentials from that file.

Note that by default the upload is disabled for all subprojects. This forces us to enable the upload in those individual submodules we want to set it up for.

Submodule build.gradle Changes

In each submodule where we want to publish to the repo, there are two sets of simple changes to make.

// define the version that we will publish to the repo as
version = '1.0.0'

// change group value here if must differ from the one in the root build.gradle.

dependencies {
  ...
}

// enable the upload
uploadArchives { enabled true }

We specify the version for publishing and enable the uploadArchives for the submodule.

Publish Archives

That's pretty much it.  In both the root directory as well as in the individual module directories you can issue the command gradlew uploadArchives and (if you get a good build) the artifacts will be published to the repository.

Liferay Gradle Projects

Liferay gradle projects get a similar modification as the Liferay Gradle Workspace modifications, but they're a little easier since you don't have to worry about submodules.

From the previous Liferay Gradle Workspace section above, add the same property values to the gradle.properties file. If you don't have a gradle.properties file, you can create one with the listed properties.

Build.gradle Changes

The changes we make to the build.gradle file are similar, but are still different enough:

buildscript {
  dependencies {
    classpath group: "com.liferay", name: "com.liferay.gradle.plugins", version: "3.3.9"
  }

  repositories {
    maven {
      url "http://localhost:888/archiva/repository/liferay"
    }
  }
}

apply plugin: "com.liferay.plugin"
apply plugin: 'java'
apply plugin: 'maven'

repositories {
  mavenLocal()

  maven {
    url "http://localhost:888/archiva/repository/liferay"
  }
}

// define the group and versions for the artifacts
group = 'com.dnebinger.gradle'
version = '1.0.3'

dependencies {
  ...
}

// define a task for the javadocs
task javadocJar(type: Jar, dependsOn: javadoc) {
  classifier = 'javadoc'
  from 'build/docs/javadoc'
}

// define a task for the sources
task sourcesJar(type: Jar) {
  classifier = 'sources'
  from sourceSets.main.allSource
}

// list all of the artifacts
artifacts {
  archives jar
  archives javadocJar
  archives sourcesJar
}

// configure the publication stuff
uploadArchives {
  repositories.mavenDeployer {
    repository(url: project.publishUrl) {
      authentication(userName: project.publishUsername, password: project.publishPassword)
    }
  }
}

That's pretty much it.  Like the previous section, this build.gradle file supports creating the javadoc and source jars and will upload those using the same gradlew uploadArchives command.

Liferay SDK

In the Liferay SDK we have to configure Ivy to support the artifact publication. And actually this is quite easy because Liferay has already configured ivy to support their internal deployments to an internal Sonatype server, we just have to override the properties.

In your build.username.properties file in the root of the SDK, you need to just add some property overrides:

sonatype.release.url=http://localhost:888/archiva/repository/liferay/
sonatype.release.hostname=localhost
sonatype.release.username=dnebing
sonatype.release.password=dnebing1

That is your configuration for publishing. After building your plugin, i.e. using the ant war command, just issue the ant publish command to push the artifact to the repository. If you run ant jar-javadoc before the ant publish, your javadocs will be generated so they'll be available for publishing too. There's also an ant jar-source target available, but I didn't see where it was being uploaded to the repository, so that might not be supported by the SDK scripts.

One thing I did find, though, is that in each plugin you plan on publishing, you should edit the ivy.xml file in the plugin directory. The ivy.xml file has, as the first tag element, the line:

<info module="hello-portlet" organisation="com.liferay">

The organisation attribute is actually going to be the group id used during publishing so, unless you want all of your plugins to be in the com.liferay group, you'll want to edit the file to set it to what you need it to be.

I did check the templates and there doesn't seem to be a way to configure it.  The templates are all available in the SDK's tools/templates directory, so you could go into all of the individual ivy.xml files and set the value you want, that way as you create new plugins using the templates the default value will be your own.

Note that this only applies if you create plugins using the command line; I'm honestly not sure if you are using the Liferay IDE that the templates in the SDK folder are actually the ones the IDE uses for new plugin project creation.

Liferay Maven Projects

Liferay Maven projects are, well, simple projects based on Maven.  I'm not going to dive into all of the files here, but suffice it to say you add your repository servers, usually in your settings.xml file, and then you add to the pom file:

<distributionManagement>
  <repository>
    <id>archiva.internal</id>
    <name>Internal Release Repository</name>
    <url>http://localhost:888/archiva/repository/internal</url>
  </repository>
  <snapshotRepository>
    <id>archiva.snapshots</id>
    <name>Internal Snapshot Repository</name>
    <url>http://localhost:888/archiva/repository/snapshots</url>
  </snapshotRepository>
</distributionManagement>

With just this addition, you can use the mvn deploy command to push up your artifacts. Additionally, you can add support for publishing the javadocs and sources too: https://stackoverflow.com/questions/4725668/how-to-deploy-snapshot-with-sources-and-javadoc

Liferay Maven Workspace

For publishing purposes, the Liferay Maven workspace is merely a collection of submodules. This means that Maven pretty much is going to support the publication of submodules after you complete the configuration and pom changes mentioned in the previous Liferay Maven Projects section.

Normally the mvn deploy command will publish all of the submodules, but you can selectively disable submodule publish by configuring the plugin in the submodule pom: http://maven.apache.org/plugins/maven-deploy-plugin/faq.html#skip

Conclusion

This blog post ended up being a lot bigger than what I originally planned, but it does contain a great deal of information.

We reviewed how to set up an Archiva instance to use for your repository proxy.

We checked out the changes to make in each one of the Liferay development frameworks to leverage the repository proxy to pull all dependencies from the repository proxy, whether that proxy is Apache Archiva, Artifactory or Nexus.

We also learned how to configure each of the development frameworks to support publishing of the artifacts to share with the development team.

A lot of good stuff, if you ask me. I hope you find these details useful and, of course, if you have any comments leave them below and any questions, well post them to the forum and we'll help you out.

As a test, I timed the complete build of the https://github.com/liferay/liferay-blade-samples after deleting my ~/.m2/repository folder (purging my local system repository). Without the repository proxy, downloading all of the dependencies and completing the build took 69 seconds (note I have gigabit ethernet at home, so your numbers are going to vary from that). After purging ~/.m2/repository again and configuring for the repository proxy (pre-populated with artifacts), downloading all of the dependencies and completing the build took 45 seconds.

That is almost a 35% reduction in build time, and it means that 35% of the total build is taken to download artifacts even over my gigabit ethernet.

If you do not have gigabit ethernet where you're at, it would not surprise me to see that time to download increase, taking the % of reduction up along with it.