Improving Test Performance on Liferay

Recently, we upgraded the JSF Portlet Bridge Test Compatibility Kit (TCK) from Selenium 1.0 to our new test framework which uses Selenium 2.53. The TCK contains about 220 tests, and on Liferay 7.0 before we upgraded, it took around 8 minutes and 30 seconds to execute. In contrast, the TCK took 1 minute and 30 seconds to run on Apache Pluto, the portlet container reference implementation. In order to speed up test execution for Liferay Portal, we decided to try a few simple tricks and changes to reduce the total test time, and… we succeeded! Here’s a breakdown for how fast each browser ran the TCK on my machine before our upgrade, after our upgrade, and after we tweaked the TCK with some Liferay specific tricks:

Browser	Selenium `1.0` Test Time	Selenium `2.53` Test Time	Selenium `2.53` w/Exclusive Window State Test Time
HtmlUnit*	-	-	~00:00:45
Chrome	-	~00:04:30	~00:01:30
Firefox	~00:08:30 (w/FF v`21.0`)	~00:06:00 (w/FF v`46.0`)	~00:02:30
JBrowser* (experimental)	-	-	~00:02:30
PhantomJS*	-	~00:16:15	~00:10:30

* Headless browser.

As you can see, we made some pretty big improvements just by upgrading to a modern version of Selenium. Running tests with HtmlUnit also provided a nice boost over other browsers. Before I talk in-depth about HtmlUnit though, I want to begin by explaining the Liferay-specific tweaks and tricks that we used to speed up the TCK. These techniques are specific to Liferay but not to Selenium or HtmlUnit, so they may be useful in improving the performance of Liferay portlet tests regardless of the test framework or browsers used.

Exclusive Window State

The main performance boost is thanks to Liferay’s custom “exclusive” window state.¹ In Liferay, any portlet can be accessed in the exclusive state with a render URL containing the portlet’s id (p_p_id) and p_p_state=exclusive:

http://localhost:8080/my/portlets/page?p_p_id=myPortletName_WAR_myPortletWarName&p_p_lifecycle=0&p_p_state=exclusive

Liferay’s exclusive window state feature was created when Liferay only supported Portlet 1.0, and it provides similar functionality to the RESOURCE_PHASE of the Portlet 2.0 lifecycle. When Liferay Portal returns a portlet’s markup in the exclusive state, the markup contains only the portlet’s markup fragment without <html>, <head>, or <body> tags. Thankfully, every browser we use for testing handles the incomplete markup gracefully and renders the portlet testable. Providing the bare-bones portlet markup, the exclusive state doesn’t render any portal markup or frontend resources (such as CSS or JS). This greatly reduces the time it takes for a browser to load the page. The exclusive window state is useful for quickly testing backend functionality and rendering. I would not recommend testing portlets with complex JavaScript or CSS in the exclusive state. In fact, if you want to test a portlet with any external JavaScript resources at all, you will need to make sure that the resource’s script tag is rendered in the portlet’s body (<div>) markup rather than the portal’s <head> section (which isn’t rendered).² Certainly, using the exclusive state will not work for all tests, but it can provide significant speed improvements for certain tests.

Pop Up Window State

If you cannot easily move your portlet’s resources into your portlet’s body or if you rely on Liferay Portal’s frontend features (JS and/or CSS), try using the “pop_up” window state instead. In Liferay, any portlet can be accessed in the pop up state with a render URL containing the portlet’s id (p_p_id) and p_p_state=pop_up:

http://localhost:8080/my/portlets/page?p_p_id=myPortletName_WAR_myPortletWarName&p_p_lifecycle=0&p_p_state=pop_up

When a portlet is in the pop up state, the portal renders the <html>, <head>, and <body> tags including JavaScript and CSS resources. However, as with the exclusive state, the portal does not render any portal markup or portlets besides the one referenced in the URL. Even though using the exclusive state yielded greater performance benefits, using the pop up state still significantly sped up our tests (probably by about 25%).

Of course, neither of these two states should be used for testing portlets which may interact with each other, since they only render a single portlet. These states also don’t help if you are trying to test other window states (as we do in the TCK). However, while upgrading the TCK, we found another way you can speed up your tests if you are using Selenium.

HtmlUnit

Of all the Selenium compatible browsers that we test with, HtmlUnit is by far the fastest. And since HtmlUnit is headless, it can be run on a CI server without a window manager. HtmlUnit is not perfect. Its JavaScript engine has trouble running more complicated or cutting edge code, so I wouldn’t recommend it for testing newer browser features like the History API (which SennaJS uses) or HTML5. Nonetheless, it is an excellent browser for testing simple pages quickly. So how do you use HtmlUnit? Well if you are using Selenium, simply add the HtmlUnit dependency to your project (make sure you get the latest one):

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>htmlunit-driver</artifactId>
    <version>2.23.2</version>
    <scope>test</scope>
</dependency>

Then just change your WebDriver implementation to HtmlUnitDriver:

WebDriver webDriver = new HtmlUnitDriver();

The End.

Well… not really. HtmlUnit is a simple tool, but unfortunately configuring it to behave like every other browser is over-complicated and poorly documented. First, you will need to specify the following dependencies to avoid several NoClassDefFoundError/ClassNotFoundException issues:

<dependency>
    <groupId>xml-apis</groupId>
    <artifactId>xml-apis</artifactId>
    <version>1.4.01</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.eclipse.jetty.websocket</groupId>
    <artifactId>websocket-client</artifactId>
    <version>9.2.18.v20160721</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
    <scope>test</scope>
</dependency>

Second, I recommend disabling (or filtering) HtmlUnit’s incredibly verbose logging:

// Uncomment to enable HtmlUnit's logs conditionally when the log level is FINER or higher.
// if (logLevel.intValue() > Level.FINER.intValue()) {
LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log",
    "org.apache.commons.logging.impl.NoOpLog");
Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
Logger.getLogger("org.apache.commons.httpclient").setLevel(Level.OFF);
// }

Third, you should extend HtmlUnitDriver:

public class CustomHtmlUnitDriver extends HtmlUnitDriver {

Extending HtmlUnitDriver allows you to force HtmlUnit not to throw exceptions when JavaScript errors occur, and it allows you to silence CSS errors:

@Override
protected WebClient modifyWebClient(WebClient initialWebClient) {

    WebClient webClient = super.modifyWebClient(initialWebClient);

    // Don't throw exceptions when JavaScript Errors occur.
    webClient.getOptions().setThrowExceptionOnScriptError(false);

    // Uncomment to filter CSS errors.
    // if (logLevel.intValue() > Level.FINEST.intValue()) {
    webClient.setCssErrorHandler(new SilentCssErrorHandler());
    // }

    return webClient;
}

Fourth, you should note that HtmlUnit does not load any images by default,³ so any tests which require images to be loaded should manually call a method like the one below to load them:

public void loadImages() {

    HtmlPage htmlPage = (HtmlPage) lastPage();
    DomNodeList<DomElement> imageElements = htmlPage.getElementsByTagName("img");

    for (DomElement imageElement : imageElements) {

        HtmlImage htmlImage = (HtmlImage) imageElement;

        try {

            // Download the image.
            htmlImage.getImageReader();
        }
        catch (IOException e) {
            // do nothing.
        }
    }
}

Finally, always initialize HtmlUnit with JavaScript enabled (and emulate a popular browser):

WebDriver webDriver = new CustomHtmlUnitDriver(BrowserVersion.FIREFOX_45, true);

If you want a complete example of how we use HtmlUnit in Liferay Faces, see HtmlUnitDriverLiferayFacesImpl.java and Browser.java.

Hopefully, the advice in this post will help you speed up your tests in Liferay. If you’ve found any other tricks that improve your tests’ performance, please post them in the comments below.

Liferay Faces Team Lead, Neil Griffin, suggested using the exclusive window state to speed up tests. ↵
Liferay Faces Bridge automatically handles this case and moves JS and CSS resource markup into the portlet’s body (<div>) if the exclusive state is being used. ↵
Once HtmlUnit 2.25 and a compatible HtmlUnitDriver are released, you will be able to configure HtmlUnit to download all images automatically. ↵

0 Comments

Please sign in to comment.

Menu Display

Blogs

Category Filter

Improving Test Performance on Liferay

Exclusive Window State

Pop Up Window State

HtmlUnit

More Blog Entries

Liferay Design Patterns - Flexible Entity Presentation

Liferay IDE 3.1 Milestone 2 Released