Integrating Liferay Commerce using an ETL (Part 3: Liferay Components for Talend Open Studio)

DISCLAIMER: This blog   post has Spanish and English version.

Finished the step of Part 2: Configuring Liferay Commerce in 3 steps

In the previous two blog posts, we worked with the Talend Open Studio and Liferay Commerce configuration. Today, we can start building the job that will perform the product upload to the portal and that can also be used for future product updates.

                                          ...before a brief explanation about the components of Liferay in Talend…

All Liferay Talend components are based on a connection and I/O with Liferay's Headless APIs, whether they are out-of-the-box (/o/ headless-commerce ...) or custom ones developed through RESTBuilder.

When using one of Liferay’s Talend components you select the specific endpoint you want to carry out I/O operations against. Some examples of valid API endpoints that can be used inside components are: 

  • /headless-commerce-admin-catalog/v1.0

  • /headless-commerce-admin-order/v1.0

List of Components

  • Connection Component
    Establishes the connection to the Liferay Portal server. You must set the root domain like "http://localhost:8080" or "http://yourserver.com" and it can be reused by all the other components, thus avoiding having to configure a separate connection for each component. 

  • Liferay Input Component
    It sends GET requests to the configured headless API, receives the data and converts it to the schema described by OpenAPI Specification and then passes it to the Talend stream.

  • Liferay Output Component
    It receives the data, converts it to the schema described by OpenAPI Specification and sends POST / PUT requests with data to the configured headless API.

  • Liferay Batch File
    It receives the records in the format described by the OpenAPI Specification schema and adds them to a batch file. 

  • Liferay Batch Output File

It sends the batch file to the endpoint "headless-batch-engine". This file must be generated by the previous component "Liferay Batch File".

Documentation about the components: https://github.com/liferay/liferay-portal/tree/master/modules/etl/talend

Catalog sync

We start with Talend Open Studio, MySql and Liferay Commerce (already installed and with the job opened if you followed Part 1 and 2 of this series).

1. Delete the component “tLogRow” that we already had in the project. 

2. In the “Palette” we look for the component “tMap” and we drag it to the Design area.


 

3. Change the query of “tDBInput” component to select * from products, so it fetches all product columns, as per in the image below (don’t forget the step to propagate changes which we saw in the first part of this series).


 

4. Now drag the red arrow from “tDBInput” component to “tMap” component.

5. Select “tMap” and basic configuration will appear, click on “Map Editor”

6. Check that everything is correct. It should look like the image below, with the “tMap” automatically including all the columns from our table. 

 7. Click on Ok.

Note: Through this component, we can process the data of the products that we will send to the Liferay Commerce catalog. As it can be seen in the image above, the input fields (all the columns of the DB) are already reflected as input in our tMap.

8. Search in the “Palette” the “tLiferayOutput” component and drag it to the Design area.


 

9. Double-click on “tLiferayOutput” and you will see the basic configuration; set the following data.

  • Liferay Host URL: include the hostname and port of your Liferay Portal server.
  • User ID: include a user that has the appropriate permissions for the headless API call (as here we are testing locally, if you do not want to register a user and configure it to access the API, you can use the Liferay Portal administrator user).

  • Password: include the password for the user above. 

  • OpenAPI Module Path: include the API endpoint. In our case, we’ll use "/headless-commerce-admin-catalog/v1.0".

10. After filling in the fields referenced above, click on the “Operation Path” button and select “/products” and then click on Ok.

11. Drag the arrow to the Liferay component and accept the message related to“get schema” when prompted.

                                                                                      ...We are almost there ...

12. We will need to define 4 additional “contexts” (fields) , for the extra data required by Liferay that does not come from our database. 


 

13. Double-click on the “tMap” component and you will see the mapping settings. Here we can map the fields that we want, especially those we saw in “SwaggerHub”. 

Note: you may find problems with the Talend components after opening and closing them. If you see an exclamation mark and a WARNING message saying that you need to install an external “jar.” (like in the image below)                                                                   ***Click on install, and write down the path of the dependencies that you will need.***


 

Here we finish the components configuration. You can now see how easy it is, it only takes 3 components: one for the data source, another for the field mapping and finally the "tLiferayOutput" component to send the data to Liferay Commerce.

We might wonder… if we run it, will it work? 

The answer is no, because although we connected the "tMap" component, we have not processed the data to send it to the format expected by the Headless API that we are calling ("/headless-commerce-admin-catalog/v1.0 ").

We will see how to do it and end this series in the next and final blog.

PS: Thanks to Bernard Mc Closkey (Commerce Solution Architect for EMEA) for helping me on the translation of this Blog post.