Recently, I was faced with the task of importing a large number of products from several warehouses and  product images into Liferay Commerce.  The upsert code itself to import the data was fairly straightforward, but the challenge was the sheer number of records inserted initially. The import was bound to be long-running, no doubt, and was going to run in the middle of the night anyway, but I wanted to find any way I could to "trim the fat." Let me share with you a few tips I learned from others and my own experience that helped me slim down the import.

A Little Bit of Background

Because importing each product into the Liferay Commerce catalog involves more than just 1 table, it was going to be a large number of records for each table. I couldn't really use some batch update tool that updates records in bulk or ActionableDynamicQuery (which doesn't do inserts, only updates, as far as I know), so I called the Liferay Commerce API which took care of inserting/updating all the right entities for me. It was an upsert-only operation, no deletions, to avoid the risk of the wrong data getting deleted. 

Importing the images from the warehouse was going to take the longest the first time they are ever downloaded, but, after that, they wouldn't need to be updated every time. Hot linking is a way to avoid the import altogether, but that's often banned by sites for their own good. 

(In case you're wondering, Talend ETL jobs were an option, but the decision ended up being writing our own import.)

Shorter Subsequent Imports

The first time the import is run should be the longest, and the subsequent imports should be shorter. (This tip doesn't apply if all the data changes frequently and must be updated every time the import is run.) See if you can find which data does not need to be updated. Some data doesn't change frequently, like an image of a product, so it doesn't need to be updated every day, for instance. Even if it does eventually need to be updated, you could rig it so that it forces it to update, but it's great if you can keep the import shorter most of the time.

Small Transactions