Degraded performance of service call writing many items in document library

2021-03-11T17:07:31Z

In my module I import ZIP packages containg HTML files and their resources. My portlet/scheduler code takes ZIP packages one by one and call the custom service method which stores HTML content in database and resources in Document Library.

I found if this loop is moved directly to the service, the process is continuously slowed down with every iteration. I suppose data is written only after the service call succeeds, so when the loop is directly in the service method, all data are acummulated in some temporary storage what makes the process slower and slower.

I found it is especially apparent for operation with lot of resources so I assume the problems is somewhere in core LR PortletFileRepositoryUtil.addPortletFileEntry() method.

Here are durations when importing 13 different entries, but in same order using both approaches:

fast vs. slow (minutes) ============= 0:17 vs. 0:49 1:10 vs. 2:31 0:01 vs. 0:16 0:00 vs. 0:02 1:31 vs. 5:41 0:03 vs. 0:30 1:16 vs. 9:19 0:01 vs. 0:07 0:09 vs. 0:16 0:07 vs. 3:33 0:11 vs. 0:33 0:15 vs. 0:24 0:03 vs. 0:07

The fast method is finished in 10 minutes, while the slow in 30 minutes. It the number of entires is greater, the slowness is significant even more.

Here are results when same package with 1000 resources is imported three times in a row (slow method):

0:39 1:36 (+ 57 s) 2:29 (+ 110 s)

Fast variant:

Portlet/Scheduler/Rest App:

for (entry : entries) {
   myService.import(entry);
}

Service:

public void import(Entry entry) {
   addContentToDatabase(entry);
   addResourcesToDocumentLibrary(entry);
}

Slow variant:

Portlet/Scheduler/Rest App:

myService.importEntries(entries);

Service:

public void importEntries(List<Entry> entries) {
   for (entry: entries) {
      import(entry);
   }
}

That delay comes from addPortletFileEntry method:

try (InputStream inputStream = Files.newInputStream(fileSystem.getPath(resourcePath))) {
    String mimeType = MimeTypesUtil.getContentType(fileName);
    PortletFileRepositoryUtil.addPortletFileEntry(groupId, userId, className, classPK,
        portletId, folderId, inputStream, fileName, mimeType, false);
}

Is it possible to somehow keep the loop in service, but avoid that slowness? I've tried runing System.gc() in every iteration, but with no progress.

This behavior I originally found in 6.2, but it is still present in 7.3 versions. I use PostgreSQL database and standard file system for Document Library.