Down with Web Contents, Long Live Objects!

Liferay has new techniques for capturing, storing and rendering custom structured content.

Introduction

I was recently in a meeting reviewing some FreeMarker templates that were extracting web contents using a structure, parsing and processing the data, and rendering an output. Basically the implementation was kind of the classic or "legacy" way of doing specialized presentation of structured contents in Liferay.

In this case they had a Carousel implementation, so they had a web content structure with the fields for the image, the title, the summary, a link, etc. They also had the web content template in FM that would render the carousel card, then they also had an Asset Publisher w/ another template that could render the DOM for the carousel, plus there was additional theme JS/CSS that were necessary to make it all work together.

While this works, it does present a number of challenges. First and foremost, it absolutely requires developer resources to handle the FM templates and anything going on inside of the templates. Additionally, I'm just cringing at all of the FM use in general because I know that it's interpreted and also that it is probably doing too darn much and will negatively impact performance.

Additionally, the web content is hard to use from a headless perspective. I mean, sure you can get the article, but the fields may be a lot harder to get to, plus you can't do things like request three fields from the content, you like always get them all.

When you have content creators with different roles, i.e. some are allowed to create Carousel records but others are not, you need to permission your structures and templates correctly so the users who can't create Carousel records can't access the structures, but this in turn can cause issues if they are just browsing the site and it has the structured contents on it...

There's just so much not to like about structured web contents, especially given Liferay's adoption of low code/no code as a way to empower business users to be able to create and maintain the content pages.

When you're on a later version of Liferay, there actually are better ways to do this kind of thing. Come along with me while we build this out. I know that at times you're going to be like "Why is he doing that?" or "Why is he using this?", but I encourage you to follow along and do the same things and, when we get to the end and we're discussing pros/cons, it will all kind of make sense...

Challenge

So the challenge we're going to tackle here is building a Carousel in Liferay. I can hear it now, "I already have at least one carousel, why do I need another?"

The point here is not to present just another carousel, the point here is to present an alternative way to capture, store and render custom structured content. A carousel is just a convenient meme that everyone will understand.

It is taking the new and better way to approach structured data and putting this way to use in a UI element that everyone can understand.

Defining the Carousel

So our carousel is going to have the following fields:

  • A background image
  • A title
  • A summary text
  • A call to action URL

So our carousel will have all of the typical stuff, there will also be a call to action button where we can select a page to navigate to if the user clicks the button.

We could add more things on here like a "read more..." link and, you know, other custom structure fields as necessary to define what will be displayed.

Defining the Carousel Object Definition

If you're familiar with the classic way of handling structured content, you're probably already navigating to the Site menu, Content & Data, Web Content to create the structure.

But you'd be wrong. We're actually going to head to the Waffle, Control Panel, Objects so we can define our Object there.

We're actually going to use an Object for the structured data instead of a Web Content. We'll save the full list of whys for the conclusion of the article, but at this point the only thing that matters is that we can create an Object that contains all of the fields we need for rendering.

Before we go further, a bit of setup...

So I'm using DXP Q4.6 release, this should equate to CE GA 102. Newer should work, even older may work, but UIs may change and feature set, etc.

Additionally, I've navigated to the Waffle, Instance Settings, Feature Flags and have enabled the following feature flags. These will change the UI from the default views, so you'll want to enable these settings also. In newer or older versions these flags may be permanently enabled or promoted/demoted or not available at all. For the most part this shouldn't matter much as the blog doesn't pivot on these specific changes but it will give you a look at the things to come...

Oh and I've also edited the images to remove the irrelevant items, so don't worry if your list doesn't match this one exactly, just enable the same ones selected below.


Since I like to stay a bit organized, the first thing I'm going to do is create an Object Folder named Structured Contents. This is where I would create different types of objects that represent structured contents that I plan on displaying.

In the Structured Contents folder, I'm going to create a new object called Carousel Entry.

Once my entry is created, and since it's my first one, my list is pretty simple:


Click into the Carousel Entry object to define it further.

The very first decision we need to make on our object is the Scope. We have two options, we can pick Company (aka Instance) and this would be akin to making the carousel entry available to all sites (basically same as creating web content in the global site), or we can pick Site and the carousel entries would only be visible in the site that we define them for.

In my implementation, I'm going with Site because each site would have individual content creators and they each have different things they are displaying in their carousels, so its a good choice for me. In other implementations or for other structured contents, the choice may be better to use the Company scope. Just be sure to pick the right one now since we can't easily change it in the future.

Now, you might say that you need both, maybe some that are shared and others which are not... There are ways that you can do this (i.e. define a CompanyCarousel and a SiteCarousel and pull the data together using a Blueprint search or via API builder, etc), but those implementation details are beyond the scope of this blog post...

With Site for my scope, I'm also going to set a panel link as Site Administration > Content & Data because that's the obvious place to have a control panel for Carousels, yeah?

So here's another benefit for using Objects - you get specific control panels for each one. The classic way, you have to go to Web Content, maybe navigate into a folder, click the Add button, pick the right structure, then you can start setting the fields. With content objects like this one, a site admin can just go to the Carousel Entries control panel and add new entries there.

After setting the scope and the panel link, I'm also going to enable the Entry History in Audit Framework option because I want to track the changes made to the entries.

Actually I planned on enabling object translations, and the screenshots here in the blog reflect that. However, I encountered a bug in Liferay that translated fields in site-scoped objects cannot be mapped. Since I was more interested in finishing the blog than I was on solving the defect, I opened a bug report, deleted my object and re-created the object but without the translations enabled. Maybe by the time you are reading this, translation mapped fields will be fixed and you can proceed as I started, but if you hit the same issue as me, at least you'll know what the cause was.

Click the Save button (not the Publish button), then go to the Fields tab to define the fields for the carousel entries.

I've jumped ahead and added my 4 fields. They are pretty straight forward:


The only one that needs more explanation is the Background Image guy. It's type is Attachment, and for the Select Files dropdown I selected the Upload or Select from Documents and Media Item Selector. There's not much other configuration you can do directly in the form, but once you add the field and then click on it in the list, the right-side fly out gives you a few more options. I've edited mine as follows:


By limiting the file types to images, I know that I'm only going to allow the user to select image files. I've also set the max file size to 0 because even though I'd prefer images of 100MB or less, I just can't guarantee that all images will conform to that limitation.

We have what we need now for the Carousel Entry object, so we can go to the Details tab and now Publish the Object Definition.

In my own implementation, I did take a moment before publishing to define a default Layout (in the Layouts tab) and View (in the Views tab). These aren't really necessary, but they will be used by the generated Control Panel, so defining these will let you control the presentation. You can add this before publishing the object or come back and add after publishing the object. Just remember that the default Layout and View will be used by the Control Panel.

At this point we have defined an Object that represents what our structured data represents, in this case it is an individual carousel entry. We also have a control panel entry that content admins can use to define new carousel entries, so we have the data entry partially covered.

The next part we need to address is rendering the carousel with our new carousel entry objects.

Building The Carousel Display

The cornerstone of the carousel display is going to involve using a Collection Display fragment on a content page. The Collection Display fragment is bound to a collection (or a collection provider) and typically handles the rendering of each item in the collection, but also is done using fragments and mappings to fields within the item (in this case a Carousel Entry) to display.

What we want to display is a standard sort of Carousel, so the large background image, highlighted title and summary detail, plus a "read more" type button to get to further detail.

Now to keep things simple (from an implementation perspective), I am going to use https://swiperjs.com/ as the foundational implementation. This will require using a hierarchy of divs with assigned classes...

To make this happen, we'll use a Header, a Paragraph, and a Button fragments to display the title, summary and call to action. In addition, we'll need a container fragment for rendering the individual carousel entry (Swiper calls them slides), (will have the assigned background image and necessary Swiper classes), and an outer container fragment that will wrap the collection display and pull all of the Swiper stuff together...

So basically I'll need two custom fragments, one for the container at the item level, and the other is the outer container that controls the carousel. I'll start with the item-level container...

Creating the Carousel Slide Container

So we're going to create a custom fragment! So I navigated to Design -> Fragments, then I created a new Fragment Set titled Structured Content Fragments, and in there I created a new Basic Fragment which I called the Carousel Slide.

The Carousel Slide HTML content was really all that I defined for the fragment, and it consists of:

<div class="swiper-slide carousel-slide"
    data-lfr-background-image-id="carousel-slide-background-img">
  <lfr-drop-zone></lfr-drop-zone>
</div>

So our <div /> has two classes assigned, one is the swiper-slide which is used by SwiperJS, the other is carousel-slide which can be used to style the content of the slide itself.

The <div /> also has a background image assigned, and using the data-lfr-background-image-id attribute will allow us to map to an image selected when the fragment is used. With this in place, when we are setting up the fragment in the UI we'll be able to map to the image file that is assigned in the Carousel Entry.

The <lfr-drop-zone /> tag inside of the <div /> is what turns the div into a container; we're declaring that any fragment can be dropped in the drop zone, so once this container is used on a page we'll be able to drop the Header, Paragraph and Button fragments into the container.

In fact, I created a Carousel page and then started adding the Carousel Slide fragment and dropped the Heading, Paragraph and Button fragments inside of the container. It doesn't look pretty (at the moment) and doesn't have the mapping enabled, but it certainly has the structure and items that we'll be needing:

Creating the Carousel Container

With the carousel slide out of the way, we can now move on to the carousel container itself.

It's the responsibility of this container to set up and configure SliderJS to work on the content, and even though it is a container it will only hold the Collection Display fragment.

The Collection Display fragment will be bound to the collection, then we'll be using the Carousel Slide container for each item in the collection to display.

So again I go to Design -> Fragments and in the Structured Content Fragments fragment set I created a new Carousel basic fragment. Like the Carousel Slide fragment, the HTML for the Carousel fragment is pretty simple:

<div class="swiper carousel">
  <div class="swiper-wrapper" id="swiperWrapper">
    <lfr-drop-zone></lfr-drop-zone>
  </div>
  <div class="swiper-button-next"></div>
  <div class="swiper-button-prev"></div>
  <div class="swiper-pagination"></div>
</div>

Unlike the Carousel Slide fragment, the Carousel fragment has some JS associated with it:

const editMode = document.body.classList.contains('has-edit-mode-menu');

if (!editMode) {
  const script = document.createElement('script');
  script.type = 'text/javascript';
  script.src = 'https://cdn.jsdelivr.net/npm/swiper@11/swiper-bundle.min.js';
  document.head.appendChild(script);

  const link = document.createElement('link');
  link.rel = 'stylesheet';
  link.href = 'https://cdn.jsdelivr.net/npm/swiper@11/swiper-bundle.min.css';
  document.head.appendChild(link);

  script.onload = () => {
    // eslint-disable-next-line
    var swiper = new Swiper(".carousel", {
      spaceBetween: 30,
      effect: "fade",
      navigation: {
        nextEl: ".swiper-button-next",
        prevEl: ".swiper-button-prev",
      },
      pagination: {
        el: ".swiper-pagination",
        clickable: true,
      },
    });
  };
} else {
  document.getElementsByClassName('swiper-wrapper')[0].style.zIndex = -1;
}

This script first figures out if currently in edit mode. When in edit mode, the z index for the div is modified, but otherwise nothing happens.

The magic is when not in edit mode. The JS and the CSS for Swiper are loaded, then the Swiper is initialized for the <div /> with the carousel class. When not in edit mode, the Carousel should work as expected. In edit mode, the controls will be easier to access and configure because the Swiper will not be activated, it will just render as a normal Collection Display fragment.

Building the Carousel Page

We now have all of the pieces that we need to make the Carousel so we are ready to create the page.

Before creating the page, I did go to Content & Data -> Carousel Entries (the control panel I get by setting the panel link in the object definition) and I created a few entries. Google was a great help in finding some free banner images which I downloaded and set on each of the carousel entries.

I then went to the Site Builder -> Pages and created a new Carousel page from the Blank Page template.

With the page in edit mode, I first dropped a Carousel fragment on the page, within that I dropped and configured a Collection Display fragment to use the Carousel Entries collection provider, and in the Collection Item drop zone I dropped the Carousel Slide fragment, mapping the background image to the Carousel Entry's background image, and then I dropped a Header, Paragraph and Button fragments into the Carousel Slide. I mapped the Header to the Title, the Paragraph to the Summary, and the Button Link to the Call To Action URL.

Thus my carousel was complete so I published the page.

And... nothing. The published page had my stuff on it, the DOM elements were there, I even verified that Swiper JS and CSS were being loaded...

I then took a look at some of the sample Swiper demos, and I found that their DOM was like:

<div class="swiper mySwiper">
  <div class="swiper-wrapper">
    <div class="swiper-slide">
      ...
    </div>
    <div class="swiper-slide">
      ...
    </div>
    <div class="swiper-slide">
      ...
    </div>
    <div class="swiper-slide">
      ...
    </div>
  </div>
  <div class="swiper-button-next"></div>
  <div class="swiper-button-prev"></div>
  <div class="swiper-pagination"></div>
</div>

When I checked the DOM for what my collection display fragment was using, I had <div /> inside of <div /> inside of <div />, many more levels than what SwiperJS was expecting.

So I changed the javascript on the Carousel fragment to be:

const editMode = document.body.classList.contains('has-edit-mode-menu');

if (!editMode) {
  const script = document.createElement('script');
  script.type = 'text/javascript';
  script.src = 'https://cdn.jsdelivr.net/npm/swiper@11/swiper-bundle.min.js';
  document.head.appendChild(script);

  const link = document.createElement('link');
  link.rel = 'stylesheet';
  link.href = 'https://cdn.jsdelivr.net/npm/swiper@11/swiper-bundle.min.css';
  document.head.appendChild(link);

  script.onload = () => {
		
    var slides = document.getElementsByClassName('carousel-slide');
    var slidesHtml = '';
		
    for (let item of slides) {
      slidesHtml = slidesHtml + item.outerHTML;
    }
		
    var wrapper = document.getElementById('swiperWrapper');
    wrapper.innerHTML = slidesHtml;
		
    // eslint-disable-next-line
    var swiper = new Swiper(".carousel", {
      spaceBetween: 30,
      effect: "fade",
      navigation: {
        nextEl: ".swiper-button-next",
        prevEl: ".swiper-button-prev",
      },
      pagination: {
        el: ".swiper-pagination",
        clickable: true,
      },
    });
  };
} else {
  document.getElementsByClassName('swiper-wrapper')[0].style.zIndex = -1;
}

This allowed all of the fragments to be rendered, but then I found all of the carousel-slide divs, concatenated their DOM, then overrode the innerHTML on the swiperWrapper element, basically removing all of the extra <div />s that the collection display was using that I no longer needed.

After saving and propagating the fragment, my page now looked like:


Sure, it still needs a bit of style applied to it (beyond what I had done in my testing), but the carousel itself works, the forward/backward arrows work as does the pagination dots at the bottom of the page.

So while it's not quite ready for production, it is certainly close enough to prove the thing is doable.

Stop or Continue?

So my implementation looks good, so now the question becomes, should I stop here or continue on?

Consider:

  • My Carousel Entry collection provider returns all of the entries defined for the site. As the content creator, I can use the generated control panel to add new entries, but I'd also have to use it to delete the entries that should no longer be in the carousel. I could add a boolean checkbox to the Carousel Entry to display the entry or not, then add a special Blueprint Collection Provider (enabled via a feature flag) that would return all Carousel Entries that had the boolean checked.
  • I only allow for a single carousel per site, but maybe different pages would need different carousels. I could add a key field to the Carousel Entry, plus a Blueprint Collection Provider to filter for specific keys. That way my carousel would be tied to a specific key and that key would match a set of Carousel Entries that had the same key.
  • Content creators would need to have access to the control panel to get to the Carousel Entries panel. I can use a regular old Form Container bound to the Carousel Entry object on a protected page so content creators could manage the Carousel Entries without navigating to the control panel at all.
  • In fact, I could use a "content creator" segment to add list and form views under the carousel so they could make changes in the page itself, but the segment wouldn't be available to others, they'd only see the carousel.
  • As objects, the Carousel Entries are available via the Headless API generated for the object, also available via the Headless Batch API, are subject to the new Datasets and Data Migration Center features in Liferay, so I have a long list of options available to support export/import and content promotion/demotion between environments.
  • Objects are subject to Workflow and Publications, so these tools can be leveraged to review and approve content prior to going live.

Given all of these opportunities (and probably others which I've overlooked), although I'm choosing to stop at this point, there's no reason why you couldn't continue and build out the solution you need...

Conclusion

So granted this Carousel was a contrived solution, but generally it is a good representation of what is possible if you can get away from web content structures and focus on Objects for your structured content needs.

Doing so offers the following advantages:

  • You can get a specific control panel for your structured content, making creating and maintaining these a lot easier because they are permissioned separately and don't push every solution into web content and the web content display.
  • Eliminates the need to use FreeMarker, instead leveraging the Collection Display fragment and other fragments with field mappings to handle a customized layout but created visually without support from developers.
  • Because I get a complete Headless API for my Carousel Entry objects, I can easily access, create, update and delete entries without having to jump through the structured web content hoops.
  • Impressive number of export/import options to support moving Objects between environments where traditional Liferay tools (i.e. LAR export/import) tend to fail easily and often.
  • Using form container fragments, you can move your content creators out of the control panel entirely, giving them either generated or handmade (well, handmade in the UI, not handmade as in writing code or JSPs or what not) interfaces for maintaining content outside of the control panel.

By leveraging Objects and custom fragments, it is possible to envision a world where, instead of using OOTB Liferay entities such as blogs, wikis, etc, perhaps you build them as Objects and custom fragments for rendering?

Or taking what you previously would do in a web content structure with FreeMarker web content templates and reworking them as Objects/Fragments?

Before you start down this road, it is important to consider the disadvantages/shortcomings:

  • Objects lack some of the field types (and custom field types) that you might define for a web content structure.
  • Objects lack the kind of composition support you now have with Web Content Structures and Field Sets. Objects can have relationships, but the lack of a 1:1 relationship is still a drawback here.
  • An "Object Display" fragment is missing... I would want to be able to drop an object display fragment onto a page (like a web content display), select the object to display, then drop child fragments in the container and map to the object fields for display. This way I could drop a bunch of Object Display fragments on a page, point them at different objects, and use fragments w/ field mappings for the display. Right now you are kind of stuck going through a collection display / collection item even when you only have a single object you want to display, plus you have to leverage a blueprint to get there. Don't worry, I did open a feature request for this guy, now I just have to fight to get him added to the roadmap...

So there you have it, a brand new way to approach structured data/structured content by leveraging Objects and Fragments instead of Web Contents.

While they're not yet at feature parity, I'd still argue that the benefits you get from this approach can be worth becoming an early adopter of this technique.

Let me know in the comments below the kind of structured contents you build using this technique, what kinds of things you find easy to do, and especially call out things that you think are missing (like the Object Display fragment) so we can take them to the product team and try to get them implemented.

Blogs

I fully agree that Objects are great and very useful. So far, we have used the webcontent approach alot since it was very convenient for simple structured data. Objects are often far better suited for this, but not always. Btw.: That multilingual objects were not possible was also a showstopper for us in some cases. ​​​​

We practically never use the mapping feature, ​​what we usually do is: a) Add a template contributor that prepares the data for the Freemarker template in a nice way. The contributor creates a dto for the frontend developers and enriches it with extra information (often categories or other related information).

b​​​​​) Add a rest call returning such a dto and we render everything in Javascript.

Which approach we use depends a bit on the usecase. Note: You could, of course, use the Liferay API directly from Freemarker/Javascript, but in my experience, it is more convenient to just prepare everything in a Java class. In the early days (with serviceLocator) we did practically everything in Freemarker and it was painful and horrible.

 

About the mapping feature: I did not test it in the most current Liferay version, but we were never able to get the mappings to work conveniently according to our requirements. Sure, there might be some simple contents, where it works, but these were usually replaced by Fragments + direct editing anyway. Freemarker performance: Maybe this is outdated, it was a while ago, but I once tried to improve the performance of pages with (usually) 10 webcontents or so on them. My initial assumption was that Freemarker must be the problem. But when I measured it, Freemarker wasn't the bottleneck, it was really neglible. Again, this could be outdated, but in the early days of Fragments, somebody (I think Pavel S.) measured fragments with/without Freemarker once and found a 7% performance difference.

​​​​​​​So, while we prefer the Javascript + Rest Service approach nowadays, Freemarker isn't that bad.

Hi Christoph,

Long time no see :) I hope you're coming to the Devcon in November?

What you wrote here is almost exactly how we approach the development in 7.4. We've developed a template contributor to provide enriched data to Fragments or used the old Journal Templates for rendering specific sections of the Display Page (e.g., those with repeatable fields). Then, we map these to regular paragraph fragments. Liferay's provided mappings are effective only for straightforward scenarios, anything more complex requires additional coding.

What do you use as a REST service, Liferay Headless API or you build your custom? I find the built-in Headless API (either GraphQL or REST) still quite difficult to use and not fit for public websites. It's not available for unauthenticated users, which makes sense as the visitors should not be able to browse all the article metadata.

I also agree that doing anything in freemarker is painful and horrible!

Hi Krzysztof, I am the Product Manager for the Headless Infrastructure and I'd like to know if you could comment on the difficulties you face so we can improve their capabilities. About APIs not being available for guest users, that's just part of our policy of being secure by default, so we do not make it easy for unwanted users to attack your solution. However, it is possible to provide access for Guest users. For that you need to configure Service Access Policies, so they only have access to the APIs you want to. You can find information about how to configure it here: https://learn.liferay.com/w/dxp/headless-delivery/consuming-apis/making-unauthenticated-requests

Hi Pablo,

First of all: the documentation. It's short: ​​​​​​​https://learn.liferay.com/w/dxp/headless-delivery/apis-with-rest-builder/producing-and-implementing-apis-with-rest-builder

And there's a lot of things I don't know.

How do I get the User in my rest service? Or the request? It's probably in the thread local, but I'd rather write something like this:

public Response doSomething(MyDTO dat, @Context User user, @Context Company company)

and have it directly available. (I can do this, I have implemented custom contexts).

Also: How can I return multiple response codes from the implementation? With a custom implementation I can send 200 and 204 and whatever I want from the same method:

return Response.status(someCalculatedResponseStatus).build();

Just throwing an exception and 200 OK does not cut it.

Maybe this is all a documentation issue, but ...

- Public/Protected: Pretty much all our APIs need to be accessible public and private. I don't want to configure Service Access Policies. I don't want to click through several environments to allow access to a new method. Using annotations and osgi properties + custom code is far more convenient.

 

Also: I really rarely need an openapi specification, most calls are just used in the project and not shared with external implementors. So, it doesn't save me anything. It is simply easier to implement this stuff directly.

Hi Pablo,

Thank you for joining our discussion, it's great to be able to discuss the issues directly with the author!

I played with the API again today (on 2023.Q4.2 hotfix-134) and I tried to open /o/headless-delivery/v1.0/structured-contents/{articleId} to the public as we often need access to web content fields. The major issue is that it still reveals too much information, like author, all the web content technical details but also internal categories (from which I revoked Guest permission!). So it is definitely not fit for public use for now. 

What I noticed as well is that it's not very consistent in terms of how it uses the underlying Liferay data, for example the GraphQL API uses siteKey (group key) to identify the site and the REST API uses the old groupId (called siteId, which also creates some confusion). For retrieving web content, some methods use articleId and others use resourcePrimKey from what I remember (both named in a different, non-liferay way). What's more, many of these values are difficult to obtain on Display Pages or within Collections, where they could potentially be used. What I usually need to implement is to retrieve the Web Content data from request (either as JournalArticle or as InfoItem within the Collection), get the appropriate key, and save it somewhere on the page so the front-end developer can call the API from JS.

I'll keep experimenting with the API and let you know if I have any new findings. I'll also mention you on slack discussion with David, about how much data gets revealed by the API.

Thanks for your comment, I'll take a look. For example, about "The major issue is that it still reveals too much information, like author, all the web content technical details" -> we try to expose no more information that it's already available in the UI (like here I'm seeing the author of your comment), or for the web content technical details, the needed information to be able to render/consume the content. I'm referring to field Types, contentStructureId, etc. About the internal categories, I'm reviewing it internally. About "for example the GraphQL API uses siteKey (group key) to identify the site and the REST API uses the old groupId (called siteId, which also creates some confusion)" -> as we started with rest, we went for siteId, but then realized that it's not easy to use, so for GraphQL we did a better job with siteKey. For not breaking compatibility, we kept in REST the name of the parameter to siteId, but actually, it can work with the siteName as well. "For retrieving web content, some methods use articleId and others use resourcePrimKey from what I remember" -> we expanded the possibilities, so now we support many parameters to retrieve the content. "(both named in a different, non-liferay way)" -> we consciouly decoupled the API datamodel from the db data model, so in case we decide to change the db data model, we can keep the API working I hope my answers at least gives you context about why our APIs work the way they do.

Thank you for the clarifications, they shed light on the design decisions behind the current API architecture. In general, the API is very useful and it really accelerates the development. It's easier to use it to retrieve the data from the backend, much better than using traditional local services.

However, I have a suggestion regarding the OpenAPI schema documentation. For instance, the getStructuredContent method simply states: "Retrieves the structured content via its ID". This doesn't say much and a potential developer could go to the Web Content in Liferay, open an article, copy the "Article ID", paste it in the API and... receive 404. A brief note clarifying that the required parameter is the resourcePrimKey from the JournalArticle would help and save a lot of time. Moreover, it could say that this method fetches the most recent approved version of the article (btw, for some reason there is no info on the article version in the reply). Otherwise, newcomers to Liferay will very quickly get frustrated with the product as they won't be able to get the simple things done. Not because of the API itself, but because of the lack of knowledge and documentation.

Another area of concern is the data exposure. The current approach to privacy is well-suited for Intranets or Internal Knowledge Bases, where each user is a known and trusted entity. For public, corporate websites though, the rules are different. The unauthenticated visitors should only access content as it is presented within fragments or rendered by Freemarker templates. They don't have access to Liferay's UI. Disclosing the identity of an employee who published the article is undesirable and will not be allowed by any security audit. Also, all the other metadata like creation or publication dates are very often considered internal and should not be revealed.

Because of the above, for now, we can't expose the current API to the public. Fortunately, we discovered the great restClient which already helps us to skip all the terrible Freemarker development on top of journalArticle's XML and local services.

Hi Krzysztof, About the IDs, we wanted to "decouple" the API world from the internal world, so the ID you need to use to retrieve a single content is the one provided when asking for the collection: "http://localhost:8080/o/headless-admin-content/v1.0/sites/{siteId}/structured-contents". If at some point we decide (I know, 99% chance of not happening) change Ids from resourcePrimKey to ArticleId, we will not break the API contract. " it could say that this method fetches the most recent approved version of the article" -> yes, it fetches the last version only of the approved articles. This was decided taking into account your comment about data exposure, in delivery we do not have information about version. To retrieve all articles in different status and with version information, you can use the headless admin content API: http://localhost:8080/o/headless-admin-content/v1.0/sites/{siteId}/structured-contents Regarding your comment about data exposure: "Disclosing the identity of an employee who published the article is undesirable and will not be allowed by any security audit. Also, all the other metadata like creation or publication dates are very often considered internal and should not be revealed." I understand your concern, however that depends on the use case. We have customers who use web content for blogging, or to have a section like "news" in their public websites where information about the author, or publication dates are needed, so our reasoning was to not expose more information that we are exposing for example for blogs or information you can expose in the UI.  Having said this, we are working on a feature, currently in Beta, that allows you to define custom endpoints for Objects, exposing only the information you are interested in. It's still in beta: https://learn.liferay.com/w/dxp/headless-delivery/api-builder As we expand the Object use cases, you will be able to apply it to more places.

Hello Pablo, 

I understand the concept of separating the API world from the internal world, but my concern is the documentation. I have been working with Liferay for many years, so I can identify the Liferay entities that each API artifact represents (even by going through the sources). However, If I give this API to a front-end developer not familiar with Liferay, he'll be completely lost. All the entity and parameter names are different than in Liferay (both UI and the database) and the only documentation that says about it (https://help.liferay.com/hc/en-us/articles/360028727072-OpenAPI-Profiles) is very limited and actually is outdated. For example, it mentions ContentSet (in DXP 2023.Q4.2 it seems to be renamed to ContentSetElement), which represents the internal AssetListEntry (a term known only to Liferay experts) and there is no mention of Collections at all. Other labels are similar, API introduces a third layer of labeling for Liferay artifacts (after UI and database/services) so it brings even more confusion to the platform that is already very complex.

That brings out a problem that Liferay has had for many years. It is a really feature-rich and versatile platform, but due to the lack of documentation and common best practices (caused by some architectural inconsistencies), a lot of Liferay projects are poorly implemented, not fully leveraging the Liferay powers and with lots of custom coding that could have been avoided.

In terms of the used keys, it makes total sense if the application is built entirely on top of the API (so the keys come from collections). However, in a Liferay world, there are lots of "hybrid" apps, that utilize Collection Fragments or Display Pages from where we need to call the API to reach certain data that are not retrievable by standard fragment mappings.

I also tried the API builder some time ago and it looked very promising. From what I understand though, it works only with Objects, so we can't use it until we migrate away from Web Content.

Thank you for mentioning the headless-admin-content API, I haven't looked into that yet, so I'll check it in the next couple of days.

Thanks again for your comments and your input to this discussion!

Hi Krzysztof,

yes, it's a while. I will most likely go to the Devcon, especially since it's basically "nearby" in Budapest. The virtual thing was not my piece of cake, but a real Devcon ...

We usually use the contributors directly from the fragment or execute macros (we deploy them and use <#include "/<bundle>_SERVLET_CONTEXT_/path/xy.ftl" /> to load them, but your solution is pretty nice too. I really like the mapping to paragraphs idea. ​​​​​​​Since we currently deploy the fragments using the fragment toolkit, it didn't matter to me, but since the toolkit was deprecated with 2024.Q1, I was already pondering how to do it in the future.

We also build our custom REST services with the OSGI whiteboard. More convenient.

 

The fragments toolkit is deprecated, yes, but not going away soon. Team is working on another mechanism for fragments export/import, and the toolkit will not go away before that mechanism is launched and tested.

In the interim, you are safe to use the toolkit...

Another thing I rediscovered a couple of days ago (I've heard about it on the last Devcon for the first time) was the restClient tool which allows you to call every headless-delivery or custom objects API from the ftl template (check https://help.liferay.com/hc/en-us/articles/14840864106765-Accessing-to-Objects-custom-fields-in-an-Aplication-Diplay-Template).

Then, getting a full article rendered with a specific template is just:

<#assign documentCard = restClient.get("/headless-delivery/v1.0/structured-contents/${resourcePrimKey}/rendered-content/SOME_TEMPLATE") />

That works really nice and I start replacing my custom contributors written in java with these calls.

 

With all things FreeMarker, you should always do performance tests on things like this, preferrably using an Asset Publisher with 20+ instances rendered on a page, then look at your page rendering performance.

It's not going to tell you how the site will operate when thousands of users are trying to invoke FM at the same time for all of this rendering, but it can tell you if perhaps you're adding overhead that you might not otherwise see until under load.

That's a good point, thanks! I'll look closer at the performance of these templates.

Since it is in a FreeMarker template, the template is making the rest call (so server side synchronous processing), retrieving the result, parsing the json, keeping part of it around for the render needs, discarding the response when done (so a GC activity), ...

Just because restClient() is in there won't mean there isn't a cost involved with using it...

Regarding performance: The recent addition of "builtin" glowroot is really awesome. We are using it for several years now and it helped us uncover and fix lots of performance issues. The small cost of instrumentation was always quickly offset by the metrics you get.

https://learn.liferay.com/w/dxp/system-administration/using-glowroot-with-liferay

(Note: If you use an older Liferay version, glowroot can be added "manually". There's a guide here in the blog somewhere)

The best part about the Glowroot integration is the inclusion of a custom plugin which monitors FreeMarker rendering times, created by our very own Fabian Bouché...

The plugin can help visualize FreeMarker-based performance issues.

This is a super walkthrough to help implementation teams consider the pros and cons of the (implementation) design choices that one might make at this inflection point of LR Objects.  For many of our implementations, the defect related to Localization support would be a show stopper, so hopefully that is resolved soon enough.  

The good news for both of you is that localization is supported in Objects, it's just a bug in the mapping aspects, and I've opened a ticket on that aspect so I'd expect it to get resolved at some point.

The value in mapping, you don't have to worry about parsing web content or extracting field values in any way, with Objects the fields are just there and accessible, especially when using the headless APIs.

Any time you're using FM, there's always a risk of FM abuse, even when you use a custom contributor. For example, if you only need 1 field from your structure, even using a contributor, you may need to parse the whole content just to get a single field value. And if you're not careful how you are getting your field values, you could be parsing the data multiple times.

Objects doesn't suffer from any of that, it's one of the reasons I think Objects is really the better option for structured data than a web content structure and FM is.

Sure there may be gaps and bugs, but those can be fixed. The web content is what it is, and is not going to be "fixable" for many scenarios.

Give the fragment mappings another shot, they're getting better with each release and are super convenient whether using Objects or structured web contents.

Very interesting use case for objects, thanks! We also have been looking into objects as replacements for webcontent but are running into too much restrictions for this to be possible. However, we do use it for very simple data that just needs to be displayed in lists. I.e. like how you would use dynamic data lists in the past.

I hope in the future it would be more flexible, because I love the field validation possibilities of objects, since that is something many customers ask for in webcontent to help editors with their content.

I do have to disagree with you on the following:

"So here's another benefit for using Objects - you get specific control panels for each one." 

That is indeed nice if you do not have many objects/ content types. But most of our customers have a lot of content types that would crowd the interface.  I think it would be great if it would be possible to choose a category key for objects, just as you would for system settings/portlets so you can group objects in the interface. 

"So here's another benefit for using Objects - you get specific control panels for each one." 

That is unfortunately a problem also for us. We developed one site-scoped app with 5 object types, but for now, it is used only on one site. Currently, all these control panel entries appear on all other sites, even the ones that will never use it. I imagine that if we start creating more of these, the Control Panel will become unmanageable.

It would be great to have the following options:

1) Grouping the Objects in some sort of an Application, that would have just one entry in the Control Panel

2) For Site-scoped Applications, enabling or disabling an Application for a specific site.

Well, you could not use a control panel at all.

When you enable the "Show Widget in Page Builder" option, you'll have the same interface but as a widget that you can drop on a page of the site.

You can set up a page, drop the widget on it, permission the page for the select group that have access, and leave the panel link undefined.

Only use the page on the site(s) that needs it, exclude it from the rest.