REST Builder Limitations / Observations

RE: RE: REST Builder Limitations / Observations

2024-02-06T18:22:10Z

@Javier Gamarra
This might be a weird place to ask this; however, I really needed an expert's opinion on this.

I am trying to find a way to build a rest-builder that in the rest-openapi.yaml file I do not have to create a separate component for the built-in Liferay commerce product. Instead I want to reference the same class the L iferay headless product api uses.

Thanks.

RE: RE: REST Builder Limitations / Observations

2021-04-24T08:38:25Z

Maybe backed by an abstract method or an adapter or builder that would make it clear this would be the place to implement it?

Got it

Liferay source code leverages several approaches including the anti-pattern double braces - as David N. references in his blog post series on the headless api - so it's a little confusing what is the right way to do it.

The double braces are not the recommended way, as you mention it's an anti pattern. Liferay, being a large organization, we often find that we have to adhere to code conventions we don't personally like with the goal of achieving a consistent codebase, double braces being one example. We don't have a team recommendation because we are not providing tools/methods in how to convert entities to DTOs but that will change in the future. Right now if you are not going to share the conversion across different modules just using setters is enough. If you plan on sharing the conversion the DTOConverter would be the way to go.

Lastly, does the fact that you have just opened LPS-130916 mean that I can not yet leverage ElasticSearch aggregations via the Headless API framework?

No, I created LPS-130916 some days ago because I want to improve the API for defining and exposing which fields can be filtered/searched/aggregated but right now you can do it and it works. Javier will share the drafts he prepared about it and we can guide you in the implementation/use. But basically you can aggregate on fields indexed by keyword or modify the index to allow aggregating by "text-indexed" fields (an Elastic limitation). An example query like:

curl "http://localhost:8080/o/headless-delivery/v1.0/sites/Guest/structured-contents/?restrictFields=actions&aggregationTerms=userName,title,dateCreated,dateModified,entryClassPK,version,status,roleId,folderId&fields=title" \
     -u 'test@liferay.com:test'

Should return something like (with just one content created):

{
  "actions": {},
  "facets": [
    {
      "facetCriteria": "entryClassPK",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "92276"
        }
      ]
    },
    {
      "facetCriteria": "roleId",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "39613"
        },
        {
          "numberOfOccurrences": 1,
          "term": "39615"
        }
      ]
    },
    {
      "facetCriteria": "userName",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "test test"
        }
      ]
    },
    {
      "facetCriteria": "version",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "1.3"
        }
      ]
    },
    {
      "facetCriteria": "folderId",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "0"
        }
      ]
    },
    {
      "facetCriteria": "title",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "a new hope"
        }
      ]
    },
    {
      "facetCriteria": "dateModified",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "20210424082455"
        }
      ]
    },
    {
      "facetCriteria": "dateCreated",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "20210413205209"
        }
      ]
    },
    {
      "facetCriteria": "status",
      "facetValues": [
        {
          "numberOfOccurrences": 1,
          "term": "0"
        }
      ]
    }
  ],
  "items": [
    {
      "title": "A new hope"
    }
  ],
  "lastPage": 1,
  "page": 1,
  "pageSize": 20,
  "totalCount": 1
}

RE: RE: REST Builder Limitations / Observations

2021-04-23T21:11:55Z

Javier: Thanks for the prompt and comprehensive response!

I will work on getting you those code samples early next week.

Responses to your responses:

very helpful to know about the available properties for rest-config.yaml. ty.
In terms of the conversion convention - it's not that I need the framework to auto-magically do it for me - rather that it would be nice to have an agreed upon approach. Maybe backed by an abstract method or an adapter or builder that would make it clear this would be the place to implement it?
Liferay source code leverages several approaches including the anti-pattern double braces - as David N. references in his blog post series on the headless api - so it's a little confusing what is the right way to do it.

Lastly, does the fact that you have just opened LPS-130916 mean that I can not yet leverage ElasticSearch aggregations via the Headless API framework? Our 6.2 app that we are rewriting on 7.3.5 makes heavy use of these and so I would like to know if this is something I should be expecting to use in the near term, or whether I need to go my own way and leverage the ElasticSearch client libraries directly. (I did have to do this for our 6.2 app - so it wouldn't be an unexpected piece of news). I would just like to know in order to plan accordingly.

Sincerely

RE: REST Builder Limitations / Observations

2021-04-22T20:56:14Z

Ohhh, really nice feedback, thank you so much! :D We'll act on it right away.

Point by point this is our comments:

buildREST Gradle task is rewriting parts of my rest-openapi.yaml file.

I hate that. It's a configuration that was thought to be internal of liferay and because of several issues is the default. We are going to change that here: LPS-130965. You can disable it in the rest-config.yaml, here are some of the properties. The most annoying ones are forcePredictableSchemaPropertyName and forcePredictableOperationId.

If I attempt to use the schema -> items -> $ref -> XXXResource then RESTBuilder fails with GraphQL errors

This looks like a bug, can you paste what you try to generate?

I'm using 7.3.5 GA6 - but I am not seeing consistent support for LPS-106992 in the framework:

We're going to share the documentation about it. Aggregations are tricky because we rely on the fields that Elasticsearch support that are not all the fields that can be filterable/sortable/searchable because it depends on how they have been indexed. Our idea in LPS-130916 is to expose can you can do which each field.

I found myself needing to define a component called MultipartBody to reference within my path definition.

Yeah, you should not need to do this... it should pick and let you use the existing one. Can you share the endpoint definition (via email javier.gamarra@liferay.com or slack if it has sensitive parts).

Error outputs from the buildREST gradle task is very cryptic. Makes it a very time consuming process to understand what the problem is - especially when loading it into Open API validators seem to show no issues

Yep, this is 100% true. We have made some improvements but there is a lot of margin to improve.

Need to quote every path definition

We were not aware of it, we'll investigate it here.

There must be an entry in the tags section for every path definition

All should not be needed, only the ones that can not be linked to a class (like delete or multiparts) but those are already too many, we are working on it here.

Every responses section seems to need both a application/json AND a application/xml specified under contents

Again a force flag, we recommend them disabling them, we'll change it to false by default.

I tried creating a resource property with the name abstract and that choked the builder

Hehe, this looks like because abstract is a reserved keyword in Java. We should validate it beforehand.

In order to make our application fully headless - I have had to create endpoints to update our user profiles - since none of these are currently exposed via the Liferay Headless Admin User API endpoint.

Can you share them? and we'll integrate them in the product. We lack a lot of "actions", specially those that are very linked to operations in the portal.

Do I still need to define an OSGI Web module for my application - even though it is supposed to be entirely headless? It appears that we still need to define Portlet referencing permission defintions in default.xml for our root resource actions. Is this true? Are there any resources available which could help me with migrating permissions from a 6.2.x web application to a 7.3.x+ OSGI modularized Headless API application?

Here I'm a bit of a loss. I think the portlet is still needed but maybe someone from security or core can help pointing to resources. We can also ask in the community slack directly.

Some API signatures require lambda functions - for example within the Search indexing framework

The SearchUtil interface is tough... we have to simplify/revamp it.

The system could benefit from a clear convention for converting between ServiceBuilder entities and Resource DTOs

Interesting... this is the second time we receive this feedback but we could not pinpoint the needs... what would you expect? to create automatically an API exposing all fields? or have a filtering/conversion layer that receives everything and removes/adds?

Calling a YYResourceImpl from another XXResourceImpl -- for example in the case of delegating the population of DTOs to their specific ResourceImpl class when you have a Resource that is composed of others -- requires that you pre-populate that delegation ResourceImpl's contexts.

REST Builder should generate some *ResourceFactory classes to simplify doing this automatically

I think this was all. We'll fix the bugs quickly, and please share us all the issues you find :)

REST Builder Limitations / Observations

2021-04-21T22:08:02Z

I thought it might be useful to other developers to put these observations / limitations in one place as pertains to REST Builder / Headless API development.

To the Liferay developers who work on these components - I would really appreciate some feedback on where exactly things stand with each of these. (Thanks in advance!)

buildREST Gradle task is rewriting parts of my rest-openapi.yaml file.
It appears that there is a component in the build process that is trying to make my schema "better"? Or it's trying to make it conform to some expected format. However I have no idea what that format looks like - and so I don't know what I should be changing it to.
It does keep changing my response schema entities (e.g. images) for items. I believe this has something to do with the Page concept within the headless API - but I haven't found any documentation that specifcally indicates what I should be doing.
If I attempt to use the schema -> items -> $ref -> XXXResource then RESTBuilder fails with GraphQL errors
I'm using 7.3.5 GA6 - but I am not seeing consistent support for LPS-106992 in the framework:
1. Within the LR codebase I see rest-open-api.yaml files with the query parameter: aggregationTerms defined - and I do see what appears to be the result of that in the method signatures of the related xxResourceImpl classes - with their conversion into Aggregation.
  However - I cannot find a single example within the API Explorer where I can enter a value for the aggregationTerms query input.
2. Also, when I add an aggregationTerms query input to my path definition I am not getting an Aggregation generated. I am currently however getting:
  1. Filter (from including filter)
  2. Pagination (from including pageSize and page)
  3. Sort (from including sort)
3. I don't see any documentation on this topic (I did ask Javier de Arcos Velilla for access to his google doc preview).
In order to be able to create a custom endpoint to upload a file to the server - I found myself needing to define a component called MultipartBody to reference within my path definition. While this ends up working overall, I end up getting a clash between my resource and the com.liferay.portal.vulcan.multipart.MultipartBody class. This requires me to need to clean up my imports on the affected classes each time I run the buildREST gradle task. Am I missing something that would not require me to define my own variant?
Error outputs from the buildREST gradle task is very cryptic. Makes it a very time consuming process to understand what the problem is - especially when loading it into Open API validators seem to show no issues
Gotchas that I've run in to from using the buildREST gradle task (some of which are listed in REST Builder Conventions:
1. Need to quote every path definition
  1. This mostly trips me up because I am moving between a separate OpenAPI editor and my Liferay workspace. I have gotten around this for now by only copying in new sections, rather than the entire YAML file each time
2. There must be an entry in the tags section for every path definition
3. Every responses section seems to need both a application/json AND a application/xml specified under contents
4. I tried creating a resource property with the name abstract and that choked the builder
In order to make our application fully headless - I have had to create endpoints to update our user profiles - since none of these are currently exposed via the Liferay Headless Admin User API endpoint:
1. mark email as verified
2. accepting terms of service
3. uploading a profile image
Since this current project represents a migration from a 6.x based application to 7.3.x - I am running into issues relating to how exactly to migrate Permissions. The permissions documentation is confusing at best and I am uncertain as to what exactly is and isn't supported for 7.3.5+. Some questions on this topic:
1. Do I still need to define an OSGI Web module for my application - even though it is supposed to be entirely headless? It appears that we still need to define Portlet referencing permission defintions in default.xml for our root resource actions. Is this true?
2. Are there any resources available which could help me with migrating permissions from a 6.2.x web application to a 7.3.x+ OSGI modularized Headless API application?
Some API signatures require lambda functions - for example within the Search indexing framework
The system could benefit from a clear convention for converting between ServiceBuilder entities and Resource DTOs
Calling a YYResourceImpl from another XXResourceImpl -- for example in the case of delegating the population of DTOs to their specific ResourceImpl class when you have a Resource that is composed of others -- requires that you pre-populate that delegation ResourceImpl's contexts.
Is there any established pattern for doing this that's baked into the framework?

Again - thanks in advance for any information you can provide on these topics.

Sincerely,

Chris D.