Introduction

In my recent blog, Introduction to Liferay Objects, I used Objects and Liferay OOTB facilities to have a list of Course Registrations and UI interfaces for submitting a new Course Registration, listing them, and for admin users the ability to approve or deny them.

It worked well, but it didn't handle maybe some expected requirements. For example, the approve/deny buttons could change the status of any registration, not just the pending ones. The buttons were not aware if the admin had permission to perform only one of the two actions, things like that.

The next blog I started which, at this point, is yet to be released will be building an application to handle the course registrations. It is still going to use the Object that we created in that blog, it's just going to be using more of the generated headless APIs for doing all of the work.

And at this point, the blog is basically finished in draft mode and I also have the custom application ready to go too.

However, I have been plagued with errors being returned when I use any of the headless APIs that are not GET requests. I keep getting 409 (Conflict) return codes which make no sense because there should be no conflict. Also I'm using the test@liferay.com admin user, so permissions shouldn't be an issue, but I don't really know that for certain at this point.

So I needed to track this down and thought that others facing some sort of unexplained API failures might need to do the same, so I thought I'd create a blog describing the things I've done to identify what my problem is and maybe that would come in handy for them too.

Debugging Setup

First I knew I was going to be in the debugger, but instead of debugging a custom application, I was going to be debugging Liferay.

My blog post and application are running on GA 112, so I downloaded the source for that version (well, actually I've forked the https://github.com/liferay/liferay-portal repo, so really I just checked out the 7.4.3-GA112 branch, but a download would have worked also).

Before actually loading the project into the IDE, I do like to build on the command line. This ensures that all necessary artifacts for the build will be available to the IDE when I do load it in.

Building Liferay requires JDK8, so that is a prerequisite. Then use the command line:

$ export ANT_OPTS="$ANT_OPTS -Xms8g -Xmx8g" && ant setup-sdk setup-yarn && \
    ant all && ij

That last ij piece, that is responsible for building the IntelliJ project files for the portal. I'm a big IntelliJ fan and the ij script does a great job at making all of the various folders and subfolders into proper IntelliJ projects. You can download the ij script from here: https://github.com/holatuwol/liferay-intellij.

With these steps complete, you can now go into IntelliJ and just Open the project. IntelliJ will pull it all in, index all of the code, do all of its typical things. Note that Liferay itself is a big project and will take some time to load. Also if you are using a cloned repo, IntelliJ will process all of that history too, so that can add some time to load.

Finally, in order to debug a running Liferay portal, you need to start it up in debug mode. Follow the steps that I've already outlined here: https://liferay.dev/blogs/-/blogs/blade-outside-of-the-ide#intellij-debugging. The only difference is that you should add the Debug configuration to the liferay-portal project, not a custom Liferay workspace that you might have around.

All that remains is to start up your Liferay GA 112 in debug mode, then have IntelliJ connect via remote debugging to the Tomcat instance.

Debugging Objects

Debugging Liferay is really easy for most code, even regular Headless calls. You just find a class that has your entry point, such as the ResourceImpl class that is tied to your entity. If we were trying to debug Structured Content API calls though Headless Delivery, you'd open the StructuredContentResourceImpl class since that has all of the entry points you might need to set breakpoints on.

But this won't work for your custom objects because, well, they don't have a unique implementation. It certainly looks like they should since your custom objects are available in /o/api just like custom RESTBuilder modules are, but that is really just a façade created for us by the Objects framework.

So to set a reasonable breakpoint for the debugger, we'll need to understand a bit more about how Objects is actually wired.

At startup, all custom object definitions will be processed by the com.liferay.object.rest.internal.deployer.ObjectDeployerImpl class. This class will actually use details from the Object Definition as well as methods from the ObjectEntryResourceImpl and will register the JaxRS application with the appropriate name, path, and declared object action methods. This occurs during startup, so if you want to intercept this execution you'll need to add the JPDA_SUSPEND="y" argument to your debug.sh script to have Tomcat wait for the debugger to attach before starting up.

So all actual methods for custom objects will come from the com.liferay.object.rest.internal.resource.v1_0.ObjectEntryResourceImpl class, so you can actually set breakpoints on the public methods of this class to intercept calls.

There are the generic delete, get, patch, post and put methods here for the various activities you want to intercept. Of particular interest may be the putObjectEntryObjectActionObjectActionName() and putScopeScopeKeyByExternalReferenceCodeObjectActionObjectActionName() methods. These are the two entry points used to invoke the standalone object actions that may be defined for your Object.

Now the reason why you may find this to be a challenging place to debug is that it is the one ResourceImpl class that handles all of the custom objects, not just the one that you might be interested in. When you set a breakpoint on, say, getObjectEntriesPage(), this method will be called for every type of object list that is going to be returned, there's just a different instance that would be tied to each individual Object that you have defined.

If you can control your debugging so you're only checking one object at a time, it does make things a bit easier. But if you have an app making a bunch of different calls, it may prove to be challenging to skip the breakpoints in objects you're not worried about.

A great way to debug the objects is actually to use the /o/api page for your specific object type. You can invoke the methods there and hit the breakpoints without a lot of extra noise.

Speaking of /o/api, it may be interesting to set some breakpoints on com.liferay.object.rest.internal.vulcan.openapi.contributor.ObjectEntryOpenAPIContributor. This class is responsible for contributing things to the OpenAPI schema to reflect what is in your custom Object. If you're wondering why a standalone action might not be showing up or how the properties are being handled, this is a good place to check.

The Situation

Okay, so the reason I'm writing this...

Basically I was doing a GET on one of the Course Registration instances using the URL http://localhost:8080/o/c/courseregistrations/scopes/32815/by-external-reference-code/6f704673-30af-a1f3-5ea2-0e6f95b6f7ed. This would run successfully and return the JSON

{
  "actions": {
    "approveRegistration": {
      "method": "PUT",
      "href": "http://localhost:8080/o/c/courseregistrations/
         by-external-reference-code/6f704673-30af-a1f3-5ea2-0e6f95b6f7ed/
         object-actions/approveRegistration"
    },
    "denyRegistration": {
      "method": "PUT",
      "href": "http://localhost:8080/o/c/courseregistrations/
         by-external-reference-code/6f704673-30af-a1f3-5ea2-0e6f95b6f7ed/
         object-actions/denyRegistration"
    },
    "permissions": {
      "method": "GET",
      "href": "http://localhost:8080/o/c/courseregistrations/34905/permissions"
    },
    "get": {
      "method": "GET",
      "href": "http://localhost:8080/o/c/courseregistrations/34905"
    },
    "replace": {
      "method": "PUT",
      "href": "http://localhost:8080/o/c/courseregistrations/34905"
    },
    "update": {
      "method": "PATCH",
      "href": "http://localhost:8080/o/c/courseregistrations/34905"
    },
    "delete": {
      "method": "DELETE",
      "href": "http://localhost:8080/o/c/courseregistrations/34905"
    }
  },
  "creator": {
    "additionalName": "",
    "contentType": "UserAccount",
    "familyName": "Test",
    "givenName": "Test",
    "id": 20122,
    "name": "Test Test"
  },
  "dateCreated": "2024-03-16T04:32:35Z",
  "dateModified": "2024-04-14T23:44:58Z",
  "externalReferenceCode": "6f704673-30af-a1f3-5ea2-0e6f95b6f7ed",
  "id": 34905,
  "keywords": [],
  "scopeKey": "Masterclass",
  "status": {
    "code": 0,
    "label": "approved",
    "label_i18n": "Approved"
  },
  "taxonomyCategoryBriefs": [],
  "registrationStatus": {
    "key": "denied",
    "name": "Denied"
  },
  "notes": "Nothing to say here.",
  "r_courseAttendee_userERC": "aa3313fd-42a0-267b-8366-8c38ef011432",
  "r_courseAttendee_userId": 20122,
  "course": {
    "key": "projectManagerCertificateJuly2024",
    "name": "Project Manager Certificate - July, 2024"
  },
  "courseAttendeeERC": "aa3313fd-42a0-267b-8366-8c38ef011432"
}

I would then display the Approve and Deny buttons and, depending upon which one you clicked, I'd issue a PUT call to the relevant action URL.

Now when I did this, I would always get an error back, a 409 Conflict as the result. But this made no sense to me because nothing that I was doing should have triggered any kind of conflict.

Hence the need to debug the situation. I needed to find out why the PUT actions were failing.

Debugging The Call

So actually I set up breakpoints on all of the classes I discussed earlier.

In ObjectDeployerImpl I set a breakpoint on deploy() so I could see what was being registered from my Object Definition.

In ObjectEntryOpenAPIContributor, I set a breakpoint on contribute() so I could see what was being added to the schema and would appear on the /o/api page for my application.

And, in ObjectEntryResourceImpl I set a breakpoint on putScopeScopeKeyByExternalReferenceCodeObjectActionObjectActionName() because this was the method that should be invoked for either the approve or deny action URLs.

On startup, the first breakpoint I hit was the deploy() method. I could see that my JaxRS application was being properly registered and everything seemed to be wired correctly, so no problem there.

I then tested my React code. In my api() method (which you'll get to see when I publish the blog), I added a console.log() statement to show what I was invoking, and I could see that it was using the URL http://localhost:8080/o/c/courseregistrations/by-external-reference-code/6f704673-30af-a1f3-5ea2-0e6f95b6f7ed/object-actions/approveRegistration, but it was still getting the 409 failure response.

At this point I'm thinking maybe there is something wrong in how the approveRegistration is implemented, so I open a tab to /o/api to try it there manually. As soon as I did this, I hit the next breakpoint in the contribute() method, so I could verify that yes, in fact my two standalone actions were being added correctly to the OpenAPI schema.

At this point I had verified the JaxRS application was registered correctly, and also the right things were being added to the OpenAPI schema, so from a registration perspective everything was looking fine.

So the next step was to find one of the actions and try to run it on /o/api, and that's what I did:

I plugged in my key values and clicked Execute, and wouldn't you know it, it worked just fine and had a return code of 204!

Oh, and the debugger stopped in the putScopeScopeKeyByExternalReferenceCodeObjectActionObjectActionName() call so clearly it was getting to the right method to handle the action URL...

When I tried invoking the action from React, I only got the 409 and never hit the breakpoint, so something before the resource impl was failing and returning the 409. I used the stack trace in the debugger to set breakpoints at every method in the stack hoping that I'd be able to find one that triggered the failure, but that was kind of fruitless. First, the stack was in the bowels of CXF (the implementation Liferay uses for the JaxRS applications), and second I couldn't trace exactly where in CXF it was kicking out and returning the 409.

Now I'm starting to go a little crazy...

I'm wondering why the /o/api call would work but it would fail for my React app.

I eliminated perms as an issue because I'm logged in as the Test admin account and besides, the 409 didn't have anything to do with a permissions issue. Plus I checked the headers that I was sending with the React call and they matched exactly the headers used in the curl example, so the call should have worked.

The Clue

So it was the checking of the headers between my React call and the curl example from /o/api that gave me the clue to the mystery...

My header values matched exactly, but I noticed that the URLs didn't match up...

The curl URL example is http://localhost:8080/o/c/courseregistrations/scopes/32815/by-external-reference-code/6f704673-30af-a1f3-5ea2-0e6f95b6f7ed/object-actions/approveRegistration, but the URL in the approveRegistration action href is http://localhost:8080/o/c/courseregistrations/by-external-reference-code/6f704673-30af-a1f3-5ea2-0e6f95b6f7ed/object-actions/approveRegistration. It's missing a scopes portion in the path!

The Bug

So yes, I had actually been beating my head against the wall on this for days. Not straight through, I had code to finish and, even though I knew the failure was still there, I could think about it while wrapping the other code up.

But I had found a bug. The action URLs are supposed to be HATEOAS so I expected them to just work, but for site-scoped objects, the action URLs are wrong because they don't include the necessary scope portion of the path.

I opened a bug on this issue: https://liferay.atlassian.net/browse/LPD-23177

I also changed my code so it wouldn't use the action URLs, instead I'd manually construct the proper, complete URL to invoke the action. This was all it took to have my React code successfully invoke the standalone actions.

Conclusion

Did I need to debug the Objects headless APIs to find the problem here that turned out to be a product bug?

Maybe not, I mean if my first thought had been to check what URL was being used and whether it matched what was used in /o/api, maybe I could have identified the bug without all of the debugging.

Since that wasn't my first thought, going through the steps to debug the process was really my only option. And, had the failure been something different, debugging may have been the only way to find the issue and either resolve it or report it.

Anyways, I thought this was a useful exercise and that sharing it with you might provide some insight if you ever find yourself needing to debug the bowels of the Objects headless APIs.

The moral of this story, if there is one - always check your URL to ensure that they are really valid!