Add orderable column to users (or: how customization degrades)

RE: Add orderable column to users (or: how customization degrades)

2018-12-21T14:53:10Z

I have really enjoyed reading this thread and I think I can definitely, as a third party, relate to both sides of this argument. I think as developers of the extensions, sometimes we forget that Liferay is in fact a product company and a big part of what you buy is just that, what you buy. As David has said, in the past it has been so open that some of the "closed" aspects of 7.x can sometimes be annoying -- no argument there. Like you I have hit a wall a few times and found myself doing what I needed to do to meet the requirements. So Tobias, I definitely see and agree, in part, with some of what you are saying.

There is one part though that I think was unwarranted though.

"I really hope that all my complains are not in vain and someone from the lead developers at Liferay is listening as well. I often have to create comparisons of implementation costs for projects and from my point of view the momentum is shifting towards the competitors of Liferay 7.1 now."

... As Milen mentioned, complaining in the forums is not the way to get your voice heard. This is meant for community based support and engagement. With that said, I do applaud the detail that you put into the post as it provides amazing detail for anyone else who needs to do this. Being an open source product though, and given that you have a cleaner solution, I would think a PR with the changes and a issues.liferay.com that references both this thread and the PR is a better way to ensure the complaints are not made in vain. I would imagine that the development team at Liferay is under plenty of pressure from all sides and I doubt they spend much time checking these threads.

Now, while I am a fan of the content, there is something you said that I take exception with. The comment about Liferay 7.1 and the other competitors in the market place.The insunuation you are making is that -

1. The quality issue you found is systemic in the product -- you have not surveyed 3.2 million lines of code to validate that you have not just found a one off

2. Your code quality issue is trivial at best. It's annoying to see so much code but it's not as if 1000 methods are being called in place of one -- yes it should be cleaned up, but really, it's a low priority. I'd rather see some of the new roadmap features over dedupe activities -- and I bet most paying customers would agree

3.You are insinuating that competitors in the market place don't have similar issues with their code, or other areas where the frustration you experienced with Liferay would surface. I have worked with the competitors ... Adobe, Oracle, IBM, Vignette (back in the day), beleive me, they SUCK in comparison. My biggest frustration with all of them has always been how you "get what you get" and extending the product is an unsupported nightmare. "

Yeah, you can customize login. You just have to write your own"

-- as opposed to overriding one class in the whole pipeline. You want to rant about duplicating code? that is the worst -- if you can even get your hands on it since the license agreement forbids you to unpack jars and look at it.

You are of course entitled to your opinion but the problem I have with such a bold statement is that while you may be familiar with the product, others that come along here may not be. They might read your statement and take it to heart -- falsely assuming that Liferay is not a great product and can't be a great solution for them.

Like you, I am a seasoned technologist and having been a consultant for all but 1 year of my career I have worked with dozens upon dozens of products. ALL of them have their points of weakness, and ALL of them have points of strength. I have done many many projects with customers where when I started there was down right hatred for the product (Liferay). Quite quickly though, there was the realization that the issue was in the implementation, not the product. Almost all of those same clients, by the time the engagement was done, were transformed into advocates for Liferay.

If you, as you say, are responsible for evaluating competing products for your clients, I should hope that your criteria uses something far more relevant than something that 90% of clients will never see. There is always room for improvement -- both in Liferay, and our own code.

RE: Add orderable column to users (or: how customization degrades)

2018-12-21T01:37:21Z

Tobias Liefke:

Someone who maybe knows the pros and cons of lambdas (@David: sorry I couldn't resist).

[snip]

p.s. @David If you still like to hear my thoughts about your arguments - send me a PM / mail and I will be happy to answer them.

LOL.

I know the pros and cons... Nostalgia just gets me me sometimes for those good old anonymous inner classes...

RE: Add orderable column to users (or: how customization degrades)

2018-12-20T22:54:12Z

Thank you all for your affirmative and encouraging words. I'm a little overwhelmed now.

And I really will try to give further input, but I have to admit that I'm not prepared to do so. And looking at the upcoming holidays it will be hard to me too look into all these further readings.

But I'd like to repeat my points:

If it is easy for you to add a new functionality, it is easy for me and others as well ("Eat your own dog food").
Try to keep an eye on code quality. I mean, I would look to have at least one person in each development team who knows that Tiobe is not an asteroid. Someone who knows what premature optimization means and why one should try to avoid it. Someone who maybe knows the pros and cons of lambdas (@David: sorry I couldn't resist).

As much as I would like to step in here, I'm not the one that can do that, as I can't split myself into that many pieces. Besides that Liferay is not my only field of interest and my voluntary time is already occupied by other projects (even though I try to help Olaf dealing with all the stackoverflow questions from time to time).

And I'd like to add another thought: I wouldn't think too much about all possible usecases, as I wouldn't even catch half of the use cases that exist out there. What I mean: Don't think too much about possible extension points. As long as you have a clean architecture (the OSGI decision was a big step in that direction and Lexicon 2.0 seems to me like one as well) and as long as you have an eye on clean code, there won't be many complaints about extensions. After all it is still Open Source - if I can't build a feature with the existing extension points, I still can create a patch and add a pull request. As I already said, the reason for my post was not the lack of extension points, but the lack of code quality.

Finally I would like to add that I really would have liked to go to the Devcon - but the schedule was (as usually) tight at the end of the year. But after these encouraging words towards the upcoming version I will start a new try next year.

p.s. @David If you still like to hear my thoughts about your arguments - send me a PM / mail and I will be happy to answer them.

RE: Add orderable column to users (or: how customization degrades)

2018-12-20T19:11:49Z

Hey Tobias,

Following up Brian's comment, you can track https://issues.liferay.com/browse/LPS-83832 where we've already started the work to bring for example the new table components into DXP.

As UI goes, we've started migrating our component strategy away from jsps and towards a data-oriented model, where templates don't have logic and all they do is render some data. That makes it easier to extend, since the only thing you need to change is the underlying data being passed down to the component. Unfortunately, the data creation moment might not be itself exposed if the developers didn't account for it, which then locks down the extension options.

What we want to attempt, after what it's already proven to work in Commerce, is to provide an extensibility-first approach, so internal Liferay developers won't unintentionally shut extension points down. One option is that every major component (tag) can be fetched an osgi filter, or component id and that we adopt that by default. With that, external developers should be able to register OSGi components with higher ranking to extend the data that gets generated and/or passed down to the components.

What do you think? Do you think this might address some of the existing problems you're experiencing?

This being said, in your posts, you clearly hint that this is more than a mere UI problem. I think we've had similar extension problems recently (ie: externally adding a new supported language) that cut across many different domains (database, models, localization, apps, themes...) which need a more holistic approach. How do you think such extension needs could be addressed to the developer's satisfaction?

Thanks for bearing with us. As Brian says, we'll fix this!

PS: Special kudos to Marco Leo for coming up and pursuing some of the ideas to improve our take on UI extensibility.

RE: Add orderable column to users (or: how customization degrades)

2018-12-20T04:38:21Z

Tobias,

You make very valid points. We're going to fix the 1.) code duplication and 2.) make it easier to customize columns like you could back in 6.2.x. There's no reason things simple should be complicated.

The Commerce team already has a workaround they've been using for 7.1.x. See this commit.

I have a sprint in January where I will 1.) digest the problem as you've described (and which Marco Leo has brought up to me before) and 2.) bring in Marco's solution to the portal. Then, I'm going to assign someone to de-duplicate and apply the pattern every where.

We will fix this. You have my word.

RE: Add orderable column to users (or: how customization degrades)

2018-12-20T02:26:40Z

Hi Tobias,

I'm sorry for all the trouble you had to go through when trying to extend Liferay functionality. I'm Liferay's Engineering Lead of Search and I agree with you that sorting on a few custom fields shouldn't be so hard and take so much code. We've been listening to similar reports and in the roadmap of our next version, sort improvements were selected as a priority currently in active development.

https://issues.liferay.com/browse/LPS-84969

I wanted to say the level of detail you provided is remarkable and will be extremely helpful in making sure we implement this feature just right. This is an immensely useful, specific, end-to-end use case and we will make sure it gets properly covered. I would love to hear any other insights you may have from your experience with customization of Liferay Search.

Thank you,

André

RE: Add orderable column to users (or: how customization degrades)

2018-12-19T20:20:59Z

Tobias Liefke:

Conclusion:

I had to write a lot of code and had to use a lot of inside knowledge of Liferay - which may change already by the next patch version. Just to add some simple sortable date columns.

...

Thus this post is more a call for higher code quality in Liferay than it is a tutorial on how to add sortable columns in Liferay.

Hi Tobias,

I for one tend to agree with you. It absolutely should not require such big effort to achieve the goal you wanted to achieve. Period.

That said, I don't disagree with David. I know there is a reason for everything and it is often very hard balance between innovation and backward compatibility and bunch of other things. Especially for a product with such history and codebase. Sadly sometimes we do sacrifice developer experience to achieve other goals. But It's not my intention to make excuses or justify decisions I don't have enough information about.

What I would like to suggest to you is to consider sharing your expectations and ideas of how things should work in the form of feature requests in Jira. I understand this is what you try to do here but there are 2 problems with that. First is that forums are not a place that all product team members monitor. Second, you did a great job describing the problem but it would be better if you could explain the expected solution. Feature requests can help with both - reach the right people and give them your expectations about how things should work instead. It also allow to start a discussion with the teams building the product.

If you choose to do that and experience long delays before teams respond, please feel free to contact me directly and will do my best to have respective teams look at your proposals.

Best,
Milen

RE: Add orderable column to users (or: how customization degrades)

2018-12-19T16:53:30Z

Tobias Liefke:

Some overrides are easily implemented and still supported. JSP fragment bundles, for one. So there are ways to affect customization without forcing a rewrite of one of Liferay's modules.

But JSP fragment bundles are an example for problems on every patch version - because you can't extend them. Thus you have to check for changes in these JSPs on every upgrade. From my point of view they are much worse compared to extending private classes - where in many cases the compiler already indicates incompatibilities.

There are solutions for this coming down the pike. Liferay is building a new method to inject into a JSP page at top and bottom (and possibly elsewhere, I don't have all of the details yet), but this would offer spaces that are mixed in with the Liferay core JSP and help to keep customizations separate from core. I don't know yet if it will be effective for all use cases, but it should accommodate some.

Tobias Liefke:

Well, arguments about code formatting, style and maintainability are often somewhat subjective.

I wonder if it is lack of knowledge or ignorance that you are mixing code formatting with code quality. While code formatting is somehow subjective and has only one rule (consistency), there is a common sense of code quality. In each book and on every page about that topic you will find nearly the same rules with the same argumentation. You just need to read and understand them to get my point.

I don't like the tone this is taking on...

Code quality can also be subjective. Is an anonymous inner class vs a lambda a question of quality? Is readability a measure of quality and, if so, readable by whom? A line, duplicated 3 times, may actually peform better than a loop with a counter and a test involved, but one is preferred over another from a review standpoint while the other is preferred from a performance standpoint.

It is often easy to lob criticisms about quality, style and maintenance issues at someone else's code. But at the end of the day, you won't know what is important to them or why. The boilerplate code this was initially about, that stuck out for you in your review. But obviously for Liferay it made sense to build it that way and makes sense to keep it that way.

The Liferay code base is a large system with teams of folks working on separate feature areas; all code changes get merged into a single git repo where CI verifies that all tests pass and there is also the final gatekeepers led by Brian Chan. Trust me, if they thought the code had quality issues they would have removed it by now and, because it is in there, suggests to me that there is a reason it is in there the way it is.

While I totally agree that it appears that having the boilerplate code looks like a quality issue, since I don't know the reason why it is the way it is and don't know if it serves some special purpose for Liferay, I'm not going to make a big post saying there are code quality issues. Instead I'd try to get to one of the numerous Liferay events that are held each year and attended by an overwhelming number of Liferay staff, including Engineering, and I'd start a conversation to find out why the code is like the way it is. Or even post to the community Slack channel, where some from Engineering hang out.

RE: Add orderable column to users (or: how customization degrades)

2018-12-19T12:57:32Z

Some overrides are easily implemented and still supported. JSP fragment bundles, for one. So there are ways to affect customization without forcing a rewrite of one of Liferay's modules.

But JSP fragment bundles are an example for problems on every patch version - because you can't extend them. Thus you have to check for changes in these JSPs on every upgrade. From my point of view they are much worse compared to extending private classes - where in many cases the compiler already indicates incompatibilities.

Well, arguments about code formatting, style and maintainability are often somewhat subjective.

I wonder if it is lack of knowledge or ignorance that you are mixing code formatting with code quality. While code formatting is somehow subjective and has only one rule (consistency), there is a common sense of code quality. In each book and on every page about that topic you will find nearly the same rules with the same argumentation. You just need to read and understand them to get my point.

Arguing about code quality is like arguing about the global warming: You can ignore it, you can say that it is subjective - but in the end it will catch you resp. your successors. And the costs will be exponentially higher than the costs that you would have right now.

Liferay's been doing the OSGi thing for like 3 years now, so I'm sure they have their own ways of dealing with code maintenance issues.

I've been doing OSGI for more than 15 years now and I don't see what code duplication has to do with OSGI.

I really hope that all my complains are not in vain and someone from the lead developers at Liferay is listening as well. I often have to create comparisons of implementation costs for projects and from my point of view the momentum is shifting towards the competitors of Liferay 7.1 now.

RE: Add orderable column to users (or: how customization degrades)

2018-12-18T16:53:18Z

Tobias Liefke:

And I don't think that one of the Liferay architects or one of the Liferay customers will follow your arguments: Rebuild the whole user administration portlet just for an additional sorting option? Let's rebuild the whole portal then, because I need some other functionalities in other areas as well which were not foreseen by the developers.

Well, that's a little extreme... I'm sorry if I gave the impression that this was the only choice. Some overrides are easily implemented and still supported. JSP fragment bundles, for one. So there are ways to affect customization without forcing a rewrite of one of Liferay's modules.

The suggestion, though, is to stay within the lines of allowed customization vs ones that cross the line. Rather than crossing the line, the recommended path would be to create your own.

Valid paths towards customizations include:

JSP Fragment bundles
Service Pre/Post actions
Component overrides
Portlet filters
Service wrappers
Etc

The question comes down to whether your goals can be achieved by some other manner than extending and/or overriding Liferay internal classes.

Tobias Liefke:

I tried to show, that it is not only about one extension point - I needed to create changes all over the place (which everybody would need to do for a new order criteria). This indicates a problem in the whole framework, because the Pareto case for ordering is exactly that: order by one of the fields of the model entity.

This kind of use case will be helpful to Liferay to highlight complexities in approaching some customizations.

Tobias Liefke:

I'm not talking about what I need, I'm talking about what Liferay needs to do to keep their code maintainable.

Well, arguments about code formatting, style and maintainability are often somewhat subjective.

Liferay's been doing the OSGi thing for like 3 years now, so I'm sure they have their own ways of dealing with code maintenance issues.

RE: Add orderable column to users (or: how customization degrades)

2018-12-18T07:57:45Z

Just a personal thing, but I never understood why lambdas were supposed to be so much better than straight-up OOP...

Unfortunately the code just makes them all look boilerplate. Each string is used 3 times, once for the column identifier, once for the sort parameter value in the url, and once for the label to use for the value. Since Liferay uses the same value for all 3 cases, this appears to be boilerplate because you get the same three lines using the same constant string over and over and over. But that's just Liferay, there is the option to override and use different keys or identifiers or ... The fact that Liferay tends to use the same value is representative of the consistency they use in their code. The "buildOrderByDropdownItems(...)" idea would only work when all of the values would be the same, a case that is not always guaranteed.

I don't understand why lambdas are not straight-up OOP. And by the way - I even didn't say anything about lambdas, I only had to use them, because DropdownItemList.add expects one. So please let's focus on the main points of this discussion.

The code doesn't only look boilerplate, it is boilerplate. Just because I want to allow different implementation options I don't need to duplicate code. Offering a default implementation with default options won't prevent someone from using his own implementation, if he needs another option. But this should be no excuse to duplicate the same code a hundred times.

Consistency is not about writing the same code again and again, it is about using the same patterns. Right now if you check the different sources of _getOrderByDropdownItems you will already notice a slight deviation in the implementations (without impact on the functionality) - which will eventually lead to no consistency at all.

I don't want to start a whole new debate on code quality - there are enough websites and books out there dealing with that topic. And if you think that Liferay doesn't duplicate code or that this is a good thing to do so, you should take some time to read them.

... But for Liferay, this was a big problem. Customers would open support tickets, but many times the problems ended up getting traced back to their custom code overrides. What was even worse was when a customer project was over, the dev team that knew about the customizations either rolled off or their contract was up, so the client's devops are left with a response of "the problem is in your custom code" and no way to fix it. It was never that Liferay support didn't want to help the client, they just didn't have the staff or the development expertise to do it. Ultimately though the over-customization aspects of the portal, the thing that we developers enjoy and use extensively, well this ended up impacting relationships at many levels.

OSGi, with its modularization and extensibility, well it also presented an opportunity to reign in the over-customization aspects. By creating defined extension points, the intent is that the platform should still be extendible, but the extensions do not themselves step on Liferay code; the extensions are more "bolt on" so the lines between Liferay code and custom code are much clearer.

While I understand how OSGI helps extending Liferay, I don't see how OSGI does help the support team here - as they still don't know about custom code overrides?

So, what does this mean for your code? Well I think it means you've been trying to force Liferay to be just as over-customizable as it used to be, but much to your own chagrin. Your code is now brittle because it extends and overrides code which is not meant to be extended or overridden. The ViewUsersManagementToolbarDisplayContext class you're extending is not exported from the bundle; I don't know how you've included it in order to extend it (either by embedding the jar or using my blog post to force a package export), but you could be subject to future bugs if Liferay adjusts their version but yours does not get the same adjustments in a timely manner.

A bundle fragment is the key, as it has access to the private packages. I had to create one anyway to implement the columns in the JSP files. And I'm used to check all extensions for incompatibilities on each patch version, even if they extend code which is meant to be extended, because I've learned that even that is not as stable as you try to make me think. One of the reasons for that is the code duplication all over the place.

Basically, as a Liferay developer, there's a new mindset you need to consider adopting. If there is an extension point, use it. If there is not an extension point, I think the idea is that you should "roll your own". That might seem harsh, but again this keeps the lines clear from Liferay's user admin panel and your own users admin panel. Having your own admin panel isolates your code and protects against changes to Liferay's internal code, and you are free to use the Liferay APIs to provide similar functionality, but the difference for Liferay is that bugs in your custom admin panel are, well, your bugs and not Liferay's.

Me, now I personally sit on both sides of this fence... I can see the benefits it has for Liferay, but as a developer like yourself I would hate to develop and maintain my own user admin portlet just so I could add an email column to the table. Like you, I too want to extend what Liferay provides and only layer in the changes I need to get that column without having to create and maintain the whole thing myself.

But, when we choose that road, we're specifically going down a path that we should know is not supported and might break as early as the next fix pack or GA, let alone the next major version.

Is there a better way? Sure! Liferay thinks they have most of the necessary extension points in place, but they don't believe they have all of them and they are open to adding new ones. Pop over to issues.liferay.com and open a ticket asking for a new extension point and it could make it in. If you're a client, you can use the new help.liferay.com to open a support ticket and work the extension point that way.

Remember their goal is not to lock you out of customizing Liferay, they just want to define a controlled way to do it where you can customize the platform but your changes are independent and immunized from changes in the Liferay internals.

I think you misinterpreted one of the main points of my post. When talking about maintainability and readibility I meant for the developers at Liferay themselves, not for Liferay extension developers. Put it that way: The easier it is for developers at Liferay to implement a new field and add it as column to the UI, the easier it is for us extension developers to add similar functionality.

And I don't think that one of the Liferay architects or one of the Liferay customers will follow your arguments: Rebuild the whole user administration portlet just for an additional sorting option? Let's rebuild the whole portal then, because I need some other functionalities in other areas as well which were not foreseen by the developers.

I tried to show, that it is not only about one extension point - I needed to create changes all over the place (which everybody would need to do for a new order criteria). This indicates a problem in the whole framework, because the Pareto case for ordering is exactly that: order by one of the fields of the model entity. And that is possible with just one line of code, not with that bunch of duplicated code - if you do it right (I've seen enough frameworks which do this). Again: I'm not talking about what I need, I'm talking about what Liferay needs to do to keep their code maintainable.

RE: Add orderable column to users (or: how customization degrades)

2018-12-17T15:04:46Z

Tobias Liefke:

Conclusion:

I had to write a lot of code and had to use a lot of inside knowledge of Liferay - which may change already by the next patch version. Just to add some simple sortable date columns.

By stepping through the code I found many places where simle coding standards like the DRY principle were violated - with huge drawbacks for maintainability, extensibility and readability. Liferay 7.1 has about 3.2 million code lines in Java files (not counting empty / commented lines, JSP files and other templates). This is an increase by ~ 300,000 code lines (11%) compared to Liferay 7.0. Some might say this is an indicator for more functionality. But for me this is an indicator for more sources of errors and vulnerabilities, more code nobody is aware of, more code to maintain, more code you need to overwrite for your own extensions.

Thus this post is more a call for higher code quality in Liferay than it is a tutorial on how to add sortable columns in Liferay.

Without taking any sides, let me kind of explain...

In Liferay 6 and earlier, all code was subject to override. Every JSP, every class, every resource file... It was super easy for developers to get in there and override what was necessary to get their jobs done.

But for Liferay, this was a big problem. Customers would open support tickets, but many times the problems ended up getting traced back to their custom code overrides. What was even worse was when a customer project was over, the dev team that knew about the customizations either rolled off or their contract was up, so the client's devops are left with a response of "the problem is in your custom code" and no way to fix it. It was never that Liferay support didn't want to help the client, they just didn't have the staff or the development expertise to do it. Ultimately though the over-customization aspects of the portal, the thing that we developers enjoy and use extensively, well this ended up impacting relationships at many levels.

OSGi, with its modularization and extensibility, well it also presented an opportunity to reign in the over-customization aspects. By creating defined extension points, the intent is that the platform should still be extendible, but the extensions do not themselves step on Liferay code; the extensions are more "bolt on" so the lines between Liferay code and custom code are much clearer.

When you see things such as a bundle w/o a package export for a class you want to override or an internal, private method, well these are just some of the ways in which the over-customization have been reigned in.

With the clean lines between Liferay code and custom code, another correlary to this is that the Liferay code is now subject to change w/o necessarily broadcasting the change to you. As long as the extension points still exist and the contracts don't change, they may change up the internals at any time because those changes are internal and should not bleed over to the custom code via the extension points.

So, what does this mean for your code? Well I think it means you've been trying to force Liferay to be just as over-customizable as it used to be, but much to your own chagrin. Your code is now brittle because it extends and overrides code which is not meant to be extended or overridden. The ViewUsersManagementToolbarDisplayContext class you're extending is not exported from the bundle; I don't know how you've included it in order to extend it (either by embedding the jar or using my blog post to force a package export), but you could be subject to future bugs if Liferay adjusts their version but yours does not get the same adjustments in a timely manner.

Basically, as a Liferay developer, there's a new mindset you need to consider adopting. If there is an extension point, use it. If there is not an extension point, I think the idea is that you should "roll your own". That might seem harsh, but again this keeps the lines clear from Liferay's user admin panel and your own users admin panel. Having your own admin panel isolates your code and protects against changes to Liferay's internal code, and you are free to use the Liferay APIs to provide similar functionality, but the difference for Liferay is that bugs in your custom admin panel are, well, your bugs and not Liferay's.

Me, now I personally sit on both sides of this fence... I can see the benefits it has for Liferay, but as a developer like yourself I would hate to develop and maintain my own user admin portlet just so I could add an email column to the table. Like you, I too want to extend what Liferay provides and only layer in the changes I need to get that column without having to create and maintain the whole thing myself.

But, when we choose that road, we're specifically going down a path that we should know is not supported and might break as early as the next fix pack or GA, let alone the next major version.

Is there a better way? Sure! Liferay thinks they have most of the necessary extension points in place, but they don't believe they have all of them and they are open to adding new ones. Pop over to issues.liferay.com and open a ticket asking for a new extension point and it could make it in. If you're a client, you can use the new help.liferay.com to open a support ticket and work the extension point that way.

Remember their goal is not to lock you out of customizing Liferay, they just want to define a controlled way to do it where you can customize the platform but your changes are independent and immunized from changes in the Liferay internals.

RE: Add orderable column to users (or: how customization degrades)

2018-12-17T14:27:44Z

Tobias Liefke:

But instead of a straight forward buildOrderByDropdownItems("first-name", "last-name, "screen-name"), there is the same boilerplate code again and again.

Just a personal thing, but I never understood why lambdas were supposed to be so much better than straight-up OOP...

Unfortunately the code just makes them all look boilerplate. Each string is used 3 times, once for the column identifier, once for the sort parameter value in the url, and once for the label to use for the value. Since Liferay uses the same value for all 3 cases, this appears to be boilerplate because you get the same three lines using the same constant string over and over and over. But that's just Liferay, there is the option to override and use different keys or identifiers or ... The fact that Liferay tends to use the same value is representative of the consistency they use in their code. The "buildOrderByDropdownItems(...)" idea would only work when all of the values would be the same, a case that is not always guaranteed.

RE: Add orderable column to users (or: how customization degrades)

2018-12-17T10:39:57Z

It turns out that this was only the tip of the iceberg:

To ensure that the email address is indexed, I had to add that property to my portal-ext.properties:

index.sortable.text.fields=firstName,jobTitle,lastName,name,screenName,title,emailAddress

(which I had already done in 6.2 for one of my instances - and I needed some time to find out why email addresses weren't sorted on my other instances).

Now I wanted to add some more columns, thus I improved my ViewUsersManagementToolbarDisplayContext.java:

public class ViewUsersManagementToolbarDisplayContext
        extends com.liferay.users.admin.web.internal.display.context.ViewUsersManagementToolbarDisplayContext {

    private HttpServletRequest request;

    public ViewUsersManagementToolbarDisplayContext(HttpServletRequest request, RenderRequest renderRequest,
            RenderResponse renderResponse, String displayStyle, String navigation, int status) {
        super(request, renderRequest, renderResponse, displayStyle, navigation, status);
        this.request = request;
    }

    @Override
    public List<DropdownItem> getFilterDropdownItems() {
        List<DropdownItem> items = super.getFilterDropdownItems();
        addOrderColumn(items, "email-address", "email-address");
        addOrderColumn(items, "createDate", "create-date");
        addOrderColumn(items, "lastLoginDate", "Login");
        addOrderColumn(items, "lockoutDate", "locked");
        addOrderColumn(items, "jobTitle", "job-title");
        return items;
    }

    private void addOrderColumn(List<DropdownItem> items, final String columnKey, String columnLabel) {
        DropdownGroupItem sortGroup = (DropdownGroupItem) items.get(items.size() - 1);
        DropdownItemList itemList = (DropdownItemList) sortGroup.get("items");
        itemList.add(dropdownItem -> {
            dropdownItem.setActive(Objects.equals(getOrderByCol(), columnKey));
            dropdownItem.setHref(getPortletURL(), "orderByCol", columnKey);
            dropdownItem.setLabel(LanguageUtil.get(this.request, columnLabel));
        });
    }

}

But this isn't enough, as there is a mapper function com.liferay.portlet.usersadmin.util.UsersAdminImpl.getUserOrderByComparator which maps the column names to a comparator. And that function returns the UserLastNameComparator for every unknown column, which includes all my date columns. UsersAdmin is an old fashioned Liferay service class, which means it resides in portal-impl.jar and is sometimes referenced with UsersAdminUtil and sometimes injected with @Reference.

In 6.2 I would have written a hook or ext-plugin to wrap that service in my own implementation class (and use spring-ext.xml to install it), but for 7.1 I decided to write a new plugin which contains my wrapper class, with some additional tricks from my magic bag:

@Component(immediate = true, service = UsersAdmin.class)
public class MyUsersAdminImpl implements UsersAdmin {

    private UsersAdmin wrapped;

    @Activate
    public void activate() {
        // Install our implementation on top of the existing implementation
        if (UsersAdminUtil.getUsersAdmin() instanceof MyUsersAdminImpl) {
            this.wrapped = ((MyUsersAdminImpl) UsersAdminUtil.getUsersAdmin()).getWrapped();
        } else {
            this.wrapped = UsersAdminUtil.getUsersAdmin();
        }
        new UsersAdminUtil().setUsersAdmin(this);
    }    

    @Deactivate
    public void deactivate() {
        // Reinstall the previous implementation
        if (UsersAdminUtil.getUsersAdmin() == this) {
            new UsersAdminUtil().setUsersAdmin(this.wrapped);
        }
    }

    ... all UsersAdmin functions delegation to wrapper ...

}

Now I implemented my own comparator, but instead of one per property (as done in Liferay) I like a more generic aproach:

public class GenericComparator<T> extends OrderByComparator<T> {
    private static final long serialVersionUID = 1L;

    private final boolean ascending;

    private final String orderBy;

    private final String[] orderByFields;

    private final Comparator<T> comparator;

    public GenericComparator(String field, Comparator<T> comparator) {
        this(field, comparator, true);
    }

    public GenericComparator(String field, Comparator<T> comparator, boolean ascending) {
        this(field + (ascending ? " ASC" : " DESC"), new String[] { field }, comparator, ascending);
    }

    public GenericComparator(String orderBy, String[] orderByFields, Comparator<T> comparator, boolean ascending) {
        this.orderBy = orderBy;
        this.orderByFields = orderByFields;
        this.comparator = comparator;
        this.ascending = ascending;
    }

    @Override
    public int compare(T entity1, T entity2) {
        final int value = this.comparator.compare(entity1, entity2);
        return this.ascending ? value : -value;
    }

    @Override
    public String getOrderBy() {
        return this.orderBy;
    }

    @Override
    public String[] getOrderByFields() {
        return this.orderByFields;
    }

    @Override
    public boolean isAscending() {
        return this.ascending;
    }

}

That comparator I can use in my own getUserOrderByComparator:

@Component(immediate = true, service = UsersAdmin.class)
public class MyUsersAdminImpl implements UsersAdmin {

    ...

    @Override
    public OrderByComparator<User> getUserOrderByComparator(String orderByCol, String orderByType) {
        // Support the default comparators
        if (orderByCol.indexOf('-') >= 0) {
            return this.wrapped.getUserOrderByComparator(orderByCol, orderByType);
        }

        // Support additional (generic) comparators
        final boolean orderByAsc = orderByType.equals("asc");
        try {
            final Method getter = User.class.getMethod("get" + StringUtil.upperCaseFirstLetter(orderByCol));
            return new GenericComparator<>(orderByCol, (a, b) -> {
                try {
                    return ((Comparable<Object>) getter.invoke(a)).compareTo(getter.invoke(b));
                } catch (IllegalAccessException | InvocationTargetException e) {
                    throw new IllegalArgumentException(e);
                }
            }, orderByAsc);
        } catch (NoSuchMethodException | SecurityException e) {
            return this.wrapped.getUserOrderByComparator(orderByCol, orderByType);
        }
    }

}

And this isn't still enough, because all the date columns are not indexed. Unfortunately I can't add these to index.sortable.text.fields from above, because they are not of type text.

Thus I added a UserIndexPostProcessor to my plugin:

@Component(immediate = true, service = IndexerPostProcessor.class, 
        property = { "indexer.class.name=com.liferay.users.admin.internal.search.UserIndexer" })
public class UserIndexerPostProcessor extends BaseIndexerPostProcessor {

    public void postProcessDocument(Document document, Object obj) throws Exception {
        final User user = (User) obj;
        // Add date fields for sorting to index
        document.addDateSortable("createDate", user.getCreateDate());
        document.addDateSortable("lastLoginDate", user.getLastLoginDate());
        document.addDateSortable("lockoutDate", user.getLockoutDate());
    }

    @Override
    public void postProcessSearchQuery(BooleanQuery searchQuery, BooleanFilter booleanFilter,
            SearchContext searchContext) throws Exception {
        // Ensure that datefields are mapped correctly
        final Sort[] sorts = searchContext.getSorts();
        if (sorts != null) {
            for (int i = 0; i < sorts.length; i++) {
                final Sort sort = sorts[i];
                if (sort.getType() == Sort.STRING_TYPE && sort.getFieldName().endsWith("Date")) {
                    sorts[i] = new Sort(sort.getFieldName() + "_Number_sortable", Sort.LONG_TYPE, sort.isReverse());
                }
            }
        }
    }

}

And after reindexing the users I finally could sort by all these new columns.

Conclusion:

I had to write a lot of code and had to use a lot of inside knowledge of Liferay - which may change already by the next patch version. Just to add some simple sortable date columns.

By stepping through the code I found many places where simle coding standards like the DRY principle were violated - with huge drawbacks for maintainability, extensibility and readability. Liferay 7.1 has about 3.2 million code lines in Java files (not counting empty / commented lines, JSP files and other templates). This is an increase by ~ 300,000 code lines (11%) compared to Liferay 7.0. Some might say this is an indicator for more functionality. But for me this is an indicator for more sources of errors and vulnerabilities, more code nobody is aware of, more code to maintain, more code you need to overwrite for your own extensions.

Thus this post is more a call for higher code quality in Liferay than it is a tutorial on how to add sortable columns in Liferay.

Add orderable column to users (or: how customization degrades)

2018-11-05T22:17:21Z

I am just migrating a project from Liferay 7.0 to 7.1.

That project has a fragment for com.liferay.users.admin.web to add some columns to the user administration.

In Liferay 6.2 I had a hook where I just needed to add a column like this to /html/portlet/users_admin/user/search_columns.jspf :

<liferay-ui:search-container-column-text
    href="<%= rowURL %>"
    name="email-address"
    orderable="<%= true %>"
    property="emailAddress"
/>

In Liferay 7.0 I still needed to add that column, but the orderable flag was ignored (allthough it still exists, even for the predefined columns which is quite confusing).

Instead I had to adjust the file view_flat_users.jspf and change the management-bar-sort:

<liferay-frontend:management-bar-sort
     orderByCol="<%= searchContainer.getOrderByCol() %>"
     orderByType="<%= searchContainer.getOrderByType() %>"
     orderColumns='<%= new String[] {"first-name", "last-name", "screen-name", "email-address" } %>'
     portletURL="<%= portletURL %>"
/>

And now in Liferay 7.1 with the new "Clay" framework all this is still not enough.

I have to override ViewUsersManagementToolbarDisplayContext.getFilterDropdownItems():

    public List<DropdownItem> getFilterDropdownItems() {
        List<DropdownItem> items = super.getFilterDropdownItems();
        DropdownGroupItem sortGroup = (DropdownGroupItem) items.get(items.size() - 1);
        ((DropdownItemList) sortGroup.get("items")).add(dropdownItem -> {
            dropdownItem.setActive(Objects.equals(getOrderByCol(), "email-address"));
            dropdownItem.setHref(getPortletURL(), "orderByCol", "email-address");
            dropdownItem.setLabel(LanguageUtil.get(this.request, "email-address"));
        });
        return items;
    }

(and I still need to modify view_flat_users.jsp or init.jsp to reference my class instead of the default one and I need to modify search_columns.jspf to include my column).

To make things worse, the method ViewUsersManagementToolbarDisplayContext._getOrderByDropdownItems is private. And there are about 130 "_getOrderByDropdownItems()" implementations and many look the same. But instead of a straight forward buildOrderByDropdownItems("first-name", "last-name, "screen-name"), there is the same boilerplate code again and again.

I've got the feeling that even if we now are able to override everything with the OSGI fragments, the new frameworks make it difficult to customize anything, because of many implentation specific details, which are only accessible with much boilerplate code and are subject to change even on patch version upgrades. Am I the only one with that impression?