RE: LAR imports/exports

thumbnail
تم تعديل 25126 منذ 19 سنوات من الدقائق. New Member المشاركات: 22 تاريخ الإنضمام: 3‏/8‏/06 المشاركات الحديثة
We recently upgraded to Liferay 4.2.1 in hopes that we could begin utilizing LAR importing/exporting to push changes from developement/staging environments directly to our production cluster. However, when attempting a LAR import/export from development to staging we ran into the following issues:

* Layouts more than 1 level deep that were deleted in development were not deleted from staging after the import.
* We have a large number of pages in our public community, and the import took a minute or two to complete. During this time, the public pages were not browseable; requests would time out.

Are others experiencing similar issues with LARs? As another question: what approaches are people taking to push portal changes from one environment to the next? We're finding ourselves having to export and import databases directly, which is a laborious process in a clustered environment.
thumbnail
تم تعديل 15091 منذ 19 سنوات من الدقائق. Junior Member المشاركات: 52 تاريخ الإنضمام: 13‏/5‏/06 المشاركات الحديثة
check if this issue is fixed in the trunk. if not raise an issue on that in JIRA.

regarding publishing/staging ... this is on the roadmap for 4.3.0 (Owen). see also here
thumbnail
تم تعديل 25126 منذ 19 سنوات من الدقائق. New Member المشاركات: 22 تاريخ الإنضمام: 3‏/8‏/06 المشاركات الحديثة
I have opened one JIRA ticket for one of our primary concerns (layout child pages) LEP-2594. I have not opened any regarding site downtime during an import.

I actually had not looked at the 4.3.0 (Owen) roadmap before now. I see that publishing/staging is listed on there, but the wiki page it links to doesn't contain any information. What features are expected with regards to staging/publishing in 4.3.0?
thumbnail
تم تعديل 10527 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1197 تاريخ الإنضمام: 8‏/2‏/05 المشاركات الحديثة
Yes, I can see the problem now. It's more because the LAR does not delete anything really... this is something I'm working on fixing.

Currently, if there is a layout in the LAR, it will overwrite the existing one... but does nothing with it's children, because there isn't one in the LAR (it was deleted).

So what we need are import/export options which we can set, like so:

- merge LAR content with existing (overwrites exiting, doesn't delete existing)
- overwrite existing completely with LAR (overwrites existing, deleting what's missing, like missing Layouts...)
- don't import older?
- etc...

Any other scenarios you can think of? Oh, I am adding this so that each DataHandler can support it's own options, so developers can have import/export options in their DataHandler show up on the import/export UI. Every portlet which has a DataHandler will be listed on the page and show it's options... this will provide fine grained control over import/export.

Sometimes you'll want to import/export users, other times you won't...
thumbnail
تم تعديل 10527 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1197 تاريخ الإنضمام: 8‏/2‏/05 المشاركات الحديثة
Ray Auge:
It's more because the LAR does not delete anything really...


This is wrong for the code in SVN (which deletes non existent layouts), but might be true depending on the version you're using... Figured I better clear that up.
thumbnail
تم تعديل 25126 منذ 19 سنوات من الدقائق. New Member المشاركات: 22 تاريخ الإنضمام: 3‏/8‏/06 المشاركات الحديثة
To be perfectly honest, I haven't come across a really deep community discussion around LARs and the use cases, so I really welcome all comments to this post. That said, I'd like to offer my thoughts on LARs and the uses cases as I see them, because I'm beginning to think I'm not on the same page as the developers of LAR functionality.

Right now, the only way to move a "site" (community, etc.) from one instance of liferay to another is through the use of a LAR. In the case of moving a site, it will always be the preferred behavior to delete and replace. That is, I can't think of a use case where I would want to preserve something when I'm trying to migrate a site. (Our use case here is moving changes from a development environment to a staging, or from staging to production, etc.)

Often, it's not that case that we need, or even want, to move an entire site. For example, if a page were added to a dev environment, we most likely only want to copy this page over to a stage environment; carrying the whole community as a LAR is needless bulky and slower, plus it runs into the issues of what to do with other duplicate pages, etc. In these common use cases, a single page, or multiple-but-individually-selectable set of pages would be preferred over a full community import/export. In the case of individual layouts, it would be necessary to have an export checkbox like "include child layouts."

The last of my major concerns is with the importing of LARs: during an import in production, our site does not respond to requests. It seems there is a blocking mechanism in place as the LAR is imported. So in order to push a community from staging to production, we need to employ a fairly complex process of removing one node from our production cluster, pointing it to a new copy of the production database, import the LAR into this "renegade" node, then one-by-one switch our other nodes to this new cluster. (Because of the issues with not deleting child pages, etc., we actually just export and import the full database, but this will cause problems for us as we begin working with multiple communities.) I'm not sure if there is a different algorithm that could be used, such as blocking only the layout currently being replaced and doing one at a time, or something like that, but blocking the entire production site has some significant financial implications for us.

With these use cases in mind, most of the merging abilities laid out in Ray Auge's post don't seem necessary applicable to me. At least for my organization, we will rarely, if ever, want to be merging communities; we'll either want a full community replacement or a simple replacement/addition of a single layout (maybe with children).

That said, we haven't worked with a lot of the user/security settings yet. I do see the need for import/export options about weather to bring user permissions along or not. Admittedly, I have not thought about this issue in as much detail yet.

Thoughts?
thumbnail
تم تعديل 10527 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1197 تاريخ الإنضمام: 8‏/2‏/05 المشاركات الحديثة
Daniel Robert:
To be perfectly honest, I haven't come across a really deep community discussion around LARs and the use cases, so I really welcome all comments to this post.


This is as good a time as any. emoticon

Daniel Robert:
In the case of moving a site, it will always be the preferred behavior to delete and replace. That is, I can't think of a use case where I would want to preserve something when I'm trying to migrate a site. (Our use case here is moving changes from a development environment to a staging, or from staging to production, etc.)
...
Often, it's not that case that we need, or even want, to move an entire site. For example, if a page were added to a dev environment, we most likely only want to copy this page over to a stage environment; carrying the whole community as a LAR is needless bulky and slower, plus it runs into the issues of what to do with other duplicate pages, etc. In these common use cases, a single page, or multiple-but-individually-selectable set of pages would be preferred over a full community import/export. In the case of individual layouts, it would be necessary to have an export checkbox like "include child layouts."


This is a great idea. With the changes I'm just about finished with, this should be very easy to implement. BTW, to argue the point about "merging", this is the type of merge I had in mind, adding individual pieces of new content or data to existing. But, there are other slightly different use cases besides development staging.

Daniel Robert:
The last of my major concerns is with the importing of LARs: during an import in production, our site does not respond to requests. It seems there is a blocking mechanism in place as the LAR is imported. So in order to push a community from staging to production, we need to employ a fairly complex process of removing one node from our production cluster, pointing it to a new copy of the production database, import the LAR into this "renegade" node, then one-by-one switch our other nodes to this new cluster. (Because of the issues with not deleting child pages, etc., we actually just export and import the full database, but this will cause problems for us as we begin working with multiple communities.) I'm not sure if there is a different algorithm that could be used, such as blocking only the layout currently being replaced and doing one at a time, or something like that, but blocking the entire production site has some significant financial implications for us.


I think the culprit here is likely the LayoutSet (the owner of very layout in a community), which is likely locked in a hibernate transaction. I'll look into this.

Daniel Robert:
With these use cases in mind, most of the merging abilities laid out in Ray Auge's post don't seem necessary applicable to me. At least for my organization, we will rarely, if ever, want to be merging communities; we'll either want a full community replacement or a simple replacement/addition of a single layout (maybe with children).


My idea of merging is exactly the one you pointed out above, many new pages added among existing pages, without overwriting, or deleting existing.
thumbnail
تم تعديل 10527 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1197 تاريخ الإنضمام: 8‏/2‏/05 المشاركات الحديثة
Daniel Robert:
That said, we haven't worked with a lot of the user/security settings yet. I do see the need for import/export options about weather to bring user permissions along or not. Admittedly, I have not thought about this issue in as much detail yet.


The current LAR functionality in Liferay is currently limited in the sense that there is no real control over what is happening, so it's hard to really get a grasp of it's use cases due to it being outwardly basic functionality.

So what exactly are we trying to do with LARs? Is it a development stagging feature? Is it an archiving feature? Is it a packaging feature?

Well, it's mostly all of these. Currently, you wouldn't think so, or you think it's pretty underdeveloped, and it is. But this is because there is no outward control over its exact feature set. And this is still being fleshed out.

Ultimately my idea is to provide a fairly rich API for handling LARs. One feature we added today was the ability for the DataHandler classes to have access to the ZipWriter from withing the Handler itself. This will allow DataHandler to load/store binary files directly withing the LAR as opposed to in serial form. Another piece which is missing that I'm working on currently, is providing fine grained controls to DataHandler developers to implement different behaviors durng import/export accoring to user selected options.

The use case I refer to for the design is the LMS (Learning Management System), and in particular with "Online Courses". Online Courses can be very well represented as Communities in Liferay. In this context a Community, being a distinct "web site" for the sake of education, is probably the most transient type of online community, in the sense that it's users have access for limited amount of time, its data is purged frequently (every term?), may require many exact copies (sectioning of students, max per instructor, etc.).

In this scenario there is a specific need to be able to frequently store complete record of all data in a given "course". It might be due to verionning of content, duplication, or for pure archival purposes. In certain countries it is legislated that paper and/or electronic record be kept for a minimum number of years. Often where it isn't legislated, it's institutional policy.

Under an ASP model, where the client does not have control of the underlying data store, only a client interface, there is a need to provide a means to the client of both loading and unloading content and/or data to and from their application. A client may have made a full backup of their site, and for whatever reason wants to restore only the content from it, without all the included user data...

So you see, there are several use cases we have to take into account. So we need more control over individual behavior. Also providing a good API for developers to leverage these features is also very important.

Currently, it's not possible the go from version X to version Y of the portal, but I'm going to try having something partially implemented sometime in the near future.
thumbnail
تم تعديل 25126 منذ 19 سنوات من الدقائق. New Member المشاركات: 22 تاريخ الإنضمام: 3‏/8‏/06 المشاركات الحديثة
Ray Auge:

So what we need are import/export options which we can set, like so:

- merge LAR content with existing (overwrites exiting, doesn't delete existing)
- overwrite existing completely with LAR (overwrites existing, deleting what's missing, like missing Layouts...)
- don't import older?
- etc...

Any other scenarios you can think of?


I suppose the options for export would be different than the options for import. I could envision options like this:

Exports:
  • Include themes?
  • Include layouts? (is this desired functionality?)
  • pages to export: (provide user a mechanism to select individual pages/groups of pages, defaults to entire community)


This list is definitely missing user/security options. I wonder if such settings belong here or in the import list, or both.

Imports:
  • Replace all? or Merge content? (either-or, like a radio-button)
  • Import security settings?
  • Import themes/layouts?


Clearly I need to think about this list longer...it's an interesting question. To be honest, most of these are just "niceties" to me at the moment, until I can apply LARs effectively in our current use case.

Ray Auge:
Oh, I am adding this so that each DataHandler can support it's own options, so developers can have import/export options in their DataHandler show up on the import/export UI.


I actually was not familiar with data handlers until you said this. This is something we'll definitely keep in mind as we develop additional portlets. Is there a good reference on using this feature somewhere? I assume you're adding some additional methods to the PortletDataHandler interface to support the import/output display? That's pretty cool.
thumbnail
تم تعديل 21831 منذ 19 سنوات من الدقائق. Regular Member المشاركات: 149 تاريخ الإنضمام: 7‏/2‏/06 المشاركات الحديثة
We'll try to write up our use case (deployment/Staging/Workflow) and post it here. I think it would be helpful if we know of other usecases to do the same. That will allow us to figure out where we can tackle the problem with a common solution and where the vertical requirements are. Once we identify enough vertical requirements to recognize patterns we can discuss frameworks to facilitate the pluggability of special case implementations.

In the end we want to strive for simple, functional and correct -- everything else is an optimization.
thumbnail
تم تعديل 18941 منذ 19 سنوات من الدقائق. Expert المشاركات: 405 تاريخ الإنضمام: 28‏/6‏/06 المشاركات الحديثة
Russ Danner:
I think it would be helpful if we know of other usecases to do the same.


My use case is we are using the "Community Virtual Hosting" to host individual web sites for our customers (a group of businesses in the same industry). Since each client has "about the same" needs as far as content is concerned, I have a two .LAR files that represent the "starting point" of their "public" internet site and their "private" intranet site. Using the "CMS + Portal = Web Site" strategy, in these .LAR files I have articles that represent the site's default content. In the articles themselves, I have embedded certain "merge variable" place holders for things like company name, company phone, etc. I use a custom data handler for the import that calls Velocity to merge an Organization's demographic data with the articles when the site is first created.

Now, here is one issue relevant to this discussion: We supply a couple of custom themes for the users to choose from (and once we get into full scale production, we plan to have a very large suite of themes for users to choose from). Creating a single "default site" that is generic enough to work with ALL of the themes sacrafices appearance for the ability to quickly change from theme to theme (i.e. we emphasized "function" over "form" when creating our default site).

The "home page" in particular suffers the most. For any one particular theme, a different "layout" of the articles (and different settings, like hiding borders, titles, etc.) may be more desireable than the "defaults" we are using. So, I had this vision of a "page library" or a "layout library" where once a user has selected a theme to use, they could select from a page library for a "home page" that is more appropriate for their theme. It would basically be the same article(s) and portlets for any given page, they just may be layed out differently, and their appearance may also be different.

Taking this a step further (I just thought of this while I was typing :grinemoticon, what would REALLY be cool (and easier for my less than web savy customers) would be to have a "layout only .LAR" tied with each theme. A user would select a theme, and all of the pages' layouts' change to match something more "appropriate" for that theme. Doing that would mean I'd want to import only certain aspects of the .LAR - for each page, I'd want to have the layout template, the portlet arrangements, and the portlet's appearance configuration (am I showing a border, what is the title, etc.). I would NOT want the articles themeslves re-imported, nor any of the other content that might be stored in the .LAR. Basically, I'm talking about an "appearance only" .LAR
thumbnail
تم تعديل 10527 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1197 تاريخ الإنضمام: 8‏/2‏/05 المشاركات الحديثة
I wanted to place here a couple of the prototype modifications to the Import/Export/Staging views to get some of your feedback.

This view illustrates a few of the things I'm consideing changing.


the first thing you'll probably notice are the publish/copy buttons have moved from above the left layout tree to their own tab under the Import/Export tab. I do think that perhaps the name of the Import/Export tab could be changed, but the idea is to clarify the concept that publishing/staging operations can be affected by the Import/Export options, such as the DataHandlers and how they they might affect data, etc... But more notably, the option to select which layouts are to be handled.

Next you'll notice the dynamic per/DataHandler options bellow, here you see that this community has the Journal and Journal Content portlets and that they have options which can affect their behaviour. (the actual logic on what they do currently is simply to enable them or not, but might have staging specific options, as any number of options is possible per DataHandler.

Anyway, here is the view with the "Selected Layouts":


One thing that isn't there is the option which affects how missing layouts are affected. i think this one is critical and I want to make sure it's laid out properly.

Anyway, food for thought. Let me know if you think this could be helpful to your use cases.
thumbnail
تم تعديل 18941 منذ 19 سنوات من الدقائق. Expert المشاركات: 405 تاريخ الإنضمام: 28‏/6‏/06 المشاركات الحديثة
Ray Auge:
Let me know if you think this could be helpful to your use cases.


Ray -

This looks great! As soon as I get my first round of production portlets finished, I'll check out the latest code so I can get some of this.

In looking at the above, the first thing that came to my mind is "some way to save the settings". Perhaps an XML file or some other format that describes the export options. Then, what I would do would be to capture my export configuration in some persistent form, and then manually or programatically apply that export configurtion to FUTURE exports. I suppose that ideally, the options would be tied to the community being exported.

I already have a single community that serves as my "template" community. If I need to make a change to the "default content", I modify that community, then export its results. What I may need to do is have SEVERAL template communities - one for each theme. I'd then need to export each community variation with only specific settings. Of course, I could keep track of those manually, but it would be nice to have to only set it up once.

You mentioned that it was your plan to have an API to allow finer control over how the export is done. I think you are on the right track. Given what you've already written about, I think you've got most, if not all, of the stuff I'd need for my use case.
thumbnail
تم تعديل 21831 منذ 19 سنوات من الدقائق. Regular Member المشاركات: 149 تاريخ الإنضمام: 7‏/2‏/06 المشاركات الحديثة
Well I haven't been good about getting our use posted -- the least I can do is give some feedback here.

I see the activate staging check box. When I click this what am i turning on? How to I describe what staging is?

In this implementation the job of exporting and importing configuration from one environment to another feels (to me) to be a technical task. That is fine. With an API in place it may be possible to hide the configuration behind simple workflow -- the user getting a notification could the approve or deny a page which would then trigger the export/import from on environment to another based on something preconfigured.


The other thing that I noticed was the lack of a way to express a page removal.
thumbnail
تم تعديل 25246 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1519 تاريخ الإنضمام: 7‏/8‏/06 المشاركات الحديثة
Ray that looks great. Will there be a possibility to publish to a remote portal instance to have different public and staging portals?
thumbnail
تم تعديل 10527 منذ 19 سنوات من الدقائق. Liferay Legend المشاركات: 1197 تاريخ الإنضمام: 8‏/2‏/05 المشاركات الحديثة
Mika Koivisto:
Will there be a possibility to publish to a remote portal instance to have different public and staging portals?


Hopefully in the not too distant future. emoticon

In the meantime, I'd like to ask you to have a look at this article I just finished writing on the new framework. (still needs a few screen shots and such but it's 98% done).

PortletDataHandlers