RE: Elastic search Distance search with liferay 7

Jiten Vaghela, modified 7 Years ago.

Elastic search Distance search with liferay 7

New Member Posts: 20 Join Date: 8/22/16 Recent Posts

Hello All.

I have one requirement, In which I need to search record based on geo distance search.

I tried a lot but I didn't get any example of it:

My requirement is :
lets say we have Car owners with some location[lat,lon]stored in custom entity and have indexed document(index document have lat, lon value).
example:
"lon":"73.191948",
"lat":"19.186354",


{
	"_index":"liferay-20116",
	"_type":"LiferayDocumentType",
	"_id":"com.test.model.CarOwner_PORTLET_2",
	"_score":1,
	"_source":{
		"entryClassPK":"2",
		"scopeGroupId_sortable":20143,
		"entryClassName":"com.test.model.CarOwner",
		"groupId":"20143",
		"lon":"73.191948",
		"createDate_sortable":1508248053466,
		"stagingGroup":"false",
		"uid":"com.test.model.CarOwner_PORTLET_2",
		"scopeGroupId":"20143",
		"status_sortable":1,
		"groupId_sortable":20143,
		"modified":"20171108065033",
		"modified_sortable":1510123833893,
		"lat":"19.186354",
		"status":"1",
		"createDate":"20171017134733"
	}
}

who's location coordinates are latitude: 40, longitude: -70.
Based on above location coordinate, I want to find out those Car owner who is within 200KM range.

I referred below link for geo search with elasticsearch:
GeoDistance Search with ElasticSearch

I have used BooleanQuery to search CarOwner based on other criterion.

Can anyone help me out?

Jorge Díaz, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

Liferay Master Posts: 753 Join Date: 1/9/14 Recent Posts

You have to use the Geo* filters, that are located in com.liferay.portal.kernel.search.filter. For example: https://github.com/liferay/liferay-portal/blob/master/portal-kernel/src/com/liferay/portal/kernel/search/filter/GeoDistanceFilter.java

You have more information about Liferay 7 search filters in following pages:

Filters can be applied to queries, see: https://github.com/liferay/liferay-portal/blob/3ad9823522f69dfe74ce69c2616e454a9f28769f/portal-kernel/src/com/liferay/portal/kernel/search/Query.java#L52-L54

Jiten Vaghela, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

New Member Posts: 20 Join Date: 8/22/16 Recent Posts

Thanks Jorge Díaz for your valuable response.

As you say, I made some changes like below:

Indexer:


protected Document doGetDocument(CarOwner carOwner) throws Exception {
		Document document = getBaseModelDocument(CLASS_NAME, carOwner);
		
		AssetCategory provinceCategory = _assetCategoryLocalService.getCategory(carOwner.getProvinceId());
		if(provinceCategory != null){
			AssetCategoryProperty latitudeProperty = _assetCategoryPropertyLocalService.getCategoryProperty(provinceCategory.getCategoryId(), CarOwnerConstant.LATITUDE);
			AssetCategoryProperty longitudeProperty = _assetCategoryPropertyLocalService.getCategoryProperty(provinceCategory.getCategoryId(), CarOwnerConstant.LONGITUDE);
			if(latitudeProperty !=null  &amp;&amp; longitudeProperty != null){
				document.addGeoLocation("location", Double.parseDouble(latitudeProperty.getValue()), Double.parseDouble(longitudeProperty.getValue()));
			}			
		}
		
		document.addText(CarOwnerConstant.REGION_ID, String.valueOf(carOwner.getProvinceId()));
		document.addNumber(Field.GROUP_ID, carOwner.getGroupId());
		document.addNumber(Field.SCOPE_GROUP_ID, carOwner.getGroupId());
		document.addDate(Field.MODIFIED_DATE, carOwner.getModifiedDate());
		document.addNumber(Field.STATUS, carOwner.getStatus());
		document.addDate(Field.CREATE_DATE, carOwner.getCreateDate());
	    return document;
	}

I changed this line:
document.addGeoLocation("location", Double.parseDouble(latitudeProperty.getValue()), Double.parseDouble(longitudeProperty.getValue()));

So now the Indexed document JSON is:


{
    "_index":"liferay-20116",
    "_type":"LiferayDocumentType",
    "_id":"com.test.model.CarOwner_PORTLET_2",
    "_score":1,
    "_source":{
        "entryClassPK":"2",
        "scopeGroupId_sortable":20143,
        "entryClassName":"com.test.model.CarOwner",
        "groupId":"20143",
        "createDate_sortable":1508248053466,
        "stagingGroup":"false",
        "uid":"com.test.model.CarOwner_PORTLET_2",
        "scopeGroupId":"20143",
        "status_sortable":1,
        "groupId_sortable":20143,
        "modified":"20171108065033",
        "modified_sortable":1510123833893,
        "regionId":"53711",
        "location": "lat: 19.186354, lon: 73.191948",
        "status":"1",
        "createDate":"20171017134733"
    }
}

Now, I have "location": "lat: 19.186354, lon: 73.191948", instead of separate lat. lon value.

As you say, I need to use Geo* filters. I used filter as below:


public BooleanQuery generateRegionQuery(List<string> regionIdList, Set<long> carOwnerIds) {

	try {

		_log.debug("Startd generating query for region filter");

		BooleanQuery fullQuery = new BooleanQueryImpl();

		BooleanQuery entityQuery = generateBooleanQuery(Field.ENTRY_CLASS_NAME, CarOwner.class.getName(),
				QueryTermLevel.REQUIRED);
		BooleanQuery regionIdQuery = new BooleanQueryImpl();

		for (String regionId : regionIdList) {
			System.out.println("regionId ::"+regionId);
			BooleanQuery cityQuery = generateBooleanQuery(CarOwnerConstant.REGION_ID, regionId,
					QueryTermLevel.REQUIRED);
			regionIdQuery.add(cityQuery, BooleanClauseOccur.SHOULD);
		}

		GeoLocationPoint geoLocationPinPoint = new GeoLocationPoint(18.106659,83.395554);
		GeoDistance geoDistance = new GeoDistance(50, DistanceUnit.KILOMETERS);
		GeoDistanceFilter geoDistanceFilter = new GeoDistanceFilter("location", geoLocationPinPoint, geoDistance);
		System.out.print("Distance :"+geoDistanceFilter);
		BooleanQuery query  =  new BooleanQueryImpl();
		query.setPostFilter(geoDistanceFilter);
		
		BooleanQuery idsQuery = generateCarOwnerIdsQuery(carOwnerIds, Field.ENTRY_CLASS_PK);
		fullQuery.add(entityQuery, BooleanClauseOccur.MUST);
		fullQuery.add(regionIdQuery, BooleanClauseOccur.MUST);
		fullQuery.add(idsQuery, BooleanClauseOccur.MUST);
		fullQuery.add(query, BooleanClauseOccur.MUST);

		_log.debug("Query is generated for region filter");

		return fullQuery;
	} catch (Exception e) {
		_log.error("Error in generating query for region filter");
		_log.error(e.getMessage());
	}
	return null;
}
</long></string>

I have added below filter :
GeoLocationPoint geoLocationPinPoint = new GeoLocationPoint(18.106659,83.395554);
GeoDistance distance = new GeoDistance(50, DistanceUnit.KILOMETERS);
GeoDistanceFilter geoDistance = new GeoDistanceFilter("location", geoLocationPinPoint, distance);
System.out.print("Distance :"+geoDistance);
BooleanQuery query = new BooleanQueryImpl();
query.setPostFilter(geoDistance);

Still it can't works, emoticon

It display "Distance : (cached=null, executionOption=null)" in console.

Do you have sample code for the same?

Jorge Díaz, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

Liferay Master Posts: 753 Join Date: 1/9/14 Recent Posts

You can try activating Liferay debug traces from Liferay portal control panel

Open control panel => configuration => server administration => log levels => add category
Set com.liferay.portal.search category to DEBUG level

That configuration should dump to log the queries that are sent to search engine, check the "location" section

Jiten R. Vaghela, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

New Member Posts: 20 Join Date: 8/22/16 Recent Posts

Hello Jorge Díaz,

Thanks for your response.

I did the same as you described above.
I didn't get any DEBUG log on server console(Note: I started tomcat in DEBUG mode using ./catalina.sh jpda start).
Then I set com.liferay.portal.search.elasticsearch to DEBUG level. I got debug logs with boolean query and boolean query in JSON format.
Example:


05:54:23,436 INFO  [http-nio-8080-exec-9][ElasticsearchIndexSearcher:195] Searching {booleanClauses=[{MUST({booleanClauses=[{MUST({booleanClauses=[{MUST({booleanClauses=[{SHOULD({analyzer=null, className=MatchQuery, cutOffFrequency=null, field=entryClassName, fuzziness=null, fuzzyTranspositions=null, lenient=null, maxExpansions=null, minShouldMatch=null, operator=null, prefixLength=null, slop=null, type=null, value=com.test.model.CarOwner})}, {SHOULD({analyzer=null, className=MatchQuery, cutOffFrequency=null, field=entryClassName, fuzziness=null, fuzzyTranspositions=null, lenient=null, maxExpansions=null, minShouldMatch=null, operator=null, prefixLength=null, slop=null, type=PHRASE_PREFIX, value=com.test.model.CarOwner})}], className=BooleanQueryImpl})}, {SHOULD({analyzer=null, className=MatchQuery, cutOffFrequency=null, field=entryClassName, fuzziness=null, fuzzyTranspositions=null, lenient=null, maxExpansions=null, minShouldMatch=null, operator=null, prefixLength=null, slop=null, type=PHRASE, value=com.test.model.CarOwner})}], className=BooleanQueryImpl})}], className=BooleanQueryImpl})}, {MUST({booleanClauses=[{SHOULD({booleanClauses=[{MUST({booleanClauses=[{MUST({booleanClauses=[{SHOULD({analyzer=null, className=MatchQuery, cutOffFrequency=null, field=regionId, fuzziness=null, fuzzyTranspositions=null, lenient=null, maxExpansions=null, minShouldMatch=null, operator=null, prefixLength=null, slop=null, type=null, value=53717})}, {SHOULD({analyzer=null, className=MatchQuery, cutOffFrequency=null, field=regionId, fuzziness=null, fuzzyTranspositions=null, lenient=null, maxExpansions=null, minShouldMatch=null, operator=null, prefixLength=null, slop=null, type=PHRASE_PREFIX, value=53717})}], className=BooleanQueryImpl})}, {SHOULD({analyzer=null, className=MatchQuery, cutOffFrequency=null, field=regionId, fuzziness=null, fuzzyTranspositions=null, lenient=null, maxExpansions=null, minShouldMatch=null, operator=null, prefixLength=null, slop=null, type=PHRASE, value=53717})}], className=BooleanQueryImpl})}], className=BooleanQueryImpl})}], className=BooleanQueryImpl})}, {MUST({booleanClauses=[], className=BooleanQueryImpl})}], className=BooleanQueryImpl} took 4 ms

05:54:23,439 INFO  [http-nio-8080-exec-9][ElasticsearchIndexSearcher:501] The search engine processed {_  "bool" : {_    "must" : [ {_      "bool" : {_        "must" : {_          "bool" : {_            "must" : {_              "bool" : {_                "should" : [ {_                  "match" : {_                    "entryClassName" : {_                      "query" : "com.test.model.CarOwner",_                      "type" : "boolean"_                    }_                  }_                }, {_                  "match" : {_                    "entryClassName" : {_                      "query" : "com.test.model.CarOwner",_                      "type" : "phrase_prefix"_                    }_                  }_                } ]_              }_            },_            "should" : {_              "match" : {_                "entryClassName" : {_                  "query" : "com.test.model.CarOwner",_                  "type" : "phrase",_                  "boost" : 2.0_                }_              }_            }_          }_        }_      }_    }, {_      "bool" : {_        "should" : {_          "bool" : {_            "must" : {_              "bool" : {_                "must" : {_                  "bool" : {_                    "should" : [ {_                      "match" : {_                        "regionId" : {_                          "query" : "53717",_                          "type" : "boolean"_                        }_                      }_                    }, {_                      "match" : {_                        "regionId" : {_                          "query" : "53717",_                          "type" : "phrase_prefix"_                        }_                      }_                    } ]_                  }_                },_                "should" : {_                  "match" : {_                    "regionId" : {_                      "query" : "53717",_                      "type" : "phrase",_                      "boost" : 2.0_                    }_                  }_                }_              }_            }_          }_        }_      }_    }, {_      "bool" : { }_    } ]_  }_} in 2ms [Sanitized]

Query in JSON format:


{
  "bool" : {
    "must" : [ {
      "bool" : {
        "must" : {
          "bool" : {
            "must" : {
              "bool" : {
                "should" : [ {
                  "match" : {
                    "entryClassName" : {
                      "query" : "com.test.model.CarOwner",
                      "type" : "boolean"
                    }
                  }
                }, {
                  "match" : {
                    "entryClassName" : {
                      "query" : "com.test.model.CarOwner",
                      "type" : "phraseprefix"
                    }
                  }
                } ]
              }
            },
            "should" : {
              "match" : {
                "entryClassName" : {
                  "query" : "com.test.model.CarOwner",
                  "type" : "phrase",
                  "boost" : 2.0
                }
              }
            }
          }
        }
      }
    }, {
      "bool" : {
        "should" : {
          "bool" : {
            "should" : {
              "term" : {
                "entryClassPK" : "2"
              }
            }
          }
        }
      }
    }, {
      "bool" : {
        "should" : {
          "term" : {
            "status" : "1"
          }
        }
      }
    } ]
  }
}

I didn't find any filter in a query, Even I added GeoDistanceFilter.

Strange, emoticon

I can't understand what is happen with code.
Will you please take a look on Below code.


public BooleanQuery generateRegionQuery(List<string> regionIdList, Set<long> carOwnerIds) {

    try {

        _log.debug("Startd generating query for region filter");

        BooleanQuery fullQuery = new BooleanQueryImpl();

        BooleanQuery entityQuery = generateBooleanQuery(Field.ENTRY_CLASS_NAME, CarOwner.class.getName(),
                QueryTermLevel.REQUIRED);
        BooleanQuery regionIdQuery = new BooleanQueryImpl();

        for (String regionId : regionIdList) {
            System.out.println("regionId ::"+regionId);
            BooleanQuery cityQuery = generateBooleanQuery(CarOwnerConstant.REGION_ID, regionId,
                    QueryTermLevel.REQUIRED);
            regionIdQuery.add(cityQuery, BooleanClauseOccur.SHOULD);
        }

        GeoLocationPoint geoLocationPinPoint = new GeoLocationPoint(18.106659,83.395554);
        GeoDistance geoDistance = new GeoDistance(50, DistanceUnit.KILOMETERS);
        GeoDistanceFilter geoDistanceFilter = new GeoDistanceFilter("location", geoLocationPinPoint, geoDistance);
        System.out.print("Distance :"+geoDistanceFilter);
        BooleanQuery query  =  new BooleanQueryImpl();
        query.setPostFilter(geoDistanceFilter);
        
        BooleanQuery idsQuery = generateCarOwnerIdsQuery(carOwnerIds, Field.ENTRY_CLASS_PK);
        fullQuery.add(entityQuery, BooleanClauseOccur.MUST);
        fullQuery.add(regionIdQuery, BooleanClauseOccur.MUST);
        fullQuery.add(idsQuery, BooleanClauseOccur.MUST);
        fullQuery.add(query, BooleanClauseOccur.MUST);

        _log.debug("Query is generated for region filter");

        return fullQuery;
    } catch (Exception e) {
        _log.error("Error in generating query for region filter");
        _log.error(e.getMessage());
    }
    return null;
}
</long></string>

Do I missed anything?
If you don't mind, Can you please suggest me some code snippet for GeoDistanceFilter? if you have.
OR
Any example of GeoDistanceFilter in liferay?
Thanks.
G10

Jorge Díaz, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7 (Answer)

Liferay Master Posts: 753 Join Date: 1/9/14 Recent Posts

Hi Jiten,

I think your problem is both query.setPreBooleanFilter and query.setPostFilter must be applied to parent query, in case of having a query with subqueries, the filters applied in child queries are ignored.

So you have to create a filter and apply to the parent query that you are going to execute.

About setPreBooleanFilter vs setPostFilter, it is always better to apply (pre boolean) filters because the post filters have some performance issues in elasticsearch, see: https://www.elastic.co/guide/en/elasticsearch/guide/current/_post_filter.html

If your filter is not a BooleanFilter you can always create a BooleanFilter and add your filter inside it:

		BooleanFilter booleanFilter = new BooleanFilter();
		booleanFilter.add(geoDistanceFilter,BooleanClauseOccur.MUST);

Here the whole example using a filter, applied to parent query, try executing following code:

/* creating geo filter */
GeoLocationPoint geoLocationPinPoint = new GeoLocationPoint(18.106659,83.395554);
GeoDistance geoDistance = new GeoDistance(50, DistanceUnit.KILOMETERS);
GeoDistanceFilter geoDistanceFilter = new GeoDistanceFilter("location", geoLocationPinPoint, geoDistance);

/* creating boolean filter */
BooleanFilter booleanFilter = new BooleanFilter();
booleanFilter.add(geoDistanceFilter,BooleanClauseOccur.MUST);

/* main query */
BooleanQuery query = new BooleanQueryImpl();
query.addRequiredTerm(Field.ENTRY_CLASS_NAME, CarOwner.class.getName());
query.setPreBooleanFilter(booleanFilter);

/* execute query */
SearchContext searchContext = new SearchContext();
Hits hits = IndexSearcherHelperUtil.search(searchContext, query);

/* print output */
Document[] docs = hits.getDocs();
if (docs != null) { for (int j = 0; j < docs.length; j++) { System.out.println("" + j + " - "+docs[j]); } }

Jiten R. Vaghela, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

New Member Posts: 20 Join Date: 8/22/16 Recent Posts

Hi Jorge Díaz,

Thanks for your guidance, Now Query works fine with parent query.

Nice explanation, Good job bro. emoticon

As I discussed above. I need to get all the carOwner who are with in 50KM of area.

Actually, In this case we carOwner, Suppose I have multiple carOwner with different location.
For example:
1. Car Owner A : location {23.022505, 72.571362},
2. Car Owner B : location {21.17024, 72.831061},
3. Car Owner C : location {18.52043, 73.856744}
Now, I wanna search for all carOwners Who are within 50KM area of any of the above 3.
It means I need to add multiple GeoDistanceFilter, Right?

Is it possible to add multiple GeoDistanceFilter in parent query? emoticon

Jorge Díaz, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

Liferay Master Posts: 753 Join Date: 1/9/14 Recent Posts

I think it is possible, but I have never done it.

Try doing the following:
1) Create the geoDistance filters:

GeoDistanceFilter geoDistanceFilter1 = new GeoDistanceFilter("location", ....);
GeoDistanceFilter geoDistanceFilter2 = new GeoDistanceFilter("location", ....);
GeoDistanceFilter geoDistanceFilter3 = new GeoDistanceFilter("location", ....);

2) Create a boolean filter and add the three created GeoDistanceFilter. Here you have two options BooleanClauseOccur.SHOULD or BooleanClauseOccur.MUST.
- BooleanClauseOccur.MUST: the three geoDistanceFilter must be fulfill by results:

BooleanFilter booleanFilter = new BooleanFilter();
booleanFilter.add(geoDistanceFilter1,BooleanClauseOccur.MUST);
booleanFilter.add(geoDistanceFilter2,BooleanClauseOccur.MUST);
booleanFilter.add(geoDistanceFilter3,BooleanClauseOccur.MUST);

- BooleanClauseOccur.SHOULD: only one of the geoDistanceFilter has to be fulfill by results:

BooleanFilter booleanFilter = new BooleanFilter();
booleanFilter.add(geoDistanceFilter1,BooleanClauseOccur.SHOULD);
booleanFilter.add(geoDistanceFilter2,BooleanClauseOccur.SHOULD);
booleanFilter.add(geoDistanceFilter3,BooleanClauseOccur.SHOULD);

Jorge Díaz, modified 7 Years ago.

RE: Elastic search Distance search with liferay 7

Liferay Master Posts: 753 Join Date: 1/9/14 Recent Posts

Hi all,

An update about one of my comments:

Jorge Díaz:

Hi Jiten, I think your problem is both [b]query.setPreBooleanFilter[/b] and [b]query.setPostFilter[/b] must be applied to parent query, in case of having a query with subqueries, the filters applied in child queries are ignored.

It could be consider a issue/limitation of Liferay, see: LPS-85607

Community

Company

Feedback

Ask Questions and Find Answers

Important:

Ask is now read-only. You can review any existing questions and answers, but not add anything new.

But - don't panic! While ask is no more, we've replaced it with discuss - the new Liferay Discussion Forum! Read more here here or just visit the site here:

discuss.liferay.com

RE: Elastic search Distance search with liferay 7