LuceneSearchService - sort order of German 'Umlaute'

[chrisw14]

Hi,
is it possible in the implementation of imixs-workflow to tell lucene that it should order the german Umlaute correctly? Like 'Ü' should be 'Ue' and so on...

I think there are possibilities for lucene but I don't know how to implement it in imixs-workflow:
https://lucene.apache.org/core/4_0_0/analyzers-common/org/apache/lucene/analysis/de/GermanNormalizationFilter.html
https://lucene.apache.org/core/4_0_0/analyzers-common/org/apache/lucene/analysis/charfilter/MappingCharFilterFactory.html

Thank you!

[rsoika]

This is an interesting question which I can't answer for now.
In Imixs-Workflow we have this property

"lucence.analyzerClass" which defaults to the org.apache.lucene.analysis.standard.ClassicAnalyzer

You can overwrite this property with the imixs.properties file.

For my first understanding if you want to use the GermanNormalizationFilter you need to implement your own Analyzer. For example:

package my.project;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.de.GermanNormalizationFilter;
import org.apache.lucene.analysis.standard.StandardTokenizer;

public class MyCustomAnalyzer extends Analyzer {
	@Override
	protected TokenStreamComponents createComponents(String fieldName) {
		StandardTokenizer standardTokenizer = new StandardTokenizer();
		GermanNormalizationFilter germanNormalizeationFilter = new GermanNormalizationFilter(standardTokenizer);
		return new TokenStreamComponents(standardTokenizer, germanNormalizeationFilter);
	}
}

Than you can activate the custom analyzer in your project with a imixs.properties entry :

# add custom lucene analyzer
lucence.analyzerClass=my.project.MyCustomAnalyzer

But I am not sure if this is an workable solution. Can you check this?

[chrisw14]

I tried it out but can't found the mentioned imported classes:
org.apache.lucene.analysis.Analyzer;
org.apache.lucene.analysis.de.GermanNormalizationFilter;
org.apache.lucene.analysis.standard.StandardTokenizer;

In my pom I have:

	<dependency>
		<groupId>org.apache.lucene</groupId>
		<artifactId>lucene-analyzers-common</artifactId>
		<version>6.3.0</version>
	</dependency>

Where can I found these classes?

[rsoika]

The lucene dependencies needed are:

			<dependency>
				<groupId>org.apache.lucene</groupId>
				<artifactId>lucene-core</artifactId>
				<version>${lucene.version}</version>
			</dependency>
			<dependency>
				<groupId>org.apache.lucene</groupId>
				<artifactId>lucene-analyzers-common</artifactId>
				<version>${lucene.version}</version>
			</dependency>
			<dependency>
				<groupId>org.apache.lucene</groupId>
				<artifactId>lucene-queryparser</artifactId>
				<version>${lucene.version}</version>
			</dependency>

[chrisw14]

Yes I already had this dependencies in the pom.xml but the packages mentioned above cannot be found.
It looks like this:

[rsoika]

look if you can build your project with mvn install. It looks like you have project setup problem in eclipse

[chrisw14]

Mvn install has built the project with success but in the class the imports cannot be resolved.
Several lucene packages are included in the java build path:

[rsoika]

So it's a eclipse problem. You can try the project command "maven -> update project".
If this does not help please ask in the eclipse community. Let's concentrate here on the lucene question.

[chrisw14]

Okay I tried it with the compile errors and added the line to the imixs.properties file but the sort order of the german "Umlaute" hasn't changed. There were no other errors.

Now I added a line in the class with an output in the console but it never appears, so I think the class has never been called.

[rsoika]

Yes you are right. This is indeed a bug of the LuceneUpdateService. I opened a new issue #429

[rsoika]

I fixed this now in release 4.4.1-SNAPSHOT. Please try if this works for you.

[chrisw14]

Yes, the class will be called in 4.4.1-SNAPSHOT.
Then I really have this compiler problem:

java.lang.Error: Unresolved compilation problems: 
	The import org.apache.lucene.analysis.Analyzer cannot be resolved
	The import org.apache.lucene.analysis.de.GermanNormalizationFilter cannot be resolved
	The import org.apache.lucene.analysis.standard.StandardTokenizer cannot be resolved
	Analyzer cannot be resolved to a type
	TokenStreamComponents cannot be resolved to a type
	The method createComponents(String) of type LuceneGerman must override or implement a supertype method
	StandardTokenizer cannot be resolved to a type
	StandardTokenizer cannot be resolved to a type
	GermanNormalizationFilter cannot be resolved to a type
	GermanNormalizationFilter cannot be resolved to a type
	TokenStreamComponents cannot be resolved to a type

Okay, now I added the jars manually to the build path.
The class is called but the sort order is the same as before!?

[rsoika]

concerning the compilation problems: you need to check your project and IDE setup I think.

concerning the sort order problem: how is the call of the find method looking now? How looks your imixs.properties 'lucence.indexFieldListNoAnalyze'. This is a important setting for sorting. Is your sorting field listed there?

[chrisw14]

Okay, I will set it up again.

The call of the find method:
results = workflowService.getDocumentService().find(pageQuery, -1, 0, "title", false);
And of course I added the "title" to the lucence.indexFieldListNoAnalyze and reindexed the lucene index over the imixs-admin panel.