Add Xapian Omega solution to haystack backend to fix long term issues

[doctormo]

Inside the xapian project, they have solved the "Term Too Long" error by providing two different options inside their omega side project, one is to truncate the terms and the other is to hash the terms.

See: https://lists.xapian.org/pipermail/xapian-discuss/2007-March/003450.html for example.

This commit adds this capability to this haystack backend. It's been tested with ascii, urls, japonese and strings of emoji on python 3.6, django 1.11.15 in both management command and real time updating.

[alexsilva]

Xapian developers recommend a different method:

https://trac.xapian.org/wiki/FAQ/UniqueIds#Workingroundthetermlengthlimit

Based on the link above I did something different:

Fix long term

[doctormo]

@alexsilva Isn't that documentation specific to unique IDs. But in either case, you could add SPLIT as one of the methods for the above code instead of having it be specific cast as one item.

[pcolmer]

Can one of these proposed solutions please be incorporated into the xapian-haystack package? I've had to hack a patch into our copy in order to allow Mailman 3 to successfully index emails containing long URLs but it would be much better/cleaner if xapian-haystack incorporated a solution permanently.

[claudep]

I guess we'll have to wait for a rebased patch including tests.

[odhiambo]

Can one of these proposed solutions please be incorporated into the xapian-haystack package? I've had to hack a patch into our copy in order to allow Mailman 3 to successfully index emails containing long URLs but it would be much better/cleaner if xapian-haystack incorporated a solution permanently.

@pcolmer what exactly did you do to enable Mailman 3 to successfully index the emails?
I am stuck at the same point.
TIA

[anarcat]

Xapian developers recommend a different method:

https://trac.xapian.org/wiki/FAQ/UniqueIds#Workingroundthetermlengthlimit

Based on the link above I did something different:

Fix long term

wait, what's going on here? that commit looks nice and merged in that repo, which even has a 3.2.0 tag, but it was never shipped here.

is there really a 3.2.0 xapian-haystack release out there? according to pypi, we're still at 3.2.0, so i'm really wondering what's going on with that tag...

update: digging in that repo's history, it seems like this patch wasn't kept and instead something like this PR was kept.

[anarcat]

i've applied this patch in production here (with @msapiro's tweak) to workaround "long term" bugs, and so far it seems to work better! at least, before, the indexer was crashing after 3-4 minutes, and now it's been going for 7 minutes without crash, which is a pretty good sign.

i'll report back here once the indexing has completed, but that could take hours.

if this works well, i plan to ship this patch in the debian package of this software to get wider testing in debian unstable, and, again, report back here. i hope this can be merged!

[doctormo]

@anarcat Please feel free to rebase current master with these changes and I'll close this PR for now.

[anarcat]

@anarcat Please feel free to rebase current master with these changes and I'll close this PR for now.

done in #238

btw so far my tests work okay. i ran out of disk space during our first indexing (xapian seems to take more space than whoosh?), which corrupted the index, but i'm reindexing now and i'm 1/3 of the way through, without issues.