Possible speed-up for droid-detect

[tokee]

In DroidDetector.getMimeTypeFromResults, the first result from droid detection is used for the MIME-type and the rest of the droid results are discarded, except that the total number of results are used to determine whether or nor to muck about with the version number.

The problem here is that droid tries all signatures on the input, which is not ideal when only 1 result is really used. It seems simple to add an exit-on-first-hit to the droid code in (InternalSignatureCollection.getMatchingSignatures), which, assuming random order of signatures and matches, would cut processing time in half. With a bit of prioritization of the signatures where the most common MIME types are checked first, there should be a significant speed-gain.

The price is that it will not be possible to determine whether or not there were multiple signature matches. This means that the version number will either be (more) un-reliable or always missing. This might call for making it the optimization optional: "Slow parsing with version" or "Fast parsing without version".

[tokee]

I have opened digital-preservation/droid#185 - if that is accepted, I hope it is simple to update nanite to (optionally) take advantage of this.

If accepted, I also hope to add re-ordering of signatures to droid, to get a drastic speed-up instead of "just" a doubling of speed. I have not done a feasibility study of that part.

[anjackson]

Hi @tokee note that DROID/PRONOM rely on HasPriorityOverFileFormatID relationships between signatures to ensure correct matches (rather than using a global priority value as Tika does). To ensure correct results, the first match also needs to be the highest priority match.

Also, in all cases, all binary signatures for recognised container formats need to be run so that the second-level of identification (based on scanning container contents) can function.

I'm wary of mucking around with DROIDs logic too much here. If you don't need format versions, it's better to simply switch off DROID processing in the indexer all together.

EDIT, e.g. change this to false: https://github.com/ukwa/webarchive-discovery/blob/master/configs/jisc6.conf#L95-L96