Custom hyphenation table

[dkager]

A question from the transcribers is if they can have a custom hyphenation table on top of the Dutch table from OpenOffice. This would be used to deal with corner cases not caught by the upstream table. The custom table would be a supplement to the OpenOffice one. Ideally the location of the custom table should be specified in an option for the conversion.
A side question was where the upstream table is located in the first place. In libhyphen?

@bertfrees Could you shed some light on this for our developer?

[bertfrees]

Some thoughts:

• Currently the hyphenator is selected as follows:

  • Try the Hyphen table
  • If for some reason this doesn't work the TeX table is chosen

  If you want to be able to extend the Hyphen table you must disable the fallback.

• I got the Hyphen table from the LibreOffice source code. The Hyphen table is based on the Tex table. Libhyphen is merely a software library, it doesn't include any actual tables.

• A question is where the custom hyphenation table would live. Some agencies have their hyphenation table inside their own module. Another possibility is to provide it as input to the job. Based on your explanation this sounds like the preferred option since transcribers want to be able to specify the location. It does mean however that the file must be uploaded before running a job.

• You have several option for "extending" the hyphenation table. This is what other agencies do:

  • Celia has a custom hyphenation table, manually written, but based on an existing table, in their module.
  • SBS has a custom hyphenation table in their module which is composed of an existing table and a supplement generated from a list of exception words (words that were previously hyphenated wrong).
  • NLB has a custom hyphenation table in their module that is composed of a base table generated from a comprehensive list of hyphenated words and a supplement with patterns for all non-standard hyphenations.

  Even more approaches are possible, but what is most appropriate really depends on the situation.

[dkager]

Thanks for the quick response, will CC this to our developer.
The SBS solution and the per-job table sound nice. We use the CLI, so uploading isn't a big problem. Maybe we could set a default table path if one isn't supplied with a job.

[bertfrees]

The SBS solution isn't ideal in the sense that it results in a long table (one pattern for each exception word) and adding one exception word fixes only that word (which you could also consider an advantage because you don't accidentally break something).

NLB's solution is the best one in my opinion, but in order to use this approach you must have a big corpus of correctly hyphenation words at your disposal.

[dkager]

NLB's solution is the best one in my opinion but in order to use this approach you must have a big corpus of correctly hyphenation words at your disposal.

That, plus the fact that the transcribers specifically want to have a supplement table to deal with exceptions. We'll have to study the workflow a bit more. But it's good to know that there are options.