v5.0.0

This release incremented the major version number to v5 for the API breaking changes
related to various conversion functions for Jyutping romanization.
Please see the notes below for details.

Added

• Added yale_to_jyutping to convert Yale romanization into Jyutping.
• characters_to_jyutping, jyutping_to_ipa, jyutping_to_yale,
  yale_to_jyutping, and jyutping_to_tipa now all accept str | list[str]
  as input. A str is treated as a single word with no word segmentation;
  a list[str] carries explicit word segmentation, one word per element.
• Added stringify_yale to take the output of jyutping_to_yale and return
  a string with apostrophes for disambiguating syllable boundaries.
• parse_jyutping now tolerates whitespace between syllables in its input
  string.

Changed

• Breaking: characters_to_jyutping, jyutping_to_ipa,
  jyutping_to_yale, yale_to_jyutping, jyutping_to_tipa, and g2p now
  use a single space to separate the per-syllable representations within
  each word in their output. For example,
  characters_to_jyutping('廣東話') returns [('廣東話', 'gwong2 dung1 waa2')]
  rather than the previous [('廣東話', 'gwong2dung1waa2')]. The underlying
  rime-cantonese data file (chars_to_jyutping.json) is regenerated with
  the same convention so that downstream consumers can recover syllables
  via str.split().
• Breaking: g2p now returns list[tuple[str, str | None]] (the IPA
  element is a single space-separated string), replacing the previous
  list[tuple[str, list[str] | None]] shape.

Removed

• Breaking: removed the return_as keyword argument from
  jyutping_to_ipa, jyutping_to_yale, and yale_to_jyutping.
  These functions now always return a list of strings.
• Breaking: removed the apostrophe-disambiguation behavior that
  jyutping_to_yale previously applied under return_as="string". The
  space between syllables in the new output resolves the same ambiguity.
  Please note:
  • The new stringify_yale takes the output of jyutping_to_yale and returns
    a string with apostrophes for disambiguating syllable boundaries.
  • In yale_to_jyutping input, apostrophes are still accepted but are now
    treated identically to whitespace as a syllable-boundary hint within a
    single word.

Fixed

• Fixed jyutping_to_yale mapping for the kw onset, which was incorrectly
  mapped to k.

v4.3.0

Added

• Added a g2p wrapper function for grapheme-to-phoneme conversion.

v4.2.0

Added

• Added the CantoMap corpus.
• Added from_git and from_url to CHAT to support fetching datasets from remote sources.

Changed

• Bumped rustling to >= 0.8.0.

v4.1.0

Added

• segment() for word segmentation now takes a boolean keyword argument offsets
  to optionally give the (start, end) indices of the segmented words.

Changed

• Bumped rustling to >= 0.7.0, for retrieving offsets in word segmentation.

Fixed

• segment() for word segmentation now splits a mixed Cantonese/English unit as separate
  words if it's not already a recognized mixed word.

v4.0.0

The major version bump from v3 to v4 is due to backward-incompatible yet minor
API changes triggered by the switch from the PyLangAcq + wordseg
dependencies to Rustling for handing CHAT data, word segmentation, and
part-of-speech tagging.

Added

• Added the new function jyutping_to_ipa for Jyutping-to-IPA conversion.
• The characters_to_jyutping function can now take a list of strings as input
  with user-provided word segmentation.
• Added support for Python 3.11, 3.12, 3.13, and 3.14.

Changed

• Switched to Rustling as the underlying engine
  for CHAT data parsing, word segmentation, and part-of-speech tagging;
  dropped PyLangAcq and wordseg as dependencies.
• CHATReader has been renamed CHAT and switched from PyLangAcq's legacy pure-Python
  parser to a Rust-based parser from Rustling, with various API changes
  for method names, arguments, etc.
• The word segmentation model has been updated to a semi-supervised hybrid approach
  that combines a DAG and hidden Markov model.
  The segment function for word segmentation no longer accepts an argument
  for a custom segmenter.
• For both word segmentation and part-of-speech tagging, the persisted models
  shipped with the package are now zstd-compressed FlatBuffers binaries.

Removed

• Dropped support for Python 3.7, 3.8, and 3.9.

Fixed

• Fixed word segmentation so that spaces between English words
  in the user input are now honored as word boundaries.
• If parse_text is given an empty input or None,
  now an empty CHAT instance is returned.
• If parse_text is given a non-empty list of utterances,
  then any empty utterance (e.g., None, "")
  will now be represented by an empty Utterance instance inside the resulting CHAT output.
• Fixed the HKCanCor-to-UD mapping for G1 mapped to VERB not V.

v3.4.0

[3.4.0] - 2021-12-28

Added

• Added the parse_text for analyzing Cantonese text data.
• Characters-to-Jyutping conversion:
  The characters_to_jyutping function now has the segmenter kwarg for
  customizing word segmentation.
• Added support for Python 3.10.
• Turned on Windows testing on CircleCI.
• Added pyproject.toml. Related to preferring setup.cfg for specifying
  build metadata and options.

Changed

• Characters-to-Jyutping conversion:
  For the characters_to_jyutping function,
  in case rime-cantonese and HKCanCor don't agree,
  rime-cantonese data (more accurate) is preferred.
• Updated the rime-cantonese data to the latest 2021.05.16 release,
  improving both characters-to-Jyutping conversion and word segmentation.
• Updated the PyLangAcq dependency to v0.16.0, allowing PyCantonese's CHATReader
  to use the new methods to_chat, to_strs, info, head, and tail.
• Switched to setup.cfg to fully specify build metadata and options,
  while keeping a minimal setup.py for backward compatibility.
  Related to the new pyproject.toml.

Removed

• Dropped support for Python 3.6.

Security

• Turned on safety and bandit checks at CircleCI builds.

v3.3.1

[3.3.1] - 2021-05-14

Fixed

• Allowed PyLangAcq v0.14.* for real.

v3.3.0

[3.3.0] - 2021-05-14

Changed

• Allow PyLangAcq v0.14.*, thereby adding the new features of the filter method to CHATReader
  and optional parallelization for CHAT data processing.

Fixed

• Fixed the search method of CHATReader when by_tokens is False.

v3.2.4

[3.2.4] - 2021-05-07

Fixed

• Fixed the previously inoperational methods append, append_left, extend, and extend_left
  of the class CHATReader through the upstream PyLangAcq package.
• Retrained the part-of-speech tagger, after the minor character fix from v3.2.3.
• Raised NotImplementedError for the method ipsyn of CHATReader,
  since the upstream method works only for English.

v3.2.3

[3.2.3] - 2021-04-12

Fixed

• Fixed character issues in the built-in HKCanCor data: 𥄫