ENH v4.2.0

Author: jacksonllee

.github/dependabot.yml

@@ -1,13 +1,5 @@
 version: 2
 updates:
-  - package-ecosystem: "cargo"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/"
-    schedule:
-      interval: "weekly"
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:

.gitignore

@@ -15,3 +15,6 @@ docs/generated/
 
 # Benchmarks
 benchmarks/_baseline/
+
+# From maturin develop
+src/pycantonese/lib_pycantonese.dylib.dSYM/

CHANGELOG.md

@@ -13,6 +13,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 ### Fixed
 ### Security
 
+## [4.2.0] - 2026-03-27
+
+### Added
+- Added the CantoMap corpus.
+- Added `from_git` and `from_url` to `CHAT` to support fetching datasets from remote sources.
+
+### Changed
+- Bumped rustling to >= 0.8.0.
+
 ## [4.1.0] - 2026-03-14
 
 ### Added

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "pycantonese"
-version = "4.1.0"
+version = "4.2.0"
 edition = "2024"
 
 [lib]
@@ -10,7 +10,7 @@ path = "src/rust/lib.rs"
 
 [dependencies]
 pyo3 = { version = "0.28", features = ["abi3-py310"] }
-rustling = { version = "0.7.0", default-features = false, features = ["pyo3"] }
+rustling = { version = "0.8.0", default-features = false, features = ["pyo3"] }
 regex = "1"
 
 [features]

Cargo.toml

@@ -51,13 +51,11 @@ Using conda::
 
    conda install -c conda-forge pycantonese
 
-For Pyodide, install the WASM wheels (the ``.whl`` files with ``emscripten`` in the filename)
-from the GitHub releases of
-`Rustling <https://github.com/jacksonllee/rustling/releases>`_ and
-`PyCantonese <https://github.com/jacksonllee/pycantonese/releases>`_.
+PyCantonese also works
+`in JavaScript <https://docs.pycantonese.org/stable/quickstart.html#javascript>`_.
 
 Ready for more?
-Check out the `Quickstart <https://pycantonese.org/quickstart.html>`_ page.
+Check out `Quickstart <https://docs.pycantonese.org/stable/quickstart.html>`_.
 
 Links
 -----
@@ -98,6 +96,7 @@ Please note that PyCantonese includes data from the following sources,
 all of which are permissively licensed:
 
 - Hong Kong Cantonese Corpus (CC BY)
+- CantoMap (GPL-3.0)
 - rime-cantonese (CC BY 4.0)
 - Common Voice Cantonese (Mozilla Public License 2.0)
 - Cantonese-Traditional Chinese Parallel Corpus (CC0 1.0 Universal)

README.rst

@@ -14,6 +14,7 @@ Corpus Data
 
     read_chat
     hkcancor
+    cantomap
     CHAT
     CHAT.search
 
@@ -58,6 +59,12 @@ Natural Language Processing
 .. autoclass:: pycantonese.corpus.Token
 
 
+:class:`~pycantonese.corpus.Utterance`
+--------------------------------------
+
+.. autoclass:: pycantonese.corpus.Utterance
+
+
 :class:`~pycantonese.jyutping.Jyutping`
 ---------------------------------------
 

docs/api.rst

@@ -91,5 +91,5 @@
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {
     "python": ('https://docs.python.org/3/', None),
-    "rustling": ("https://rustling.readthedocs.io/stable/", None),
+    "rustling": ("https://docs.rustling.io/stable/", None),
 }

docs/conf.py

@@ -17,17 +17,21 @@ For these reasons, PyCantonese adopts the `CHAT <https://talkbank.org/0info/manu
 corpus format from CHILDES and TalkBank.
 CHAT is widely used, well-documented, and rich for linguistic annotations.
 PyCantonese uses the library
-`Rustling <https://rustling.readthedocs.io/stable/chat.html>`_ to parse CHAT data files.
+`Rustling <https://docs.rustling.io/stable/chat.html>`_ to parse CHAT data files.
 For a primer on the CHAT data format, please see
-`here <https://docs.pylangacq.org/stable/transcriptions.html#chat-format>`_.
+`here <https://docs.rustling.io/stable/chat/transcriptions.html#chat-format>`_.
 
 
 Built-in Data
 -------------
 
-Currently, PyCantonese comes with one built-in corpus, the
-`Hong Kong Cantonese Corpus <https://github.com/fcbond/hkcancor>`_
-(HKCanCor; license: CC BY), via the function :func:`~pycantonese.hkcancor`:
+PyCantonese ships with two built-in corpora: HKCanCor and CantoMap.
+
+HKCanCor
+^^^^^^^^
+
+The `Hong Kong Cantonese Corpus <https://github.com/fcbond/hkcancor>`_
+(HKCanCor; license: CC BY) is available via the function :func:`~pycantonese.hkcancor`:
 
 .. code-block:: python
 
@@ -46,11 +50,39 @@ They have been converted to CHAT for incorporation into PyCantonese.
 On the format conversion, please consult this
 `readme <https://github.com/jacksonllee/pycantonese/blob/main/src/pycantonese/data/hkcancor/README.md>`_.
 
+CantoMap
+^^^^^^^^
+
+The `CantoMap <https://github.com/gwinterstein/CantoMap>`_ corpus
+(license: GPL-3.0) is a collection of contemporary Hong Kong Cantonese
+conversation recordings from MapTask exercises.
+It is available via the function :func:`~pycantonese.cantomap`:
+
+.. code-block:: python
+
+    import pycantonese
+    cantomap = pycantonese.cantomap()
+    cantomap.n_files  # number of data files
+    # 99
+    len(cantomap.words())  # number of words as segmented from all the utterances
+    # 118572
+
+CantoMap is word-segmented and annotated for Jyutping romanization.
+Part-of-speech tags (HKCanCor tagset) are added by PyCantonese's POS tagger
+during the conversion from ELAN to CHAT format.
+
+The original CantoMap source files are in the ELAN annotation format (``.eaf``).
+Because the CHAT format inherently requires word-segmented data,
+it is a natural fit for the word-segmented, Jyutping-annotated CantoMap data.
+The ELAN files have been converted to CHAT for incorporation into PyCantonese.
+On the format conversion, please consult this
+`readme <https://github.com/jacksonllee/pycantonese/blob/main/src/pycantonese/data/cantomap/README.md>`_.
+
 
 CHILDES and TalkBank Data
 -------------------------
 
-For corpora other than HKCanCor, PyCantonese provides the function :func:`~pycantonese.read_chat`
+For corpora beyond the built-in ones, PyCantonese provides the function :func:`~pycantonese.read_chat`
 to read in Cantonese data in the CHAT format.
 
 As of 2026, CHAT datasets are publicly available from `TalkBank <https://talkbank.org/>`_.
@@ -63,35 +95,43 @@ before you can download the full transcripts as a ZIP archive to your local driv
 
 Here are the Cantonese-related TalkBank datasets (in alphabetical order):
 
-* `Child Heritage Chinese Corpus <https://childes.talkbank.org/access/Biling/CHCC.html>`_
+* `Child Heritage Chinese Corpus <https://talkbank.org/childes/access/Biling/CHCC.html>`_
 
-* `Guthrie Bilingual Corpus <https://childes.talkbank.org/access/Biling/Guthrie.html>`_
+* `Guthrie Bilingual Corpus <https://talkbank.org/childes/access/Biling/Guthrie.html>`_
 
-* `HKU-70 Corpus <https://childes.talkbank.org/access/Chinese/Cantonese/HKU.html>`_
+* `HKU-70 Corpus <https://talkbank.org/childes/access/Chinese/Cantonese/HKU.html>`_
 
-* `Lee-Wong-Leung Corpus <https://childes.talkbank.org/access/Chinese/Cantonese/LeeWongLeung.html>`_
+* `Lee-Wong-Leung Corpus <https://talkbank.org/childes/access/Chinese/Cantonese/LeeWongLeung.html>`_
 
 * `Mandarin-Cantonese-English EACMC Corpus <https://talkbank.org/childes/access/Biling/EACMC.html>`_
 
-* `Yip-Matthews Bilingual Corpus <https://childes.talkbank.org/access/Biling/YipMatthews.html>`_
+* `Yip-Matthews Bilingual Corpus <https://talkbank.org/childes/access/Biling/YipMatthews.html>`_
 
 
 Custom Data
 -----------
 
-If you have your own CHAT data locally and would
-like PyCantonese to handle it, :func:`~pycantonese.read_chat`
-takes a path that can be a ZIP archive, a local directory, or a single CHAT file.
-
-If more fine-grained control is needed when reading data, please check out
-:class:`~pycantonese.CHAT`, particularly the following methods:
-
-* :py:meth:`~pycantonese.CHAT.from_zip`
-* :py:meth:`~pycantonese.CHAT.from_dir`
-* :py:meth:`~pycantonese.CHAT.from_files`
-* :py:meth:`~pycantonese.CHAT.from_strs`
-
-The CHAT parser comes from `Rustling <https://rustling.readthedocs.io/stable/chat.html>`_,
-which both PyCantonese and PyLangAcq use.
-For more on reading CHAT data in general, please see
-`PyLangAcq's documentation <https://docs.pylangacq.org/stable/read.html>`_.
+If you have your own CHAT data,
+:func:`~pycantonese.read_chat` accepts a local ZIP archive,
+a local directory, or a single ``.cha`` file path.
+
+For more control over how data is read, the :class:`~pycantonese.CHAT` class
+provides the following class methods:
+
+* :py:meth:`~pycantonese.CHAT.from_zip` -- local ZIP archive
+* :py:meth:`~pycantonese.CHAT.from_dir` -- local directory
+* :py:meth:`~pycantonese.CHAT.from_files` -- one or more local file paths
+* :py:meth:`~pycantonese.CHAT.from_strs` -- in-memory strings
+* :py:meth:`~pycantonese.CHAT.from_git` -- Git repository (cloned and cached)
+* :py:meth:`~pycantonese.CHAT.from_url` -- URL to a ZIP archive (downloaded and cached)
+
+The CHAT parser is powered by `Rustling <https://docs.rustling.io/stable/chat.html>`_,
+with Cantonese-specific additions for Jyutping romanization, Chinese characters,
+and a general corpus search function.
+
+If your data is in one of the
+`formats supported by Rustling <https://docs.rustling.io/stable/>`_,
+you can use Rustling to parse it, apply any processing you need,
+and create a :class:`~pycantonese.CHAT` object via :py:meth:`~pycantonese.CHAT.from_strs`.
+For an example of this workflow, see how the CantoMap ELAN data is
+`converted for use in PyCantonese <https://github.com/jacksonllee/pycantonese/blob/main/src/pycantonese/data/cantomap/README.md>`_.

docs/data.rst

@@ -42,7 +42,7 @@ Note:
   :func:`~pycantonese.CHAT.utterances`, :func:`~pycantonese.CHAT.search`, etc).
 * Since CHAT is designed for conversational data and your input data is a string,
   :func:`~pycantonese.parse_text` attempts simple utterance segmentation
-  (by the Chinese full-width punctuation marks ``{"，", "！", "。"}``
+  (by the Chinese full-width punctuation marks ``{"。", "！", "？"}``
   as well as the end-of-line character ``"\n"``).
 * By default, a dummy participant ``"X"`` is assigned to each utterance.
   To provide your own participant, pass it to the ``participant`` keyword argument of
@@ -143,20 +143,15 @@ This way, you can more easily share the processed data with your colleagues,
 reload the data (see :ref:`data`) for further processing and analysis in your workflow,
 and so forth.
 
-With a :class:`~pycantonese.CHAT` object, simply call the :func:`~pycantonese.CHAT.to_chat`
-method with a local file path.
+With a :class:`~pycantonese.CHAT` object, simply call the :func:`~pycantonese.CHAT.to_files`
+method with a local directory path.
 
 .. skip: start
 
 .. code-block:: python
 
-    file_path = "result.cha"
-    corpus.to_chat(file_path)
-
-    # If you're running code on Google Colab,
-    # you can download the file like this:
-    from google.colab import files
-    files.download(file_path)
+    dir_path = "output"
+    corpus.to_files(dir_path)
 
 .. skip: end