Bundle of more than 30 new text objects for Neovim.
-
Updated
Apr 13, 2026 - Lua
#
subword
Here are 19 public repositories matching this topic...
Korean text normalization and language preparation package for LM in Kaldi-based ASR system
-
Updated
Apr 23, 2020 - Python
Simple-to-use scoring function for arbitrarily tokenized texts.
-
Updated
Feb 19, 2025 - Python
Keyword Search Recipe for Subword ASR
-
Updated
Jul 12, 2019 - Shell
Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)
-
Updated
Nov 6, 2018 - Python
johnny - a neural network graph based DEPendency Parser
-
Updated
Mar 25, 2021 - Python
Effective Subword Segmentation for Text Comprehension (TASLP 2019)
-
Updated
Nov 6, 2018 - C++
Unsupervised Word Segmentation using Minimum Description Length for Neural Machine Translation (NMT)
-
Updated
Dec 21, 2018 - C++
A causal intervention framework to learn robust and interpretable character representations inside subword-based language models
-
Updated
Jul 10, 2023 - Jupyter Notebook
An implementation of subword division algorithm proposed in T. Mikolov (2012).
-
Updated
Sep 25, 2019 - HTML
This repository contains source code implementation of assignments for NTU's MSAI course AI6127 on Deep Neural Networks for Natural Language Processing (2019 Sem 2).
-
Updated
Dec 11, 2020 - Jupyter Notebook
A framework for generating subword vocabulary from a tensorflow dataset and building custom BERT tokenizer models.
machine-learning deep-learning tensorflow machine-translation vocabulary-builder bert subword wordpiece berttokenizer tensorflow-text
-
Updated
Jul 6, 2021 - Python
-
Updated
Jan 8, 2023 - Jupyter Notebook
🧠 Build and explore a minimal Byte Pair Encoding tokenizer in TypeScript, training and encoding text using raw UTF-8 bytes without external libraries.
nlp open-source education encoding machine-learning typescript ai compiler decoding developer-tools text-processing language-models utf8 subword bun bpe byte-pair-encoding llm
-
Updated
Jun 9, 2026 - TypeScript
Com la tokenització fractura la morfologia catalana i si una segmentació conscient dels morfemes recupera la geometria. Provat en 3 llengües indoeuropees (català, castellà, anglès): el català es fragmenta ~1,7× més que l'anglès; forçar el tall morfèmic recupera la composicionalitat (robust a portadora i replicat en castellà).
nlp morphology catalan gemma tokenization subword salamandra llm qwen low-resource-language representation-geometry open-weight-models
-
Updated
May 28, 2026 - Python
Tokenization is a way of separating a piece of text into smaller units called tokens. Here, tokens can be either words, characters, or subwords. Hence, tokenization can be broadly classified into 3 types – word, character, and subword (n-gram characters) tokenization.
cat nlp count tensorflow tokenizer natural-language character sentence keras-classification-models subword nerual-network imdb-dataset deep-learning-architectures rnn-keras smaller-units tokenizer-nlp
-
Updated
Jun 30, 2021 - Jupyter Notebook
Subword Neural Machine Translation
-
Updated
Jun 20, 2017 - Python
From-scratch Byte Pair Encoding (BPE) tokenizer in TypeScript using Bun
nlp open-source education encoding machine-learning typescript ai compiler algorithms tokenizer decoding developer-tools text-processing language-models utf8 subword bun bpe byte-pair-encoding llm
-
Updated
Feb 11, 2026 - TypeScript
The concept of DAWGs is based on: Blumer, A. et al. (1985). The smallest automation recognizing the subwords of a text. Theoretical Computer Science, 40, 31–55.
nlp tree parsing tree-structure theoretical-computer-science dawg subword subword-segmentation subwords
-
Updated
Sep 13, 2022 - Java
Improve this page
Add a description, image, and links to the subword topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the subword topic, visit your repo's landing page and select "manage topics."