Search results
2 packages found
Sort by: Default
- Default
- Most downloaded this week
- Most downloaded this month
- Most dependents
- Recently published
This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.1
published version 0.6.2, 10 months ago0 dependents licensed under $ISC
10
Powered By [ArcoDesign](https://arco.design/)
published version 0.0.1, 6 months ago0 dependents licensed under $MIT
9