@lordluceus/mistral-tokenizer
TypeScript icon, indicating that this package has built-in type declarations

1.2.2 • Public • Published

🌬️ mistral-tokenizer-ts 🌬️

This repository is a TypeScript fork of mistral-tokenizer-js, initially developed by imoneoi.

It is designed to maintain the core functionality of the tokenizer while leveraging the advantages of TypeScript for better type safety and developer experience.

Please refer to the original repository for more context and the foundational work behind the tokenizer.

Install

npm install mistral-tokenizer-ts

Usage

import { MistralTokenizer } from 'mistral-tokenizer-ts'

const tokenizer = new MistralTokenizer()

// Encode.
const tokens = tokenizer.encode('Hello world!')

// Decode.
const decoded = tokenizer.decode([1, 22557, 1526, 28808])

Tests

npm run test

Credit

Readme

Keywords

Package Sidebar

Install

npm i @lordluceus/mistral-tokenizer

Weekly Downloads

5

Version

1.2.2

License

MIT

Unpacked Size

3.88 MB

Total Files

16

Last publish

Collaborators

  • lordluceus