Search results
26 packages found
Rich text and markdown tokenization made easy.
Maintenance: 26%. Quality: 64%. Popularity: 2%.
React component to convert a text string into React.ReactNodes
Maintenance: 33%. Quality: 52%. Popularity: 0%.
A simple, Twitter-aware tokenizer.
- tokenise
- tokenize
- tokenising
- tokenizing
- tokeniser
- tokenizer
- token
- NLP
- language
- text
- strings
- stanford
- dlatk
Maintenance: 27%. Quality: 51%. Popularity: 3%.
The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.
- ESnext
- deep-clone
- package manager
- String.prototype.matchAll
- ie
- queue
- linewrap
- Microsoft
- trimRight
- ArrayBuffer#slice
- parser
- starter
- collection.es6
- commander
- View more
Maintenance: 25%. Quality: 51%. Popularity: 5%.
A CLI tool to concatenate all text files in your CWD with headers for GPT prompt engineering.
- text
- concatenate
- CLI
- Current Working Directory
- GPT
- tokens
- tokenizer
- ChatGPT
- prompt engineering
- token-count
- text manipulation
- file concatenation
- command line interface
- GPT-3
- View more
Maintenance: 28%. Quality: 52%. Popularity: 0%.
Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.
Maintenance: 8%. Quality: 60%. Popularity: 0%.
The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.
- shrinkwrap
- sigint
- symlink
- lockfile
- AsyncIterator
- take
- JSON
- Uint32Array
- core
- serializer
- accessor
- random
- Underscore
- jQuery
- View more
Maintenance: 25%. Quality: 51%. Popularity: 0%.
Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.
Maintenance: None. Quality: 60%. Popularity: 4%.
A tokenizer for Google-like search queries
Maintenance: None. Quality: 61%. Popularity: 2%.
Split text into tokens.
Maintenance: None. Quality: 62%. Popularity: 1%.
A tokenizer for Google-like search queries
Maintenance: None. Quality: 61%. Popularity: 0%.
A super fast html-parser stream that outputs tag, text and closing nodes.
Maintenance: None. Quality: 64%. Popularity: 1%.
Tokenize Chinese texts into words.
Maintenance: None. Quality: 53%. Popularity: 2%.
A super fast html-parser stream that outputs tag, text and closing nodes.
Maintenance: None. Quality: 60%. Popularity: 1%.
Library to tokenize text to paragraphs, sentences, subsentences and words
Maintenance: None. Quality: 53%. Popularity: 0%.
Markdown parser and lexer. A fork of marked.js maintained for Assemble.
Maintenance: None. Quality: 54%. Popularity: 3%.
jQuery plugin that allows a user to type keywords, which will be broken up into tokens/tags and displayed, similarly to tagging a post on Tumblr or Stack-Overflow.
Maintenance: None. Quality: 51%. Popularity: 1%.
Not a replacement for REGEX, but an alternative that is far more readable and a bit more flexible
Maintenance: None. Quality: 51%. Popularity: 0%.
A CJK text tokenizer
Maintenance: None. Quality: 42%. Popularity: 2%.
Tokenizer for Vietnamese in Nodejs and Javascript
Maintenance: None. Quality: 42%. Popularity: 3%.