Search results
27 packages found
Rich text and markdown tokenization made easy.
React component to convert a text string into React.ReactNodes
The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.
- ESnext
- deep-clone
- package manager
- String.prototype.matchAll
- ie
- queue
- linewrap
- Microsoft
- trimRight
- ArrayBuffer#slice
- parser
- starter
- collection.es6
- commander
- View more
A simple, Twitter-aware tokenizer.
- tokenise
- tokenize
- tokenising
- tokenizing
- tokeniser
- tokenizer
- token
- NLP
- language
- text
- strings
- stanford
- dlatk
The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.
- ES
- regular-expression
- scheme
- es2016
- crypt
- ramda
- async
- ansi
- helpers
- Array
- circular
- package manager
- assert
- pretty
- View more
The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.
- shrinkwrap
- sigint
- symlink
- lockfile
- AsyncIterator
- take
- JSON
- Uint32Array
- core
- serializer
- accessor
- random
- Underscore
- jQuery
- View more
A CLI tool to concatenate all text files in your CWD with headers for GPT prompt engineering.
- text
- concatenate
- CLI
- Current Working Directory
- GPT
- tokens
- tokenizer
- ChatGPT
- prompt engineering
- token-count
- text manipulation
- file concatenation
- command line interface
- GPT-3
- View more
Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.
Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.
Split text into tokens.
A tokenizer for Google-like search queries
A tokenizer for Google-like search queries
A super fast html-parser stream that outputs tag, text and closing nodes.
Tokenize Chinese texts into words.
A super fast html-parser stream that outputs tag, text and closing nodes.
Library to tokenize text to paragraphs, sentences, subsentences and words
Markdown parser and lexer. A fork of marked.js maintained for Assemble.
jQuery plugin that allows a user to type keywords, which will be broken up into tokens/tags and displayed, similarly to tagging a post on Tumblr or Stack-Overflow.
Not a replacement for REGEX, but an alternative that is far more readable and a bit more flexible
Simple, but powerful lexical scanner that is a more minimal implementation of X-Scanner
- y-scanner
- yscanner
- x-scanner
- xscanner
- stringscanner
- scanner
- string
- text
- textscanner
- lex
- lexer
- lexical
- parse
- parser
- View more