❤
npm
Sign UpSign In

hx-tokenizer

1.0.0 • Public • Published

tokenizer Build Status npm version XO code style

Split text into tokens.

Install

$ npm install --save hx-tokenizer

Usage

const tokenizer = require("hx-tokenizer");
const tokens = tokenizer.tokenize("Lorem ipsum, something.");
// tokens == ["Lorem", "ipsum", ",", "something", "."]
const text = tokenizer.join(tokens);
// text == "Lorem ipsum, something."

API

tokenizer.tokenize(text)

tokenizer.join(tokens)

License

MIT © Hexacta

/hx-tokenizer/

    Package Sidebar

    Install

    npm i hx-tokenizer

    Repository

    github.com/hexacta/tokenizer

    Homepage

    github.com/hexacta/tokenizer#readme

    Weekly Downloads

    0

    Version

    1.0.0

    License

    MIT

    Last publish

    Collaborators

    • pomber
    Try on RunKit
    Report malware