😄 HappyNodeTokenizer

A basic Twitter aware tokenizer for Javascript environments.

A Typescript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz.

Features

• Accurate port of both libraries (run npm run test)
• Typescript definitions
• Uses generators / memoize for efficiency
• Customizable and easy to use

Install

  npm install --save happynodetokenizer

Usage

HappyNodeTokenizer exports a function called tokenizer() which takes an optional configuration object (See "The Options Object" below).

Example

import { tokenizer } from 'happynodetokenizer';

const text = 'RT @ #happyfuncoding: this is a typical Twitter tweet :-)';

// these are the default options
const opts = {
  'mode': 'stanford',
  'normalize': undefined,
  'preserveCase': true,
};

// create a tokenizer instance with our options
const myTokenizer = tokenizer(opts);

// calling myTokenizer returns a generator function
const tokenGenerator = myTokenizer(text);

// you can turn the generator into an array of token objects like this:
const tokens = [...tokenGenerator()];

// you can also convert token objects to array of strings like this:
const values = Array.from(tokens, (token) => token.value);

Output

The tokens variable in the above example will look like this:

[
  { end: 1, start: 0, tag: 'word', value: 'rt' },
  { end: 3, start: 3, tag: 'punct', value: '@' },
  { end: 19, start: 5, tag: 'hashtag', value: '#happyfuncoding' },
  { end: 20, start: 20, tag: 'punct', value: ':' },
  { end: 25, start: 22, tag: 'word', value: 'this' },
  { end: 28, start: 27, tag: 'word', value: 'is' },
  { end: 30, start: 30, tag: 'word', value: 'a' },
  { end: 38, start: 32, tag: 'word', value: 'typical' },
  { end: 46, start: 40, tag: 'word', value: 'twitter' },
  { end: 52, start: 48, tag: 'word', value: 'tweet' },
  { end: 56, start: 54, tag: 'emoticon', value: ':-)' }
]

Where preserveCase in the Options Object is false, each result object may also contain a variation property which presents the token as originally matched if it differs from the value property. E.g.:

[
  { end: 1, start: 0, tag: 'word', value: 'rt', variation: 'RT' },
  { end: 3, start: 3, tag: 'punct', value: '@' },
  { end: 19, start: 5, tag: 'hashtag', value: '#happyfuncoding' },
  ...
  { end: 46, start: 40, tag: 'word', value: 'twitter', variation: 'Twitter' },
  ...
]

The Options Object

The options object and its properties are optional. The defaults are:

{
  'mode': 'stanford',
  'normalize': undefined,
  'preserveCase': true,
};

mode

string - valid options: stanford (default), or dlatk

stanford mode uses the original HappyFunTokenizer pattern. See Github.

dlatk mode uses the modified HappierFunTokenizing pattern. See Github.

normalize

string - valid options: "NFC" | "NFD" | "NFKC" | "NFKD" (default = undefined)

Normalize strings (e.g., when set, mañana becomes manana).

Normalization is disabled with set to null or undefined (default).

preserveCase

boolean - valid options: true, or false (default)

Preserves the case of the input string if true, otherwise all tokens are converted to lowercase. Does not affect emoticons.

Tags

HappyNodeTokenizer outputs an array of token objects. Each token object has three properties: idx, value and tag. The value is the token itself, the idx is the token's original index in the output, the tag is a descriptor based on one of the following depending on which opt.mode you are using:

Tag	Stanford	DLATK	Example
phone	✔️	✔️	[phone]
url	❌	✔️	http://www.youtube.com
url_scheme	❌	✔️	http://
url_authority	❌	✔️	[0-3]
url_path_query	❌	✔️	/index.html?s=search
htmltag	❌	✔️	<em class='grumpy'>
emoticon	✔️	✔️	>:(
username	✔️	✔️	@somefaketwitterhandle
hashtag	✔️	✔️	#tokenizing
punct	✔️	✔️	,
word	✔️	✔️	hello
<UNK>	✔️	✔️	(anything left unmatched)

Testing

To compare the results of HappyNodeTokenizer against HappyFunTokenizer and HappierFunTokenizing, run:

npm run test

The goal of this project is to provide an accurate port of HappyFunTokenizer and HappierFunTokenizing. Therefore, any pull requests with test failures will not be accepted.

Acknowledgements

Based on HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz.

Uses the "he" library by Mathias Bynens under the MIT license.

License

(C) 2017-24 P. Hughes. All rights reserved.

Shared under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license.