huba-gramophone

Gramophone

Extracts most frequently used keywords and phrases from text. It excludes common stop words. It can be configured to extract arbitary length phrases (ngrams) rather than just keywords.

// Get the top two most commonly used phrases of two more words in the Stream Handbook
request('https://github.com/substack/stream-handbook/blob/master/readme.markdown')
  .pipe(gramophone.stream({ngrams: 2, html: true, limit: 2}))
  .on('data', console.error.bind(console));

Would write out:

readable stream
writable stream

API

• gramophone.extract()
• gramophone.stream()
• gramophone.transformStream()

---

gramophone.extract(text[, options])

Synchronously extracts keywords from the text. By default it returns any keyword phrases that occur more than once. It also removes any common English words. It returns the results reverse ordered by frequency i.e. the first result is the most common phrase.

keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar')

Returns ['beep beep', 'foo bar'].

Option: score

Returns each keyword as an object where term is the keyword and tf is the number of times the phrase was used i.e. the term frequency. Off by default.

keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar', {score: true})

Returns [{term: 'beep beep', tf: 3}, {term: 'foo bar', tf: 2}].

Option: limit

Returns the top N results. The default is to not limit the results.

keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar', {limit: 1})

Returns ['beep beep'].

Option: flatten

Returns all occurrences of the ngram. Useful for passing data to Natural's TF-IDF function. Note: the original order is not maintained. Off by default.

keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar', {flatten: true})

Returns ['beep beep', 'beep beep', 'beep beep', 'foo bar', 'foo bar'].

Option: html

Extracts the keywords from html text elements. The default is false.

keyword.extract('<strong>beep</strong>, <strong>beep</strong> and <strong>foo</strong>', {html: true})

Returns ['beep', 'foo'].

Option: min

Only returns results with greater than or equal to N occurences. The default value is 2.

keyword.extract('beep and beep and beep and foo and foo', {min: 3})

Returns ['beep'].

Option: ngrams

If ngrams is a number (N), only look for phrases with N words. If ngrams is a list ([N1, N2]), only look for the phrases with N1 or N2 words etc.. The defualt is too look for [1, 2, 3] word ngrams.

keyword.extract('beep and beep and beep bop boop and foo and foo bar', {ngrams: [2, 3]})

Returns ['beep bop boop', 'foo bar'].

Option: stopWords

Add extra stopWords to be used in addition to the English set.

keyword.extract('foo et bar et foo et bar et foo', {stopWords: ['et']})

Returns ['foo', 'bar'].

Option: startWords

Any words in this list are whitelisted even if they are a stop word.

keyword.extract('foo and bar with foo and bar', {startWords: ['and']})

Returns ['foo and bar']

Option: stem

Apply stemming before extracting keywords. The returned keyword will be the most frequently used word.

keyword.extract('fooing and foo and fooing', {stem: true})

Returns ['fooing']

Option: cutoff

Allows you to specify the cutoff for determining whether to include a phrase that is a component of another phrase. E.g. should "node" and "runs" be extracted as keywords as well as "node runs".

A component phrase is filtered based on the following formula:

phrase freq. / component phrase freq. >= 1 - cutoff

E.g., let's say you have some text that includes the phrase "node runs" 20 times, "node" 40 times and "runs" 22 times. If the cutoff was 0.5 (the default), "node" would be included as 20 / 40 >= 1 - 0.5. However, "runs" would not be returned as a keyword as 20 / 22 < 1 - 0.5.

Wow. I could probably make this more intuitive. Open to suggestions.

Option: alternativeTokenizer

Gramophone uses Natural to tokenize and extract ngrams. By default, natural uses its WordTokenizer which splits words using /\W+/ as a separator regular expression.

This causes words like "London's" to be split into two tokens rather than one. It also breaks up accented words, using the accented character as the separator.

Setting alternativeTokenizer to true uses a more generous regular expression that respects these characters.

keyword.extract('Lörem Ipsüm Lörem Ipsüm.', {alternativeTokenizer: true})

---

gramophone.stream([options])

Returns a through stream that reads in the text stream and emits keywords based on the options passed. It uses the same options as extract. Note: this stream behaves like a sink and will buffer the stream completely before emitting keywords.

See first example.

---

gramophone.transformStream([options])

Returns a through stream that reads in the stream and emits keywords for each data read. By default, it assumes that each data read in a string. Alternatively the stream can read and write to objects. To read the text from an object property, specify the from option. If you want to write the keywords back to the object, also specify the to option.

var stream = gramophone.transformStream({from: 'text', to: 'keywords'});
stream.write({ text: 'foo and bar and foo'});
stream.end();

Emits the data: { text: 'foo and bar and foo', keywords: [foo] }.

Related projects

• node-alchemy: a cloud based keyword extraction service.
• natural: a fantastic natural language processing library for node.js. Checkout Tf-Idf if you're looking to extract keywords based on their relative frequency to other documents. If people are interested, I might add Tf-Idf support to gramophone.

Licence & copyright

gramophone is Copyright (c) 2012 B.J. Rossiter.

gramophone is licensed under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.