limon

The pluggable JavaScript lexer. Limon = Lemon. 🍋

Examples

This is not finished yet, but go to examples directory and look deeply there. 🎉
We have few initial examples:

• advanced - making the lexer to not be "on per character basis", by overwriting the .tokenize method using plugin.
• semver - tokenize semver string or alike
• simple - tokenize simple string (for example: a > (b + 2))
• csv - tokenize CSV string, and partially parsing
• more upcoming... going to port the PostCSS Tokenizer

And finally, after all, benchmarking.

Install

npm i limon --save

Usage

For more use-cases see the tests

const limon = require('limon')
const prevNext = require('limon-prev-next')
 
limon
  .use(prevNext())
  .use(function (app) {
    return function (ch, i, input) {
      // console.log('prev is:', this.prev())
      // console.log('next is:', this.next())
 
      if (/\s/.test(ch)) {
        this.tokens.push(['whitespace', ch, i])
        return
      }
      if (/\W/.test(ch)) {
        this.tokens.push(['symbol', ch, i])
        return
      }
      if (/\d/.test(ch)) {
        this.tokens.push(['digit', ch, i])
        return
      }
      this.tokens.push(['letter', ch, i])
    }
  })
 
var tokens = limon.tokenize('a > (b + 2)')
console.log(tokens)
// =>
// [ [ 'letter', 'a', 0 ],
//   [ 'whitespace', ' ', 1 ],
//   [ 'symbol', '>', 2 ],
//   [ 'whitespace', ' ', 3 ],
//   [ 'symbol', '(', 4 ],
//   [ 'letter', 'b', 5 ],
//   [ 'whitespace', ' ', 6 ],
//   [ 'symbol', '+', 7 ],
//   [ 'whitespace', ' ', 8 ],
//   [ 'digit', '2', 9 ],
//   [ 'symbol', ')', 10 ] ]

Limon

Initialize Limon with input and options. Both are completely optional. You can pass plugins and tokens to options.

Params

• input {String}: String value to tokenize, or if object it is assumed options.
• options {Object}: Optional options, use it to pass plugins or tokens.

Example

var Limon = require('limon').Limon
var lexer = new Limon('foo bar')
 
// or pass only options
var limon = new Limon({ foo: 'bar' })
var tokens = limon.tokenize('baz qux')

Related

• lazy-arrayify: We are lazy, also lazy-cached and browserify-ready - just arrayify, falsey values… more | homepage
• lazy-cache: Cache requires to be lazy-loaded when needed. | homepage
• lazy-utils: Most of the used mostly everywhere utils: extend-shallow, isobject, is-extendable (used in… more | homepage
• limon-prev-next: Plugin for limon pluggable lexer that adds prev and next methods. | homepage
• postjson: Transforming JSON with plugins. | homepage

Contributing

Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.
But before doing anything, please read the CONTRIBUTING.md guidelines.

Charlike Make Reagent