tokenizer
A simple tokenizer written in javascript for Node.JS
Example
var fs = ;var tokenizer = ; tokenizerdebug = true;tokenizer;tokenizer;tokenizer;var tokens = tokenizer; console;
where the source file is:
fn hello_world
puts 'hello world'
will output:
Starting tokenizer --
fn hello_world
puts 'hello world'
-- --
word token: fn
whitespace token:
word token: hello_world
newline token:
whitespace token:
word token: puts
whitespace token:
word token: 'hello
whitespace token:
word token: world'
newline token:
-- Tokenizing complete --
[ 'fn',
' ',
'hello_world',
'\n',
' ',
'puts',
' ',
'\'hello',
' ',
'world\'',
'\n' ]
-- --
Parsed 11 tokens