SQL tokenizer
analyzes SQL statements and converts it into a list of tokens
Installation | Usage | License
Installation
With npm do
npm install sql-tokenizer
Usage
Create a tokenize function using CommonJS.
const tokenize =
You can also create a tokenize function with ES6 syntax.
const tokenize =
Turn SQL statement into tokens.
// ['select', ' ', '*', ' ', 'from', ' ', 'revenue']
Quotes are handled properly.
// ['select', ' ', "'O''Reilly'", ' ', 'as', ' ', '"book shelf"']
Indentation is preserved.
// '\n',// 'SELECT', ' ', 'COUNT', '(', '*', ')', ' ', 'AS', ' ', 'num', '\n',// 'FROM', ' ', '(', '\n',// '\t', 'SELECT', ' ', '*', '\n',// '\t', 'FROM', ' ', 'mytable', '\n',// '\t', 'WHERE', ' ', 'yyyymmdd', '=', '20170101', '\n',// '\t\t', 'AND', ' ', 'country', ' ', 'IN', ' ', '(', "'IT'", ',', "'US'", ')', '\n',// ')', '\n'
The tokenizer function accepts an optional array of operators, which defaults to SQL92-operators. The following example shows how to extend the SQL92 operators list with PostgreSQL bitwise operators.
const sql92Operators = const tokenizer = const operators = sql92Operators const tokenize =