@imnull/csv-reader

CSVReader

const { CSVReader } = require('./dist')

const reader = new CSVReader({
    head: [
        {
            name: 'id',
            parse: Number,
            index: 0,
        },
        'hash',
        'type',
        'prefix',
        {
            name: 'datetime',
            parse(s) {
                return new Date(s)
            }
        },
    ],
    /** Give then progress of scaning file */
    // callback: (progress) => {
    //     console.log(`percent ${Math.floor(progress * 1e6) / 1e4}%`)
    // },
    /** Filter the rows you want */
    // filter: (row) => {
    //     return row.datetime.getSeconds() === 31
    // },
})

reader.parse(file).then(res => {
    console.log(res)
})

parse

Conver then CSV data to a key-value object by the options config.

TReaderOptions.head

type TReaderOptions = {
  //...
  head: (string | { [k in keyof THead]?: THead[k] })[]
  //...
}

It's a array of string or a optional THead.

If string, it will be the name of the column at index of itself.

If THead, it has defined the column name, column index, and data parser.

type THead = {
    name: string;
    parse(raw: string): any;
    index: number;
}

matrix

It just convert the plain text to a string[][] format. For example, if you what to insert the data into some database table, a array of row values is enough.

const { CSVReader } = require('@imnull/csv-reader')

console.time('parse')
reader.matrix('some local file').then(rows => {
    console.log(rows)
})

So you got rows just like:

[ 'HeartRate', '1678076194000', '1678076194992', '0', '6', '0', '' ],
[ 'HeartRate', '1678076195000', '1678076195992', '0', '6', '0', '' ],
[ 'HeartRate', '1678076196000', '1678076196992', '0', '6', '0', '' ],
[ 'HeartRate', '1678076197000', '1678076197992', '0', '6', '0', '' ],
[ 'HeartRate', '1678076198000', '1678076198992', '0', '6', '0', '' ],
[ 'HeartRate', '1678076199000', '1678076199992', '0', '6', '0', '' ],
...

And then

rows.forEach(cols => {
  db.exec('INSERT INTO SOME_TABLE(c1,c2,c3,c4)VALUES(?,?,?,?)', cols)
})

Performance

Use CSVReader.matrix:

[demo.cvs * 100] parsed
  - 100,000 rows
  - 5 columns
  - 11,082,295 bytes

608.049ms

----------------

[demo.cvs * 1000] parsed
  - 1,000,000 rows
  - 5 columns
  - 111,822,896 bytes

6.009s

----------------

[demo2.cvs]
  - 86,280 rows
  - 7 columns
  - 3,955,613 bytes

560.668ms

----------------

[demo3.cvs]
  - 1,438 rows
  - 2 columns
  - 106,337,864 bytes

268.806ms

----------------

[demo3.cvs * n]
  - 23,008 rows
  - 2 columns
  - 1,701,405,824 bytes

3.119s

Version

1.1.0

Add methods CSVReader.matrix and CSVReader.matrixSync.

1.0.1

fix some bug