qs-iconv

Character encoding for qs (and by extension: request) and querystring.

Why?

Sending requests in Node.js usually encodes url using the utf-8 character code (both with querystring and qs). This means that sending a String like こんにちは！ will become %E3%81%93%E3%82%93%E3%81%AB%E3%81%A1%E3%81%AF%EF%BC%81. Unfortunately not all servers are implemented in utf8 (particularly in Japan ShiftJIS is still very often seen) and those servers might expect form requests to be sent using another encoding.

Example: If the servers speaks shift_jis then the above string should be sent as: %82%B1%82%F1%82%C9%82%BF%82%CD%81I.

(Note: Browsers also send form data to servers with shift_jis support as shift_jis encoded String)

How?

This library implements the encoder and decoder option for qs after installing it you can use it like:

(encoder)

var qsIconv = require('qs-iconv')
var qs = require('qs')
qs.stringify({test: 'こんにちは！'}, {encoder: qsIconv.encoder('shift_jis')})

(decoder)

var qsIconv = require('qs-iconv')
var qs = require('qs')
qs.parse('%82%B1%82%F1%82%C9%82%BF%82%CD%81I', {decoder: qsIconv.decoder('shift_jis')})

Querystring

Node.js comes with require('querystring') right out of the box. It has fewer features than qs but should work in quite a few cases as well. This library supports querystring as well! Here is how you would encode the above example using querystring.

var qsIconv = require('qs-iconv')
var querystring = require('querystring')
var tmpEscape = querystring.escape
querystring.escape = qsIconv.encoder('shift_jis')
querystring.stringify({test: 'こんにちは！'})
querystring.escape = tmpEscape

Of course you unescape works as well:

var qsIconv = require('qs-iconv')
var querystring = require('querystring')
var tmpUnescape = querystring.unescape
querystring.unescape = qsIconv.decoder('shift_jis')
querystring.parse('%82%B1%82%F1%82%C9%82%BF%82%CD%81I')
querystring.unescape = tmpUnescape

Request

Most likely you will not come in touch with iconv through qs or querystring but rather through request. Here is how you can make a shift_jis POST request using this library:

var qsIconv = require('qs-iconv')
var request = require('request')
// inter-locale.com explains encoding in depth
request('https://encoding-server-jsdxxvgqvo.now.sh/', {
    method: 'POST',
    data: {
        field1: 'こんにちは！'
    },
    qsStringifyOptions: {
        encoder: qsIconv.encoder('shift_jis')
    },
    qsParseOptions: {
        decoder: qsIconv.decoder('shift_jis')
    }
})

(Note: request supports this library from version 2.72.0)

Iconv

This package uses by default iconv-lite which is a JavaScript-only variant of iconv. But since its stripped down to the bone and might be missing some tricky edge cases this library also allows to take the iconv instead. For it to work you need to have iconv installed additionally to qs-iconv!

var qsIconv = require('qs-iconv')
qsIconv.decoder('shift_jis', true)
qsIconv.encoder('shift_jis', true)

Performance Remark

Both iconv and iconv-lite are not optimized for de-/encoding single charaters but rather a string. Unless we have good support for character encoding this library will add a reasonable performance toll.

License

ISC

(Fixes and other contributions welcome)