Node.js Utils
This module shares helpers among all commercetools Node.js components.
Table of Contents
Getting Started
SphereUtils = require 'sphere-node-utils'Logger = SphereUtilsLoggerTaskQueue = SphereUtilsTaskQueue... # or require 'sphere-node-utils'
Documentation
Helpers
Currently following helpers are provided by SphereUtils
:
Logger
ExtendedLogger
TaskQueue
Sftp
ProjectCredentialsConfig
ElasticIo
userAgent
Repeater
Logger
Logging is supported by the lightweight JSON logging module called Bunyan.
The Logger
can be configured with following options
logConfig: levelStream: 'info' # log level for stdout stream levelFile: 'debug' # log level for file stream path: './sphere-node-utils-debug.log' # where to write the file stream name: 'sphere-node-utils' # name of the application serializers: request: reqSerializer # function that maps the request object with fields (uri, method, headers) response: resSerializer # function that maps the response object with fields (status, headers, body) src: false # includes a log of the call source location (file, line, function). # Determining the source call is slow, therefor it's recommended not to enable this on production. silent: false # don't instantiate the {Bunyan} logger, instead use `console` streams: # a list of streams that defines the type of output for log messages level: 'info'stream: processstdout level: 'debug'path: './sphere-node-utils-debug.log'
A
Logger
instance should be extended by the component that needs logging by providing the correct configuration
require 'sphere-node-utils' module.exports = # we can override here some of the configuration options @appName: 'my-application-name' @path: './my-application-name.log'
A Bunyan
logger can also be created from another existing logger. This is useful to connect sub-components of the same application by sharing the same configuration.
This concept is called child logger.
require 'sphere-node-utils' @appName: 'my-application-name' myLogger = logConfig # assume we have a component which already implements logging appWithLogger = logConfig: logger: myLogger # now we can use `myLogger` to log and everything logged from the child logger of `AppWithLogger` # will be logged with a `widget_type` field, meaning the log comes from the child logger
Once you configure your logger, you will get JSON stream of logs based on the level you defined. This is great for processing, but not for really human-friendly.
This is where the bunyan
command-line tool comes in handy, by providing pretty-printed logs and filtering. More info here.
# examples # this will output the content of the log file in a `short` format bunyan sphere-node-connect-debug.log -o short00:31:47.760Z INFO sphere-node-connect: Retrieving access_token...00:31:48.232Z INFO sphere-node-connect: GET /products # directly pipe the stdout stream jasmine-node --verbose --captureExceptions test | ./node_modules/bunyan/bin/bunyan -o short00:34:03.936Z DEBUG sphere-node-connect: OAuth constructor initialized. config: 00:34:03.933Z DEBUG sphere-node-connect: Failed to retrieve access_token, retrying...1
console
Silent logs, use You can pass a silent
flag to override the level functions of the bunyan
logger (debug, info, ...) to print to stdout / stderr using console.
ExtendedLogger
An ExtendedLogger
allows you to wrap additional fields to the logged JSON object, by either defining them on class instantiation or by chaining them before calling the log level method.
Under the hood it uses the Logger
Bunyan
object
logger = additionalFields: project_key: 'foo' another_field: 'bar' logConfig: # see config above (Logger) streams: level: 'info'stream: processstdout # then use the logger as usual loggerinfo id: 123'Hello world'# => {"name":"sphere-node-utils","hostname":"Nicolas-MacBook-Pro.local","pid":25856,"level":30,"id":123,"project_key":"foo","another_field":"bar","msg":"Hello world","time":"2014-04-17T10:54:05.237Z","v":0} # or by chaining loggerwithFieldtoken: 'qwerty'info id: 123'Hello world'# => {"name":"sphere-node-utils","hostname":"Nicolas-MacBook-Pro.local","pid":25856,"level":30,"id":123,"project_key":"foo","another_field":"bar", "token": "qwerty","msg":"Hello world","time":"2014-04-17T10:54:05.237Z","v":0}
TaskQueue
A TaskQueue
allows you to queue promises (or function that return promises) which will be executed in parallel sequentially, meaning that new tasks will not be triggered until the previous ones are resolved.
require 'sphere-node-utils' = -> setTimeout -> resolve true 5000task = maxParallel: 50 # default 20 taskaddTask callMethen # result == true catch
# adding and executing multiple tasks callMename = -> setTimeout -> resolve name 5000taskQueue = task1Promise = taskQueueaddTask callMe'task1'task2Promise = taskQueueaddTask callMe'task2' Promisealltask1Promisetask2Promisethen # result == ['task1', 'task2'] catch
Available methods:
setMaxParallel
sets themaxParallel
parameter (default is20
). If < 1 or > 100 it throws an erroraddTask
adds a task (promise) to the queue and returns a new promise
Sftp
Provides promised based wrapped functionalities for some SFTP
methods
listFiles
stats
readFile
not implemented yetsaveFile
not implemented yetgetFile
putFile
safePutFile
renameFile
safeRenameFile
removeFile
openSftp
close
downloadAllFiles
The client using the
Sftp
helper should take care of how to send requests to manage remote files, by controlling e.g. concurrency viaTaskQueue
and/or functions ofBluebird
promise API
ProjectCredentialsConfig
Allows to read SPHERE.IO credentials from a file or via environment variables, based on the project_key
.
From a file
By default the module will try to read the credentials from the following locations (descending priority):
- ./.sphere-project-credentials
- ./.sphere-project-credentials.json
- ~/.sphere-project-credentials
- ~/.sphere-project-credentials.json
- /etc/sphere-project-credentials
- /etc/sphere-project-credentials.json
The versions of these without the .json
extension consist of a series of lines, each of which contains a project key, client ID and client secret, separated by colons:
<project-key1>:<client-id1>:<client-secret1>
<project-key2>:<client-id2>:<client-secret2>
The JSON versions are structured as follows:
{
"<project-key1>": {
"client_id":"<client-id1>",
"client_secret":"<client-secret1>"
},
"<project-key2>": {
"client_id":"<client-id2>",
"client_secret":"<client-secret2>"
}
}
Example usage:
const PROJECT_KEY = 'your-project-key' ProjectCredentialsConfig
From environment variables
This is a little bit more restricted, since you can only define one set of credentials with the environment variables. Nevertheless this is very useful for deployments, where you really only need one set of credentials per deployment. You can define your credentials using these variables:
export SPHERE_PROJECT_KEY="your-project-key"export SPHERE_CLIENT_ID="your-client-id"export SPHERE_CLIENT_SECRET="your-client-secret"
ElasticIo
(Coming soon)
UserAgent
A synchronous module that builds user_agent according to the standard specified by commercetools
sphere-node-sdk module must be installed in the node_modules because it's required to build the user_agent
const user_agent =
Example of returned user_agent
1.16.2 Node.js/v6.5.0 (darwin; x64) sphere-node-utils/0.8.6
Repeater
A Repeater allows to execute a promise function and recover from it in case of errors, for a certain number of times before giving up.
By default the initial task will be retried unless a new task is returned from the recover function (see example below).
The only method exposed is execute
, which accepts 2 arguments:
- task: a
Function
that returns a Promise - recover: a
Function
that returns a Promise, called when the task fails
Following options are supported when initializing a new Repeater
:
attempts
(Int) how many times the task should be repeated, if failed, before giving up (default 10)timeout
(Int) the delay between attempts before retrying, inms
(default 100)timeoutType
(String) the type of the timeout (default c)c
constant delay always the same timeoutv
variable delay timeout grows with the attempts count (also using a random component)
client = ...repeater = attempts: 10 = clientproductsbyIdproductIdupdatepayloadrepeaterexecute -> updateTaskpayload if estatusCode is 409 # this means a concurrent modification, so we retry to update with # a new task by retrieving a new product version first = -> # task must be a function clientproductProjectionsstagedtruebyIdproductIdfetch then newPayload = _extend payloadversion: resultbodyversion updateTasknewPayload # now we must resolve the recover function with the new task # If we just want to retry the initial task then simply resolve an empty promise # Promise.resolve() Promiseresolve newTask else # we should not retry in this case, so simply bubble up the error Promisereject e
Mixins
Currently following mixins are provided by SphereUtils
:
Qutils
processList
Qutils
Deprecated in favour of
Bluebird
promises
A collections of Q utils (promise-based)
require 'sphere-node-utils'
processList
Process each element in the given list using the function fn
(called on each iteration).
The function fn
has to return a promise that should be resolved when all elements of the page are processed.
list = key: '1'key: '2'key: '3'processList list # elems is an array doSomethingWithelems # it's a promise then -> # something else anotherPromise
Note that the argument passed to the process function is always an array, containing a number of elements defined by
maxParallel
option
You can pass some options as second argument:
accumulate
whether the results should be accumulated or not (defaulttrue
). If not, an empty array will be returned from the resolved promise.maxParallel
how many elements from the list will be passed to the processfn
function (default1
)
Examples
(Coming soon)
Contributing
In lieu of a formal styleguide, take care to maintain the existing coding style. Add unit tests for any new or changed functionality. Lint and test your code using Grunt. More info here
Releasing
Releasing a new version is completely automated using the Grunt task grunt release
.
grunt release // patch releasegrunt release:minor // minor releasegrunt release:major // major release
License
Copyright (c) 2014 SPHERE.IO Licensed under the MIT license.