winston-opensearch

transport for the winston logging toolkit.

Most of code is from the winston-elasticsearch version 0.17.3

Features

logstash compatible message structure.
Thus consumable with kibana.
Date pattern based index names.
Custom transformer function to transform logged data into a different message structure.
Buffering of messages in case of unavailability of ES. The limit is the memory as all unwritten messages are kept in memory.

Compatibility

I tested Winston 3.8.2, Opensearch 2.7 for AWS usage.

Installation

npm install --save winston winston-opensearch

Usage

const winston = require('winston');
const { OpensearchTransport } = require('winston-opensearch');

const esTransportOpts = {
  level: 'info',
};
const esTransport = new OpensearchTransport(esTransportOpts);
const logger = winston.createLogger({
  transports: [esTransport],
});
// Compulsory error handling
logger.on('error', (error) => {
  console.error('Error in logger caught', error);
});
esTransport.on('error', (error) => {
  console.error('Error in logger caught', error);
});

The winston API for logging can be used with one restriction: Only one JS object can only be logged and indexed as such. If multiple objects are provided as arguments, the contents are stringified.

Options

level [info] Messages logged with a severity greater or equal to the given one are logged to ES; others are discarded.
index [none | when dataStream is true, logs-app-default] The index to be used. This option is mutually exclusive with indexPrefix.
indexPrefix [logs] The prefix to use to generate the index name according to the pattern <indexPrefix>-<indexSuffixPattern>. Can be string or function, returning the string to use.
indexSuffixPattern [YYYY.MM.DD] a Day.js compatible date/ time pattern.
transformer [see below] A transformer function to transform logged data into a different message structure.
useTransformer [true] If set to true, the given transformer will be used (or the default). Set to false if you want to apply custom transformers during Winston's createLogger.
ensureIndexTemplate [true] If set to true, the given indexTemplate is checked/ uploaded to ES when the module is sending the first log message to make sure the log messages are mapped in a sensible manner.
indexTemplate [see file index-template-mapping.json] the mapping template to be ensured as parsed JSON. ensureIndexTemplate is true and indexTemplate is undefined
flushInterval [2000] Time span between bulk writes in ms.
retryLimit [400] Number of retries to connect to ES before giving up.
dataStream [false] Use Opensearch datastreams.
healthCheckTimeout [30s] Timeout for one health check (health checks will be retried forever).
healthCheckWaitForStatus [yellow] Status to wait for when check upon health. See its API docs for supported options.
healthCheckWaitForNodes [>=1] Nodes to wait for when check upon health. See its API docs for supported options.
client An opensearch client instance. If given, the clientOpts are ignored.
clientOpts An object passed to the ES client. See its docs for supported options.
waitForActiveShards [1] Sets the number of shard copies that must be active before proceeding with the bulk operation.
pipeline [none] Sets the pipeline id to pre-process incoming documents with. See the bulk API docs.
buffering [true] Boolean flag to enable or disable messages buffering. The bufferLimit option is ignored if set to false.
bufferLimit [null] Limit for the number of log messages in the buffer.
source [none] the source of the log message. This can be useful for microservices to understand from which service a log message origins.
internalLogger [console.error] A logger of last resort to log internal errors.

Logging of OpenSearch Client

The default client and options will log through console.

Interdependencies of Options

When changing the indexPrefix and/or the transformer, make sure to provide a matching indexTemplate.

Transformer

The transformer function allows mutation of log data as provided by winston into a shape more appropriate for indexing in Opensearch.

The default transformer generates a @timestamp and rolls any meta objects into an object called fields.

Params:

logdata An object with the data to log. Properties are:
- timestamp [new Date().toISOString()] The timestamp of the log entry
- level The log level of the entry
- message The message for the log entry
- meta The meta data for the log entry

Returns: Object with the following properties

@timestamp The timestamp of the log entry
severity The log level of the entry
message The message for the log entry
fields The meta data for the log entry

The default transformer function's transformation is shown below.

Input A:

{
  "message": "Some message",
  "level": "info",
  "meta": {
    "method": "GET",
    "url": "/sitemap.xml",
    ...
  }
}

Output A:

{
  "@timestamp": "2019-09-30T05:09:08.282Z",
  "message": "Some message",
  "severity": "info",
  "fields": {
    "method": "GET",
    "url": "/sitemap.xml",
    ...
  }
}

The default transformer can be imported and extended

Example

const { OpensearchTransformer } = require('winston-opensearch');
const esTransportOpts = {
  transformer: (logData) => {
    const transformed = OpensearchTransformer(logData);
    transformed.fields.customField = 'customValue';
    return transformed;
  },
};
const esTransport = new OpensearchTransformer(esTransportOpts);

Note that in current logstash versions, the only "standard fields" are @timestamp and @version, anything else is just free.

A custom transformer function can be provided in the options initiation.

Events

error: in case of any error.

Example

An example assuming default settings.

Log Action

logger.info('Some message', {});

Only JSON objects are logged from the meta field. Any non-object is ignored.

Generated Message

The log message generated by this module has the following structure:

{
  "@timestamp": "2019-09-30T05:09:08.282Z",
  "message": "Some log message",
  "severity": "info",
  "fields": {
    "method": "GET",
    "url": "/sitemap.xml",
    "headers": {
      "host": "www.example.com",
      "user-agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
      "accept": "*/*",
      "accept-encoding": "gzip,deflate",
      "from": "googlebot(at)googlebot.com",
      "if-modified-since": "Tue, 30 Sep 2019 11:34:56 GMT",
      "x-forwarded-for": "66.249.78.19"
    }
  }
}

Target Index

This message would be POSTed to the following endpoint:

http://localhost:9200/logs-2019.09.30/log/

So the default mapping uses an index pattern logs-*.