qpp-shared-logger-node

A simple configurable wrapper around the winston application logger and the morgan http access logger.

This logger is opinionated. Configure it with a minimal set of options, and get Splunk-searchable, PII-scrubbed, and QPP-compliant logs with minimal effort.

Installation

$ npm install qpp-shared-logger-node --save

Usage

This module accepts configuration, sets up the logger, then yields a winston Logger object for you to use.

It is recommended that you set up your project with a log.js module that you can import from the rest of your project; log.js is where you will configure the shared logger and expose the winston Logger.

First, pass your configuration to the module, then get access to the Logger object.

log.js:

const sharedLogger = require('qpp-shared-logger-node');

sharedLogger.configure({
    projectSlug: 'my-app-name'
});

module.exports = sharedLogger.logger; // a winston Logger

In other parts of your code:

const logger = require('log');

logger.info('fetching data', { id: object.id, sort: 'asc' });

Logging with context fields

The contextLogger function returns a logger object that includes a set of fields with each log message. The fields are merged with those passed to a logging method.

For example, this can be used to log a request id in all entries related to a single client request:

req.logger = sharedLogger.contextLogger({ requestId: uuidv1(), url: req.url });
req.logger.info('started');
req.logger.error(message, { error: err });

Field values in the log call take priority over values in the context fields.

Context Logging Defaults

There are defaults that should be added to the context of the beginning with each request if possible.

Attribute Name	Attribute Value	Description
requestId	UUID	This is a unique ID that will allow for tracking events throughout the livetime of the request. This can be populated either through uuid or express request id or some other preferred UUID generator method.
oktaId	string	This value should be pulled from the auth service if possible
applicationBuild or gitSHA	string	This should be the build id or the git SHA of the current running application

const requestContext = {
  requestId: uuidv1(),
  oktaId: AuthService(), // The auth service should return the okta id.
  applicationBuild: process.env.BUILD_ID // what constant the application would be storing this information
}

req.logger = sharedLogger.contextLogger(requestContext);

Configuration

Key	Description	Allowed Values
projectSlug (Required)	The name of your app, to be included in the log metadata and the log file path.	any string

Environment Defaults

This module will read process.env.NODE_ENV to determine your application's "environment", and configure these settings out-of-the-box:

Environment	Application log path	Log level
development	console	debug
test	./app.log	silly
production	/var/log/qpp-service/{{ projectSlug }}/app.log	info

Environment	Access log path	Format
development	console	dev
test	./access.log	combined
production	/var/log/qpp-service/{{ projectSlug }}/access.log	combined

Advanced Application Log Configuration

You may override the defaults:

Key	Description	Allowed Values	Default
addlFormats	This key accepts an array of Winston compatible logform formats that are then added to the logger. This allows you to provide formats for unique scenarios that are not covered by the default formats.	An array of logform formats.	[]
format	Format logs should be written in. Default is json format. More on winston fromats can be found https://www.npmjs.com/package/winston#formats.	json, simple, prettyPrint, logstash	json
environment	Override the "node environment" that your app is running in. Using the conventional values will automatically configure various log settings for you. Conventional values are development, test, and production. Deployed code should run with NODE_ENV=production	process.env.NODE_ENV
logDirectory	A valid directory where the log file should be written, or "console" to write to stdout. Will throw if the directory does not exist or is not writable.	console, or an absolute dir	built from environment
logFilenamePrefix	The application log filename will be built from this prefix, by default it will be app.YYYYMMDD.log	any string	app
datePattern	A string representing the moment.js date format to be used for rotating. The meta characters used in this string will dictate the frequency of the file rotation. For example, if your datePattern is simply 'HH' you will end up with 24 log files that are picked up and appended to every day. (default: 'YYYYMMDD')	any string containing a standard date pattern	YYYYMMDD
logFileExtension	The log file extension to use.	string	log
logLevel	All log messages at this level or higher will be logged. none effectively turns off logging.	none, error, warn, info, verbose, debug, silly	chosen based on environment
logTimestamps	Add timestamps to log entries	true or false	true
logColorize	If true, log messages will be sent colorized (most valuable when logging to the console)	true or false	false
redactKeys	An array of keys to scrub from the log metadata	an array of lowercase strings	['email', 'firstname', 'lastname', 'password', 'ptan', 'tin', 'userid', 'username']
redactRegexes	An array of regular expressions representing string values to scrub	an array of regular expressions (string or RegExp)	[]
maxDays	The maximum number of days to keep logs for.	A number, in days	0 (No deletion)
rotationMaxsize	The max size the log file should reach before it is rotated.	a size, in bytes. For example, 1M = 1000000. Or 'none' to never rotate logs	50000000 (50M)
splunkSettings	Adding the Splunk configuration settings will add Splunk http transport via the winston-splunk-httplogger package	object	undefined

Splunk Transport Configuration

Key	Description	Allowed Values	Default
url	URL string to pass to url.parse. This will try to set host, path, protocol, port, url. Any of these values will be overwritten if the corresponding property is set on config	http://localhost:8888	undefined
token	The Splunk HTTP Event Collector token

WARNING - If the Splunk transport fails to connect to Splunk, log messages will be lost during the outage. DO NOT rely on this logger alone if you need to have guaranteed delivery of all logs to Splunk.

Advanced HTTP Access Log Configuration

Override defaults.

Key	Description	Allowed Values	Default
accessLog.logDirectory	A valid directory where the log file should be written, or "console" to write to stdout. Will throw if the directory does not exist or is not writable.	console, or an absolute dir	built from environment
accessLog.logFilenamePrefix	The access log filename will be built from this prefix, by default it will be access.YYYYMMDD.log	any string	access
accessLog.format	The log format. Note that combined and common are well-known as NCSA log formats.	combined, common, dev, short, tiny, none	chosen based on environment
accessLog.rotationMaxsize	The max size the log file should reach before it is rotated.	a size, in bytes. For example, 1M = 1000000. Or 'none' to never rotate logs	50000000 (50M)
accessLog.maxFiles	The maximum number of rotated logs to keep around. Logs rotated after this will be removed.	a count, in number of files	None (No deletion)

Considerations for production logging

• Splunk ingesters are set up to find log files under /var/log, so always send your production logs here or to a subdirectory within.

• To differentiate logs emitted from the DEV or IMP or PROD deployed environments, be sure the name is embedded in one or more of the Splunk-searchable fields. For example, configure your log directory with the "deployed environment" embedded as a subdir: /var/log/qpp-service/{{ dev|imp|prod }}/{{ projectSlug }}/

  So on Halloween, Splunk will index the source of log messages as /var/log/qpp-service/imp/projectSlug/app.20171031.log. A sample Splunk query to pick up your IMP logs: source=/var/log/qpp-service/imp/projectSlug/*.

• Suggestion: use an environment var (not NODE_ENV) to communicate the name of the environment from the OS to the app. The auth service, for example, uses the ENVIRONMENT var.

PII (Personally-Identifiable Information) and log scrubbing

Log messages are composed of a message (String) and metadata (key-value). Keys in the metadata that match one of the redactKeys will not have their value logged. Note that the logger does not search the String message for data to scrub. Please keep log messages simple, and add metadata with proper keys.

Compatibility

Tested with node v10.15.3 and node v12.15.0

Development

$ brew install gitleaks # Installs gitleaks for pre-commit secret detection. If not running on Mac, see https://github.com/zricethezav/gitleaks/releases
$ npm install       # install dependencies
$ npm  test         # run tests, also report coverage in ./coverage/index.html
$ npm run  format   # run prettier to format code
$ npm run lint      # run eslint to check code

Release Process

The release process is semi-automated via github actions. A number of steps are necessarily left manual (such as versioning) and require intervention from the user.

1. Create a release branch release/* either off of master to pull all changes, or by cherry-picking only certain changes.

2. Bump the version using npm version <patch | minor | major>.

3. Push the release branch to github.

4. Github actions will automatically create a release and tag based off the version in package.json.

5. Review the draft release page and publish it as a pre-release.

6. Github actions will automatically publish a package to npm. Additionally, a new pull request will be created to backfill master from release if necessary.

Want to Contribute?

Want to file a bug or contribute some code? Read up on our guidelines for contributing.

Public Domain

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

See the formal LICENSE file.

Resources

• https://jira.cms.gov/browse/QPPELIG-130
• https://jira.cms.gov/browse/QPPELIG-131
• https://confluence.cms.gov/display/QPPGUIDE/Shared+Logger
• https://confluence.cms.gov/display/QPPPLAT/Logging+Standard+Proposal
• https://confluence.cms.gov/display/QPPGUIDE/Shared+Library+V1+Milestone
• https://www.owasp.org/index.php/Logging_Cheat_Sheet
• https://www.npmjs.com/package/winston
• https://www.npmjs.com/package/morgan
• https://github.com/i0natan/nodebestpractices/blob/master/sections/production/setnodeenv.md