@telemetry-js/telemetry

Telemetry

Modular metrics for Node.js. Collect, process and publish metrics, picking only the metrics that you need.

Table of Contents

Click to expand

• Highlights ✨
• Usage
• Examples
  • Basic
  • Aggregation
  • But I Just Want To Publish A One-Time Metric
• Available Plugins
  • Collectors
  • Schedules
  • Processors
  • Publishers
• Naming Guide
  • Metric Names
  • Metric Tags
  • Plugin Package Names
• API
• Install
• Acknowledgements
• License

Highlights ✨

• Plugin-based and evented
• A plugin serves one of 4 roles:
  1. Collector: emits metrics (when "pinged")
  2. Processor: decorates or combines metrics
  3. Schedule: pings other plugins on an interval
  4. Publisher: publishes metrics
• Add custom metrics sourced from a function, counter or your own plugin
• Add tags (a.k.a. dimensions) to metrics
• Includes plugins for automatic tags like instanceid on EC2
• Locally aggregate before publishing metrics, to save bandwidth
• Publish to CloudWatch, AppOptics, Logz.io Metrics, stdio or your own publisher.

Usage

To get started you'll need 3 things.

1. The telemetry module (this)

This module controls what and when to collect and publish metrics. It groups plugins into "tasks". One task might publish a memory metric with a name tag to CloudWatch every 5 minutes, while another task publishes the same metric with a name, instanceid and region tag to AppOptics every minute. If the desired tags and interval are the same for both publishers, you can use a single task.

Tasks are also useful to manage the lifetime of metrics. Say we have an application with long-running background jobs that we want metrics on. Each job can have its own telemetry task and add for example a job_id tag to its metrics.

2. At least one collector and publisher

Without a collector, there are no metrics. Without a publisher, the metrics don't go anywhere.

3. A schedule like @telemetry-js/schedule-simple.

Most collectors follow a pull-based model: they must be "pinged" before they emit metrics. A schedule does just that, typically pinging your plugins on a fixed interval.

Examples

Basic

This example collects Node.js memory metrics every 30 seconds and sends them to CloudWatch with a name tag.

const telemetry = require('@telemetry-js/telemetry')()
const mem = require('@telemetry-js/collector-nodejs-memory')
const simple = require('@telemetry-js/schedule-simple')
const tag = require('@telemetry-js/processor-tag')
const cloudwatch = require('@telemetry-js/publisher-cloudwatch')

telemetry.task()
  .collect(mem)
  .schedule(simple, { interval: '30s' })
  .process(tag, { name: 'my-app' })
  .publish(cloudwatch)

// Start tasks
await telemetry.start()

Aggregation

This example collects metrics every 30 seconds, locally aggregates them, and sends summary metrics to CloudWatch and AppOptics every 5 minutes. Those summary metrics have a min, max, sum and count of recorded values, and are called "statistic sets" in CloudWatch.

const telemetry = require('@telemetry-js/telemetry')()
const fn = require('@telemetry-js/collector-function')
const counter = require('@telemetry-js/collector-counter')
const disk = require('@telemetry-js/collector-disk')
const simple = require('@telemetry-js/schedule-simple')
const summarize = require('@telemetry-js/processor-summarize')
const tag = require('@telemetry-js/processor-ec2-instance-id')
const cloudwatch = require('@telemetry-js/publisher-cloudwatch')
const appoptics = require('@telemetry-js/publisher-appoptics')

// Example of custom metric that takes value from a function
const rand = fn.sync('myapp.random.count', { unit: 'count' }, Math.random)
const errors = counter.delta('myapp.errors.delta')

telemetry.task()
  .collect(rand)
  .collect(errors)
  .collect(disk, { metrics: ['*.percent'] })
  .schedule(simple, { interval: '30s' })
  .process(summarize, { window: '5m' })
  .process(tag)
  .publish(cloudwatch)
  .publish(appoptics, { token: '***' })

await telemetry.start()

// Elsewhere in your app
errors.increment(1)

But I Just Want To Publish A One-Time Metric

Got you:

const appoptics = require('@telemetry-js/publisher-appoptics')
const single = require('@telemetry-js/metric').single

const publisher = appoptics({ token: '***' })
const metric = single('myapp.example.count', { unit: 'count', value: 2 })

publisher.publish(metric)

await publisher.flush()

Available Plugins

Collectors

Name	Description
disk	Free, available and total disk space
net	TCP, UDP, ICMP, IP metrics
sockstat	/proc/net/sockstat metrics
nodejs-gc	Node.js garbage collection duration
nodejs-memory	Node.js memory (RSS, heap, external)
nodejs-event-loop-duration	Node.js event loop duration
nodejs-event-loop-lag	Node.js event loop lag
osmem	Free, used and total memory
counter	A counter incremented by you
function	Collect metric value from a function
redis	Redis metrics
incidental	Record incidental values
stopwatch	Record durations
aws-lb	AWS LB node count
dmesg	Count certain kernel messages

Schedules

Name	Description
simple	Collect metrics on a fixed interval

Processors

Name	Description
summarize	Locally summarize metrics within a time window
tag	Add your own tags
ecs-tags	Add common tags for ECS container
ec2-instance-id	Add instanceid tag, fetched from metadata
ec2-instance-tags	Copy all instance tags, fetched from EC2 API
ec2-instance-name	Copy only the name tag (if set)
ec2-instance-region	Add region tag, fetched from metadata
debug	Log metrics and task lifecycle events with debug

Publishers

Name	Description
appoptics	AppOptics
cloudwatch	CloudWatch
logzio-metrics	Logz.io Metrics
ndjson	Write NDJSON to a stream
debug	Log metrics and task lifecycle events with debug

Naming Guide

Metric Names

Lowercase, namespaced by dots (e.g. myapp.foo.bar.count), prefixed with project (myapp.), suffixed with unit (.count).

Metric names from Telemetry plugins are prefixed with telemetry. Custom metrics of your app should be prefixed with your appname. If however, your custom metric is not app-specific, then you could instead use metrics tags to differentiate apps and/or runtime contexts. You might as well write a Telemetry plugin at that point. Do it!

Metric Tags

Lowercase, only a-z. E.g. instanceid, name, project, environment.

Plugin Package Names

Follow the format <role>-<name>, where:

• <role> is one of collector, processor, publisher, schedule, or, if the plugin serves multiple roles, then simply plugin
• <name> typically matches the metric name (for collectors), purpose (for processors) or publisher name. If the plugin is a collector that emits multiple metrics (e.g. disk.free.bytes, disk.total.bytes) then use the longest common prefix as <name> (e.g. disk).

API

Yet to document.

Install

With npm do:

npm install @telemetry-js/telemetry

Acknowledgements

This project is kindly sponsored by Reason Cybersecurity Ltd.

License

MIT © Vincent Weevers