reliable-get

Reliable HTTP get wrapper (with cache and serve stale on error), best wrapped around things you dont trust very much.

Behaviour

If reliable-get makes a request that times out or errors, its callback will receive both an error object and a previously cached response, if one is present in cache. You can then decide whether to ignore the error and use the cached response, or not.

Basic usage

var ReliableGet = require('reliable-get');
var config = {
  cache:{
    engine:'memorycache'
  }
};
var rg = new ReliableGet(config);
rg.on('log', function(level, message, data) {
  // Wire up to your favourite logger
});
rg.on('stat', function(type, key, value) {
 // Wire up to your favourite stats library (e.g. statsd)
});
rg.get({url:'http://www.google.com'}, function(err, response) {
   console.log(response.content);
});

Configuration options

When you create an instance of reliable-get you need to specify the cache configuration. This then applies across all requests.

var config = {
  cache:{
    engine:'redis',
    url:'redis://localhost:6379?db=0'
  }
};

You can also pass a property requestOpts to pass options to be used in request. Example:

var config = {
  cache: {
    engine: 'redis',
    url: 'redis://localhost:6379?db=0'
  },
  requestOpts: {
    forever: true,
    followRedirect: false
  }
}

Property	Description	Example / Default	Required
cache.engine	Cache to use, redis/memcached/memorycache/nocache	nocache	No
cache.engine.url	URL to redis	localhost:6379	No
cache.compress	Use snappy compression	false	No
cache.namespace	Prefix for redis keys	''	No
cache.hosts	Array of host:port combinations for memcached	[]	No
cache.autodiscover	Use Elasticache Auto Discovery	false	No

GET options

When making a get request, you need to provide a basic options object:

rg.get({url:'http://www.google.com'}, function(err, response) {
   console.log(response.content);
});

Property	Description	Example / Default	Required
url	Service to get	http://my-service.tes.co.uk	Yes
timeout	Timeout for service	5000	No
cacheKey	Key to store cached value against	my-service_tes_co_uk	No
tags	List of tags (surrogate keys)	[]	No
cacheTTL	TTL of cached value in ms	1 minute (60000)	No
explicitNoCache	Do not cache under any circumstances	false	No
headers	Headers to send with request		No
tracer	Unique value to pass with request		No
type	Type of request, used for statsd and logging		No
statsdKey	Key that statsd events will be posted to		No
eventHandler	Object (see below) for logging and stats		No

Example from a Compoxure backend request:

var options = {
  url: targetUrl,
  cacheKey: targetCacheKey,
  cacheTTL: targetCacheTTL,
  timeout: utils.timeToMillis(backend.timeout || DEFAULT_LOW_TIMEOUT),
  headers: backendHeaders,
  tracer: req.tracer,
  statsdKey: 'backend_' + utils.urlToCacheKey(host)
};

From a compoxure fragment request:

var options = {
  url: url,
  timeout: timeout,
  cacheKey: cacheKey,
  cacheTTL: cacheTTL,
  explicitNoCache: explicitNoCache,
  headers: optionsHeaders,
  tracer: req.tracer,
  statsdKey: statsdKey
};

The options object is fully passed down to the request.

Response fields

The library will decorate with response with some useful keys that you may need to use, there are following:

• stale - is added to response when the request to origin failed and a stale cached version is returned instead
• cached - is true if there was a cache hit, otherwise false
• realTiming - show the time it took for the response to be returned

Tags

Some store (memorycache/redis) supports assigning one or more tags to a certain resource. They are used only if the resource is cached, to purge all cache entries with the same tag.

Configuration

Cache configuration

The cache object accept any config value accepted by redis. It also takes:

• config.cache.engine: nocache, memorycache, memcached, redis (use nocache/memorycache for testing only!)
• config.cache.url: redis dsn, it is translated to the connection parameters
• config.cache.compress: enable snappy compression on cached items
• config.cache.namespace: adds this string as a prefix to any key. Useful to share redis with other services or migrations
• config.cache.hosts: Memcached cluster nodes addresses as <host>:<port> combinartions in an array
• config.cache.autodiscover: enable AWS Elasticache Auto Discovery of Memcached cache cluster nodes

Request Configuration

The "config.requestOpts" contains the default configuration passed to "request".