serverless-offline-streams
This plugin provides support for event driven systems using Serverless Offline
Supported Features
SQS
- Bind to both real AWS SQS queues and SQS compatible MQ instances at the same time. It is recommended to include an ElasticMQ instance as part of your test stack (see the example for a sample docker-compose file.
- Auto-create SQS queues from Resource definitions
- Create SQS queues via plugin configuration
- Auto-scan lambda function definitions and bind queues to appropriate handlers
- Resolve direct arn, Ref, Fn::Import and FN::GetAtt queue bindings
- Create, Delete and purge queues on stack start (configurable)
- Poll queues for new messages and invoke bound handlers
- Remove messages from queues on success
DynamoDB
- DynamoDB Streams events
- DynamoDB Streams event filtering
DynamoDB Streams
- Bridges the DynamoDB Local plugin and Serverless Offline
- Binds DynamoDB streams event mappings to the appropriate lambdas
- Supports Event Filtering
Installation
npm install serverless-offline-streams
In your serverless.yml:
plugins:
- serverless-offline-streams
Configuration
custom:
serverless-offline-streams:
dynamodb:
endpoint: http://localhost:8000 # Required - dynamodb local endpoint
tableNames: # Optional. See below for explanation of table names mapping
tableKey1: tableName1
tableKey2: tableName2
sqs:
enabled: false # Optional. Whether to activate SQS queue event mappings
host: http://127.0.0.1:8050 # Required. Host & port of elasticmq instance
localQueueManagement:
createFromResources: true # Optional. If true, will scan defined Resources for queues and create them according to the config
removeOnStart: true # Optional. If true, will remove all existing queues in elasticmq on startup
purgeOnStart: false # Optional. If true, will purge all existing queues in elasticmq on startup
additionalQueues: # Optional. Additional queues to create on startup
- name: 'Queue1', # Queue name to bind handlers to
aliases: # Aliases for the queue - Useful when binding to output variables in other stacks
- "Queue1Alias1"
- "Queue1Alias2"
visibilityTimeout: 5 # Optional. Queue VisibilityTimeout
delaySeconds: 5 # Optional. Queue DelaySeconds
remote: # Optional. If present, the plugin assumes the queue is a real AWS queue instead of a local elasticmq one
queueUrl: # Required. The full URL of the queue, in AWS (in the form http(s)://sqs.REGION.amazonaws.com/ACCOUNT/QUEUE-NAME)
pollConfig: # Optional. See below for an explanation of how polling works
strategy: backoff # Optional. Either backoff or fixed-interval
drainQueues: false # Optional. Whether to keep retrieving messages from a queue until there are no messages, if a message is found
messageBatchSize: 10 # Optional. How many messages to retrieve in each batch
fixedIntervalMs: 5000 # Optional. Only applicable to fixed-interval strategy
backOffType: double # Optional. Only applicable to backoff strategy. Either double or step
minIntervalMs: 100 # Optional. Only applicable to backoff strategy
maxIntervalMs: 5000 # Optional. Only applicable to backoff strategy
intervalStepMs: 100 # Optional. Only applicable to backoff strategy with step backoffType
SQS Configuration
Polling
The plugin will poll all bound queues at regular intervals, based on its polling strategy (see below), attempting to retrieve messageBatchSize messages on each poll.
If at least one message is found on a queue, and drainQueues is true, then the plugin will continue to retrieve messages until the queue is empty.
Warning: If a queue receives a lot of messages and drainQueues is set to true, this may cause starvation of handling to the other queues, as the thread will be blocked attempting to drain the busier queue. It is better to allow a round robin approach where each poll interval will attempt to retrieve messageBachSize messages from each queue in turn.
Polling Strategies
The plugin supports two polling strategies:
Backoff
This is the default strategy. The plugin will first poll every minIntervalMs milliseconds. If no messages are received on any queue, the polling interval will increase, up to maxIntervalMs. If a message is received, the polling will drop back down to minIntervalMs (on the basis that more messages are likely to follow)
The way in which the polling interval increases is determined by the backoffType property:
- step - The polling interval will increase by intervalStepMs each time
- double - The polling interval will increase by doubling each time (e.g. if minIntervalMs is 100, the next interval will be 200, then 400, then 800 etc.)
Fixed Interval
The plugin will poll every fixedIntervalMs, regardless of whether messages are found or not
DynamoDB Streams Configuration
Table Names
- If you are using only string ARNs, the plugin will be able to extract the table names from the ARN.
- If you are using a Ref function, the plugin will scan your resources to locate the table
- However, if you are using a cross stack reference, you must provide a mapping in the
tableNames
configuration block
The entry value should be the dynamo table name and the key should be the name of the cross stack reference after interpolation. For example, given the following event mapping and having stage set to 'dev':
handler: src/handler.handler
events:
- stream:
arn: !ImportValue ${self:provider.stage}StreamArn
type: dynamodb
batchSize: 10
startingPosition: TRIM_HORIZON
The import value key would resolve to "devStreamArn" so the tableNames block would be as follows:
custom:
serverless-offline-dynamodb-streams-handler:
tableNames:
devStreamArn: my-dynamo-table
Usage with Typescript
In order for the plugin to correctly locate your transpiled handlers, set the 'location' property of serverless-offline This should be compatible with a number of transpilers that put transpiled code in a separate build directory (tested with the serverless-esbuild plugin)
custom:
serverless-offline:
location: .esbuild/.build
Filter Patterns
See the Filter Patterns Guide
Future work:
- Supporting AWS EventBridge