Canoe
Paddle downstream with Canoe, an S3 utility library for Node.js. Built on the AWS Node SDK.
Install
npm install canoe --save
Usage
Create a new Canoe
instance by passing an instance of AWS.S3
from the aws-sdk
module.
var AWS = Canoe = ; var s3 = ;var canoe = s3;
Methods
createPrefixedReadStream
Create a readable stream of all objects whose keys match a given prefix.
Usage
Combines multiple objects into a single readable stream, preserving the order of the objects.
// This will stream all objects starting with "path/to/things/name_"// For example, it would match "path/to/things/name_1", "path/to/things/name_2", etccanoe
createWriteStream
A Node 0.10-friendly writable stream interface ("streams2") for uploading objects to S3.
Usage
Creates a writable stream for a given S3 bucket/key. The stream will be returned immediately and also passed to an optional callback.
The stream will be writable when it's returned, but not actually ready to send data to S3 yet (data will be buffered internally in the meantime). If you use the immediately returned stream, be sure to respect false
-y return values, as Node's readable.pipe()
does. The stream will be fully ready when the callback is run.
The stream will emit a writable
event when it's ready to send data to S3. A close
event will be emitted when the stream is fully done consuming (uploading) the data; this will happen after the finish
event.
Basic usage:
var writableStream = canoe;writableStream;writableStream;
Example: Piping a large file
// Create a stream and use it immediatelyvar writeable = canoe; // For fun, let's keep track of how much memory we need.// We'll print the memory peak once the upload finishes.// Peak will stay fairly consistent regardless of file size.var maxMemory = 0;; writeable; // Imagine you have some massive file you want to upload that doesn't fit into memoryfs;
Example: Writing chunks of data
// Create a stream and wait to use it in a callbackvar s3Params = Bucket: 'random-access-memories' Key: 'instant.crush';canoe;
Contributing
See the contributing documentation.
Author
Evan Solomon (personal website), supported by The Obvious Corporation.
License
Copyright 2013 The Obvious Corporation.
Licensed under the Apache License, Version 2.0. See the top-level file LICENSE.txt
and (http://www.apache.org/licenses/LICENSE-2.0).