@cumulus/aws-client
TypeScript icon, indicating that this package has built-in type declarations

20.1.2 • Public • Published

@cumulus/aws-client

Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.

⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.

Usage

npm install @cumulus/aws-client

Interacting with Localstack

To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:

NODE_ENV=test

API

Modules

CloudFormation
CloudwatchEvents
DynamoDb
Kinesis
Lambda
S3
SNS
SQS
STS
SecretsManager
StepFunctions

Classes

DynamoDbSearchQueue

Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.

S3ListObjectsV2Queue

Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.

S3ObjectStore

Class to use when interacting with S3

Functions

createKey(params)Promise.<Object>

Create a KMS key

See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.

encrypt(KeyId, Plaintext)Promise.<string>

Encrypt a string using KMS

decryptBase64String(ciphertext)string

Decrypt a KMS-encrypted string, Base 64 encoded

CloudFormation

CloudFormation~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>

Describes a given CloudFormation stack

See CloudFormation.Stack

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.Stack> - The resources belonging to the stack

Param Type Description
StackName string The name of the CloudFormation stack to query

CloudFormation~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>

Describes the resources belonging to a given CloudFormation stack

See CloudFormation.StackResources

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.StackResources> - The resources belonging to the stack

Param Type Description
StackName string The name of the CloudFormation stack to query

CloudFormation~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

Get parameter values for the given CloudFormation stack

Kind: inner method of CloudFormation
Returns: Promise.<Object> - Object keyed by parameter names

Param Type Description
stackName string The name of the CloudFormation stack to query
parameterKeys Array.<string> Key names for the stack parameters that you want to return

CloudwatchEvents

CloudwatchEvents~putEvent(name, schedule, state, [description], [role]) ⇒ Promise.<CloudWatchEvents.PutRuleResponse>

Create a CloudWatch Events rule

Kind: inner method of CloudwatchEvents

Param Type Description
name string the rule name
schedule string a ScheduleExpression
state string the state of the rule
[description] string
[role] string a Role ARN

DynamoDb

DynamoDb.scan ⇒ Promise.<Object>

Call DynamoDb client scan

See DocumentClient.scan() for descriptions of params and the return data.

Kind: static property of DynamoDb

Param Type
params Object

DynamoDb.createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>

Create a DynamoDB table and then wait for the table to exist

Kind: static method of DynamoDb
Returns: Promise.<Object> - the output of the createTable call

Param Type Description
params Object the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#createtable

DynamoDb.deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise

Delete a DynamoDB table and then wait for the table to not exist

Kind: static method of DynamoDb

Param Type Description
params Object the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#deletetable

DynamoDb~get(params) ⇒ Promise.<Object>

Call DynamoDb client get

See DocumentClient.get() for descriptions of params and the return data.

Kind: inner method of DynamoDb
Throws:

  • RecordDoesNotExist if a record cannot be found
Param Type Description
params Object
params.tableName string Table name to read
params.item GetCommandInput.Key Key identifying object to get
params.client DynamoDBDocument Instance of a DynamoDb DocumentClient
params.getParams Object Additional parameters for DocumentClient.get()

DynamoDb~parallelScan(params) ⇒ Promise

Do a parallel scan of DynamoDB table using a document client.

See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan. See DocumentClient.scan().

Kind: inner method of DynamoDb

Param Type Description
params Object
params.totalSegments number Total number of segments to divide table into for parallel scanning
params.scanParams ScanInput Params for the DynamoDB client scan operation See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
params.processItemsFunc function Function used to process returned items by scan
[params.dynamoDbClient] DynamoDBDocument Instance of Dynamo DB document client
[params.retryOptions] pRetry.Options Retry options for scan operations

Kinesis

Kinesis~describeStream(params, retryOptions) ⇒ Promise.<Object>

Describe a Kinesis stream.

Kind: inner method of Kinesis
Returns: Promise.<Object> - The stream description response

Param Type Description
params Object
params.StreamName string A Kinesis stream name
retryOptions Object Options passed to p-retry module

Lambda

Lambda~isEventBridgeEvent()

Bare check for EventBridge shape

Kind: inner method of Lambda

Lambda~invoke()

Invoke a Lambda function

Kind: inner method of Lambda

S3

S3.getS3Object

Deprecated

Gets an object from S3.

Kind: static property of S3

S3.recursivelyDeleteS3Bucket ⇒

Delete a bucket and all of its objects from S3

Kind: static property of S3
Returns: the promised result of S3.deleteBucket

Param Description
bucket name of the bucket

S3.listS3ObjectsV2(params) ⇒ Promise.<Array>

Fetch complete list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3
Returns: Promise.<Array> - resolves to an array of objects corresponding to the Contents property of the listObjectsV2 response

Param Type Description
params Object params for the s3.listObjectsV2 call

S3.listS3ObjectsV2Batch(params)

Fetch lazy list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3

Param Description
params params for the s3.listObjectsV2 call

S3~s3Join(...args) ⇒ string

Join strings into an S3 key without a leading slash

Kind: inner method of S3
Returns: string - the full S3 key

Param Type Description
...args string | Array.<string> the strings to join

S3~parseS3Uri(uri) ⇒ Object

parse an s3 uri to get the bucket and key

Kind: inner method of S3
Returns: Object - Returns an object with Bucket and Key properties

Param Type Description
uri string must be a uri with the s3:// protocol

S3~buildS3Uri(bucket, key) ⇒ string

Given a bucket and key, return an S3 URI

Kind: inner method of S3
Returns: string - an S3 URI

Param Type Description
bucket string an S3 bucket name
key string an S3 key

S3~s3TagSetToQueryString(tagset) ⇒ string

Convert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'

Kind: inner method of S3
Returns: string - tags query string

Param Type Description
tagset Array.<Object> S3 TagSet array

S3~deleteS3Object(bucket, key) ⇒ Promise

Delete an object from S3

Kind: inner method of S3
Returns: Promise - promise of the object being deleted

Param Type Description
bucket string bucket where the object exists
key string key of the object to be deleted

S3~headObject(Bucket, Key, retryOptions) ⇒

Get an object header from S3

Kind: inner method of S3
Returns: returns response from S3.headObject as a promise

Param Description
Bucket name of bucket
Key key for object (filepath + filename)
retryOptions options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed

S3~s3ObjectExists(params) ⇒ Promise.<boolean>

Test if an object exists in S3

Kind: inner method of S3
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating if the object exists

Param Description
params same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property

S3~waitForObjectToExist()

Wait for an object to exist in S3

Kind: inner method of S3

Param Default Description
params.interval 1000 interval before retries, in ms
params.timeout 30000 timeout, in ms

S3~s3PutObject(params)

Put an object on S3

Kind: inner method of S3

Param Description
params same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

S3~putFile()

Upload a file to S3

Kind: inner method of S3

S3~s3CopyObject()

Copy an object from one location on S3 to another

Kind: inner method of S3

S3~promiseS3Upload()

Upload data to S3

see https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage

Kind: inner method of S3

S3~streamS3Upload()

Upload data to S3 using a stream

Kind: inner method of S3

S3~getObjectReadStream()

Get a readable stream for an S3 object

Kind: inner method of S3

S3~downloadS3File()

Downloads the given s3Obj to the given filename in a streaming manner

Kind: inner method of S3

S3~getObjectSize()

Get the size of an S3 object

Kind: inner method of S3

S3~s3GetObjectTagging()

Get object Tagging from S3

Kind: inner method of S3

S3~s3PutObjectTagging()

Puts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property

Kind: inner method of S3

S3~getObject()

Gets an object from S3.

Kind: inner method of S3

S3~waitForObject()

Get an object from S3, waiting for it to exist and, if specified, have the correct ETag.

Kind: inner method of S3

S3~getObjectStreamContents()

Transform streaming response from S3 object to text content

Kind: inner method of S3

S3~getTextObject()

Fetch the contents of an S3 object

Kind: inner method of S3

S3~getJsonS3Object()

Fetch JSON stored in an S3 object

Kind: inner method of S3

S3~fileExists()

Check if a file exists in an S3 object

Kind: inner method of S3

S3~deleteS3Files(s3Objs)

Delete files from S3

Kind: inner method of S3

Param Description
s3Objs An array of objects containing keys 'Bucket' and 'Key'

S3~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise

Upload the file associated with the given stream to an S3 bucket

Kind: inner method of S3
Returns: Promise - A promise

Param Type Description
fileStream ReadableStream The stream for the file's contents
bucket string The S3 bucket to which the file is to be uploaded
key string The key to the file in the bucket
s3opts Object Options to pass to the AWS sdk call (defaults to {})

S3~listS3Objects()

List the objects in an S3 bucket

Kind: inner method of S3

S3~deleteS3Buckets(buckets) ⇒ Promise

Delete a list of buckets and all of their objects from S3

Kind: inner method of S3
Returns: Promise - the promised result of S3.deleteBucket

Param Type Description
buckets Array list of bucket names

S3~calculateObjectHash(params)

Calculate the cryptographic hash of an S3 object

Kind: inner method of S3

Param Type Description
params Object
params.s3 S3 an S3 instance
params.algorithm string cksum, or an algorithm listed in openssl list -digest-algorithms
params.bucket string
params.key string

S3~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>

Validate S3 object checksum against expected sum

Kind: inner method of S3
Returns: Promise.<boolean> - returns true for success
Throws:

  • InvalidChecksum - Throws error if validation fails
Param Type Description
params Object params
params.algorithm string checksum algorithm
params.bucket string S3 bucket
params.key string S3 key
params.expectedSum number | string expected checksum
[params.options] Object crypto.createHash options

S3~getFileBucketAndKey(pathParams) ⇒ Array.<string>

Extract the S3 bucket and key from the URL path parameters

Kind: inner method of S3
Returns: Array.<string> - [Bucket, Key]

Param Type Description
pathParams string path parameters from the URL bucket/key in the form of

S3~createBucket(Bucket) ⇒ Promise

Create an S3 bucket

Kind: inner method of S3

Param Type Description
Bucket string the name of the S3 bucket to create

S3~createS3Buckets(buckets) ⇒ Promise

Create multiple S3 buckets

Kind: inner method of S3

Param Type Description
buckets Array.<string> the names of the S3 buckets to create

S3~multipartCopyObject(params) ⇒ Promise.<{etag: string}>

Copy an S3 object to another location in S3 using a multipart copy

Kind: inner method of S3
Returns: Promise.<{etag: string}> - object containing the ETag of the destination object

note: this method may error if used with zero byte files. see CUMULUS-2557 and https://github.com/nasa/cumulus/pull/2117.

Param Type Default Description
params Object
params.sourceBucket string
params.sourceKey string
params.destinationBucket string
params.destinationKey string
[params.sourceObject] S3.HeadObjectOutput Output from https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property
[params.ACL] string an S3 Canned ACL
[params.copyTags] boolean false
[params.chunkSize] number chunk size of the S3 multipart uploads

S3~copyObject()

Copy an S3 object to another location in S3

Kind: inner method of S3

Param Default Description
params.ACL an S3 Canned ACL
params.copyTags false
params.chunkSize chunk size of the S3 multipart uploads

S3~moveObject() ⇒ Promise.<undefined>

Move an S3 object to another location in S3

Kind: inner method of S3

Param Type Default Description
[params.ACL] string an S3 Canned ACL
[params.copyTags] boolean false
[params.chunkSize] number chunk size of the S3 multipart uploads

SNS

SNS~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>

Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.

Kind: inner method of SNS

Param Description
snsTopicArn SNS topic ARN
message Message object
retryOptions options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions

SNS~createSnsTopic(snsTopicName) ⇒

Create an SNS topic with a given name.

Kind: inner method of SNS
Returns: - ARN of the created SNS topic

Param Description
snsTopicName Name of the SNS topic

SQS

SQS~createQueue()

Create an SQS Queue. Properly handles localstack queue URLs

Kind: inner method of SQS

SQS~sendSQSMessage()

Send a message to AWS SQS

Kind: inner method of SQS

SQS~receiveSQSMessages()

Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.

Kind: inner method of SQS

SQS~isSQSRecordLike()

Bare check for SQS message Shape

Kind: inner method of SQS

SQS~parseSQSMessageBody()

Extract SQS message body

Kind: inner method of SQS

SQS~deleteSQSMessage()

Delete a given SQS message from a given queue.

Kind: inner method of SQS

SQS~sqsQueueExists()

Test if an SQS queue exists

Kind: inner method of SQS

STS

SecretsManager

StepFunctions

StepFunctions.describeExecution(params) ⇒ Promise.<DescribeExecutionOutput>

Call StepFunctions DescribeExecution

See StepFunctions.describeExecution() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param Type
params DescribeExecutionInput

StepFunctions.describeStateMachine(params) ⇒ Promise.<DescribeStateMachineOutput>

Call StepFunctions DescribeStateMachine

See StepFunctions.describeStateMachine() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param Type
params DescribeStateMachineInput

StepFunctions.getExecutionHistory(params) ⇒ Promise.<GetExecutionHistoryOutput>

Call StepFunctions GetExecutionHistory

See StepFunctions.getExecutionHistory() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param Type
params GetExecutionHistoryInput

StepFunctions.listExecutions(params) ⇒ Promise.<Object>

Call StepFunctions ListExecutions

See StepFunctions.listExecutions() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param Type
params Object

StepFunctions~executionExists(executionArn) ⇒ Promise.<boolean>

Check if a Step Function Execution exists

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: inner method of StepFunctions

Param Type Description
executionArn string the ARN of the Step Function Execution to check for

DynamoDbSearchQueue

Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.

Kind: global class

dynamoDbSearchQueue.empty() ⇒ Promise.<Array>

Drain all values from the searchQueue, and return to the user. Warning: This can be very memory intensive.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Array> - array of search results.

dynamoDbSearchQueue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table

dynamoDbSearchQueue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns 'null'.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table

S3ListObjectsV2Queue

Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.

Kind: global class

s3ListObjectsV2Queue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description

s3ListObjectsV2Queue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns 'null'.

Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description

S3ObjectStore

Class to use when interacting with S3

Kind: global class

s3ObjectStore.signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

Returns an HTTPS URL that can be used to perform a GET on the given object store URL

Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:

  • TypeError - if the URL is not a recognized protocol or cannot be parsed
Param Type Description
objectUrl string the URL of the object to sign
[options] string options to pass to S3.getObject
[queryParams] string a mapping of parameter key/values to put in the URL
presignOptions RequestPresigningArguments presignOptions

s3ObjectStore.signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

Returns an HTTPS URL that can be used to perform a HEAD on the given object store URL

Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:

  • TypeError - if the URL is not a recognized protocol or cannot be parsed
Param Type Description
objectUrl string the URL of the object to sign
[options] string options to pass to S3.getObject
[queryParams] string a mapping of parameter key/values to put in the URL
presignOptions RequestPresigningArguments presignOptions

createKey(params) ⇒ Promise.<Object>

Create a KMS key

See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.

Kind: global function

Param Type
params Object

encrypt(KeyId, Plaintext) ⇒ Promise.<string>

Encrypt a string using KMS

Kind: global function
Returns: Promise.<string> - the Base 64 encoding of the encrypted value

Param Type Description
KeyId string the KMS key to use for encryption
Plaintext string the string to be encrypted

decryptBase64String(ciphertext) ⇒ string

Decrypt a KMS-encrypted string, Base 64 encoded

Kind: global function
Returns: string - the plaintext

Param Type Description
ciphertext string a KMS-encrypted value, Base 64 encoded

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.


Generated automatically using npm run build-docs

Readme

Keywords

Package Sidebar

Install

npm i @cumulus/aws-client

Weekly Downloads

171

Version

20.1.2

License

Apache-2.0

Unpacked Size

196 kB

Total Files

49

Last publish

Collaborators

  • cumuluspublisher
  • nsidc_kovarik
  • jennyhliu
  • npauzenga
  • ppilone84
  • etcart