dynamodb-spec-generator

A command line tool to generate documentation for a DynamoDB table and its access patterns, to assign in the design phase of a well-factored DynamoDB table

Purpose
Installaton
Usage
Examples
- Relational Modeling

Purpose

Designing a DynamoDB table is very different than a SQL database because one must decide upfront what access patterns the table needs to support and pre-denormalize the data.

Additionally, a well-factored DynamoDB table, with overloading GSIs, sparse indexes, local secondary indexes, composite keys (etc.) can be quite hard to document and understand.

dynamodb-spec-generator was created to help alleviate this issue, to make it easier to design a DynamoDB according to the access patterns. All you have to do is provide it with a valid DynamoDB spec file and it will output a markdown document.

Installation

dynamodb-spec-generator is packaged as an npm package and requires node >= 8.0.0:

# install the command globally 
$ npm install dynamodb-spec-generator -g

Alternatively, if you are working inside a nodejs project, you can add it as a dependency:

# Install it as a development dependency 
$ npm install dynamodb-spec-generator --save-dev

If you install it as a dependency, make sure to run it with npx like so:

$ npx dynamodb-spec-generator

Usage

dynamodb-spec-generator takes a single argument, the path to the spec file, and will output the markdown to stdout:

$ npx dynamodb-spec-generator spec.json

If you want to instead save it to a file, pass the -o flag:

$ npx dynamodb-spec-generator spec.json -o README.md

You can also have it watch for changes to the spec file and regenerate the markdown whenever you make a change by passing the --watch flag:

$ npx dynamodb-spec-generator spec.json -o README.md --watch

Examples

Relational Modeling

This example attempts to recreate the DynamoDB docs for converting a normalized SQL relational database into a single DynamoDB table with two Global Secondary Indexes.

The spec file relationalModeling.json

{
  "service": "HR Api Backend",
  "version": "1.0.0",
  "description": "A recreation of the Relational Modeling example in the DynamoDB Docs",
  "author": "Eric Allam",
  "tableDefinition": {
    "TableName": "HR-Table",
    "KeySchema": [
      {
        "AttributeName": "PK",
        "KeyType": "HASH"
      },
      {
        "AttributeName": "SK",
        "KeyType": "RANGE"
      }
    ],
    "AttributeDefinitions": [
      {
        "AttributeName": "PK",
        "AttributeType": "S"
      },
      {
        "AttributeName": "SK",
        "AttributeType": "S"
      },
      {
        "AttributeName": "Data",
        "AttributeType": "S"
      }
    ],
    "ProvisionedThroughput": {
      "ReadCapacityUnits": 5,
      "WriteCapacityUnits": 5
    },
    "BillingMode": "PAY_PER_REQUEST",
    "GlobalSecondaryIndexes": [
      {
        "IndexName": "gsi1",
        "KeySchema": [
          {
            "AttributeName": "SK",
            "KeyType": "HASH"
          },
          {
            "AttributeName": "Data",
            "KeyType": "RANGE"
          }
        ],
        "Projection": {
          "ProjectionType": "ALL"
        },
        "ProvisionedThroughput": {
          "ReadCapacityUnits": 5,
          "WriteCapacityUnits": 5
        }
      },
      {
        "IndexName": "gsi2",
        "KeySchema": [
          {
            "AttributeName": "GSI-Bucket",
            "KeyType": "HASH"
          },
          {
            "AttributeName": "Data",
            "KeyType": "RANGE"
          }
        ],
        "Projection": {
          "ProjectionType": "ALL"
        },
        "ProvisionedThroughput": {
          "ReadCapacityUnits": 5,
          "WriteCapacityUnits": 5
        }
      }
    ]
  },
  "accessPatterns": [
    {
      "title": "Look up Employee Details by Employee ID",
      "type": "get",
      "index": "main",
      "params": {
        "partition": "HR-EMPLOYEE1",
        "sort": { "value": "EMPLOYEE1", "operator": "=" }
      }
    },
    {
      "title": "Query Employee Details by Employee Name",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": { "value": "EMPLOYEE1" },
        "sort": { "value": "John", "operator": "begins_with" }
      }
    },
    {
      "title": "Get an employee's current job details only",
      "index": "main",
      "type": "query",
      "params": {
        "partition": { "value": "HR-EMPLOYEE1" },
        "sort": { "value": "v0", "operator": "begins_with" }
      }
    },
    {
      "title": "Get Open Orders for a customer for a date range",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": { "value": "CUSTOMER1" },
        "sort": { "value": "OPEN#2019-01", "operator": "begins_with" }
      }
    },
    {
      "title": "Show all Orders in OPEN status for a date range across all customers",
      "description": "Query in parallel for the range [0..N] to get all shards",
      "type": "query",
      "index": "gsi2",
      "params": {
        "partition": "Bucket-6",
        "sort": {
          "operator": "between",
          "minValue": "OPEN#2019-01",
          "maxValue": "OPEN#2019-02"
        }
      }
    },
    {
      "title": "All Employees Hired Recently",
      "index": "gsi1",
      "type": "query",
      "params": {
        "partition": "HR-CONFIDENTIAL",
        "sort": { "value": "2019-01-01", "operator": ">" }
      }
    },
    {
      "title": "Find all employees in a certain warehouse",
      "index": "gsi1",
      "type": "query",
      "params": {
        "partition": "WAREHOUSE1"
      }
    },
    {
      "title": "Get all OrderItems for a Product including warehouse location inventories",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": "PRODUCT1",
        "filters": [
          {
            "attribute": "Warehouse1",
            "operator": "attribute_exists"
          }
        ]
      }
    },
    {
      "title": "Get customers by Account Rep",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": "EMPLOYEE1",
        "filters": [
          {
            "attribute": "Address",
            "operator": "attribute_exists"
          }
        ]
      }
    },
    {
      "title": "Get orders by Account Rep and date",
      "description": "Scatter/Gather to query all statuses (OPEN, PENDING, FULFILLED)",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": "EMPLOYEE1",
        "sort": { "value": "OPEN#2019-01-12", "operator": "=" }
      }
    },
    {
      "title": "Get all employees with specific Job Title",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": "v0"
      }
    },
    {
      "title": "Get inventory by Product and Warehouse",
      "type": "get",
      "index": "main",
      "params": {
        "partition": "OE-PRODUCT1",
        "sort": { "value": "PRODUCT1", "operator": "=" }
      }
    },
    {
      "title": "Get total product inventory",
      "type": "get",
      "index": "main",
      "params": {
        "partition": "OE-PRODUCT1",
        "sort": {
          "value": "PRODUCT1",
          "operator": "="
        }
      }
    },
    {
      "title": "Get Account Reps ranked by Order Total and Sales Period",
      "type": "query",
      "index": "gsi1",
      "params": {
        "partition": "2018-Q4",
        "order": "DESC"
      }
    }
  ],
  "records": [
    {
      "PK": "HR-EMPLOYEE1",
      "SK": "EMPLOYEE1",
      "Data": "John Smith",
      "StartDate": "01-12-2019"
    },
    { "PK": "HR-EMPLOYEE1", "SK": "v0", "Data": "Principle Account Manager" },
    {
      "PK": "HR-EMPLOYEE1",
      "SK": "HR-CONFIDENTIAL",
      "Data": "2019-02-12",
      "Employee": "John Smith",
      "Salary": 50000
    },
    {
      "PK": "HR-EMPLOYEE1",
      "SK": "WAREHOUSE1",
      "Data": "2019-02-15",
      "Employee Name": "John Smith"
    },
    {
      "PK": "HR-EMPLOYEE1",
      "SK": "2018-Q4",
      "Data": "5000",
      "Employee Name": "John Smith"
    },
    {
      "PK": "HR-EMPLOYEE2",
      "SK": "2018-Q4",
      "Data": "10000",
      "Employee Name": "John Smith"
    },
    {
      "PK": "OE-ORDER1",
      "SK": "CUSTOMER1",
      "Data": "OPEN#2019-01-18",
      "GSI-Bucket": "Bucket-6"
    },
    {
      "PK": "OE-ORDER1",
      "SK": "PRODUCT1",
      "Data": "OPEN#2019-01-18",
      "GSI-Bucket": "Bucket-4",
      "UnitPrice": "$89.99"
    },
    {
      "PK": "OE-ORDER1",
      "SK": "EMPLOYEE1",
      "Data": "OPEN#2019-01-12",
      "OrderTotal": 2500
    },
    {
      "PK": "OE-PRODUCT1",
      "SK": "PRODUCT1",
      "Data": "Quickcrete Cement - 50lb bag",
      "Warehouse1": 46,
      "Warehouse2": 12
    },
    {
      "PK": "OE-CUSTOMER1",
      "SK": "CUSTOMER1",
      "Data": "Ace Building Supplies",
      "Address": "1600 Penn"
    },
    {
      "PK": "OE-CUSTOMER1",
      "SK": "EMPLOYEE1",
      "Data": "Ace Building Supplies",
      "Address": "1600 Penn"
    }
  ]
}

Will generate this markdown:

HR Api Backend DynamoDB Spec

A recreation of the Relational Modeling example in the DynamoDB Docs

Table Spec
Access Patterns
Indexes
- Main
- Gsi1
- Gsi2
Author

Table Spec

Params to create the table using the CLI or the AWS SDK:

{
  "TableName": "HR-Table",
  "KeySchema": [
    {
      "AttributeName": "PK",
      "KeyType": "HASH"
    },
    {
      "AttributeName": "SK",
      "KeyType": "RANGE"
    }
  ],
  "AttributeDefinitions": [
    {
      "AttributeName": "PK",
      "AttributeType": "S"
    },
    {
      "AttributeName": "SK",
      "AttributeType": "S"
    },
    {
      "AttributeName": "Data",
      "AttributeType": "S"
    }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 5,
    "WriteCapacityUnits": 5
  },
  "BillingMode": "PAY_PER_REQUEST",
  "GlobalSecondaryIndexes": [
    {
      "IndexName": "gsi1",
      "KeySchema": [
        {
          "AttributeName": "SK",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "Data",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 5,
        "WriteCapacityUnits": 5
      }
    },
    {
      "IndexName": "gsi2",
      "KeySchema": [
        {
          "AttributeName": "GSI-Bucket",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "Data",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 5,
        "WriteCapacityUnits": 5
      }
    }
  ]
}

createTable Using the CLI:

$ aws dynamodb create-table --table-name HR-Table --cli-input-json create-table.json

Using the AWS SDK:

const DynamoDB = require("aws-sdk/clients/dynamodb");
 
const service = new DynamoDB({ region: process.env.AWS_REGION });
 
service.createTable(tableJson, (err, data) => {
  console.log(data);
});

Access Patterns

Look up Employee Details by Employee ID

Perform a DocumentClient.get against the Main index:

{
  "TableName": "HR-Table",
  "Key": {
    "PK": "HR-EMPLOYEE1",
    "SK": "EMPLOYEE1"
  }
}

Matching Records

PK (HASH)	SK (RANGE)
HR-EMPLOYEE1	EMPLOYEE1	Data: John Smith	StartDate: 01-12-2019

Query Employee Details by Employee Name

Perform a DocumentClient.query against the Gsi1 index with a begins_with(#Data, :Data) condition on the sort key:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK and begins_with(#Data, :Data)",
  "ExpressionAttributeNames": {
    "#SK": "SK",
    "#Data": "Data"
  },
  "ExpressionAttributeValues": {
    ":SK": "EMPLOYEE1",
    ":Data": "John"
  },
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
EMPLOYEE1	John Smith	PK: HR-EMPLOYEE1	StartDate: 01-12-2019

Get an employee's current job details only

Perform a DocumentClient.query against the Main index with a begins_with(#SK, :SK) condition on the sort key:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#PK = :PK and begins_with(#SK, :SK)",
  "ExpressionAttributeNames": {
    "#PK": "PK",
    "#SK": "SK"
  },
  "ExpressionAttributeValues": {
    ":PK": "HR-EMPLOYEE1",
    ":SK": "v0"
  }
}

Matching Records

PK (HASH)	SK (RANGE)
HR-EMPLOYEE1	v0	Data: Principle Account Manager

Get Open Orders for a customer for a date range

Perform a DocumentClient.query against the Gsi1 index with a begins_with(#Data, :Data) condition on the sort key:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK and begins_with(#Data, :Data)",
  "ExpressionAttributeNames": {
    "#SK": "SK",
    "#Data": "Data"
  },
  "ExpressionAttributeValues": {
    ":SK": "CUSTOMER1",
    ":Data": "OPEN#2019-01"
  },
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
CUSTOMER1	OPEN#2019-01-18	PK: OE-ORDER1	GSI-Bucket: Bucket-6

Show all Orders in OPEN status for a date range across all customers

Query in parallel for the range [0..N] to get all shards

Perform a DocumentClient.query against the Gsi2 index with a #Data BETWEEN :DataMin AND :DataMax condition on the sort key:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#GSI-Bucket = :GSI-Bucket and #Data BETWEEN :DataMin AND :DataMax",
  "ExpressionAttributeNames": {
    "#GSI-Bucket": "GSI-Bucket",
    "#Data": "Data"
  },
  "ExpressionAttributeValues": {
    ":GSI-Bucket": "Bucket-6",
    ":DataMin": "OPEN#2019-01",
    ":DataMax": "OPEN#2019-02"
  },
  "IndexName": "gsi2"
}

Matching Records

GSI-Bucket (HASH)	Data (RANGE)
Bucket-6	OPEN#2019-01-18	PK: OE-ORDER1	SK: CUSTOMER1

All Employees Hired Recently

Perform a DocumentClient.query against the Gsi1 index with a #Data > :Data condition on the sort key:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK and #Data > :Data",
  "ExpressionAttributeNames": {
    "#SK": "SK",
    "#Data": "Data"
  },
  "ExpressionAttributeValues": {
    ":SK": "HR-CONFIDENTIAL",
    ":Data": "2019-01-01"
  },
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
HR-CONFIDENTIAL	2019-02-12	PK: HR-EMPLOYEE1	Employee: John Smith	Salary: 50000

Find all employees in a certain warehouse

Perform a DocumentClient.query against the Gsi1 index:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK",
  "ExpressionAttributeNames": {
    "#SK": "SK"
  },
  "ExpressionAttributeValues": {
    ":SK": "WAREHOUSE1"
  },
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
WAREHOUSE1	2019-02-15	PK: HR-EMPLOYEE1	Employee Name: John Smith

Get all OrderItems for a Product including warehouse location inventories

Perform a DocumentClient.query against the Gsi1 index with filter params:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK",
  "ExpressionAttributeNames": {
    "#SK": "SK",
    "#Warehouse1": "Warehouse1"
  },
  "ExpressionAttributeValues": {
    ":SK": "PRODUCT1"
  },
  "IndexName": "gsi1",
  "FilterExpression": "attribute_exists(#Warehouse1)"
}

Matching Records

SK (HASH)	Data (RANGE)
PRODUCT1	Quickcrete Cement - 50lb bag	PK: OE-PRODUCT1	Warehouse1: 46	Warehouse2: 12

Get customers by Account Rep

Perform a DocumentClient.query against the Gsi1 index with filter params:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK",
  "ExpressionAttributeNames": {
    "#SK": "SK",
    "#Address": "Address"
  },
  "ExpressionAttributeValues": {
    ":SK": "EMPLOYEE1"
  },
  "IndexName": "gsi1",
  "FilterExpression": "attribute_exists(#Address)"
}

Matching Records

SK (HASH)	Data (RANGE)
EMPLOYEE1	Ace Building Supplies	PK: OE-CUSTOMER1	Address: 1600 Penn

Get orders by Account Rep and date

Scatter/Gather to query all statuses (OPEN, PENDING, FULFILLED)

Perform a DocumentClient.query against the Gsi1 index with a #Data = :Data condition on the sort key:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK and #Data = :Data",
  "ExpressionAttributeNames": {
    "#SK": "SK",
    "#Data": "Data"
  },
  "ExpressionAttributeValues": {
    ":SK": "EMPLOYEE1",
    ":Data": "OPEN#2019-01-12"
  },
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
EMPLOYEE1	OPEN#2019-01-12	PK: OE-ORDER1	OrderTotal: 2500

Get all employees with specific Job Title

Perform a DocumentClient.query against the Gsi1 index:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK",
  "ExpressionAttributeNames": {
    "#SK": "SK"
  },
  "ExpressionAttributeValues": {
    ":SK": "v0"
  },
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
v0	Principle Account Manager	PK: HR-EMPLOYEE1

Get inventory by Product and Warehouse

Perform a DocumentClient.get against the Main index:

{
  "TableName": "HR-Table",
  "Key": {
    "PK": "OE-PRODUCT1",
    "SK": "PRODUCT1"
  }
}

Matching Records

PK (HASH)	SK (RANGE)
OE-PRODUCT1	PRODUCT1	Data: Quickcrete Cement - 50lb bag	Warehouse1: 46	Warehouse2: 12

Get total product inventory

Perform a DocumentClient.get against the Main index:

{
  "TableName": "HR-Table",
  "Key": {
    "PK": "OE-PRODUCT1",
    "SK": "PRODUCT1"
  }
}

Matching Records

PK (HASH)	SK (RANGE)
OE-PRODUCT1	PRODUCT1	Data: Quickcrete Cement - 50lb bag	Warehouse1: 46	Warehouse2: 12

Get Account Reps ranked by Order Total and Sales Period

Perform a DocumentClient.query against the Gsi1 index:

{
  "TableName": "HR-Table",
  "KeyConditionExpression": "#SK = :SK",
  "ExpressionAttributeNames": {
    "#SK": "SK"
  },
  "ExpressionAttributeValues": {
    ":SK": "2018-Q4"
  },
  "ScanIndexFoward": false,
  "IndexName": "gsi1"
}

Matching Records

SK (HASH)	Data (RANGE)
2018-Q4	10000	PK: HR-EMPLOYEE2	Employee Name: John Smith
2018-Q4	5000	PK: HR-EMPLOYEE1	Employee Name: John Smith

Indexes

Main

PK (HASH)	SK (RANGE)
HR-EMPLOYEE1	EMPLOYEE1	Data: John Smith	StartDate: 01-12-2019
HR-EMPLOYEE1	v0	Data: Principle Account Manager
HR-EMPLOYEE1	HR-CONFIDENTIAL	Data: 2019-02-12	Employee: John Smith	Salary: 50000
HR-EMPLOYEE1	WAREHOUSE1	Data: 2019-02-15	Employee Name: John Smith
HR-EMPLOYEE1	2018-Q4	Data: 5000	Employee Name: John Smith
HR-EMPLOYEE2	2018-Q4	Data: 10000	Employee Name: John Smith
OE-ORDER1	CUSTOMER1	Data: OPEN#2019-01-18	GSI-Bucket: Bucket-6
OE-ORDER1	PRODUCT1	Data: OPEN#2019-01-18	GSI-Bucket: Bucket-4	UnitPrice: $89.99
OE-ORDER1	EMPLOYEE1	Data: OPEN#2019-01-12	OrderTotal: 2500
OE-PRODUCT1	PRODUCT1	Data: Quickcrete Cement - 50lb bag	Warehouse1: 46	Warehouse2: 12
OE-CUSTOMER1	CUSTOMER1	Data: Ace Building Supplies	Address: 1600 Penn
OE-CUSTOMER1	EMPLOYEE1	Data: Ace Building Supplies	Address: 1600 Penn

Gsi1

SK (HASH)	Data (RANGE)
EMPLOYEE1	John Smith	PK: HR-EMPLOYEE1	StartDate: 01-12-2019
v0	Principle Account Manager	PK: HR-EMPLOYEE1
HR-CONFIDENTIAL	2019-02-12	PK: HR-EMPLOYEE1	Employee: John Smith	Salary: 50000
WAREHOUSE1	2019-02-15	PK: HR-EMPLOYEE1	Employee Name: John Smith
2018-Q4	5000	PK: HR-EMPLOYEE1	Employee Name: John Smith
2018-Q4	10000	PK: HR-EMPLOYEE2	Employee Name: John Smith
CUSTOMER1	OPEN#2019-01-18	PK: OE-ORDER1	GSI-Bucket: Bucket-6
PRODUCT1	OPEN#2019-01-18	PK: OE-ORDER1	GSI-Bucket: Bucket-4	UnitPrice: $89.99
EMPLOYEE1	OPEN#2019-01-12	PK: OE-ORDER1	OrderTotal: 2500
PRODUCT1	Quickcrete Cement - 50lb bag	PK: OE-PRODUCT1	Warehouse1: 46	Warehouse2: 12
CUSTOMER1	Ace Building Supplies	PK: OE-CUSTOMER1	Address: 1600 Penn
EMPLOYEE1	Ace Building Supplies	PK: OE-CUSTOMER1	Address: 1600 Penn

Gsi2

GSI-Bucket (HASH)	Data (RANGE)
Bucket-6	OPEN#2019-01-18	PK: OE-ORDER1	SK: CUSTOMER1
Bucket-4	OPEN#2019-01-18	PK: OE-ORDER1	SK: PRODUCT1	UnitPrice: $89.99

Author

Spec authored by Eric Allam and generated by dynamodb-spec-generator

DynamoDB Spec

TODO

Ability to use a live table
Code generation
Link to the Indexes in the Access Pattern descriptions
Query counts
Query projection
Query attributes to get
Consistent reads
Shards
Scatter/Gather access patterns
Pagination
Local Secondary Indexes
TTL attribute
Updates/Puts
Embed the spec json in the generated markdown as a comment
Ability to group access patterns by section
Add more information about each index instead of just showing records