MemTable
Light weight in memory database with powerful indexing and querying with lodash and highland node compatible streams. A data cache with similar operations to a database but in memory with syncronous calls. Adds unique/non-unique data indexing and data querying using standard lodash (syncronous) and highland (asyncronous streams) on top of standard JS iterators. Use in conjunction with permament data store by watching for data mutations on change handler.
Install
npm install memtable
const Memtable = require('memtable')
Why
Cache your data with powerful indexing and querying capabilities built on JS iterators, lodash and node streams using highland-js. Query data with very little overhead. Memtable supports unique and non unique indexes, secondary and compound indexes. entries. Data is kept internally in native Set and Map structures. Listen to memtable changes to syncronize data back to a persistent data store.
v2.0
Version 2 brings non-unique and required indexes, functional indexes, named indexes and more robust index queries. You now must specify the name of the index, and can provide objects, arrays, functions or strings to query your indexed data. This version uses ES6 iterators and stores data in native Sets and Maps. Event emitter depedency has been removed for a single top level callback to watch for changes on the table. Cache warming is done by passing data through to setSilent to prevent events from triggering rather than passing the data set in on construction. There is also now a pre-save hook to allow for mutations or data validations before saving data.
Basic Usage
const Table = const validateUser = //all options optional const usersTable = //sets user data in table let user =users //returns user, but a version of the user missing unnecessary props user = users //query table by "login" property and return a single result user = users //query table by "email" property and return a single result user = users //returns all users in an iterator const allUsers = users //returns true, since primary id exists in table var exists = users
Restore and Persist data
Basic example of how to "warm" the memtable cache from database, then watch for changes on memtable to send changes back to database.
//assume we have a user model that uses promises to persist data to database var UserDb = var Table = //keep store eventually consistent with memory model //for better consistency, persist data asyncronously, then update memtable on success. { } //pass change handler on init, or call listen(cb) const userCache = //this is equivalent to change handler on init, //only one listener will every be registered though userCache //initialize memtable cache from persistent storage //set silent will prevent change handler from firing during init UserDb
Advanced Indexes
Memtable is pretty flexible with data indexes, this is how to use some of the more advanced features.
//lets define some advanced secondary indexes for a userconst indexes =//compound indexname:'fullname'index:'first''last'required:falseunique:false//uses a function to both validate email address and form it to some//standard representationname:'email'required:true unique:true
Querying
API
Initialization Options
var Table = table =
//default option values options = primary:'id' //default primary key property, objects must at least have this property defined indexes: //list secondary indexable properties as strings, must be unique
Set
Save object to table, or replace existing object. Will trigger onChange callback. Will throw if required properties are not set. Will also throw if unique secondary id collides with another item already in table.
//add a single object, returns the object back. Throws if required properties are not defined. var result = table //add an array of objects, returns the list of objects back var result = table
Update
Update properties on an existing item in the table. Will throw if trying to update primary key(s). Will merge properties into the existing object in the table and trigger onChange callback.
//update item with the primaryid, returns the updated object. var result = table //update item by secondary id, specify the secondary key, secondary id and the updates to make var result = table
Get
Get an object from table. Will throw if object does not exist.
//get object by primary id, returns the stored object. var result = table //get objects by a list of primary ids var result = table //get an object by secondary id, returns stored object. var result = table //get objects by a list of secondary ids var result = table //get object by composite id. Must specify composite id as an array of values //Must specify the values to search as an array in the same order as the ids var result = table //get a list of objects by composit ids. specify array of composite values. var result = table
Has
Check if an object exists in the table. Will never throw. returns only true or false for each object checked.
//"has" will always return true or false, will not throw //result is true if object with id "primaryid" exists var result = table //check if object exists by secondary unique property, will not throw //result is true if object with secondary id "secondaryid" exists var result = table //check if list of ids exist, will not throw //result is array of true/false values var result = table //check if list of unique secondary ids exist, will not throw //result is array of true/false values var result = table //check if object exists by composite id var result = table //check if a list of objects exist by composite ids var result = table
Remove
Remove an object from memory and trigger onRemove callback. Throws if object does not exist. Each remove returns the object removed.
//remove object by primary id var result = table //remove objects by list of primary ids var result = table //remove object by secondary id var result = table //remove objects by list of secondaryids var result = table //remove object by composite id. Must specify composite id as an array of values var result = table //remove objects by composite ids. specify array of array of composite values. var result = table
Map, Filter, Reduce, Each, Sort
Internally uses lodash's "map", "filter", "reduce" and "each" functions over the entire table. You can accomplish the same thing with "list" getting all the data as an array, but this saves you an iteration over the table. Order of iteration is not gauranteed.
//filter over all items in table, returns an array of items which filter returned true for //see lodash filter var result = table //map over all items in table, returns an array of mapped values //see lodash map var result = table //reduce over all items in table, with optional second paramter to seed the reduce //see lodash reduce var result = table //iterate over all items in table, for side effects. // see lodash each table //returns sorted array where you specify the properties to sort by //specify 'asc' or 'desc' to change sort order, default is ascending //see lodash orderBy var result = table
Search
Search over specified "searchable" properties in each item in the table. Case sensitive or insensitive. Partial matches returned. For custom search use table.filter and define your own.
//specify partial search and filters all objects which match. Must have specified them as "searchable" in initialization properties. var result = table
Lodash Sequence
Wrap table in a lodash sequence, letting you chain commands. Must call value at end to get results. Syncronous operations. See lodash
//wraps just table values var result = tablevalue //if you want primary keys and values use this //which will wrap they primary table object and return value,key var result = tablevalue
Highland Streams
Wrap the table in a highland stream, which gives you access to the node stream api as well as highlands api. Asyncronous operations. See highland
//highland is a node compatible stream which emits values from the table one by one table //if you want key value pairs do this: table
Drop
Clear the table. Does not return anything
table
List
Get an array of all objects in the table.
var result = table
State
Get the entire table state as an object. Will include all secondary and composite indexes.
//returns an object which represents the table in memory var result = table