LinvoDB
LinvoDB is a Node.js/NW.js/Electron persistent DB with MongoDB / Mongoose-like features and interface.
Features:
- MongoDB-like query language
- Persistence built on LevelUP - you can pick back-end
- NW.js/Electron friendly - JS-only backend is level-js or Medea
- Performant - steady performance unaffected by DB size - queries are always indexed
- Auto-indexing
- Live queries - make the query, get constantly up-to-date results
- Schemas - built-in schema support
- Efficient Map / Reduce / Limit
Coming soon:
- Streaming cursors
- Distributed dataset
Relationship to NeDB
LinvoDB is based on NeDB, the most significant core change is that it uses LevelUP as a back-end, meaning it doesn't have to keep the whole dataset in memory. LinvoDB also can do a query entirely by indexes, meaning it doesn't have to scan the full database on a query.
In general:
- LinvoDB is better for large datasets (many objects, or large objects) because it doesn't keep the whole DB in memory and doesn't need to always scan it
- LinvoDB does the entire query through the indexes, NeDB scans the DB
- Both LinvoDB and NeDB play well with NW.js (node-webkit). LinvoDB can be initialized with the JS-only level-js back-end.
- NeDB is ultra-fast because the DB is in memory, LinvoDB's performance is comparible to MongoDB. LinvoDB is faster for large datasets.
- LinvoDB has live queries, map/reduce and schema support.
- Both LinvoDB and NeDB are unsuitable for huge datasets (big data)
- Combining NeDB's in-memory data and LinvoDB's full-indexed queries would yield even better performance. If you want to sacrifice memory for query performance, you can use LinvoDB with a backend that works like that or with LevelDB + increased LRU cache
Install, Initialize, pick backend
Install:
npm install linvodb3 level-js # For NWjs using level-jsnpm install linvodb3 leveldown # For pure nodejs using LevelDB
Initialize:
var LinvoDB = ; // The following two lines are very important// Initialize the default store to level-js - which is a JS-only store which will work without recompiling in NW.js / ElectronLinvoDBdefaultsstore = db: ; // Comment out to use LevelDB instead of level-js// Set dbPath - this should be done explicitly and will be the dir where each model's store is savedLinvoDBdbPath = process; var Doc = "doc" /* schema, can be empty */
Initialization, detailed:
var LinvoDB = ;var modelName = "doc";var schema = ; // Non-strict always, can be left emptyvar options = ;// options.filename = "./test.db"; // Path to database - not necessary // options.store = { db: require("level-js") }; // Options passed to LevelUP constructor var Doc = modelName schema options; // New model; Doc is the constructor LinvoDBdbPath // default path where data files are stored for each modelLinvoDBdefaults // default options for every model
Insert / Save
The native types are String
, Number
, Boolean
, Date
and null
. You can also use
arrays and subdocuments (objects). If a field is undefined
, it will not be saved.
If the document does not contain an _id
field, one will be automatically generated (a 16-characters alphanumerical string). The _id
of a document, once set, cannot be modified.
// Construct a single document and then save itvar doc = a: 5 now: test: "this is a string" ;docb = 13; // you can modify the doc doc; // Insert document(s)// you can use the .insert method to insert one or more documentsDoc;Doc; // Save document(s)// save is like an insert, except it allows saving existing document tooDoc;
Querying
Use find
to look for multiple documents matching you query, or findOne
to look for one specific document. You can select documents based on field equality or use comparison operators ($lt
, $lte
, $gt
, $gte
, $in
, $nin
, $ne
, $regex
, $exists
). You can also use logical operators $or
, $and
and $not
. See below for the syntax.
var Planet = "planet" /* schema, can be empty */ // Let's say our datastore contains the following collectionPlanet; // end of .save()
Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)
The syntax is { field: { $op: value } }
where $op
is any comparison operator:
$lt
,$lte
: less than, less than or equal$gt
,$gte
: greater than, greater than or equal$in
: member of.value
must be an array of values$ne
,$nin
: not equal, not a member of$exists
: checks whether the document posses the propertyfield
.value
should be true or false$regex
: checks whether a string is matched by the regular expression. Contrary to MongoDB, the use of$options
with$regex
is not supported, because it doesn't give you more power than regex flags. Basic queries are more readable so only use the$regex
operator when you need to use another operator with it (see example below)
// $lt, $lte, $gt and $gte work on numbers and stringsPlanet; // When used with strings, lexicographical order is usedPlanet // Using $in. $nin is used in the same wayPlanet; // Using $existsPlanet; // Using $regex with another operatorPlanet;
Array fields
When a field in a document is an array the query is treated as a query on every element and there is a match if at least one element matches.
// If a document's field is an array, matching it means matching any element of the arrayPlanet; // This also works for queries that use comparison operatorsPlanet; // This also works with the $in and $nin operatorPlanet;
Logical operators $or, $and, $not
You can combine queries using logical operators:
- For
$or
and$and
, the syntax is{ $op: [query1, query2, ...] }
. - For
$not
, the syntax is{ $not: query }
Planet; Planet; // You can mix normal queries, comparison queries and logical operatorsPlanet;
Sorting and paginating
If you don't specify a callback to find
, findOne
or count
, a Cursor
object is returned. You can modify the cursor with sort
, skip
and limit
and then execute it with exec(callback)
.
var Planet = "planet" /* schema, can be empty */ var doc1doc2doc3doc4; Planet; // end of .save
Counting documents
You can use count
to count documents. It has the same syntax as find
. For example:
// Count all planets in the solar systemPlanet; // Count all documents via cursorPlanet;
Map / Reduce / Filter / Aggregate
Besides the standard pagination and sorting Cursor methods, we have the filter
, map
and reduce
modifiers.
Before seeing the examples, you should know that you can combine any of these modifiers in any order/way and all will be executed. For example, you can run a regular query with .find and then run a reduce on it.
No matter how you combine those modifiers, the order of execution is: query, filter, sort, limit/skip, map, reduce, aggregate.
The basic syntax is:
Cursor.map(function(val){ return val })
Cursor.reduce(function reducer(a,b), initial);
Cursor.filter(function(val) { return true /* or false*/ }); // truthy / falsy values accepted
Cursor.aggregate(function(res) { /* do something to the result of the query right before serving */ return res })
// Let's assume this datasetvar Planet = "planet" /* schema, can be empty */ Planet; // end .save()
Live Queries
Once you have a Cursor
object, returned by calling find
without a callback, you can turn it into a live query, meaning the .res
property will always be up-to-date results from the query. Of course, all modifiers, such as limit
, skip
, sort
, map
, reduce
, filter
and aggregate
will still apply.
An event will be emitted when the result is updated - liveQueryUpdate
on the model itself.
Seriously consider if live queries can be utilized in your application - if you need particular results continuously, using live queries is extremely efficient, since you don't have to re-read the database but results are kept up-to-date as you update the documents.
// Let's assume this datasetvar Planet = "planet" /* schema, can be empty */ Planet; // end .save()
Angular Disclaimer
If you plan to use Live Queries with AngularJS and update scope on the liveQueryUpdated
event please be careful. First, I recommend using $digest
when possible instead of $apply
(dirty-check only the current scope). Second, I recommend debouncing the event before running the $scope.$apply()
event to avoid $apply being called many times because of heavy DB use at a moment.
Updating
Re-saving a document
doc.save()
- you can use save
on a document instance to re-save it, therefore updating it.
// Let's use the same example collection as in the "finding document" part// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false }// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true }// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }// { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true } Planet;
Atomic updating
Doc.update(query, update, options, callback)
will update all documents matching query
according to the update
rules:
query
is the same kind of finding query you use withfind
andfindOne
update
specifies how the documents should be modified. It is either a new document or a set of modifiers (you cannot use both together, it doesn't make sense!)- A new document will replace the matched docs
- The modifiers create the fields they need to modify if they don't exist, and you can apply them to subdocs. Available field modifiers are
$set
to change a field's value,$unset
to delete a field and$inc
to increment a field's value. To work on arrays, you have$push
,$pop
,$addToSet
,$pull
, and the special$each
. See examples below for the syntax.
options
is an object with two possible parametersmulti
(defaults tofalse
) which allows the modification of several documents if set to trueupsert
(defaults tofalse
) if you want to insert a new document corresponding to theupdate
rules if yourquery
doesn't match anything. If yourupdate
is a simple object with no modifiers, it is the inserted document. In the other case, thequery
is stripped from all operator recursively, and theupdate
is applied to it.
callback
(optional) signature:err
,numReplaced
,newDoc
numReplaced
is the number of documents replacednewDoc
is the created document if the upsert mode was chosen and a document was inserted
Note: you can't change a document's _id.
// Let's use the same example collection as in the "finding document" part// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false }// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true }// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }// { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true } // Replace a document by anotherPlanet; // Set an existing field's valuePlanet; // Setting the value of a non-existing field in a subdocument by using the dot-notationPlanet; // Deleting a fieldPlanet; // Upserting a documentPlanet; // If you upsert with a modifier, the upserted doc is the query modified by the modifier// This is simpler than it sounds :)Planet; // If we insert a new document { _id: 'id6', fruits: ['apple', 'orange', 'pear'] } in the collection,// let's see how we can modify the array field atomically // $push inserts new elements at the end of the arrayPlanet; // $pop removes an element from the end (if used with 1) or the front (if used with -1) of the arrayPlanet; // $addToSet adds an element to an array only if it isn't already in it// Equality is deep-checked (i.e. $addToSet will not insert an object in an array already containing the same object)// Note that it doesn't check whether the array contained duplicates before or notPlanet; // $pull removes all values matching a value or even any query from the arrayPlanet;Planet; // $each can be used to $push or $addToSet multiple values at once// This example works the same way with $addToSetPlanet;
Removing
Removing a document instance
// if you have the document instance at hand, you can justDoc;
Removing from the collection
Doc.remove(query, options, callback)
will remove all documents matching query
according to options
query
is the same as the ones used for finding and updatingoptions
only one option for now:multi
which allows the removal of multiple documents if set to true. Default is falsecallback
is optional, signature: err, numRemoved
// Let's use the same example collection as in the "finding document" part// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false }// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true }// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }// { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true } // Remove one document from the collection// options set to {} since the default for multi is falsePlanet; // Remove multiple documentsPlanet;
Events
// Hook-likeDoc // Will be called before saving a document - no matter if using save, insert or update methods. You can modify the document in this event, it's essentially a hookDoc // Will be called before saving a new document - again, no matter if using save/insert/update methods. You can modify the document in this eventDoc // Before removing a document; called with the document about to be removed Doc // When a document is constructed // After operation is completeDoc // Called after inserting new documents is complete; docs is an array of documentsDoc // Called after updating documents is complete; docs is an array of documentsDoc // Called after removing documents is complete; ids is an array of ids
Schemas
You can define a schema for a model, allowing you to enforce certain properties to types (String, Number, Date), set defaults and also define properties with getter/setter. Since schema support is implemented deep in LinvoDB, you can query on fields which are getter/setter-based and rely that types/defaults are always going to be enforced.
NOTE: when constructing a model with a schema, please specify options object after the schema, otherwise schema will be treated as options: new LinvoDB(name, schema, options)
Schemas are defined as an object of specs for each property. The spec can have properties:
type
- the type to be enforced, can be String, Number, Date along with "string", "number", "date" alternative syntax. Can also be a RegExp instance in case you want to validate against that expression.default
- the default value; must comply to the type obviouslyenumerable
- whether this property will be enumerableget
- getter, cannot be used with type/defaultset
- setter, cannot be used with type/defaultindex
,sparse
,unique
- booleans, whether to create an index and it's options
If type is all you need, you can shorthand the property to the type only, e.g. { name: String }
.
You can also define a property as an "array of" by setting it to [spec]
, for example [String]
for an array of strings.
Nested objects are supported.
var Person = "person" name: type: String default: "nameless" // default value age: Number // shorthand to { type: ... } created: Date address: // nested object line1: String line2: String department: type: String index: true // you can use the schema spec to define indexes favNumbers: Number // array of firstName: { return thisname0 } ; var p = ;// p is { name: 'nameless', age: 0, created: /* date when created */, address: { line1: "", line2: "" }, favNumbers: [] } pname = 23;// p.name becomes "23" pcreated = "10/23/2004"; // p is 23 October 2004, date object pfavNumbers;pfavNumbers; // favNumbers will be [22, 42] ; the string will be cast to a numberpfavNumbers; // nothing happens, can't cast// p.favNumbers is [22, 42] pname = "John Smith"; // p.firstName is "John" p;
Model - static & instance methods
// var doc = new Doc(); // create a new instance// Or get it from query results docdocdoc; // returns a copy of the document
You can define additional functions for both the model and the document instances.
Planet;Planet; Planetmethod"findSameSystem" { return Planet };Planet;
Indexing
Indexing in LinvoDB is automatic, although you can turn that off ({autoindex: false}
in model options, not recommended). Defining indexes, in case you need you enforce a unique constraint, happens with Doc.ensureIndex({ fieldName: "name", unique: true })
.
The full syntax is Doc.ensureIndex(options, cb)
, where callback is optional and get passed an error if any (usually a unique constraint that was violated). ensureIndex
can be called when you want, even after some data was inserted, though it's best to call it at application startup. The options are:
- fieldName (required): name of the field to index. Use the dot notation to index a field in a nested document.
- unique (optional, defaults to
false
): enforce field uniqueness. Note that a unique index will raise an error if you try to index two documents for which the field is not defined. - sparse (optional, defaults to
false
): don't index documents for which the field is not defined. Use this option along with "unique" if you want to accept multiple documents for which it is not defined.
You can remove a previously created index with Doc.removeIndex(fieldName, cb)
.
NOTE compound indexes are currently not supported.
Promises with Bluebird
Even though LinvoDB does not support Promises out-of-the-box, it can easily be made promise-friendly using Bluebird's promisification feature:
var LinvoDB = ;var Promise = ; var Planet = 'planet' {}; Promise;// As of this line, LinvoDB APIs now have promise-returning methods with *Async suffix.// All the callback-based APIs are still there and will work as before. Planet; // or, if you use ES7 async / await: try var docs = await Planet; // use docs somehow catch err // handle errors
Utilization
Stremio - LinvoDB was created specifically because NeDB started to behave suboptimally with >300 movie/series metadata objects, which were pretty large. Reduced memory usage from ~500MB to ~100MB. Live queries, schemas and map/reduce helped create a much cleaner codebase.
If you wish to add something here, contact me at ivo@linvo.com
License
See License
Help
Pull requests are always welcome. :)