gstore cache
Advanced Cache Layer for the Google Datastore Node.js API
gstore cache speeds up your Datastore entities fetching by providing an advanced cache layer
for the @google-cloud/datastore Key and Query API.
Deprecated
❗️ This package is deprecated. Use now the nsql-cache + nsql-cache-datastore packages for your Google Datastore entities caching.
Highlight
- Define multiple cache stores with different TTL thanks to node-cache-manager.
- LRU memory cache out of the box to speed up your application right away.
- Datastore and objects are converted to unique string ids easy to cache.
- Advanced cache (when using node_redis) that automatically saves your queries in Redis "Sets" by Entity Kind. You can then set an infinite TTL (time to live) for your queries and only invalidate the cache when you add, edit or delete an entity Kind.
Please don’t forget to star this repo if you found it useful :)
Installation
npm install gstore-cache --save# or yarn add gstore-cache
Info: gstore-cache is fully integrated into gstore-node. If you are not (yet) using gstore-node to manage your Datastore entities, have a look at the project te see see what you're missing!
Usage example
Datastore <Key>
const Datastore = ;const gstoreCache = ; const datastore = ;const cache = gstoreCache; const key = datastore; /** * The "keys.read()" helper will * - Look for the entity in the cache * - If not found, fetch it from the Datastore * - Prime the cache with the entity fetched from the Datastore. */cachekeys; /** * You can also pass several keys. * gstore-cache will first check the cache and only fetch from the Datastore * the keys that were *not* found in the cache. * * In the example below, only the "key3" would be passed to datastore.get() and * fetched from the Datastore */const key1 = datastore; // this entity is in the cacheconst key2 = datastore; // this entity is in the cacheconst key3 = datastore; cachekeys;
The "gstoreInstance.keys.read()" helper above is syntactic sugar for the following:
const Datastore = ;const gstoreCache = ; /** * After you initialized the cache (once during application bootstrap) * you can get its instance anywhere calling "instance()". */const cache = gstoreCache; const datastore = ;const key = datastore; cachekeys ;
Datastore <Query>
const Datastore = ;const gstoreCache = ; const datastore = ;const cache = gstoreCache; const query = datastore ; /** * Just like with the Keys, the "queries.read()" helper will * - Look for the query in the cache * - If not found, run the query on the Datastore * - Prime the cache with the response from the query. */cachequeries;
The "gstoreInstance.queries.read()" helper is syntactic sugar for the following:
const Datastore = ;const gstoreCache = ; const datastore = ;const cache = gstoreCache; const query = datastore ; cachequeries ;
Advanced Queries Caching
gstore cache has an advanced cache mechanism for the queries when you provide a Redis client.
If you provide a Redis store then when you read() or set() a query, gstore cache not only saves the response of the query in the cache(s), but it also detects the Entity Kind of the query and saves a reference of the query in a Redis Set.
This means that you can safely have the query data in the cache infinitely until you either add, edit or delete an entity of the same Kind.
// server.js const Datastore = ;const gstoreCache = ;const redisStore = ; const datastore = ; const cache = gstoreCache; // ...
// ...some handler const query = datastore ; // with read()cachequeries ; // or with set()query ; // You can now invalidate the cache only when// you create/edit or delete a "Posts" entity. const key = datastore;const data = title: 'My Post' ; datastore ;
API
gstoreCache
gstoreCache.init(options)
Initialize gstore cache. You only needs to do it once, on application bootstrap.
-
options: An object with the following properties:
- datastore: a @google-cloud/datastore instance
- config: an object of configuration (optional)
The config object has the following properties:
-
stores: An array of "cache-manager" stores. Each store is an object that will be passed to the
cacheManager.caching()
method. Read the docs to learn more about node cache manager.Important: Since version 2.7.0 "cache-manager" allows you to set, get and delete multiple keys (with mset, mget and del). The store(s) you provide here must support this feature.
At the time of this writting only the "memory" store and the "node-cache-manager-redis-store" support it. If you provide a store that does not support mset/mget you can still use gstore-cache but you won't be able to set or retrieve multiple keys/queries at once.
// Multi stores example const Datastore = ;const gstoreCache = ;const redisStore = ; const datastore = ; gstoreCache;
- ttl: An object of TTL configuration for Keys and Queries. This is where you define the TTL (Time To Live) in seconds for the Key caching and Query caching. You can override this value on any read/set/mset call later.
const config = // ... ttl: keys: 600 // 10 minutes queries: 5 // 5 seconds ;
In case you have multiple stores, you can have a different TTL value for each store.
const config = // ... ttl: stores: memory: keys: 300 // 5 minutes queries: 5 redis: keys: 60 * 60 * 24 // 1 day queries: 0 // infinite ;
- cachePrefix: An object of configuration for naming the cache keys. Each cache key will be prepended with a prefix that you can set here.
const config = // ... cachePrefix: keys: 'prefix-for-keys:' queries: 'prefix-for-queries:' ;
This is the complete configuration with the default values:
const config = stores: store: 'memory' max: 100 // max number of items in the LRU memory cache ttl: keys: 60 * 10 // 10 minutes queries: 5 // 5 seconds // the "stores" configuration is only needed when you provide multiple stores stores: memory: keys: 60 * 5 // 5 minutes queries: 5 redis: keys: 60 * 60 * 24 // 1 day queries: 0 // infinite cachePrefix: keys: 'gck:' queries: 'gcq:' ; const datastore = ; // Initialize gstore cache with the datastore instance and the configgstoreCache;
gstoreCache.instance()
Get the gstore cache instance.
gstoreCacheInstance.keys
read(key|Array<key> [options, fetchHandler]])
read is a helper that will: check the cache, if no entity(ies) are found in the cache, it will fetch the entity(ies) in the Datastore. Finally it will prime the cache with the entity(ies).
-
key: a Datastore Key or an Array of Datastore Keys. If it is an array of keys, only the keys that are not found in the cache will be passed to the fetchHandler.
-
options: an optional object of options.
ttl: 900 // custom TTL value // For multi-stores it can also be an object ttl: memory: 300 redis: 3600
- fetchHandler: an optional function handler to fetch the keys. If it is not provided it will default to the
datastore.get()
method.
const gstoreCache = ;const Datastore = ; const datastore = ;const cache = gstoreCache; const key = datastore; /** * 1. Basic example (using the default fetch handler) */cachekeys ; /** * 2. Example with a custom fetch handler that first gets the key from the Datastore, * then runs a query and add the entities from the response to the fetched entity. */const fetchHandler = datastore ;; cachekeys ; // or with a custom TTLcachekeys ;
get(key)
Retrieve an entity from the cache passing a Datastore Key
const key = datastore; cachekeys;
mget(key [, key2, key3, ...])
Retrieve multiple entities from the cache.
const key1 = datastore;const key2 = datastore; cachekeys;
set(key, entity [, options])
Add an entity in the cache.
- options: an optional object of options.
ttl: 900 // custom TTL value // For multi-stores it can also be an object ttl: memory: 300 redis: 3600
const key = datastore; datastore;
mset(key, entity [, key(n), entity(n), options])
Add multiple entities in the cache.
- options: an optional object of options.
ttl: 900 // custom TTL value // For multi-stores it can also be an object ttl: memory: 300 redis: 3600
const key1 = datastore;const key2 = datastore; datastore;
del(key [, key2, key3, ...])
Delete one or multiple keys from the cache
const key1 = datastore;const key2 = datastore; // Single keycachekeys; // Multiple keyscachekeys;
gstoreCacheInstance.queries
read(query [, options, fetchHandler])
read is a helper that will: check the cache, if the query is not found in the cache, it will run the query on the Datastore. Finally it will prime the cache with the response of the query.
-
query: a Datastore Query.
-
options: an optional object of options.
ttl: 900 // custom TTL value // For multi-stores it can also be an object ttl: memory: 300 redis: 3600
- fetchHandler: an optional function handler to fetch the query. If it is not provided it will default to the
query.run()
method.
const gstoreCache = ;const Datastore = ; const datastore = ;const cache = gstoreCache; const query = datastore ; /** * 1. Basic example (using the default fetch handler) */cachequeries ; /** * 2. Example with a custom fetch handler. */const fetchHandler = q ;; cachequeries ;
get(query)
Retrieve a query from the cache passing a Datastore Query
const query = datastore; cachequeries;
mget(query [, query2, query3, ...])
Retrieve multiple queries from the cache.
const query1 = datastore;const query2 = datastore; cachequeries;
set(query, data [, options])
Add a query in the cache
- options: an optional object of options.
ttl: 900 // custom TTL value // For multi-stores it can also be an object ttl: memory: 300 redis: 3600
const query = datastore; query;
mset(query, data [, query(n), data(n), options])
Add multiple queries in the cache.
- options: an optional object of options.
ttl: 900 // custom TTL value // For multi-stores it can also be an object ttl: memory: 300 redis: 3600
const query1 = datastore;const query2 = datastore; Promiseallquery1 query2 ;
kset(key, value, entityKind|Array<EntityKind> [, options])
Important: this method is only available if you provided a Redis store during initialization.
If you have a complex data resulting from several queries and targeting one or multiple Entiy Kind, you can cache it and link the Entity Kind(s) to it. Let's see it in an example:
const gstoreCache = ;const cache = gstoreCache; /** * Handler to fetch all the data for our Home Page */const fetchHomeData = { // Check the cache first... cache;};
clearQueriesEntityKind(entityKind|Array<EntityKind>)
Delete all the queries linked to one or several Entity Kinds.
// ... continuing from the example above. // Create a new "Posts" Entity Kindconst key = datastore;const data = title: 'My new post' text: 'Body text of the post' ; datastore ;
del(query [, query2, query3, ...])
Delete one or multiple queries from the cache
const query1 = datastore;const query2 = datastore; // Single querycachequeries; // Multiple queriescachequeries;
"cache-manager" methods bindings (get, mget, set, mset, del, reset)
gstore cache has bindings set to the underlying "cache-manager" methods get, mget, set, mset, del and reset. This allows you to cache any other data you need. Refer to the cache-manager documentation.
const gstoreCache = ;const cache = gstoreCache; cache; cache; cache; cache; cache; // Clears the cachecache;
Development setup
Install the dependencies and run the tests. gstore-caches lints the code with eslint and formats it with prettier so make sure you have both pluggins installed in your IDE.
# Run the tests npm installnpm test # Coverage npm run coverage # Format the code (if you don't use the IDE pluggin) npm run prettier
To run the integration tests you need to launch the Local Datastore emulator and a local Redis server.
# Local Datastore # Make sure you have the emulator installed # More info: https://cloud.google.com/datastore/docs/tools/datastore-emulator # # The following command will create a "local-datastore" folder inside the project # where the Local Datastore will keep the entities gcloud beta emulators datastore start --data-dir=$PWD/local-datastore # Redis server (Mac Os or Linux) # From inside the folder where redis is located: ./redis-server
Release History
- 1.0.0
- First Release
Meta
Sébastien Loix – @sebloix
Distributed under the MIT license. See LICENSE
for more information.
Contributing
- Fork it (https://github.com/sebelga/gstore-cache/fork)
- Create your feature branch (
git checkout -b feature/fooBar
) - Commit your changes (
git commit -am 'Add some fooBar'
) - Push to the branch (
git push origin feature/fooBar
) - Rebase your feature branch and squash (
git rebase -i master
) - Create a new Pull Request