QueryLayer

QueryLayer is a client-side JavaScript library that provides an adaptive query abstraction layer for data-driven applications. It provides a consistent interface for responding to SQL-semantic queries adaptively with either client-side data transformations or back-end requests.

Installation

(Coming Soon!)

npm install query-layer

Features

• Standard Query Format: QueryLayer provides a common representation for SQL-semantic queries across your application, allowing components to express their data requirements without knowledge of any back-end system, and making it easy to change query approaches as the application grows.

• Modular Data Connectors: QueryLayer does not offer support for any backend database out of the box (though we may add this in the future). Instead, it supports a modular interface for data connectors, allowing it to be used with almost any backend system that accepts some variant of SQL semantics.

• Fast Client-Side Query Processing: QueryLayer supports high-performance tabular data transformations in the client, supporting the same arbitrarily complex queries that might be sent to a back-end system.

• Intelligent Caching: QueryLayer uses an LRU caching approach for loaded data. In addition to standard cache lookups for repeated queries, QueryLayer can run new queries on top of cached data, e.g. calculating grand totals or applying additional filters, avoiding unnecessary back-end calls.

• Strongly Typed Builder API: QueryLayer offers clean, legible builder functions for queries and expressions, supporting a large subset of SQL select syntax. The builder functions provide static type safety using Flow, and the resulting queries can be fully validated at runtime.

• Full Expression DSL: QueryLayer supports parsing, interpretation, and validation of a SQL-style expression language, allowing your application to create complex queries from user input. QueryLayer offers both syntactic validation of the DSL and semantic validation of the query, with human-readable error reporting.

Basic Usage

Creating a QueryLayer instance for static data:

const mySchema = {
  id: 'my-schema',
  fields: [
    {id: 'city', type: 'string'},
    {id: 'metrics1', type: 'number'},
    {id: 'ts', type: 'time'},
  ],
};

const queryLayer = createStaticQueryLayer({
  data: DATA,
  schema: mySchema,
});

Using the builder functions to create a query:

const query = createQuery(
  mySchema,
  select(string('city'), sum(number('metrics1'))),
  where(gte(time('ts'), literal(new Date('2019-01-01T00:00Z')))),
  groupBy(string('city')),
  orderBy(desc(sum(number('metrics1'))))
);

Loading a query:

await queryLayer.loadQuery({query});

Your Questions Answered

Why would I need this?

Applications for data visualization and analysis, e.g dashboard applications, exploratory data analysis tools, etc., generally opt for either front-end or back-end data processing, with significant tradeoffs:

• Front-end data processing happens in client-side memory and is generally very fast on smaller datasets. This allows for much more fluid user interactions, e.g. seeing near-immediate results when scrubbing a timeline or changing the aggregation. But performance breaks down as the size of the dataset gets bigger, freezing or even crashing the UI.

• Back-end data processing can scale horizontally, potentially handling datasets of arbitrary size. However, even in the best case, these queries require network calls, and you lose the responsiveness of in-memory dataset manipulation, making many interactions slow and painful for the end user.

QueryLayer addresses this issue by supporting both modes, requesting data when needed and reusing loaded data whenever possible. The rest of your application can send queries without worrying about whether they're handled on the client side or in a back-end database, and QueryLayer will adaptively pick the best approach.

SQL-semantic?

SQL query semantics can be expressed in a variety of syntaxes, but the basic elements are SELECT, FROM, WHERE, GROUP BY, and ORDER BY clauses, where each clause is either an expression or an expression list. Expressions can be arbitrarily nested, e.g. SUM(a * b) or (foo OR bar) AND baz. While most SQL databases process queries using a string syntax, QueryLayer uses builder functions to create a query AST (abstract syntax tree), which is easy to validate, manipulate, and reformat for a variety of back-end APIs. The AST is human-legible, the builder functions even more so, making it easy to understand and reason about your queries.

What about GraphQL?

QueryLayer and GraphQL focus on very different use cases. QueryLayer's main concern is tabular data transformation, e.g. analytical queries over a single dataset, particularly with datasets where the schema is only available at runtime. GraphQL is much better suited for ORM applications where the data structures are known in advance, and where the relationships between different models is a key concern, but offers poor support for the kind of arbitrary user-specified queries required in many data analysis applications.

Development

To run tests (using Jest):

yarn test

To lint (using ESLint):

yarn lint

To type-check (using Flow):

yarn flow

To format the code (using Prettier):

yarn prettier