frxt

freetext filter for sql-esque data

About

frxt is a refractor and update to the library json-ctrl by combining several of the smaller dependencies (json-cnf, json-tkn, json-tim, and jsqlon). As such there are many moving pieces for which this README.md will try to explain and address.

Assumptions

A core assumption of this library is that the stored data is SQL-eqsue. In other words, that the JSON object's outer most keys correspond to record ids and that each record (an object) is structured as if in a SQL table i.e. each record contains the same fields:

json = {
  id1: {
    field1: value1f1,
    field2: value1f2,
    field3: value1f3,    
  },
  // ...
  idn: {
    field1: valueNf1,
    field2: valueNf2,
    field3: valueNf3,    
  }
}

It may be the case that users need to overwrite the default behavior of how field values are treated. For this three objects exists in which overriding functions can be set. These include:

• fieldRenderFunctions: how a given field's value should be altered when rendering it on the page.
• fieldSortByFunctions: how a given field's value should be altered when comparing against other values for sorting.
• fieldFilterFunctions: how a given field's value should be altered when comparing against specified filters.

To overwrite a field's function (e.g. fieldX), a function with the following signature is expected:

function overwriteFieldX(id, field, record) {
  let value = record[field]
  // manipulate value to be returned.
  return value
}

then set the corresponding object (e.g. fieldRenderFunctions) to have this function with the field name as the key:

fieldRenderFunctions = {
  fieldX: overwriteFieldX
}

Example

let json = {
  'banana': {
    'fruit': 'banana'
  },
  'pear': {
    'fruit': 'pear'
  },
  'apple': {
    'fruit': 'apple'
  }
}
 
let inputString = 'fruit is banana or type contains app'
let tknFieldMap = {
  'fruit': 'fruit',
  'Type': 'fruit',
  'name': 'fruit',
}
 
let [valid, error] = frxt.extractFiltersFromText({
  input:inputString,
  tknFieldMap:tknFieldMap,
  tknFunctionMap:frxt.configs.defaultCNFFunctionMap,
  tknConditionalMap:frxt.configs.defaultCNFConditionalMap,
  fieldTokens:undefined,
  functionTokens:undefined,
  conditionalTokens:undefined,
  scrubInputPattern:frxt.configs.PunctuationPattern,
  cnfLogicConfig:frxt.configs.defaultCNFLogicConfig,
  cnfFunctionsConfig:frxt.configs.defaultCNFFunctionConfig,
  cnfConditionalConfig:frxt.configs.defaultCNFConditionalConfig,
  debug:false
})
expect(valid.length).toBe(2)
expect(error.length).toBe(0)
expect(valid[0].filter.field).toBe('fruit')
expect(valid[1].filter.field).toBe('fruit')
 
let filters = valid.map(({filter})=>filter)
 
let res = frxt.conjunctiveNormalFormFilter(
  json,
  filters,
  frxt.configs.defaultCNFLogicConfig,
  frxt.configs.defaultCNFFunctionConfig,
  frxt.configs.defaultCNFConditionalConfig,
  {}
)
expect(res.length).toBe(2)
expect(res[0]).toBe('banana')
expect(res[1]).toBe('apple')

Filtering

Filtering of the records in the stored JSON occurs in four steps:

1. records are passed through the specified conjunctive normal form filters. (This utilizes specified fieldFilterFunctions).
2. the records passing step 1 are passed through the global RegEx (This utilizes specified fieldRenderFunctions, as this is what the user sees).
3. the records are sorted via TimSort from the given sortSpecification (This utilizes the specified fieldSortByFunctions).
4. the records of the specified "page" are returned.

Sorting

Sorting is handled via the state variable sortSpecification. sortSpecification is an array of objects, where each object has the following structure:

sortSpec = {
  field: fieldName,
  isAscending: boolean
}

This specification allows for the fields, in the order provided in sortSpecification to undergo TimSort.

Pagination

Pagination is provided for controlling which records are show to the user. It is worthwhile that this is local pagination, therefore all the data is already in memory. This stems from the combination of functions applied to the data outlined in the section regarding filtering. Consider the question "what are the five elements on page two?". If filters, RegEx and sorting are in play, there is no way to know this without finding all records that pass and sorting them to know which will be on page two.

What's Inside?

• defaults: default configurations
• filter: conjunctive normal form filter for SQL-esque JSON
• freetext: extract filters from user provided freeform text
• jsonsql: utilities for SQL-esque JSON
• sort: timsort for SQL-esque JSON
• types: typescript types for this package
• utils: grab bag of utility functions

Configuration

Utility Functions

There is a lot which can be configured to achieve the desired behavior. As these configurations are mostly independent, they are split up into sub-configs. These are passed to the utility functions from the components.

I already touched on the fieldRenderFunctions, fieldSortByFunctions, fieldFilterFunctions above. In addition there are:

1. configCNFLogic: The logical options for CNF.
2. configCNFFunctions: The functions which can be applied during CNF.
3. configCNFConditionals: The conditionals (or tests) which can be applied during CNF.
4. configCNFExtractors:
5. configTKNFunctionMap: A mapping of tokens to functions in the CNF.
6. configTKNConditionalMap: A mapping of tokens to conditionals in the CNF.

configCNFLogic

The two logical operations supported are and and or. You can change how they are displayed in VFilterView as so:

{
  and: {
    display: "∧"
  },
  or: {
    display: "∨"
  }
}

configCNFFunctions

These are the known functions and corresponding types they work on for the free text filter.

{
  identity: {
    display: "x → x",
    function: (x) => x,
    types: ["number", "string", "array:number", "undefined"]
  },
  abs: {
    display: "abs",
    function: (x) => Math.abs(x),
    types: ["number"]
  },
  mean: {
    display: "mean",
    function: (nums) => (nums.reduce((add, val) => add + val) / nums.length),
    types: ["array:number"]
  },
  max: {
    display: "max",
    function: (nums) => Math.max(...nums),
    types: ["array:number"]
  },
  min: {
    display: "min",
    function: (nums) => Math.min(...nums),
    types: ["array:number"]
  },
  length: {
    display: "len",
    function: (arr) => arr.length,
    types: ["array", "array:number", "array:string", "array:object"]
  }
}

configCNFConditionals

Here you find which conditionals exist

{
  eq: {
    display: "=",
    conditional: (a, b) => {
      // intentional == rather than === to allow for 1 == "1" to be true
      return a == b;
    },
    types: [
      "array", "array:string", "array:object", "array:number",
      "number", "string", "undefined"
    ]
  },
  neq: {
    display: "≠",
    types: [
      "array", "array:string", "array:object", "array:number",
      "number", "string", "undefined"
    ],
    conditional: (a, b) => a != b,
  },
  lt: {
    display: "<",
    types: ["number"],
    conditional: (a, b) => a < b,
  },
  gt: {
    display: ">",
    types: ["number"],
    conditional: (a, b) => a > b,
  },
  lte: {
    display: "≤",
    types: ["number"],
    conditional: (a, b) => a <= b,
  },
  gte: {
    display: "≥",
    types: ["number"],
    conditional: (a, b) => a >= b,
  },
  ss: {
    display: "⊂",
    types: ["array", "array:string", "array:number", "string"],
    conditional: (a, b) => {
      let includes = false
      if (Array.isArray(a)) {
        for (let i = 0; i < a.length; i++) {
          if (a[i] == b) includes = true
          if (includes) return includes
        }
      } else if (typeof a === 'string') {
        return a.includes(b)
      } else { return includes }
      return includes
    }
  },
  nss: {
    display: "⟈",
    types: ["array", "array:string", "array:number", "string"],
    conditional: (a, b) => {
      let includes = true
      if (Array.isArray(a)) {
        return a.every(e => e != b)
      } else if (typeof a === 'string') {
        return !a.includes(b)
      }
      return includes
    }
  }
}

configTKNFunctionMap

There are the default tokens that map tokens (the keys) to the corresponding values in configCNFFunctions.

{
  max: "max",
  min: "min",
  maximum: "max",
  minimum: "min",
  length: "length",
  len: "length",
  mean: "mean",
  median: "median",
  average: "mean",
  ave: "mean",
  identity: "identity"
}

configTKNConditionalMap

There are the default tokens that map tokens (the keys) to the corresponding values in configCNFConditionals.

{
  eq: "eq",
  is: "eq",
  equal: "eq",
  "=": "eq",
  "!=": "neq",
  "≠": "neq",
  ">": "gt",
  "≥": "gte",
  ">=": "gte",
  "<": "lt",
  "≤": "lte",
  "<=": "lte",
  "is not": "neq",
  neq: "neq",
  "not equal to": "neq",
  gt: "gt",
  "greater than": "gt",
  "less than": "lt",
  lt: "lt",
  "less than or equal to": "lte",
  "greater than or equal to": "gte",
  "member of": "ss",
  substring: "ss",
  contains: "ss",
  includes: "ss"
}