@openstapps/es-mapping-generator

Tools to convert and validate StAppsCore

What is the ES Mapping Generator for?

The ES Mapping Generator is for converting TypeScript interfaces to Elasticsearch mappings, with support for Integer and Float types, Dates, etc.

Why is this a separate package?

The ES Mapping Generator relies on an outdated version of TypeDoc, that can't be updated. Because of that, it then also relies on an outdated version of TypeScript, resulting in a cascade effect things that can't be updated. Because of that, we decided to isolate it to the best of our abilities.

How to use the Elasticsearch Mapping generator

The mapping generator is intended to be used by the backend directly, but it can also be used to generate these files manually.

Generating the mapping files by hand

To generate the mapping files by hand, you need a local copy of the core-tools and the core, both need to be built first. After that you can run

node lib/cli.js mapping path/to/core path/to/destination ignoredTag1,ignoredTag2,ignoredTag3

If you don't pass in any ignored tags, you will likely be prompted with errors, despite the core being absolutely correct. This is because there are some tags that are not relevant to Elasticsearch, but the program has no direct way to tell which ones simply lack an implementation and which ones can be ignored. Currently the ignored tags include minlength, pattern and see.

Generating the mapping directly from another TypeScript program

This is the more easy way, and it gives you direct access to the generated mapping as a (mostly typesafe) object. To use it, call generateTemplate, However you will first need to generate a ProjectReflection of the core you are working with. If you use the core as a dependency, you can for example use

const map = generateTemplate(
  getProjectReflection(resolve('node_modules', '@openstapps', 'core', 'src')),
  ignoredTags,
  false,
);

to generate the mappings. Note that the result object contains both a list of errors in map.errors and the actual mapping in map.template. You can also specify whether you want the generator to show any errors while generating the mappings in the console in the last parameter, true (default) being show errors and false to suppress them. That said it is very easy to replace all type: "MISSING_PREMAP", type: "PARSE_ERROR", type: "TYPE_CONFLICT" with dynamic: true (you can take these exactly like written here and run a replace over the file).

Fixing a generated mapping by hand

If you get errors when generating the mappings, the mappings might not work, however they will still be generated to the programs best efforts. Most small issues can be fixed after the mapping was generated as a temporary solution and without changing anything in the mapper's code. This however requires some understanding of how mappings work.

The output of the program can easily reach 25.000 lines, but you can find errors quickly by searching for MISSING_PREMAP, PARSE_ERROR and TYPE_CONFLICT. When you reach them you can then manually replace them with your code.

As a last resort you can also replace all errors with dynamic types, this should get the mapping working, but it is NOT RECOMMENDED as a fix other than using it locally.