@madxnl/madhatter

Description

Framework for quickly starting up new graphql projects based on nestjs

Nest framework TypeScript starter repository.

New version

npm version <nummer>
git push && git push --tags

Link to Madhatter

Linking no longer works with npm / yarn without causing library conflicts. Thus we need to create a package for madhatter and install it in the project. To create a package from the madhatter library, run the following commands:

cd madhatter
npm pack

This will create a tarball file in the madhatter directory. This can be installed in the project by running:

npm install ../madhatter/madhatter-x.y.z.tgz

New modules/files

npm run generate-index

Settings

You can use environment variables to change certain behavior of MadHatter:

Key	description	default
SECURE_FILE_DOWNLOADS	Enables the default security header on file downloads	true

Madhatter basics

What is it?

Madhatter is a framework we developed to help build our APIs faster and more consistently. It is built on top of NestJS, written entirely in TypeScript, and uses GraphQL as the API interface. It uses a PostgreSQL database with TypeORM, but also provides an Elasticsearch instance for high performance queries. It provides a set of baseclasses that supports basic behavior (CRUD Actions) for your entities out of the box, and ensures integrity throughout the project. It also provides a CLI tool to help scaffold the files neccessary for the entities yourself (see later chapter).

Baseclasses

Entities are organized into groups of *.model.ts, *.resolver.ts and *.service.ts files for model definition, API interface definition, and business logic implementation respectively.

Madhatter provides BaseModel, GenericResolver and BaseService classes to help with the implementation of these files.

BaseModel simply extends your model with unique ID and timestamps.
GenericResolver provides a default interface for all CRUD operations in the form of query Query, and create, update and delete Mutations.
BaseService being the most comprehensive of all:
- It provides implementation for all CRUD operations, including hooks like beforeUpsert and afterUpsert.
- It provides an interface to configure whether to index the entity in Elasticsearch or not, and which fields to include in the indexing.
- It provides logic for both query in the PostgreSQL database and search in the Elasticsearch container.
- It provides validation rules that are invoked before saving the entity.

Madhatter CLI

To help scaffolding the necessary files, madhatter also provides a CLI tool in a separate package (@madxnl/madhatter-cli). It is using Handlebars templating to generate *.model.ts, *.resolver.ts and *.service.ts for entity definitions located in the madhatter.yml file. By running the madhatter generate command, it generates for each entity:

A *.model.ts file with a model definition extended from BaseModel with the corresponding fields defined in the madhatter.yml file, and next to it a return type and an input type (required for the GraphQL schema).
A *.service.ts file with a service extended from BaseService, with its models repository injected, and empty validation rules defined.
A *.resolver.ts file with a resolver extended from GenericResolver, with its service injected, and ResolveField defined for each relation.

IMPORTANT! The CLI tool is also able to overwrite existing files, so can be useful even when just adding a new field on an entity. However, it is basing its templating on the following comment placed on top of files. If this comment is removed, the file will be omitted for the CLI tool. (Meaning once you start adding custom logic to the service and resolver files you should remove the comment, and can no longer use the CLI tool on those files.)

/**
* This file is auto generated by the MadHatter framework. Do not modify the
* contents of this file unless you really need to.
*
* Remove this comment if you want to prevent any future overrides.
*/

An example for entity definition in the madhatter.yml file:

modules:
  - name: User
    models:
      - name: Organization
        fields:
          - "name: string"
          - "email?: string"
          - "status: OrganizationStatus: 0"
          - "users: User[]"

The above example would generate a User module (see NestJS documentation about modules) with an organization.model.ts, organization.service.ts and organization.resolver.ts files in it. The organization model would be defined with:

A name field of type string that is a required field.
An email field of type string that is an optional field.
A status field of enum type OrganizationStatus that is a required field, and has a default value of 0.
A users relation of type OneToMany.

(One would obviously still need to define User and OrganizationStatus separately.)

How we store data

We store the data in a PostgreSQL database, but we also have an Elasticsearch (ES) container that you can optionally use to index specific fields on specific models. Thus, the ES container is no more than a replica next to the primary SQL database that is always kept in sync.

How to index entities in the ES container

For ES indexing we support indexing each field separately (including relations), and we also provide an extra data field where you can dump data from different fields to allow for omnisearch. This can be configured separately from the field indexing!

As an example, on a User model we could index the email and organization.id fields separately, while we could dump the firstName, lastName and email fields in the data field. In this case the corresponding ES document would look like this:

{
  "email": "john@doe.com",
  "organization": {
    "id": "12345678-1234-1234-1234-123456789012"
  },
  "data": "John Doe john@doe.com"
}

To mark an entity for ES indexing simply set the indexModel property in its *.service.ts file to true. This would by default index all its primitive fields (no relations) for both indexing separately and for dumping in the data fields. Though this works, it is not recommended for a couple reasons:

You might accidentally index sensitive fields like password.
After adding a new field to a model that would also be indexed, and from that point on your ES documents would be inconsistent.

It is much better practice to define the fields you want to index yourself. To do so, define the searchFilterFields and searchQueryFields in the service file for specifying the fields to index, and which fields to dump in the data field, respectively. Following our previous example this would look like:

  public searchFilterFields() {
    return {
      email: true,
      organization: {
        id: true,
      },
    }
  }

  public searchQueryFields() {
    return {
      firstName: true,
      lastName: true,
      email: true,
    }
  }

Query or search

Whenever looking up an entity you can decide whether to use the SQL database (query) or the ES container (search). All of the madhatter internal logic uses SQL since most of the lookups are based on ID and that being an index in the SQL tables it is just as performant. The main use case for ES comes in handy of course when you want to search on specific fields, or perform omnisearch.

For this purpose the GenericResolver already implements logic in its query method to decide whether to perform query on SQL or search on ES.

To understand this decision we first have to talk about the arguments of the query method. GenericResolver provides this method for every entity, and generates a GraphQL Query operation on the schema, with the entity's name being the operation name by default. The arguments added by default are:

input: an array of objects typed as the entity itself, this argument is used to specify which fields to search or query upon. The array notation represents an OR operation. If any of the fields other than the id is defined here, then the ES search will be invoked. If only the id is defined then Postgres query will be invoked.
queryArgs: General query arguments we want to use for pagination, sorting, omnisearch, etc. Properties on this argument are:
- page, size: pagination variables.
- queryString: The string field used for omnisearch. This searches on the data field of the ES document.
- sort: An array of type SortArg that is directly being mapped to elasticsearch sort query. Recommended for sorting on multiple fields, since changing the order of the sort means simply changing the order of the array.
- orderBy: An object with key-value pairs, where the keys are fields to be sorted on and the values are the sort order. Madhatter does not provide native type for this, you are required to create the type definition by extending the QueryArgs type. This is also being mapped to elasticsearch sort query, but due to how NestJS processes input variables, the type definition will determine the order of the mapping, and henceforth the order of the sorting. Therefore this is NOT RECOMMENDED for sorting on multiple fields!

If any of the properties (except for the pagination variables) are set on the queryArgs argument, then the ES search will be invoked.

So in general for a root.Query operation Madhatter solves the question for you whether to use SQL or ES. When resolving relations (methods with @ResolveField decorator) you have the freedom to decide for yourself.

Example type definition on the client side:

@InputType()
class UserOrderBy {
  @Field(() => Order, { nullable: true })
  email: Order
}

@InputType()
export class UserQueryArgs extends PartialType(QueryArgs(UserInputSchema)) {
  @Field(() => UserOrderBy, { nullable: true })
  orderBy?: UserOrderBy
}

How we mutate data

For creating and updating entities the GenericResolver provides two methods: create and update, and the BaseService defines some elaborate mechanisms revolving around the upsert method.

One can tell by first glance that the create and update methods are exactly the same by default. This is because throughout the entire system we operate under the assumption that creating and updating ( = upserting ) entities should involve the same steps, the only difference being whether an id field is already defined in the input argument or not. Thus, both create and update resolver methods end up calling the upsert method from the BaseService.

Calling the upsert method without an ID parameter would correspond to creating a new instance of the entity semantically:

this.service.upsert(
  { 
    name: "New distributor" 
  }
)

Whereas calling the upsert method with an ID parameter would correspond to updating an existing instance of the entity semantically:

this.service.upsert(
  { 
    id: "12345678-1234-1234-1234-123456789012",
    name: "Updated distributor"
  }
)

Important to note that in this case upsert performs a partial update, thus only the fields defined in its argument will be modified, and the rest preserved.

upsert can also handle relations, and as an example one could create or update users through upserting an organization:

this.service.upsert(
  { 
    id: "12345678-1234-1234-1234-123456789012",
    users: [
      { 
        name: "New user" 
      },
      { 
        name: "Updated name for this other user",
        id: "12345678-5678-5678-5678-123456789012",
      }
    ] 
  }
)

In this case a new user would be created, the existing one updated, and the rest of the users belonging to this organization preserved.

IMPORTANT! Throughout the madhatter ecosystem we always stick to using upsert method to mutate data, and we very strongly advise to stick to this conviction in your project as well. There are many things hooked into the upsert flow, such as:

Taking care of updating relations
Syncing the ES container with the updated data if needed
Providing hooks for custom logic before and after the save ('beforeUpsert', 'afterUpsert', event handlers with @On decorator)

Validation rules

In the create and update method of the GenericResolver madhatter performs validation checks that can be configured in the entity's service file. You can also invoke this validation method at any given place if you wish. It is typed such that each fields on the entity must be defined under the validationRules object. Each entry in this method is a callback that receives the to-be-updated value, and the entire model instance as arguments. To throw an error, return a string as the error message to be thrown, to approve the value return undefined.

Example:

    passwordResetHash(value: string, model: User): Promise<void | string> | void | string {
      if (value != null) {
        return 'Password reset hash cannot be set directly, use the proper mutation instead'
      }
      return undefined
    }

Starting from version 0.7.14, validationRules field on a service is deprecated. Please use the @ValidationRule decorator instead on a class, and define the rules on the class as static methods. Example:

import { ValidationRule } from '@madxnl/madhatter';
import { Address } from './address.model';
import { Injectable } from '@nestjs/common';

@ValidationRule(Address)
@Injectable()
export class AddressValidator {
  public static city(value: string, model: Address): void | string {
    if (value !== 'Amsterdam') {
      return 'City must be Amsterdam'
    }
    return undefined
  }
}

Do not forget to then register this class as a provider in an appropriate module.

Hooks and event listeners

Madhatter provides different mechanisms for custom logic to be executed upon an upsert operation:

beforeUpsert and afterUpsert hooks

These are methods that are invoked before and after the upsert operation and are hooked into the operation lifecycle (upsert won't return until these methods have returned as well).

Both receive the input and model arguments, the input argument being the one that contains the data the upsert was called with, the model being the entity's current state in the database.

  public async beforeUpsert(input: User, model: User): Promise<void> {
    ...
  }

@On decorators

These are decorators that can be used on methods of services, and are invoked upon the upsert operation. They are event-based i.e. they operate outside the operation lifecycle, and are invoked asynchronously.

The @On decorator takes a string argument that is the name of the event to listen to, with : format.

The handler function receives a single argument, which is the data that was passed to the upsert method.

  @On('Team:update')
  async onTeamUpdate(model: Team) {
    ...
  }