@microfleet / core

Opinionated Web and (Micro)Services Toolkit

Lead Maintainer: Vitaly Aminev

Easily design, develop & distribute your code with opinionated microservices toolkit. It's designed to provide highly extensible, defined structure so that you only have to worry about implementing business logic and nothing else. Make sure that every minute you spend counts.

Installation

yarn add @microfleet/core or npm i @microfleet/core

Getting started

What is @microfleet?

First of all it's a middle ground between usability and flexibility, ease of use and scalability. Unless you are Facebook or Google it's very likely that you can handle hundreds of thousands of TPS on a bunch of small and slow VMs with @microfleet. It's not a walk in the park, but after grasping basic concepts of this - you'd be able to create robust microservices within minutes.

A lot of components could still be improved and a lot of common use-cases handled, we've been working on this since 2015 and we are commited to constantly improving this toolkit as we believe that almost every app out there shares common functionality. We want to make a toolkit that allows you to hack together production grade apps within days to be able to prove your startup concept, roll out safely and actually handle load if your idea is the next Big One.

Having this concept in mind there is already a solid basis for this:

• @microfleet/users - user management service with practically anything you might think of in that matter
• @microfleet/files - distributed file management with the help of S3-like storage
• @microfleet/payments - loathed paypal payments implementation. This one likely needs a lot of love, even though it's still used in production
• @microfleet/chat - chat service, real-time and scalable with moderator capabilities. Once again a middle ground between complicated messenger apps and hello-world chats
• @microfleet/mailer - sends emails, what else! includes queue, quality of service & integrations with some of the popular providers
• @microfleet/phone - if mails aren't good enough you can always send a text message
• @microfleet/social - hub for social services
• @microfleet/calendar - be able to create calendar of events for your app, radio station or anything else
• @microfleet/organizations - this one is WIP, provides meta layer to include shared-accounts / organization-like accounts

Most of the time it's easy to combine these services in a lego-like fashion providing you with a mix of desired functionality.

Using @microfleet

So, how do we use it? After many iterations we have what we believe is a great structure for the code, which allows for easy service creation. Here is an example of action:

function addAction({ params }) {
  return params[0] + params[1];
}

module.exports = addAction;

How do we test it? Easiest for us is https://github.com/jakubroztocil/httpie, go ahead and install it if you still haven't

It's clear that params could be anything and our little action handler won't do any good in that way as we have no means of enforcing validation of input arguments. Rejoice, this was one of the first things we've done. By default every handler will look for a relevant json-schema under <project_root>/schemas/<actionName>.json. Lets create that file now with the following content:

{
  "$id": "add",
  "type": "array",
  "items": {
    "minItems": 2,
    "maxItems": 2,
    "type": "number"
  }
}

It will ensure that payload is validated, it must be an array and contain 2 numbers. You get the gist of it. Let's try running request without any payload:

http localhost:3000/mservice/add

HTTP/1.1 400 Bad Request
Connection: keep-alive
Date: Mon, 22 May 2017 21:11:32 GMT
Transfer-Encoding: chunked
cache-control: no-cache
content-encoding: gzip
content-type: application/json; charset=utf-8
vary: accept-encoding

{
    "error": "Bad Request",
    "message": "add validation failed: data should be array",
    "name": "ValidationError",
    "statusCode": 400
}

And with the correct payload:

echo '[1,2]' | http POST localhost:3000/mservice/add
HTTP/1.1 200 OK
Connection: keep-alive
Date: Mon, 22 May 2017 21:21:33 GMT
Transfer-Encoding: chunked
cache-control: no-cache
content-encoding: gzip
content-type: application/json; charset=utf-8
vary: accept-encoding

3

Now on top of it you can build pretty much anything - routes will be created automatically, network topology will follow your configuration, input would be validated. Make sure to check real-world examples with or without database usage. We recommend starting with the @microfleet/mailer service as it contains only 2 actions, but clearly shows most of the concepts.

Available Plugins

To ensure smooth and fast development, there are many core plugins that build-up on core functionality. At this point they are all bundled with core toolkit, but expect them to be externalized as toolkit matures.

Essentials

This includes common functionality plugins:

• validator - json-schema based input data validation
• logger - request logger
• router - logic layer for multi-transport router

Validator

packages/plugins-logger

Based on @microfleet/validation allows to easily validate input args based on json-schema. Create directory schemas and populate it with schemas, where names correspond to actions.

Adds following API to the microservice instance:

• .validator - instance of ms-validation
• .validate<T>(schema: string, input: T) => Promise<Error | T> - Promise-based API that resolves/rejects based
• .validateSync<T>(schema: string, input: T) => { error?: Error, doc: T } - sync API, which always returns an object. If validation failed - it populates error property with an instance of Error. Doc is a modified version of input arg with coercion, defaults, filtering of props and so on.

Logger

packages/plugins-logger

Creates bunyan logger, with streams based on passed configuration and extends microservice instance with the following methods:

• .log - instance of bunyan logger

Router

packages/plugins-logger

Initializes router, which scans folders for actions and builds routing tree for for enabled transports. Router controls request lifecycle, which tries to mimic hapi.js lifecycle as closely as possible, with unified interface for multiple transports. Currently supports AMQP (@microfleet/transport-amqp), HTTP (hapi.js) and Socket.IO. Default sensible configurations are provided.

Stock configuration looks for all **/*.js files in src/actions directory, enables hapi.js based HTTP handler on port 3000. On top of it enables 2 router extension, which provide request logging & automatic json-schema matching for actions based on their names.

Request Lifecycle

We've struggled to make sure that requests loop the same regardless of transport selected and the lifecycle was adopted from hapi.js model as it proved to be extremely easy to follow and extend.

Lifecycle goes in the following way:

1. Request - checks that there is an ServiceAction for the route passed from the transport, throws 404 error or maintance mode error.
   • preRequest: (this: Microfleet, request: ServiceRequest) => void
   • request: performs request
   • postRequest: can handle result or errors (ServiceRequest.error) from request part
2. Auth - performs authentication if ServiceAction.auth is present.
   • preAuth - (this: Microfleet, request: ServiceRequest) => void
   • auth: adds auth property contains credentials to ServiceRequest
   • postAuth: can handle result or errors (ServiceRequest.error) from auth part
3. Validate - performs validation based on json-schema if ServiceAction.schema is defined
   • preValidate: (this: Microfleet, request: ServiceRequest) => void
   • validate: performs validation
   • postValidate: handle validation result or errors (ServiceRequest.error) if handlers present
4. Allowed - performs arbitrary validation before passing control to actual function handler
   • preAllowed: (this: Microfleet, request: ServiceRequest) => void
   • allowed: performs arbitrary code defined in ServiceAction.allowed
   • postAllowed: handle result or errors (ServiceRequest.error) from allowed part
5. Handler - runs userland endpoint code:
   • preHandler: (this: Microfleet, request: ServiceRequest) => void
   • handler: performs userland code
   • postHandler: handle response or error from handler
6. Response - runs response handler, which makes standard error / standard responses:
   • preResponse: (this: Microfleet, request: ServiceRequest) => void
   • response: pushes data to the requester
   • postResponse: can modify finalized data that goes to the client

Transports

• amqp - AMQP transport based on @microfleet/transport-amqp, requires RabbitMQ
• hapi - hapi implementation for http
• socket.io - enabled websockets on top of http, therefore, requires hapi plugin to be enabled

Databases

• redis cluster - clustered redis implementation, uses ioredis client
• redis sentinel - HA redis implementation, no sharding, uses ioredis client
• knex - high-level API for SQL based databases (PostgreSQL, MySQL, MariaDB, etc)
• elasticsearch - elasticsearch connector
• cassandra - cassandra connector

Roadmap

As with any healthy toolkit - there is always plenty to add. These are some of the major features that we are working towards. Our goal to ensure that most of the apps can be created by writing a simple integration layer with your business logic and a bunch of human-readable configuration

• [ ] more docs

  • [ ] verbose validator configuration example
  • [ ] verbose logger configuration example
  • [ ] verbose router configuration example
  • [ ] verbose AMQP transport configuration docs
  • [ ] verbose HTTP transports configuration example
  • [ ] verbose socket.io transport configuration example
  • [ ] redis documentation
  • [ ] knex documentation
  • [ ] elasticsearch configuration documentation
  • [ ] cassandra configuration docs

• [ ] authentication framework:

  • [ ] ubiquitous way to pass authorization tokens disregarding the transport
  • [ ] RBAC route configuration

• [ ] transport-agnostic inter-service communication API

  • [ ] service discovery integration:
    • [ ] consul
    • [ ] etcd
  • [ ] high level messaging API
  • [ ] app-level healthchecks

• [ ] Tracing API: visibility into transactions

  • [ ] tracing integration via http://opentracing.io/
  • [ ] Monitoring dashboard

Sponsorship

Development of the @microfleet generously supported by contributions from individuals and corporations. If you are benefiting from @microfleet and would like to help keep the project financially sustainable, please send an email to Vitaly Aminev

Current Supporters

• Makeomatic

License