This package has been deprecated

Author message:

Moved to @hoast/hoast.

hoast

A modular file processor focused on creating a simple ecosystem for task automation.

Table of contents

• Elevator pitch
• Introduction
• Installation and usage
  • Command line interface
  • Script
  • Options
  • Debugging
• Modules
  • Build-in
  • Usage
  • Making
• Known issues

Elevator pitch

Creating a static page generator can be incredibly easy as is show below.

const Hoast = require(`hoast`);
const read = Hoast.read,
      frontmatter = require(`hoast-frontmatter`),
      layout = require(`hoast-layout`),
      transform = require(`hoast-transform`);
 
Hoast(__dirname, {
  patterns: [
    `*`,
    `!(layouts)`
  ],
  patternOptions: {
    all: true,
    extended: true
  }
})
  // Read file content.
  .use(read())
  // Extract front matter.
  .use(frontmatter())
  // Transform markdown.
  .use(transform({
    patterns: `*.md`
  }))
  // Layout files.
  .use(layout({
    directory: `layouts`,
    layouts: `page.hbs`,
    patterns: `*.html`
  }))
  // Process.
  .process();

See the static page generator example for the full example with in depth comments including dependencies and source directory.

Introduction

hoast is a modular file processor focused on creating a simple ecosystem for task automation. The original objective was to generate webpages using a minimal system, but in addition to static page generation it can also be used for a range of different applications.

The system has been written with several goals it mind, to be small in size as well as easy to use and improve. The result is a modular approach some of the advantages are:

1. An incredibly small base system, at just a few hundred lines of code and only three dependencies.
2. Highly customizable as modules are task specific therefore you only include the modules you want. This ensures you have a deeper understanding of the build process.
3. Modules are quick to create and allow a large portion of people to contribute.
4. Maintenance is easier as the code base is sectioned up and simpler to follow along to.

Another key advantage is the environment the system has been developed in, Node.js, which utilizes JavaScript the language most commonly known by web developers who this project targets with static page generation.

The order in which hoast works can be broken down into three main steps.

1. From the source directory all files are scanned.
2. Each module manipulates the information presented.
3. The results are written to the destination directory.

Installation and usage

Install hoast using npm after you have your project initialized.

$ npm install hoast

Command line interface

To learn about to hoast command run $ hoast -h.

Create a JSON configuration file with the options and modules.

{
  "options": {},
  "modules": {
    "read": {},
    "hoast-transform": {
      "patterns": "*.md"
    }
  }
}

See options for more detail about the property.

The modules property ...

If you want to re-use the same module multiple times you can wrap each module in their object and change the modules property in an array as seen below.

{
  "options": {},
  "modules": [
    {
      "read": {},
    }, {
      "hoast-transform": {
        "patterns": "A/*.md"
      }
    }, {
      "hoast-transform": {
        "patterns": "B/*.md"
      }
    }
  ]
}

Do not forget to install the modules as well.

Script

// Include library.
const Hoast = require(`hoast`);
// Get build-in module.
const read = Hoast.read;
 
// Initialize hoast.
Hoast(__dirname)
  // Add module.
  .use(read())
  // Start process.
  .process();

constructor(directory, options)

As with any constructor it initializes the object.

Returns the hoast instance.

• directory: The working directory, most often __dirname.
  • Type: String
  • Required: Yes
• options: See options for more detail about the parameter.
  • Type: Object
  • Required: No

use(module)

Adds the module to the modules stack.

Returns the hoast instance.

• module: See using modules and making modules for more detail about the parameter.
  • Type: Function
  • Required: Yes

process(options)

An asynchronous function which goes through the three steps mentioned in the introduction. It scans the files in the source directory, cycles through each module, and writes the result to the destination directory.

Returns the hoast instance.

• options: See options for more detail about the parameter.
  • Type: Object
  • Required: No

Asynchronously

hoast can be used asynchronously via script, two examples are given below.

const Hoast = require(`hoast`);
const read = Hoast.read;
 
Hoast(__dirname)
  .use(read())
  .process()
  // On end.
  .then(function(hoast) {
    console.log(`Processing successfully finished!`);
  })
  // On error.
  .catch(function(error) {
    console.error(error);
  });

const Hoast = require(`hoast`);
const read = Hoast.read;
 
const build = async function() {
  let hoast = Hoast(__dirname);
  try {
    await hoast
      .use(read())
      .process();
  }
  // On error.
  catch(error) {
    console.error(error);
  }
  // On end.
  console.log(`Processing successfully finished!`);
};
 
build();

As you can see in both examples the hoast object is still available for further usage.

options

The options object which can be given using the constructor or process functions.

• source The directory to process files from.
  • Type: String
  • Default: source
• destination: The directory to write the processed files to.
  • Type: String
  • Default: destination
• remove: Whether to remove all files in the destination directory before processing.
  • Type: String
  • Default: false
• patterns: Glob patterns to match directory and file paths with. If a directory or file path does not match during scanning of the source it will not be further explored.
  • Type: String or Array of strings
    • Default: []
• patternOptions: Options for the glob pattern matching. See planckmatch options for more details on the pattern options.
  • Type: Object
  • Default: {}
• patternOptions.all: This options is added to patternOptions, and determines whether all patterns need to match instead of only one.
  • Type: Boolean
  • Default: false
• concurrency: Maximum number of files to process at once.
  • Type: Number
  • Default: Infinity
• metadata: Metadata that can be used by modules.
  • Type: Object
  • Default: {}

The defaults are only applied when the constructor is called, the process' options parameter overrides what is set earlier.

Debugging

When making a hoast module I highly recommend activating the debug logs by setting up the environment variables as well as the debug module.

On Windows the environment variable is set using the set command.

set DEBUG=*,-not_this

Note that PowerShell uses different syntax to set environment variables.

$env:DEBUG = "*,-not_this"

Modules

As mentioned before the modules handle the logic that transforms the file information into the desired result. They are chained one after another and fed the results of the module that came before it. At the start the only information available is provided by the scanner function which performs a recursive search of the source directory and returns an array with objects of which an example can be seen below.

{
  path: `mark/down.md`,
  stats: {
    dev: 2114,
    ino: 48064969,
    mode: 33188,
    nlink: 1,
    uid: 85,
    gid: 100,
    rdev: 0,
    size: 527,
    blksize: 4096,
    blocks: 8,
    atimeMs: 1318289051000,
    mtimeMs: 1318289051000,
    ctimeMs: 1318289051000,
    birthtimeMs: 1532801006990
  }
}

For more detail about stats see the node documentation. The functions as well as the atime, mtime, ctime, and birthtime are not included.

Build-in

There is a single build-in module which reads the content of the files. It adds a content property as either a string or buffer depending if it is utf-8 encoded.

{
  content: {
    type: `string`,
    data: `Hello there.`
  }
}

{
  content: {
    type: `Buffer`,
    data: [ 71, 101, 110, 101, 114, 97, 108, 32, 75, 101, 110, 111, 98, 105, 46 ]
  }
}

This module has to be called before the process function or any module that expects the content property to be set.

Usage

The following example copies only the markdown files from the source directory to the destination directory.

const Hoast = require(`hoast`);
const read = Hoast.read,
      filter = require(`hoast-filter`);
 
Hoast(__dirname)
  // Filter to only include .md files.
  .use(filter({
    patterns: `*.md`
  }))
  .use(read())
  .process()

In this example you can see why the read function is not done automatically beforehand. Since you can now eliminate items from being further processed using the filter module before the read call is made.

You can re-use the modules after you have called process as seen below. The following script will copy all files from the sourceA and sourceB directories into the default destination directory.

const Hoast = require(`hoast`);
const read = Hoast.read;
 
Hoast(__dirname)
  // Read file content.
  .use(read())
  .process({
    source: `sourceA`
  })
  .then(function(hoast) {
    return hoast.process({
      source: `sourceB`
    });// Read will automatically be used again.
  });

If you want to start with a fresh set of modules all you have to do is remove out hoast.modules property. The following script first copies all markdown files from the default source directory to the destination directory, and then copies all text files from the same source directory to the destination directory.

const Hoast = require(`hoast`);
const read = Hoast.read,
      filter = require(`hoast-filter`);
 
Hoast(__dirname)
  // Filter out everything but markdown files.
  .use(filter({
    patterns: `*.md`
  })) // Only .md files available afterwards.
  .use(read())
  .process()
  .then(function(hoast) {
    // Removing out modules array.
    hoast.modules = [];
    return hoast
      // Filter out everything but text files.
      .use(filter({
        patterns: `*.txt`
      })) // Only .txt files available afterwards.
      .use(read())
      .process();
  });

See the examples directory for more in depth usage.

Making

In the simplest form the script below is a hoast module. The first time it will be called as a function and arguments can be passed on so properties can be initialized or validated. The return of the function is another function which will be called every time files need to be processed. hoast is the hoast instance and has the options property assigned via the constructor or process call. The files argument is an array files scanned and ready to me transformed.

module.exports = function(options) {
  // Prepare anything.
  
  // Return module.
  return function(hoast, files) {
    // Perform logic.
  };
};

For future proving and compatibility with the command-line tool use a single options parameter in the exported function.

See the remove module for an example.

Asynchronous

The modules can also be asynchronously by either adding the async keyword or using a promise.

module.exports = function(options) {
  
  // Return asynchronous module.
  return async function(hoast, files) {
    // Perform asynchronous logic.
  };
};

Overriding files

You can also return a new files array if you need to overwrite the existing one, however it is recommended to iterate over the files using the forEach function instead of map or filter. Use this power carefully!

module.exports = function(options) {
  
  return function(hoast, files) {
    
    // Override files.
    return files;
  };
};

See the filter module for an example.

hoast helpers

The hoast instance includes several helpers which can be used to perform task common across modules. These helpers can be accessed via the helper property of the hoast instance. See the list below for available functions.

• createDirectory(directory): Create a directory at the given path.
  • directory: The path to the desired directory.
    • Type: string
    • Required: Yes
• deepAssign(target, ...sources): Deeply assign two or more objects together.
  • target: Object to assign to.
    • Type: Object
    • Required: Yes
  • sources: Objects to read properties from.
    • Type: Object
    • Required: No
• matchExpressions(value, expressions, all): Match a value to regular expressions.
  • value: Value to match to.
    • Type: String
    • Required: Yes
  • expressions: Regular expressions to match with.
    • Type: String or Array of strings
    • Required: Yes
  • all: Whether all or any patterns need to match in order to return a positive result.
    • Type: Boolean
    • Default: false
• parsePatterns(patterns, options, path): Parse glob patterns into regular expressions.
  • patterns: Glob patterns.
    • Type: String or Array of strings
    • Required: Yes
  • options: Parse options, see planckmatch's parse options for more detail.
    • Type: Object
    • Required: No
• removeFiles(files): Remove directories and/or files from storage.
  • files: Paths of directories and/or files.
    • Type: Array of strings
    • Required: Yes
• removeFiles.single(files): Remove a directory or file from storage.
  • file: Path of directory or file.
    • Type: String
    • Required: Yes
• scanDirectory(directory, expressions, all): Retrieve stats of files within a given directory.
  • directory: Path to directory.
    • Type: String
    • Required: Yes
  • expressions: Regular expressions to match with.
    • Type: String or Array of strings
    • Required: No
  • all: Whether all or any expressions need to match.
    • Type: Boolean
    • Default: false
• writeFiles(directory, files): Write files to storage.
  • directory: Path to directory where files are to be written to.
    • Type: String
    • Required: Yes
  • files: List of files with path property with the relative path to the given directory and a content property with the files content.
    • Type: Array of objects
    • Required: Yes
• writeFiles.single(directory, file): Write file to storage.
  • directory: Path to directory where file is to be written to.
    • Type: String
    • Required: Yes
  • file: Object with path property with the relative path to the given directory and a content property with the file's content.
    • Type: Object
    • Required: Yes

Before and after

Some modules might require to be perform logic after all options are initialized or after all files are written to storage. To accommodate this you can add a before and after function to your main method. These functions can also be asynchronously and are called in the order the modules were added.

module.exports = function(options) {
  
  const method = function(hoast, files) {
    // Perform logic across files.
  };
  
  method.before = function(hoast) {
    // Called before the files are scanned from storage.
  };
  
  method.after = function(hoast) {
    // Called after the files are written to storage.
  };
  
  // Return functions.
  return method.
};

See the changed module for an example.

Known issues

• Access modes of directories and files are not transferred.