Zephyr Binary Log Transformer

Intro

This library implements a parser and two transform streams (one for browser and one for node.js) that can read binary log format of Zephyr RTOS (not pushed to main stream yet) and print it in a human-readable form.

The purpose of the binary format is to free space used by the log buffers on memory-constrained chips. To turn the binary format on use LOG_FORMAT_BINARY=y option in your KConfig.

In order for the library to be able to transform string pointers to the text, you need to provide the parser with a string map and the base address of the strings section in the RAM. Check LogParserOptions interface for more details.

Usage

To install the library use npm as usual:

npm install zephyr-binary-log-transformer --save

You may either use the parser directly or use one of the wrapping transform streams instead:

Node.js transform stream

Below is an example of how to implement a log transformer using JLink RTT Telnet in node.js.

The first step involves parsing a string map from a JSON file. The rodata_data field includes the string map in format "offset": "string value" and the rodata_sh_addr contains the address of the rodata section in the memory.

import * as fs from 'fs';
import * as net from 'net';
import { promisify } from 'util';
import { ParserEventLevel } from 'zephyr-binary-log-transformer/lib/parser';
import { LogStreamNode } from 'zephyr-binary-log-transformer/lib/log-stream-node';

const readFile = promisify(fs.readFile);

async function doIt() {
    const filePath = process.argv[2] || './zephyr-resources.json';

    try {
        const fileContent = await readFile(filePath);
        const data = JSON.parse(fileContent.toString('ascii'));
        const stringMap = new Map();
        Object.entries(data.rodata_data).forEach(([key, value]) => stringMap.set(+key, value));
        const stringsOffset = BigInt(data.rodata_sh_addr);

        const stream = new LogStreamNode({ stringMap, stringsOffset });
        stream.addEventListener((level, event) => console.log(`[${level}]: ${event}`), ParserEventLevel.Warning);
        stream.on('data', data => console.log(data.toString('ascii')));

        const socket = new net.Socket();
        socket.connect(19021, 'localhost');
        socket.pipe(stream);
    } catch (err) {
        console.error('Failed to read resources', err);
    }
}

doIt().then(
    () => console.log('Finished')
)  

Browser transform stream

Below is an example of how you can implement similar functionality in a web browser. This time the source is Web Serial API.

import { LogStreamBrowser } from 'zephyr-binary-log-transformer/lib/log-stream-browser';

async function startLogCapture() {
  const port = await navigator.serial.requestPort();
  await port.open(options);
  const transformer = new LogStreamBrowser({ stringMap, stringsOffset });
  const stream = port.readable.pipeThrough(new TransformStream(transformer));
  const reader = stream.getReader();

  readLoop(reader);
}

async function readLoop(reader) {
  while (true) {
    const { value, done } = await reader.read();
    if (value) {
      appendLogLine(value);
    }
    if (done) {
      reader.releaseLock();
      break;
    }
  }
}

Parser

If the transform streams don't fit your needs, you may use the parser directly. It's API is pretty straight-forward

import { LogParser, ParserEventLevel, ParserEventListener } from './parser';

const parser = new LogParser({ stringMap, stringsOffset });
parser.addMessageListener(msg => console.log(msg));         // Emitted for every transformed log line
parser.addEventListener(listener, ParserEventLevel.Debig);  // Optional, it may be used for diagnostics, details follow in the description

parser.feed(data);  // Send chunk of binary data to the parser any ArrayLike<number> value can be used

API Details

LogParser

Constructor

The constructor accepts a single parameter of type LogParserOptions, it has the following members:

• stringMap: Map<number, string> (required): Maps offsets in the strings section in the firmware memory to concrete strings
• stringsOffset: biging (required): Offset of the strings section in the firmware memory
• emitIgnored: boolean (optional): If true, the parser will emit messages from characters it ignored (default is false)

The emitIgnored flag may come handy when the incoming data does also include some raw strings that are not binary encoded. This is for example the case of the JLink RTT Telnet where the incoming data start with a string header describing the version of JLink and data may be interleaved with string messages informing about dropped log lines. If the parser is waiting for the message preamble (0xfe) and this flag is set, it stores all the incoming characters different from 0xfe to a buffer. Once it receives the message preamble, it emits this whole buffer (parsed as string) as a message.

Message Event

All the transformed messages (and eventually ignored characters) are emitted asynchronously through this event. You attach a listener using addMessageListener(listener: MessageListener): void and eventually remove it using removeMessageListener(listener: MessageListener): void.

Diagnostic Events

If you have troubles with your log output, you may want to check the diagnostic events of the parser. Events have different levels from debug to error:

• Error: Emitted only under serious conditions when something prevents parsing the message. Examples are missing entry in the string map or wrong binary data.
• Warning: Emitted when parser ignores some incoming data
• Info: Informative messages, currently not used
• Debug: Detailed information about the parser state, incoming data and their handling

You can listen to diagnostic events using addEventListener(listener: ParserEventListener, minLevel: ParserEventLevel): void. minLevel specifies the minimum event level that you are interested in with debug being the lowest and error the highest. The listener then receives both the event level and event message (type ParserEventListener = (level: ParserEventLevel, message: string) => void;).

Data Feed

You feed the data into the parser using the feed(chunk: ArrayLike<number>): void method. You may feed the data in as they arrive, no matter if they are aligned to messages or not.

LogStreamNode

Both streams are essentially just wrappers around the LogParser class that adapt it to the appropriate stream API.

Constructor

You instantiate the LogStreamNode class using the same LogParserOptions interface as the LogParser class.

'data' event

Instead of the message listener, you consume the standard 'data' event of the stream.

Diagnostics

The stream still exposes the diagnostic events using the addEventListener(listener: ParserEventListener, minLevel: ParserEventLevel): void

Feeding data

To feed the data into the parser you write to the stream. Standard usage of the transform stream is to pipe it after your input stream.

LogStreamBrowser

Constructor

You instantiate the LogStreamBrowser class using the same LogParserOptions interface as the LogParser class.

Reader

Instead of the message listener, you use the stream reader using stream.getReader() and reading from it in a loop as shown in the example above.

Diagnostics

The stream still exposes the diagnostic events using the addEventListener(listener: ParserEventListener, minLevel: ParserEventLevel): void

Feeding data

To feed the data into the parser you write to the stream. Standard usage of the transform stream is to pipe your input stream through the transformer, e.g.

readable.pipeThrough(new TransformStream(new LogStreamBrowser(stringMap, stringsOffset)))

Format description

• 1B Message Preamble (0xfe)
• 1B Message Header
  • 1 Top-most bit = message type (0 = STD, 1 = HEXDUMP)
  • 3 bits = severity (0 = None, 1 = Error, 2 = Warning, 3 = Info, 4 = Debug)
  • 4 bits = variadic arguments count (for STD message)
• 4B Timestamp
• 8B Pointer to message string

STD Message Variadic Arguments:

• 1B Header
  • Topmost bit = type (0 = Uint, 1 = StringZ)
  • 7 lower bits = length in bytes (for Uint argument)
• Argument value

HEXDUMP Message:

• 4B length of the data
• Data

RAW String messages are STD message with severity none, timestamp and string pointer 0 and 1 variadic StringZ argument