Autobot: Bot Testing for Humans
Autobot is a multi-platform bot testing framework designed for humans. It is designed to have concise, simple test files with strong flexibility and extensibility.
Features:
- Human readable/writable YAML dialogue files
- Supports any bot platform.
- Comes with connectors for botframework
- Implement a new connector with only two methods:
send
andonReply
.
- Parallel execution of tests
- Test against multiple acceptable answers
- Wildcard matching (
<NUMBER>
,<WORD>
,<*>
) - Support for testing rich attachments (
<IMAGE>
,<CARDS>
) - Support for regular expressions (
<(st|nt|nd)>
) - Show diffs on error
- Conversation branches
- Full unicode support
- Support for preambles and loading variables from locale/translation files
Dialogue Format
Simple Chat file
Title: "Simple Greetings"Dialogue: - Human: Hello - Bot: - Hi - Hello - Bot: How are you? - Human: I'm good - Human: Yourself? - Bot: I'm good too!
Complex chat file with variables, branching, and wildcards
Title: "Branching conversation"Dialogue: - Human: Hello - Bot: Hi <WORD> // Wildcard match - Human: How are you? - Bot: I am <$emotion> // variable loaded from external file - Wait: 5000 - Human: How's my campaign doing? - Bot: Your impressions are up 20% since last week! Would you like to see more? - 1: - Human: Sure - Bot: Here's a graph of your impressions in the last month. - Bot: <IMAGE> 2: - Human: No - Bot: Anything else I can help with? - Human: What countries is this campaign live in? - 1: - Bot: Your campaign is live in <NUMBER> countries right now - Human: Show me ad spend by country - Bot: Here you go - Bot: <IMAGE> 2: - Bot: Your campaign is only live in 1 country right now - Human: Which one? - Bot: <WORD>
Special Tags
You can use the following tags in your test dialogue files:
Tag | Meaning |
---|---|
<*> |
Matches anything, including whitespaces |
<WORD> |
A single word without whitespaces |
<IMAGE> |
An image attachment |
<CARDS> |
A card attachment |
<(REGEX)> |
Any regex expression, i.e. <([0-9]{2})> |
<$VARNAME> |
An expression from the locale/translation file |
Install
To install from npm
$ npm install -g auto-bot
To install from source
$ npm run build
$ npm install -g ./
Usage
You must have a config file autobot.yml
in the current directory or a parent directory.
$ autobot
Usage: autobot [options] <chatPath>
autobot is an multi-platform bot testing framework. It requires an autobot.yml config file to be in the working directory or a parent.
Options:
-V, --version output the version number
-v --verbose Enable full logging including bot queries and responses
-j --json Enable seeing the direct JSON responses from the client
-c, --config <autobot.yml> autobot.yml config file to use (default current directory and parents)
-h, --help output usage information
To run a specific test:
$ autobot ./path/to/test.yml
To run all tests recursively in a directory:
$ autobot ./path/to/directory
To see the dialogue in the console, add the -v
flag
$ autobot -v ./path/to/test.yml
Discovered 1 tests (1 branches)
HUMAN testuser-simple-0: Hello
BOT testuser-simple-0: Hi!
Config File Format
The config file autobot.yml
takes the following fields:
client: botframework
directLineSecret: <SECRET IF USING BOT FRAMEWORK>
localeFiles:
- locale/en/locale.json
timeout: 20000 (timeout in ms)
preamble:
- Human: Hey there
- Bot: <*>
- Human: Hi
Extending to a new platforms
autobot is designed to be super simple to integrate a new platform. You only need to implement the Client
interface
found in src/clients/client_interface.ts
, in either javascript or typescript.
There are only 2 mandatory methods, and 2 optional methods to implement:
export class MyNewClient extends Client {
send(message) {
// Mandatory
}
onReply(callback) {
// Mandatory
}
onReady(callback) {
// Optional
}
close() {
// Optional
}
}