tryorama
An end-to-end/scenario testing framework for Holochain applications, written in TypeScript.
Tryorama allows you to write test suites about the behavior of multiple Holochain nodes which are networked together, while ensuring that test nodes in different tests do not accidentally join a network together.
npm install @holochain/tryorama
Take a look at the sample below, or skip to the Conceptual Overview for a more in depth look.
Sample usage
Check out this heavily commented example for an idea of how to use tryorama
import { Orchestrator, Config } from '../../src'
// Point to your DNA file and give it a nickname.
// The DNA file can either be on your filesystem...
const dnaBlog = Config.dna('~/project/dnas/blog.dna.json', 'blog')
// ... or on the web
const dnaChat = Config.dna('https://url.to/your/chat.dna.json', 'chat')
// Set up a Conductor configuration using the handy `Conductor.config` helper.
// Read the docs for more on configuration.
const mainConfig = Config.gen(
{
blog: dnaBlog, // agent_id="blog", instance_id="blog", dna=dnaBlog
chat: dnaChat, // agent_id="chat", instance_id="chat", dna=dnaChat
},
{
// specify a bridge from chat to blog
bridges: [Config.bridge('bridge-name', 'chat', 'blog')],
}
})
// Instatiate a test orchestrator.
// It comes loaded with a lot default behavior which can be overridden, including:
// * custom conductor spawning
// * custom test result reporting
// * scenario middleware, including integration with other test harnesses
const orchestrator = new Orchestrator()
// Register a scenario, which is a function that gets a special API injected in
orchestrator.registerScenario('proper zome call', async (s, t) => {
// Declare two players using the previously specified config,
// and nickname them "alice" and "bob"
const {alice, bob} = await s.players({alice: mainConfig, bob: mainConfig})
// You have to spawn the conductors yourself...
await alice.spawn()
// ...unless you pass `true` as an extra parameter,
// in which case each conductor will auto-spawn
const {carol} = await s.players({carol: mainConfig}, true)
// You can also kill them...
await alice.kill()
// ...and re-spawn the same conductor you just killed
await alice.spawn()
// now you can make zome calls,
await alice.call('chat', 'messages', 'direct_message', {
content: 'hello world',
target: carol.agentAddress('chat')
})
// you can wait for total consistency of network activity,
await s.consistency()
// and you can make assertions using tape by default
const messages = await carol.call('chat', 'messages', 'list_messages', {})
t.equal(messages.length, 1)
})
// Run all registered scenarios as a final step, and gather the report,
// if you set up a reporter
const report = await orchestrator.run()
// Note: by default, there will be no report
console.log(report)
Conceptual overview
To understand Tryorama is to understand its components. Tryorama is a test Orchestrator for writing tests about the behavior of multiple Holochain nodes which are networked together. It allows the test writer to write Scenarios, which specify a fixed set of actions taken by one or more Players, which represent Holochain nodes that may come online or offline at any point in the scenario. Actions taken by Players include making zome calls and turning on or off their Holochain Conductor.
Orchestrators
Test suites are defined with an Orchestrator
object. For most cases, you can get very far with an out-of-the-box orchestrator with no additional configuration, like so:
import {Orchestrator} from '@holochain/try-o-rama'
const orchestator = new Orchestrator()
The Orchestrator constructor also takes a few parameters which allow you change modes, and in particular allows you to specify Middleware, which can add new features, drastically alter the behavior of your tests, and even integrate with other testing harnesses. We'll get into those different options later.
The default Orchestrator, as shown above, is set to use the local machine for test nodes, and integrates with the tape
test harness. The following examples will assume you've created a default orchestrator in this way.
Scenarios
A Tryorama test is called a scenario. Each scenario makes use of a simple API for creating Players, and by default includes an object for making tape assertions, like t.equal()
. Here's a very simple scenario which demonstrates the use of both objects.
// `s` is an instance of the Scenario API
// `t` is the tape assertion API
orchestrator.registerScenario('description of this scenario', async (s, t) => {
// Use the Scenario API to create two players, alice and bob (we'll cover this more later)
const {alice, bob} = await s.players({alice: config, bob: config})
// start alice's conductor
await alice.spawn()
// make a zome call
const result = await alice.call('some-instance', 'some-zome', 'some-function', 'some-parameters')
// another use of the Scenario API is to automagically wait for the network to
// reach a consistent state before continuing the test
await s.consistency()
// make a test assertion with tape
t.equal(result.Ok, 'the expected value')
})
Each scenario will automatically kill all running conductors as well as automatically end the underlying tape test (no need to run t.end()
).
Players
A Player represents a Holochain user running a Conductor. Therefore, the main concern in configuring a Player is providing configuration for its underlying Conductor.
Conductor configuration
Much of the purpose of Tryorama is to provide ways to generate conductor configurations (TODO: we need documentation on conductor configs) that meet the following criteria:
- Common configs should be easy to generate
- Any conductor config should be possible
- Conductors from different scenarios must remain independent and invisible to each other
Config
helper
Simple config with the
- Common configs should be easy to generate
Let's look a common configuration. Here is an example of how you might set up three Players, all of which share the same conductor config which defines a single DNA instance named "chat" using a DNA file at a specified path -- a reasonable setup for writing scenario tests for a single DNA. It's made really easy with a helper called Config.gen
.
import {Config, Orchestrator} from '@holochain/try-o-rama'
const orchestrator = new Orchestrator()
// Config.gen is a handy shortcut for creating a full-fledged conductor config
// from as little information as possible
const commonConfig = Config.gen({
// `Config.dna` generates a valid DNA config object, i.e. with fields
// "id", "file", "hash", and so on
chat: Config.dna('path/to/chat.dna.json')
})
orchestrator.registerScenario(async (s, t) => {
const {alice, bob, carol} = await s.players({
alice: commonConfig,
bob: commonConfig,
carol: commonConfig,
})
})
Config.gen
also takes an optional second parameter. The first parameter allows you to specify the parts of the config which are concerned with DNA instances, namely "agents", "dnas", "instances", and "interfaces". The second parameter allows you to specific how the rest of the config is generated. Config
also has other helpers for generating other parts. For instance, to turn off logging, there is an easy way to do it like so:
const commonConfig = Config.gen(
{
chat: Config.dna('path/to/chat.dna.json')
},
{
logger: Config.logger(false)
}
)
Config.gen
offers a lot of flexibility, which we'll explore more in the next sections.
Config.gen
More fine-grained instance setup with
- Any conductor config should be possible
Config.gen
can be used in a slightly more explicit way. The first argument can take an object, as shown, which is a handy shorthand for quickly specifying instances with DNA files. If you need more control, you can define instances in a more fine-grained way using an array:
const dnaConfig = Config.dna('path/to/chat.dna.json', 'chat')
// this
Config.gen({
myInstance: dnaConfig
myOtherInstance: dnaConfig
})
// is equivalent to this
Config.gen([
{
id: 'myInstance',
agent: {
id: 'myInstance',
name: name1, // NB: actually generated by Tryorama, it's necessary for agent names to be distinct across all conductors...
public_address: 'HcS----------...',
keystore_file: 'path/to/keystore',
},
dna: {
id: 'chat',
file: 'path/to/chat.dna.json',
}
},
{
id: 'myOtherInstance',
agent: {
id: 'myOtherInstance,
name: name2, // NB: actually generated by Tryorama, it's necessary for agent names to be distinct across all conductors...
public_address: 'HcS----------...',
keystore_file: 'path/to/keystore',
},
dna: {
id: 'chat',
file: 'path/to/chat.dna.json',
}
}
])
Advanced setup with configuration seeds
- Any conductor config should be possible
If you need your conductor to be configured in a really specific way, fear not, Tryorama can handle that. However, you'd better have a good understanding of how Holochain conductor configuration works, as well as what requirements Tryorama has in order to run tests. In the previous example we used Config.gen
to create configuration with as little hassle as possible. Let's see what's going on under the hood and how we can write a fully customized conductor config. But, let's also remember the third point:
- Conductors from different scenarios must remain independent and invisible to each other
To achieve the independence of conductors, Tryorama ensure that various values are unique. It uses UUIDs during DNA config as well as for Agent IDs to ensure unique values; it ensures that it automatically creates temp directories for file storage when necessary, adding the paths to the config. So how can we let Tryorama handle these concerns while still generating a custom config? The answer is in a key concept:
Players are configured by giving them functions that generate their config. For instance, when you call Config.gen
, it's actually creating a function for you like this:
// this
const config = Config.gen({alice: dnaConfig})
// becomes this
const config = ({playerName, uuid, configDir, adminPort, zomePort}) => {
return {
persistence_dir: configDir,
agents: [/* ... */],
dnas: [/* ... */],
instances: [/* ... */],
interfaces: [/* ... */],
network: {/* ... */},
// and so on...
// basically, a complete Conductor configuration in JSON form
}
})
Such a function is called a config seed, since the function is reusable across all scenarios.
Config seeds take an object as a parameter, with five values:
-
scenarioName
(TODO): the name of the current scenario, i.e.registerScenario(scenarioName, ...)
-
playerName
: the name of the player for this conductor, e.g."alice"
-
uuid
: a UUID which is guaranteed to be the same within a scenario but unique between different scenarios -
configDir
: a temp dir created specifically for this conductor -
adminPort
: a free port on the machine which is used for the admin Websocket interface, used to get privileged info from the conductor -
zomePort
: a free port on the machine which is used for the normal Websocket interface, used to e.g. make zome calls
The constraints generated configs must abide by
Under the hood, Tryorama generates unique and valid values for these parameters and generates unique configurations by injecting these values into the seed functions. If you are writing your own config seed, you can use or ignore these values as needed, but you must be careful to set things up in a way that Tryorama can work with to drive the test scenarios:
- There must be an admin interface running over WebSockets at
adminPort
- There must be an interface running over WebSockets at
zomePort
including all instances - All agents within a scenario must have a unique name (even across different conductors!)
- You must incorporate the UUID or some other source of uniqueness into the DNA config's
uuid
field, to ensure that conductors in different tests do not attempt to connect to each other on the same network
Config.gen
Using seed functions in Since configuring a full config that properly uses these injected values is really tedious and error-prone, especially for the part concerning agents and instances, Config.gen
also accepts functions using the usual seed arguments. So if you need to set up your dpki config using some of these values, you could do so:
Config.gen(
{alice: dnaConfig},
({playerName, uuid}) => {
return {
dpki: {
instance_id: 'my-instance',
init_params: JSON.stringify({
someValueThatNeedsToBeUnique: uuid,
someValueThatWantsToBeThePlayerName: playerName,
})
}
}
}
)
You can also use seed functions in the first parameter of Config.gen
TODO: verify and document this