⭐ Star us on GitHub — it motivates us a lot! 🚀
This package provides a Scramjet Command Line Interface to communicate with Transform Hub and Cloud Platform. The document is focused mainly on the Scramjet Transform Hub part.
-
Install the Scramjet CLI.
npm install -g @scramjet/cli
Once installed, the Scramjet CLI is available under the
si
command that stands for Scramjet Interface. -
Install the autocompletion script.
To use Scramjet CLI with command hints, please install completion script. It depends on bash-completion so first make sure it's already installed by runningtype _init_completion
.Below command installs completion script in
~/.bashrc
.si completion install
Running command
si completion bash
prints script to the terminal.
- Show help
- Manage config
- Show Scramjet Hub parameters
- Sequence operations
- Instance operations
- Topic operations
Check the available commands by typing si --help
in the terminal.
Usage: si [options] [command]
This is a Scramjet Command Line Interface to communicate with Transform Hub and Cloud Platform.
Options:
-v, --version Display current CLI version
--config <name> Set global configuration profile
--config-path <path> Set global configuration from file
-h, --help display help for command
Commands:
hub Allows to run programs in different data centers, computers or devices in local network
config|c Config contains default Scramjet Transform Hub (STH) and Scramjet Cloud Platform (SCP) settings
sequence|seq Operations on a Sequence package, consisting of one or more functions executed one after another
instance|inst [command] Operations on the running Sequence
topic Manage data flow through topics operations
completion completion operations
util|u Various utilities
Show subcommand help by providing --help
or -h
option after each as in example below.
Usage
si [command] --help
Example
si sequence -h
Config contains default Scramjet Transform Hub (STH) and Scramjet Cloud Platform (SCP) settings.
There are 3 configuration types inside CLI for different purpose:
- Default configuration - contains default values for the Scramjet Transform Hub (STH) settings.
- Profile configuration - contains personalized values for the Scramjet Transform Hub (STH) and Scramjet Cloud Platform settings.
- Session configuration - contains values for the current session.
Default SI configuration is a global configuration file used by all CLI processes. This is a basic configuration used by CLI when no other is chosen or set.
This is an example of a 'default.json' file:
{
"configVersion": 1,
"apiUrl": "http://127.0.0.1:8000/api/v1",
"middlewareApiUrl": "",
"env": "development",
"scope": "",
"token": "",
"log": {
"debug": false,
"format": "pretty"
}
}
Properties description:
Property | Description | Default value |
---|---|---|
configVersion |
A version of the config. | 1 |
apiUrl |
Hub API url. Applicable only to self hosted Hub. | "http://127.0.0.1:8000/api/v1" |
middlewareApiUrl |
Scramjet Cloud Platform Space url. To be set during authorization process. | "" |
env |
Set the environment you will work in: - "production" → for Scramjet Cloud Platform environment, - "development" → to communicate with Transform Hub (self hosted Hub environment). |
"development" |
scope |
A default scope that should be used when session starts. | "" |
token |
Scramjet Cloud Platform authorization token. To be set during authorization process. | "" |
log |
"log" property contains two additional properties: - "debug" → if error occurs stack trace can be shown/hide with the "true" /"false" option, - "format" → log messages display can be set to "pretty" or "json" . |
"debug" : false "format" : "pretty"
|
Every value can be set for every config property by using command si config set [command]
The CLI development environment communicates with the Transform Hub. If this is your first installation of Scramjet CLI, the environment should already be set to development since it is the default value of the CLI environment.
💡 An environmental value that is set to production allows to use commands of the Scramjet Cloud Platform. We encourage you to sign up for the SCP Beta.
If you wish to use the open source STH on your own machine, please make sure the STH API URL is set correctly in the CLI. By default the STH starts the API server on http://0.0.0.0:8080/api/v1
. The API URL can be set in the CLI with the following command:
si config set apiUrl http://0.0.0.0:8080/api/v1
💡 An URL pattern looks like this:
http://<localhost|IPaddress>:<portNumber>/api/v1
Profile configuration refers to all configurations that can be manipulated and set by user. Profile configuration can be provided in two ways:
-
Path to outside configuration file provided by user (read only file, requires all configuration fields in file):
si <someCliCommand> --config-path some/path/toFile
process.env: SI_CONFIG_PATH=some/path/toFile si <someCliCommand>
Below the example of a
profile-config.json
file saved on the disk:{ "configVersion": 1, "apiUrl": "http://127.0.0.1:8000/api/v1", "middlewareApiUrl": "https://api.beta.scramjet.cloud/api/v1", "env": "production", "scope": "", "token": "generatedToken", "log": { "debug": false, "format": "pretty" } }
and its use from the command line:
# check profile $ si c profile ls Available profiles: default -> production # check current profile settings $ si c p Current profile: production { configVersion: 1, apiUrl: 'http://127.0.0.1:8000/api/v1', middlewareApiUrl: '', env: 'development', scope: '', token: '', log: { debug: false, format: 'pretty' } } # si command execution with external profile configuration $ si hub info --config-path /home/user/profile-config.json { id: 'sth-0', info: { created: '2022-08-10T09:22:42.011Z', lastConnected: '2022-08-10T09:22:42.014Z' }, healthy: true, isConnectionActive: true }
$ SI_CONFIG_PATH=/home/user/profile-config.json si hub info { id: 'sth-0', info: { created: '2022-08-10T09:22:42.011Z', lastConnected: '2022-08-10T09:22:42.014Z' }, healthy: true, isConnectionActive: true }
The examples above show that CLI uses the external profile configuration provided by user over the configuration set in CLI profile.
-
Profile name to configuration handled and manipulated by CLI resources files:
si <someCliCommand> --config <profileName>
$ si c profile ls Available profiles: default -> production testing $ si hub version --config testing { service: '@scramjet/host', apiVersion: 'v1', version: '0.27.0', build: 'becd14d' }
process.env: SI_CONFIG=profileName si <someCliCommand>
$ si c profile ls Available profiles: default -> production testing $ SI_CONFIG=testing si hub version { service: '@scramjet/host', apiVersion: 'v1', version: '0.27.0', build: 'becd14d' }
-
si config profile use <profileName>
- sets a<profileName>
as a default profile for all shells
$ si config profile use testing $ si config profile ls Available profiles: default production -> testing
There is always at least a default profile file ('default.json') if none of the above method is used to set config profile. Order of precedence for profile configuration is added (if profile config is set with many options only highest precedence is used).
When user sets any value in si config
, by calling si config set [command]
, for example: si config set log --format pretty
the changes will appear in the currently used profile. If config is provided from user file, exception will be thrown:
$ si c set env development --config-path /home/user/my-profile-config.json
Error: Unable to write for read only configuration
It means that si config set [command]
will change the profile config properties only, not the properties in the config file provided by user.
Profiles can be managed by a CLI command si config profile [command]
Creating profiles is optional. As mentioned before, there is always a default config available. CLI works with it when no profile is created, but it can be quite useful to have some profiles when working with multiple environments, scopes or spaces at the same time.
This is the list of available si config profile
commands.
list|ls # Show available configuration profiles
use <name> # Set configuration profile as default to use
create <name> # Create new configuration profile
remove <name> # Remove existing profile configuration
To create a profile:
- Create a new profile using command
si config profile create <profileName>
. When creating a new profile, next to the default configuration file named 'default.json', a new configuration file named 'newProfile.json' is created with all the property values set to default. After creation, the new profile is NOT automatically set as default. You have to switch to the target profile if you want to set new values for it. - List all profiles using command
si config profile list
to see your new profile on the list. The arrow points at the profile which is currently in use. Also a 'si-config.json' file indicates which profile is currently in use. - Switch to the new profile using command
si config profile use profileName
This is how the above steps execution would look in the terminal:
# list profiles
$ si config profile ls
Available profiles:
-> default
# create profile named 'production'
$ si config profile create production
# create profile named 'development'
$ si config profile create development
# list profiles
$ si config profile ls
Available profiles:
-> default
development
production
# switch to 'development' profile
$ si config profile use development
# list profiles to confirm profile change
$ si config profile ls
Available profiles:
default
-> development
production
- Set new values. You can set new values in your profile by using command
si config set [command]
. - Display your config content with
si config print
(too lazy to type? Roger that! Usesi c p
instead).
# print out current profile config
$ si c p
Current profile: development
{
configVersion: 1,
apiUrl: 'http://127.0.0.1:8000/api/v1',
middlewareApiUrl: '',
env: 'development',
scope: '',
token: '',
log: { debug: false, format: 'pretty' }
}
# change apiUrl property value
$ si config set apiUrl "http://127.0.0.1:9000/api/v1"
# set debug property to true
$ si config set log --debug true
# set log format to json
$ si config set log --format json
# print out the config again to see the changes
$ si c p
Current profile: development
{"configVersion":1,"apiUrl":"http://127.0.0.1:9000/api/v1","middlewareApiUrl":"","env":"development","scope":"","token":"","log":{"debug":true,"format":"json"}}
- Delete profile with
si config profile remove <profileName>
. If you are deleting the profile you are currently using, you will be switched to thedefault
profile right after the deletion.
# list profiles
$ si config profile ls
Available profiles:
default
-> development
production
# remove profile 'development'
$ si config profile remove development
# list profiles again to check if the profile was removed
$ si config profile ls
Available profiles:
-> default
production
- It is possible to reset all config values at once by using command
si config reset all
.
# print out current profile config
$ si c p
Current profile: development
{
configVersion: 1,
apiUrl: 'http://127.0.0.1:9000/api/v1',
middlewareApiUrl: '',
env: 'production',
scope: '',
token: '',
log: { debug: true, format: 'pretty' }
}
# reset all the config values
$ si config reset all
# print out the config again to see the changes
$ si c p
Current profile: development
{
configVersion: 1,
apiUrl: 'http://127.0.0.1:8000/api/v1',
middlewareApiUrl: '',
env: 'development',
scope: '',
token: '',
log: { debug: false, format: 'pretty' }
}
Session configuration is a configuration used inside of CLI to handle all lastXxx
parameters. Configuration is kept alive in terminal (shell session time of life) that uses CLI (as outcome every terminal will have it's own lastXXX params). With this solution, you can move freely between terminal sessions and communicate with your Hubs and Sequences even more effectively.
A session configuration file named session-config.json
is created locally in the temporary directory named after session id, for example: /tmp/.si-tmp/4020249
. After the session terminates, the directory is removed along with its contents. The configuration session-config.json
file is created at the moment of initiating the command related to packing and running the Sequence. This action results in the first write to the session configuration file as a value assignment to the parameters "lastPackagePath"
, "lastInstanceId"
and "lastSequenceId"
.
To print out the current session configuration, use si config session
command.
{
lastPackagePath: '',
lastInstanceId: '',
lastSequenceId: '',
lastSpaceId: '',
lastHubId: '',
sessionId: '4099770'
}
"lastInstanceId"
and "lastSequenceId"
values are set automatically while running si seq deploy
or si seq send
and si seq start
commands. They can be also set manually with commands:
-
si seq use <sequenceID>
-
si inst use <instanceID>
This becomes useful when you want to change the Sequence or Instance ID assigned to alias (-
).
Lets set an example to show how it works.
# print out current session config
$ si config session
{
lastPackagePath: '',
lastInstanceId: '',
lastSequenceId: '',
lastSpaceId: '',
lastHubId: '',
sessionId: '4125483'
}
# deploy a sequence
$ si seq deploy /hello.tar.gz
SequenceClient {
_id: '0f424150-5f01-420a-85d0-54ebf4701fc3',
host: HostClient {
apiBase: 'http://127.0.0.1:8000/api/v1',
client: ClientUtils {
apiBase: 'http://127.0.0.1:8000/api/v1',
fetch: [Function (anonymous)],
normalizeUrlFn: [Function: normalizeUrl]
}
},
sequenceURL: 'sequence/0f424150-5f01-420a-85d0-54ebf4701fc3'
}
InstanceClient {
host: HostClient {
apiBase: 'http://127.0.0.1:8000/api/v1',
client: ClientUtils {
apiBase: 'http://127.0.0.1:8000/api/v1',
fetch: [Function (anonymous)],
normalizeUrlFn: [Function: normalizeUrl]
}
},
_id: '907ed318-7260-47ce-9b83-6bd95da60429',
instanceURL: 'instance/907ed318-7260-47ce-9b83-6bd95da60429'
}
# print out the session config again to check if the values have been assigned
$ si config session
{
lastPackagePath: '',
lastInstanceId: '907ed318-7260-47ce-9b83-6bd95da60429',
lastSequenceId: '0f424150-5f01-420a-85d0-54ebf4701fc3',
lastSpaceId: '',
lastHubId: '',
sessionId: '4125483'
}
# start the sequence again using alias `-`
$ si seq start -
InstanceClient {
host: HostClient {
apiBase: 'http://127.0.0.1:8000/api/v1',
client: ClientUtils {
apiBase: 'http://127.0.0.1:8000/api/v1',
fetch: [Function (anonymous)],
normalizeUrlFn: [Function: normalizeUrl]
}
},
_id: '75fdb61a-e473-4af6-a0b1-59e2b6c1486b',
instanceURL: 'instance/75fdb61a-e473-4af6-a0b1-59e2b6c1486b'
}
# check session config, after starting the sequence again, lastInstanceId is set to the new value
$ si config session
{
lastPackagePath: '',
lastInstanceId: '75fdb61a-e473-4af6-a0b1-59e2b6c1486b',
lastSequenceId: '0f424150-5f01-420a-85d0-54ebf4701fc3',
lastSpaceId: '',
lastHubId: '',
sessionId: '4125483'
}
# change lastInstanceId to another instance id
$ si inst use 907ed318-7260-47ce-9b83-6bd95da60429
# print out the session config again to check if the lastInstanceId has been changed
$ si config session
{
lastPackagePath: '',
lastInstanceId: '907ed318-7260-47ce-9b83-6bd95da60429',
lastSequenceId: '0f424150-5f01-420a-85d0-54ebf4701fc3',
lastSpaceId: '',
lastHubId: '',
sessionId: '4125483'
}
Configuration file session-config.json
creation can be also initialized while setting up the environment for Scramjet Cloud Platform, for example:
$ si config session
{
lastPackagePath: '',
lastInstanceId: '',
lastSequenceId: '',
lastSpaceId: '',
lastHubId: '',
sessionId: '340908'
}
$ si config set json '{"middlewareApiUrl": "https://api.beta.scramjet.cloud/api/v1", "env": "production", "token": "generatedToken"}'
$ si config session
Defaults set to: Space: org-a10d5cb5-4ca53c2e2a05-manager, Hub: sth-0
{
lastPackagePath: '',
lastInstanceId: '',
lastSequenceId: '',
lastSpaceId: 'org-a10d5cb5-4ca53c2e2a05-manager',
lastHubId: 'sth-0',
sessionId: '340908'
}
# Properties 'lastSpaceId and 'lastHubId' has been set automatically to some values.
Properties lastSpaceId
and lastHubId
can be also set manually with commands:
si space use <spaceId>
si hub use <hubId>
si config [command]
print|p # Print out the current configuration
set # Add properties to the global config
reset # Reset configuration value to default
profile|pr # Select and work with user profiles
si config print
$ si c p
Current profile: default
{
configVersion: 1,
apiUrl: 'http://127.0.0.1:8000/api/v1',
middlewareApiUrl: '',
env: 'development',
scope: '',
token: '',
log: { debug: false, format: 'pretty' }
}
si config set [command]
json <json> # Set configuration properties from a json object
apiUrl <url> # Specify the Hub API Url
log [options] # Specify log options
middlewareApiUrl <url> # Specify middleware API url
scope <name> # Specify default scope that should be used when session starts
token <jwt> # Specify platform authorization token
env <production|development> # Specify the environment
si config reset [command]
apiUrl # Reset apiUrl
log # Reset logger
middlewareApiUrl # Reset middlewareApiUrl
token # Reset token
env # Reset env
all # Reset all configuration
si config profile [command]
list|ls # Show available configuration profiles
use <name> # Set configuration profile as default to use
create <name> # Create new configuration profile
remove <name> # Remove existing profile configuration
Usage:
si config set [options] [command]
Commands:
json <json> set configuration properties from json object
apiUrl <url> specify the Hub API Url (default: http://127.0.0.1:8000/api/v1)
log [options] specify log options
middlewareApiUrl <url> specify middleware API url
scope <name> specify default scope that should be used when session start
token <jwt> specify platform authorization token (default: )
env <production|develop> specify the environment (default: development)
Usage:
si config use [options] [command]
Commands:
apiUrl <url> specify the hub API url (current: http://127.0.0.1:8000/api/v1)
Usage:
index config set log [options]
Options:
--debug <boolean> specify log to show extended view (default: true)
--format <format> specify format between "pretty" or "json" (default: pretty)
-h, --help display help for command
Allows to run programs in different data centers, computers or devices in local network
Usage:
si hub [command] [options...]
Options:
-h, --help display help for command
Commands:
list|ls list the hubs
info display info about the hub
logs pipe running hub log to stdout
load monitor CPU, memory and disk usage on the Hub
set <apiUrl>
Sequence (Transform Sequence) is a list of chained functions with a lightweight application business logic that contains a developer code. The minimal number is one function.
Usage: index sequence|seq [command] [options...]
Operations on a Sequence package, consisting of one or more functions executed one after another
Options:
-h, --help display help for command
Commands:
list|ls Lists all available Sequences
pack [options] <path> Create archived file (package) with the Sequence for later use
send [options] <package> Send the Sequence package to the Hub
update <query> <package> Updates sequence with given name
use|select <id> Select the Sequence to communicate with by using '-' alias instead
of Sequence id
start [options] <id> Start the Sequence with or without given arguments
deploy|run [options] <path> Pack (if needed), send and start the Sequence
get <id> Obtain a basic information about the Sequence
delete|rm <id> Delete the Sequence from the Hub
prune [options] Remove all Sequences from the current scope (use with caution)
💡 Argument id - the Sequence id to start or '-' for the last uploaded.
Before starting the Sequence the package has to be prepared. The package is an archived directory with the Sequence and all its required dependencies. The package can be easily created with the command presented below.
Usage:
si seq pack [options] <path>
Options:
-c, --stdout output to stdout (ignores -o)
-o, --output <file.tar.gz> output path - defaults to dirname
-h, --help display help for command
Hub accepts only Sequence package that is compressed into *tar.gz
format. Command si seq send <path-to-file>
will pack the directory (if it wasn't packed earlier) and send it to the Hub (STH or SCP Hub).
Now the Sequence can be executed on the Hub. The Sequence Instance will be created and run.
Usage: si sequence start [options] <id>
Start the Sequence with or without given arguments
Arguments:
id Sequence id to start or '-' for the last uploaded
Options:
-f, --config-file <path-to-file> Path to configuration file in JSON format to be passed to the
Instance context
-s, --config-string <json-string> Configuration in JSON format to be passed to the Instance
context
--output-topic <string> Topic to which the output stream should be routed
--input-topic <string> Topic to which the input stream should be routed
--args <json-string> Arguments to be passed to the first function in the Sequence
--limits <json-string> Instance limits
-h, --help display help for command
All above steps can be done by single command: si seq deploy
, so Sequence will be packed (if needed), sent and started.
Usage:
si sequence deploy|run [options] <path>
Pack (if needed), send and start the Sequence
Options:
-o, --output <file.tar.gz> Output path - defaults to dirname
-f, --config-file <path-to-file> Path to configuration file in JSON format to be passed to the
Instance context
-s, --config-string <json-string> Configuration in JSON format to be passed to the Instance
context
--args <json-string> Arguments to be passed to the first function in the Sequence
-h, --help display help for command
Instance (Sequence Instance) is a running Sequence, that can process an enormous amount of data on the fly without losing persistence.
Usage:
si inst [command] [options...]
Operations on the running Sequence
Options:
-h, --help display help for command
Commands:
list|ls List the Instances
use <id> Select the Instance to communicate with by using '-' alias instead
of Instance id
health <id> Display Instance health status
info <id> Display the info about the Instance
log <id> Pipe the running Instance log to stdout
kill <id> Kill the Instance without waiting for the unfinished task
stop <id> <timeout> End the Instance gracefully waiting for the unfinished tasks
input [options] <id> [file] Send a file to input, if no file given the data will be read
directly from the console input (stdin)
output <id> Pipe the running Instance output to stdout
stdio|attach <id> Listen to all stdio - stdin, stdout, stderr of the running Instance
event Show event commands
stdin <id> [file] Send a file to stdin, if no file given the data will be read from
stdin
stderr <id> Pipe the running Instance stderr stream to stdout
stdout <id> Pipe the running Instance stdout stream to stdout
💡 Argument id - the instance id or '-' for the last one started or selected.
Usage:
si inst event [command] [options...]
Show event commands
Options:
-h, --help display help for command
Commands:
emit|invoke <id> <eventName> [payload] Send event with eventName and a JSON formatted event
payload
on [options] <id> <eventName> Get the last event occurrence (will wait for the first
one if not yet retrieved)
Publish/subscribe operations allow to manage data flow.
Usage:
si topic [command] [options...]
Manage data flow through topics operations
Options:
-h, --help display help for command
Commands:
get [options] <topic-name> Get data from topic
send [options] <topic-name> [<file>] Send data on topic from file, directory or directly
through the console
ls List information about topics
See the code documentation here: scramjetorg/transform-hub/docs/cli/modules.md
To find out more about Scramjet Interface, please check out our official documentation website.
This package is part of Scramjet Transform Hub.
Scramjet Transform Hub is a deployment and execution platform. Once installed on a server, it will allow you to start your programs and keep them running on a remote machine. You will be able to start programs in the background or connect to them and see their output directly on your terminal. You will be able to pipe your local data to the program, as if it was running from your terminal. You can start your server in AWS, Google Cloud or Azure, start it on your local machine, install it on a Raspberry Pi or wherever else you'd like.
There's no limit what you can use it for. You want a stock checker? A chat bot? Maybe you'd like to automate your home? Retrieve sensor data? Maybe you have a lot of data and want to transfer and wrangle it? You have a database of cities and you'd like to enrich your data? You do machine learning and you want to train your set while the data is fetched in real time? Hey, you want to use it for something else and ask us if that's a good use? Ask us via email or hop on our Scramjet Discord!
- Scramjet, the company behind Transform Hub
- The Scramjet Framework - functional reactive stream processing framework
- The Transform Hub repo on github
- You can see the Scramjet Transform Hub API docs here
- You can see the CLI documentation here, but
si help
should also be quite effective. - Don't forget to ⭐ this repo if you like it,
subscribe
to releases and keep visiting us for new versions and updates. - You can open an issue - file a bug report or a feature request here
This module is licensed under AGPL-3.0 license.
The Scramjet Transform Hub project is dual-licensed under the AGPL-3.0 and MIT licenses. Parts of the project that are linked with your programs are MIT licensed, the rest is AGPL.
We accept valid contributions and we will be publishing a more specific project roadmap so contributors can propose features and also help us implement them. We kindly ask you that contributed commits are Signed-Off git commit --sign-off
.
We provide support for contributors via test cases. If you expect a certain type of workflow to be officially supported, please specify and implement a test case in Gherkin
format in bdd
directory and include it in your pull request. More info about our BDD test you will find here.
The project need's your help! There's lots of work to do and we have a lot of plans. If you want to help and be part of the Scramjet team, please reach out to us, on discord or email us: opensource@scramjet.org.
Do you like this project? It helped you to reduce time spent on delivering your solution? You are welcome to buy us a coffee ☕ Thanks a lot! 😉
- There's also a Paypal donation link if you prefer that: