Core Library for xkite, a Kafka Integrated Testing Environment
xkite-core provides a comprehensive prototyping, testing, and monitoring toolset for Apache Kafka. Use xkite to bootstrap your next project, or install our library into an existing project. Built by developers, for developers.
Dependencies
The latest stable versions of:
Installation
-
Clone this Repository
git clone https://github.com/oslabs-beta/xkite-core.git -
Install Dependencies
cdinto the cloned repository and runnpm install
Who Uses xkite-core
How It Works
The xkite-core library is, as the name suggests, the core library for xkite.
To interface with xkite-core, simply import the Kite class into your project.
In xkite, Kite provides the underlying functionality for configuring a Docker Compose YAML configuration, managing docker containers (configure, run, pause and shutdown), interfacing with remote xkite servers, and providing configuration settings for developers to easily connect to Kafka instances.
Kite Class Data Types
Click to expand details.
KiteState
Overall state of Kite, provided by Kite.getState().
type KiteState =
| 'Unknown'
| 'Init'
| 'Configured'
| 'Running'
| 'Paused'
| 'Shutdown';
KiteServerState
State of remote Kite connection, provided by Kite.getServerState().
type KiteServerState = 'Disconnected' | 'Connected';
KiteConfig
Input object to create a docker instances. Used for Kite.configure().
interface KiteConfig {
kafka: KiteKafkaCfg;
db?: dbCfg;
sink?: sinkCfg;
grafana?: grafanaCfg;
prometheus?: prometheusCfg;
}
KiteSetup
Response object from Kite.getSetup() available after Kite is configured.
interface KiteSetup {
dBSetup?: dbCfg;
kafkaSetup: KafkaSetup;
spring?: { port: number };
prometheus?: { port: number };
grafana?: { port: number };
zookeeper?: { ports: number[] };
jmx?: { ports: number[] };
jupyter?: { port: number };
spark?: { port: number[] };
docker?: { services: string[] };
}
KafkaSetup
Response object from Kite.getKafkaSetup() available after Kite is configured.
interface KafkaSetup {
clientId: string;
brokers: Array<string>;
ssl?: boolean;
}
KiteConfigFile
Format of the configuration file object provided by Kite.getConfigFile().
Note: the fileStream object is a stream of the docker-compose.yml file generated from Kite.configure()
interface KiteConfigFile {
header?: any;
fileStream: Buffer;
}
dbCfg
Configuration object as a part of the KiteConfig object. It defines which data source the user wants configured.
interface dbCfg {
name: 'postgresql' | 'ksql';
port?: number | undefined;
postgresql?: {
username: string;
password: string;
dbname: string;
};
ksql?: {
schema_port?: number | undefined;
};
kafkaconnect?: {
port?: number | undefined;
};
}
sinkCfg
Configuration object as a part of the KiteConfig object. It defines which data sink the user wants configured
interface sinkCfg {
name: 'jupyter' | 'spark';
port?: number;
rpc_port?: number;
kafkaconnect?: {
port?: number | undefined;
};
}
grafanaCfg
Configuration object as a part of the KiteConfig object. It defines which port the user wants their grafana interface to be configured on
interface grafanaCfg {
port?: number | undefined;
}
prometheusCfg
Configuration object as a part of the KiteConfig object. It defines prometheus settings the user wants configured such as port, scrape and evaluation intervals
interface prometheusCfg {
port?: number | undefined;
scrape_interval?: number; //seconds
evaluation_interval?: number; //seconds
}
Kite Class Methods
Click to expand details.
configure()
configures the Kite class for either a local or remote docker session using a KiteConfig object or a server string pointing to a configured Kite server instance such as "http://localhost:3000".Note: If no input is give a default configuration will be used.
Type Definition:
configure: (arg?: string | KiteConfig | undefined) => Promise<'void'>;
Example:
const { Kite } = require('xkite-core');
await Kite.configure(); // configure local default
// or
await Kite.configure('http://localhost:3000'); // configure remote
// or
deploy()
deploys all configured docker instances from Kite.configure(). If the Kite serverState === "Connected" then this deployment will happen on the remote server.
Type Definition:
deploy: () => Promise<void>;
Example:
const { Kite } = require('xkite-core');
await Kite.deploy();
pause()
pauses any/all running docker instances. If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
pause: (service?: string[] | undefined) => Promise<any>;
Example:
const { Kite } = require('xkite-core');
await Kite.pause(['kafka1', 'kafka2']); // pauses kafka1 and kafka2 docker services
await Kite.pause(); // pauses all docker instances
unpause()
Unpauses any/all running docker instances. If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
unpause: (service?: string[] | undefined) => Promise<any>;
Example:
const { Kite } = require('xkite-core');
await Kite.unpause(['kafka1', 'kafka2']); // unpauses kafka1 and kafka2 docker services
await Kite.unpause(); // unpauses all docker instances
shutdown()
Shuts down all running or paused docker instances and removes all configured volumes. If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
shutdown: () => Promise<any>;
Example:
const { Kite } = require('xkite-core');
await Kite.shutdown();
getSetup()
Retrieves the KiteSetup object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getSetup: () => KiteSetup | Promise<KiteSetup>;
Example:
const { Kite } = require('xkite-core');
const setup = await Kite.getSetup();
getKafkaSetup()
Retrieves the KafkaSetup object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getSetup: () => KiteSetup | Promise<KiteSetup>;
Example:
const { Kafka } = require('kafkajs')
const { Kite } = require('xkite-core');
const kafkaSetup = await Kite.getKafkaSetup();
const kafka = new Kafka({
...kafkaSetup,
clientId: 'myapp'
})
...
getDBSetup()
Retrieves the dBCfg object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getDBSetup: () => dbCfg | Promise<dbCfg | undefined>;
Example:
const { Kite } = require('xkite-core');
const dBSetup = await Kite.getDBSetup();
getConfig()
Retrieves the KiteConfig object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getConfig: () => KiteConfig | Promise<KiteConfig>;
Example:
const { Kite } = require('xkite-core');
const config = await Kite.getConfig();
getConfigFile()
Retrieves the KiteConfig object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getConfig: () => KiteConfig | Promise<KiteConfig>;
Example:
const { Kite } = require('xkite-core');
const config = await Kite.getConfig();
getKiteState()
Retrieves the current KiteState. If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getKiteState: () => KiteState | Promise<KiteState>;
Example:
const { Kite } = require('xkite-core');
const state = await Kite.getState();
getKiteServerState()
Retrieves the state of remote connection with the Kite server.
Type Definition:
getKiteServerState: () => KiteServerState;
Example:
const { Kite } = require('xkite-core');
const serverState = await Kite.getServerState();
getPackageBuild()
Retrieves the current package.zip file from Kite. This file contains the full set of dependencies to replicate the docker ecosystem sans xkite-core. If the Kite serverState === "Connected" then this command will be initiated on the remote server.
Type Definition:
getPackageBuild: () => Promise<KiteConfigFile | Error>;
Example:
const { Kite } = require('xkite-core');
const fs = require('fs');
const pkg = await Kite.getPackageBuild();
fs.writeFileSync(
path.resolve(__dirname, 'package.zip'),
Buffer.from(pkg.fileStream)
);
Docker Images
xkite uses the following docker images to provide their associated services: