Zeetl
An ETL for ZEE. Run in it Docker or import it as a Node libary.
Requirements
- Postgres
- Node 16.x or Docker
Run in Docker
The ETL can be started with Docker. Its config is set exclusively via environment variables. The following table dictates options and requirements.
Environment Variable | Description | Default | Required |
---|---|---|---|
STAKING_PROGRAM_ID | The address of the ZEE staking program | - | true |
SOLANA_URL | URL of the Solana RPC API | - | true |
POSTGRES_PASSWORD | Postgres password | - | true |
POSTGRES_USER | Postgres user | admin | false |
POSTGRES_HOST | Postgres host | localhost | false |
POSTGRES_PORT | Postgres port | 5432 | false |
POSTGRES_DATABASE | Postgres database | zeetl | false |
SOLANA_RATE_LIMIT | Min time between API requests in ms | 250 | false |
LOG_LVL | The log level | info | false |
LOG_SILENT | Set logs to silent | false | false |
The easiest way to pass environment variables is to create a .env
file containing your configuration. Here is an example:
POSTGRES_PASSWORD=dummy
SOLANA_URL=https://api.devnet.solana.com
SOLANA_RATE_LIMIT=250
STAKING_PROGRAM_ID=7vo1tfi7A7DfLi5viwb1eNwv9WUuphV2QS3TNv1nPUo5
Next, migrate the database schema.
// Insert the correct version.
docker run --rm --network host --env-file .env ormembaar/zeetl:VERSION bin/migrate
Finally, start the ETL.
// Again, insert the version
docker run -d --network host --env-file .env --name=zeetl ormembaar/zeetl:VERSION
Import as library
The ETL can also be used as a library.
const connection = new Connection('https://api.devnet.solana.com', 'confirmed');
const etl = new ETL({
connection,
postgresURL: 'postgres://...',
stakeProgramId: '7vo1tfi7A7DfLi5viwb1eNwv9WUuphV2QS3TNv1nPUo5',
rateLimit: 250
});
etl.on('error', (err) => log.error('ETL error', err));
etl.start();
Then you can query the database directly via the Prisma client embedded in the ETL.
const foo = await etl.client.stakeInitInstruction.findFirst();
console.log(foo);
/*
{
id: 1,
signatureId: 4,
payer: 'FuGYQLy3jPhdUvRiAMtz71TdXwSdEVuV8xHq2XDPBAE6',
staker: '5hWFiih4FNjGBJghozv1ZYBh4nEXyTNoCHSJgW1gg2Nj',
stakerFund: 'BvQM3N9fw7bodh7eAjEvrmBeGkazi9wbQKHgMyEQpPGK',
stakerBeneficiary: 'AKdjvmGqwbPQg3AtZeEzZ7fPHuDjRnXAWbznU2YzfHmt',
communityAccount: 'ALismzdo2CZk96rAq9VMwCfRcHzhfCaA8JmjiJvc4UC',
stakeAccount: 'BqRvpXgiT7svyn9Y83Ss6gqKkBcTECgLg7AtqnXFcFbX',
zeeTokenMint: 'E3iGfQ6AdPLT2Q39JMNKnC78hEvoNTgqCmt7dydSAzpc'
}
*/
See the Prisma docs for help using the client. The client is generated from prisma/schema.prisma
, which should be used for reference.