carbites
Chunking for CAR files. Split a single CAR into multiple CARs.
Install
npm install carbitesUsage
Carbites supports 3 different strategies:
- Simple (default) - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID.
- Rooted - like simple, but creates a custom root node to ensure all blocks in a CAR are referenced.
- Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR has the same root CID, but contains a different portion of the DAG.
Simple
import { CarSplitter } from 'carbites'
import { CarReader } from '@ipld/car'
import fs from 'fs'
const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))
const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs
const splitter = new CarSplitter(bigCar, targetSize) // (simple strategy)
for await (const car of splitter.cars()) {
// Each `car` is an AsyncIterable<Uint8Array>
}bafkqaaa as recommended.
Rooted
Instead of an empty CID, carbites can generate a special root node for each split CAR that references all the blocks and the original roots (only in the first CAR). To do this, use the RootedCarSplitter constructor. When reading/extracting data from the CARs, the root node should be discarded.
Example
import { RootedCarSplitter } from 'carbites/rooted'
import { CarReader } from '@ipld/car/reader'
import * as dagCbor from '@ipld/dag-cbor'
import fs from 'fs'
const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))
const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs
const splitter = new RootedCarSplitter(bigCar, targetSize)
const cars = splitter.cars()
// Every CAR has a single root - a CBOR node that is an tuple of `/carbites/1`,
// an array of root CIDs and an array of block CIDs.
// e.g. ['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']]
const { done, value: car } = await cars.next()
const reader = await CarReader.fromIterable(car)
const rootCids = await reader.getRoots()
const rootNode = dagCbor.decode(await reader.get(rootCids[0]))
console.log(rootNode[0]) // /carbites/1
console.log(rootNode[1]) // Root CIDs (only in first CAR)
/*
[
CID(bafybeictvyf6polqzgop3jt32owubfmsg3kl226omqrfte4eyidubc4rpq)
]
*/
console.log(rootNode[2]) // Block CIDs (all blocks in this CAR)
/*
[
CID(bafybeictvyf6polqzgop3jt32owubfmsg3kl226omqrfte4eyidubc4rpq),
CID(bafyreihcsxqhd6agqpboc3wrlvpy5bwuxctv5upicdnt3u2wojv4exxl24),
CID(bafyreiasq7d2ihbqm5xvhjjzlmzsensuadrpmpt2tkjsuwq42xpa34qevu)
]
*/The root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.
Treewalk
Every CAR file has the same root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.
Example
import { TreewalkCarSplitter } from 'carbites/treewalk'
import { CarReader } from '@ipld/car/reader'
import * as dagCbor from '@ipld/dag-cbor'
import fs from 'fs'
const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))
const [rootCid] = await bigCar.getRoots()
const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs
const splitter = new TreewalkCarSplitter(bigCar, targetSize)
for await (const car of splitter.cars()) {
// Each `car` is an AsyncIterable<Uint8Array>
const reader = await CarReader.fromIterable(car)
const [splitCarRootCid] = await reader.getRoots()
console.assert(rootCid.equals(splitCarRootCid)) // all cars will have the same root
}CLI
npm i -g carbites
# Split a big CAR into many smaller CARs
carbites split big.car --size 100MB --strategy simple # (default size & strategy)
# Join many split CARs back into a single CAR.
carbites join big-0.car big-1.car ...
# Note: not a tool for joining arbitrary CARs together! The split CARs MUST
# belong to the same CAR and big-0.car should be the first argument.API
-
class CarSplitter -
class CarJoiner class RootedCarSplitterclass RootedCarJoinerclass TreewalkCarSplitterclass TreewalkCarJoiner
class CarSplitter
Split a CAR file into several smaller CAR files.
Import in the browser:
import { CarSplitter } from 'https://cdn.skypack.dev/carbites'Import in Node.js:
import { CarSplitter } from 'carbites'Note: This is an alias of SimpleCarSplitter - the default strategy for splitting CARs.
constructor(car: CarReader, targetSize: number)
Create a new CarSplitter for the passed CAR file, aiming to generate CARs of around targetSize bytes in size.
cars(): AsyncGenerator<AsyncIterable<Uint8Array> & RootsReader>
Split the CAR file and create multiple smaller CAR files. Returns an AsyncGenerator that yields the split CAR files (of type AsyncIterable<Uint8Array>).
The CAR files output also implement the RootsReader interface from @ipld/car which means you can call getRoots(): Promise<CID[]> to obtain the root CIDs.
static async fromBlob(blob: Blob, targetSize: number): CarSplitter
Convenience function to create a new CarSplitter from a blob of CAR file data.
static async fromIterable(iterable: AsyncIterable<Uint8Array>, targetSize: number): CarSplitter
Convenience function to create a new CarSplitter from an AsyncIterable<Uint8Array> of CAR file data.
class CarJoiner
Join together split CAR files into a single big CAR.
Import in the browser:
import { CarJoiner } from 'https://cdn.skypack.dev/carbites'Import in Node.js:
import { CarJoiner } from 'carbites'Note: This is an alias of SimpleCarJoiner - a joiner for the the default CAR splitting strategy.
constructor(cars: Iterable<CarReader>)
Create a new CarJoiner for joining the passed CAR files together.
car(): AsyncGenerator<Uint8Array>
Join the CAR files together and return the joined CAR.
class RootedCarSplitter
Split a CAR file into several smaller CAR files ensuring every CAR file contains a single root node that references all the blocks and the original roots (only in the first CAR). When reading/extracting data from the CARs, the root node should be discarded.
Import in the browser:
import { RootedCarSplitter } from 'https://cdn.skypack.dev/carbites/rooted'Import in Node.js:
import { RootedCarSplitter } from 'carbites/rooted'The API is the same as for CarSplitter.
Root Node Format
The root node is a dag-cbor node that is a tuple of the string /carbites/1, an array of root CIDs (only seen in first CAR) and an array of block CIDs (all the blocks in the CAR). e.g. ['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']].
Note: The root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.
class RootedCarJoiner
Join together CAR files that were split using RootedCarSplitter.
The API is the same as for CarJoiner.
class TreewalkCarSplitter
Split a CAR file into several smaller CAR files. Every CAR file has the same root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.
Import in the browser:
import { TreewalkCarSplitter } from 'https://cdn.skypack.dev/carbites/treewalk'Import in Node.js:
import { TreewalkCarSplitter } from 'carbites/treewalk'The API is the same as for CarSplitter.
class TreewalkCarJoiner
Join together CAR files that were split using TreewalkCarSplitter.
The API is the same as for CarJoiner.
Releasing
You can publish by either running npm publish in the dist directory or using npm run publish.
Contribute
Feel free to dive in! Open an issue or submit PRs.
License
Dual-licensed under MIT + Apache 2.0