Gatty
I put the following in a big kettle:
- isomorphic-git
- append-only event logs
- Ink and Switch’s essay “Local-first software”
- BYOS (bring your own storage)
- browser and Node.js
then simmered the pot for a few months, and this package is what resulted. After finding that “gappy”, evoking Git and append-only, was reserved on NPM, I settled on “gatty”.
Installation
I assume you know and have Git and possibly Node.js.
Node.js setup In your Node.js project:
$ npm i --save isomorphic-gatty
and in your JavaScript/TypeScript code:
;
Browser setup If you’re making a browser app without Node, grab index.bundle.min.js, rename it gatty.bundle.min.js
and invoke it in your HTML:
It’s around 384 kilobytes unzipped, roughly 100 kilobytes gzipped.
Usage and API
Gatty is intended to support a user-facing local-first application. “Local-first” means the app keeps all user data local, and uses Gatty/isomorphic-git as one strategy for backup and multi-device data synchronization. Specifically, Gatty saves a stream of events to a git repo and synchronizes it with a remote git server (e.g., GitHub, Gitlab, Gogs, Azure Repos, etc.). The “events” are just plain strings that your app generates and understands: Gatty doesn’t know anything about them.
The envisioned use case is your app periodically calls Gatty, each time giving it the following:
- new events generated by your app (plain strings—if your app generates anything richer,
JSON.stringify
it first), - a event unique identifier associated with each event—perhaps a timestamp or a random string (or both), and
- the last event unique identifier Gatty sync’d for you (empty string if you’ve never sync’d with Gatty).
Gatty in turn will return
- pairs of unique IDs & events (both plain strings) not generated on this device, i.e., by your app running on another device,
- another event unique identfier that represents the last event your app–device has synchronized, that you can use next time.
This way, the only extra thing you app keeps track of in order to use Gatty is a single stringy unique identifier.
N.B. Gatty currently doesn’t handle offline detection. Your app should make an effort to determine online status, and invoke Gatty when it has network connectivity. As we test how this works, we’ll update this section with tips.
setup
setup: Partial<Gatty>, url: string: Promise<Gatty>
where the second argument
url: string
, the URL to clone from
is required while all the arguments of the first object are optional and passed directly to isomorphic-git:
corsProxy: string
, a CORS proxy like https://cors.isomorphic-git.org to route all requests—necessary if you intend to push to some popular Git hosts like GitHub and Gitlab, but not to others like Gogs and Azure Repos. This proxy will see your username, tokens, Git repo information, so…branch: string
, the branch of the repo you want to work with,depth: number
, how many commits back to fetch,since: Date
, how far back in calendar terms to fetch,username: string
, username for authentication (usually pushing requires this),password: string
, plaintext password for authentication (don’t use this, figure out how to use a token),token: string
, a token with hopefully restricted scope for authentication.
The returned value is a promisified object of type Gatty
, which includes these options and a couple of other internal things.
sync
syncgatty: Gatty, lastSharedUid: string, uids: string, events: string: Promise<>
Given a
gatty
object returned bysetup
,lastSharedUid
, a string representing the event unique identifier that Gatty told you it’s synchronized (use''
, the empty string, if you’ve never synchronized),uids
, an array of unique identifiers (plain strings),events
, an array of events (plain strings),
Gatty will pull the latest version of the repo from the URL you gave it during setup
, add the new events you just gave it, and find and returns the (promisified) events that you haven’t seen (newEvents
), as an array of id–event pairs. It also returns newSharedUid
, the unique identifier of the last synchronized event that you have to keep track of for future calls to sync
.
Repo format
Currently Gatty will create two directories:
_events/
, containing line-delimited JSON files:1
2
, etc. These filenames are base-36-encoded (1
through9
, thena
throughz
, then10
, etc.). Each file contains several JSON-encoded arrays:[unique id, event text]
, separated by a newline. New files will be created when the last one’s size exceeds a threshold, currently 9 kilobytes.
_uniques/
, containing one file per event. The filename is a filenamified version of the event’s unique identifier:uid1
uid2
, or whatever identifiers you picked. Each file contains a string:${path to file in _events}-${number of characters to skip to get to first character of the event}
. For example:_events/3-412
means: to get the event attached to this unique identifier, open_events/3
file, and skip 412 characters (JavaScript characters, i.e., UTF-16, alas).
In future, this storage format might change as we figure out which of the many deficiencies in this scheme wind up mattering most 😅.
Dev
Tape and node-git-server for local testing. (Manually tested with syncing to GitHub (see index.html
).)
Browserify for bundling.
Google Closure Compiler for minification and dead-code elimination.
TypeScript for sanity.
TODO. Create a little webapp that demonstrates this with, e.g., a GitHub or something (not Gist since Gist doesn’t allow subdirectories).
Contact
If you have a GitHub account, please create an issue to get in touch, otherwise e-mail etc. is available in Ahmed Fasih’s contact info