Approaching 0.1

Follow @aquameta

We're approaching 0.1 release! Here's a quick summary of development that needs to be done before we get there.

1. bundle blob store

Right now each commit is stored as a whole new copy of all the fields in all the rows in the commit. This isn't very efficient, since most commits are slight modifications of the previous commit. I started here for simplicity, but now that the data model is pretty solid and things seem to be functioning ok, we can start to do some simple optimizations and make sure tests keep passing.

The better way to do this is to to handle it with a hash table, similar to git's blob store. To implement a hash table in the database is pretty simple. The primary key of the blob table is the sha of the contents, and the only other field is "value". There are triggers on the table so that when you INSERT into it, it runs a sha256 on the value and then if it already exists, discards the insert. That way each value in any field in any bundle is only stored once.

There will literally be only one true, one 1, etc in the entire repository. This makes things really efficient, and it's totally trivial to implement in PostgreSQL.

2. bundle push/pull

Merge changes from one bundle repo into another. A push asks a remote repository "Hey what commits do you have for this bundle?" and then sends to the remote any new commits. A pull works in just the opposite direction.

HTTP push happens from inside the database. To accomplish this, I'm writing a simple HTTP client for PostgreSQL, and an Aquameta Endpoint client on top of that. Under the hood it uses PL/Python and liburl2.

WebRTC push behaves similarly to the HTTP one, but instead of going over HTTP directly, it proxies through the browser and over to another user's browser over WebRTC. That way two parties can both be behind firewalls and push/pull to each other without exposing ports.

3. central bundle repo

Aquameta's central package repository, like npm, pypi, apt etc. but for bundles.

4. bundle everything

After layer 0 meta and layer 1 bundle, all other layers in Aquameta can be bundles. Right now, they're .sql files in /core, but with meta+bundle, they can just be installed via the bundle system. So, I'm going to bundle up event, www, widget, filesystem and www. We'll be down to just two .sql files, meta and bundle. Everything else will be a bundle, and we'll be fully bootstrapped. In theory, future system updates should just be bundle pulls.


comments powered by Disqus