Commits that could get me committed.
Did I say that I was nerd-sniped by the AT Protocol? Maybe not loudly, but my close friends knew that I’d thrown myself into a new side project and may have wondered about the wisdom of that (frequently I’ve had side projects of side projects of side…). “How hard could it be,” I wondered, and at the beginning of February I decided that it wouldn’t be that hard and that I could probably have something working before I left for Vancouver for ATmosphereConf.
But it was challenging. It helped a lot to have an instance of the Bluesky PDS running behind IO so that I could observe its traffic, and it also helped to have already built the client side of ATProto OAuth into IO, and to have just built slink, which gave me a way to interact with XRPC in Go on my own terms. However, I didn’t vibe code it and kept with my characteristic distaste for dependencies, so I wound up writing some key parts (like CID, CBOR, CAR, MST operations and OAuth) by hand and from scratch. One other thing that helped a lot was to start with the Bluesky PDS data structures. As Fred Brooks wrote:
Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.
I haven’t decided what I want to do with this other than use it myself for a while. Like IO, it has a TUI and an SSH control interface. Just looking at the start screen raises lots of ideas of things that could be added to it.
I call the PDS smile because originally I thought I was building something very personal for a single user who, for whatever reason, might want to run multiple repos. I thought of it as a hobbit-hole, which Tolkien called a smial (💡!), but I didn’t want to forever be correcting the spelling for people who heard the name, and smile isn’t a bad thing to present to the world.
Currently I have one instance of smile that I run on my laptop and publish to the world using a Digital Ocean droplet, IO, and an SSH reverse tunnel. Did you know about these? All I do is run this on my laptop:
ssh -R 8080:0.0.0.0:8080 cloudhost
where cloudhost is the name of my droplet in my Tailnet. Then when I run smile locally, it listens to port 8080, which the tunnel makes available on port 8080 of my droplet, which IO publishes using an ingress. (That port is firewalled in case anyone is tempted to hit it directly.)
For entertainment and an opportunity to mock my commit hygiene, here is the commit history for what I’ve done so far.
Tue Feb 03 initial commit
Tue Feb 03 add cli functions to list contents of a did's database
Tue Feb 03 reorganize list commands
Tue Feb 03 add a get command for repo blocks
Tue Feb 03 get repo block contents as json, get a list of record keys
Tue Feb 03 add tests and functions to compute merkle tree depth of keys
Wed Feb 04 tid handling
Wed Feb 04 hashing and cid
Wed Feb 04 parse cid-links stored in CBOR
Wed Feb 04 tighten-up display of CIDs in JSON exports
Wed Feb 04 it appears that I can now regenerate all hashes in the MST
Wed Feb 04 one more missing piece, converting base32-encoded CIDs to tagged-CBOR CIDs
Thu Feb 05 add stub for password verification test command
Thu Feb 05 password hashing and salting that matches the bluesky pds
Thu Feb 05 dump the mst
Thu Feb 05 tweak tree display
Fri Feb 06 expand mst-related code
Fri Feb 06 mst progress
Fri Feb 06 tree checking
Fri Feb 06 add key miner
Fri Feb 06 "check tree" is a separate command and collects all errors
Fri Feb 06 stub for delete command, split traverse from dump and have both write on stdout
Fri Feb 06 steps toward record deletion
Fri Feb 06 insert command structure and a few helpers
Fri Feb 06 general cleanup and notes on the next steps
Sat Feb 07 reorganize CLI to put command implementations under internal
Sat Feb 07 just a few quick thoughts
Sun Feb 08 consistency and style preferences
Mon Feb 09 tree dump/check/traverse now can take a --cid argument to work on a subtree
Mon Feb 09 set up test commands for splitting and merging subtrees
Mon Feb 09 round-trip cid representations (with apologies to my future self)
Mon Feb 09 deeper round-tripping of nodes
Mon Feb 09 first partial implementation of tree splitting
Mon Feb 09 tree splitting in the repo store
Mon Feb 09 tree merging in the repo store
Mon Feb 09 entry deletion
Mon Feb 09 possibly-working tree insertion
Mon Feb 09 trim any empty nodes from the top of a tree after deleting an entry
Tue Feb 10 support empty trees
Tue Feb 10 debug insertion problems + several new test commands
Tue Feb 10 minor streamlining
Tue Feb 10 pass around a sqlite pool instead of a did
Tue Feb 10 more database-related changes, beginning mst tests
Tue Feb 10 mst tests
Wed Feb 11 fix noisy test
Wed Feb 11 slightly tighten-up tree level computation
Wed Feb 11 tighten-up record deletion function by building-in the necessary cleanup steps.
Wed Feb 11 clean up / polish the mst package
Wed Feb 11 garbage collection
Wed Feb 11 simple car reader
Wed Feb 11 scaffolding to read/write CAR files
Wed Feb 11 custom CAR file reading and writing
Wed Feb 11 simplify CID encoding, remove multihash dependency
Wed Feb 11 significantly reduce dependency on go-cid
Wed Feb 11 removing dependency on go-cid
Wed Feb 11 delete parse cid command
Wed Feb 11 move cid code into a dedicated package
Thu Feb 12 cid tests
Thu Feb 12 move timestamp identifiers to a dedicated package
Thu Feb 12 replace the encoders package with more specific packages under "encoding"
Thu Feb 12 clean up and test mst package
Thu Feb 12 rename "mst" package to "tree"
Thu Feb 12 group storage-related packages under storage
Thu Feb 12 improve command organization
Thu Feb 12 centralize setting log level
Thu Feb 12 add stub for server
Thu Feb 12 move server stub into an internal package
Thu Feb 12 stubs for some early PDS handlers
Sat Feb 14 add subscribe test command
Sat Feb 14 reading the firehose (uninterpreted)
Sat Feb 14 subscribe repos
Sun Feb 15 experimental dumping of cbor contents of car files
Sun Feb 15 remove unnecessary dependency on github.com/multiformats/go-base32, encoding/base32 is fine
Sun Feb 15 I tried to use encoding/base32 for tids, but it didn't work out.
Sun Feb 15 gitignore
Sun Feb 15 update readme
Sun Feb 15 It's "smile" now.
Sun Feb 15 basic subscription commands -- working!
Sun Feb 15 getting started with drisl
Mon Feb 16 drisl decoding is generally working
Mon Feb 16 roundtripping drisl from the firehose
Mon Feb 16 progress.. passing all tests
Mon Feb 16 so long and thanks for all the fish, fxamacker/cbor
Mon Feb 16 fix the car file reading test command
Mon Feb 16 it's DRISL everywhere now
Mon Feb 16 general cleanup
Mon Feb 16 fix a couple of bugs found testing with the firehose
Mon Feb 16 tweak some names and remove straggling map[any] instances
Mon Feb 16 car tests
Mon Feb 16 restore "DagCBOR" after it was accidentally changed to "DagDRISL"
Tue Feb 17 add some handler stubs
Tue Feb 17 going back to "CBOR" (from "DRISL") DRISL is a mouthful and I' personally not ready to walk away from general CBOR (even though it's not currently needed for ATProto).
Wed Feb 18 expand time/tid commands to get time in ATProto format.
Wed Feb 18 add more debug printing in subscribe repos cli command
Wed Feb 18 rename the encoding directory to encoders to match IO.
Wed Feb 18 rewrite cbor encoder to use reflection
Wed Feb 18 sample websocket server handler
Thu Feb 19 just something fun
Thu Feb 19 add a test and fix a bug in tid encoding
Thu Feb 19 Add lexicons as a submodule
Thu Feb 19 go mod tidy
Thu Feb 19 Add github action Updated Go version and added slink installation step.
Thu Feb 19 add token subcommands
Fri Feb 20 first UI mock
Sun Feb 22 Create UI and commmand structure (largely pulled from IO) (#1)
Sun Feb 22 remove inappropriate space from error prefix
Sun Feb 22 clean up/organize ssh server
Sun Feb 22 configuration
Sun Feb 22 ui cleanup and organization
Mon Feb 23 improve cobra command organization, add common code to PersistentPreRunE.
Mon Feb 23 hook up repo storage in command initialization
Mon Feb 23 live data in repos table on main screen
Mon Feb 23 repo browsing
Mon Feb 23 import token-related and resolution code from slink
Mon Feb 23 add some missing files and rename "service" to "smileserver"
Mon Feb 23 add missing module (blocked by .gitignore)
Mon Feb 23 progress on com.atproto.server.createSession
Mon Feb 23 pretty good com.atproto.server.createSession
Mon Feb 23 better com.atproto.server.createSession, preparing for more.
Tue Feb 24 token implementations
Tue Feb 24 clean up basic auth and token management
Tue Feb 24 atproto-proxy
Tue Feb 24 serve web did
Tue Feb 24 add com.atproto.server.describeServer with hard-coded values
Tue Feb 24 app.bsky.actor.getPreferences
Tue Feb 24 com.atproto.sync.listRepos
Tue Feb 24 more repo-related stubs
Wed Feb 25 list records (wip) with list options
Wed Feb 25 com.atproto.repo.(getRecord,listRecords)
Wed Feb 25 slightly reorganize storage packages
Wed Feb 25 big package reorg
Wed Feb 25 reorganize webserver package
Wed Feb 25 reorganize webserver
Wed Feb 25 minor cleanup of main server function
Wed Feb 25 delete mocks
Wed Feb 25 update README
Wed Feb 25 stubs for more table accessors
Thu Feb 26 add stubs for a minimum viable pds
Thu Feb 26 add some identity-related xrpc methods and tweak stubs for unimplemented methods
Thu Feb 26 com.atproto.sync.listBlobs
Thu Feb 26 improvements to com.atproto.repo.listRecords
Thu Feb 26 com.atproto.sync.getBlob
Thu Feb 26 com.atproto.sync.getRepoStatus
Thu Feb 26 add a couple of missing "unimplemented" responses
Thu Feb 26 first version of com.atproto.sync.getRepo (does not support "since")
Thu Feb 26 com.atproto.sync.getRecord
Thu Feb 26 com.atproto.repo.uploadBlob
Thu Feb 26 app.bsky.actor.putPreferences
Thu Feb 26 add repo revision to tree operations
Fri Feb 27 reading the sequence database
Fri Feb 27 steps forward, improved subscribeRepos client and tweaked stubs
Fri Feb 27 com.atproto.sync.subscribeRepos
Fri Feb 27 auth for a few remaining methods
Fri Feb 27 input reading for record mutators
Fri Feb 27 mutator planning + did cache
Fri Feb 27 update slink version
Sat Feb 28 add command to recompute record hashes, refine mutator plans
Sat Feb 28 improvements to tree.Find() for use in mutators
Sat Feb 28 tree record replacement
Sat Feb 28 add progress info to README
Sat Feb 28 put list options structs with the tables where they are used
Sat Feb 28 unverified push to build mutator methods
Sat Feb 28 create/delete kinda works
Sat Feb 28 clean up create/delete/put
Sun Mar 01 add collect function to get cids of blocks in the current revision
Sun Mar 01 use -l flag to set debug logging level in selected tests
Sun Mar 01 collect ops and build the car for sequence storage
Sun Mar 01 saving repo_seq entries, now we have subscriptions
Sun Mar 01 add com.atproto.server.checkAccountStatus and manual garbage collection
Sun Mar 01 scaffolding for com.atproto.repo.applyWrites
Sun Mar 01 restructure mutator handlers to prepare to include them in applyWrites
Sun Mar 01 hook up the rest of com.atproto.repo.applyWrites
Sun Mar 01 create mutation package for mutator cores
Sun Mar 01 Update README to mark all methods built, dropping com.atproto.server.activateAccount
Mon Mar 02 polish "mutation" code
Mon Mar 02 improve mutator calling structure
Mon Mar 02 minor naming adjustment
Mon Mar 02 avoid possible key collisions in applyWrites by offsetting new creation rkeys with the operation index
Mon Mar 02 add function to extract blob references from records
Mon Mar 02 blob management
Mon Mar 02 I can create dids! It's ugly.
Mon Mar 02 add a did update command and do a little cleanup. it's still very messy but works!
Mon Mar 02 tweak the home page.
Mon Mar 02 polish did create/update a bit and store keys in the "standard" place.
Tue Mar 03 lots of cleanup to the did operations
Tue Mar 03 did chain verification (only works for secp256k1 keys)
Tue Mar 03 README updates
Tue Mar 03 create table statements
Tue Mar 03 list accounts command
Tue Mar 03 commands to check and set passwords
Tue Mar 03 use hostname setting to identify the server
Tue Mar 03 cleanup the "base"
Tue Mar 03 ok, I don't like this suffix but I'll use it for consistency.
Tue Mar 03 new account setup
Tue Mar 03 use lowercase in health response
Tue Mar 03 return real links in describeServer response
Tue Mar 03 Dockerfile
Wed Mar 04 add simple cors middleware
Wed Mar 04 fix some problems with jwts and atproto proxying
Wed Mar 04 ensure that blob storage directories exist before saving blob data
Wed Mar 04 give cors clients what they ask for
Wed Mar 04 make the logging of proxied headers optional
Wed Mar 04 use slink.Link to represent cid-links
Thu Mar 05 work in progress to fix cbor encoding
Thu Mar 05 fix cbor encoder and all tests
Thu Mar 05 Update slink to fix marshalling problem
Thu Mar 05 save a sync event when new accounts are created.
Thu Mar 05 debugging com.atproto.sync.subscribeRepos -- the indigo relay can crawl without errors
Thu Mar 05 fix test
Thu Mar 05 restructure a branch to avoid a potential crash
Thu Mar 05 tagged values are a private detail of cbor encoding
Thu Mar 05 add "since" to repo export
Thu Mar 05 fix typo in temp dir name
Fri Mar 06 revise home directory setup to be more generic (for reuse across apps).
Fri Mar 06 smile doesn't use XDG_CACHE_HOME
Tue Mar 10 initial oauth scaffolding
Tue Mar 10 add hash recomputation to DID verifier
Wed Mar 11 Gathering info for PAR.
Wed Mar 11 some early cleanup of PAR handler
Wed Mar 11 break out oauth package to handle the core operations
Thu Mar 12 a string across the chasm of oauth signin
Thu Mar 12 oauth login kind of works
Thu Mar 12 debugging errors in blob handling, add missing xrpc function handler
Thu Mar 12 fix a couple of errors causing data to be lost in transmission
Fri Mar 13 fix problem with blobs
Fri Mar 13 just tell them that account emails are confirmed
Fri Mar 13 cool down logging
Fri Mar 13 tweak oauth html and add a "session" hidden input to consent form
Fri Mar 13 collect and display information about subscribers
Fri Mar 13 ui tweaks
Fri Mar 13 handle nullable cid-links
Sat Mar 14 add sync command and reduce delay between subscription messages
Sun Mar 15 tweak "home" page
Mon Mar 16 fix "put" record to create or replace, make oauth requested scope explicit
Mon Mar 16 generate a nonce for DPoP
Tue Mar 17 rearrange some functions in xrpc handlers
Tue Mar 17 require DPoP nonce for PAR requests
Tue Mar 17 initial (messy) DPoP support for XRPC
Tue Mar 17 major refactoring of web error handling
Tue Mar 17 another wave of error refactoring
Tue Mar 17 fix crash in message creation with invalid blobs
Tue Mar 17 stub for verify records command
Tue Mar 17 a slow way to list repos that allows some to be non-public
Tue Mar 17 I'm going to remove the accounts database. This starts with a passwords table in core.
Tue Mar 17 move authrequests and tokencodes to core
Tue Mar 17 it's gone! the accounts db is merged into core and dropped
Wed Mar 18 more robustly handle uninitialized repo databases