Use Bolt for IO internal storage and general data persistence.
From the start, IO used SQLite for persistent in-application data storage. But this week, we removed our dependency on SQLite and now instead use Bolt.
Strictly speaking, we're using etcd-io/bbolt since the original boltdb/bolt is archived.
Pros
- We were already using SQLite in a very tightly-constrained way. Partly out of laziness (or programmer efficiency), IO configuration and many state elements were stored in SQLite as JSON documents. This practice developed and matured over time and made it easy to move to the Bolt key-value store.
- This reduces the build complexity of IO. SQLite is written in C, and while a Go version exists, we had previously decided to use the C version with CGO. That was based on performance evaluations at heavy traffic loads that were primarily related to traffic storage. As we've progressed, this seems less important and may have been a premature optimization. Instead, the higher build complexity of CGO has been a here-and-there stumbling block, and while we could have moved back to an all-Go SQLite, time has clarified that all IO needs is a key-value store, so something simpler and lighter-weight would suffice. Bolt seemed like the right choice: it had reportedly-good performance and a tiny go.mod.
- This reduces the conceptual complexity of IO data storage. Now all storage operations are essentially puts and gets, which makes it easier to think about future enhancements that might allow multiple IOs to redundantly share state.
- Traffic storage now stores each traffic item as a protobuf-encoded blob.
Cons
- Bolt is said to be relatively slow for writes, but this is only an issue for traffic logging and can be mitigated by architectural decisions.
- Traffic logging used SQLite indexes to make display faster. We initially address this with Bolt by saving traffic in two buckets: each item goes into one global bucket and one API-specific bucket. This doubles the amount of space required to store each item, but brings simplicity and could be addressed in future revisions.