FSDB
FSDB is a collection of Go libraries providing a key-value
store on your file system.
(Example code)
Why?
tl;dr: It’s for larger (10k+ per entry), less latency sensitive data store.
Most key-value store libraries are optimized for small, in-memory size.
Even for on-disk libraries, they are usually not optimized for larger (10k+)
values.
Also on-disk libraries usually uses write amplify for better performance,
which means they will take more disk space than the actual data stored.
FSDB store the data as-is or use optional gzip compression,
making it a better solution for companies that need to store huge amount of data
and is less sensitive to data latency.
FSDB could also be used as the last layer of your layered key-value stores:
Use in-memory libraries for samllest, most time sensitive data;
Use other on-disk libraries for larger, less time sensitive data;
And use FSDB for largest, least latency sensitive data.
Further more, FSDB provided a hybrid implementation,
which allows you to put some of your data on a remote bucket
(AWS S3, Google Cloud Storage, etc.),
providing an exra layer for larger and higher latency data.
It can also be used to implement mocks of remote libraries.
For example, package bucket
uses local FSDB to implement its
mock for testing.
Highlights
FSDB has a minimal overhead,
which means its performance is almost identical to the disk I/O performance.
The local implementation has no locks.
The hybrid implementation only has an optional row lock,
please refer to the
package documentation
for more details.
The local and hybrid implementation has the same interface,
which means you can use the local implementation first,
and move the the hybrid implementation later as your data grows.
Packages Index
- Package fsdb
defines the interface. It does not provide implementations.
- Package local
provides the local implementation.
- Package hybrid
provides the hybrid implementation.
- Package bucket
defines the bucket interface.
It does not provide implementations.
Implementations for AWS S3 and
Google Cloud Storage
can be found in external libraries.
There’s also an
implementation
based on
Go-Cloud
Blob
interface so you can use any Go-Cloud Blob implementation in hybrid FSDB.
Test
All packages provide its own tests can be run with go test.
If you want to test every package within this repository,
on Go 1.11+ you can use go test ./....
There are some tests with sleep calls,
you can skip them by running go test -short instead.
Package local
provides a benchmark test can be run with go test -bench ..
License
BSD License.
评论