toplingdb

中文版

ToplingDB: A Persistent Key-Value Store for External Storage

ToplingDB is developed and maintained by Topling Inc. See ToplingDB Branch Name Convention.

Quick Start

ToplingDB requires C++17, gcc 8.3 or newer is recommended, clang also works.

ToplingDB is forked from RocksDB, much faster than RocksDB (see the db_bench comparisons), you can download ToplingDB Enterprise or compile it yourself:

Compile & run db_bench

sudo yum -y install git libaio-devel gcc-c++ gflags-devel zlib-devel bzip2-devel libcurl-devel liburing-devel snappy-devel jemalloc-devel
#sudo apt-get update -y && sudo apt-get install -y libjemalloc-dev libaio-dev libgflags-dev zlib1g-dev libbz2-dev libcurl4-gnutls-dev liburing-dev libsnappy-dev libbz2-dev liblz4-dev libzstd-dev
git clone https://github.com/topling/toplingdb
cd toplingdb
make -j`nproc` db_bench DEBUG_LEVEL=0
sudo make install-dev-shared DEBUG_LEVEL=0 PREFIX=/some/path # installs shared library and headers; install-shared alone installs only the library

After download+uncompress or compile, you can run bundled db_bench.sh(need port 2011), then use ToplingDB in C++, or in Java, Rust.

During compilation, a precompiled ToplingZipTable (90-day trial) may be downloaded automatically. If you need to disable the ZipTable Builder trial, pass WITH_TOPLING_ROCKS=0 WITH_CSPP_MEMTABLE=0 to make; the Makefile requires CSPP to be disabled when Topling Rocks is disabled (or contact us). CSPPMemTable is also distributed this way.

Introduction

ToplingDB’s submodule rockside is the entry point of ToplingDB, see SidePlugin wiki.

ToplingDB has much more key features than RocksDB:

1. SidePlugin enables users to write a json(or yaml) to define DB configs
2. Embedded Http Server enables users to view almost all DB info on web, this is a component of SidePlugin
3. Embedded Http Server enables users to online change db/cf options and all db meta objects(such as MemTabFactory, TableFactory, WriteBufferManager …) without restart the running process
4. Many improvements and refactories on RocksDB, aimed for performance and extendibility
5. memtable as wal log index, omit Flush MemTable to L0, reduce write amp, further improves for large MemTable
6. Topling transaction lock management, 5x faster than rocksdb
7. MultiGet with concurrent IO by fiber/coroutine + io_uring, much faster than RocksDB’s async MultiGet
8. Topling de-virtualization, de-virtualize hotspot (virtual) functions, and key prefix caches, bechmarks
9. Topling zero copy for point search(Get/MultiGet) and Iterator
10. Topling memtable as log index, omit memtable flush to L0
11. Builtin SidePlugins for existing RocksDB components(Cache, Comparator, TableFactory, MemTableFactory…)
12. Builtin Prometheus metrics support, this is based on Embedded Http Server
13. Many bugfixes for RocksDB, a small part of such fixes was Pull Requested to upstream RocksDB

Crash-safe recovery

See the crash-safe recovery guide for the recovery mechanism, configuration, and limitations. For the underlying data-structure principles, see The Isomorphism Between Wait-Free Reads and Crash Safety. DB::Open after an abnormal exit is timed in crash_recover_bench.md.

ToplingDB cloud native DB services

1. MyTopling(MySQL on ToplingDB), MyTopling on aliyun
2. Todis(Redis on ToplingDB)

ToplingDB Components

With SidePlugin mechanics, plugins/components can be physically separated from core toplingdb

1. Can be compiled to a separated dynamic lib and loaded at runtime
2. User code need not any changes, just change json/yaml files
3. Topling’s non-open-source enterprise plugins/components are delivered in this way

Repository dir structure

toplingdb
 \__ sideplugin
      \__ rockside                 (submodule , sideplugin core and framework)
      \__ topling-zip              (auto clone, zip and core lib)
      \__ cspp-memtable            (auto clone, sideplugin component)
      \__ cspp-wbwi                (auto clone, sideplugin component)
      \__ topling-sst              (auto clone, sideplugin component)
      \__ topling-rocks            (auto clone, sideplugin component)
      \__ topling-zip_table_reader (auto clone, sideplugin component)
      \__ toplingdb-fs             (auto clone when enabled, sideplugin component)
      \__ topling-dcompact         (auto clone, sideplugin component)
           \_ tools/dcompact       (dcompact-worker binary app)

Repository	Permission	Description (and components)
ToplingDB	public	Top repository, forked from RocksDB with our fixes, refactories and enhancements
rockside	public	This is a submodule, contains: SidePlugin framework and Builtin SidePlugins Embedded Http Server and Prometheus metrics
cspp-wbwi (WriteBatchWithIndex)	public	With CSPP and carefully coding, CSPP_WBWI is 20x faster than rocksdb SkipList based WBWI
cspp-memtable	private	(CSPP is Crash Safe Parallel Patricia trie) MemTab, which outperforms SkipList on all aspects: 3x lower memory usage, 7x single thread performance, perfect multi-thread scaling
topling-sst	public	1. SingleFastTable(designed for L0 and L1) 2. VecAutoSortTable(designed for MyTopling bulk_load). 3. Deprecated ToplingFastTable, CSPPAutoSortTable
topling-dcompact	public	Distributed Compaction with general dcompact_worker application, offload compactions to elastic computing clusters, much more powerful than RocksDB’s Remote Compaction; see the dcompact db_bench comparisons
topling-rocks	private	For build ToplingZipTable, an SST implementation optimized for RAM and SSD space, aimed for L2+ level compaction, which uses topling dedicated searchable in-memory data compression algorithms
topling-zip_table_reader	public	For read ToplingZipTable by community users, builder of ToplingZipTable is in topling-rocks

To simplify compilation, ToplingDB’s Makefile automatically clones component repositories. Access to private source repositories requires authorization; when the ToplingZipTable Builder or CSPP MemTable source is unavailable, the Makefile can download precompiled trial objects. A failed private clone therefore does not by itself mean that the component is absent from the build. See the trial-component switches in Quick Start.

Configurable features

For performance and simplicity, ToplingDB disabled some RocksDB features by default:

Feature	Control MACRO
Dynamic creation of ColumnFamily	ROCKSDB_DYNAMIC_CREATE_CF
User level timestamp on key	TOPLINGDB_WITH_TIMESTAMP
Wide Columns	TOPLINGDB_WITH_WIDE_COLUMNS
fabricated features for read	TOPLINGDB_WITH_FABRICATED_COMPLEXITY

To enable these features, add -D${MACRO_NAME} to var EXTRA_CXXFLAGS, such as build ToplingDB for java with dynamic ColumnFamily:

make -j`nproc` EXTRA_CXXFLAGS='-DROCKSDB_DYNAMIC_CREATE_CF' rocksdbjava

---

---

---

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat ([email protected]) and Jeff Dean ([email protected])

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.