ape-dts

English | 中文

ApeDTS

A production-ready Rust binary (~17 MB compressed) for multi-engine snapshot migration, CDC, and data validation.

ApeDTS is a lightweight, standalone data migration and CDC engine. It handles snapshot migration, continuous synchronization, data validation, correction, and in-flight transformation across databases, message queues, and analytical engines. Designed for cloud-native, stateless deployments.

Key features

• Lightweight by design — Runs as a single standalone process, with no external runtime components or intermediate storage.
• Mature multi-engine coverage — Supports currently validated homogeneous and heterogeneous paths spanning MySQL, PostgreSQL, MongoDB, Redis, Kafka, TiDB, StarRocks, ClickHouse, and Doris.
• Production-ready reliability — Provides checkpoint-based recovery for snapshot and CDC tasks, Prometheus-compatible metrics, and task-level observability.
• Source- and task-specific parallelism — Uses dedicated parallel strategies for different sources, targets, and task types to improve throughput.
• Advanced data capabilities — On supported paths, validates, corrects, and reviews data; filters and routes at the database, table, and column levels; and applies lightweight Lua transformations in flight.

Supported task types

Currently supported mature task types:

	mysql -> mysql	pg -> pg	mongo -> mongo	redis -> redis	mysql -> kafka	pg -> kafka	mysql -> starrocks	mysql -> clickhouse	mysql -> tidb	pg -> starrocks	pg -> clickhouse	mysql -> doris	pg -> doris
Snapshot	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
CDC	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
Data check/revise/review	✔	✔	✔						✔
Structure migration	✔	✔	✔				✔	✔	✔	✔	✔	✔	✔

Quick starts

CLI

dtscli is a lightweight local CLI for creating and managing ApeCloud DTS tasks. It can generate task configs, start dt-main, list tasks, stream logs, and stop, restart, or delete local task records.

For installation and detailed usage, see dt-cli/README.md.

Tutorial

• prerequisites
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql -> starrocks
• mysql -> doris
• mysql -> clickhouse
• mysql -> tidb
• mysql -> kafka -> consumer
• pg -> starrocks
• pg -> doris
• pg -> clickhouse
• pg -> kafka -> consumer
• snapshot + cdc without data loss
• modify data by lua

Run tests

Refer to test docs for details.

More docs

• Configurations
  • config details
• Error handling
  • error codes
• Structure tasks
  • migration
  • check
  • check by Liquibase
• Snapshot tasks
  • data migration
  • data check
  • data revise
  • data review
  • resume at breakpoint
  • multiple tables in parallel
• CDC tasks
  • data sync
  • heartbeat to source database
  • two-way data sync
  • generate sqls from CDC
  • resume at breakpoint
• Custom consumers
  • mysql/pg -> kafka -> consumer
• Data processing
  • modify data by lua
• Monitor
  • monitor info
  • position info
• Task templates
  • mysql -> mysql
  • pg -> pg
  • mongo -> mongo
  • redis -> redis
  • mysql/pg -> kafka
  • mysql -> starrocks
  • mysql -> doris
  • mysql -> clickhouse
  • pg -> starrocks
  • pg -> doris
  • pg -> clickhouse

Benchmark

ape-dts: 2.0.26.1; Environment: Alibaba Cloud, same VPC; MySQL: 8.4.11 8c16g; Full load: 8 tables × 4 million rows; CDC: 8 tables × 1 million baseline rows, 4 million UPDATEs

• MySQL -> MySQL, Snapshot

Node Specs	Rows/s	Source MySQL CPU / RSS (MiB)	Target MySQL CPU / RSS (MiB)
1c2g	109067	7.28% / 8725	112.59% / 14235
2c4g	131783	9.18% / 8636	156.01% / 14291
4c8g	133283	9.06% / 8634	172.36% / 14254

• MySQL -> MySQL, CDC

Node Specs	UPDATE/s	Source MySQL CPU / RSS (MiB)	Target MySQL CPU / RSS (MiB)
1c2g	23477	2.71% / 1989	69.38% / 6915
2c4g	51171	5.73% / 1947	191.92% / 5627
4c8g	52538	5.63% / 2061	193.22% / 6058

• More benchmark details

Contributing

Structure

Modules

• dt-main: program entry

• dt-precheck: pre-check, to minimize interruptions during subsequent data operations by identifying issues early for fast failure

• dt-connector: extractors + sinkers for databases

• dt-pipeline: pipeline to connect extractors and sinkers

• dt-parallelizer: parallel algorithms

• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations

• dt-common: common utils, basic data structures, metadata management

• dt-tests: integration tests

• related sub module: mysql binlog connector in rust

Crate features

The dt-main crate provides several optional components which can be enabled via Cargo [features]:

• metrics: Enable Prometheus format task metrics HTTP service interface. See the task metrics reference for metric names, units, and semantics. After enabling this feature, you can customize the metrics service with the following configuration:

  [metrics]
  # http service host
  http_host=127.0.0.1
  # http service port
  http_port=9090
  # http service worker count
  workers=2
  # prometheus metrics const labels
  labels=your_label1:your_value1,your_label2:your_value2
  

Build

• Minimum supported Rust version (MSRV) The current minimum supported Rust version (MSRV) is 1.85.0.
• cargo build
• build images

Checklist

• run cargo clippy --all-targets --all-features --workspace fix all clippy issues.

Community

If you have any questions, you can reach out to us through:

• ApeDTS GitHub Discussions

• ApeDTS Wechat Account with note ape-dts: