DBLab Engine
⚡ Blazing-fast PostgreSQL cloning and branching 🐘
🛠️ Build powerful dev/test environments.
🔃 Cover 100% of DB migrations with CI tests.
💡 Quickly verify ChatGPT ideas to get rid of hallucinations.
Available for any PostgreSQL, including self-managed and managed services* like AWS RDS, GCP Cloud SQL, Supabase, and Timescale.
It can be installed and used anywhere: across all cloud environments and on-premises.
*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren’t possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.
Why DBLab?
- Build dev/QA/staging environments using full-scale, production-like databases.
- Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
- Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
- Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.
For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.
Try it yourself right now:
How it works
Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.
Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.
Read more:
Where to start
Case studies
Features
- Speed & scale
- Blazing-fast cloning of PostgreSQL databases – clone in seconds, irrespective of database size
- Theoretical max of snapshots/clones: 264 (ZFS, default)
- Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default)
- Support & technologies
- Supported PostgreSQL versions: 10–18
- Thin cloning (CoW) technologies: ZFS and LVM
- UI for manual tasks and API & CLI for automation
- Packaged in Docker containers for all components
- PostgreSQL containers
- Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
- Customization capabilities for containers (docs)
- Docker container and PostgreSQL configuration parameters in the DBLab config
- Source database requirements
- Location flexibility: self-managed PostgreSQL, AWS RDS, GCP Cloud SQL, Azure, etc.—no source adjustments needed.
- No ZFS or Docker requirements for source databases
- Data provisioning & retrieval
- Simple-mode setup: paste source URL + password, get a probed configuration in one click (UI Configuration → Simple tab)
- Expert-mode form for both logical (dump/restore) and physical (WAL-G, pgBackRest) retrieval
- Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
- Partial data retrieval in logical mode (specific databases/tables)
- Database renaming during snapshot creation (
databaseRename option)
- Continuous update in physical mode
- Periodic full refresh in logical mode without downtime
- Recovery & management
- Fast Point in Time Recovery (PITR) for physical mode
- Auto-deletion of unused clones
- Time-based auto-deletion of unused branches and snapshots (
retention config; safe-only, never force-deletes dependents)
- Snapshot retention policies in DBLab configuration
- Clones, branches & snapshots
- “Deletion protection” for clones, branches, and snapshots (blocks manual and auto-deletion, plus count-based retention for snapshots)
- Persistent clones withstand DBLab restarts
- “Reset” command for data version switching
- “Upgrade” command to move a single clone to a newer PostgreSQL major version in place
- Resource quotas: CPU, RAM
- Monitoring & security
/healthz API endpoint (no auth), extended /status endpoint (API docs)- Prometheus metrics endpoint (
/metrics) for monitoring
- Netdata module for insights
Clone major upgrade
A single clone can be moved to a newer PostgreSQL major version without touching the rest of the
instance, which makes it practical to test an upgrade against production-like data:
dblab clone upgrade my-clone
The target version is not a parameter: it follows from the upgrade image the instance is
configured with, which is built on that major’s binaries. The engine reports it in
GET /status as cloneUpgrade.targetVersion, the CLI prints it, and the UI shows it on the
confirmation dialog.
The same action is available in the UI on the clone page and over the API as
POST /clone/{id}/upgrade.
The clone’s Postgres is shut down cleanly, pg_upgrade --link converts its data directory in
place, and the clone restarts on an image of the target major. Because --link hard-links the
data files instead of copying them, the upgrade is fast and consumes almost no extra space. The
request returns as soon as the clone enters the UPGRADING state; watch the clone status for the
result.
Enabling it. Set provision.pgUpgradeImage to an upgrade image of the major you want clones
upgraded to. That image is the single source of the target: the engine reads the major out of a
release tag (:17, :17-0.8.0, :17-0.8.0-glibc236), and re-reads it from the image’s own
PG_MAJOR before every upgrade, so a tag repointed at another build is caught before any data
moves. A tag outside that grammar — a digest pin, latest, a CI build — states no major, so the
engine fetches the image in the background at startup and reads it from there; until that lands
the upgrade reports itself unavailable. When pgUpgradeImage is unset the feature is simply
unavailable and the endpoint says so; nothing else about the instance changes. The UI keeps the
“Upgrade clone” button visible but disabled in that state, and its tooltip carries the reason
reported on /status, so an unset image is one hover away from being diagnosed.
provision:
pgUpgradeImage: "postgresai/pg-upgrade:17"
pgUpgradeTimeout: 3h
pgUpgradePullTimeout: 1h
pgUpgradeTimeout bounds a single upgrade run; it defaults to three hours. When the budget runs
out, the upgrade container is removed and the clone is put back on the version its data directory
still holds, exactly as it would be after any other failure.
pgUpgradePullTimeout bounds each of the two image pulls that precede it, and defaults to one
hour. The upgrade image carries the target major plus the server packages of the four preceding
ones, so the first upgrade on an instance downloads several gigabytes; pre-pulling both images
makes the step a no-op. Both pulls happen while the clone is still serving traffic, so exceeding
the budget leaves the clone untouched and running. The two budgets are separate on purpose — a
slow pull must not eat into the time pg_upgrade gets once the clone is already stopped.
An upgrade request may name an image explicitly instead of letting the engine substitute the major
in the current tag. provision.upgradeImageAllowList says which repositories such a request may
name. When it is empty, as it is by default, only the repository of the clone image is accepted:
the one databaseContainer.dockerImage points at, plus the repository of the clone’s own image
when the clone has already been upgraded to another one. That default accepts exactly the image
the engine would derive itself. List repositories to allow other ones, or "*" to allow any
repository the instance can reach.
provision:
upgradeImageAllowList:
- "postgresai/extended-postgres"
Releases up to v4.2.0 accepted any repository when the list was empty. An instance that relied on
that must now set the list to ["*"].
How it can end. Almost every outcome leaves the clone running:
| Status |
Meaning |
OK |
The clone runs the target version; dbVersion reports it. |
WARNING |
The upgrade was not applied. Either nothing had been converted and the clone still runs its original version, or conversion had begun and the clone was rebuilt from its origin snapshot — in which case data written since the clone was created is lost. The message says which, and points at the pg_upgrade log under <clone dir>/upgrade/logs/. When the postmaster that pg_upgrade started failed, its own fatal error, with the DETAIL and HINT lines that follow it, is appended to the cause, and the run’s pg_upgrade_output.d/<timestamp>/ directory (server log, internal log, schema dumps) is preserved under the same logs/ directory; the five most recent failed runs are kept. |
WARNING, clone not running |
The upgrade settled on disk but the clone container did not come back up. The data directory is intact and the upgrade is still recorded as pending, so the next engine start finishes or undoes it and brings the clone back. |
FATAL |
Recovery itself failed and there is nothing left to finish. Reset or delete the clone. |
Rolling back. Resetting the clone returns it to the version the instance is configured with,
because a reset re-provisions from the origin snapshot using the engine-wide image.
Things to know.
- The target must be newer than the clone’s current major and at most four majors ahead — the
upgrade image carries binaries for the four preceding versions. A clone outside that range is
refused; upgrading it means pointing
pgUpgradeImage at a different major.
- Only majors DBLab ships a default configuration for can be targeted (10–18 today).
- The image tag keeps its extension bundle and glibc suffix; only the major changes
(
…:16-0.8.0-glibc236 → …:17-0.8.0-glibc236). Changing the glibc build across an upgrade
would change collation behaviour, and pg_upgrade does not reindex. Pass --docker-image to
override the choice; when its tag names a major, that major has to be the one being upgraded to.
pg_upgrade verifies that every extension in the source database has a matching library for
the new major. An extension outside the image’s set fails this check, and the clone rolls back
untouched.- Clones using non-default tablespaces are refused: those live outside the clone dataset.
- In physical mode the sync instance and the pool are untouched — the upgrade only ever acts on a
clone’s own dataset. The instance-wide
dockerImage must still match the source major.
- Iteration 1 limitation: a snapshot created from an upgraded clone does not record its major
version, so clones created from that snapshot start on the instance-wide image and will fail to
start. Avoid snapshotting upgraded clones until snapshot-level version resolution lands.
How to contribute
Support us on GitHub/GitLab
The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐
Spread the word
- Tweet about DBLab and mention @Database_Lab.
- Share a link to this repository on your favorite social media platform.
Share your experience
If DBLab has been a vital tool for you, tell the world about your journey. Use the logo from the ./assets folder for a visual touch. Whether it’s in documents, presentations, applications, or on your website, let everyone know you trust and use DBLab.
HTML snippet for lighter backgrounds:
<a href="http://databaselab.io">
<img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>
For darker backgrounds:
<a href="http://databaselab.io">
<img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>
Propose an idea or report a bug
For proposals, bug reports, and participation in development, see our Contributing Guide.
Reference guides
How-to guides
You can find more in the “How-to guides” section of the documentation.
Miscellaneous
License
The DBLab source code is licensed under the OSI-approved open source license Apache 2.0.
Reach out to the PostgresAI team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.
Many thanks to our amazing contributors!
Translations
Making DBLab more accessible to engineers around the globe is a great help for the project. Check details in the translation section of contributing guide.
This README is available in the following translations:
👉 How to make a translation contribution
评论