redis-replicator

Table of Contents (中文说明)

• 1. Redis-replicator
  • 1.1. Brief Introduction
  • 1.2. Chat with Author
  • 1.3. Contact the Author
• 2. Installation
  • 2.1. Requirements
  • 2.2. Maven Dependency
  • 2.3. Install from Source Code
  • 2.4. Select a Version
• 3. Simple Usage
  • 3.1. Basic Usage
  • 3.2. Backup Remote RDB Snapshot
  • 3.3. Backup Remote Commands
  • 3.4. Convert RDB to Dump Format
  • 3.5. RDB Check
  • 3.6. Scan and PSYNC
  • 3.7. Other Examples
• 4. Advanced Topics
  • 4.1. Command Extension
    • 4.1.1. Write a Command
    • 4.1.2. Write a Command Parser
    • 4.1.3. Register the Parser
    • 4.1.4. Handle Command Event
    • 4.1.5. Putting It All Together
  • 4.2. Module Extension
    • 4.2.1. Compile Redis Test Modules
    • 4.2.2. Uncomment in redis.conf
    • 4.2.3. Write a Module Parser
    • 4.2.4. Write a Command Parser
    • 4.2.5. Register Parsers and Handle Events
    • 4.2.6. Putting It All Together
  • 4.3. Stream
  • 4.4. Write Your Own RDB Parser
  • 4.5. Redis URI
• 5. Other Topics
  • 5.1. Built-in Command Parsers
  • 5.2. EOFException
  • 5.3. Trace Event Log
  • 5.4. SSL Connection
  • 5.5. Authentication
  • 5.6. Avoid Full Sync
  • 5.7. Lifecycle Events
  • 5.8. Handle Huge Key-Value Pairs
  • 5.9. Redis 6 Support
    • 5.9.1. SSL Support
    • 5.9.2. ACL Support
  • 5.10. Redis 7 Support
    • 5.10.1. Function
  • 5.11. Redis 7.4 Support
    • 5.11.1. TTL Hash
• 6. Contributors
• 7. Consulting
• 8. References
• 9. Supported By
  • 9.1. 宁文君
  • 9.2. YourKit
  • 9.3. IntelliJ IDEA
  • 9.4. Redisson

1. Redis-replicator

1.1. Brief Introduction

Redis Replicator is an implementation of the Redis Replication protocol written in Java. It can parse, filter, and broadcast RDB and AOF events in real-time. It can also synchronize Redis data to a local cache or a database. In this document, Command refers to writable commands (e.g., set, hmset) and excludes readable commands (e.g., get, hmget). Supports Redis 8.4.x and older versions.

1.2. Chat with Author

1.3. Contact the Author

[email protected]

2. Installation

2.1. Requirements

• Compile: JDK 9+
• Runtime: JDK 8+
• Maven: 3.3.1+
• Redis: 2.6 - 8.4

2.2. Maven Dependency

<dependency>
    <groupId>com.moilioncircle</groupId>
    <artifactId>redis-replicator</artifactId>
    <version>3.11.0</version>
</dependency>

2.3. Install from Source Code

# Step 1: Install JDK 11+ for compilation
# Step 2: Clone the repository
git clone https://github.com/leonchen83/redis-replicator.git
# Step 3: Navigate to the project directory
cd redis-replicator
# Step 4: Build the project
mvn clean install package -DskipTests

2.4. Select a Version

Redis Version	redis-replicator Version
[2.6, 8.4.x]	[3.11.0, ]
[2.6, 8.2.x]	[3.10.0,3.10.0]
[2.6, 8.0.x]	[3.9.0, 3.9.0]
[2.6, 7.2.x]	[3.8.0, 3.8.1]
[2.6, 7.0.x]	[3.6.4, 3.7.0]
[2.6, 7.0.x-RC2]	[3.6.2, 3.6.3]
[2.6, 7.0.0-RC1]	[3.6.0, 3.6.1]
[2.6, 6.2.x]	[3.5.2, 3.5.5]
[2.6, 6.2.0-RC1]	[3.5.0, 3.5.1]
[2.6, 6.0.x]	[3.4.0, 3.4.4]
[2.6, 5.0.x]	[2.6.1, 3.3.3]
[2.6, 4.0.x]	[2.3.0, 2.5.0]
[2.6, 4.0-RC3]	[2.1.0, 2.2.0]
[2.6, 3.2.x]	[1.0.18] (not supported)

3. Simple Usage

3.1. Basic Usage

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof KeyStringValueString) {
            KeyStringValueString kv = (KeyStringValueString) event;
            System.out.println(new String(kv.getKey()));
            System.out.println(new String(kv.getValue()));
        } else {
            // ...
        }
    }
});
replicator.open();

3.2. Backup Remote RDB Snapshot

See RdbBackupExample.java

3.3. Backup Remote Commands

See CommandBackupExample.java

3.4. Convert RDB to Dump Format

You can use DumpRdbVisitor to convert an RDB file to the Redis DUMP format.

Replicator r = new RedisReplicator("redis:///path/to/dump.rdb");
r.setRdbVisitor(new DumpRdbVisitor(r));
r.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (!(event instanceof DumpKeyValuePair)) return;
        DumpKeyValuePair dkv = (DumpKeyValuePair) event;
        byte[] serialized = dkv.getValue();
        // We can use the Redis RESTORE command to migrate this serialized value to another Redis instance.
    }
});
r.open();

3.5. RDB Check

You can use SkipRdbVisitor to check the correctness of an RDB file.

Replicator r = new RedisReplicator("redis:///path/to/dump.rdb");
r.setRdbVisitor(new SkipRdbVisitor(r));
r.open();

3.6. Scan and PSYNC

By default, redis-replicator uses the PSYNC command, pretending to be a replica, to receive commands. An example is as follows:

Replicator r = new RedisReplicator("redis://127.0.0.1:6379");
r.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        System.out.println(event);
    }
});

r.open();

However, on some cloud services, the PSYNC command is prohibited. In such cases, you can use the SCAN command instead:

Replicator r = new RedisReplicator("redis://127.0.0.1:6379?enableScan=yes&scanStep=256");
r.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        System.out.println(event);
    }
});

r.open();

3.7. Other Examples

See examples

4. Advanced Topics

4.1. Command Extension

4.1.1. Write a Command

@CommandSpec(command = "APPEND")
public static class YourAppendCommand extends AbstractCommand {
    private final String key;
    private final String value;

    public YourAppendCommand(String key, String value) {
        this.key = key;
        this.value = value;
    }
    
    public String getKey() {
        return key;
    }
    
    public String getValue() {
        return value;
    }
}

4.1.2. Write a Command Parser

public class YourAppendParser implements CommandParser<YourAppendCommand> {
    @Override
    public YourAppendCommand parse(Object[] command) {
        return new YourAppendCommand(new String((byte[]) command[1], UTF_8), new String((byte[]) command[2], UTF_8));
    }
}

4.1.3. Register the Parser

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
replicator.addCommandParser(CommandName.name("APPEND"), new YourAppendParser());

4.1.4. Handle Command Event

replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if(event instanceof YourAppendCommand){
            YourAppendCommand appendCommand = (YourAppendCommand)event;
            // Your code goes here
        }
    }
});

4.1.5. Putting It All Together

See CommandExtensionExample.java

4.2. Module Extension

4.2.1. Compile Redis Test Modules

cd /path/to/redis-4.0-rc2/src/modules
make

4.2.2. Uncomment in redis.conf

loadmodule /path/to/redis-4.0-rc2/src/modules/hellotype.so

4.2.3. Write a Module Parser

public class HelloTypeModuleParser implements ModuleParser<HelloTypeModule> {
    @Override
    public HelloTypeModule parse(RedisInputStream in, int version) throws IOException {
        DefaultRdbModuleParser parser = new DefaultRdbModuleParser(in);
        int elements = parser.loadUnsigned(version).intValue();
        long[] ary = new long[elements];
        int i = 0;
        while (elements-- > 0) {
            ary[i++] = parser.loadSigned(version);
        }
        return new HelloTypeModule(ary);
    }
}

public class HelloTypeModule implements Module {
    private final long[] value;

    public HelloTypeModule(long[] value) {
        this.value = value;
    }

    public long[] getValue() {
        return value;
    }
}

4.2.4. Write a Command Parser

public class HelloTypeParser implements CommandParser<HelloTypeCommand> {
    @Override
    public HelloTypeCommand parse(Object[] command) {
        String key = new String((byte[]) command[1], Constants.UTF_8);
        long value = Long.parseLong(new String((byte[]) command[2], Constants.UTF_8));
        return new HelloTypeCommand(key, value);
    }
}

@CommandSpec(command = "hellotype.insert")
public class HelloTypeCommand extends AbstractCommand {
    private final String key;
    private final long value;

    public long getValue() {
        return value;
    }

    public String getKey() {
        return key;
    }

    public HelloTypeCommand(String key, long value) {
        this.key = key;
        this.value = value;
    }
}

4.2.5. Register Parsers and Handle Events

public static void main(String[] args) throws IOException {
    Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
    replicator.addCommandParser(CommandName.name("hellotype.insert"), new HelloTypeParser());
    replicator.addModuleParser("hellotype", 0, new HelloTypeModuleParser());
    replicator.addEventListener(new EventListener() {
        @Override
        public void onEvent(Replicator replicator, Event event) {
            if (event instanceof KeyStringValueModule) {
                System.out.println(event);
            }
            
            if (event instanceof HelloTypeCommand) {
                System.out.println(event);
            }
        }
    });
    replicator.open();
}

4.2.6. Putting It All Together

See ModuleExtensionExample.java

4.3. Stream

Since Redis 5.0, a new data structure called STREAM has been added. Redis-replicator parses STREAM data as follows:

Replicator r = new RedisReplicator("redis://127.0.0.1:6379");
r.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof KeyStringValueStream) {
            KeyStringValueStream kv = (KeyStringValueStream)event;
            // Key
            String key = kv.getKey();
            
            // Stream
            Stream stream = kv.getValueAsStream();
            // Last stream ID
            stream.getLastId();
            
            // Entries
            NavigableMap<Stream.ID, Stream.Entry> entries = stream.getEntries();
            
            // Optional: Groups
            for (Stream.Group group : stream.getGroups()) {
                // Group PEL (Pending Entries List)
                NavigableMap<Stream.ID, Stream.Nack> gpel = group.getPendingEntries();
                
                // Consumers
                for (Stream.Consumer consumer : group.getConsumers()) {
                    // Consumer PEL (Pending Entries List)
                    NavigableMap<Stream.ID, Stream.Nack> cpel = consumer.getPendingEntries();
                }
            }
        }
    }
});
r.open();

4.4. Write Your Own RDB Parser

• Write a YourRdbVisitor that extends RdbVisitor.
• Register your RdbVisitor with the Replicator using the setRdbVisitor method.

4.5. Redis URI

Before version 2.4.0, RedisReplicator was constructed as follows:

Replicator replicator = new RedisReplicator("127.0.0.1", 6379, Configuration.defaultSetting());
Replicator replicator = new RedisReplicator(new File("/path/to/dump.rdb"), FileType.RDB, Configuration.defaultSetting());
Replicator replicator = new RedisReplicator(new File("/path/to/appendonly.aof"), FileType.AOF, Configuration.defaultSetting());
Replicator replicator = new RedisReplicator(new File("/path/to/appendonly.aof"), FileType.MIXED, Configuration.defaultSetting());

Since version 2.4.0, we have introduced the Redis URI concept to simplify the RedisReplicator construction process:

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
Replicator replicator = new RedisReplicator("redis:///path/to/dump.rdb");
Replicator replicator = new RedisReplicator("redis:///path/to/appendonly.aof");

// Configuration setting example
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379?authPassword=foobared&readTimeout=10000&ssl=yes");
Replicator replicator = new RedisReplicator("redis:///path/to/dump.rdb?rateLimit=1000000");
Replicator replicator = new RedisReplicator("rediss://user:[email protected]:6379?rateLimit=1000000");

5. Other Topics

5.1. Built-in Command Parsers

Command	Command	Command	Command	Command	Command
PING	APPEND	SET	SETEX	MSET	DEL
SADD	HMSET	HSET	LSET	EXPIRE	EXPIREAT
GETSET	HSETNX	MSETNX	PSETEX	SETNX	SETRANGE
HDEL	UNLINK	SREM	LPOP	LPUSH	LPUSHX
LREM	RPOP	RPUSH	RPUSHX	ZREM	ZINTERSTORE
INCR	DECR	INCRBY	PERSIST	SELECT	FLUSHALL
FLUSHDB	HINCRBY	ZINCRBY	MOVE	SMOVE	BRPOPLPUSH
PFCOUNT	PFMERGE	SDIFFSTORE	RENAMENX	PEXPIREAT	SINTERSTORE
ZADD	BITFIELD	SUNIONSTORE	RESTORE	LINSERT	ZREMRANGEBYLEX
GEOADD	PEXPIRE	ZUNIONSTORE	EVAL	SCRIPT	ZREMRANGEBYRANK
PUBLISH	BITOP	SETBIT	SWAPDB	PFADD	ZREMRANGEBYSCORE
RENAME	MULTI	EXEC	LTRIM	RPOPLPUSH	SORT
EVALSHA	ZPOPMAX	ZPOPMIN	XACK	XADD	XCLAIM
XDEL	XGROUP	XTRIM	XSETID	COPY	LMOVE
BLMOVE	ZDIFFSTORE	GEOSEARCHSTORE	FUNCTION	SPUBLISH	HPERSIST
HSETEX	HPEXPIREAT	XACKDEL	XDELEX	MSETEX

5.2. EOFException

When event consumption is too slow and the backlog of events exceeds the Redis backlog limit, Redis will actively disconnect from the slave. When Redis-replicator reconnects, it will perform a full synchronization. To avoid this situation, you need to set the parameter client-output-buffer-limit slave 0 0 0.

For more details, please refer to redis.conf.

client-output-buffer-limit slave 0 0 0

WARNING: This setting may cause the Redis server to run out of memory in some cases.

5.3. Trace Event Log

• Set the log level to debug.
• If you are using Log4j2, add a logger as shown below:

<Logger name="com.moilioncircle" level="debug">
    <AppenderRef ref="YourAppender"/>
</Logger>

Configuration.defaultSetting().setVerbose(true);
// As a Redis URI parameter
"redis://127.0.0.1:6379?verbose=yes"

5.4. SSL Connection

System.setProperty("javax.net.ssl.keyStore", "/path/to/keystore");
System.setProperty("javax.net.ssl.keyStorePassword", "password");
System.setProperty("javax.net.ssl.keyStoreType", "your_type");

System.setProperty("javax.net.ssl.trustStore", "/path/to/truststore");
System.setProperty("javax.net.ssl.trustStorePassword", "password");
System.setProperty("javax.net.ssl.trustStoreType", "your_type");

Configuration.defaultSetting().setSsl(true);

// Optional settings
Configuration.defaultSetting().setSslSocketFactory(sslSocketFactory);
Configuration.defaultSetting().setSslParameters(sslParameters);
Configuration.defaultSetting().setHostnameVerifier(hostnameVerifier);

// As a Redis URI parameter
"redis://127.0.0.1:6379?ssl=yes"
"rediss://127.0.0.1:6379"

If you prefer not to use System.setProperty, you can configure it programmatically as follows:

RedisSslContextFactory factory = new RedisSslContextFactory();
factory.setKeyStorePath("/path/to/redis/tests/tls/redis.p12");
factory.setKeyStoreType("pkcs12");
factory.setKeyStorePassword("password");

factory.setTrustStorePath("/path/to/redis/tests/tls/redis.p12");
factory.setTrustStoreType("pkcs12");
factory.setTrustStorePassword("password");

SslConfiguration ssl = SslConfiguration.defaultSetting().setSslContextFactory(factory);
Replicator replicator = new RedisReplicator("rediss://127.0.0.1:6379", ssl);

5.5. Authentication

Configuration.defaultSetting().setAuthUser("default");
Configuration.defaultSetting().setAuthPassword("foobared");

// As a Redis URI parameter
"redis://127.0.0.1:6379?authPassword=foobared&authUser=default"
"redis://default:[email protected]:6379"

5.6. Avoid Full Sync

Adjust the Redis server settings as follows:

repl-backlog-size
repl-backlog-ttl
repl-ping-slave-period

The repl-ping-slave-period MUST be less than Configuration.getReadTimeout(). The default Configuration.getReadTimeout() is 60 seconds.

5.7. Lifecycle Events

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
final long start = System.currentTimeMillis();
final AtomicInteger acc = new AtomicInteger(0);
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if(event instanceof PreRdbSyncEvent) {
            System.out.println("pre rdb sync");
        } else if(event instanceof PostRdbSyncEvent) {
            long end = System.currentTimeMillis();
            System.out.println("time elapsed:" + (end - start));
            System.out.println("rdb event count:" + acc.get());
        } else {
            acc.incrementAndGet();
        }
    }
});
replicator.open();

5.8. Handle Huge Key-Value Pairs

As mentioned in 4.4. Write Your Own RDB Parser, this tool has a built-in Iterable Rdb Parser to handle huge key-value pairs. For more details, please refer to: [1] HugeKVFileExample.java [2] HugeKVSocketExample.java

5.9. Redis 6 Support

5.9.1. SSL Support

cd /path/to/redis
./utils/gen-test-certs.sh
cd tests/tls
openssl pkcs12 -export -CAfile ca.crt -in redis.crt -inkey redis.key -out redis.p12
cd /path/to/redis
./src/redis-server --tls-port 6379 --port 0 --tls-cert-file ./tests/tls/redis.crt \
     --tls-key-file ./tests/tls/redis.key --tls-ca-cert-file ./tests/tls/ca.crt \
     --tls-replication yes --bind 0.0.0.0 --protected-mode no

System.setProperty("javax.net.ssl.keyStore", "/path/to/redis/tests/tls/redis.p12");
System.setProperty("javax.net.ssl.keyStorePassword", "password");
System.setProperty("javax.net.ssl.keyStoreType", "pkcs12");

System.setProperty("javax.net.ssl.trustStore", "/path/to/redis/tests/tls/redis.p12");
System.setProperty("javax.net.ssl.trustStorePassword", "password");
System.setProperty("javax.net.ssl.trustStoreType", "pkcs12");

Replicator replicator = new RedisReplicator("rediss://127.0.0.1:6379");

5.9.2. ACL Support

Replicator replicator = new RedisReplicator("redis://user:[email protected]:6379");

5.10. Redis 7 Support

5.10.1. Function

Since Redis 7.0, FUNCTION is supported, and its structure is stored in the RDB file. You can use the following method to parse a FUNCTION.

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof Function) {
            Function function = (Function) event;
            function.getCode();
                
            // Your code goes here
        }
    }
});
replicator.open();

You can also parse a FUNCTION into serialized data and use FUNCTION RESTORE to restore it to a target Redis instance.

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
replicator.setRdbVisitor(new DumpRdbVisitor(replicator));
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof DumpFunction) {
            DumpFunction function = (DumpFunction) event;
            byte[] serialized = function.getSerialized();
            // Your code goes here
            // You can use FUNCTION RESTORE to restore the serialized data to a target Redis instance
        }
    }
});
replicator.open();

5.11. Redis 7.4 Support

5.11.1. TTL Hash

Since Redis 7.4, TTL HASH is supported, and its structure is stored in the RDB file. You can use the following method to parse a TTL HASH.

Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof KeyStringValueTTLHash) {
            KeyStringValueTTLHash skv = (KeyStringValueTTLHash) event;
            // Key
            byte[] key = skv.getKey();
            
            // TTL Hash
            Map<byte[], TTLValue> ttlHash = skv.getValue();
            for (Map.Entry<byte[], TTLValue> entry : ttlHash.entrySet()) {
                System.out.println("field: " + Strings.toString(entry.getKey()));
                System.out.println("value: " + Strings.toString(entry.getValue().getValue()));
                System.out.println("field ttl: " + entry.getValue().getExpires());
            }
        }
    }
});
replicator.open();

6. Contributors

• Leon Chen
• Adrian Yao
• Trydofor
• Argun
• Sean Pan
• René Kerner
• Maplestoria
• Special thanks to Kevin Zheng

7. Consulting

Commercial support for redis-replicator is available. The following services are currently offered:

• Onsite consulting: $10,000 per day
• Onsite training: $10,000 per day

You may also contact Baoyi Chen directly at [email protected].

8. References

• rdb.c
• Redis RDB File Format
• Redis Protocol specification
• Redis Replication
• Redis-replicator Design and Implementation

9. Supported By

9.1. 宁文君

January 27, 2023, was a sad day as I lost my mother, 宁文君. She was always encouraging and supportive of my work on this tool. Every time a company started using it, she would get as excited as a child and motivate me to continue. Without her, I could not have maintained this tool for so many years. Even though I haven’t achieved much, she was always proud of me. R.I.P, and may God bless her.

9.2. YourKit

YourKit is kindly supporting this open source project with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit’s leading software products: YourKit Java Profiler and YourKit .NET Profiler.

9.3. IntelliJ IDEA

IntelliJ IDEA is a Java Integrated Development Environment (IDE) for developing computer software. It is developed by JetBrains (formerly known as IntelliJ), and is available as an Apache 2 Licensed community edition, and in a proprietary commercial edition. Both can be used for commercial development.

9.4. Redisson

Redisson, a Redis-based In-Memory Data Grid for Java, offers distributed objects and services (BitSet, Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, Bloom filter, Remote service, Spring cache, Executor service, Live Object service, Scheduler service) backed by a Redis server. Redisson provides a more convenient and easier way to work with Redis. Redisson objects provide a separation of concerns, allowing you to focus on data modeling and application logic.