XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library
Full Documentation
| 中文文档
| README_CN.md
XuanCe is an open-source ensemble of Deep Reinforcement Learning (DRL) algorithm implementations.
We call it as Xuan-Ce (玄策) in Chinese.
“Xuan (玄)” means incredible and magic box, “Ce (策)” means policy.
DRL algorithms are sensitive to hyper-parameters tuning, varying in performance with different tricks,
and suffering from unstable training processes, therefore, sometimes DRL algorithms seems elusive and “Xuan”.
This project gives a thorough, high-quality and easy-to-understand implementation of DRL algorithms,
and hope this implementation can give a hint on the magics of reinforcement learning.
We expect it to be compatible with multiple deep learning backends(
PyTorch,
TensorFlow, and
MindSpore),
and hope it can really become a zoo full of DRL algorithms.
Paper link: https://arxiv.org/pdf/2312.16248.pdf
Table of Contents:
Features
- :school_satchel: Highly modularized.
- :thumbsup: Easy to learn, easy for installation, and easy for usage.
- :twisted_rightwards_arrows: Flexible for model combination.
- :tada: Abundant algorithms with various tasks.
- :couple: Supports both DRL and MARL tasks.
- :key: High compatibility for different users. (PyTorch, TensorFlow2, MindSpore, CPU, GPU, Linux, Windows, MacOS, etc.)
- :zap: Fast running speed with parallel environments.
- :computer: Distributed training with multi-GPUs.
- 🎛️ Support automatically hyperparameters tuning.
- :chart_with_upwards_trend: Good visualization effect with tensorboard or wandb tool.
Algorithms
:point_right: DRL
- DQN: Deep Q Network [Paper]
- Double DQN: DQN with Double Q-learning [Paper]
- Dueling DQN: DQN with Dueling Network [Paper]
- PER: DQN with Prioritized Experience Replay [Paper]
- NoisyDQN: DQN with Parameter Space Noise for Exploration [Paper]
- DRQN: Deep Recurrent Q-Network [Paper]
- QRDQN: DQN with Quantile Regression [Paper]
- C51: Distributional Reinforcement Learning [Paper]
- PG: Vanilla Policy Gradient [Paper]
- NPG: Natural Policy Gradient [Paper]
- PPG: Phasic Policy Gradient [Paper] [Code]
- A2C: Advantage Actor Critic [Paper] [Code]
- SAC: Soft Actor-Critic [Paper] [Code]
- SAC-Discrete: Soft Actor-Critic for Discrete Actions [Paper] [Code]
- PPO-Clip: Proximal Policy Optimization with Clipped Objective [Paper] [Code]
- PPO-KL: Proximal Policy Optimization with KL Divergence [Paper] [Code]
- DDPG: Deep Deterministic Policy Gradient [Paper] [Code]
- TD3: Twin Delayed Deep Deterministic Policy Gradient [Paper][Code]
- P-DQN: Parameterised Deep Q-Network [Paper]
- MP-DQN: Multi-pass Parameterised Deep Q-network [Paper] [Code]
- SP-DQN: Split Parameterised Deep Q-Network [Paper]
:point_right: Model-Based Reinforcement Learning (MBRL)
:point_right: Multi-Agent Reinforcement Learning (MARL)
- IQL: Independent Q-learning [Paper] [Code]
- VDN: Value Decomposition Networks [Paper] [Code]
- QMIX: Q-mixing networks [Paper] [Code]
- WQMIX: Weighted Q-mixing networks [Paper] [Code]
- QTRAN: Q-transformation [Paper] [Code]
- DCG: Deep Coordination Graphs [Paper] [Code]
- IDDPG: Independent Deep Deterministic Policy Gradient [Paper]
- MADDPG: Multi-agent Deep Deterministic Policy Gradient [Paper] [Code]
- IAC: Independent Actor-Critic [Paper] [Code]
- COMA: Counterfactual Multi-agent Policy Gradient [Paper] [Code]
- VDAC: Value-Decomposition Actor-Critic [Paper] [Code]
- IPPO: Independent Proximal Policy Optimization [Paper] [Code]
- MAPPO: Multi-agent Proximal Policy Optimization [Paper] [Code]
- MFQ: Mean-Field Q-learning [Paper] [Code]
- MFAC: Mean-Field Actor-Critic [Paper] [Code]
- ISAC: Independent Soft Actor-Critic
- MASAC: Multi-agent Soft Actor-Critic [Paper]
- MATD3: Multi-agent Twin Delayed Deep Deterministic Policy Gradient [Paper]
- IC3Net: Individualized Controlled Continuous Communication Model [Paper] [Code]
- CommNet: Communication Neural Net [Paper][Code]
:point_right: Contrastive Reinforcement Learning (CRL)
- CURL: Contrastive Unsupervised Representation Learning for Sample-Efficient Reinforcement Learning [Paper] [Code]
- SPR: Data-Efficient Reinforcement Learning with Self-Predictive Representations [Paper] [Code]
- DrQ: Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels [Paper] [Code]
Environments
Cart Pole
|
Pendulum
|
Acrobot
|
MountainCar
|
Bipedal Walker
|
Car Racing
|
Lunar Lander
|
Ant
|
HalfCheetah
|
Hopper
|
HumanoidStandup
|
Humanoid
|
InvertedPendulum
|
...
|
Adventure
|
Air Raid
|
Alien
|
Amidar
|
Assault
|
Asterix
|
Asteroids
|
...
|
GoToDoorEnv
|
LockedRoomEnv
|
MemoryEnv
|
PlaygroundEnv
|
...
|
Helix
|
Single-Agent Hover
|
Multi-Agent Hover
|
...
|
Simple Push
|
Simple Reference
|
Simple Spread
|
Simple Adversary
|
...
|
Example 1
|
Example 2
|
Example 3
|
Example 4
|
...
|
:point_right: Installation
:computer: XuanCe can run at Linux, Windows, MacOS, and EulerOS, etc.
Step 1: Set up a Python environment
We recommend installing Anaconda to manage your Python environment.
(You can also download a specific Anaconda installer from here.)
Then open a terminal and create/activate a new conda environment (Python >= 3.8 is recommended):
conda create -n xuance_env python=3.8 && conda activate xuance_env
Step 2: Install XuanCe
pip install xuance
This command does not include the dependencies of deep learning backends. To install the XuanCe with
deep learning tools, you can type pip install xuance[torch] for PyTorch,
pip install xuance[tensorflow] for TensorFlow2,
pip install xuance[mindspore] for MindSpore,
and pip install xuance[all] for all dependencies.
Note: Some extra packages should be installed manually for further usage.
Click here to see more details for installation.
:point_right: Quickly Start
Train a Model
import xuance
runner = xuance.get_runner(algo='ppo',
env='classic_control',
env_id='CartPole-v1')
runner.run(mode='train')
Test the Model
import xuance
runner = xuance.get_runner(algo='ppo',
env='classic_control',
env_id='CartPole-v1')
runner.run(mode='test')
Visualize the results
Tensorboard
You can use tensorboard to visualize what happened in the training process. After training, the log file will be
automatically generated in the directory “.results/” and you should be able to see some training data after running the
command.
$ tensorboard --logdir ./logs/dqn/torch/CartPole-v0
Weights & Biases (wandb)
XuanCe also supports Weights & Biases (wandb) tools for users to visualize the results of the running implementation.
How to use wandb online? :arrow_right: https://github.com/wandb/wandb.git/
How to use wandb offline? :arrow_right: https://github.com/wandb/server.git/
Benchmarks
XuanCe provides an official benchmark pipeline for evaluating DRL and MARL algorithms.
To avoid increasing the size of the main repository,
official benchmark results (including evaluation curves, summary tables, and pretrained models)
are maintained in a separate repository:
👉 https://github.com/agi-brain/xuance-benchmarks
Users can either:
- Run benchmarks locally using the provided pipeline, or
- Directly inspect and reuse the official benchmark results without rerunning experiments.
(Note: You can also post your questions on Stack Overflow.)
(QR code for QQ group and WeChat official account)
QQ group 1
|
QQ group 2
|
Official account (WeChat)
|
Citations
If you use XuanCe in your research or development, please cite the paper:
@article{liu2023xuance,
title={XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library},
author={Liu, Wenzhang and Cai, Wenzhe and Jiang, Kun and Cheng, Guangran and Wang, Yuanda and Wang, Jiawei and Cao, Jingyu and Xu, Lele and Mu, Chaoxu and Sun, Changyin},
journal={arXiv preprint arXiv:2312.16248},
year={2023}
}
评论