Documents to Markdown Converter Server
This backend server is a robust, scalable solution for effortlessly converting a wide range of document formats—including PDF, DOCX, PPTX, CSV, HTML, JPG, PNG, TIFF, BMP, AsciiDoc, and Markdown—into Markdown. Powered by Docling (IBM’s advanced document parser), this service is built with FastAPI, Celery, and Redis, ensuring fast, efficient processing. Optimized for both CPU and GPU modes, with GPU highly recommended for production environments, this solution offers high performance and flexibility, making it ideal for handling complex document processing at scale.
Comparison to Other Parsing Libraries
| Original PDF |
 |
| Docling-API |
Marker |
 |
 |
| PyPDF |
PyMuPDF4LLM |
 |
 |
Features
Environment Setup (Running Locally)
Prerequisites
- Python 3.12 or higher
- Poetry (Python package manager)
- Redis server (for task queue)
1. Install Poetry (if not already installed)
curl -sSL https://install.python-poetry.org | python3 -
2. Clone and Setup Project
git clone https://github.com/drmingler/docling-api.git
cd docling-api
poetry install
Create a .env file in the project root:
REDIS_HOST=redis://localhost:6379/0
ENV=development
MAX_SIZE_PER_FILE_MB=100
MAX_BATCH_SIZE_MB=500
4. Start Redis Server
Start Redis locally (install if not already installed):
For MacOS:
brew install redis
brew services start redis
For Ubuntu/Debian:
sudo apt-get install redis-server
sudo service redis-server start
5. Start the Application Components
- Start the FastAPI server:
poetry run uvicorn main:app --reload --port 8080
- Start Celery worker (in a new terminal):
poetry run celery -A worker.celery_config worker --pool=solo -n worker_primary --loglevel=info
- Start Flower dashboard for monitoring (optional, in a new terminal):
poetry run celery -A worker.celery_config flower --port=5555
6. Verify Installation
- Check if the API server is running:
curl http://localhost:8080/health
- Test Celery worker:
curl -X POST "http://localhost:8080/documents/convert" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "document=@/path/to/test.pdf"
- Access monitoring dashboard:
Development Notes
- The API documentation is available at http://localhost:8080/docs
- Redis is used as both message broker and result backend for Celery tasks
- The service supports both synchronous and asynchronous document conversion
- For development, the server runs with auto-reload enabled
Environment Setup (Running in Docker)
For detailed Docker instructions, including CPU mode, GPU mode, service URLs, verification commands, configuration, and troubleshooting, see docs/docker.md.
- Clone the repository:
git clone https://github.com/drmingler/docling-api.git
cd docling-api
- Optional: create a
.env file if you need to override the default Docker Redis connection:
REDIS_HOST=redis://redis:6379/0
ENV=production
MAX_SIZE_PER_FILE_MB=100
MAX_BATCH_SIZE_MB=500
CPU Mode
To start the service using CPU-only processing, use the following command. You can adjust the number of Celery workers by specifying the --scale option. In this example, 1 worker will be created:
docker compose up --build --scale celery_worker=1
GPU Mode (Recommend for production)
For production, it is recommended to enable GPU acceleration, as it significantly improves performance. Use the command below to start the service with GPU support. You can also scale the number of Celery workers using the --scale option; here, 3 workers will be launched:
docker compose -f docker-compose.gpu.yml up --build --scale celery_worker=3
Service Components
The service will start the following components:
API Usage
Synchronous Conversion
Convert a single document immediately:
curl -X POST "http://localhost:8080/documents/convert" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "document=@/path/to/document.pdf" \
-F "extract_tables_as_images=true" \
-F "image_resolution_scale=4"
Asynchronous Conversion
- Submit a document for conversion:
curl -X POST "http://localhost:8080/conversion-jobs" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "document=@/path/to/document.pdf"
- Check conversion status:
curl -X GET "http://localhost:8080/conversion-jobs/{job_id}" \
-H "accept: application/json"
Batch Processing
Convert multiple documents asynchronously:
curl -X POST "http://localhost:8080/batch-conversion-jobs" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "documents=@/path/to/document1.pdf" \
-F "documents=@/path/to/document2.pdf"
Configuration Options
image_resolution_scale: Control the resolution of extracted images (1-4)extract_tables_as_images: Extract tables as images (true/false)CPU_ONLY: Build argument to switch between CPU/GPU modesMAX_SIZE_PER_FILE_MB: Maximum size of each uploaded file in MB (default: 100)MAX_BATCH_SIZE_MB: Maximum combined size of files in a batch upload in MB (default: 500)CORS_ALLOW_ORIGINS: Comma-separated list of allowed origins (default: *). Credentials are only sent when explicit origins are configured.CELERY_BROKER_URL / CELERY_RESULT_BACKEND: Optional explicit Celery URLs. If unset, both fall back to REDIS_HOST.
Health
GET /health — liveness probe. Returns 200 {"status": "ok"} when the API process is up.GET /health/ready — readiness probe. Reports whether the Celery broker connection is reachable.
The limits apply to both synchronous conversions and asynchronous conversion jobs. Uploads over either limit return 413 Payload Too Large.
Monitoring
- Access the Flower dashboard to monitor Celery tasks and workers
- View task status, success/failure rates, and worker performance
- Monitor resource usage and task queues
Architecture
The service uses a distributed architecture with the following components:
- FastAPI application serving the REST API
- Celery workers for distributed task processing
- Redis as message broker and result backend
- Flower for task monitoring and management
- Docling for the file conversion
- GPU mode provides significantly faster processing for large documents
- CPU mode is suitable for smaller deployments or when GPU is not available
- Multiple workers can be scaled horizontally for increased throughput
License
The codebase is under MIT license. See LICENSE for more information
Acknowledgements
- Docling the state-of-the-art document conversion library by IBM
- FastAPI the web framework
- Celery for distributed task processing
- Flower for monitoring and management
评论