haproxy-openmanager

HAProxy OpenManager

Modern, web-based management interface for HAProxy load balancers with multi-cluster support, agent-based pull architecture, and automated deployment capabilities.

Table of Contents

1. Project Overview
   • Agent Pull Architecture
   • Features at a Glance
2. Screenshots
   • Multi-Cluster & Agent Setup
   • Dashboard - Real-time Monitoring
   • HAProxy Configuration
   • Configuration Import & Viewer
   • Apply Management & Version Control
   • Security & Certificate Management
   • ACME Automation
   • Site Wizard - Guided Multi-Step Host Setup
   • WAF Management
   • IP Inventory
3. Key Capabilities
   • Configuration & Entity Management
   • Agent & Cluster Management
   • Version Control & Change Management
   • Monitoring & Security
   • ACME Auto SSL / Let’s Encrypt
   • Integration & API
4. Architecture
   • System Architecture with Agent Pull Model
   • Component Details
   • Agent Pull Communication Flow
5. Features & User Interface
   • Pool Management
   • Cluster Management
   • Agent Management
   • Dashboard
   • Frontend Management
   • New Site Wizard
   • Backend Servers
   • Configuration
   • Apply Management
   • SSL Certificates
   • ACME Auto SSL
   • WAF Management
   • IP Inventory
   • HA / VIP Management
   • User Management
   • Settings
6. Getting Started - First-Time Usage
7. Installation
   • Docker Installation
   • Kubernetes/OpenShift Installation
   • Standalone Installation
   • Local Development
8. Configuration
9. API Reference
10. Development
11. Troubleshooting
12. Contributing
13. Use Cases
14. License
15. Author
16. Support

Project Overview

HAProxy OpenManager is a comprehensive management platform designed to simplify HAProxy administration across multiple environments. It provides a unified interface for managing remote HAProxy instances through an agent-based pull architecture, supporting multi-cluster configurations with centralized control.

Agent Pull Architecture

This project implements an agent pull architecture where management operations are executed through a lightweight agent service (haproxy-agent) installed on each target HAProxy server:

Conceptual Hierarchy

Pool (Logical Grouping)
  └── Cluster (HAProxy Cluster Definition)
       └── Agent 1 → HAProxy Server 1
       └── Agent 2 → HAProxy Server 2
       └── Agent 3 → HAProxy Server 3

Setup Flow

1. Create Agent Pool: First, create a pool to logically group agents (e.g., “Production Pool”)
2. Create Cluster: Select the pool and create a cluster (e.g., “Production Cluster with 3 nodes”)
3. Deploy Agents: Generate installation scripts for each HAProxy server, selecting the same pool
4. Result: Multiple agents in one pool, all managing one cluster

How It Works

1. Automatic Polling: Agents periodically poll the backend for configuration tasks (every 30 seconds)
2. Local Execution: Agents apply changes to local haproxy.cfg and reload HAProxy service
3. No Direct Push: Backend never directly pushes changes - agents pull on their schedule
4. Unified Management: All agents in a pool receive same cluster configuration

This architecture provides better security (no inbound connections to HAProxy servers), resilience (agents can retry on failure), and scalability (backend doesn’t track agent state).

Features at a Glance

✅ Agent-Based Pull Architecture - Secure, scalable management without inbound connections
✅ Multi-Cluster & Pool Management - Organize and manage multiple HAProxy clusters from one interface
✅ Frontend/Backend/Server CRUD - Complete entity management with visual UI
✅ New Site Wizard - Step-by-step guided setup for a complete proxied host (frontend + backend + servers + SSL/ACME) in one consolidated atomic apply
✅ Bulk Config Import - Import existing haproxy.cfg files with smart SSL auto-assignment
✅ Version Control & Rollback - Every change versioned with one-click restore capability
✅ Real-Time Monitoring - Live stats, health checks, and performance dashboards
✅ SSL Certificate Management - Centralized SSL with expiration tracking
✅ CSR Creation (v1.9.0) - Generate a private key + CSR in-app (RSA 2048/4096, ECDSA P-256/P-384, full subject + SANs), have it signed by any external CA, then import the signed certificate — the key never leaves the server
✅ ACME Auto SSL (Let’s Encrypt) - Automated certificate issuance, renewal, and deployment via ACME protocol
✅ ACME DNS-01 Challenge (v1.8.0) - TXT-record validation for internal/isolated clusters (no public port 80) and wildcard certificates; pluggable DNS providers (Manual + Cloudflare + GoDaddy (v1.10.0)), opt-in, HTTP-01 unchanged
✅ ACME Certificate Diagnostic Panel - Automated preflight that checks agent readiness, DNS resolution, port 80 reachability, and ACME challenge ACL before issuing certificates
✅ WAF Rules - Web Application Firewall management and deployment
✅ Agent Script Versioning - Update agents via UI (Monaco editor) with auto-upgrade
✅ Token-Based Agent Auth - Secure token management with revoke/renew
✅ IP Inventory - Cross-cluster IP search to identify agents, VIPs, and backend servers by IP
✅ HA / VIP (Keepalived) Management - Create virtual IPs from the UI; the agent installs & configures Keepalived (unicast VRRP) with a HAProxy health-check so the VIP fails over automatically; live MASTER/BACKUP detection per node
✅ Role-Based User Management - Admin and user roles with granular permissions and access control
✅ User Activity Audit Logs - Complete audit trail of all system events
✅ Request/Response Log (v1.11.0) - Every inbound API call (GETs and errors included) and every outbound HTTP call the backend makes (ACME, Cloudflare, GoDaddy, agents) in one filterable timeline, with redacted, size-capped bodies and operator-configurable retention
✅ REST API - Full programmatic access for automation and CI/CD integration

Screenshots

Multi-Cluster & Agent Setup

Multi-Cluster Management - Cluster selector dropdown showing multiple clusters with agent pools and online agent counts - switch between different clusters

Agent Management - Agent setup wizard with platform selection (Linux/macOS) and architecture (x86_64/ARM64)

Agent Management - Generated installation script ready for deployment to target servers

Dashboard - Real-time Monitoring

Dashboard - Overview tab showing system metrics, active servers, SSL certificates, and cluster agent health

Dashboard - Performance Trends with request rate and response time heatmap (24h view)

Dashboard - Health Matrix showing backend servers with detailed health check status and response times

Dashboard - Response Time Heatmap (24h) with color-coded latency visualization per backend

HAProxy Configuration

Frontend Management - List view with SSL/TLS bindings, sync status, and config status

Frontend Management - Edit modal for configuring bind address, protocol mode, default backend, SSL/TLS, ACLs, and advanced options

Backend Configuration - Backend server and pool configuration page

Backend Management - Backends & Servers tab with 165 servers across 56 backends, health status overview

Configuration Import & Viewer

Bulk Config Import - Paste existing haproxy.cfg and parse configuration

Bulk Config Import - Parsed entities showing 56 frontends, 55 backends, 165 servers with SSL and configuration recommendations

Configuration Viewer - Real-time configuration viewer to pull haproxy.cfg from live agents

Apply Management & Version Control

Apply Management - Pending changes and deployment history with version tracking

Version History - Configuration diff viewer showing added/removed lines with summary

Version History - Complete version history with restore capabilities and deployment status

Security & Certificate Management

Security Management - Agent API token management with active tokens, expiration tracking, and revoke capabilities

ACME Automation - Automated SSL Certificates

ACME Automation - Setup guide with prerequisites checklist, certificate statistics, and order management

ACME Automation - Active certificate orders with PENDING status tracking and renewal schedule

ACME Settings - ACME/SSL Automation configuration with provider selection, staging mode, and auto-renewal settings

Site Wizard - Guided Multi-Step Host Setup

New Site Wizard - Multi-step guided form for creating a complete proxied host (domains + backend pool + servers + frontend + SSL/ACME) with live HAProxy config validation and atomic apply

WAF Management

WAF Management - WAF rule configuration with request filtering, rate limiting, and advanced options

IP Inventory - Cross-Cluster IP Search

IP Inventory - Unified view of all IPs across clusters with search capability to identify which cluster, agent, or backend server an IP belongs to

Key Capabilities

Configuration & Entity Management

• Frontend Management: Create, edit, update, and delete frontend configurations with SSL bindings, ACL rules, and routing
• Backend Management: Full CRUD operations for backend pools with load balancing algorithms and health checks
• Server Management: Add, update, remove, and configure backend servers with weights, maintenance mode, and connection limits
• Bulk Configuration Import: Import existing haproxy.cfg files to instantly manage all frontends, backends, and servers without manual recreation
  • Smart SSL Auto-Assignment: Automatically matches and assigns SSL certificates during bulk import if they exist in SSL Management with matching names and SYNCED status - eliminates manual SSL configuration after import
• Configuration Synchronization: All entities (backends, frontends, servers, SSL certificates, WAF rules) auto-sync to agents
• Configuration Viewer: View live HAProxy configuration pulled from agents via Configuration Management page

Agent & Cluster Management

• Agent-Based Management: Lightweight agent service for secure, pull-based HAProxy management
• Pool Management: Organize agents into logical pools for better cluster organization
• Multi-Cluster Support: Manage multiple HAProxy clusters from a single interface with cluster selector
• Agent Token Management: Secure token-based agent installation with token revoke and renew capabilities
• Agent Script Editor: Monaco code editor for updating agent scripts via UI (not binary) - create new versions, rollback, and auto-upgrade all agents
• Platform-Specific Scripts: Auto-generated installation scripts for Linux and macOS (x86_64/ARM64)
• HA / VIP (Keepalived) Management: Create a virtual IP from the UI and the agents install & configure Keepalived (VRRP) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops — opt-in, with live MASTER/BACKUP per node, multi-distro install, cluster-driven config path, and Apply/Reject staging

Version Control & Change Management

• Apply Management: Centralized change tracking and deployment status monitoring across all agents
• Version Control: Every configuration change creates a versioned snapshot with complete history
• Configuration Restore: Restore any previous configuration version with one-click rollback capability
• Diff Visualization: See exactly what changed between versions with line-by-line comparison
• Change Tracking: Track who made changes, when, and what entities were affected

Monitoring & Security

• Real-time Monitoring: Live statistics, health checks, and performance metrics from HAProxy stats socket
• SSL Management: Centralized certificate management with expiration tracking and zero-downtime updates
• WAF Rules: Web Application Firewall rule management and deployment for security protection
• User Management: Role-based access control with admin and user roles
• User Activity Logs: Complete audit trail of all user actions, configuration changes, and system events

ACME Auto SSL / Let’s Encrypt

• Automated Certificate Issuance: Request SSL certificates from Let’s Encrypt or any ACME-compatible CA directly from the UI
• Automatic Renewal: Hourly background task monitors certificate expiry and creates renewal orders before expiration
• Auto-Completion of Validated Orders (v1.4.0): Independent 60-second background task drives CA-validated orders through finalize -> download -> save without requiring user intervention; multi-replica safe via PostgreSQL FOR UPDATE SKIP LOCKED atomic claim, plus per-order session-level pg_advisory_lock to serialize concurrent completion attempts (UI “Complete” + auto-task race protection)
• Zero-Touch Deployment: Renewed certificates are automatically applied through the same PENDING -> APPLIED pipeline as manual SSL updates, with agent notification
• Stuck Order Detection (v1.4.0): Setup wizard surfaces orders that the CA has validated but not yet downloaded, with one-click Complete action and automatic 60-second retry
• Multi-Provider Support: Configurable ACME directory URL supports Let’s Encrypt, ZeroSSL, Google Trust Services, Buypass, and custom CAs
• HTTP-01 Challenge: Built-in challenge responder with automatic HAProxy routing injection; reserved backend name _acme_challenge_backend is auto-managed and protected from manual edits / agent sync collisions
• DNS-01 Challenge (v1.8.0 — Issue #35): Validate via a DNS TXT record instead of HTTP on port 80, for internal/isolated clusters with no public ingress and for wildcard certificates (*.example.com). Pluggable per-account DNS provider (Manual + Cloudflare + GoDaddy (v1.10.0); credentials encrypted at rest and verified on save), same PENDING → APPLIED pipeline, bounded automatic retry on propagation lag, and a DNS-01 event timeline. Opt-in via a global setting; HTTP-01 behaviour is unchanged. (See the DNS-01 Challenge subsection under ACME Auto SSL below.)
• ACME Account Management: Register, view, and deactivate ACME accounts from the UI
• Staging Mode: Test certificate issuance with Let’s Encrypt staging environment before production
• Custom Staging Endpoint (v1.4.0): Optional staging_url_override setting lets you point staging mode at a private ACME test CA (e.g. Pebble) without touching the production directory URL
• External Account Binding (EAB): Support for CAs that require EAB (ZeroSSL, Google Trust Services). Enter the EAB Key ID and HMAC Key globally in Settings, or per-account in the Register Account dialog (a per-account value overrides the global setting; leave it blank to use the global one)
• Structured Error Diagnostics (v1.4.0): All ACME failures (challenge, finalize, download) persist structured JSON to letsencrypt_orders.error_detail for clear post-mortem analysis
• Audit Logging (v1.4.0): Every ACME operation (request, revoke, CA-chain import, account ops) is captured in user_activity_logs for compliance review
• ACME Diagnostic Panel (v1.5.0 — Issue #13): Live pre-flight + post-failure diagnostics (DNS / port-80 / routing / account / agents) and merged event timeline (acme_order_events + correlated user_activity_logs) accessible from the ACME Automation page; humanized error rendering for 11+ RFC8555 problem types with backwards-compatible fallback for legacy plain-string error_detail; per-user 5/min rate-limit
• New Site Setup Wizard (v1.5.0 — Issue #14): Single guided flow that creates a Backend + Servers + HTTP Frontend (and optional HTTPS Frontend with chosen SSL mode: ACME / Upload / Existing / None) in one atomic transaction, with diff preview, draft persistence (PEM stripped), and ACME-staged order completion gated by agent confirmation. Per-mode reject path cleanly rolls back including any wizard-staged ACME orders.
• Backward Compatible: ACME-managed and manually uploaded certificates coexist seamlessly; existing SSL workflows are completely unaffected

Integration & API

• REST API: Complete API for programmatic access and integration with CI/CD pipelines
• Agent Pull Architecture: Secure polling model - no inbound connections required to HAProxy servers
• Redis Cache: High-performance metrics caching for dashboard with configurable retention

Architecture

System Architecture with Agent Pull Model

Component Details

Agent Pull Communication Flow - Detailed Entity Management

This detailed sequence diagram shows the complete lifecycle of configuration management, including entity creation, agent polling, configuration application, SSL management, WAF rules, and agent upgrade process.

Key Flow Features:

• 🔄 Continuous Polling: Agents poll every 30s for tasks, stats collection, and version checks
• 📄 Config Management: haproxy.cfg backup → validate → apply → rollback if needed
• 🔐 SSL Handling: Secure download, file permissions (chmod 600), automatic frontend binding
• 🤖 ACME Auto-Renewal: Automatic certificate renewal via ACME protocol with same Apply pipeline as manual SSL
• 🛡️ WAF Integration: Stick-tables generated and integrated into config
• 📊 Stats Collection: Parallel stats upload using socat + stats socket
• 🔄 Auto-Upgrade: Agents detect new versions and self-upgrade with zero-touch deployment
• ✅ Validation: Every config change is validated before HAProxy reload
• 📝 Activity Logging: All operations logged in JSON format for audit trail

Features & User Interface

Pool Management - Agent Pool Organization

• Pool Creation: Create logical pools to group HAProxy agents
• Pool Assignment: Associate clusters with specific pools
• Pool Overview: View all agents within a pool
• Multi-Pool Support: Manage multiple pools for different environments (prod, staging, etc.)
• Pool-based Filtering: Filter agents and clusters by pool

Cluster Management - Multi-Cluster HAProxy Administration

• Cluster Selector: Top navigation cluster selector for context switching
• Cluster Creation: Define clusters and associate with agent pools
• Cluster Status: Monitor cluster health and connectivity
• Default Cluster: Set preferred default cluster
• Cross-Cluster View: Compare metrics across multiple clusters
• Cluster-Scoped Operations: All configurations are cluster-specific

Agent Management - HAProxy Agent Deployment & Monitoring

Agent Script Management

Agent scripts are not hardcoded in the project. They are stored in the database and can be updated via the UI:

• Script Editor: Edit agent installation scripts directly in the UI
• Platform Support: Separate scripts for Linux and macOS (x86_64/ARM64)
• Version Control: Create new agent versions with changelog
• Automatic Upgrade: Deploy new agent versions to all agents with one click
• Upgrade Process: Agents check backend version → Download new script → Self-upgrade → Restart

Agent Version Update Flow:

Agent Operations

• Agent Registration: Automatic agent registration after installation
• Installation Scripts: Generate platform-specific scripts (Linux/macOS ARM/x86)
• Script Download: One-click download of customized installation scripts
• Agent Status: Real-time agent connectivity and health monitoring
• Platform Detection: Automatic detection of OS and architecture
• Last Seen: Track when agents last connected to backend (updated every 30s)
• Batch Operations: Upgrade multiple agents simultaneously

Agent Logs & Monitoring

• Log Location: /var/log/haproxy-agent/agent.log on each agent server
• Log Format: Structured JSON for easy parsing and log aggregation
• Log Fields: timestamp, level, message, agent_id, cluster_id, task_id
• Activity Tracking: All agent activities (config updates, SSL changes, task execution)
• Log Integration: JSON format allows easy integration with ELK, Splunk, Datadog
• Log Rotation: Automatic rotation with configurable retention

Example Log Entry (JSON):

{
  "timestamp": "2025-01-15T10:30:45Z",
  "level": "INFO",
  "agent_id": "prod-haproxy-01",
  "cluster_id": 5,
  "task_id": 123,
  "message": "SSL certificate updated successfully",
  "details": {
    "cert_name": "wildcard.example.com",
    "action": "update",
    "haproxy_reload": "success"
  }
}

Dashboard - Main Overview & Real-time Monitoring

The dashboard displays comprehensive real-time metrics collected by agents from HAProxy stats socket. Agents use socat to query the stats socket every 30 seconds and send CSV data to the backend, which parses and stores it in Redis for high-performance access.

Tab-Based Navigation:

1. Overview Tab:

   • System-wide metrics: Total requests, active sessions, error rate
   • Frontend/Backend/Server counts
   • SSL certificate status and expiration warnings
   • Quick health status indicators

2. Frontends Tab:

   • Request rate per frontend with sparkline charts
   • Session distribution across frontends
   • Status (UP/DOWN) monitoring
   • Filter by specific frontends

3. Backends Tab:

   • Backend health matrix (all servers at a glance)
   • Request distribution percentages
   • Response time averages
   • Active/backup server status

4. Performance Trends Tab:

   • Response time trends (1h, 6h, 24h, 7d, 30d views)
   • Request rate trends with interactive zoom
   • Error rate analysis over time
   • Session trends and capacity planning

5. Capacity & Load Tab:

   • Current vs max connections
   • Queue depth monitoring
   • CPU/Memory utilization (if available)
   • Load distribution heatmaps

6. Health Matrix Tab:

   • Detailed server health status
   • Last status check timestamps
   • Check duration and latency
   • Failure reasons and diagnostics

Data Collection Flow:

• Agents → Stats Socket (show stat) → CSV → Backend API → Redis → Dashboard UI
• Agent Log Path: /var/log/haproxy-agent/agent.log (JSON format for easy log aggregation)
• Update Interval: 30 seconds (configurable)
• Data Retention: 30 days in Redis (configurable)

Frontend Management - Virtual Host & Routing Configuration

• Frontend CRUD: Create, edit, delete, and duplicate frontend configurations
• Protocol Support: HTTP, TCP, and health check mode configurations
• Binding Configuration: IP address, port, and SSL binding options
• Backend Assignment: Default backend selection and routing rules
• SSL Configuration: Certificate assignment and SSL protocol settings
• ACL Rules: Access control rules with pattern matching
• Redirect Rules: HTTP to HTTPS redirects and custom redirections
• Advanced Options: Connection limits, timeouts, and performance tuning
• Search & Filter: Real-time search and status-based filtering

New Site Wizard - Guided Multi-Step Host Setup

The New Site Wizard is the recommended entry point for creating a complete proxied host. It collects every piece of configuration a host needs across a five-step guided form and creates the matching frontend + backend + servers + SSL binding as a single atomic apply.

• 5-step guided flow:
  1. Domains - host header / SNI / wildcard validation, IDN/Punycode normalization, port + bind-address collision check
  2. Backend - pool name, load-balance algorithm, health check method/URI, cookie-based persistence (RFC 6265 and HAProxy-parser-safe character validation)
  3. Servers - per-server address/port/weight, optional CA bundle reference (cluster-RBAC enforced), backup-server + cookie-value validators
  4. SSL - three modes: existing certificate (cluster-RBAC), PEM upload (chain validation + SAN/CN match), or ACME order (account binding + preflight)
  5. Review + Dry-Run - live POST /api/sites/preview runs the proposed config through haproxy -c and surfaces every WARNING/ERROR with a marker comment that pinpoints the offending block before anything touches the database
• Draft persistence: every step auto-saves to a server-side draft (30-day TTL, 50 drafts per user cap, cluster-scoped, PEM material stripped at rest)
• Resume across sessions: drafts can be reopened from a different browser; cluster swap mid-wizard surfaces a confirmation prompt to prevent cross-cluster contamination
• Atomic apply: frontend + backend + servers + SSL binding land as a single PENDING change reviewable from Apply Management - accept and apply, or reject as a whole
• Cluster RBAC: every step honours user_pool_access and cluster RBAC; SSL certificate references are validated against the target cluster at both preview AND create time
• Hardening: IDN/Punycode safe slug generation, cookie/header injection guards, path-traversal protection on SSL filenames, per-user-per-minute rate limit on create + preview, advisory locks against concurrent creation
• Backwards compatible URLs: legacy /proxied-hosts/new route continues to resolve to the new wizard

Backend Servers - Server Pool Management

• Server Management: Add, edit, remove, and configure backend servers
• Health Checks: HTTP/TCP health check configuration and monitoring
• Load Balancing: Round-robin, least-conn, source, and URI algorithms
• Server Weights: Dynamic weight adjustment for load distribution
• Maintenance Mode: Enable/disable servers without removing configuration
• Backup Servers: Failover server configuration for high availability
• Connection Limits: Per-server connection and rate limiting
• Monitoring Dashboard: Real-time server status and performance metrics

Configuration - HAProxy Config File Management

• Syntax Editor: Monaco editor with HAProxy syntax highlighting
• Real-time Validation: Configuration syntax checking and error detection
• Configuration Templates: Pre-built templates for common setups
• Version Control: Configuration versioning and rollback capabilities
• Backup & Restore: Automatic backups and manual restore functionality
• Direct Editing: Raw HAProxy configuration file editing
• Deployment: Apply configuration changes with validation
• Help System: Integrated HAProxy directive documentation
• Real-time Config Pull: Pull current haproxy.cfg from live HAProxy servers
  • View actual running configuration
  • Compare with managed configuration
  • Import existing configurations
  • Useful for migrating from manual to managed setup

Apply Management - Change Tracking & Deployment

Change Tracking & Versioning

Every configuration change creates a new version with complete tracking:

• Automatic Versioning: Each apply creates a new version number (e.g., v1.0.123)
• Version Snapshots: Complete configuration state saved for each version
• Diff Visualization: See exactly what changed between versions
  • Green: Added lines
  • Red: Removed lines
  • Yellow: Modified lines
• Change Summary: High-level summary of changes (added frontends, modified backends, etc.)
• Version Metadata: Timestamp, user, cluster, affected entities

Deployment & Status

• Change Tracking: Track all pending configuration changes before apply
• Apply Status: Real-time deployment status per agent
• Bulk Apply: Deploy changes to all agents in a cluster simultaneously
• Success/Failure Tracking: Monitor which agents successfully applied changes
• Retry Mechanism: Retry failed deployments per agent
• Apply Logs: Detailed logs for each deployment operation with error messages

Version History & Restore

Version History Viewer:

• View all historical configuration versions
• Compare any two versions side-by-side
• See who made changes and when
• Filter by date, user, or cluster

Configuration Restore:

• Restore any previous configuration version
• One-click rollback to last known good configuration
• Restore process:
  1. Select previous version from history
  2. Review diff (current vs target version)
  3. Confirm restore
  4. Agents pull restored configuration
  5. HAProxy reloaded with previous config

Use Cases:

• Rollback after problematic deployment
• Audit configuration changes
• Compare production vs staging configurations
• Disaster recovery

SSL Certificates - Centralized TLS/SSL Certificate Management

HAProxy OpenManager provides powerful centralized SSL/TLS certificate management with both global and cluster-specific scopes:

SSL Certificate Scopes

1. Global SSL Certificates:

• Defined once, distributed to all clusters and all agents
• Perfect for wildcard certificates or shared certificates
• When updated, all agents across all clusters automatically receive the new certificate
• Use case: *.example.com certificate used across all environments

2. Cluster-Specific SSL Certificates:

• Scoped to a specific cluster only
• Only agents in that cluster receive these certificates
• Use case: Environment-specific certificates (prod, staging, dev)

Certificate Operations

Upload & Distribution:

1. User uploads certificate via UI (PEM format: cert + private key)
2. Backend stores certificate in database
3. Agents poll backend and detect new/updated certificates
4. Agents download certificates to local disk (/etc/haproxy/certs/)
5. Agents update haproxy.cfg frontend bindings
6. HAProxy is safely reloaded with zero downtime

Binding to Frontends:

• SSL certificates are automatically bound to frontends based on configuration
• Frontend definition specifies which certificate to use
• Format: bind *:443 ssl crt /etc/haproxy/certs/cert-name.pem

Centralized SSL Update Process:

User Updates SSL in UI → All Agents Poll Backend (30s) 
  → Download New Certificate → Update Frontend Bindings 
  → Validate Config → Reload HAProxy (zero downtime)

CSR Workflow (external / corporate CAs) — v1.9.0

For certificates signed by an external or corporate CA, the CSR tab on the SSL Certificates page covers the whole flow without the private key ever leaving the server:

1. Create CSR: pick a name (becomes the certificate name / on-agent file path), Common Name, optional SANs and subject fields (O/OU/L/ST/C/email), and a key algorithm (RSA 2048/4096 or ECDSA P-256/P-384). The backend generates the key + CSR; only the CSR PEM is shown (copy or download as .csr).
2. Get it signed: submit the CSR to your Certificate Authority.
3. Import: paste the signed certificate (+ optional chain), choose Global or cluster-specific scope and usage type. The backend verifies the certificate matches the stored key, rejects expired certs, warns on SAN drift, and creates a normal SSL certificate entry (source: CSR).
4. Deploy: the imported certificate goes through the standard PENDING → Apply Management → agent pull pipeline like any other certificate.

Key Features

• Certificate Upload: PEM format certificate and private key upload
• ACME Automation: Automatic certificate issuance and renewal via Let’s Encrypt / ACME protocol (see ACME Auto SSL)
• Certificate Store: Centralized repository with encryption at rest
• Expiration Monitoring: Automatic expiration tracking with alerts (30, 15, 7 days)
• Global Update: Update one certificate, deploy to all clusters simultaneously
• Domain Binding: Associate certificates with specific domains/frontends
• Certificate Validation: Syntax and format validation before deployment
• Renewal Tracking: Certificate renewal status and history
• Source Tracking: Certificates display their source (“Manual” or “Auto (ACME)”) for clear management
• ACME Protection: ACME-managed certificates are protected from manual content edits to prevent accidental overwrites
• Security Profiles: SSL/TLS protocol and cipher suite configuration
• Zero Downtime: Safe HAProxy reload ensures no dropped connections

ACME Auto SSL - Automated Certificate Management

HAProxy OpenManager includes a built-in ACME client for automated SSL/TLS certificate management using the ACME protocol (RFC 8555). This enables zero-touch certificate issuance and renewal from Let’s Encrypt and other ACME-compatible Certificate Authorities.

How ACME Auto SSL Works

Auto-Renewal Flow

Certificates are automatically renewed before expiration through a background task that runs hourly:

Key behavior: Auto-renewed certificates follow the exact same Apply pipeline as manual SSL updates. The cert transitions PENDING -> APPLIED automatically, agents are notified, and cross-cluster propagation works identically to manual Apply. No separate deployment mechanism is used.

ACME Certificate Diagnostic Panel

Available from the ACME Automation list (Diagnose button on each order, or by clicking the order’s status tag), the Diagnostic Panel runs a full preflight checklist before a certificate is issued or renewed and surfaces every blocker in a single operator-friendly view:

• Agent reachability - verifies that at least one agent in each target cluster is online and within the last-seen window
• DNS resolution - resolves every domain on the order (A / AAAA / CNAME) and flags wildcards and non-resolvable names with the exact upstream resolver error
• Port 80 reachability - checks that the HTTP-01 challenge port is reachable from the public internet via the agent’s egress path
• ACME challenge ACL preview - renders the acl1 !acl1 + http-request return block that will be injected at apply time, so the operator can see exactly what HAProxy will receive
• HSTS / rate-limit policy collision check - warns if existing frontend rules would short-circuit the challenge route
• Account binding check - confirms a valid ACME account exists for the selected provider + cluster combination
• Event Log timeline - merged view of typed acme_order_events rows and correlated user_activity_logs entries; auto-tails every 5s while the order is in pending/processing
• Humanized error display - covers 11+ RFC8555 problem types (badNonce, caa, connection, rateLimited, unauthorized, …) with full backwards compatibility for legacy plain-string error details
• Operator-friendly error envelope - every failure returns { correlation_id, field, cause, remediation } so root-causing is one API call away

ACME Features

ACME Configuration

ACME settings are managed through Settings > ACME / SSL Automation tab:

Cluster ACME Setup

For HTTP-01 challenges to work, each cluster needs ACME challenge routing enabled:

1. Go to Cluster Management > Edit cluster
2. Enable ACME Challenge Routing toggle
3. Optionally set a custom ACME backend URL
4. Apply changes

When enabled, HAProxy configuration is automatically injected with:

• An ACL matching /.well-known/acme-challenge/ requests
• A use_backend rule directing challenge traffic to the management backend
• A dedicated _acme_challenge_backend section

This injection only affects HTTP-mode frontends and is completely removed when ACME is disabled.

ACME Distributed Architecture

Understanding how HTTP-01 challenges work in a distributed HAProxy environment is critical for successful certificate issuance:

┌─────────────────┐    DNS A Record     ┌──────────────────┐
│  Let's Encrypt   │ ──────────────────► │  HAProxy Node(s) │
│  (CA Server)     │  HTTP GET :80       │  (VIP / Public)  │
└─────────────────┘  /.well-known/      └────────┬─────────┘
                      acme-challenge/             │
                      {token}                     │ ACL match →
                                                  │ use_backend
                                                  │ _acme_challenge_backend
                                                  ▼
                                         ┌──────────────────┐
                                         │  OpenManager      │
                                         │  (Backend Server) │
                                         │  Serves token     │
                                         │  response from DB │
                                         └──────────────────┘

Key Points:

1. Let’s Encrypt connects to your HAProxy, not directly to OpenManager
2. DNS must resolve the requested domain to your HAProxy node’s public IP (or VIP/NAT)
3. Port 80 must be open from the internet to HAProxy for HTTP-01 validation
4. HAProxy routes the /.well-known/acme-challenge/ path to OpenManager via the injected backend
5. OpenManager serves the challenge token response stored in the database
6. The MANAGEMENT_BASE_URL or acme.challenge_backend_url setting tells HAProxy where to find OpenManager

Network Scenarios:

DNS-01 Challenge — Internal/Isolated Clusters & Wildcards (v1.8.0 — Issue #35)

The default HTTP-01 challenge validates over port 80, so the domain must resolve publicly to an HAProxy node with ACME Challenge Routing enabled. DNS-01 validates via a DNS TXT record (_acme-challenge.<domain>) instead, so it needs no inbound port 80 and no public ingress to the HAProxy node. Use it for:

• Internal / isolated clusters (behind a VPN/firewall, no public port 80) where you still control the domain’s DNS.
• Wildcard certificates (*.example.com) — which can only be issued via DNS-01.

DNS-01 is opt-in and fully backward compatible: it is disabled until an administrator enables it, and existing HTTP-01 certificates are completely unaffected.

• Enable it: Settings → ACME / SSL Automation → DNS-01 Challenge (advanced) → turn on Enable DNS-01 Challenge and Save. While off, DNS-01 options are hidden and no DNS-01 orders can be created.
• Per-account provider: in ACME Automation, create (or reconfigure) an ACME account with Challenge Method = DNS-01 and a DNS Provider. Provider credentials are verified before saving and encrypted at rest (Fernet, mirroring the VRRP/MFA secret pattern); they are never returned by the API or written to logs.
• Supported providers: Manual (publish the TXT record yourself in any DNS — including fully internal DNS — then click Verify; works everywhere but cannot auto-renew unattended), Cloudflare (API token with Zone:DNS:Edit + Zone:Read; the TXT record is created and cleaned up automatically and renews unattended), and GoDaddy (v1.10.0) (a Production API Key + Secret pair from developer.godaddy.com/keys — the first key that dashboard issues is an OTE/test key and is rejected; the zone must be in the same GoDaddy account, which needs at least one registered domain before GoDaddy allows DNS API access at all. A Personal Access Token works too: paste it as the API Key and leave the Secret blank — that is the forward path as GoDaddy retires the sso-key scheme. TXT records are created and cleaned up automatically and renew unattended). The provider interface is pluggable — more providers can be added without changing the issuance flow.
• Same pipeline: after validation the certificate follows the normal PENDING → APPLIED flow (assign to clusters / Apply Management) and the agent serves it — identical to HTTP-01 from finalize onward, with zero agent or rendered-config changes for DNS-01.
• Manual flow: the order detail shows the exact _acme-challenge.<domain> record name + TXT value (copyable); publish it and click I’ve added the records — Verify. For Cloudflare and GoDaddy it is automatic.
• Resilience: a propagation-lag failure is recovered by a bounded fresh-order retry chain (1 original + 3 retries with increasing backoff, kept under Let’s Encrypt’s rate limits); any orphaned TXT record is cleaned up by a reconcile sweep. The order detail shows a DNS-01 event timeline (publish → validation → cleanup).
• Wildcards: *.example.com is validated at _acme-challenge.example.com; it does not cover the apex — add example.com as a separate name if you need both (the providers handle the two coexisting TXT values automatically).
• Scope (this release): the Site Wizard remains HTTP-01-only; issue DNS-01 / wildcard certificates from ACME Automation.

ACME Quick Start Guide

Follow these steps to obtain your first Let’s Encrypt certificate:

Step 1: Configure ACME Settings

• Navigate to Settings > ACME / SSL Automation tab
• Select your ACME provider (default: Let’s Encrypt)
• For testing, enable Staging Mode to avoid rate limits
• Save changes

Step 2: Register an ACME Account

• On the SSL Certificates > ACME Automation page, find the ACME Account card
• Click Register Account and provide a valid email address
• Accept the Terms of Service

Step 3: Enable ACME on Clusters

• Go to Cluster Management > Edit your cluster
• Enable ACME Challenge Routing toggle
• Set the correct ACME Backend URL if OpenManager is on a different server
• Save the cluster configuration

Step 4: Apply Configuration

• Go to Apply Management and apply the pending HAProxy configuration changes
• This injects the ACME challenge routing rules into HAProxy

Step 5: Verify Challenge Routing

Test that the challenge path is reachable through HAProxy:

curl -v http://your-haproxy-ip/.well-known/acme-challenge/test

Expected: HTTP 404 from OpenManager (not HAProxy’s default 503). This confirms routing works.

Step 6: Ensure DNS Resolution

• Your domain(s) must have DNS A/AAAA records pointing to your HAProxy node’s public IP
• Verify: dig +short yourdomain.com should return the HAProxy IP

Step 7: Request Certificate

• Go to SSL Certificates > ACME Automation tab
• Click New Certificate and follow the wizard
• The UI will guide you through prerequisite checks before submission

Troubleshooting ACME

If certificate issuance fails, check the following:

Diagnostic Logging: OpenManager provides detailed ACME logging with prefixes:

• ACME: — Order creation, challenge responses, finalization
• ACME-CHALLENGE: — Incoming challenge token requests and responses

Review application logs to trace the complete ACME flow when troubleshooting issues.

WAF Management - Web Application Firewall

• Rate Limiting: Request rate limiting by IP, URL, or custom patterns
• IP Filtering: Whitelist/blacklist IP addresses and CIDR ranges
• Geographic Blocking: Country-based access restrictions
• DDoS Protection: Automated DDoS detection and mitigation rules
• Custom Rules: User-defined security rules with regex patterns
• Attack Monitoring: Real-time attack detection and logging
• Rule Templates: Pre-configured security rule templates
• Statistics Dashboard: WAF activity analytics and blocked request metrics

IP Inventory - Cross-Cluster IP Search & Discovery

The IP Inventory page provides a unified view of all IP addresses across every cluster, independent of the global cluster selector. It enables quick identification of which cluster, agent, or backend server a given IP belongs to.

Key Features:

• Cross-Cluster Search: Search any IP address across all clusters simultaneously
• IP Type Identification: Instantly determine if an IP is an agent server IP, a keepalived VIP, or a backend server address
• Unified Results Card: Search results show matched IP type (Server IP, VIP, Backend), agent/server name, cluster association, and status
• Agent Tab: Browse all agents with cluster name, hostname, agent IP, VIP (keepalive), VRRP state (MASTER/BACKUP), status, platform, and last seen time
• Backend Servers Tab: Browse all backend servers with cluster, backend name, server address, status, weight, and check status
• Cluster Color Coding: Each cluster is assigned a consistent color tag for quick visual identification
• Auto-Refresh: Data refreshes automatically every 30 seconds
• Filters: Filter by cluster and status within each tab

Use Cases:

• Identify which HAProxy cluster an IP belongs to during incident response
• Find which agent is running as MASTER or BACKUP in a keepalived pair
• Locate a backend server IP across multiple clusters
• Audit all IP addresses managed by the platform

HA / VIP Management - Keepalived Virtual IP Failover

The HA / VIP page manages highly-available virtual IPs backed by Keepalived (VRRP) directly from the UI — no SSHing into nodes to install/configure Keepalived by hand. It builds on the agent pull-architecture: you define the VIP centrally, click Apply, and the agents converge.

What it does:

• VIP dashboard: lists each virtual IP with its pool, VRID, per-node interface, and a live MASTER / BACKUP / FAULT column per node (refreshed every 30s from the existing keepalive-state heartbeat pipeline).
• Creation form: enter the virtual IP (+ prefix), pick the pool, and select which of the pool’s HAProxy nodes participate — each with a network interface (from the node’s reported interfaces), a VRRP role (MASTER/BACKUP) and a priority. VRID auto-allocates per pool; an optional VRRP secret can be set.
• Hands-off automation: on Apply, each member’s keepalived.conf is rendered and delivered; the agent installs Keepalived if missing, writes the config + a HAProxy health-check (vrrp_script + track_script), validates with keepalived -t, and starts the service. When HAProxy drops on the active node, the health-check lowers VRRP priority and the VIP fails over automatically to a backup.
• Several VIPs per node, several addresses per VIP (v1.12.0): a node can belong to more than one VIP — each VIP is its own vrrp_instance (its own VRID, role and priority) and the node’s keepalived.conf holds all of them. A VIP can also carry up to 19 additional addresses next to its virtual IP; they sit in the same virtual_ipaddress block, on the same interface, and fail over together.

How it works (pull-based, isolated):

• New tables vip_instances + vip_members; the agent polls GET /api/agents/{name}/keepalived-config (key-authenticated, agent-bound) and converges. It is fully isolated from the global HAProxy apply flow — VIP changes never regenerate haproxy.cfg.
• Cluster-driven path: the keepalived.conf location is a cluster setting (keepalived_config_path, default /etc/keepalived/keepalived.conf) delivered to the agent dynamically — just like the HAProxy paths — so non-standard installs are supported with zero per-node effort. The default is what the keepalived service loads on every distro; if you set a non-default path, ensure the keepalived unit is configured to load it.

Apply / Reject staging:

• VIP edits are staged as PENDING and only go live on Apply, which records an applied snapshot. Reject discards pending changes and restores the previous applied state (a never-applied VIP is discarded). Editing a live VIP never disrupts it until you re-Apply.

Enterprise safety:

• Opt-in & backward compatible — nodes/clusters without a VIP do nothing new.
• Never clobbers a hand-managed Keepalived — if an unmanaged keepalived.conf exists, the agent reports “externally managed” and leaves it untouched.
• Multi-distro install — Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine (apt/dnf/yum/zypper/apk).
• Reliable detection across platforms — MASTER/BACKUP is detected via systemd journal, syslog, and a portable interface-based check (exact-match, all VIPs).
• Secure — the VRRP secret is encrypted at rest and never returned by the API; the delivery endpoint requires the node’s own API key.
• Scope — VRRP targets bare-metal / VMware / on-prem L2 networks; the UI clearly notes that AWS/Azure/GCP don’t honor VRRP/gratuitous-ARP. macOS agents

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。