CDC
Capture every mutation. Emit to any destination.
WAL-subscription-based change capture that emits structured events to queues, webhooks, files, or custom consumers. Over 100K events/sec throughput.
Change Data Capture (CDC) is an ecosystem component that tails every committed mutation on any WeftKit engine and emits a structured change event to any destination you configure — message queues, webhooks, file streams, or custom HTTP consumers.
CDC provides at-least-once delivery with resume tokens, so downstream consumers can restart and replay events from any checkpoint. Dead letter queues capture failed deliveries for analysis and retry.
How It Works
Step-by-step walkthrough of the internal architecture.
WAL Hook Registration
CDC registers a low-overhead hook directly on the database module's WAL. No additional writes, no performance penalty beyond change event serialization.
Event Serialization
Each change is serialized into a structured event with table name, operation type, before/after row snapshots, transaction ID, and monotonic position.
Routing & Filtering
Events are filtered by schema, table, or operation type. Transformation functions can enrich or reshape events before delivery.
Emitter Dispatch
Filtered events are dispatched to configured emitters: Queue (MPSC), Webhook (HTTP POST), File (append-only log), or a custom HTTP consumer of your choice.
Delivery Guarantee
Each emitter tracks delivery position. Failed deliveries go to the dead letter queue. Resume tokens allow consumers to replay from any checkpoint.
Use Cases
Common deployment patterns and scenarios.
Event-Driven Microservices
Stream database changes to Kafka, RabbitMQ, or NATS for service coordination and eventual consistency patterns.
Audit & Compliance Logging
Capture every data mutation with full before/after snapshots for GDPR, SOC2, and HIPAA audit trails.
Real-Time Analytics
Stream change events to analytical systems for live dashboards, monitoring, and anomaly detection.
Cache Invalidation
Invalidate application caches immediately on data change without polling or manual cache management.
Configuration Reference
TOML configuration options for this component.
[cdc]
# Emitter type: "queue" | "webhook" | "file" | "custom"
emitter = "queue"
# Queue capacity (in-memory, backpressure applied)
queue_capacity = 65536
# Dead letter queue path
dlq_path = "/var/lib/weftkit/cdc-dlq"
# Schema filter (empty = capture all)
schemas = ["public"]
# Batch size for queue emitter
batch_size = 100