Serverless Persistence (WKP-1) — WeftKit Docs

WeftKit's WKP-1 Serverless Persistence contract lets every WeftKit database engine persist its state to a customer-provided HTTPS endpoint — a Lambda, a Cloud Function, an Azure Function, or any HTTPS URL — instead of requiring an always-on sidecar container. Combined with a horizontally scalable container platform (ECS Fargate, Cloud Run, ACI, plain Kubernetes with rolling restarts), it lets you run a WeftKit database as a stateless-looking, auto-scaling service whose durable state lives outside the pod entirely.

Why this exists

Running a managed relational DB (RDS, Cloud SQL, Cosmos DB) forces you into vertical scaling and a fixed monthly bill. Running WeftKit as a container in an ephemeral runtime gives you horizontal scaling and second-by-second billing — but the container disk vanishes when the pod cycles. WKP-1 solves the ephemerality without introducing a second always-on service:

The engine pushes its durable state to your function on a schedule and right before shutdown.
On boot, the engine pulls the latest state back and is immediately ready.
Scale to zero, scale out, recycle freely — your durable state is in your object store, guarded by your IAM.

Which images implement WKP-1

| Image | WKP-1 | Why | |-------|-------|-----| | weftkit/relational | ✅ | Stateful database engine | | weftkit/document | ✅ | Stateful database engine | | weftkit/graph | ✅ | Stateful database engine | | weftkit/inmemory | ✅ | Stateful database engine (persists snapshots + WAL) | | weftkit/keyvalue | ✅ | Stateful database engine | | weftkit/vector | ✅ | Stateful database engine | | weftkit/markdown | ✅ | Stateful database engine | | weftkit/filestore | ✅ | Stateful database engine | | weftkit/gateway | ✗ | Stateless router | | weftkit/pool | ✗ | Stateless connection proxy | | weftkit/discovery | ✗ | Cluster membership rebuilds from heartbeats | | weftkit/playground | ✗ | Web UI only, no durable data |

Every database engine speaks the same protocol — one function implementation serves every engine you run.

How it works


text
                ┌────────────────────────────┐
                │  Your Lambda / Cloud Func  │───▶  S3 / GCS / Blob
                └────────────────────────────┘
                           ▲  ▲
          POST /persist    │  │  GET /restore
                           │  │
  ┌──────────────┬─────────┴──┴─────────┬──────────────┐
  │ weftkit/     │ weftkit/              │ weftkit/     │
  │ relational   │ vector                │ filestore    │
  │  (Fargate)   │  (Fargate)            │  (Fargate)   │
  └──────────────┴──────────────────────┴──────────────┘

Every engine, on a cron tick and on SIGTERM:
  POST {endpoint}/persist  → uploads a compressed delta/snapshot

Every engine, on boot, if restore-on-boot is enabled:
  GET  {endpoint}/restore  → downloads the latest, hydrates state

Health probe:
  GET  {endpoint}/health   → 200 OK when storage is reachable

Configuration (env vars)

Identical surface on every database engine image:

| Variable | Required | Default | Meaning | |----------|----------|---------|---------| | WEFTKIT_PERSIST_MODE | yes | disabled | disabled | serverless | container | | WEFTKIT_PERSIST_ENDPOINT | yes for serverless | — | HTTPS URL of your function | | WEFTKIT_PERSIST_SECRET | yes for serverless | — | HMAC key — same string on both sides | | WEFTKIT_PERSIST_SCHEDULE | no | */15 * * * * | Cron schedule for periodic push | | WEFTKIT_PERSIST_ON_SHUTDOWN | no | true | Final flush on SIGTERM | | WEFTKIT_PERSIST_RESTORE_ON_BOOT | no | true | GET /restore on cold start if volume is empty | | WEFTKIT_PERSIST_STRATEGY | no | incremental | full | delta | incremental (full daily + deltas) | | WEFTKIT_PERSIST_MAX_RETRIES | no | 5 | Per push/pull attempt | | WEFTKIT_PERSIST_COMPRESSION | no | zstd | zstd | gzip | none | | WEFTKIT_PERSIST_TENANT | no | default | Multi-tenant namespace sent to the function | | WEFTKIT_PERSIST_INSTANCE_ID | no | <uuid> | Stable per-instance id; defaults to a random UUID on first boot, persisted to the volume |

The WKP-1 protocol

Three endpoints. One contract. Same for every engine.

POST `/persist`

Called on each schedule tick and before shutdown.


http
POST /persist HTTP/1.1
Host: persist.example.com
Content-Type: application/octet-stream
Content-Encoding: zstd
Content-Length: <bytes>
X-WeftKit-Protocol: v1
X-WeftKit-Engine: relational
X-WeftKit-Version: 0.1.0
X-WeftKit-Tenant: default
X-WeftKit-Instance-Id: 5d7e3c6b-2b05-43d5-9bb7-ec3d0d3bfb46
X-WeftKit-Strategy: incremental
X-WeftKit-Snapshot-Kind: delta        // "full" | "delta"
X-WeftKit-Cursor-From: wkc_0000_0018
X-WeftKit-Cursor-To:   wkc_0000_0024
X-WeftKit-Checksum: sha256=<hex>
X-WeftKit-Timestamp: 1711999200
X-WeftKit-Request-Id: 2b28f8c8-1b2a-4c5b-8c40-3f1d8d52f012
X-WeftKit-Signature: v1=<hmac-sha256>

<zstd-compressed binary body>

Success:


http
HTTP/1.1 200 OK
Content-Type: application/json

{ "cursor": "wkc_0000_0024", "stored_at": 1711999201 }

Errors:

| Status | Meaning | Engine behaviour | |--------|---------|------------------| | 200 | Stored | Advance local cursor, continue | | 202 | Accepted (async) | Advance local cursor, continue | | 400 | Malformed | Log and drop — never retry | | 401 | Bad signature / expired timestamp | Alert, retry with fresh timestamp | | 409 | Cursor gap | Switch to full strategy, push again | | 413 | Payload too large | Split and retry | | 429 | Throttled | Honour Retry-After, exponential back-off | | 5xx | Transient | Retry up to MAX_RETRIES, then buffer locally until next tick |

GET `/restore`

Called on boot when the data volume is empty (or WEFTKIT_PERSIST_RESTORE_ON_BOOT=true and no cursor is recorded).


http
GET /restore?tenant=default&engine=relational&cursor=&latest=true HTTP/1.1
X-WeftKit-Protocol: v1
X-WeftKit-Engine: relational
X-WeftKit-Instance-Id: 5d7e3c6b-2b05-43d5-9bb7-ec3d0d3bfb46
X-WeftKit-Timestamp: 1711999300
X-WeftKit-Request-Id: 7f3e2a11-09b0-4d74-95b6-1f0a7d7c3b60
X-WeftKit-Signature: v1=<hmac-sha256>

Success:


http
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Encoding: zstd
X-WeftKit-Cursor: wkc_0000_0024
X-WeftKit-Checksum: sha256=<hex>
X-WeftKit-Snapshot-Kind: full

<zstd-compressed binary body>

Nothing persisted yet: 204 No Content. Engine boots with a fresh volume — first-run behaviour.

GET `/health`

Liveness of the persistence path — fires at boot and periodically.


http
GET /health HTTP/1.1
X-WeftKit-Protocol: v1
X-WeftKit-Timestamp: 1711999300
X-WeftKit-Signature: v1=<hmac-sha256>

Return 200 OK if the function can reach its backing store. Any non-2xx makes the engine log a warning but continue serving reads.

Signing

Every request is signed with HMAC-SHA256 over the canonical string:

v1
{X-WeftKit-Timestamp}
{X-WeftKit-Request-Id}
{METHOD}
{PATH + "?" + QUERY}
{sha256(body) or "-" for empty}

Signature header format: v1=<hex-hmac-sha256>.

Engines and functions must reject any request with a timestamp skew greater than 300 seconds. Signature verification is constant-time.

Payload format

The body is opaque to your function — just store and return the bytes. Internally, WeftKit engines use:

Framing: a single compressed blob per request
Default compression: zstd (level 6)
Full snapshot: the engine's page-level state dump plus a WAL cursor
Delta: WAL segments from Cursor-From to Cursor-To
Incremental: one full per day per tenant, delta every tick in between

Your function does not need to understand the content — only store it keyed by tenant, engine, and request ID.

Suggested object-store layout

Your function should key each uploaded blob roughly like:

weftkit/
  {tenant}/
    {engine}/
      {instance-id}/
        full/
          2026-04-21T00:00:00Z_wkc_0000_0019.zst
        delta/
          2026-04-21T00:15:00Z_wkc_0000_0019_to_0024.zst

On GET /restore, return the most recent full plus every delta after its cursor, concatenated (or return one full if the engine asked for that strategy).

Reference implementations

Ready-to-deploy function source code for each cloud lives under /examples/persistence/. Each implementation is self-contained, ~150 lines, and pluggable with your preferred object store.

| Target | Folder | Notes | |--------|--------|-------| | AWS Lambda → S3 | /examples/persistence/aws-lambda-s3/ | Lambda Function URL + S3 | | Google Cloud Function → GCS | /examples/persistence/gcp-cloud-function-gcs/ | 2nd-gen Cloud Functions + GCS | | Azure Function → Blob Storage | /examples/persistence/azure-function-blob/ | Node 20 HTTP trigger + Azure Blob | | Self-hosted Node / Express → any S3-compatible | /examples/persistence/node-selfhosted/ | MinIO, Cloudflare R2, Wasabi — anything @aws-sdk/client-s3 speaks |

Example: single engine on ECS Fargate + Lambda


yaml
# ECS task definition excerpt
containerDefinitions:
  - name: weftkit-relational
    image: weftkit/relational:0.1.0
    portMappings:
      - { containerPort: 20000 }
    environment:
      - name: WEFTKIT_PERSIST_MODE          value: serverless
      - name: WEFTKIT_PERSIST_ENDPOINT      value: https://xxx.lambda-url.us-east-1.on.aws
      - name: WEFTKIT_PERSIST_SECRET        valueFrom: arn:aws:secretsmanager:...:wkp-secret
      - name: WEFTKIT_PERSIST_SCHEDULE      value: "*/5 * * * *"
      - name: WEFTKIT_PERSIST_ON_SHUTDOWN   value: "true"
      - name: WEFTKIT_PERSIST_RESTORE_ON_BOOT value: "true"

No sidecars. No PVCs. Scale to N replicas — each replica identifies itself via Instance-Id, persists independently, and restores its own state on cold start. If you only ever run one replica at a time (for a strongly consistent engine like relational), the engine's internal sequencer ensures a replacement pod picks up where the previous one left off.

Multi-engine, one function

The same function handles every engine. The engine header X-WeftKit-Engine tells it which subtree to write to:


yaml
# docker-compose.yml (self-hosted demo)
services:
  relational:
    image: weftkit/relational:0.1.0
    environment:
      WEFTKIT_PERSIST_MODE: serverless
      WEFTKIT_PERSIST_ENDPOINT: https://persist.example.com
      WEFTKIT_PERSIST_SECRET: ${WKP_SECRET}
  vector:
    image: weftkit/vector:0.1.0
    environment:
      WEFTKIT_PERSIST_MODE: serverless
      WEFTKIT_PERSIST_ENDPOINT: https://persist.example.com
      WEFTKIT_PERSIST_SECRET: ${WKP_SECRET}
  filestore:
    image: weftkit/filestore:0.1.0
    environment:
      WEFTKIT_PERSIST_MODE: serverless
      WEFTKIT_PERSIST_ENDPOINT: https://persist.example.com
      WEFTKIT_PERSIST_SECRET: ${WKP_SECRET}

One endpoint, one secret, three engines — each gets its own subtree in your object store automatically.

Sizing and SLOs

| Dimension | Target | |-----------|--------| | Push overhead on write path | < 1% at default 15 min schedule | | Restore time (1 GiB relational) | < 45 s cold start | | Maximum payload per POST | 100 MiB compressed (split larger snapshots) | | Function execution budget | Typically 10–30 s per push; 60+ s for large filestore restores | | Recommended Lambda memory | 512 MiB (push) / 1024 MiB (restore) | | Timestamp clock skew | ±300 s | | Retry budget per tick | 5 attempts with exponential back-off up to 60 s |

Fallback: container-mode persistence

If you are already running a stateful cluster (Kubernetes with PVCs, bare-metal with local disks) and prefer an always-on central bridge, WEFTKIT_PERSIST_MODE=container keeps the legacy weftkit/persistence image in play. Every engine supports both modes — pick per-engine.

FAQ

Can I use this with multiple replicas of the same engine? Yes. Each replica has its own Instance-Id and writes to its own subtree. The engine's internal sequencer prevents split-brain on recovery.

What happens if my Lambda is unreachable for a while? Engines buffer durable state locally (RAM or scratch disk if available), retry with exponential back-off up to WEFTKIT_PERSIST_MAX_RETRIES, and warn. Final flush on SIGTERM has a 30 s timeout.

Is the traffic encrypted? Always — every request is HTTPS. The HMAC provides integrity; combine with IAM-restricted Lambda URLs or mTLS for private endpoints.

Do I need a database per tenant? No. A single engine handles many tenants; WEFTKIT_PERSIST_TENANT selects the subtree in your object store. If you need hard isolation per tenant, run separate engine containers.

Next steps

Browse the reference implementations
Enable Serverless Persistence in the Compose Builder
Read the Deployment Guide for platform-specific notes (ECS Fargate, Cloud Run, ACI, Kubernetes)