DewDB, from first write to replicated storage.

1. Introduction

DewDB is a lightweight distributed document database written in Rust. It stores JSON documents, creates collections on first write, and exposes a native REST API.

A shard owns an append-only write-ahead log, an in-memory BTreeMap index, snapshots, and compaction. An optional router hashes a collection:id pair with xxh64 and forwards requests to the configured primary.

• JSON CRUD, Merge Patch, bulk writes, queries, and collection administration.
• WAL durability, crash recovery, snapshots, compaction, and inline caching.
• WAL-frame replication, repair, snapshot resync, election, and write concerns.
• Commit-gated visibility plus health, JSON, Prometheus, tracing, and optional auth.

2. Installation

Use a Rust toolchain compatible with the Rust 2024 edition.

git clone <your-dewdb-repository-url>
cd dewdb
cargo build --release

The output is target/release/dewdb on Unix-like systems and target\release\dewdb.exe on Windows.

3. Quick start

Collections are created on first write. Start with one standalone primary.

{
  "node_id": "standalone-1",
  "role": "shard",
  "shard_role": "primary",
  "listen_addr": "127.0.0.1:8081",
  "data_dir": "./standalone-data"
}

cargo run -- --config node.json

curl -X PUT "http://127.0.0.1:8081/collections/users/docs/ada?w=majority" \
  -H "content-type: application/json" \
  -d '{"value":{"name":"Ada","age":36}}'

curl http://127.0.0.1:8081/collections/users/docs/ada

4. Configuration

Every node starts from a JSON configuration file.

Authentication

{
  "auth": {
    "internal_secret": "cluster-secret",
    "api_keys": ["public-client-key"],
    "upstream_api_key": "public-client-key"
  }
}

internal_secret protects /internal/*. Public API keys protect client routes./health stays open; /metrics requires a key when API keys are configured.

5. Running a cluster

A single primary is a quorum of one. A three-node replica set can tolerate one failed node for majority writes. Every node needs the same voting topology, and routers need the same complete, non-overlapping hash map.

cargo run -- --config node-1.json
cargo run -- --config node-2.json
cargo run -- --config node-3.json
cargo run -- --config router.json

6. REST API

Collection routes include GET /collections, DELETE /collections/:name,POST /compact, and POST /snapshot. Router administration can return207 Multi-Status for partial fan-out results.

7. Working with documents

Documents are arbitrary JSON values under string IDs. PUT replaces the complete value;PATCH applies RFC 7396. A null root patch is rejected—use DELETE instead.

8. Querying

Filters support dotted paths, plain equality, and $gt, $gte, $lt,$lte, $ne, and $in. Queries also accept key bounds, limit, one sort key, projections, an opaque cursor, and read=primary|replica.

curl --get "http://127.0.0.1:8080/collections/users/query" \
  --data-urlencode 'filter={"age":{"$gte":18}}' \
  --data-urlencode 'sort=age:desc' \
  --data-urlencode 'fields=name,profile.city' \
  --data-urlencode 'limit=25'

9. Replication and durability

The primary appends a CRC-protected WAL frame, fsyncs through group commit, stages it as non-visible, and sends it to replicas. Quorum acknowledgements advance a per-collection commit index, then the committed prefix is applied to the read-visible index.

• w=1 waits for local primary durability.
• w=majority waits for a strict majority including the primary.
• w=all waits for every configured acknowledgement.
• w=N waits for an explicit acknowledgement count.

Replicas repair predecessor gaps when history is retained. Divergence or compacted-away history triggers a JSON/base64 collection snapshot resync; streaming snapshots remain roadmap work.

10. Consensus and visibility

Consensus-managed collections persist applied.meta. Restart replays only entries through that applied watermark and restages newer durable frames, preventing restart from exposing an uncommitted suffix.

11. Monitoring

GET /health reports node role, term, uptime, status, and reasons. GET /metrics returns JSON;GET /metrics?format=prometheus exposes Prometheus text. Logging supports text or JSON and includes the node ID.

12. Operations

For a clean backup, stop a node gracefully or create a collection snapshot, then copy the collection directory. Prefer primary-driven resync for replica recovery rather than copying live files by hand.

13. Current limitations

• Term and vote persistence is not crash-safe enough for full Raft guarantees.
• Election freshness is a database-level summary, not a full per-log comparison.
• Replication does not yet persist leader-driven nextIndex/matchIndex follower progress.
• A new leader may need a current-term write to carry inherited uncommitted entries forward.
• Quorum reads, membership changes, online resharding, and joint consensus are future work.

14. Troubleshooting