DewDB, from the first write to a sharded cluster.

1. Introduction

One binary, one JSON config file per node, local files for storage, HTTP as the protocol. There is no external coordinator, no separate metadata service, and no driver to install.

A shard owns an append-only write-ahead log, an in-memory sorted index with an inline value cache, snapshots and compaction. A router holds no data: it hashes collection:key, finds the owning group in its cluster view, and forwards. Each group replicates and fails over independently.

• JSON CRUD, merge patch, bulk writes, queries, aggregation and collection administration.
• WAL durability, crash recovery, index snapshots, background compaction and an inline read cache.
• Leader-driven replication, gap repair, snapshot resync, pre-vote elections and write concerns.
• Committed-only visibility, change streams, webhook delivery, health, JSON and Prometheus metrics, and tiered credentials.

2. Installation

Windows, PowerShell

iwr https://windows.dewdb.com -useb | iex

This installs DewDB for the current user under %LOCALAPPDATA%\DewDB and adds it to your user PATH, so it does not need Administrator privileges. Open a new terminal after installing to pick up the PATH change.

dewdb --version
dewdb init
dewdb

Windows, GitHub Releases

Releases are published at github.com/dewdb/dewdb/releases/latest. Download the Windows x86_64 ZIP, extract it, and run:

.\dewdb.exe --version
.\dewdb.exe init
.\dewdb.exe

Each release page carries a SHA256SUMS.txt if you want to verify the download before extracting.

Build from source

You need a Rust toolchain compatible with the Rust 2024 edition.

cargo build --release
./target/release/dewdb --config examples/dew.json

examples/dew.json is a ready-made single-node config on port 8090. With no --config, the binary reads dew.json in the working directory. --version and --help are answered before any config file is read, so they work before you have written one.

3. Quick start

dewdb init writes a starter dew.json. That is all a single node needs: it leads itself, commits on its own fsync, and is ready immediately.

{
  "node_id": "n1",
  "role": "shard",
  "shard_role": "primary",
  "listen_addr": "127.0.0.1:8081",
  "data_dir": "./data"
}

Collections are created on first use, with no schema and no create step.

curl -s -X PUT localhost:8081/collections/users/docs/u1 \
  -H 'content-type: application/json' \
  -d '{"value":{"name":"ada","age":36,"profile":{"city":"london"}}}'

curl -s --get localhost:8081/collections/users/query \
  --data-urlencode 'filter={"age":{"$gte":30},"profile.city":"london"}' \
  --data-urlencode 'sort=age:desc' \
  --data-urlencode 'fields=name,profile.city'

{"items":[{"name":"ada","profile":{"city":"london"}}],"next_cursor":null}

Ctrl-C shuts the node down cleanly: pending group-commit waiters are fsynced and the durable position is recorded before the process exits.

4. Configuration

One JSON object per node. Only node_id, role and listen_addr are required, plus shard_role for a shard that should serve as primary. The whole config tree rejects unknown fields, so a typo fails the boot instead of silently taking a default.

Further blocks tune timing and limits: flow_control, maintenance, rebalance, data_movement, read_cache, changefeed, webhooks and logging. Each has a working default.

5. REST API

All bodies are JSON, and errors are {"error":"…"}.

Documents

Administration

Status codes worth knowing

Nothing unrecognised is reinterpreted. An unknown ?w=, an unknown ?read=, an unparseable sort or a cursor that does not belong to its query is a 400 rather than a silent default.

6. Working with documents

Documents are arbitrary JSON values under string keys. POST mints a UUIDv4; PUT takes the key you supply and replaces the whole value; PATCH applies JSON Merge Patch, so a null field removes it, nested objects merge, and anything else replaces.

• A delete appends a tombstone, so it replicates, commits and recovers exactly like a write.
• A bulk write lands as one group commit, and ownership is checked for every key before anything is written — the batch is accepted whole or refused whole.
• Routing hashes collection:key, so two collections distribute across a cluster independently.
• A collection name is 1–128 characters of a-z0-9._-, checked after percent-decoding and again before it becomes a directory.
• Reads never create a collection. A read of one that does not exist is a 404.

7. Querying

GET /collections/users/query
      ?filter={"age":{"$gte":30}}
      &sort=tier:asc,age:desc
      &fields=name,profile.city
      &start=u&end=v
      &limit=50&cursor=…
      &read=quorum

Operators

Conditions on one field and across fields are ANDed. A document missing the path does not match — $exists: false and $not are the two ways to select for absence. Ordered comparison stays inside one JSON type: $gt: 1 never admits a string, and $gt: "b" never admits a number.

Sorting, projection and paging

• Sort on up to eight dotted paths, each ascending or descending, using a total order across JSON types. The document key decides where they all tie.
• fields=a,b.c returns only the named paths, rebuilding nested structure.
• Every query returns an opaque next_cursor. A sorted cursor is one position for the whole cluster; an unsorted cursor is a keyspace position pinned to the layout it began on.
• limit defaults to 100 and is capped at 10 000. max_docs bounds the walk behind it: 100 000 documents per shard per request by default.
• keys=true returns a parallel keys array; keys=embed returns each row as {id, value}.

Read preferences

8. Secondary indexes

POST   /collections/users/indexes    {"name":"by_age","field":"age"}
GET    /collections/users/indexes
DELETE /collections/users/indexes/by_age

• A definition is a replicated log entry, so it commits under a quorum, survives a leader change and reaches a replica that was down for it.
• The postings are derived, never written to disk: a node that opens a collection rebuilds every index it holds a definition for.
• A build runs in the background, and the planner uses the index only once it is ready — a query is never answered from a half-filled index.
• An index narrows which keys are read, never which rows are returned. Equality, $in, ordered comparison, $prefix, a single $type and $exists: true each name a span of the postings.
• A cluster-wide catalogue carries definitions to a shard group that gains the collection later, after a new shard, a handover, a rebalance or a restart.
• Eight indexes per collection, 64 bytes of index name, 256 bytes and 16 segments of field path.

9. Aggregation

GET /collections/orders/aggregate
      ?filter={"tier":"gold"}
      &group=region,plan
      &metrics=count,sum:amount,avg:amount,min:signed_up,max:signed_up
      &read=quorum&max_docs=100000

Totals are computed on the shard and merged through the totals rather than through the answers: every metric travels with the count behind it, so a router sums sum and count across groups and divides once. An average of shard averages is not an average, and this shape makes that unrepresentable.

• count, sum, avg, min and max, grouped on up to four dotted paths.
• Every response carries scanned beside matched, so the cost of a request is in the answer.
• At most 10 000 distinct groups and 16 metrics. A spent read budget refuses by default; partial=true opts into totals over what was read, labelled partial.
• The same filter, index use and read preference as a query.

10. Change streams and CDC

curl -N --get localhost:8081/collections/orders/changes \
  --data-urlencode 'filter={"status":"paid"}'

event: open
data: {"collection":"orders","position":412}

event: change
id: 413
data: {"lsn":413,"op":"insert","key":"o1","value":{"region":"eu","amount":35,"status":"paid"}}

• Events are published where entries are applied, never where they are appended, so a subscriber never sees a write a leader change can take back.
• Four kinds: insert, update, delete and drop. A drop is one event, not one delete per key.
• Every event's lsn is also its SSE id, so ?after= and a browser's Last-Event-ID both resume exactly where the stream stopped.
• A position older than the buffer is refused with 410 and the resume_floor that still works, rather than silently skipped.
• The same feed is available over a WebSocket at /changes/ws, with the same parameters and the same refusals.
• A subscription may be opened before the collection exists, and captures its first writes.

Webhook delivery

For a consumer that is not running when the change happens, register an endpoint and the same feed is pushed to it.

• Each delivery carries HMAC-SHA256(secret, "<timestamp>.<body>"), with the timestamp inside the signed string. The secret is write-only and never appears in a response.
• Retried until acknowledged, with the same batch and delivery id, on a delay that doubles to a cap and carries a per-subscription spread.
• After a 2xx the position advances through a majority-committed entry, so a restart or a promoted replica resumes where the endpoint got to.
• delivered, attempts, failures, gaps and last_error report what has been happening.

11. Write concerns and durability

Every mutation is a CRC-checked frame appended to a write-ahead log, fsynced through group commit, staged as non-visible, and replicated. Quorum acknowledgements advance a per-collection commit index, and the committed prefix is applied to the read-visible index.

• w=1 waits for local leader durability.
• w=majority waits for a strict majority including the leader.
• w=all waits for every configured acknowledgement.
• w=N waits for an explicit acknowledgement count.

wtimeout bounds the wait. A write that is durable but short of its concern answers 202 with acks and required in the body, so a client that asked for durability and was answered 200 got the durability it asked for.

Recovery truncates a torn tail rather than trusting it, and a restart replays what was committed. Index snapshots bound replay work and are written atomically through a temp file plus rename.

12. Replication and failover

Three nodes, each with its own port and data directory. The primary lists its replicas and peers; a replica is the same file with shard_role set to replica and primary_addr pointing at the leader.

{
  "node_id": "n1",
  "role": "shard",
  "shard_role": "primary",
  "listen_addr": "127.0.0.1:9501",
  "data_dir": "./n1",
  "replicas": ["http://127.0.0.1:9502", "http://127.0.0.1:9503"],
  "peers":    ["http://127.0.0.1:9502", "http://127.0.0.1:9503"]
}

curl -s -X PUT 'localhost:9501/collections/users/docs/u1?w=majority' \
  -H 'content-type: application/json' -d '{"value":{"name":"ada"}}'

Stop the leader. Within the heartbeat timeout plus election jitter, a survivor takes the term and accepts writes, with no operator step and no external arbiter. A replica that fell behind repairs from WAL frames, or takes a full collection snapshot if it fell behind further than retention.

• Term, leadership and vote are persisted as one value, and a node never acts on either before it is durable.
• Pre-vote elections and log-freshness checks keep a node that cannot win from disturbing a healthy leader.
• Commit counts only entries from the current term, and membership moves through a joint configuration.
• Add a replica at runtime as a learner, then promote it with POST /cluster/configuration.

13. Sharding

A router is a node with role: "router". It holds no data: it hashes collection:key, finds the owning group in its cluster view, and forwards. Each group replicates and fails over independently.

                    ┌─► group A (leader + replicas)
client ──► router ──┤
                    └─► group B (leader + replicas)

• Queries and aggregations fan out and merge: sorted pages are merged in order, and aggregate totals are combined through their counts.
• Adding a shard is POST /cluster/migrate: keys move in paced batches while both sides keep serving.
• During a migration a destination's committed copy stays invisible until the ring gives it ownership, so a router sees exactly one row while both groups hold the key.
• A collection may be narrower than the ring. The router counts a shard holding none of it as absence, not as a failure.
• Optional automatic rebalancing reconciles placement after membership changes.

14. Security

{
  "auth": {
    "internal_secret": "cluster-secret",
    "api_keys": ["client-key-1"],
    "admin_keys": ["admin-key-1"],
    "upstream_api_key": "client-key-1"
  }
}

• API keys or bearer tokens guard the public API. Send them as x-api-key or Authorization: Bearer ….
• admin_keys is a separate tier over topology, collection drops, index definitions and webhook routes — listing webhooks included, because a listing names where a node posts documents.
• internal_secret guards /internal/*, the node-to-node routes.
• upstream_api_key is what a router presents to the shards behind it.
• /health stays reachable without credentials; /metrics requires a key when API keys are configured.
• Credentials reload without a restart.

15. Monitoring

GET /health answers 200 ok, or 503 degraded with the reasons, which is what a load balancer wants. The body also carries role, leadership, term, cluster version, uptime and collection count.

{"status":"ok","node_id":"n1","role":"shard","leader":true,"term":0,
 "cluster_version":1,"uptime_secs":3,"collections":0,"reasons":[]}

GET /metrics returns JSON: per-collection documents, WAL bytes, live and dead bytes, last and applied LSN, cached documents, a change-feed block, a replication block, a cluster summary, router state, and per-route request counts with errors and p50/p95/p99 latency. ?format=prometheus exposes the same data as Prometheus text.

Logging is structured tracing output, text or one JSON object per line, both stamped with node_id and a subsystem target. RUST_LOG overrides logging.level.

16. Client snippets

There is no driver to install. Any HTTP client is a first-class client — this is the whole integration.

const base = 'http://localhost:8080';
const headers = { 'content-type': 'application/json', 'x-api-key': 'client-key-1' };

export async function put(collection, id, value, w = 'majority') {
  const r = await fetch(`${base}/collections/${collection}/docs/${id}?w=${w}`, {
    method: 'PUT', headers, body: JSON.stringify({ value }),
  });
  const body = await r.json();
  if (r.status === 202) console.warn('write concern not met', body); // durable, under-replicated
  else if (!r.ok) throw new Error(body.error ?? r.statusText);
  return body;
}

export async function* queryAll(collection, filter) {
  let cursor = null;
  do {
    const q = new URLSearchParams({ limit: '500', filter: JSON.stringify(filter) });
    if (cursor) q.set('cursor', cursor);
    const page = await fetch(`${base}/collections/${collection}/query?${q}`, { headers })
      .then(r => r.json());
    yield* page.items;
    cursor = page.next_cursor;
  } while (cursor);
}

A browser normally reaches a change stream through your own backend, which relays the stream same-origin and forwards Last-Event-ID. The upstream first-application walkthrough has a minimal SSE proxy and the browser side that goes with it.

17. Troubleshooting

This page covers the paths most applications need. For every config field, endpoint, status code, on-disk format and internal protocol, read the reference manual, or start with the install section.