From 1461b41a36f84d0f719e90d7794b33b30295d4c3 Mon Sep 17 00:00:00 2001 From: Silas Brack Date: Sat, 7 Mar 2026 11:05:05 +0100 Subject: [PATCH] Progress --- .gitignore | 2 +- TODO.md | 54 +++++++++++++----------------------------------------- 2 files changed, 14 insertions(+), 42 deletions(-) diff --git a/.gitignore b/.gitignore index 2c4918c..417367c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,2 @@ /target -*.db +*.db* diff --git a/TODO.md b/TODO.md index a65f78f..093fcaf 100644 --- a/TODO.md +++ b/TODO.md @@ -5,55 +5,27 @@ - [x] SQLite schema + pragmas + ReadPool + WriterHandle (`db.rs`) - [x] Config parsing with validation (`config.rs`) - [x] Error types with axum IntoResponse (`error.rs`) -- [x] Consistent hash ring with virtual nodes (`hasher.rs`) -- [x] Volume HTTP client with .key sidecar writes (`volume.rs`) +- [x] Consistent hash ring with virtual nodes + stability tests (`hasher.rs`) +- [x] Volume HTTP client — key-as-path, no sidecar files (`volume.rs`) - [x] Health checker background task (`health.rs`) -- [x] HTTP handlers: GET (302 redirect), PUT (fan-out), DELETE, HEAD, list (`server.rs`) +- [x] HTTP handlers with pure decision functions extracted (`server.rs`) - [x] CLI with clap subcommands (`main.rs`) -- [x] Wiring: state, routes, health checker startup +- [x] Rebuild tool — recursive nginx autoindex walk (`rebuild.rs`) +- [x] Rebalance tool with --dry-run (`rebalance.rs`) +- [x] Integration tests — 8 tests against real nginx (Docker Compose) +- [x] Unit tests — 17 tests (hasher, server pure fns, rebalance planning) +- [x] Key-as-path simplification (removed content-addressed paths + .key sidecars) ## Remaining -### Core features - -- [ ] **Rebuild tool** (`tools/rebuild.rs`) - - Scan all volumes for `.key` sidecar files (via nginx autoindex JSON) - - Read each `.key` file to recover the original key name - - Verify the blob exists alongside each `.key` - - Reconstruct SQLite index, merging replicas (same key on multiple volumes) - - Uses `db::bulk_put` for efficient batch inserts - -- [ ] **Rebalance tool** (`tools/rebalance.rs`) - - Load current index, compute desired placement via new ring config - - Diff current vs desired volumes per key - - Copy blobs to new volumes, delete from old ones - - Update SQLite records - - `--dry-run` flag to preview without acting - - Progress output: keys moved, bytes transferred, errors - -### Refactoring - -- [ ] **Extract pure decision logic from handlers** (see PHILOSOPHY.md) - - `put_key`: separate "compute placement" from "execute IO" - - `get_key`: separate "pick healthy volume" from "build redirect" - - Make fan-out results processing a pure function +### Worth doing - [ ] **Typed volume errors** — replace `String` errors in `volume.rs` with a proper enum - -### Testing - -- [ ] **Unit tests for `db.rs`** — CRUD with an in-memory SQLite -- [ ] **Integration test harness** (see TESTING.md) - - Docker Compose with 3 nginx volume containers - - Rust integration tests that spin up the full server - - PUT/GET/DELETE/HEAD happy path - - Failure test: kill a volume mid-test, verify reads still work from replica - - Rebalance test: add 4th volume, rebalance, verify all keys accessible -- [ ] **Property-based tests for hasher** — verify ring invariants under add/remove - -### Polish - +- [ ] **Unit tests for `db.rs`** — CRUD round-trip with in-memory SQLite - [ ] **Graceful shutdown** — drain in-flight requests, flush writer + +### Nice to have + - [ ] **Metrics** — request count, latency histograms, volume error rates - [ ] **Request ID / tracing spans** — per-request trace context - [ ] **Config reload** — SIGHUP to reload config without restart