# Testing Strategy ## Layers ### 1. Unit tests (no IO) Pure logic that can be tested with `cargo test`, no containers needed. **`hasher.rs`** (already done) - Deterministic placement for the same key - All volumes appear when requesting full replication - Even distribution across volumes - Correct key path format **`db.rs`** (TODO) - Open an in-memory SQLite (`:memory:`) - Test put/get/delete/list_keys/all_records round-trip - Test upsert behavior (put same key twice) - Test soft delete (deleted flag) - Test bulk_put **Pure decision functions** (TODO, after refactor) - Given a record and a set of healthy volumes, which volume to redirect to? - Given fan-out results (list of Ok/Err), which volumes succeeded? Should we rollback? - Given current vs desired volume placement, what needs to move? ### 2. Volume client tests (mock HTTP) Use a lightweight HTTP server in-process (e.g. `axum` itself or `wiremock`) to test `volume.rs` without real nginx. - PUT blob + .key sidecar → verify both requests made - GET blob → verify body returned - DELETE blob → verify both blob and .key deleted - DELETE non-existent → verify 404 is treated as success - Health check → respond 200 → healthy; timeout → unhealthy ### 3. Integration tests (real nginx) Full end-to-end with Docker containers. These are slower but catch real issues. #### Setup ```yaml # docker-compose.test.yml services: vol1: image: nginx:alpine volumes: - ./tests/nginx.conf:/etc/nginx/conf.d/default.conf - vol1_data:/data ports: ["3101:80"] vol2: image: nginx:alpine volumes: - ./tests/nginx.conf:/etc/nginx/conf.d/default.conf - vol2_data:/data ports: ["3102:80"] vol3: image: nginx:alpine volumes: - ./tests/nginx.conf:/etc/nginx/conf.d/default.conf - vol3_data:/data ports: ["3103:80"] volumes: vol1_data: vol2_data: vol3_data: ``` ```nginx # tests/nginx.conf server { listen 80; root /data; location / { dav_methods PUT DELETE; create_full_put_path on; autoindex on; autoindex_format json; } } ``` ```toml # tests/test_config.toml [server] port = 3100 replication_factor = 2 virtual_nodes = 100 [database] path = "/tmp/mkv-test/index.db" [[volumes]] url = "http://localhost:3101" [[volumes]] url = "http://localhost:3102" [[volumes]] url = "http://localhost:3103" ``` #### Test cases **Happy path** 1. PUT `/hello` with body `"world"` → 201 2. HEAD `/hello` → 200, Content-Length: 5 3. GET `/hello` → 302 to a volume URL 4. Follow redirect → body is `"world"` 5. GET `/` → list contains `"hello"` 6. DELETE `/hello` → 204 7. GET `/hello` → 404 **Replication verification** 1. PUT `/replicated` → 201 2. Read SQLite directly, verify 2 volumes listed 3. GET blob from both volume URLs directly, verify identical content **Volume failure** 1. PUT `/failtest` → 201 2. Stop vol1 container 3. GET `/failtest` → should still 302 to vol2 (healthy replica) 4. PUT `/new-during-failure` → should fail if replication_factor can't be met, or succeed on remaining volumes depending on ring placement 5. Restart vol1 **Rebuild** 1. PUT several keys 2. Delete the SQLite database 3. Run `mkv rebuild` 4. GET all keys → should all still work **Rebalance** 1. PUT several keys with 3 volumes 2. Add a 4th volume to config 3. Run `mkv rebalance --dry-run` → verify output 4. Run `mkv rebalance` → verify keys migrated 5. GET all keys → should all work #### Running integration tests ```bash # Start volumes docker compose -f docker-compose.test.yml up -d # Wait for nginx to be ready sleep 2 # Run integration tests cargo test --test integration # Tear down docker compose -f docker-compose.test.yml down -v ``` The integration test binary (`tests/integration.rs`) starts the mkv server in-process on a random port, runs all test cases sequentially (shared state), then shuts down. ### 4. What we don't test - Performance / benchmarks (follow-up) - TLS, auth (not implemented) - Concurrent writers racing on the same key (SQLite serializes this correctly by design) - Blobs > available RAM (streaming is a follow-up)