mkv/TESTING.md

Testing Strategy

Layers

1. Unit tests (no IO)

Pure logic that can be tested with cargo test, no containers needed.

hasher.rs (already done)

• Deterministic placement for the same key
• All volumes appear when requesting full replication
• Even distribution across volumes
• Correct key path format

db.rs (TODO)

• Open an in-memory SQLite (:memory:)
• Test put/get/delete/list_keys/all_records round-trip
• Test upsert behavior (put same key twice)
• Test soft delete (deleted flag)
• Test bulk_put

Pure decision functions (TODO, after refactor)

• Given a record and a set of healthy volumes, which volume to redirect to?
• Given fan-out results (list of Ok/Err), which volumes succeeded? Should we rollback?
• Given current vs desired volume placement, what needs to move?

2. Volume client tests (mock HTTP)

Use a lightweight HTTP server in-process (e.g. axum itself or wiremock) to test volume.rs without real nginx.

• PUT blob + .key sidecar → verify both requests made
• GET blob → verify body returned
• DELETE blob → verify both blob and .key deleted
• DELETE non-existent → verify 404 is treated as success
• Health check → respond 200 → healthy; timeout → unhealthy

3. Integration tests (real nginx)

Full end-to-end with Docker containers. These are slower but catch real issues.

Setup

# docker-compose.test.yml
services:
  vol1:
    image: nginx:alpine
    volumes:
      - ./tests/nginx.conf:/etc/nginx/conf.d/default.conf
      - vol1_data:/data
    ports: ["3101:80"]

  vol2:
    image: nginx:alpine
    volumes:
      - ./tests/nginx.conf:/etc/nginx/conf.d/default.conf
      - vol2_data:/data
    ports: ["3102:80"]

  vol3:
    image: nginx:alpine
    volumes:
      - ./tests/nginx.conf:/etc/nginx/conf.d/default.conf
      - vol3_data:/data
    ports: ["3103:80"]

volumes:
  vol1_data:
  vol2_data:
  vol3_data:

# tests/nginx.conf
server {
    listen 80;
    root /data;

    location / {
        dav_methods PUT DELETE;
        create_full_put_path on;
        autoindex on;
        autoindex_format json;
    }
}

# tests/test_config.toml
[server]
port = 3100
replication_factor = 2
virtual_nodes = 100

[database]
path = "/tmp/mkv-test/index.db"

[[volumes]]
url = "http://localhost:3101"

[[volumes]]
url = "http://localhost:3102"

[[volumes]]
url = "http://localhost:3103"

Test cases

Happy path

1. PUT /hello with body "world" → 201
2. HEAD /hello → 200, Content-Length: 5
3. GET /hello → 302 to a volume URL
4. Follow redirect → body is "world"
5. GET / → list contains "hello"
6. DELETE /hello → 204
7. GET /hello → 404

Replication verification

1. PUT /replicated → 201
2. Read SQLite directly, verify 2 volumes listed
3. GET blob from both volume URLs directly, verify identical content

Volume failure

1. PUT /failtest → 201
2. Stop vol1 container
3. GET /failtest → should still 302 to vol2 (healthy replica)
4. PUT /new-during-failure → should fail if replication_factor can't be met, or succeed on remaining volumes depending on ring placement
5. Restart vol1

Rebuild

1. PUT several keys
2. Delete the SQLite database
3. Run mkv rebuild
4. GET all keys → should all still work

Rebalance

1. PUT several keys with 3 volumes
2. Add a 4th volume to config
3. Run mkv rebalance --dry-run → verify output
4. Run mkv rebalance → verify keys migrated
5. GET all keys → should all work

Running integration tests

# Start volumes
docker compose -f docker-compose.test.yml up -d

# Wait for nginx to be ready
sleep 2

# Run integration tests
cargo test --test integration

# Tear down
docker compose -f docker-compose.test.yml down -v

The integration test binary (tests/integration.rs) starts the mkv server in-process on a random port, runs all test cases sequentially (shared state), then shuts down.

4. What we don't test

• Performance / benchmarks (follow-up)
• TLS, auth (not implemented)
• Concurrent writers racing on the same key (SQLite serializes this correctly by design)
• Blobs > available RAM (streaming is a follow-up)