Schema evolution for metering

​

Current flow

1. The Durable Object buffer stores new usage and verification records in SQLite.
2. The flush pipeline sends the same batch to Tinybird and raw R2 NDJSON files.
3. The buffer deletes rows only after Tinybird and R2 both confirm the range.
4. Compaction merges raw R2 files by day for lakehouse queries.

​

Compatibility contract

• Every event includes schema_version.
• Prefer additive changes, such as a new nullable field.
• Keep old fields during the migration window so dashboards and SQL consumers continue to work.
• Treat a schema change as a multi-sink rollout, not as one table change.

​

Safe Change Types

• Add nullable field
• Add optional enum value
• Add derived field while keeping original field

​

Breaking Change Types

• Removing fields used by dashboards
• Renaming fields without compatibility alias
• Changing semantic type (example: numeric to string)
• Changing partition/path conventions without reader fallback

​

Rollout Order

1. Readers first
   • Update dashboards and SQL queries to support the old and new schema.
   • Update compaction and manifest readers to accept both shapes.
2. Writers second
   • Update Durable Object, Tinybird, and R2 payload writers.
   • Keep old fields during the transition when possible.
3. Validation window
   • Check ingestion, query errors, and compaction output.
4. Cleanup
   • Remove old fields only after every consumer is migrated.

​

Data safety during flush

• Deletion rule: The flush can complete one destination at a time, but it deletes rows only after Tinybird and R2 both succeed for the range.
• Idempotency: Usage ID and cursor checks prevent duplicate counts during a retry.
• Metadata: Metadata is deduplicated and linked by meta_id. Usage and verification records can still join after compaction.
• Access control: Realtime API endpoints require a key. Dashboard WebSocket connections require a session token.

​

Dashboard and SQL impact

SELECT *
FROM usage
WHERE schema_version IN (1, 2)

​

Impact on Compaction

• Compaction is file-level and idempotent.
• Compaction does not change the schema. It merges raw NDJSON lines.
• If a schema changes during the day, compacted output can contain mixed schema_version values.

​

Operational Checklist

• Add/update schema_version where payload shape changed.
• Verify Tinybird ingestion schemas accept the new fields.
• Verify R2 manifests and readers resolve old and new file layouts.
• Verify SQL queries run for both versions.
• Run manual compaction for target day and validate output.
• Monitor realtime endpoint and websocket dashboards during rollout.