Time-Series Data: Characteristics and Storage Needs

Every IoT deployment is fundamentally a machine for producing time-series data: a temperature every 10 seconds, a vibration FFT bin every second, a GPS fix every minute. The volume is predictable and relentless — a single sensor reporting every 10 s generates 8,640 points/day, and a fleet of 10,000 such sensors produces approximately 31.5 billion points/year. Choosing the wrong storage engine for this workload is one of the most common architecture mistakes in IoT systems: relational databases that work fine for 10,000 rows fall over at 10 billion, and the failure mode is usually discovered in production, not in the design review.

What Makes Time-Series Data Different

Time-series data has a specific statistical and access-pattern signature that general-purpose databases are not optimized for:

• Append-mostly writes. New points arrive continuously and are almost always appended at the "now" edge of the timeline. Updates to old records are rare; deletes are usually bulk (retention expiry), not row-by-row.
• Time as the primary query dimension. Nearly every query is "give me values for metric X between t1 and t2," often for a specific device/tag. This is fundamentally different from relational workloads dominated by joins on arbitrary keys.
• High write throughput, moderate read complexity. Millions of inserts per second are common in industrial or fleet telemetry; reads are typically range scans, aggregations, or downsampled views rather than point lookups.
• Value redundancy and slow-changing deltas. Consecutive samples are often very close in value (a temperature drifting by 0.1 °C between readings), which makes time-series data highly compressible — often 10:1 to 100:1 with delta and run-length encoding.
• Metric cardinality vs. tag cardinality. A metric (e.g., temperature) has few distinct names, but tags (device_id, location, firmware_version) can multiply the number of unique series into the millions — this "cardinality explosion" is the main scaling risk in time-series systems.
• Retention decay. Raw resolution matters most for recent data; older data is usually only needed at reduced resolution (rollups), which enables aggressive downsampling and compression as data ages.

Data Model: The Time-Series Point

A canonical time-series record has four logical parts:

timestamp   : 64-bit epoch (ms or ns precision)
metric name : e.g. "engine.temp", "pump.vibration.rms"
tags/labels : key-value metadata (device_id=42, site="plant-3")
value       : numeric (int/float), occasionally string/boolean

The combination of metric name + tag set defines a unique series; the timestamp+value pairs within that series form the actual time-series. This is the model used by InfluxDB, Prometheus, TimescaleDB (hypertables), and OpenTSDB, with minor naming differences.

Storage Engine Approaches

The general pattern across all purpose-built engines: partition by time (so old data can be dropped or compacted as a unit) and store values column-wise within a time window (so compression algorithms can exploit the low variance between consecutive samples).

Compression: Why It Works So Well

Time-series compression exploits two properties:

1. Timestamp regularity — if samples arrive at a fixed interval, storing only the interval (delta-of-delta encoding) collapses a 64-bit timestamp to often just 1–2 bits per point.
2. Value smoothness — consecutive sensor values differ by small amounts, so delta encoding plus XOR-based floating-point compression (as used in Facebook's Gorilla format, adopted by Prometheus and InfluxDB) can represent most points in under 2 bytes.

Worked example: A vibration sensor samples at 1 kHz (1,000 samples/s), each sample a 4-byte float, for one axis, running continuously.

• Raw rate: 1,000 samples/s × 4 bytes = 4,000 bytes/s = 4 kB/s
• Per day: 4 kB/s × 86,400 s = 345.6 MB/day per sensor
• For 500 machines (3 axes each = 1,500 series): 345.6 MB × 1,500 ≈ 518 GB/day

With Gorilla-style compression achieving a typical 10:1 ratio on smoothly varying vibration data, storage drops to ≈52 GB/day — the difference between needing a small NAS and needing a mid-size storage cluster over a year. This single calculation is why compression algorithm choice is a first-order architecture decision, not an implementation detail.

Check: 518 GB/day × 365 ≈ 189 TB/year raw vs. ≈19 TB/year compressed — both numbers are in the range operators actually report for dense vibration monitoring fleets, confirming the estimate is reasonable.

Downsampling and Retention Policies

Because old raw data is rarely queried at full resolution, most systems apply a tiered retention strategy:

Continuous aggregation (materialized rollups computed as data arrives, e.g., TimescaleDB continuous aggregates or Prometheus recording rules) avoids expensive re-scans of raw data for dashboards and keeps long-range queries fast regardless of the underlying raw data volume.

Practical Design Implications

• Choose the partition key carefully. Partitioning by time bucket (e.g., 1-day chunks) plus a hashed tag (device_id) prevents both "hot" write partitions and unbounded per-partition growth.
• Control tag cardinality deliberately. Avoid putting continuously-varying values (raw GPS coordinates, request IDs) into tags — that creates a new series per point and can silently multiply storage and index size by orders of magnitude.
• Batch writes at the edge. Gateways should buffer and batch telemetry (e.g., 100 points per write) rather than issuing one write per sample; this amortizes per-write overhead in both LSM-based TSDBs and network protocols like MQTT.
• Separate hot and cold paths early. Real-time alerting can run off the last few minutes of raw data in memory/cache, while historical analytics reads from downsampled, compressed cold storage — mixing the two access patterns on one engine is a common cause of query latency problems.
• Plan retention policy before ingestion starts, not after storage bills arrive — retroactively downsampling years of raw data is expensive and often impossible if the raw resolution was never needed.

Key Takeaways

• Time-series data is append-mostly, time-ordered, and query-dominant on time ranges — a fundamentally different access pattern from general relational workloads.
• Purpose-built time-series stores exploit timestamp regularity and value smoothness to achieve 10–100× compression versus raw storage.
• Tag/label cardinality, not raw sample count, is the usual cause of time-series systems scaling poorly — design tags to stay low-cardinality.
• Tiered retention (raw → minute rollups → hourly/daily rollups) keeps both storage cost and query latency bounded as data ages.
• Always size storage with an explicit calculation (sample rate × size × series count × compression ratio) before committing to an engine or cloud storage tier.