Data File Formats

Data file formats define how information is structured, encoded, and stored on disc or transmitted over the wire. Choosing the right format is one of those decisions that seems small but compounds fast — it affects storage costs, query performance, schema evolution, and how easily teams across your organisation can share data. Get it wrong and you pay the tax on every read, every pipeline run, and every late-night debugging session.

The Two Families: Row vs Columnar

Before diving into individual formats, it helps to understand the fundamental architectural split.

Row-oriented formats (CSV, JSON, Avro) store all fields of a record together. They are optimised for writing complete records and reading entire rows — typical in transactional systems, APIs, and streaming pipelines.

Columnar formats (Parquet, ORC) store all values of a single column together. They are optimised for analytical queries that touch a handful of columns across millions of rows — typical in data warehouses and BI workloads.

This distinction matters because it determines which queries are fast, how well data compresses, and how much I/O your compute infrastructure actually needs to perform.

Format-by-Format Breakdown

CSV (Comma-Separated Values)

The universal lowest-common-denominator. Every tool on the planet can read a CSV. That's both its strength and its curse.

• Schema: None. Column types are inferred (often incorrectly) by the reader. No metadata, no nullability semantics.
• Compression: Poor — repetitive string values don't compress well without external gzip/bzip2.
• Use cases: Data exchange with non-technical stakeholders, quick exports, legacy integrations, seed data for tests.
• Pain points: No schema enforcement means silent type coercion bugs. Quoting and delimiter issues are surprisingly common. No support for nested data.

Verdict: Fine for small, throwaway, or human-readable data. Never use it as a production data lake format.

JSON / NDJSON (Newline-Delimited JSON)

The lingua franca of APIs and configuration. Human-readable, self-describing, and universally supported.

• Schema: Self-describing but not enforced. Any record can have any shape. JSON Schema exists but is opt-in.
• Compression: Moderate — text-heavy with lots of repeated keys. Compresses reasonably well with gzip.
• Use cases: API payloads, configuration files, event logs, semi-structured data ingestion (Bronze layer).
• Pain points: Verbose (keys repeated for every record). No native date, decimal, or binary types. Parsing is CPU-intensive at scale.

Verdict: Ideal for interchange and the early stages of a data pipeline. Avoid for large-scale analytical storage.

Apache Avro

A row-oriented binary format designed for schema evolution in data-intensive systems. The schema is embedded in the file header, so readers and writers can evolve independently.

• Schema: Mandatory and embedded. Supports schema evolution (adding/removing fields, renaming via aliases) with forward and backward compatibility rules.
• Compression: Good — binary encoding is compact, and Avro natively supports Snappy, Deflate, and Zstandard codecs at the block level.
• Use cases: Kafka message serialisation (with a Schema Registry), data pipeline interchange, write-heavy workloads, long-term event storage.
• Pain points: Not optimised for analytical queries (full row must be deserialised). Less human-readable than JSON for debugging.

Verdict: The go-to format for streaming pipelines and event buses. Pair it with a Schema Registry to enforce contracts between producers and consumers.

Apache Parquet

The dominant columnar format in the modern data stack. Originally built by Twitter and Cloudera for Hadoop, now the de facto standard for analytical workloads.

• Schema: Rich and mandatory. Supports nested data (via Dremel encoding), logical types (timestamps, decimals, UUIDs), and column-level metadata.
• Compression: Excellent — columnar layout means similar values are adjacent, enabling very high compression ratios with Snappy, Zstandard, or LZ4. Predicate pushdown skips entire row groups.
• Use cases: Data lakes, data warehouses, analytical queries, ML feature stores, any read-heavy/scan-heavy workload.
• Pain points: Not ideal for single-row lookups or frequent appends. Writing is more expensive than reading. File-level schema makes partial schema evolution more cumbersome than Avro.

Verdict: If you're building an analytical platform, Parquet is the default choice. Pair it with a table format like Apache Iceberg or Delta Lake for ACID transactions and time travel.

Apache ORC (Optimised Row Columnar)

A columnar format originally developed for the Hive ecosystem. Functionally similar to Parquet with some differences in internal architecture.

• Schema: Rich, embedded, with support for complex types (structs, lists, maps).
• Compression: Excellent — comparable to Parquet. Built-in lightweight indices (min/max per stripe, bloom filters) enable efficient predicate pushdown.
• Use cases: Hive and Presto/Trino-centric analytics, workloads with many small-range filter queries.
• Pain points: Ecosystem lock-in — strongest support is in the Hadoop/Hive/Presto world. Less broadly adopted than Parquet outside that niche.

Verdict: A strong format, but unless you're deeply invested in the Hive ecosystem, Parquet's broader tooling support usually wins.

Apache Arrow / Feather

Arrow is not a file format in the traditional sense — it's a cross-language in-memory columnar standard. Feather (Arrow IPC) is its on-disc serialisation. The key innovation: zero-copy reads. Data can be mapped directly from disc into memory without deserialisation.

• Schema: Rich columnar schema, shared across languages (Python, R, Rust, C++, Java, and more).
• Compression: Variable — the in-memory format is uncompressed for speed; the IPC/Feather format supports LZ4 and Zstandard.
• Use cases: Inter-process data exchange, in-memory analytics (Pandas, Polars, DuckDB), fast local caching, columnar flight protocol (Arrow Flight).
• Pain points: Not designed for long-term storage or data lakes. Files can be large without compression. Less mature ecosystem for schema evolution.

Verdict: The backbone of modern data tooling. You'll use Arrow indirectly even if you never create an Arrow file — it's what powers Pandas 2.x, Polars, DuckDB, and Spark's vectorised execution.

Protocol Buffers (Protobuf)

Google's language-neutral binary serialisation format, designed for RPC and service-to-service communication rather than data analytics.

• Schema: Mandatory .proto files with strong typing and code generation. Excellent forward/backward compatibility via field numbering.
• Compression: Very compact binary encoding. Smaller on the wire than Avro for many message shapes.
• Use cases: gRPC service communication, internal microservice payloads, mobile data transfer, configuration storage.
• Pain points: Not self-describing — you need the .proto file to decode the data. No native support for columnar reads. Not designed for data lake or batch analytics.

Verdict: The right tool for service communication, especially with gRPC. Not a data engineering format — don't store Protobuf in your data lake.

Comparison at a Glance

Strategic Utility (Why CTOs Should Care)

File format choices ripple through your entire data platform. Here's where they hit the bottom line:

Storage Costs

Columnar formats with good compression (Parquet, ORC) routinely achieve 5–10× compression over raw JSON or CSV. At petabyte scale, that's the difference between a manageable cloud bill and an eye-watering one.

Query Performance

Columnar formats enable predicate pushdown and column pruning — the query engine reads only the data it needs. A dashboard query touching 3 columns out of 200 in a Parquet table will read ~1.5% of the data compared to scanning the full CSV.

Schema as a Contract

Formats with embedded schemas (Avro, Parquet) turn data quality from a hope into an enforcement mechanism. Producers declare what they emit, consumers know what to expect, and breaking changes are caught at write-time rather than discovered at 3am in a broken dashboard.

Pipeline Simplicity

Matching the format to the workload avoids unnecessary serialisation/deserialisation. A common pattern:

1. Ingest events as Avro via Kafka (schema-evolved, compact, fast writes).
2. Land raw data in the Bronze layer as JSON or Avro files.
3. Transform and store in Silver/Gold as Parquet (optimised for analytical reads).
4. Serve real-time aggregations via Arrow Flight or materialised views.

Interoperability

Choosing widely adopted formats (Parquet, Avro, JSON) means your data can be consumed by any engine — Spark, Trino, DuckDB, BigQuery, Snowflake — without vendor lock-in. Proprietary or niche formats create hidden switching costs.

The right format decision is context-dependent. There's no single winner. But the most common mistake is defaulting to CSV or JSON for everything because it's "easy" — and paying the compounding cost in storage, performance, and data quality for years.

Explore Next

• Medallion Architecture — How progressive data layers (Bronze, Silver, Gold) leverage different file formats at each stage.
• Garbage In, Garbage Out — Why data quality matters — and how the right serialisation format helps enforce it.
• Event-Driven Architecture — Asynchronous messaging systems where serialisation format choice (Avro, Protobuf, JSON) directly impacts throughput.
• SQL vs NoSQL — Database paradigms that pair with different storage formats for persistence and analytics.

References

• Apache Parquet Documentation — Official documentation for the Parquet columnar storage format.
• Apache Avro Specification — The full specification for Avro's schema evolution, encoding, and RPC protocol.
• Apache ORC Project — Home page for the Optimised Row Columnar format, including benchmarks and documentation.
• Apache Arrow — The cross-language in-memory columnar format that underpins modern data tooling.
• Protocol Buffers Language Guide — Google's official guide to defining and using Protocol Buffer schemas.
• Comparison of data-serialization formats (Wikipedia)Wikipedia — Broad overview of serialisation formats, their history, and trade-offs.