Columnar storage basics: how a Parquet file is laid out and why analytical reads touch less data

What it is

The Parquet glossary (cited) defines the hierarchy: a file consists of one or more row groups; a row group is a horizontal partition of the rows and consists of a column chunk for each column; column chunks are contiguous in the file and are divided into pages; a page is the indivisible unit of encoding and compression. The file-format page (cited) states that file metadata is written after the data to allow single-pass writing, and that readers are expected to read the file metadata first to find the column chunks they are interested in. Because all values of one column within a row group are stored together, encodings can exploit their similarity: the encodings page (cited) describes dictionary encoding, which builds a dictionary of the values encountered in a column chunk, stores it in a dictionary page, and writes the values as integer indices using a run-length/bit-packing hybrid, falling back to plain encoding if the dictionary grows too big. The page index (cited) adds per-page min and max values so that range scans and point lookups can read only the pages that may contain matching rows.

Why it matters

An analytical query typically reads a few columns of many rows. A row-oriented file (CSV, JSON lines) forces the reader to parse every byte of every row; a columnar file lets it read only the chunks for the projected columns, skip whole row groups whose statistics exclude the predicate, and decode compact integer indices rather than repeated strings. The same layout is a poor fit for reading one whole record or for updating rows in place: a Parquet file is written once and replaced, not edited.

How to apply

• Sort or cluster rows by the columns most often filtered on before writing; min/max pruning only helps when values within a page are close together.
• Partition datasets into directories by a coarse key (date, region) and keep files large enough that the footer and dictionaries are amortised; thousands of tiny files cost more in metadata reads than they save.
• Use the narrowest correct types and logical types (dates, decimals, timestamps with declared unit) so that statistics and encodings work and readers agree on meaning.
• Check with the query engine's explain output that column projection and predicate pushdown actually reach the reader; a filter applied after a full scan gains nothing.

Pitfalls

High-cardinality string columns defeat dictionary encoding and fall back to plain, which inflates files. Nested and repeated fields use definition and repetition levels that some tools handle inconsistently. Compression codec choice is per column chunk; a codec the reader lacks makes the file unreadable there.