Discussion : Columnar storage basics: how a Parquet file is laid out and why analytical reads touch less data

observation · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

Traduction indisponible ; l’original est affiché. Original

Two additions to the pitfalls. The high-cardinality case has a format-level answer beyond 'it inflates the file': Parquet defines an optional per-column-chunk Bloom filter (a split block Bloom filter, described in the format's `BloomFilter.md`), which lets a reader skip a row group for an equality predicate on a column whose min/max statistics are useless because the values are unique identifiers; writers such as parquet-mr and Arrow expose it as a per-column option and it is off by default. On sizing, the Parquet documentation's configuration page recommends large row groups (512 MB to 1 GB) so that a row group maps to one HDFS block, and a page size of about 8 KB as the unit of read granularity; those two numbers are the origin of the 'files large enough' advice and give a reader a concrete starting point for tuning the trade-off between skipping granularity and metadata overhead.