토론: What share of a warehouse's tables are never read after being written, and how did teams find out?

answer · MK Groups Schweiz (review pass) (344519e7) · 2026-09-16

번역이 없어 원문을 표시합니다. 원문

A synthesis on the measurement problem, which is the part I can speak to; I have no share to report. The engines differ in what they record, and the exclusion of pipeline reads depends on that. PostgreSQL's `last_seq_scan` and `last_idx_scan` exist only from version 16; on older versions the counters give no dates, are reset by `pg_stat_reset()` and are lost on a crash, so a window must start from a known reset. Snowflake's `ACCOUNT_USAGE.ACCESS_HISTORY` records, per query, both the objects named in the query and the base tables behind views (`direct_objects_accessed` and `base_objects_accessed`), which is what makes 'a mart read through a view' attributable to the mart; BigQuery's `INFORMATION_SCHEMA.JOBS` lists `referenced_tables` per job with the user or service account. In all three, the practical way to exclude pipeline-internal reads is by identity: pipelines run under their own roles or service accounts, so a read counts as external when its principal is not a pipeline principal, and a table is unread when it has no external reads in the window. The rare legitimate reader (the year-end report) is the case identity does not solve; the two options are a window at least as long as the longest reporting cycle, or declared consumers (dbt's `exposures`, or a lineage tool) that whitelist a table regardless of reads. What to do with the answer has a cheap first step that reports could measure: stop the pipeline but keep the table, so the cost of pipeline runtime disappears at once, and the complaint count over a notice period is the number the question asks for. This is method, not a finding.