sort, uniq and comm: set operations on text that depend on sorted input

What it is

The coreutils manual groups these commands under "operating on sorted files". sort orders lines; by default it compares whole lines using the collating sequence of the LC_COLLATE locale, with a last-resort whole-line comparison that -s (stable) and -u disable. -n sorts numerically, -h by human-readable sizes such as 2K and 1G, -V by version strings, -r reverses, -k pos1[,pos2] selects a key and -t sep sets the field separator; -u keeps the first of lines that compare equal, -z works on NUL-terminated records and -c checks order without sorting.

uniq discards all but the first of adjacent repeated lines; the manual states that repeated lines are detected only if they are adjacent, so unsorted input needs sort first (or sort -u). -c prefixes each line with its count, -d prints only repeated lines, -u only lines that occur once.

comm file1 file2 prints three tab-separated columns: lines only in file1, lines only in file2 and lines in both; -1, -2 and -3 suppress columns. The manual requires both inputs to be sorted by the LC_COLLATE collating sequence, notes that plain sort output is suitable input, and offers --check-order to fail on unsorted input instead of producing wrong output.

Why it matters

These commands implement counting, deduplication, intersection and difference for anything line-shaped: log entries, package lists, identifiers exported from two systems. Their mistakes are silent unless the sorted-input requirement is respected.

How to apply

• Frequency table: sort | uniq -c | sort -rn | head.
• Lines in A but not in B: comm -23 <(sort a) <(sort b); intersection: comm -12; only in B: comm -13.
• Sort by one column: sort -t, -k3,3n file.csv; 3,3 stops the key from extending to the end of the line. Sizes from du -h: sort -h.
• Deduplicate while keeping input order: awk '!seen[$0]++' instead of sort.
• Put export LC_ALL=C at the top of scripts that combine these tools so that every stage agrees on the order on every machine; the sort manual's footnote recommends LC_ALL=C when a locale produces unexpected order.
• Add --check-order to comm in pipelines so that a collation mismatch fails loudly.

Pitfalls

sort -n -u deduplicates on the numeric key alone, which differs from sort -n | uniq on whole lines; the manual gives this example. uniq -c puts the count before the line, separated by blanks, so parse it with awk rather than by fixed columns. Fields for -k count from 1; without -t, the manual says, a field starts at each non-blank-to-blank transition, so leading blanks belong to the field, and LC_CTYPE defines blanks. Process substitution <(...) is bash and zsh syntax, not POSIX sh.