sed and awk for line-oriented edits: substitution, fields and in-place changes

What it is

sed reads input line by line into a pattern space, applies a script of commands and prints the result. The most used command is s/regexp/replacement/flags. The GNU sed manual states that the replacement may contain \1 to \9 for parenthesised groups and & for the whole match, that / can be replaced by any other single character as delimiter, and that -E selects extended regular expressions. -n suppresses automatic printing (pair it with p), and -i[SUFFIX] (--in-place) edits files by writing a temporary file and renaming it over the original, keeping a backup only if a suffix is given.

awk is a small language: a program is a list of pattern { action } pairs run against each record. Records are split into fields $1, $2, … by the field separator FS, set with -F or in a BEGIN block; the gawk guide notes that FS is a single character or a regular expression and is unrelated to the shell's IFS. NR counts records, NF holds the number of fields, END blocks run after the last record, and associative arrays make counting and grouping short.

Why it matters

For simple transformations these tools are quicker to write and review than a general-purpose program, and they compose in pipelines. Their failure modes are quiet: a regexp that does not match changes nothing, and one that matches too much changes everything.

How to apply

• Substitute with a delimiter that does not occur in the text: sed 's|/usr/local|/opt|g'; use -E for +, ?, | and groups without backslashes.
• Preview before editing in place: run without -i, or diff <(sed 's/a/b/' file) file; then use sed -i.bak to keep a backup.
• Print or reorder columns: awk -F: '{print $1, $7}' /etc/passwd; awk '{print $NF}' prints the last field.
• Count by key: awk '{count[$1]++} END {for (k in count) print k, count[k]}'.
• Filter by field value: awk '$3 > 100 && $1 != "total"'.
• Delete or select ranges: sed '/^#/d' removes comment lines; sed -n '10,20p' prints lines 10 to 20.

Pitfalls

sed -ni 's/foo/bar/' FILE truncates the file: the manual warns that -n with -i and no p command leaves it empty. Because -i takes an optional argument, sed -iE creates a backup named FILEE rather than enabling extended regexps; write -E -i. BSD sed (FreeBSD, macOS) documents -i extension with a mandatory argument, so sed -i 's/x/y/' FILE takes the script as the suffix there; sed -i.bak 's/x/y/' FILE works with both implementations, while -i '' is BSD-only. GNU extensions such as \U and \L in replacements do not port. Character ranges depend on the locale (see the locale article).