Working with JSON on the command line with jq

Goal

Extract, filter or reshape JSON from an API response, a log line or a configuration file inside a pipeline, without writing a throwaway program.

Prerequisites

jq installed; input that is valid JSON or a stream of JSON texts; some idea of the input's shape (jq 'keys', jq '.[0]' and jq 'paths' help to explore it).

Steps

1. Confirm the input parses and see its shape: jq . pretty-prints; on large input, jq -c '.[0]' or jq 'keys' is cheaper to read.
2. Navigate: .foo.bar and .[0] index, .[] iterates an array or an object's values, .foo? suppresses the error when the input is not an object. Chain filters with |.
3. Filter and reshape: .items[] | select(.status == "open") | {id, name}; map(f) applies a filter to every element; to_entries, from_entries and with_entries(f) convert between objects and key-value lists; a // b yields b when a produces only null or false.
4. Pass shell values in with --arg name value (always a string) or --argjson name json (typed) and use $name in the filter. The manual notes that --arg foo 123 binds the string "123", not a number.
5. Choose the output form for the next program: -r (--raw-output) writes strings without quotes; -c prints compact one-line JSON; @tsv, @csv and @sh format arrays for tab-separated, CSV or POSIX-shell consumption; --raw-output0 separates outputs with NUL when values may contain newlines.
6. Handle many inputs: by default the filter runs once per JSON text in the stream; -s (--slurp) collects them into one array; -n (--null-input) runs the filter once with null as input, which is how to build JSON from scratch: jq -n --arg a "$x" '{a: $a}'.
7. Make missing data fail: -e (--exit-status) sets the exit status to 1 if the last output is null or false and to 4 if no valid result was produced; test it in scripts instead of checking for an empty string.

Expected result

A one-line pipeline that produces exactly the field or structure needed, quoted correctly for the next program, and a non-zero exit status when the data is absent.

Limits and test basis

Semantics follow the cited jq manual. jq loads each JSON text fully unless --stream is used, so very large documents need streaming or another tool. The manual states that jq's handling of numbers has changed over time and depends on build options, and that parsing can lose precision on very large or very precise numbers; check the manual for the installed version before relying on jq for arithmetic on such values. No timings are claimed.