jq: a pipeline for JSON

Run the query below and bob does not come back as null. He stops existing. jq is a filter language: each filter takes JSON in and gives JSON out, so you build queries by piping small steps, and a filter does not return one value. It emits a stream of zero or more.

Transcript

A JSON document of three users flows through three jq filters one stage at a time: the array splits into users, select drops the one with no languages, and the survivors are reshaped into small objects, in the order the pipe runs them.

{"users":[
  {"name":"ada","langs":["go","c"]},
  {"name":"bob","langs":[]},
  {"name":"cy","langs":["rust"]}]}

jq -c '.users[]
  | select(.langs | length > 0)
  | {name, n: (.langs | length)}' users.json

{"name":"ada","n":2}
{"name":"cy","n":1}

The mechanism#

.users[] takes one document and emits three objects. | runs the right side once for every value on its left, so each later step sees one user at a time. select(f) emits its input when f is true and nothing otherwise. {name} is shorthand for {name: .name}.

Streams, not arrays#

To get an array back, collect the stream: [.users[] | .name] gives ["ada","bob","cy"]. , concatenates streams and binds tighter than |, so 1, 2 | . * 10 prints 10 and 20: both values go through the multiply.

The boundary#

• Missing keys are quiet. .users[0].nmae prints null and exits 0, so a typo in a key looks like missing data.
• Wrong types are loud. In jq 1.8, .a on a string exits 5 with Cannot index string with string ("a"). .a? swallows that error and emits nothing.
• jq parses each top level value completely before filtering it. For a file too big for memory, --stream emits [path, leaf] pairs instead.
• -r prints strings without quotes, which is what the next shell command in the pipe wants.

Rule of thumb#

Build a query one | at a time and run it after each step. Every prefix of a jq pipeline is itself a valid program, so delete everything after any top-level | (not one inside select(...)) and you see exactly what that stage receives. Few languages let you debug with the backspace key.