How to use jq with curl to explore an API’s JSON response
Published
Table of content
- How to install jq
- What is jq and what can we do with it?
- Processing JSON data with jq
- Extract the length of a JSON array with jq
- Getting an item at a given index from a JSON array with jq
- Getting n elements of an array with jq
- Selecting specific properties
- Using the map function to pick specific keys from array items
- Filtering data using the select function in jq
- Sorting an array of objects with jq
- Inspecting the keys of a payload
- Cheat sheet and recap
- Conclusion
- Resources
jq is a useful command-line tool to process a JSON payload quickly. I tend to use jq
in conjunction with curl
to explore third-party APIs. jq
is a powerful tool for filtering or transforming JSON data. In this article, we’ll explore a couple of its functionalities while playing with two free third-party APIs. The goal is to become more comfortable with jq
by the end of the blog post and to have started building a cheat sheet of our own.
How to install jq
To install jq
, you can download the binary on the jq
website. If you are on a Mac, you can install jq
with homebrew
:
brew install jq
For a complete set of installation instructions, check out the Github repository section on that topic.
What is jq and what can we do with it?
jq
is described in its official documentation as a “lightweight and flexible command-line JSON processor”. jq
is lightweight and quick (it is written in C). Its flexibility comes from the number of features and possibilities available when using it. You could use jq to:
- Pipe JSON data (
curl <URL> | jq .
) and get a formatted and colored output. - Filter out keys and quickly extract the data you are interested in from a JSON output.
- Sort a JSON output based on a specific key.
- Limit the JSON output to
n
elements (for instance, you might want the first five elements from a JSON array). - Retrieve all the keys of a JSON response (this might be handy when interacting with a third-party API).
- And much more!
In all the commands we’ll explore, we’ll pipe JSON data to jq
to process. Piping means taking the output of a command (such as curl returning API data) and directly passing it to another program for processing. In the shell, to pipe an output, we use the |
symbol. For instance:
curl <URL> | jq .
The previous command uses curl
to fetch the data available at a given URL. Then, it passes the data to jq
.
You don’t need to pipe data to jq
to use it. You can pass a JSON file to jq as an argument:
jq . <filepath_to_json_file>
Processing JSON data with jq
Let’s get started working with JSON arrays. For this section, we will use the JSON placeholder that provides dummy To-Do API endpoints.
curl -s https://jsonplaceholder.typicode.com/todos
With this command, we obtain a JSON array of 200 elements.
Extract the length of a JSON array with jq
If you are wondering how many items are in a JSON array, you can use the length
helper in jq
:
curl -s https://jsonplaceholder.typicode.com/todos | jq length
In your terminal, you should see a single number as an output (200
in our case).
Getting an item at a given index from a JSON array with jq
Getting the first item of a JSON array is as simple as typing jq '.[0]'
:
curl -s https://jsonplaceholder.typicode.com/todos | jq '.[0]'
To get the last item of an array, you can use jq '.[-1]'
:
curl -s https://jsonplaceholder.typicode.com/todos | jq '.[-1]'
Getting n
elements of an array with jq
To get the first five elements, you can use a similar notation to Go’s slices:
curl -s https://jsonplaceholder.typicode.com/todos | jq '.[:5]'
When you write down [:5]
, it means all the indices (starting at 0) excluding the index of 5. You can provide the starting index if you have a range in mind:
curl -s https://jsonplaceholder.typicode.com/todos | jq '.[2:5]'
The above command jq ‘[2:5]’
includes three elements (the elements at index 2, 3, and 4).
Selecting specific properties
Let’s switch things up! Many APIs include a large amount of data that we may not be interested in. Let’s use the swapi API to explore a more complex dataset.
If we retrieve the Star Wars characters using the API's people endpoint, we’ll see that the output puts all the characters in a results
key. The top-level keys in the output JSON object include pagination data.
To solely include the people, we can write:
curl -s https://swapi.py4e.com/api/people/ | jq ".results"
The jq “.results”
will process the JSON data and output the result array.
Then, we can use the previous commands we have learned. We could get the first character from the people API using the following command:
curl -s https://swapi.py4e.com/api/people/ | jq ".results | .[0]"
We can go even further than that. We can extract the keys we are interested in from the results array.
Using the map function to pick specific keys from array items
map
in jq
will do something similar to the map
array function in JavaScript. It applies a given transformation to each item in the array.
In jq
, we can build new JSON objects with the properties we want using a shorthand like {name, url}
. This syntax is equivalent to writing {“name”: .name, “url”: .url}
. In our case, we can pass this shorthand to map
:
curl -s https://swapi.py4e.com/api/people/ | jq ".results | map({name, url})"
It extracts the results
properties and maps each item in the results array to a smaller object containing only the name
and url
fields. In your terminal, you should see an output that looks like the following snippet:
[
{
"name": "Luke Skywalker",
"url": "https://swapi.py4e.com/api/people/1/"
},
{
"name": "C-3PO",
"url": "https://swapi.py4e.com/api/people/2/"
}
// more data...
]
Beyond picking the properties we are interested in, we can filter the data using jq
. Let’s look at how next!
Filtering data using the select
function in jq
Let’s imagine we are looking for one character in particular in our Star Wars API. We can use the select
function to filter by a specific property, such as name
in our case.
We can do the following:
- Use
curl
to retrieve data from a specified URL. - Combine the
map
andselect
functions to look for our character.
curl -s https://swapi.py4e.com/api/people/ | jq '.results | map(select(.name == "Obi-Wan Kenobi"))'
In the above command, we will select items with a name that matches the string “Obi-Wan Kenobi”.
If we want to be less restrictive, we can use a regex:
curl -s https://swapi.py4e.com/api/people/ | jq '.results | map(select(.name | test("Leia"; "i")))'
We pipe the .name
property of each item to the test
function. The second argument of test
is to make the matching pattern case-insensitive.
There are other functions we can use with select, such as startswith
or endswith
. For instance, let’s say we want all the movies whose title starts with 'The'. We could write the following command:
curl -s https://swapi.py4e.com/api/films/ | jq '.results | map(select(.title | startswith("The")))'
- We first pick the
.results
property from the JSON object returned by the API. - Then, we use
map
to indicate this is an operation on each list item. For each list item, we want to select the films with titles that start with “The”.
Sorting an array of objects with jq
Another useful feature of jq
is the ability to sort data using the sort_by
function.
Let’s go back to our Star Wars API. Let’s imagine we want to sort the characters by their height.
curl -s https://swapi.py4e.com/api/people/ | jq ".results | sort_by(.height | tonumber) | reverse | map({name, height})"
Let’s dissect that command:
- We curl the
people
endpoint from the Star Wars API. - We pick the
results
property as it contains the array of characters on the first page. - We sort the array by each element’s height. Because the API returns each character height as a string, we must convert each height to a number to obtain a correct sorting result.
- We pass to
sort_by
the property we want to use to sort our array. We can convert the property if needed. In the example, we usetonumber
to cast each string into a number. - Then, we can pipe the results to
reverse
if we want to reverse the order of the results. In our case, it will return an array from the character with the largest height to the character with the smallest height. - Finally, we use
map
to extract the name and height of each character.
The output looks like:
[
{
"name": "Darth Vader",
"height": "202",
},
{
"name": "Biggs Darklighter",
"height": "183",
},
{
"name": "Obi-Wan Kenobi",
"height": "182",
},
// some more sorted data...
];
Inspecting the keys of a payload
To get a quick idea of what keys a given JSON payload contains, you can use the keys
function.
curl -s https://swapi.py4e.com/api/people/ | jq "keys"
In a glimpse, we get back an array of all the keys from the API response:
["count", "next", "previous", "results"]
If we wanted to get an idea of what type of data the API has for a character, we could do:
curl -s https://swapi.py4e.com/api/people/ | jq ".results[0] | keys"
In this command, we look at the first element of the result array and get back its keys:
// output from the previous curl command
[
"birth_year",
"created",
"edited",
"eye_color",
"films",
"gender",
"hair_color",
"height",
"homeworld",
"mass",
"name",
"skin_color",
"species",
"starships",
"url",
"vehicles"
]
Cheat sheet and recap
Here's a small list of the different commands we tried in this post:
curl <URL> | jq .
: to pretty-print JSON from an API.jq . <file.json>
: to process JSON data from a local file.curl -s <URL> | jq length
: to count the number of items in an array (it will only work with JSON arrays).curl -s <URL> | jq '.[0]'
: to get the first element of an array.curl -s <URL> | jq '.[-1]'
: to get the last element of an array.curl -s <URL> | jq '[:5]'
: to get the first N elements of an array (in this case 5).curl -s <URL> | jq '.[2:5]'
: to get a slice of an array (indices 2 to 4 in this example).curl -s <URL> | jq '.results'
: to extract a specific key from a JSON object.curl -s <URL> | jq '.results[0]'
: to get the first object in an array of objects.curl -s <URL> | jq '.results | map({name, url})'
: to map array and extract specific keys from each array item.curl -s <URL> | jq '.results | map(select(.name == "Obi-Wan Kenobi"))'
: to filter array items using an exact match.curl -s <URL> | jq '.results | map(select(.name | test("Leia"; "i")))'
: to filter array items using a regular expression.curl -s <URL> | jq '.results | map(select(.title | startswith("The")))'
: to filter array items where their title starts with a given string.curl -s <URL> | jq '.results | sort_by(.height | tonumber) | reverse | map({name, height})'
: to sort by height (casting the height into a number), using a descending order.curl -s <URL> | jq 'keys'
: to list the keys at the root level.curl -s <URL> | jq '.results[0] | keys'
: to list the keys of the first item in theresults
array.
Conclusion
In this article, we explored the jq
command-line utility. It is a powerful tool for processing and transforming JSON data. We’ve only brushed the surface of what jq
can do. I hope this post provides some ideas on how to use it. As always, if you have any questions or want to chat about jq
or other command-line tools, reach out on Bluesky!