How do I merge thousands of json files with jq? - json

Question

I have thousands of individually named json files in a single windows directory. I'm trying to use jq to merge them all into a single file I can then import into a jupyter notebook.
I keep getting a permissioned denied error when I try and run the following command:
jq --slurp 'map(.[])' bill
I've tried editing the directory (bill) permissions. My file path looks like this:
\downloadedJSONfiles\AK\2019-2020_31st_Legislature\bill
I downloaded jq through chocolately. I'm using cmder

Answer 1

You are getting permission denied because you provided a directory where a path to an ordinary file is expected.
I'm going to start with the "unix" approach because jq has unix roots. The following is the command you want to use:
jq --slurp 'map(.[])' bill/a.json bill/b.json bill/c.json ...
The shell will expand the following command into the above:
jq --slurp 'map(.[])' bill/*.json
The problem is that this can easily result in a command that's too long. So you really want one the following:
# A
(
jq '.[]' bill/a.json
jq '.[]' bill/b.json
jq '.[]' bill/c.json
jq '.[]' bill/d.json
jq '.[]' bill/e.json
jq '.[]' bill/f.json
) | jq --slurp .
# B
(
cat bill/a.json
cat bill/b.json
cat bill/c.json
cat bill/d.json
cat bill/e.json
cat bill/f.json
) | jq --slurp 'map(.[])'
# C
(
jq '.[]' bill/a.json bill/b.json bill/c.json
jq '.[]' bill/d.json bill/e.json bill/f.json
) | jq --slurp .
# D
(
cat bill/a.json bill/b.json bill/c.json
cat bill/d.json bill/e.json bill/f.json
) | jq --slurp 'map(.[])'
Something equivalent can be achieved using any of the following:
# Portable. Equivalent to A
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec jq '.[]' {} \; | jq --slurp .
# Portable. Equivalent to B
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec cat {} \; | jq --slurp 'map(.[])'
# Possibly portable with tweaks. Similar to "C"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -print0 |
xargs -r0 jq '.[]' |
jq --slurp .
# Possibly portable with tweaks. Similar to "D"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -print0 |
xargs -r0 cat |
jq --slurp .
# GNU-specific. Equivalent to "C"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec jq '.[]' {} + | jq --slurp .
# GNU-specific. Equivalent to "D"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec cat {} + | jq --slurp 'map(.[])'
But you asked about Windows. In the Windows world it's up to programs to perform their own wildcard expansion. So you'd expect to be able to do
jq --slurp 'map(.[])' bill\*.json
However, jq wasn't properly ported.
Assertion failed!
Program: c:\bin\jq.exe
File: src/main.c, Line 256
Expression: wargc == argc
So like in unix, you have to pass all the files you want to process to jq as separate arguments.
Using cmd, you could use either of the following:
:: Not efficient
copy /y nul bill.jsonl
for %q in (bill\*.json) do jq ".[]" %q >>bill.jsonl
jq --slurp . bill.jsonl
del bill.jsonl
:: More efficient
copy /y nul+bill\*.json bill.jsonl
jq --slurp "map(.[])" bill.jsonl
del bill.jsonl
PowerShell is a far more advanced shell than cmd. With PowerShell, you could use
jq --slurp 'map(.[])' ( Get-Item bill\*.json )
But just like the simple unix version, the above can easily result in a command line that's too long. To avoid that, we can use the following:
# Not efficient
Get-Item bill\*.json | %{ jq '.[]' $_ } | jq --slurp .
# More efficient
%{ Get-Content bill\*.json } | jq --slurp 'map(.[])'
(%{...} is a shorthand for ForEach-Object {...}.)
Finally, I'm not familiar with Cmder.