I am using jq on a windows machine and I have the following data:
{"a":"Person","birthDay":"12","deathDay":"15", ...}
{"a":"Person","birthDay":"13","deathDay":"16", ...}
...
And I would like to have a tab separated file format:
Person 12 15
Person 13 16
I can get the values with ".a" or ".birhtDay". But how can I get all values and separate them through a tab?
Kind regards,
Snafu
TSV is fortunately a simpler format compared to CSV. Assuming the values will not have newlines or tabs, you just need to build up each of the rows of data you want in the file and print them out separating values by tabs. Just make sure you use the raw output -r option.
[ .a, .birthDay, .deathDay ] | join("\t")
Recent versions of jq (e.g. 1.5rc1 -- available via https://github.com/stedolan/jq/releases) include a filter named #tsv.
On April 15, jq was enhanced so that #tsv will produce valid tsv (one line per valid array), e.g. if an input string contains tabs, newlines, or NULs.
Related
Consider the following JSON file example.json:
{
"key1": ["arr value 1", "arr value 2", "arr value 3"],
"key2": {
"key2_1": ["a1", "a2"],
"key2_2": {
"key2_2_1": 1.43123123,
"key2_2_2": 456.3123,
"key2_2_3": "string1"
}
}
}
The following jq command extracts a value from the above file:
jq ".key2.key2_2.key2_2_1" example.json
Output:
1.43123123
Is there an option in jq that, instead of printing the value itself, prints the location (line and column, start and end position) of the value within a (valid) JSON file, given an Object Identifier-Index (.key2.key2_2.key2_2_1 in the example)?
The output could be something like:
some_utility ".key2.key2_2.key2_2_1" example.json
Output:
(6,25) (6,35)
Given JSON data and a query, there is no
option in jq that, instead of printing the value itself, prints the location
of possible matches.
This is because JSON parsers providing an interface to developers usually focus on processing the logical structure of a JSON input, not the textual stream conveying it. You would have to instruct it to explicitly treat its input as raw text, while properly parsing it at the same time in order to extract the queried value. In the case of jq, the former can be achieved using the --raw-input (or -R) option, the latter then by parsing the read-in JSON-encoded string using fromjson.
The -R option alone would read the input linewise into an array of strings, which would have to be concatenated (e.g. using add) in order to provide the whole input at once to fromjson. The other way round, you could also provide the --slurp (or -s) option which (in combination with -R) already concatenates the input to a single string which then, after having parsed it with fromjson, would have to be split again into lines (e.g. using /"\n") in order to provide row numbers. I found the latter to be more convenient.
That said, this could give you a starting point (the --raw-output (or -r) option outputs raw text instead of JSON):
jq -Rrs '
"\(fromjson.key2.key2_2.key2_2_1)" as $query # save the query value as string
| ($query | length) as $length # save its length by counting its characters
| ./"\n" | to_entries[] # split into lines and provide 0-based line numbers
| {row: .key, col: .value | indices($query)[]} # find occurrences of the query
| "(\(.row),\(.col)) (\(.row),\(.col + $length))" # format the output
'
(5,24) (5,34)
Demo
Now, this works for the sample query, how about the general case? Your example queried a number (1.43123123) which is an easy target as it has the same textual representation when encoded as JSON. Therefore, a simple string search and length count did a fairly good job (not a perfect one because it would still find any occurrence of that character stream, not just "values"). Thus, for more precision, but especially with more complex JSON datatypes being queried, you would need to develop a more sophisticated searching approach, probably involving more JSON conversions, whitespace stripping and other normalizing shenanigans. So, unless your goal is to rebuild a full JSON parser within another one, you should narrow it down to the kind of queries you expect, and compose an appropriately tailored searching approach. This solution provides you with concepts to simultaneously process the input textually and structurally, and with a simple search and ouput integration.
I have 2 huge sets of numbers in columns 1 and 2 in an excel sheet. I want to pair my first column with my second column to create a JSON file like this link here - https://github.com/python-visualization/folium/blob/master/examples/data/data3.json,
something like
[{"0500000US33009": 51289.0, "0500000US38041": 46793.0, "0500000US38043": 39857.0}]
if I had just the numbers 0500000US33009, 0500000US38041, and 0500000US38043 in column 1 and 51289.0, 46793.0, and 39857.0 in column 2. How might I do this, and make sure that the resulting JSON has quotes around the "0500000US33009"?
Thanks in advance!
if I had just the numbers ...
The following assumes that the first two columns have been extracted into a CSV or TSV file, e.g.
0500000US33009,51289.0
0500000US38041,46793.0
0500000US38043,39857.0
With the input data in this format, it would of course be very easy to use your favorite text-processing tool to create a JSON dictionary.
Assuming a simple CSV format as above, the data could also be converted into a JSON dictionary using an invocation along the lines of:
jq -Rn 'reduce inputs as $in ({};
. + ($in|split(",")|{(.[0]): .[1] | tonumber}))'
Using jq 1.6 or earlier, this would produce:
{
"0500000US33009": 51289,
"0500000US38041": 46793,
"0500000US38043": 39857
}
The change in format of the numeric values is the result of a conversion to IEEE 754 64-bit numbers, and can be avoided by using a more recent version of jq. Using the current "master" version, the result would be:
{
"0500000US33009": 51289.0,
"0500000US38041": 46793.0,
"0500000US38043": 39857.0
}
So if you're stuck with jq 1.6 or earlier and require the explicit decimal point, you might want to consider omitting |tonumber in the above program, and add a post-processing step if and as required.
Some words of caution
The jq solution above assumes there are no collisions (one key having more than one value), or rather, that if there are any collisions, then the last key-value pair should prevail.
If any of the values in the second column cannot be represented with sufficient accuracy as IEEE 754 64-bit numbers, then a significantly different strategy may be warranted.
If the number of rows is indeed very large (e.g. in the billions), then the wisdom of constructing a single ginormous JSON dictionary might be worth reconsidering.
I have a csv like below. some of columns have line break like column B below. when I doing wc -l file.csv unix is returning 4 but actually these are 3 records. I don't want to replace line break with space, I am going to load data in database using sql loader and want to load data as it is. what should I do so that unix consider line break as one record?
A,B,C,D
1,"hello
world",sds,sds
2,sdsd,sdds,sdds
Unless you're dealing with trivial cases (No quoted fields, no embedded commas, no embedded newlines, etc.), CSV data is best processed with tools that understand the format. Languages like perl and python have CSV parsing libraries available, there are packages like csvkit that provide useful utilities, and more.
Using csvstat from csvkit on your example:
$ csvstat -H --count foo.csv
Row count: 3
I have this json:
{"temperature":"21", "humidity":"12.3", "message":"Today ID 342 is running"}
I want to use jq to obtain this json:
{"temp":"21", "hum":"12.3", "id":"342"}
As you can see, what i want to do is extract the ID number 342 and put it in the new json with a different key name. I think i should use a regex but i don't know how to insert it in jq syntax.
I can create another json using the basic command:
cat old.json | jq '{temp:.temperature,hum:.humidity, id:.message}' > new.json
I know i can select substring using square brackets, but i don't want to use them because they don't take into account strings with different lengths and structure. I want to use a regex because i know that the ID number comes lways after the "ID" part.
You're right that a regex is the way to go here. Fortunately, the jq manual has a large section on using them.
jq '
{
temp: .temperature,
hum: .humidity,
id: (.message | capture("ID (?<id>[[:digit:]]+)").id)
}' <old.json >new.json
You can see this running with your sample data at https://jqplay.org/s/k-ZylbOC6W
I have several large CSV's which I would like to export to a particular JSON format but I'm not really sure how to convert it over. It's a list of usernames and urls.
b00nw33,harrypotter788.flv
b00nw33,harrypotter788.mov
b00nw33,levitation271.avi
b01spider,schimbvalutar109.avi
...
I want to export them to JSON grouped by the username like the following
{
"b00nw33": [
"harrypotter788.flv",
"harrypotter788.mov",
"levitation271.avi"
],
"b01spider": [
"schimbvalutar109.avi"
]
}
What is the JQ to do this? Thank you!
The key to a simple solution is the generic function aggregate_by:
# In this formulation, f must either always evaluate to a string or
# always to an integer, it being understood that negative integers
# might be problematic
def aggregate_by(s; f; g):
reduce s as $x (null; .[$x|f] += [$x|g]);
If the CSV can be accurately parsed by simply splitting on commas, then the desired transformation can be accomplished using the following jq filter:
aggregate_by(inputs | split(","); .[0]; .[1])
This assumes jq is invoked with the -R (raw) and -n options.
Output
With the given CSV input, the output would be:
{
"b00nw33": [
"harrypotter788.flv",
"harrypotter788.mov",
"levitation271.avi"
],
"b01spider": [
"schimbvalutar109.avi"
]
}
Handling non-trivial CSV
The above solution assumes that the CSV is as uncomplicated as the sample. If, on the contrary, the CSV cannot be accurately parsed by simply splitting at commas, a more general parser will be needed.
One approach would be to use the very robust and fast csv2json parser at https://github.com/fadado/CSV
Alternatively, you could use one of the many available "csv2tsv" parsers to generate TSV, which jq can handle directly (by splitting on tabs, i.e. split("\t") rather than split(",")).
In any case, once the CSV has been converted to JSON, the filter aggregate_by defined above can be used.
If you are interested in a jq parser for CSV, you might want to look at fromcsvfile (https://gist.github.com/pkoppstein/bbbbdf7489c8c515680beb1c75fa59f2); see also
the definitions for fromcsv being proposed at https://github.com/stedolan/jq/issues/1650#issuecomment-448050902