I have actually a situation were i run into the BIGINT Problem and the haziness with that in jq 1.5/1.6 (in a Windows enviroment).
I read the issue reports and thought that if i transform the number to a string, i can handle that. But i test it with a specific command and the result is the same
[. | { last_update: .starbase_detailed_scan.last_update_time, user_name: .starbase_detailed_scan.owner_name, alliance_id: .starbase_detailed_scan.owner_alliance_id | tostring, drydocks: .starbase_detailed_scan.num_drydocks, tier: .starbase_detailed_scan.owner_level, defence_plattform: .starbase_detailed_scan.num_defence_platforms, shield_triggered: .starbase_detailed_scan.player_shield.triggered_on, shield_end: .starbase_detailed_scan.player_shield.expiry_time, parsteel: .starbase_detailed_scan.resources["2325683920"], tritanium: .starbase_detailed_scan.resources["743985951"], dilithium: .starbase_detailed_scan.resources["2614028847"], user_id: .starbase_detailed_scan.owner_user_id, defence_rating: .starbase_detailed_scan.defense_rating }]
result:
[{"last_update":"2018-12-23T19:26:24","user_name":"Hamita40","alliance_id":"774615702811599900","drydocks":3,"tier":19,"defence_plattform":3,"shield_triggered":"0001-01-01T00:00:00","shield_end":"0001-01-01T00:00:00","parsteel":183649,"tritanium":22459,"dilithium":7074,"user_id":"a2588903decc455283c88508f6a7fedf","defence_rating":25200}]
The Alliance_id is not correct. The correct id is:
774615702811599864
Is there any Workaround?
BR
Timo
Using tostring is not going to help, because tostring only gets to see the number after the jq parser has read the input.
The jq maintainers are well-aware of this issue and indeed there is a "pull request" which addresses it:
https://github.com/stedolan/jq/pull/1752
If you wish to use an officially released version of jq, then the only available "workaround" will be to change the number in the JSON source to a string. You might wish to use the "bigint" library for handling "bigint" strings - https://github.com/joelpurra/jq-bigint
UPDATES
As of Oct 22, 2019, the version of jq at "master" preserves the precision of numbers that are read, and tostring can be used on such numbers without loss of precision, e.g.
$ jq tostring
123456789123456789123456789123456789123456789123456789
"123456789123456789123456789123456789123456789123456789"
You might alternatively wish to use gojq, the Go implementation of jq.
Related
Consider the following JSON file example.json:
{
"key1": ["arr value 1", "arr value 2", "arr value 3"],
"key2": {
"key2_1": ["a1", "a2"],
"key2_2": {
"key2_2_1": 1.43123123,
"key2_2_2": 456.3123,
"key2_2_3": "string1"
}
}
}
The following jq command extracts a value from the above file:
jq ".key2.key2_2.key2_2_1" example.json
Output:
1.43123123
Is there an option in jq that, instead of printing the value itself, prints the location (line and column, start and end position) of the value within a (valid) JSON file, given an Object Identifier-Index (.key2.key2_2.key2_2_1 in the example)?
The output could be something like:
some_utility ".key2.key2_2.key2_2_1" example.json
Output:
(6,25) (6,35)
Given JSON data and a query, there is no
option in jq that, instead of printing the value itself, prints the location
of possible matches.
This is because JSON parsers providing an interface to developers usually focus on processing the logical structure of a JSON input, not the textual stream conveying it. You would have to instruct it to explicitly treat its input as raw text, while properly parsing it at the same time in order to extract the queried value. In the case of jq, the former can be achieved using the --raw-input (or -R) option, the latter then by parsing the read-in JSON-encoded string using fromjson.
The -R option alone would read the input linewise into an array of strings, which would have to be concatenated (e.g. using add) in order to provide the whole input at once to fromjson. The other way round, you could also provide the --slurp (or -s) option which (in combination with -R) already concatenates the input to a single string which then, after having parsed it with fromjson, would have to be split again into lines (e.g. using /"\n") in order to provide row numbers. I found the latter to be more convenient.
That said, this could give you a starting point (the --raw-output (or -r) option outputs raw text instead of JSON):
jq -Rrs '
"\(fromjson.key2.key2_2.key2_2_1)" as $query # save the query value as string
| ($query | length) as $length # save its length by counting its characters
| ./"\n" | to_entries[] # split into lines and provide 0-based line numbers
| {row: .key, col: .value | indices($query)[]} # find occurrences of the query
| "(\(.row),\(.col)) (\(.row),\(.col + $length))" # format the output
'
(5,24) (5,34)
Demo
Now, this works for the sample query, how about the general case? Your example queried a number (1.43123123) which is an easy target as it has the same textual representation when encoded as JSON. Therefore, a simple string search and length count did a fairly good job (not a perfect one because it would still find any occurrence of that character stream, not just "values"). Thus, for more precision, but especially with more complex JSON datatypes being queried, you would need to develop a more sophisticated searching approach, probably involving more JSON conversions, whitespace stripping and other normalizing shenanigans. So, unless your goal is to rebuild a full JSON parser within another one, you should narrow it down to the kind of queries you expect, and compose an appropriately tailored searching approach. This solution provides you with concepts to simultaneously process the input textually and structurally, and with a simple search and ouput integration.
I have the following JSON structure:
{
"host1": "$PROJECT1",
"host2": "$PROJECT2",
"host3" : "xyz",
"host4" : "$PROJECT4"
}
And the following environment variables in the shell:
PROJECT1="randomtext1"
PROJECT2="randomtext2"
PROJECT4="randomtext3"
I want to check the values for each key, if they have a "$" character in them, replace them with their respective environment variable(which is already present in the shell) so that my JSON template is rendered with the correct environment variables.
I can use the --args option of jq but there are quite a lot of variables in my actual JSON template that I want to render.
I have been trying the following:
jq 'with_entries(.values as v | env.$v)
Basically making each value as a variable, then updating its value with the variable from the env object but seems like I am missing out on some understanding. Is there a straightforward way of doing this?
EDIT
Thanks to the answers on this question, I was able to achieve my larger goal for a part of which this question was asked
iterating over each value in an object,
checking its value,
if it's a string and starts with the character "$"
use the value to update it with an environment variable of the same name .
if it's an array
use the value to retrieve an environment variable of the same name
split the string with "," as delimiter, which returns an array of strings
Update the value with the array of strings
jq 'with_entries(.value |= (if (type=="array") then (env[.[0][1:]] | split(",")) elif (type=="string" and startswith("$")) then (env[.[1:]]) else . end))'
You need to export the Bash variables to be seen by jq:
export PROJECT1="randomtext1"
export PROJECT2="randomtext2"
export PROJECT4="randomtext3"
Then you can go with:
jq -n 'with_entries((.value | select(startswith("$"))) |= env[.[1:]])'
and get:
{
"host1": "randomtext1",
"host2": "randomtext2",
"host3": "xyz",
"host4": "randomtext3"
}
Exporting a large number of shell variables might not be such a good idea and does not address the problem of array-valued variables. It might therefore be a good idea to think along the lines of printing the variable=value details to a file, and then combining that file with the template. It’s easy to do and examples on the internet abound and probably here on SO as well. You could, for example, use printf like so:
printf "%s\t" ${BASH_VERSINFO[#]}
3 2 57 1
You might also find declare -p helpful.
See also https://github.com/stedolan/jq/wiki/Cookbook#arbitrary-strings-as-template-variables
I have a large JSON file that contains bigints with their full values--not rounded like JavaScript loves to do by default.
We have a workaround to deal with the bigints in Node.js, but I'm trying to use jq (the command-line tool) to clean up our data.
However, when I ran jq on our JSON file, it rounded all of our bigints.
Is there a way to use jq so that it doesn't round the bigints or is there perhaps another command-line tool that works on a Mac that I may use instead?
As of right now, the best jq has to offer with respect to JSON numbers is the "master" version, which preserves the external numerical value very well. The updates were made on or about 22 Oct 2019, and the "master" version of jq seems to be as safe to use as the most recent release (jq 1.6).
Examples using a recent "master" version:
jqMaster -n -M '
[0000,
10000000000000000000000000000000000000012,
1.0000000000000000000000000000000000000012,
1000000000000000000000000000000000000001210000000000000000000000000000000000000012,
0.1e123456]'
Output
[
0,
10000000000000000000000000000000000000012,
1.0000000000000000000000000000000000000012,
1000000000000000000000000000000000000001210000000000000000000000000000000000000012,
1E+123455
]
Another option would be to use “gojq”, the Go implementation of jq that uses unbounded-precision representation of integer literals.
In fact, except for one bug that has only been fixed in the “master” version of gojq as of this writing, gojq supports unbounded-precision integer arithmetic. The bug fix: https://github.com/itchyny/gojq/commit/7a1840289029c9c038d61274ceac9b8d307c0358
Consider a JSON object like
{
"foo": 42,
"baz": -12,
"bar{label1=\"value1\"}": 12.34
}
constructed by jq using some data source. The actual key names and their amount may vary, but the result will always be an object with numbers (int or float) as values. The keys may contain quotation marks, but no whitespaces.
Can I use jq to format the object into a Prometheus-compatible format so I can just use the output to push the data to a Prometheus Pushgateway?
The required result would look like
foo 42
bar{label1="value1"} 12.34
baz -12
i.e. space-separated with newlines (no \r) and without quotes except for the label value.
I can't use bash for post-processing and would therefore prefer a pure jq solution if possible.
Use keys_unsorted to get object keys (keys does the same as well but the former is faster), generate desired output by means of string interpolation.
$ jq -r 'keys_unsorted[] as $k | "\($k) \(.[$k])"' file
foo 42
baz -12
bar{label1="value1"} 12.34
And, by adding -j option and printing line feed manually as #peak suggested you can make this portable.
On a Windows platform, jq will normally use CR-LF for newlines; to prevent this, use the -j command-line option and manually insert the desired 'newline' characters like so:
jq -rj 'to_entries[] | "\(.key) \(.value)\n"' file
This must be incredibly simple but the man page makes no sense to me.
curl example.com/json gives me
{
"stats": {
"storage_server.disk_total": XXXXXXXXXX
},
"counters": {}
}
and I want to extract the value XXXXXXXXXX of the disk_total. What is the syntax to do this?
To get deeply nested values by their key:
$ jq '.. |."storage_server.disk_total"? | select(. != null)'
.. is a shortcut for the zero-argument recurse -- an analog of the XPath // operator.
For learning how to construct jq queries, it is more useful to look at the tutorial and manual than the "man" page. There's also a FAQ.
The inner key name has a period in it, and therefore the .keyname shorthand cannot be used for it. So you could write:
.stats["storage_server.disk_total"]
or if your jq allows it:
.stats."storage_server.disk_total"
These are both abbreviations for:
.stats | .["storage_server.disk_total"]
Tho dot in `storage_server.disk_total" needs to be escaped to prevent it from being interpreted as an object key separator. so you can use:
jq '.stats."storage_server.disk_total"'
assuming that XXXXXXXXXX is a valid JSON number in your real JSON.