I've encountered some difference between null and nothing, can somebody explain it? As in most languages null is considered/used to represent nothing.
The select is documented to return no output. And adding(ie. +) null to X yields X. Now consider these demonstrative examples(takes no input):
adding nothing
here we have empty object, which we update with nothing:
{} | . |= . + ({} | select (.foo == 123))
which results in
null
adding null
same template but with alternative operator to substitute nothing to null:
{} | . |= . + ({} | select (.foo == 123)//null)
which results in
{}
Can someone explain the difference nothing vs null?
null is just a regular JSON value; and conceptually, it is totally different from the absence of a value, i.e, what you termed nothing. Take a look at these for example (empty is a filter that returns nothing):
$ jq -n '[null] | length'
1
$ jq -n '[empty] | length'
0
That {} + null returns {} back, and that {} | . |= empty does exactly what del(.) does are merely design choices.
Related
I'm trying to make request in jq:
cat testfile.txt | jq 'fromjson | select(.kubernetes.pod.memory.usage.bytes != null) .kubernetes.pod.memory.usage.bytes, ."#timestamp"'
My output is:
"2019-03-15T00:24:21.733Z"
"2019-03-15T00:25:10.169Z"
"2019-03-15T00:24:47.908Z"
105889792
"2019-03-15T00:25:04.446Z"
34557952
"2019-03-15T00:25:04.787Z"
How to delete excess dates?
For example output only:
105889792
"2019-03-15T00:25:04.446Z"
34557952
"2019-03-15T00:25:04.787Z"
You just need to add a pipe after select :
cat testfile.txt | jq 'fromjson | select(.kubernetes.pod.memory.usage.bytes != null) | .kubernetes.pod.memory.usage.bytes, ."#timestamp"'
Here's a DRYer (as in dry) solution:
.["#timestamp"] as $ts | .kubernetes.pod.memory.usage.bytes // empty | ., $ts
Note that this particular use of // assumes that you wish to treat null, false, and a missing key in the same way. If not, you can still use the same idea to stay DRY.
I have the following JSON records stored in a container
{"memberId":"123","city":"New York"}
{"memberId":"234","city":"Chicago"}
{"memberId":"345","city":"San Francisco"}
{"memberId":"123","city":"New York"}
{"memberId":"345","city":"San Francisco"}
I am looking to check if there is any duplication of the memberId - ideally return a true/false and then also return the duplicated values.
Desired Output:
true
123
345
Here's an efficient approach using inputs. It requires invoking jq with the -n command-line option. The idea is to create a dictionary that keeps count of each memberId string value.
The dictionary can be created as follows:
reduce (inputs|.memberId|tostring) as $id ({}; .[$id] += 1)
Thus, to produce a true/false indicator, followed by the duplicates if any, you could write:
reduce (inputs|.memberId|tostring) as $id ({}; .[$id] += 1)
| to_entries
| map(select(.value > 1))
| (length > 0), .[].key
(If all the .memberId values are known to be strings, then of course the call to tostring can be dropped. Conversely, if .memberId is both string and integer-valued, then the above program won't differentiate between occurrences of 1 and "1", for example.)
bow
The aforementioned dictionary is sometimes called a "bag of words" (https://en.wikipedia.org/wiki/Bag-of-words_model). This leads to the generic function:
def bow(stream):
reduce stream as $word ({}; .[($word|tostring)] += 1);
The solution can now be written more concisely:
bow(inputs.memberId)
| to_entries
| map(select(.value > 1))
| (length > 0), .[].key
For just the values which have duplicates, one could write the more efficient query:
bow(inputs.memberId)
| keys_unsorted[] as $k
| select(.[$k] > 1)
| $k
Hello i managed to create this jq filter .profiles | recurse | .gameDir? | if type == "null" then "" else . end | scan("{REPLACE}.*") | sub("{REPLACE}"; "{REPLACESTRINGHERE}"). it succesfully replaces what i want (checked at jqplay.org) but now i'd like to print the full json and not just the modified strings
Adapting your query:
.profiles |= walk( if type == "object" and has("gameDir")
then .gameDir |=
(if type == "null" then "" else . end
| scan("{REPLACE}.*") | sub("{REPLACE}"; "{REPLACESTRINGHERE}"))
else .
end )
(This can easily be tweaked for greater efficiency.)
If your jq does not have walk, you can google it (jq “def walk”) or snarf its def from the jq FAQ https://github.com/stedolan/jq/wiki/FAQ
walk-free approach
For the record, here's an illustration of a walk-free approach using paths. The following also makes some changes in the computation of the replacement string -- notably it eliminates the use of scan -- so it is not logically equivalent, but is likely to be more useful as well as more efficient.
.profiles |=
( . as $in
| reduce (paths | select(.[-1] == "gameDir")) as $path ($in;
($in | getpath($path)
| if type == "null" then ""
else sub(".*{REPLACE}"; "{REPLACESTRINGHERE}")
end) as $value
| setpath($path; $value) ))
Say we have the following JSON:
[
{
"dir-1": [
"file-1.1",
"file-1.2"
]
},
"dir-1",
{
"dir-2": [
"file-2.1"
]
}
]
And we want to get the next output:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"
i.e. to get the paths to all leafs, joining items with /. Is there a way to do that on JQ?
I tried something like this:
cat source-file | jq 'path(..) | [ .[] | tostring ] | join("/")'
But it doesn't produce what I need even close.
You could take advantage of how streams work by merging the path with their values. Streams will only emit path, value pairs for leaf values. Just ignore the numbered indices.
$ jq --stream '
select(length == 2) | [(.[0][] | select(strings)), .[1]] | join("/")
' source-file
returns:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"
Here is a solution similar to Jeff Mercado's which uses tostream and flatten
tostream | select(length==2) | .[0] |= map(strings) | flatten | join("/")
Try it online at jqplay.org
Another way is to use a recursive function to walk the input such as
def slashpaths($p):
def concat($p;$k): if $p=="" then $k else "\($p)/\($k)" end;
if type=="array" then .[] | slashpaths($p)
elif type=="object" then
keys_unsorted[] as $k
| .[$k] | slashpaths(concat($p;$k))
else concat($p;.) end;
slashpaths("")
Try it online at tio.run!
Using --stream is good but the following is perhaps less esoteric:
paths(scalars) as $p
| getpath($p) as $v
| ($p | map(strings) + [$v])
| join("/")
(If using jq 1.4 or earlier, and if any of the leaves might be numeric or boolean or null, then [$v] above should be replaced by [$v|tostring].)
Whether the result should be regarded as "paths to leaves" is another matter...
How to recursively delete all keys that match a given pattern?
I have following jq config, but it doesn't seem to work:
walk( if (type == "object" and (.[] | test('.*'))) then del(.) else . end)
A robust way (with respect to different jq versions) to delete all keys matching a pattern (say PATTERN) would be to use the idiom:
with_entries(select( .key | test(PATTERN) | not))
Plugging this into walk/1 yields:
walk(if type == "object" then with_entries(select(.key | test(PATTERN) | not)) else . end)