I have the following json output available and i need to extract the value of href which is the https URL using jq processor.
I have tried using
jq -r .links.urn:vodafoneid:follow.hrefs
However this does not work ?
JSON Output:
{
"links":{
"urn:somedomainid:follow":{
"href":"https://abc.somedomain.com/ula/login?service=IDGW&channel=WEB&usecaseid=a0b51311-d14b-4733-9e6b-ba5f5deec05f&opco=DE&nonce=89e31cde-fecc-41e1-91d6-1f9f84f9c136&acr_values=explicit&scopes=phone_number&returnUrl=https%3A%2F%2Fidgw.somedomain.com%2Fauthorize%23state%3Da0b51311-d14b-4733-9e6b-ba5f5deec05f",
"type":"text/html"
}
},
"context":"FOLLOW"
}
You have an obvious typo in the field name that you are trying to use vodafoneid is not somedomainid. But in general to access a field having special characters like : in their names, do a proper quoting of the field as below.
jq --raw-output '.links."urn:somedomainid:follow".href'
jqplay.org - URL
Related
For a PromTail scrape config, I am using a JSON stage.
I have a JSON log that looks like this:
{
"#l": "info",
"foo": "bar"
}
I am looking to use the JSON stage to extract the #l property into the map.
I tried this:
- json:
expressions:
level: '"#l"'
- labels:
level:
The agent starts but no logs are scraped. If I remove the JSON stage, tons of logs come in.
What could I be doing wrong with the # escape sequence?
I have confirmed. To escape a # or ., you use double quotes.
so examples:
{
"#l": "Debug",
"foo.bar": "value"
}
'"#l"'
or
'"foo.bar"'
Source
Using a JMESPath Literal
This pipeline uses a literal JMESPath expression to parse JSON fields with special characters in the name, like # or .
I am new here so sorry if I do any mistakes while asking the question.
I have a json file that keeps updating every minute(File_1.json) with json objects. All i want to do is copy these objects to another file under a single array using the jq command.
Samples of files
File_1.json:
{
"Id":"1",
"Name":"Kiran",
"Age":"12"
}
{
"Id":"2",
"Name":"Dileep",
"Age":"22"
}
Expected Output
[
{
"Id":"1",
"Name":"Kiran",
"Age":"12"
}
{
"Id":"2",
"Name":"Dileep",
"Age":"22"
}
]
I have tried using -s(slurp) but since the code will be running once for every minute its creating multiple arrays.
If you wanted simply to append the objects in File_1.json to an existing array in (say) output.json, you could write:
jq '. + [inputs]' output.json File_1.json
This presupposes output.json contains exactly one array (or the JSON value null). So to start, you could initialize output.json by running:
echo null > output.json
If you want to take the risk and overwrite output.json, you might like to use sponge:
jq '. + [inputs]' output.json File_1.json | sponge output.json
If you want to remove duplicates and don't mind sorting the objects, you could simply append | unique to the above jq filter. If retaining the order is important, then see
https://github.com/stedolan/jq/wiki/Cookbook#using-bag-to-implement-a-sort-free-version-of-unique
I'm setting up a devops pipeline so that certain data profiles stored in JSON format can be shifted across different servers. While downloading it from the current server I need to clean up all the protected keys and unique identifiers. I'm looking for the cleanest way to do the following in JQ
Input:
{
"TopKey1":{
"some_key":"some_value"
},
"TopKey2":{
"some_key2":"some_value2"
},
"KeytoSearch":[
{
"_id":"sdf",
"non_relevant_key1":"val"
},
{
"_id":"sdfdsdf",
"non_relevant_key2":"val"
},
{
"_id":"sgf",
"non_relevant_key3":"val"
}
]
}
Output:
{
"TopKey1":{
"some_key":"some_value"
},
"TopKey2":{
"some_key2":"some_value2"
},
"KeytoSearch":[
{
"non_relevant_key1":"val"
},
{
"non_relevant_key2":"val"
},
{
"non_relevant_key3":"val"
}
]
}
In python terms if this were a dictionary
for json_object in dictionary["KeytoSearch"]:
json_object.pop("_id")
I've tried combinations of map and del but can't seem to figure out the nested indexing with this. The error messages I get are along the lines of jq: error (at <stdin>:277): Cannot index string with string "_id" which sort of tells me I haven't fundamentally understood how jq works or is to be used, but this is the route I need to go because using a Python script to clean up JSON objects is something I'd rather avoid
Going with your input JSON and assuming there are other properties in your KeytoSearch object along with the _id fields, you could just do below.
jq 'del(.KeytoSearch[]._id)'
See this jqplay.org snippet for a demo. The quotes around the property key containing _ are not needed as confirmed in one of the comments below. Some meta-characters (e.g. . in the property key values needs be accessed with quotes as ".id") needs to be quoted properly, but _ is clearly not one of them.
I've tried combinations of map and del
Good! You were probably just missing the '|=' magic ingredient:
.Keytosearch |= map( del(._id) )
alternatively, you could use a walk-path unix tool for JSON: jtc and apply changes right into the sourse json file (-f):
bash $ jtc -fpw'[KeytoSearch]<_id>l:' file.json
bash $
bash $
bash $ jtc file.json
{
"KeytoSearch": [
{
"non_relevant_key1": "val"
},
{
"non_relevant_key2": "val"
},
{
"non_relevant_key3": "val"
}
],
"TopKey1": {
"some_key": "some_value"
},
"TopKey2": {
"some_key2": "some_value2"
}
}
bash $
if given json snippet is a part of a larger JSON (and [KeytoSearch] is not addressable from the root), then replace it with the search lexeme: <KeytoSearch>l.
PS> Disclosure: I'm the creator of the jtc tool
I am new to shell scripting and I need some help.
I am trying to use jq to get values from a api response and check for its correctness.
Here is a sample for how the response looks like,
{
"data" : {
"transactionType" : "Sales",
"transactionSubType" : "DomesticSale",
"Items" : [ {
"itemID" : "2",
"itemType" : "Good",
"amount" : 5.0,
"tax" : 1.0
} ]
}
}
I am able to get the values for transactionType or transactionsubtype or even ItemID values etc as given below
jq '.data.transactionType'
jq '.data.Items[0].itemID'
for Transaction type and item id
but when it comes to values of numeric types i.e., without the quotes in it, I don't get any value.
I am using similar syntax for the numeric type also as shown below.
jq '.data.Items[0].amount'
jq '.data.Items[0].tax'
Please help!!!
Your jq invocations are fine, but the sample data is missing a final closing brace ("}"), so perhaps you were not feeding jq properly.
If you're wondering why you didn't see an error message, it's almost certainly because jq 1.5 is not very good about handling incomplete JSON. The problem has since been fixed at "master". With the current version, you'd see something like this:
parse error: Unfinished JSON term at EOF at line 15, column 0
I am trying to write a JSON path expression to extract all maps and submaps from a JSON structure. Considering the JSON:
{
"k1":"v1",
"arr": ["1","2","3" ,["7","8"] ],
"submap":
{
"a":"b",
"c":"d"
},
"submap_2":
{
"a_2":"b",
"c_2":"d",
"nested": { "x":"y" }
}
}
I would want to extract the elements "submap", "submap_2", "nested".
I've tried JSONPath expressions like:
$..*[?(#.length()>0 && #.*[0] empty true)]
This returns the structures I want, but also returns [ "7","8" ]. Is there any way to do this with JSONPath or is this better done in code?
(A neat JSONPath testing tools is here: http://jsonpath.herokuapp.com/)
(The specific implementation that I'm using is this one: https://github.com/jayway/JsonPath )
jq queries are often very similar to JSONPath queries, and I would strongly recommend that if at all possible you consider using jq.
Assuming the example data is in a file named example.json, the following invocation of jq produces the result you requested:
$ jq 'path(.. | select(type=="object")) | .[-1] | select(.)' example.json
"submap"
"submap_2"
"nested"
The output of the first filter (path(....)) consists of the full path expressions of the paths to all the JSON objects, including the top-level object itself. The remaining filters are needed to produce the exact output you requested. In practice, though, the full path expressions are probably more useful, so it might be helpful for you to see the output produced by the first filter:
$ jq -c 'path(.. | select(type=="object"))' example.json
[]
["submap"]
["submap_2"]
["submap_2","nested"]