I am parsing the output from the below curl as follows -
curl 'http://www.bom.gov.au/fwo/IDN60801/IDN60801.95765.json' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' | jq '.[][][] | select(.apparent_t>5)
How can retain only certain fields in the output. Expected output is
[
{
"sort_order":147,
"lat":-33.8,
"lon":151.1,
"apparent_t":10.9
},
{
"sort_order":148,
"lat":-23.8,
"lon":128.1,
"apparent_t":7
},
{
"sort_order":236,
"lat":-33.34,
"lon":151.74,
"apparent_t":21.9
}
]
Basically, I need to filter out only certain fields & print key, values for it. I am not able to work out th next steps for this. I'd truly appreciate any help here.
If you know the keys you want to extract, just ... extract those.
curl 'http://www.bom.gov.au/fwo/IDN60801/IDN60801.95765.json' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' |
jq '[.[][][] | select(.apparent_t>5) | {"sort_order": .sort_order, "lat": .lat, "lon": .lon, "apparent_t": .apparent_t}]'
This outputs a list of dictionaries; take out the outer [...] if you actually want just a dictionary at a time in a sequence of JSON fragments in the output (maybe combine with -c then to get one dictionary per line).
Related
I am using AWS cli to fetch cloudwatch log but after fetching the JSON output includes backslash with key and value. But I am checking this log from console then that is absolutely fine( there are without backslash).
So now my question is that how to fetch cliudwatch log without this backslash in JSON format? Because jq cannot iterate over backslash in key name.
json current output -
{
"events": [
{
"logStreamName": "xxxxx75914_CloudTrail_ap-south-1",
"timestamp": 1648750347210,
"message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"Root\",\"principalId\":\"xxxxxxxx\",\"arn\":\"arn:aws:iam::xxxxxx:root\",\"accountId\":\"xxxxxxx\",\"accessKeyId\":\"xxxxxxxxx\",\"sessionContext\":{\"attributes\":{\"creationDate\":\"2022-03-31T09:38:58Z\",\"mfaAuthenticated\":\"false\"}}},\"eventTime\":\"2022-03-31T18:07:15Z\",\"eventSource\":\"s3.amazonaws.com\",\"eventName\":\"GetObject\",\"awsRegion\":\"ap-south-1\",\"sourceIPAddress\":\"xx.xx.xx.xx\",\"userAgent\":\"[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36]\",\"requestParameters\":{\"X-Amz-Date\":\"20220331T180714Z\",\"bucketName\":\"s3trail-bucket\",\"X-Amz-Algorithm\":\"AWS4-HMAC-SHA256\",\"response-content-disposition\":\"attachment\",\"X-Amz-SignedHeaders\":\"host\",\"Host\":\"s3trail-bucket.s3.ap-south-1.amazonaws.com\",\"X-Amz-Expires\":\"300\",\"key\":\"folder2/remove_httpd.yml\"},\"responseElements\":null,\"additionalEventData\":{\"SignatureVersion\":\"SigV4\",\"CipherSuite\":\"ECDHE-RSA-AES128-GCM-SHA256\",\"bytesTransferredIn\":0.0,\"AuthenticationMethod\":\"QueryString\",\"x-amz-id-2\":\"sd35j39Gup5qnNv6xN464A66HdTiPZqc53k+rMN+RO0n97J3jBLTELDzX8fG9M6YgE0VvEMvQm0=\",\"bytesTransferredOut\":147.0},\"requestID\":\"GK9A2F1DF0Q6MBNF\",\"eventID\":\"2fab9c39-7e7c-4f16-ad5b-752ad5839cb7\",\"readOnly\":true,\"resources\":[{\"type\":\"AWS::S3::Object\",\"ARN\":\"arn:aws:s3:::s3trail-bucket/folder2/remove_httpd.yml\"},{\"accountId\":\"201043775914\",\"type\":\"AWS::S3::Bucket\",\"ARN\":\"arn:aws:s3:::s3trail-bucket\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":false,\"recipientAccountId\":\"201043775914\",\"eventCategory\":\"Data\",\"tlsDetails\":{\"tlsVersion\":\"TLSv1.2\",\"cipherSuite\":\"ECDHE-RSA-AES128-GCM-SHA256\",\"clientProvidedHostHeader\":\"s3trail-bucket.s3.ap-south-1.amazonaws.com\"}}",
"ingestionTime": 1648xxxxxxxx,
"eventId": "3676836138911xxxxxxxxxx"
},
],
"searchedLogStreams": []
}
json desired output -
My cloudwatch api is -
aws logs filter-log-events --log-group-name s3_log_2 --log-stream-names Xxxxxxx_CloudTrail_ap-south-1 --start-time 1648188352000 --end-time 1648786144000 --filter-pattern ["eventName"="GetObject"]
You can use fromjson feature inside your jq command:
jq -r ".events[].message | fromjson"
Here is how I use it with your command:
aws logs filter-log-events --log-group-name aws-logs-cloudtrail-example --log-stream-names 0123456789_CloudTrail_ap-southeast-1 --start-time 1648809754000 --end-time 1648809854000 | jq -r ".events[].message | fromjson"
You can pass json to jq to unescape it.
Use function walk() can expend nested json ( require jq >= 1.6 )
PASTE_YOUR_AWS_COMMAND_HERE |./jq 'walk(if type == "string" then . as $x | try fromjson catch $x else . end)'
Reference
jq manual
How to convert embedded (quoted) json string to json
Use jq to interpret nested JSON in JSON
This question already has answers here:
How to get key names from JSON using jq
(9 answers)
Closed 1 year ago.
lets say I have this:
https://api.apis.guru/v2/specs/adyen.com/CheckoutService/64/openapi.json
and I want to grab:
/orders
/orders/cancel
/originKeys
/paymentLinks
/paymentLinks/{linkId}
/paymentMethods
/paymentMethods/balance
/paymentSession
/payments
/payments/details
/payments/result
I want to do that on an elegant way, just grabbing the paths, what I am doing is this ugly workaround, that doesn't work well with some apis:
curl -A "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0" -ks "https://api.apis.guru/v2/specs/adyen.com/CheckoutService/64/openapi.json" | jq -r '.paths' | grep "/" | grep "{" | egrep -iv "microsoft|amz|application/json|application/jwt|\*|text/xml|text/plain|application/|multipart/form-data|text/html" | cut -d "\"" -f 2
Using jq:
$ jq -r '.paths | keys | .[]' openapi.json
/orders
/orders/cancel
/originKeys
/paymentLinks
/paymentLinks/{linkId}
/paymentMethods
/paymentMethods/balance
/paymentSession
/payments
/payments/details
/payments/result
In your case, you'd want to pipe your curl output to jq instead of using a file, of course.
The import bits compared to your use of jq is piping the .paths array to keys and that to .[] to get one element per line instead of a JSON array, and -r to avoid printing each line as a JSON string complete with quote marks.
Below is an example of a JSON payload I'd like to send. How do I wrap the 'value' value so it handles the special characters like ; and " and * and / and carriage returns better?
{
"type":"INLINE",
"name":"${varEvidence-1}",
"value":"GET /mxca/userprofile/us/submenudata.do
request_type=authreg_submit&page=)(sn%3d HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) "Gecko"/20100101
Firefox/31.0
Accept: /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
Referer: https://example.com//mxca/userprofile/us/submenudata.do
request_type=authreg_submit&page=*)(sn%3d*
Cookie:
keymaster=PMV54Bye4RKtzab8vXdJyW6S8sCDe37OJuw0vx072smBAr7GW8D3rstCzWPUsEU29aM2MLFoglHzhATZISEQyYbsyNvg%3D%3D;
gatekeeper=4C7205995D7B2AFFA76E7B1335A890FCC4924C9C1E0F554077CDE2A651DD12D0A1CE69FB46D757A75FB64130179FD9001650153DA13887D33581F621453F81F560D81CFCAFE51922B6ADABF8C5A45F932491BA1325866F29B24CB2D73328A0FDBB72F1868E208C1786310849A6E5D2332E045C90CADC559A78DEA614ACE4A18E5262DDD7D9AE9854EB6EA7C9BA8BB68DC3F5DDAEA3C930442FA25FCBAF6D25F5DC6AE0C890E01A83E2CB1E70510B537BF63E653045C3B52B0E5FE728740894D87AA5599885F72DA1FDDF0D8AA9883FB3BE035EBA65CAEC15;
blueboxvalues=f93b466e-1f61c944-ba2aec26-cf40345c;
sessioncookie=easc=D0D8264E667BF43A91A02012B400E9B5A05175762DE38B9FA2D88675680B8419FD83366DE7A706A2E3391F9B0A4FDA9CE7E7168575DF4204233E855BAAC4CD01D386D62B27213A4D7595C69AB6EA15B7770572832321047ABC1627E6F1A1ECE62FAB6532AF78A9E3E888090D3EAFA80C92730CBE395556E578CA2F137E3A121CD20
Connection: keep-alive",
"docLockerId":null
}
As pet the JSON specs any characters can constitute a string except for " and \. If you have these in your JSON string you need to escape them using \ character.
So you just have to replace the character " with \" and the character \ with \\. Any other special characters are considered as characters in string.
Here is a list of JSON special characters.
\b Backspace
\f Form feed
\n New line
\r Carriage return
\t Tab
\" Double quote
\\ Backslash character
This stack overflow post
has a ton of information regarding this question.
You could also use the specs reference for more information.
Essentially what I'm attempting to do is post to a REST API, but no matter what I do I end up with HTTP 400. Here is my extremely quick and extremely dirty code:
package require rest
package require json
::http::register https 443 ::tls::socket
set credentials {username admin password LabPass1}
set url1 [format "%s/%s" "https://127.0.0.1:8834" session]
set unformattedToken [dict get [::json::json2dict [::rest::post $url1 $credentials]] token]
set cookie [format "token=%s" $unformattedToken]
set header [list X-Cookie $cookie Content-type application/json]
set config [list method post format json headers $header]
set url [format "%s/%s" "https://127.0.0.1:8834" scans]
set uuid 7485-2345-566
set name "Testing TCL Network Scan"
set desc "Basic Network Scan using API"
set pid 872
set target 127.0.0.1
set data {{"uuid":"$uuid","settings": {"name":"$name","description":"$desc", "policy_id":"$pid","text_targets":"$target", "launch":"ONETIME","enabled":false,"launch_now":true}}}
set jsonData [json::json2dict $data]
set response [::rest::simple $url $jsonData $config]
I've tried using the above code and I've also tried removing the json::json2dict call and just sending the data. I believe, and I could be wrong, that my issue is the data is going as line-based text data:
POST /scans HTTP/1.1
Host: 127.0.0.1:8834
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 10.0) http/2.8.9 Tcl/8.6.4
Connection: close
X-Cookie: token=301b8dcdf855a29b5b902cf8d93c49750935c925a965445e
Content-type: application/json
Accept: */*
Accept-Encoding: gzip,deflate,compress
Content-Length: 270
uuid=7485-2345-566&settings=name%20%7BTesting%20TCL%20Network%20Scan%7D%20description%20%7BBasic%20Network%20Scan%20using%20API%7D%20policy_id%20872%20text_targets%20127.0.0.1%20launch%20ONETIME%20enabled%20false%20launch_now%20true
I've reviewed the JSON documentation, and the REST documentation but I'm having a hard time finding an example of posting using JSON format. Here is what this looks like in a curl command:
curl https://127.0.0.1:8834/scans -k -X POST -H 'Content-Type: application/json' -H 'X-Cookie: token= <token>' -d '{"uuid":"7485-2345-566","settings":{"name":"Testing TCL Network Scan","description":"Basic Network Scan using API", "policy_id":"872","text_targets":"127.0.0.1", "launch":"ONETIME","enabled":false,"launch_now":true}'
One problem you have is that the values in the query aren't evaluated. "uuid":"$uuid" becomes "uuid":"$uuid", for instance. This is because of the braces around the value that data is set to.
The best solution would seem to be to not create a json object and then convert it to a dict, but instead create the dict directly, like this:
set data [list uuid $uuid settings [list name $name description $desc policy_id $pid text_targets $target launch ONETIME enabled false launch_now true]]
or like this, for shorter lines:
dict set data uuid $uuid
dict set data settings name $name
dict set data settings description $desc
dict set data settings policy_id $pid
dict set data settings text_targets $target
dict set data settings launch ONETIME
dict set data settings enabled false
dict set data settings launch_now true
or by some other method.
Documentation: dict, list, set
A simple logstash example is not working for me. I want to read my apache access.log and dump it out. I use the following configuration file
input {
file {
path => "/var/log/apache2/access.log"
start_position => beginning
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout {}
}
and I cannot get json output from logstash as shown here.
root#rick-VirtualBox:/opt/logstash# ./bin/logstash -f /home/rick/log_conf/first-pipeline.conf
Logstash startup completed
2015-08-10T18:58:07.660Z rick-VirtualBox 192.168.56.1 - - [10/Aug/2015:12:46:21 -0400] "GET / HTTP/1.1" 200 427 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/600.6.3 (KHTML, like Gecko) Version/8.0.6 Safari/600.6.3"
When i use the grokdebugger on the line above it says its a COMBINEDAPACHELOG pattern, which i believe is what i am asking for.
Isn't logStash suppose to create json?
Actually json is not the default codec. According to the docs Logstash's default output codec for stdout is line.
Try:
output {
stdout { codec => json }
}
You might also consider using codec => rubydebug which also produces a very clean output.