How to extract keys from a stringified json in Logstash - json

Upon using a multi-line codec, my single json object contains escape backslashes for each quote used for the key-value pairs.
Kindly help me access/extract individuaL key pairs from such a json using logstash plugins.
"msg" => "{ \"message\": \"type=SYSCALL msg=audit(23:45:56): arch=c000003e syscall=1 success=yes exe=\"/usr/sbin/sshd\" subj=adfghjk SYSCALL=write AUID=\"awx\" UID=\"root\" GID=\"root\"\", \"log\": { \"offset\": 9274863, \"file\": { \"path\": \"/var/log/audit/audit.log\" } }}"
Regards

Related

Extract nested JSON content from JSON with use of JSON Extractor in jMeter

I have JSON content inside another JSON that I need to extract as it is, without parsing its contents:
{
"id": 555,
"name": "aaa",
"JSON": "{\r\n \"fake1\": {},\r\n \"fake2\": \"bbbb\",\r\n \"fake3\": \"eee\" \r\n}",
"after1": 1,
"after2": "test"
}
When I use JSON Extractor with JSON Path expression:
$.JSON
It returns:
"{
"fake1": {},
"fake2": "bbbb",
"fake3": "eee"
}"
when I need to get the raw string:
"{\r\n \"fake1\": {},\r\n \"fake2\": \"bbbb\",\r\n \"fake3\": \"eee\" \r\n}"
I think you need to switch to JSR223 PostProcessor instead of the JSON Extractor and use the following code:
def json = new groovy.json.JsonSlurper().parse(prev.getResponseData()).JSON
vars.put('rawString', org.apache.commons.text.StringEscapeUtils.escapeJson(json))
You will be able to refer the extracted value as ${rawString} where required.
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy: What Is Groovy Used For?
console.log(JSON.stringify(data.JSON))
Here data is your given JSON data.
At first, you have to extract your JSON/data. Then you have to stringify the JSON data using JSON.stringify().
The confusing fact you have done here is that you named your key in the JSON object as "JSON".
In js when you extract a JSON object if there is another nested JSON object you will always get JSON data by just data.key_name
where data is JSON data
key is for Nested JSON key

JSONPath: Get field starting with # in escaped json string

I want to get a nested field in a json string using JSONPath.
Take for example the following json:
{
"ID": "2ac464eb-352f-4e36-8b9f-950a24bb9586",
"PAYLOAD": "{\"#type\":\"Event\",\"id\":\"baf223c4-4264-415a-8de5-61c9c709c0d2\"}"
}
If I want to extract the #type field, I expect to do it like this
$.PAYLOAD.#type
But that doesn't seem to work..
Also tried this:
$.PAYLOAD['#type']
Do I need to use escape chars or something?
Posting my comment as an answer
"{\"#type\":\"Event\",\"id\":\"baf223c4-4264-415a-8de5-61c9c709c0d2\"}"
Isn't JSON, it's a string containing encoded JSON.
Since JsonPath can't decode such string, you'll have to use a language of your desire to decode the string.
Eg: Decoding JSON String in Java

Extracting data from raw json object block and converting into dataframe in python with keys

I have sets of data in a JSON objects, and I have to convert it into a key value pairs, but it has extreme nested strings and list, how can I get the dictionary of it and extract only required data value in a data frame, the data is like :
[{
"log": "{name:start_game,start:1638316800661,url:https:www.game.com,body:{GAME_ROOM_ID:ahsbssjs,BOOT_AMOUNT:5,WALLET_UNIT:rs,MAX_GAME_DURATION:1800000,PLAY_MODE:12,PLAYERS:[{USER_ID:12222233,USER_TOKEN:,BLOCK_AMOUNT:5},{USER_ID: 12222234,USER_TOKEN:,BLOCK_AMOUNT:5},{USER_ID:1222243,USER_TOKEN:,BLOCK_AMOUNT:5},{USER_ID:12222221,USER_TOKEN:,BLOCK_AMOUNT:5}],SHOULD_START_GAME_ON_DEBIT_FAILURE:false},end:1638316800963,status:200,error:,result:{CONCESSION_INFO:{BONUS_AMOUNT_USED:1.6400000000000001,COUPON_DISCOUNT:0,TICKET_DISCOUNT:0,TOTAL_DISCOUNT:0},MESSAGE:SUCCESSFUL,PLAYERS_WITH_DEBIT_FAILURE:[],RAKE:2,TOTAL_DEBIT_AMOUNT:20},headers:{ Content-Type:application,Authorization:Basic, partnerToken:000000000 }}"
}, {
"log": "{name:start_game,start:1638316801624,url: url:https:www.game.com,body:{GAME_ROOM_ID:ahsbssjs,BOOT_AMOUNT:10,WALLET_UNIT:rs,MAX_GAME_DURATION:1800000,PLAY_MODE:12,PLAYERS:[{USER_ID:222211112,USER_TOKEN:,BLOCK_AMOUNT:10},{USER_ID: 222211114,USER_TOKEN:,BLOCK_AMOUNT:10},{USER_ID: 222211113,USER_TOKEN:,BLOCK_AMOUNT:10},{USER_ID: 222211115 USER_TOKEN:,BLOCK_AMOUNT:10}],SHOULD_START_GAME_ON_DEBIT_FAILURE:false},end:1638316802191,status:200,error:,result:{CONCESSION_INFO:{BONUS_AMOUNT_USED:0.65,COUPON_DISCOUNT:0,TICKET_DISCOUNT:0,TOTAL_DISCOUNT:0},MESSAGE:SUCCESSFUL,PLAYERS_WITH_DEBIT_FAILURE:[],RAKE:4,TOTAL_DEBIT_AMOUNT:40},headers:{Content-Type:application,Authorization:Basic,partnerToken:000000000 }}"
}]
Output :

PromTail JSON Scrape Special Character

For a PromTail scrape config, I am using a JSON stage.
I have a JSON log that looks like this:
{
"#l": "info",
"foo": "bar"
}
I am looking to use the JSON stage to extract the #l property into the map.
I tried this:
- json:
expressions:
level: '"#l"'
- labels:
level:
The agent starts but no logs are scraped. If I remove the JSON stage, tons of logs come in.
What could I be doing wrong with the # escape sequence?
I have confirmed. To escape a # or ., you use double quotes.
so examples:
{
"#l": "Debug",
"foo.bar": "value"
}
'"#l"'
or
'"foo.bar"'
Source
Using a JMESPath Literal
This pipeline uses a literal JMESPath expression to parse JSON fields with special characters in the name, like # or .

Hive: parse string into json

I have a string column which is stored in json format
{"name":
{"type":"json",
"payload":"
{\"id\":
\"123\",
\"activities\": [....]
}
}
}
I need to parse it into json format.
and then turn it into map of <name, payload> where payload is rest of the json.
"name" is a key and not a constant string.
what is the best way to do this?