For a PromTail scrape config, I am using a JSON stage.
I have a JSON log that looks like this:
{
"#l": "info",
"foo": "bar"
}
I am looking to use the JSON stage to extract the #l property into the map.
I tried this:
- json:
expressions:
level: '"#l"'
- labels:
level:
The agent starts but no logs are scraped. If I remove the JSON stage, tons of logs come in.
What could I be doing wrong with the # escape sequence?
I have confirmed. To escape a # or ., you use double quotes.
so examples:
{
"#l": "Debug",
"foo.bar": "value"
}
'"#l"'
or
'"foo.bar"'
Source
Using a JMESPath Literal
This pipeline uses a literal JMESPath expression to parse JSON fields with special characters in the name, like # or .
Related
I want to get a nested field in a json string using JSONPath.
Take for example the following json:
{
"ID": "2ac464eb-352f-4e36-8b9f-950a24bb9586",
"PAYLOAD": "{\"#type\":\"Event\",\"id\":\"baf223c4-4264-415a-8de5-61c9c709c0d2\"}"
}
If I want to extract the #type field, I expect to do it like this
$.PAYLOAD.#type
But that doesn't seem to work..
Also tried this:
$.PAYLOAD['#type']
Do I need to use escape chars or something?
Posting my comment as an answer
"{\"#type\":\"Event\",\"id\":\"baf223c4-4264-415a-8de5-61c9c709c0d2\"}"
Isn't JSON, it's a string containing encoded JSON.
Since JsonPath can't decode such string, you'll have to use a language of your desire to decode the string.
Eg: Decoding JSON String in Java
I'm sending a message to kafka manually using kafka-console-producer where key and value are jsons. When it arrives it seems to be encoded to json again and i don't know how to construct the message to be encoded to json i want to send or how to configure either the producer I'm using or the kafka broker itself not to encode messages.
Here is example:
I have file message.json:
{"distributionID": "TB-AARTool-30","senderID": "TB-AARTool","dateTimeSent": 1635405698193,"dateTimeExpires": 1635405698193,"distributionStatus": "System","distributionKind": "Request"};{"TrialId": 3,"TrialSessionId": 5,"TrialStageId": 4}
I'm sending it using command:
./kafka-console-producer.sh --broker-list localhost:9092 --topic topic --property "parse.key=true" --property "key.separator=;" < /message.json
and when it arrives it looks like:
{
"topic": "topic",
"key": "{\"distributionID\": \"TB-AARTool-30\",\"senderID\": \"TB-AARTool\",\"dateTimeSent\": 1635405698193,\"dateTimeExpires\": 1635405698193,\"distributionStatus\": \"System\",\"distributionKind\": \"Request\"}",
"value": "{\"TrialId\": 3,\"TrialSessionId\": 5,\"TrialStageId\": 4}",
"partition": 0,
"offset": 1
}
First of all at the beginning and end of the key and value quotation marks (") were added. Second, all quotation marks (") already in the message were replaced with backslash quotation mark (\"). I just want key and value to be received by kafka as valid jsons.
I'm using:
confluentinc/cp-kafka:latest docker image which i presume is latest stable kafka server version.
kafka_2.12-3.0.0 from which I'm using bin/kafka-console-producer.sh (https://www.apache.org/dyn/closer.cgi?path=/kafka/3.0.0/kafka_2.12-3.0.0.tgz) which also contains kafka server and zookeepr but i don't use them.
Your data produced fine. Try using kafka-console-consumer instead that will not show escaped JSON strings.
Regarding the shown output, appears as if you are consuming your record from some REST API or other non-Kafka built-in tool? In that case, yes, your key and value are escaped strings because that is what was produced - strings, not JSON objects.
Therefore, you aren't able to get data like this without customizing whatever deserializer gets this output
{
"topic": "topic",
"key": {
"distributionID": ...
...
}
"value": {
"TrialId": ...
...
}
...
}
This isn't a problem for other consumers that are configured to use StringDeserializer as you can pass those strings into a JSON parser to get objects
So I have some json in which the keys might be something like this:
{
"name" : "John",
"num:itparams:enterprise:2.0:content" : {
"housing" : "5"
},
"num rooms": "12"
}
I get this json from an http request, and I need to use the evaluateJsonPath processor to create attributes from them.
name is easy, i just use $.name
But how would I access the other two? I imagine you would put them in quotes somehow to escape the special characters but just doing $."num:itparams:enterprise:2.0:content" doesnt work.
You can use the bracket for the key-value which has the special characters such as
$.['num:itparams:enterprise:2.0:content'].housing
then it will give you the evaluated result 5.
I am trying to invoke my Lambda function by passing parameters as below. it contains apostrophe(').
https://invoke_url?param=[["kurlo jack's book","Adventure Books",8.8,1]]
Stringifyed to be 'https://invoke_url?param=%5B%5B%229780786706211%22s....`
I used the mapping below to pass parameter to lambda
"query": {
#foreach($queryParam in $input.params().querystring.keySet())
"$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end
#end
}
I got following error
{"message": "Could not parse request body into json: Unrecognized character escape \'\'\' (code 39)\n at [Source: [B#5b70c341; line: 29, column: 65]"}
i have also tried after removing double quotes from mapping template. But did't work.
Be sure to add .replaceAll("\\'","'") to your request body passthrough template after .escapeJavaScript(data)
I found this bit from AWS's documentation to be very helpful for this issue:
$util.escapeJavaScript()
Escapes the characters in a string using JavaScript string rules.
Note This function will turn any regular single quotes (') into
escaped ones (\'). However, the escaped single quotes are not valid in
JSON. Thus, when the output from this function is used in a JSON
property, you must turn any escaped single quotes (\') back to regular
single quotes ('). This is shown in the following example:
$util.escapeJavaScript(data).replaceAll("\\'","'")
I don't have a solution but I have narrowed the root cause. Lambda does not seem to like single quotes to be escaped with a single slash.
If you hardcode your mapping template to look like this:
{
"query-fixed": {
"param": "[[\"kurlo jack\\'s book\",\"Adventure Books\",8.8,1]]"
}
}
my test Lambda invocation succeeds. However, if you hardcode the template to this:
{
"query-fixed": {
"param": "[[\"kurlo jack\'s book\",\"Adventure Books\",8.8,1]]"
}
}
I get the same error message that you got above. Unfortunately, the second variation is what API Gateway produces for the Lambda invocation.
A workaround might involve using the template to replace single quotes escaped with slash to two slashes. See Replace a Substring of a String in Velocity Template Language
I'll follow up with Lambda internally and update if I hear anything or have a functional workaround.
Try changing your encoding of ' to %27 as per what is is defined in this W3Schools page (ironically their example does not encodes the single quote either, I guess its because it belongs to the "supported" ASSCII set of characters)
The "query string" (the part in the hyperlink after ?) must be a string. Whatever you have constructing that must be appended to it like: https://invoke_url?a=x&b=y
In your Lambda code put:
if( event.hasOwnProperty( 'params' ) )
if( event.params.hasOwnProperty( 'querystring' ) )
params = event.params.querystring;
(obviously some extraneous checks, probably unnecessary but ehh)
In your API Gateway go to:
APIs -> api_name -> Resources -> invoke_url -> GET -> Method Execution
Under URL Query String Parameters "Add query string" a and b (or whatever)
When you hit www.com/invoke_url?a=x&b=y you can now access them with:
...
params = event.params.querystring;
console.log( params.a, params.b );
...
I am just learning to use the jsonlite library to fetch json data from a server. However in the received json response (whose structure I have no control over), there seems to be a node that I can only describe as a chunk of 'escaped' JSON data, right in the middle of the json object. How do I build a JSON object out of it? I am able to extract each such value OK but then I can't use it as is without turning it into a true JSON object.
example:
library(jsonlite)
myFakeJSON <- '"{"country": "UK","ranking": "45"}"'
json <- toJSON(myFakeJSON)
but:
> json
[1] "\"{\"country\": \"UK\",\"ranking\": \"45\"}\""
The result is not a json object.. What am I doing wrong? How do I escape (or un-escape??) the received data? Seems like something obvious, but not to me :(
I think you are making two errors. First: too many quotes, second: wrong test for JSON-hood. If you wanted to use toJSON, then you would give it an R object for conversion rather than a effort at as JSON string.
> myFakeJSON <- '{"country": "UK", "ranking": "45"}'
> fromJSON(myFakeJSON)
$country
[1] "UK"
$ranking
[1] "45"
If you just need to remove the extra double-quotes on the "outside" of the curly braces, then this regex replacement succeeds on this small example:
> json <- fromJSON(gsub("\\}\\\"", "}", gsub("\\\"\\{","{", myFakeJSON))); json
$country
[1] "UK"
$ranking
[1] "45"