I have a Python Azure Function that produces custom logging messages when the Function executes. I need to pull out some of the JSON values nested in the logging message.
How can I use Kusto to access the nested JSON within the logging message string?
Example logging message:
Desired values marked with <----------
####### EventGrid trigger processing an event:
{
"id": "long-guid",
"data": {
"api": "FlushWithClose",
"requestId": "long-guid",
"eTag": "long-guid",
"contentType": "application/octet-stream",
"contentLength": 16264, <----------------------
"contentOffset": 0,
"blobType": "BlockBlob",
"blobUrl": "https://function.blob.core.windows.net/parentdir/childdir/file.name",
"url": "https://function.dfs.core.windows.net/parentdir/childdir/file.name", <---- JUST FILE.NAME here
"sequencer": "long-guid",
"identity": "long-guid",
"storageDiagnostics": {
"batchId": "long-guid"
}
},
"topic": "/subscriptions/long-guid/resourceGroups/resourceGroup/providers/Microsoft.Storage/storageAccounts/accountName",
"subject": "/blobServices/default/containers/containerName/blobs/childDir/file.name",
"event_type": "Microsoft.Storage.BlobCreated"
} #######
I imagine it has something to do with the Kusto extend function, but piping in...
| extend parsedMessage = todynamic(message)
| project timestamp, test = parsedMessage["id"]
...yields only an empty test column
message in your specific case isn't a valid JSON payload - as it has the ###... EventGrid trigger processing an event: prefix (and a somewhat similar suffix).
That is why todynamic() isn't able to process it and why you're not able to reference properties in the JSON payload that's included in it.
Ideally, you would change the payload you ingest to be a valid JSON payload, and re-type the target column to dynamic instead of string.
If you can't do that, you can use the substring() function or parse operator to get everything but the aforementioned prefix/suffix, and parse the output of that using todynamic()
though note that doing that each time you query the data bears runtime overhead that could be avoided by following the advice above.
Related
I'm trying to save data to my MySql db from a Node method. This includes a field called attachments.
console.log(JSON.stringify(post.acf.attachments[0])); returns:
{
"ID": 4776,
"id": 4776,
"title": "bla",
"filename": "bla.pdf",
"filesize": 1242207,
"url": "https://example.com/wp-content/uploads/bla.pdf",
"link": "https://example.com/bla/",
"alt": "",
"author": "1",
"description": "",
"caption": "",
"name": "bla",
"status": "inherit",
"uploaded_to": 0,
"date": "2020-10-23 18:05:13",
"modified": "2020-10-23 18:05:13",
"menu_order": 0,
"mime_type": "application/pdf",
"type": "application",
"subtype": "pdf",
"icon": "https://example.com/wp-includes/images/media/document.png"
}
This is indeed the data I want to save to the db:
await existing_post.save({
...
attachments: post.acf.attachments[0],
)};
However, the attachments field produces a 422 server error (if I comment out this field, the other fields save without a problem to the db). I'm not getting what is causing this error. Any ideas?
I've also tried
await existing_post.save({
...
attachments: post.acf.attachments,
)};
but then it seems to just save "[object Object]" to the database.
The field in the database is defined as text. I've also tried it by defining the field as json, but that made no difference.
exports.up = function (knex, Promise) {
return knex.schema.table("posts", function (table) {
table.longtext("attachments");
});
};
The 422 error code is about the server unable to process the data you are sending to it. In your case, your table field is longtext when post.acf.attachments seems like an object. That's why it saves [object Object] to your db (It is the return value of the toString() method).
Try using
await existing_post.save({
...
attachments: JSON.stringify(post.acf.attachments),
)};
MySQL and knex both support the JSON format, I'd suggest you change the field to json. (See knex docs and mysql 8 docs). You'll stiil need to stringify your objects tho.
EDIT: I just saw that Knex supports jsonInsert (and plenty other neat stuff) as a query builder that should be useful for you.
Mysql also support a large range of cool stuffs for handling jsons
In addition, when you fetch the results in the database, you'll need to parse the JSON result to get an actual JSON object:
const acf = await knex('posts').select('acf').first();
const attachment = JSON.parse(acf.attachment;
Knex also provide jsonExtract that should fill your needs (See also the mysql json_extract
I have orchestrated a data pipe line using AWS Step function.
In last state I want to send a custom notification. I'm using an Intrinsic function States.Format to format my message and subject. It works fine for Context object element. Here, I have tested that in Message parameter.
But it doesn't work with input JSON. This is my input JSON
{
"job-param":{
"pipe-line-name":"My pipe line name", "other-keys":"other values"
}
}
"Success State": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message.$": "States.Format('Execution Id:{}, completed successfully!', $$.Execution.Id)",
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job-param.pipe-line-name)",
"TopicArn": "arn:aws:sns:us-east-1:************:sns-topic"
},
"End": true
}
While saving this state machine, it gives me following error message:
The value for the field 'Subject.$' must be a valid JSON Path
I checked Input and Result path. They have this value. I can directly use this value as parameter. This is working fine. But I can't format with other string.
"Subject.$": "$.job-param.pipe-line-name"
Alternate approach would be to call lambda to customize and trigger SNS. But I want to avoid that.
Can I request some suggestions to fix this error?
Thanks in advance!
If you want to use any name with - in your JSON then you can write your JSON Path like this:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $['job-param']['pipe-line-name'])",
But it would be easier if you change your input JSON and replace - with _:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job_param.pipe_line_name)",
I would like to configure sync between two different CRMs (Clevertap and Intercom) using Zapier and Webhooks. In general Clevertap sends the following JSON to webhook:
{
"targetId": 1548328164,
"profiles": [
{
"event_properties": {
"MSG-sms": true,
"MSG-push": true,
"businessRole": "EMPLOYEE",
"Mobile Number": "123123123123",
"Name": "Artem Hovtvianisa",
"Title": "Mr",
"Last Name": "Hovtvianisa",
"Gender": "M",
"Customer type": "Business Account Holder",
"MSG-email": true,
"First Name": "Artem",
"Last seen IP": "111.177.74.50",
"tz": "GMT+0200",
"International customer": "yes",
"isBusiness": true,
"Email": "xxxyyy#gmail.com",
"Identity": 15675
},
"objectId": "e32e4de3c1e84b2d9bab3707c92cd092",
"all_identities": [
"15675",
"xxxyyy#gmail.com"
],
"email": "xxxyyy#gmail.com",
"identity": "15675"
}
]
}
Zapier provides two types of catch webhook: regular and Raw.
Catch Raw Hook
When I use this type, JSON raw data will be handled OK and on the next step (Zapier JS code app) I am able to pass proper JSON data like in example above.
However when I use simple JS code to parse JSON object and get profiles[0] array value I get the following error "TypeError: Cannot read property '0' of undefined"
JS Code from Code step:
var result = JSON.parse(JSON.stringify(inputData));
console.log(result.profiles[0]);
return result;
Catch Hook
In case I use regular Catch Hook, hook parse data in some odd way, like this:
JSON.parse cannot recognize this structure.
Please advise how can I handle Webhook Zapier step in a proper way in order to get profiles[0] array item values?
Thanks in advance!
David here, from the Zapier Platform team. You're on the right track!
Catch Raw Hook is the way to go here. Your issue is that the data is coming in as a string and you're re-stringifying it before parsing it, which gets you back to where you came from. A simpler version:
JSON.stringify("asdf") // => "\"asdf\"", quotes in the string
JSON.parse("\"asdf\"") // => "asdf", the original string
"asdf".profiles // => undefined
undefined[0] // => error, cannot read property "0" of undefined
Instead, just parse it and you're good to go!
// all your variables are already in "inputData", so yours,
// also named inputData, must be referenced explicitly.
const result = JSON.parse(inputData.inputData);
return {result: result, firstProfile: result.profiles[0]};
I'm trying to use the hbase rest API. I've read the docs here (http://wiki.apache.org/hadoop/Hbase/Stargate), but they seem to be incomplete (doesn't cover json) and out of date.
I've also run across this gist, which is mostly what I'm going off of.
Here is the (insert) request I'm trying to send:
URI: http://server:8070/table/row1/data
Headers:
Content-Type : application/json
Accept : application/json
HTTP verb - POST
and the json payload (where key, column, and $ are all base64 encoded values):
{
"Row": {
"key": "NjQ=",
"Cell": [{
"column": "NjQ=",
"$": "NjQ="
}]
}
}
Here is the error I get:
Error 500 Can not deserialize instance of java.util.List out of START_OBJECT token
at [Source: org.mortbay.jetty.HttpParser$Input#3da0b822; line: 2, column: 5]
(through reference chain: org.apache.hadoop.hbase.rest.model.CellSetModel["Row"])
The error seems to suggest that there is something that should be an array and isn't. I've tried putting square brackets just about everywhere, and I can't get it to do anything other than change the nature of the error message slightly.
I had a look through the source code and that seems to suggest that the placement for the square brackets that I have is correct. However, as far as I can tell the entire request looks to be correct. I'm not very fluent in Java though, so maybe I'm missing something.
What is the proper JSON syntax for inserting a record with the hbase rest gateway?
This is now the correct format:
{
"Row": [{
"key": "NjQ=",
"Cell": [{
"column": "NjQ=",
"$": "NjQ="
}]
}]
}
Where:
key - is the row key
column - is the column name
$ - is the data that you are storing in the given table, column, and row.
I am trying to create a test case result using a REST client, but get this error:
"Errors": ["Cannot parse input stream due to I/O error as JSON document: Parse error: expected '{' but saw '\uFFFF' [ chars read = >>>\uFFFF<<< ]"]
I get the same error when the name of the object, testcaseresult is not specified in the request body. Here are the steps to create a test case result using a browser REST client:
a) Generate the authorize key using "GET" method and the following URL:
https://rally1.rallydev.com/slm/webservice/v2.0/security/authorize
This is the response that I get back, with the security token: "123abc..."
{"OperationResult": {"_rallyAPIMajor": "2", "_rallyAPIMinor": "0", "Errors": [], "Warnings": [], "SecurityToken": "abc123..."}}
b) Use "POST" method, and the following URL:
https://rally1.rallydev.com/slm/webservice/v2.0/testcaseresult/create?key=abc123...
notice the security token in the end.
c) here is an example of a request body:
{
"testcaseresult":
{
"Build":"1",
"Tester":"/user/777",
"Date":"2010-09-04T19:56:05.000Z",
"TestCase":"/testcase/1111",
"Verdict":"Pass"
}
}
Only the required fields and the fields you want to set need to be referenced. Notice the outer key/value pair
{
"testcaseresult":{}
}
The fields that point to a full object, like "Tester" (points to User object) and "TestCase" (points to a TestCase object that owns the result) have to be referenced by their ObjectIDs:
"Tester":"/user/777",
"TestCase":"/testcase/1111",