422 error trying to save json data to the database - mysql

I'm trying to save data to my MySql db from a Node method. This includes a field called attachments.
console.log(JSON.stringify(post.acf.attachments[0])); returns:
{
"ID": 4776,
"id": 4776,
"title": "bla",
"filename": "bla.pdf",
"filesize": 1242207,
"url": "https://example.com/wp-content/uploads/bla.pdf",
"link": "https://example.com/bla/",
"alt": "",
"author": "1",
"description": "",
"caption": "",
"name": "bla",
"status": "inherit",
"uploaded_to": 0,
"date": "2020-10-23 18:05:13",
"modified": "2020-10-23 18:05:13",
"menu_order": 0,
"mime_type": "application/pdf",
"type": "application",
"subtype": "pdf",
"icon": "https://example.com/wp-includes/images/media/document.png"
}
This is indeed the data I want to save to the db:
await existing_post.save({
...
attachments: post.acf.attachments[0],
)};
However, the attachments field produces a 422 server error (if I comment out this field, the other fields save without a problem to the db). I'm not getting what is causing this error. Any ideas?
I've also tried
await existing_post.save({
...
attachments: post.acf.attachments,
)};
but then it seems to just save "[object Object]" to the database.
The field in the database is defined as text. I've also tried it by defining the field as json, but that made no difference.
exports.up = function (knex, Promise) {
return knex.schema.table("posts", function (table) {
table.longtext("attachments");
});
};

The 422 error code is about the server unable to process the data you are sending to it. In your case, your table field is longtext when post.acf.attachments seems like an object. That's why it saves [object Object] to your db (It is the return value of the toString() method).
Try using
await existing_post.save({
...
attachments: JSON.stringify(post.acf.attachments),
)};
MySQL and knex both support the JSON format, I'd suggest you change the field to json. (See knex docs and mysql 8 docs). You'll stiil need to stringify your objects tho.
EDIT: I just saw that Knex supports jsonInsert (and plenty other neat stuff) as a query builder that should be useful for you.
Mysql also support a large range of cool stuffs for handling jsons
In addition, when you fetch the results in the database, you'll need to parse the JSON result to get an actual JSON object:
const acf = await knex('posts').select('acf').first();
const attachment = JSON.parse(acf.attachment;
Knex also provide jsonExtract that should fill your needs (See also the mysql json_extract

Related

How to parse nested JSON, within a string, using Kusto

I have a Python Azure Function that produces custom logging messages when the Function executes. I need to pull out some of the JSON values nested in the logging message.
How can I use Kusto to access the nested JSON within the logging message string?
Example logging message:
Desired values marked with <----------
####### EventGrid trigger processing an event:
{
"id": "long-guid",
"data": {
"api": "FlushWithClose",
"requestId": "long-guid",
"eTag": "long-guid",
"contentType": "application/octet-stream",
"contentLength": 16264, <----------------------
"contentOffset": 0,
"blobType": "BlockBlob",
"blobUrl": "https://function.blob.core.windows.net/parentdir/childdir/file.name",
"url": "https://function.dfs.core.windows.net/parentdir/childdir/file.name", <---- JUST FILE.NAME here
"sequencer": "long-guid",
"identity": "long-guid",
"storageDiagnostics": {
"batchId": "long-guid"
}
},
"topic": "/subscriptions/long-guid/resourceGroups/resourceGroup/providers/Microsoft.Storage/storageAccounts/accountName",
"subject": "/blobServices/default/containers/containerName/blobs/childDir/file.name",
"event_type": "Microsoft.Storage.BlobCreated"
} #######
I imagine it has something to do with the Kusto extend function, but piping in...
| extend parsedMessage = todynamic(message)
| project timestamp, test = parsedMessage["id"]
...yields only an empty test column
message in your specific case isn't a valid JSON payload - as it has the ###... EventGrid trigger processing an event: prefix (and a somewhat similar suffix).
That is why todynamic() isn't able to process it and why you're not able to reference properties in the JSON payload that's included in it.
Ideally, you would change the payload you ingest to be a valid JSON payload, and re-type the target column to dynamic instead of string.
If you can't do that, you can use the substring() function or parse operator to get everything but the aforementioned prefix/suffix, and parse the output of that using todynamic()
though note that doing that each time you query the data bears runtime overhead that could be avoided by following the advice above.

How to remove or prevent escape slashes from json-file data in azure logic app workflow

Situation
I'm using Azure Logic App to do the following steps:
Load/read a JSON file from blob storage. This file is created and written in R, using the toJSON function fromt JSONlite package
Get de file content from this JSON file
Convert it to a JSON format (using expression json())
Insert the converted output into a Azure Table Storage
Please look at the screenshot for this workflow
azure workflow
I've created a JSON-file in R (using the JSONlite package). When printed in the R console, it looks fine:
{"PartitionKey":"pdf","RowKey":"robject)test","nr":"24330087","status":true}
This is the 'formatting' I want. When in this format, it inserts smoothly with logic app into the azure table storages.
Problem
But when I write the JSON-file above to Azure Blob, and use this file as input in a logic app workflow, these escape slashes cause problems
The file from the blob storage isn't interpreted by logic app in the desired format (without escape slashed) It includes the escape slashes (). And I think this causes the problems in logic apps. I pasted a hardcoded format without these slashes. And this worked. But hard coding isn't an option here.
Below the format the logic app 'json()' expression makes of it. It includes the dreaded escapes.
[
"{\"PartitionKey\":\"pdf\",\"RowKey\":\"coolblue_factuur_1744212700.pdf\",\"kvknr\":\"24330087\",\"active_status\":true}"
]
And then this Error occurs:
{
"odata.error": {
"code": "InvalidInput",
"message": {
"lang": "en-US",
"value": "An error occurred while processing this request.\nRequestId:xxxx\nTime:2019-11-20T09:02:46.6051827Z"
}
}
}
After some online research, it looks like Logic App is having difficulty with the escape slashes () and the dubble quotes it escapes ("). So all these (\") escapes are choking the flow.
Question
How to deal with \" (escape slashes?). All I want is that logic app can correctly read the JSON-file from blob storage, convert it and instert the data into the table storage.
So ideally: convert the JSON file from blob storage into a format without escape slashes. I tried to pre process this in R, but this did not work.
Extra info
Below you can find the steps I took in logic app:
The JSON file uploaded (and fetched) from blob storage
{
"headers": {
"Pragma": "no-cache",
"Transfer-Encoding": "chunked",
"Retry-After": "15",
"Vary": "Accept-Encoding",
"x-ms-request-id": "xxxx",
"Strict-Transport-Security": "max-age=31536000; includeSubDomains",
"X-Content-Type-Options": "nosniff",
"X-Frame-Options": "DENY",
"Timing-Allow-Origin": "*",
"x-ms-apihub-cached-response": "true",
"Cache-Control": "no-store, no-cache",
"Date": "Wed, 20 Nov 2019 09:09:52 GMT",
"Location": "https://[location url]",
"Set-Cookie": "[cookieset]",
"Content-Type": "application/json",
"Expires": "-1",
"Content-Length": "452"
},
"body": {
"Id": "xyz",
"Name": "robjecttest_parameters_db.json",
"DisplayName": "robjecttest_parameters_db.json",
"Path": "/path/robjecttest_parameters_db.json",
"LastModified": "2019-11-20T09:09:39Z",
"Size": 95,
"MediaType": "application/octet-stream",
"IsFolder": false,
"ETag": "\"[etag]\"",
"FileLocator": "[filelocations]",
"LastModifiedBy": null
}
}
From the headers, the content type is 'JSON'. But the body (which is the data I want) it is 'octet-stream').
After a 'get filecontent' action:
{
"$content-type": "application/octet-stream",
"$content": "[content]"
}
Next step is to convert/cast the 'body-data' to a JSON format. Using the expression & dynamic content from logic APP
json(body('Get_blob_content'))
Use this 'output' as an entity to insert into the table storage.
You just need to replace the \" with " by the expression below:
replace(string(body('Get_blob_content')), '\"', '"')
And then we can use json() to convert it.
Update:
In your json data to insert to table storage, the data in red box below cause the problem
You can't insert into it with the second level of data.

Zapier Catch (Raw) Hook JSON parsing issue

I would like to configure sync between two different CRMs (Clevertap and Intercom) using Zapier and Webhooks. In general Clevertap sends the following JSON to webhook:
{
"targetId": 1548328164,
"profiles": [
{
"event_properties": {
"MSG-sms": true,
"MSG-push": true,
"businessRole": "EMPLOYEE",
"Mobile Number": "123123123123",
"Name": "Artem Hovtvianisa",
"Title": "Mr",
"Last Name": "Hovtvianisa",
"Gender": "M",
"Customer type": "Business Account Holder",
"MSG-email": true,
"First Name": "Artem",
"Last seen IP": "111.177.74.50",
"tz": "GMT+0200",
"International customer": "yes",
"isBusiness": true,
"Email": "xxxyyy#gmail.com",
"Identity": 15675
},
"objectId": "e32e4de3c1e84b2d9bab3707c92cd092",
"all_identities": [
"15675",
"xxxyyy#gmail.com"
],
"email": "xxxyyy#gmail.com",
"identity": "15675"
}
]
}
Zapier provides two types of catch webhook: regular and Raw.
Catch Raw Hook
When I use this type, JSON raw data will be handled OK and on the next step (Zapier JS code app) I am able to pass proper JSON data like in example above.
However when I use simple JS code to parse JSON object and get profiles[0] array value I get the following error "TypeError: Cannot read property '0' of undefined"
JS Code from Code step:
var result = JSON.parse(JSON.stringify(inputData));
console.log(result.profiles[0]);
return result;
Catch Hook
In case I use regular Catch Hook, hook parse data in some odd way, like this:
JSON.parse cannot recognize this structure.
Please advise how can I handle Webhook Zapier step in a proper way in order to get profiles[0] array item values?
Thanks in advance!
David here, from the Zapier Platform team. You're on the right track!
Catch Raw Hook is the way to go here. Your issue is that the data is coming in as a string and you're re-stringifying it before parsing it, which gets you back to where you came from. A simpler version:
JSON.stringify("asdf") // => "\"asdf\"", quotes in the string
JSON.parse("\"asdf\"") // => "asdf", the original string
"asdf".profiles // => undefined
undefined[0] // => error, cannot read property "0" of undefined
Instead, just parse it and you're good to go!
// all your variables are already in "inputData", so yours,
// also named inputData, must be referenced explicitly.
const result = JSON.parse(inputData.inputData);
return {result: result, firstProfile: result.profiles[0]};

JSON request using Postman

I am sending a raw Json requet using postman to an API service which feeds it to another web service and finally a database. I want to attach a file to the raw Json request.
I am attaching below the current request I am sending. Is it the right way? The first name and other information is going through but the attachment is not. Any suggestions?
{
"Prefix": "",
"FirstName": "test-resume-dlyon",
"LastName": "test-dlyon-resume",
"AddressLine1": "test2",
"AddressLine2": "",
"City": "Invalid Zipcode",
"State": "GA",
"Zip": "99999",
"Phone": "9999999999",
"Email": "testresumedlyon#gmail.com",
"Source": "V",
"WritingNumber": "",
"AgeVerified": true,
"AdditionalSource": "",
"EnableInternetSource": true,
"InternetSource": "",
"ExternalResult": "",
"PartnerID": "",
"SubscriberID": "15584",
"Languages": [
"English",
"Spanish"
],
"fileName": "resume",
"fileExtension": "docx",
"fileData": "UELDMxE76DDKlagmIF5caEVHmJYFv2qF6DpmMSkVPxVdtJxgRYV"
}
There is no "correct" format to attach a file to a JSON.
JSON is not multipart/form-data (which is designed to include files).
JSON is a text-based data format with a variety of data types (such as strings, arrays, and booleans) but nothing specific for files.
This means that to attach a file, you have to get creative.
For example, you could encode a file in text format (e.g. using base64), but it wouldn't be very efficient, and any Word document would result in you getting a much longer string than "UELDMxE76DDKlagmIF5caEVHmJYFv2qF6DpmMSkVPxVdtJxgRYV".
Of course, the method you use to encode the file has to be the method that whatever is reading the JSON expects you to use. Since there is no standard for this, and you have said nothing about the system which is consuming the JSON you are sending, we have no idea what that method is.
First of all, I'd recommend reading the postman API docs. They have some extremely useful information on there for using the API. Two particular articles that might of interest here are these:
Looking at it and running it through a validator like this one shows that there are no syntax errors so it must be to do with the JSON parameters the API is expecting.
Here's something you can try:
In postman, set method type to POST.
Then select Body -> form-data -> Enter your parameter name (file according to your code)
and on right side next to value column, there will be dropdown "text, file", select File. choose your image file and post it.
For rest of "text" based parameters, you can post it like normally you do with Postman. Just enter parameter name and select "text" from that right side dropdown menu and enter any value for it, hit send button. Your controller method should get called.

JSON API result format

Folks,
Designing my first API in Node.JS using restify.js. My background is not webapis, pardon my amateur questions. In any case, I would like to have the res.send(data); responses to comply with the http://jsonapi.org/format/ so that my mobile application can start utilizing the api calls. At the moment if you were to call my api, it would return data in the following format:
{"Count":1,"Items":[{"dbsource":{"S":"foo"},"id":{"S":"5002820"},"name":{"S":"fnameblah,lnameblah"},"expiration":{"S":"06/13/2015"},"type":{"S":"bar"}}]}
Actually what you see above is just a return of a DynamoDB Query call.
So the question is... do you use a special library that you can pass data to, which would format and return the data in JSON format. Which in turn you can return it via res.send(data) to the clients, or is it up to us to make 'data' JSON compliant, then return it? At the end of the day we all want the results to look like:
{
"posts": [{
"id": "1",
"title": "Rails is Omakase",
"links": {
"author": "9",
"comments": [ "5", "12", "17", "20" ]
}
}]
}
Thanks!
In server side, stringify JSON object,
//...
res.statusCode = 200;
res.setHeader('Content-Type', 'application/json');
res.send(JSON.stringify(data)); //data is JSON object
res.end();
In client side, parse JSON string accordingly.
EDIT: Corrected response content type.
JSON data from server should be a JSON string
You have to parse it back the JSON format in client.
JSON.parse(string); // return JSON object