Build a JSON document from a sqlite table - json

I've been looking at the JSON1 extension for SQLite databases as a potential tool to use to compose a JSON document from a table-valued result.
I have a result set coming out of my SQLite database that resembles something like this:
CLASS|INSTANCE|PROPERTY|FROM|TO
ABC|12345|COLOR|RED|
ABC|12345|COLOR|GREEN|
ABC|12345|WEIGHT|1|10
ABC|56789|COLOR|BLUE|
ABC|56789|HEIGHT|4.5|6.2
DEF|2345|NAME|YOMOMMA|
I've been trying to make a JSON document based off of this data to look like:
{
"ABC": {
"12345": {
"COLOR": [
{ "from": "RED", "to": "" },
{ "from": "GREEN", "to": ""}
],
"WEIGHT": [
{ "from": "1", "to": "10" }
]
},
"56789": {
"COLOR": [
{ "from": "BLUE", "to": "" }
],
"HEIGHT": [
{ "from": "4.5", "to": "6.2" }
]
}
},
"DEF": {
"2345": {
"NAME": [
{ "from": "YOMOMMA", "to": "" }
]
}
}
}
I've been trying to get the JSON1 functions to help me out with this, but it seems they don't really support a multilayered JSON format (or at least, not that I've been able to see). If I wrap my result set in a group by query, I can get the properties to nicely turn into a JSON array, but when I try to wrap that into the next level, all the JSON gets escaped (would have been nice if there was a json_raw() function, but I'd imagine implementing it would be challenging).
Can this be done (relatively) easily using SQLite/ JSON1? Is there a different/ better way to create my document than using JSON1, or am I just going to have to write code for this?

Related

Best way to wrangle json data to remove field labels

I am looking to try make it easier to use the Google Analytics data in Google sheets.
The outputs that I receive from the api look like :
[
{
"dimensionValues": [
{
"value": "id2"
},
{
"value": "(not set)"
},
{
"value": "Android"
}
]
},
{
"dimensionValues": [
{
"value": "id1"
},
{
"value": "stream name"
},
{
"value": "iOS"
}
]
}
]
Whats the most efficient method to remove the field names to make a "flatter set of arrays" similar to:
[["id2","(not set)","Android"], ["id1","stream name","iOS"]]
It feels like there should be a quick way to do this!
You can use a double map for this one. Map the value to its dimensionValues and then map those to the array elements to get the designated output.
let array = [{"dimensionValues": [{"value": "id2"},{"value": "(not set)"},{"value": "Android"}]},{"dimensionValues": [{"value": "id1"},{"value": "stream name"},{"value": "iOS"}]}]
console.log(array.map(x => x.dimensionValues.map(y => y.value)))

jOOQ JSON formatting as array of objects

I have the following (simplified) jOOQ query:
val result = context.select(
jsonObject(
key("id").value(ITEM.ID),
key("title").value(ITEM.NAAM),
key("resources").value(
jsonArrayAgg(ITEM_INHOUD.RESOURCE_ID).absentOnNull()
)
)
).from(ITEM).fetch()
Now the output that I want is:
[
{
"id": "0da04cc5-f70c-4fb3-b5c7-dc645d342631",
"title": "Title1",
"resources": [
"8b0f6d5c-67fc-47ca-be77-d1735e7721ce",
"ea0316db-1cfd-46d7-8260-5c1a4e65a0cd"
]
},
{
"id": "0f7e67e6-5187-47e2-9f1d-dab08feba38b",
"title": "Title2"
}
]
result.formtJSON() gives the following output:
{
"fields": [
{
"name": "json_object",
"type": "JSON"
}
],
"records": [
[
{
"id": "0da04cc5-f70c-4fb3-b5c7-dc645d342631",
"title": "Title 1"
}
]
]
}
Disabling the headers with result.formatJSON(JSONFormat.DEFAULT_FOR_RECORDS) will get me:
[
[
{
"id": "0da04cc5-f70c-4fb3-b5c7-dc645d342631",
"title": "Title1",
"resources": [
"8b0f6d5c-67fc-47ca-be77-d1735e7721ce",
"ea0316db-1cfd-46d7-8260-5c1a4e65a0cd"
]
}
],
[
{
"id": "0f7e67e6-5187-47e2-9f1d-dab08feba38b",
"title": "Title2"
}
]
]
where I don't want the extra array.
Further customizing the JSONformatter with result.formatJSON(JSONFormat().header(false).recordFormat(JSONFormat.RecordFormat.OBJECT)) I get:
[
{
"json_object": {
"id": "0da04cc5-f70c-4fb3-b5c7-dc645d342631",
"title": "Title1",
"resources": [
"8b0f6d5c-67fc-47ca-be77-d1735e7721ce",
"ea0316db-1cfd-46d7-8260-5c1a4e65a0cd"
]
}
},
{
"json_object": {
"id": "0f7e67e6-5187-47e2-9f1d-dab08feba38b",
"title": "Title2"
}
}
]
where I don't want the object wrapped in json_object.
Is there a way to get the output I want?
Doing it with Result.formatJSON()
This is clearly a flaw in the jOOQ 3.14.0 implementation of Result.formatJSON(). In the special case where there is only one column, and that column is of type JSON or JSONB, the column name may not really matter, and thus its contents should be flattened into the object describing the row. I've created a feature request for this: https://github.com/jOOQ/jOOQ/issues/10953. It will be available in jOOQ 3.15.0 and 3.14.4. You will be able to do this:
result.formatJSON(JSONFormat().header(false).wrapSingleColumnRecords(false));
The RecordFormat is irrelevant here. This works the same way for RecordFormat.ARRAY and RecordFormat.OBJECT
Doing it directly with SQL
Of course, you can always work around this by moving all the logic into SQL. You probably simplified your query by omitting a JOIN and GROUP BY. I'm assuming this is equivalent to what you want:
JSON result = context.select(
jsonArrayAgg(jsonObject(
key("id").value(ITEM.ID),
key("title").value(ITEM.NAAM),
key("resources").value(
select(jsonArrayAgg(ITEM_INHOUD.RESOURCE_ID).absentOnNull())
.from(ITEM_INHOUD)
.where(ITEM_INHOUD.ITEM_ID.eq(ITEM.ID))
)
))
).from(ITEM).fetchSingle().value1()
Note that JSON_ARRAYAGG() aggregates empty sets into NULL, not into an empty []. If that's a problem, use COALESCE()

In Logic Apps JSON Array while parsing throwing error for single object but for multiple objects it is working fine

While parsing JSON in Azure Logic App in my array I can get single or multiple values/objects (Box as shown in below example)
Both type of inputs are correct but when only single object is coming then it is throwing an error "Invalid type. Expected Array but got Object "
Input 1 (Throwing error) : -
{
"MyBoxCollection":
{
"Box":{
"BoxName": "Box 1"
}
}
}
Input 2 (Working Fine) : -
{
"MyBoxCollection":
[
{
"Box":{
"BoxName": "Box 1"
},
"Box":{
"BoxName": "Box 2"
}
}]
}
JSON Schema :
"MyBoxCollection": {
"type": "object",
"properties": {
"box": {
"type": "array",
items": {
"type": "object",
"properties": {
"BoxName": {
"type": "string"
},
......
.....
..
}
Error Details :-
[
{
"message": "Invalid type. Expected Array but got Object .",
"lineNumber": 0,
"linePosition": 0,
"path": "Order.MyBoxCollection.Box",
"schemaId": "#/properties/Root/properties/MyBoxCollection/properties/Box",
"errorType": "type",
"childErrors": []
}
]
I used to use the trick of injecting a couple of dummy rows in the resultset as suggested by the other posts, but I recently found a better way. Kudos to Thomas Prokov for providing the inspiration in his NETWORG blog post.
The JSON parse schema accepts multiple choices as type, so simply replace
"type": "array"
with
"type": ["array","object"]
and your parse step will happily parse either an array or a single value (or no value at all).
You may then need to identify which scenario you're in: 0, 1 or multiple records in the resultset? I'm pasting below how you can create a variable (ResultsetSize) which takes one of 3 values (rs_0, rs_1 or rs_n) for your switch:
"Initialize_ResultsetSize": {
"inputs": {
"variables": [
{
"name": "ResultsetSize",
"type": "string",
"value": "rs_n"
}
]
},
"runAfter": {
"<replace_with_name_of_previous_action>": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Check_if_resultset_is_0_or_1_records": {
"actions": {
"Set_ResultsetSize_to_0": {
"inputs": {
"name": "ResultsetSize",
"value": "rs_0"
},
"runAfter": {},
"type": "SetVariable"
}
},
"else": {
"actions": {
"Set_ResultsetSize_to_1": {
"inputs": {
"name": "ResultsetSize",
"value": "rs_1"
},
"runAfter": {},
"type": "SetVariable"
}
}
},
"expression": {
"and": [
{
"equals": [
"#string(body('<replace_with_name_of_Parse_JSON_action>')?['<replace_with_name_of_root_element>']?['<replace_with_name_of_list_container_element>']?['<replace_with_name_of_item_element>']?['<replace_with_non_null_element_or_attribute>'])",
""
]
}
]
},
"runAfter": {
"Initialize_ResultsetSize": [
"Succeeded"
]
},
"type": "If"
},
"Process_resultset_depending_on_ResultsetSize": {
"cases": {
"Case_no_record": {
"actions": {
},
"case": "rs_0"
},
"Case_one_record_only": {
"actions": {
},
"case": "rs_1"
}
},
"default": {
"actions": {
}
},
"expression": "#variables('ResultsetSize')",
"runAfter": {
"Check_if_resultset_is_0_or_1_records": [
"Succeeded",
"Failed",
"Skipped",
"TimedOut"
]
},
"type": "Switch"
}
For this problem, I met another stack overflow post which is similar to this problem. While there is one "Box", it will be shown as {key/value pair} but not [array] when we convert it to json format. I think it is caused by design, so maybe we can just add a record "Box" at the source of your xml data such as:
<Box>specific_test</Box>
And do some operation to escape the "specific_test" in the next steps.
Another workaround for your reference:
If your json data has only one array, we can use it to do a judgment. We can judge the json data if it contains "[" character. If it contains "[", the return value is the index of the "[" character. If not contains, the return value is -1.
The expression shows as below:
indexOf('{"MyBoxCollection":{"Box":[aaa,bbb]}}', '[')
The screenshot above is the situation when it doesn't contain "[", it return -1.
Then we can add a "If" condition. If >0, do "Parse JSON" with one of the schema. If =-1, do "Parse JSON" with the other schema.
Hope it would be helpful to your problem~
We faced a similar issue. The only solution we find is by manipulating the XML before conversion. We updated XML nodes which needs to be an array even when we have single element using this. We used a Azure function to update the required XML attributes and then returned the XML for conversion in Logic Apps. Hope this helps someone.

How to fetch inner node of JSON using freemarker?

Sample JSON:
{
"results": [
{
"_id": "12345",
"CustomerAccount": {
"Status": "ACTIVE",
"Address": [
{
"FormatCode": "PRIM",
"FreeFormatAddress": "ENGLAND"
},
{
"FormatCode": "SEC",
"FreeFormatAddress": "IRELAND"
}
]
},
"LegalVehicleID": "01",
"BrokerAccount": {
"Status": "ACTIVE",
"Address": [
{
"FormatCode": "PRIM123",
"FreeFormatAddress": "SG"
},
{
"FormatCode": "SEC123",
"FreeFormatAddress": "IND"
}
]
}
}
]
}
So i want to get Address as list regardless of CustomerAccount or BrokerAccount.
result.*.Address and then access FormatCode and FreeFormatAddress using FreeMarker.
Please let me know how to do so thanks.
There's no such feature in the template language itself. Such things can be achieved with a TemplateModel that was designed to do queries against JSON, similarly as such queries are possible against XML (W3C DOM to be more precise) out-of-the-box. But I'm not aware of anybody has written such thing for JSON (and even then it's question what Java objects represent the JSON... like, is it Jackson JsonNode-s?).

Trouble Parsing Array vs Non-Array JSON with JSONPath

I have JSON that looks like the below. I'm trying to use JSONPath to grab the __ content __ value where the SKU is "8A-OK9F-9LI8" AND the Component.Type == 'Principal'. Right now, I am playing around with this JSON Path Expression Tester.
This JSONPath expression grabs all of the component information I need:
$.Order..Fulfillment[?(#.SKU=='8A-OK9F-9LI8')]..Component
But filtering further such as $.Order..Fulfillment[?(#.SKU=='8A-OK9F-9LI8')]..Component[?(#.Type=='Principal')] grabs only one (I believe the Array one) of the two Component elements I need. I suspect this is because one is an Array and one is a single JSON element. Is it possible to grab this with one command or do I have to combine several commands (one for the Array and one for the single JSON element)? If so, how can I grab the other Component information that I am not currently getting with:
$.Order..Fulfillment[?(#.SKU=='8A-OK9F-9LI8')]..Component[?(#.Type=='Principal')]?
Again, my goal is to grab the "__ content__" value and filter by a specific SKU and where the Component.Type == 'Principal'. Something like:
$.Order..Fulfillment[?(#.SKU=='8A-OK9F-9LI8')]..Component[?(#.Type=='Principal')]..Amount..__content__
I'm expecting to get back ["8.49", "8.49"]
Here is the JSON I am testing with:
{
"SettlementData": {},
"Order": [
{
"OrderID": "XXX",
"Fulfillment": {
"Item": {
"SKU": "8A-OK9F-9LI8",
"Quantity": "1",
"ItemPrice": {
"Component": [
{
"Type": "Principal",
"Amount": {
"__content__": "8.49",
"currency": "USD"
}
},
{
"Type": "Tax",
"Amount": {
"__content__": "0.74",
"currency": "USD"
}
}
]
}
}
}
},
{
"OrderID": "XXX",
"Fulfillment": {
"Item": {
"SKU": "8A-OK9F-9LI8",
"Quantity": "1",
"ItemPrice": {
"Component": {
"Type": "Principal",
"Amount": {
"__content__": "8.49",
"currency": "USD"
}
}
}
}
}
}
]
}
I was able to solve this in two passes. In this example, #{sku} is a Ruby interpolated string that contains the SKU I am passing in:
$.Order..Fulfillment[?(#.SKU=='#{sku}')]..ItemPrice..[?(#.Type=='Principal')].Amount.__content__
$.Order..Fulfillment..Item[?(#.SKU=='#{sku}')]..ItemPrice..[?(#.Type=='Principal')].Amount.__content__
Using a Ruby gem "jsonpath", I was able to get the amounts I needed like this:
amount = JsonPath.on(settlement, "$.Order..Fulfillment[?(#.SKU=='#{sku}')]..ItemPrice..[?(#.Type=='Principal')].Amount.__content__")
.map(&:to_f).inject(:+)
amount2 = JsonPath.on(settlement, "$.Order..Fulfillment..Item[?(#.SKU=='#{sku}')]..ItemPrice..[?(#.Type=='Principal')].Amount.__content__")
.map(&:to_f).inject(:+)