Generating XML with SSIS - What are best practices - json

So I'm trying to generate an XML from data that I receive in a JSON file.
What I've done so far is that I had stored each field in the JSON as a keypair. So a row would be FieldName, FieldValue, and JSON_PK.
I have a second table that I created in order to create the XML. It has the JSON FieldName, equivalent XML FieldName, and indentation. The plan was to create a loop in SSIS to manually create the XML.
It was suggested to be that I use instead the FOR XML in my query.
However I've run an issue that every field is named FieldName. It's complicated by fields that hold their values like this <Form submittedDate="2020-01-01"/>
So before I go back to creating a loop to create my XML, I'm wondering what are best practices? I can't be the first one to run into this issue.
Thanks!
A quick followup because it was requested:
This is the approximate for that the JSON comes in as, except is far longer:
{
"name": "Mylist",
"id": "9e8-19c9-e5",
"templateName": "VDashboard - Plus",
"categories": [
""
],
"attributes": [
{
"name": "Division ID",
"value": "ABCD",
"Id": "123",
"units": null,
"timestamp": "1970-01-01T00:00:00.0000000Z",
"errors": null
},
{
"name": "ETA ",
"value": null,
"Id": "123",
"units": null,
"timestamp": "2021-01-25T21:24:36.2514056Z",
"errors": null
},
{
"name": "ETA Destination - Estimated Time",
"value": "1/11/2021 4:15:34 PM",
"Id": "123",
"units": null,
"timestamp": "1970-01-01T00:00:00.0000000Z",
"errors": null
}
]
}
And I need to output it as an XML File.
I need to import it into the DB because I do transformation of certain fields.
Output should look at bit like this:
2020-12-03T08:00:00-05:00
0011
My table structure looks like this. It's done so that I won't have a different table for every report:
Name VARCHAR(4) NOT NULL,
ID VARCHAR(50),
TemplateName VARCHAR(50),
AttributeName VARCHAR(50),
AttributeSubName VARCHAR(50),
AttributeValue VARCHAR(50),
AttributeID VARCHAR(50),
AttributeUnits VARCHAR(50),
AttributeTimestamp DateTime,
AttributeErrors VARCHAR(50),

I've managed to resolve the issue.
Initially, I had hoped to put the JSON into a table, and create the XML by putting those values into a string with xml tags around it.
This XML though requires both Elements and Attributes and it's contents are dynamic. Therefore creating an SQL query was far too difficult to create and harder to troubleshoot.
I found that the easiest way was to create the whole thing in C#.
I used a script task to use C#. Also, I needed a table that functions like an XREF or Data dictionary. I could join it on the contents of the JSON in order to specify what the XML should look like.
Using the object XMLWriter, I created the XML, and based on the format that the XML should take, I used the functions WriteStartElement, WriteElementString, WriteAttributeString, WriteEndElement, and WriteEndDocument I wrote the XML.

Related

Azure Insert multiple rows to table from the json

I want to automate the process of bulk inserting a json from API to SQL table. Json will look something like below with array.
[
{
"Id": "1",
"Name": "Orlando",
"Age": "23"
},
{
"Id": "2",
"Name": "Keith",
"Age": "24"
},
{
"Id": "3",
"Name": "Donna",
"Age": "23"
}
]
I will have same columns in table as well Id, Name and Age. I have many json array response from different APIs in this json format with different column names. So I am looking for more generic way of inserting the records to the tables since json node name and column in table are maintained same. I am planning to keep response type to table name mapping in some configurations, So deciding the table to which record need to be inserted is not the problem.
I just want a approach where I can insert a json to table without specifying all the column names etc. I saw few articles which suggests to use OPENJSON where I need specify the column names, If I follow this approach I will end up creating multiple stored procedures for each json to tables. So suggest me a good approach for handling the scenario using Azure logic apps or functions.
There is no silver bullet here. SQL Server is declarative by design and does not support macro substitution. This leaves Dynamic SQL.
Assuming you know the Destination Table and the Columns of the destination AND your JSON is rather simple:
Example or dbFiddle
Declare #JSON varchar(max) = '
[
{
"Id": "1",
"Name": "Orlando",
"Age": "23"
},
{
"Id": "2",
"Name": "Keith",
"Age": "24"
},
{
"Id": "3",
"Name": "Donna",
"Age": "23"
}
]
'
Declare #Dest varchar(max) = 'YourTable'
Declare #Cols varchar(max) = '[Id],[Name],[Age]'
Declare #SQL varchar(max) ='
Insert Into ' +#Dest+' (' +#Cols+ ')
Select '+#Cols+'
From (
Select RN = A.[Key]
,B.[key]
,B.[value]
From OpenJson('''+#JSON+''') A
Cross Apply OpenJson(A.Value) B
) src
Pivot (max(value) for [key] in ( '+#Cols+' ) ) pvt
'
Exec(#SQL)
Results

mySQL JSON Document Store method for inserting data into node 3 levels deep

I want to take the data from here: https://raw.githubusercontent.com/usnistgov/oscal-content/master/examples/ssp/json/ssp-example.json
which I've pulled into a mySQL database called "ssp_models" into a JSON column called 'json_data', and I need add a new 'name' and 'type' entry into the 'parties' node with a new uuid in the same format as the example.
So in my mySQL database table, "ssp_models", I have this entry: Noting that I should be able to write the data by somehow referencing "66c2a1c8-5830-48bd-8fdd-55a1c3a52888" as the record to modify.
All the example I've seen online seem to force me to read out the entire JSON into a variable, make the addition, and then cram it back into the json_data column, which seems costly, especially with large JSON data-sets.
Isn't there a simple way I can say
"INSERT INTO ssp_models JSON_INSERT <somehow burrow down to 'system-security-plan'.metadata.parties (name, type) VALUES ('Raytheon', 'organization') WHERE uuid = '66c2a1c8-5830-48bd-8fdd-55a1c3a52888'
I was looking at this other stackoverflow example for inserting into JSON:
How to create and insert a JSON object using MySQL queries?
However, that's basically useful when you are starting from scratch, vs. needing to add JSON data to data that already exists.
You may want to read https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html and explore each of the functions, and try them out one by one, if you're going to continue working with JSON data in MySQL.
I was able to do what you describe this way:
update ssp_models set json_data = json_array_append(
json_data,
'$."system-security-plan".metadata.parties',
json_object('name', 'Bingo', 'type', 'farmer')
)
where uuid = '66c2a1c8-5830-48bd-8fdd-55a1c3a52888';
Then I checked the data:
mysql> select uuid, json_pretty(json_data) from ssp_models\G
*************************** 1. row ***************************
uuid: 66c2a1c8-5830-48bd-8fdd-55a1c3a52888
json_pretty(json_data): {
"system-security-plan": {
"uuid": "66c2a1c8-5830-48bd-8fdd-55a1c3a52888",
"metadata": {
"roles": [
{
"id": "legal-officer",
"title": "Legal Officer"
}
],
"title": "Enterprise Logging and Auditing System Security Plan",
"parties": [
{
"name": "Enterprise Asset Owners",
"type": "organization",
"uuid": "3b2a5599-cc37-403f-ae36-5708fa804b27"
},
{
"name": "Enterprise Asset Administrators",
"type": "organization",
"uuid": "833ac398-5c9a-4e6b-acba-2a9c11399da0"
},
{
"name": "Bingo",
"type": "farmer"
}
]
}
}
}
I started with data like yours, but for this test, I truncated everything after the parties array.

how to prevent the json data output being alphabetically sorted and stored in mysql work?

Hi I'm trying to insert complex json data in mysql workbench but my json data being inserted alphabetically sorted. How do I get json data with same order which I passed in insert query?
Create table:
CREATE TABLE payload ( `id` INT NOT NULL,
`json` JSON NOT NULL,
PRIMARY KEY (`id`));
Insert Json
INSERT INTO payload ( id, json)
VALUES (2, '{
"request": "release",
"type": [
{
"type" : 1
}
],
"start": [
{
"type": "sample",
"id": "01",
"content": [
{
"name": "jon",
"email": "jon#gmail.com"
}
]
}
]
}');
stored json in database after select * table name:
'3', '{\"type\": [{\"type\": 1}], \"start\": [{\"id\": \"01\", \"type\": \"sample\", \"content\": [{\"name\": \"jon\", \"email\": \"jon#gmail.com\"}]}], \"request\": \"release\"}'
Actually I want to have store my json same as my inserted json in database.
Is there a way to prevent the json data being alphabetically sorted?
Thanks.
Mysql will automatically sort the keys of a JSON object before it's stored.
I had the same problem!
The solution is to change the column type to 'text' and then the order of keys will not be changed!
Use serialize before storing. The conversion will make the array unrecognizable to MySQL so when retrieved and restored with unserialize it will be in the exact same order as before storage.
The drawback is the native MySQL json functions and methods can't be used to search and manipulate the array. Serializing is good only for storing and retrieving.
The only other way is to add a key with a value representing the array's sort order. That drawback is that the array must be sorted to restore it to its original state.

SOLR post json file Default fieldtype

I have POSTAL_CODE field in my json file. If I try importing that data to SOLR using solr/post, the fieldtype is being set as 'plongs' which is not suitable for data like "108-0023". Beacause of that the data import is throwing out an error. Is there any work around for this kind of issues?
Edit:
Sample data which you might use to check it.
{
"id": "1",
"POSTAL_CODE": "1982"
},
{
"id": "2",
"POSTAL_CODE": "1947"
},
{
"id": "3",
"POSTAL_CODE": "19473"
},
{
"id": "4",
"POSTAL_CODE": "19471"
},
{
"id": "5",
"POSTAL_CODE": "1947-123"
}
In the above sample, I don't understand why 'id' is not being considered as 'plongs' or 'pints' but only 'POSTAL_CODE' has that issue. if the first element has POSTAL_CODE as, say "1947-145" then the field type is being taken as 'text_general'. Generally if the value has double quotes, (i.e., "Data": "123") shouldn't it be considered as a string value?
Remove the collection, create it as new and before you index anything, define a field POSTAL_CODE in your schema as type string. This will then index any incoming data on this field without guessing, but instead use the string type, which means it is indexed as-is.
Copied and adapted from https://lucene.apache.org/solr/guide/7_0/schema-api.html, but untested:
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"POSTAL_CODE",
"type":"string",
"stored":true }
}' http://localhost:8983/solr/yourcollectionhere/schema
I tried to import the data by creating a raw json document with the field POSTAL_CODE. Below is my json & my solr version is 7.2.1
{"array": [1,2,3],"boolean": true,"color": "#82b92c","null": null,"number": 123,"POSTAL_CODE": "108-0023"}
It is indexed as Text Field in solr below is the attached screenshot. Command I triggered to index the data is as below:
bin/post -c gettingstarted test.json
Could you please provide the sample data and version of solr on which you are facing this issue.

postgres columns attribute for json

I am currently a happy user of ngx_postgres. However I recently discover I need to do something very weird. Basically I need to produce the following json output:
{
"PatientName": {
"Tag": "00100010",
"VR": "PN",
"PersonName": [
{
"SingleByte": "Wang^XiaoDong",
"Ideographic": "王^小東"
}
]
},
},
{
"PatientName": {
"Tag": "00100010",
"VR": "PN",
"PersonName": [
{
"SingleByte": "John^Doe",
}
]
},
}
With a little reading of the DICOM standard it is easy to create (simplified) a table of equivalent for Keyword, Tag and VR:
CREATE TABLE equiv (
"Keyword" varchar(64) PRIMARY KEY,
"Tag" char(8) NOT NULL,
"VR" char(2) NOT NULL,
);
Well now the tricky part is this indirection with PatientName that I do not understand, I tried:
CREATE TABLE patientname (
"SingleByte" varchar(64) primary key,
"Ideographic" varchar(64),
);
CREATE TABLE patientlevel_impl_detail (
"PatientName" varchar(64) references patientname("SingleByte"),
"PatientID" character varying(64) NOT NULL
);
CREATE view patientlist as select
patientname."SingleByte",
patientname."Ideographic",
patientname."Phonetic",
patientlevel_impl_detail."PatientID"
FROM patientlevel_impl_detail,patientname where patientlevel_impl_detail."PatientName" = patientname."SingleByte";
However in any case a TABLE and/or a VIEW is always flatten, and instead I am getting something like:
$ curl http://localhost:8080/patients
[
{
"Tag": "00100010",
"VR": "PN",
"SingleByte": "John^Doe",
"Ideographic": null,
},
]
So I do not see how I can make PersonName an array of nested string (nested json tree).
Note: I am not using 9.3, I need to use 9.1 for now
On 9.1, do yourself a favor and get the json extension for 9.1. It will save you a lot of work.
The second thing you need to do is to create a nested data structure as a view which matches your json structure. You will use array_agg() for this:
CREATE view patientlist as select
arrayagg(patientname) as "PatientName"
patientlevel_impl_detail."PatientID"
FROM patientlevel_impl_detail,patientname
where patientlevel_impl_detail."PatientName" = patientname."SingleByte";
Then you should be able to:
SELECT row_to_json(patientlist) FROM patientlist;