Parse XML to JSON Elixir - json

anyone knows How to convert xml into json. I tried Sweetyxml but its not converting xml into json.
Using this gist
https://gist.github.com/spint/40717d4e6912d8cea929
Reading json
{:ok, xmldoc} = File.read(Path.expand("/Users/sohaibanwar/Desktop/unconfirmed_order.xml"))
{doc, []} = xmldoc |> to_charlist() |> :xmerl_scan.string()
After parsing (in screenshot), but not getting right answer
YML FILE I am using
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<order>
<orderId>35267684</orderId>
<orderToken>fa74171d-f54a-4e76-bcf2-2dd284ac6450</orderToken>
<brokerTicketId>169855177</brokerTicketId>
<section>SEC1</section>
<row>N</row>
<seats/>
<notes>Limited view. Please note that you will need to use an iOS or Android mobile device to gain entry to your event.</notes>
<quantity>2</quantity>
<cost>324.00</cost>
<event>Dave Matthews Band (Rescheduled from 7/17/2020, 7/16/2021)</event>
<eventDate>2021-08-20 19:30:00</eventDate>
<orderDate>2021-06-18 05:43:13</orderDate>
<expectedShipDate>2021-06-18 00:00:00</expectedShipDate>
<venue>Xfinity Center - MA - Mansfield, MA</venue>
<status>UNCONFIRMED</status>
<purchaseOrderId>35088971</purchaseOrderId>
<electronicDelivery>false</electronicDelivery>
<passThrough></passThrough>
<listingId>3359267717</listingId>
<productionId>3412459</productionId>
<eventId>218</eventId>
<zone>false</zone>
<barCodesRequired>false</barCodesRequired>
<transferViaURL>true</transferViaURL>
<instantTransfer>false</instantTransfer>
<instantFlashSeats>false</instantFlashSeats>
</order>
Required Results
You can paster the XML here to get the required answer
{
"order": {
"orderId": 35267684,
"orderToken": "fa74171d-f54a-4e76-bcf2-2dd284ac6450",
"brokerTicketId": 169855177,
"section": "SEC1",
"row": "N",
"seats": "",
"notes": "Limited view. Please note that you will need to use an iOS or Android mobile device to gain entry to your event.",
"quantity": 2,
"cost": 324,
"event": "Dave Matthews Band (Rescheduled from 7/17/2020, 7/16/2021)",
"eventDate": "2021-08-20 19:30:00",
"orderDate": "2021-06-18 05:43:13",
"expectedShipDate": "2021-06-18 00:00:00",
"venue": "Xfinity Center - MA - Mansfield, MA",
"status": "UNCONFIRMED",
"purchaseOrderId": 35088971,
"electronicDelivery": false,
"passThrough": "",
"listingId": 3359267717,
"productionId": 3412459,
"eventId": 218,
"zone": false,
"barCodesRequired": false,
"transferViaURL": true,
"instantTransfer": false,
"instantFlashSeats": false
}
}

Well, if your question is How to convert xml to json in Elixir the answer is fairly simple: use https://github.com/homanchou/elixir-xml-to-map and JSON encoder of your choice:
def xml2json(path) do
File.read!(path)
|> XmlToMap.naive_map()
|> Jason.encode!()
end
The problem with this answer is that XML is not isomorphic to map (you have both tags and attributes in XML and you don't in map or JSON), so the better way will be to use SweetXML (or :xmerl) and do xmap with a proper match like here (code is from SweetXML examples - https://github.com/kbrw/sweet_xml#examples):
result = doc |> xmap(
matchups: [
~x"//matchups/matchup"l,
name: ~x"./name/text()",
winner: [
~x".//team/id[.=ancestor::matchup/#winner-id]/..",
name: ~x"./name/text()"
]
],
last_matchup: [
~x"//matchups/matchup[last()]",
name: ~x"./name/text()",
winner: [
~x".//team/id[.=ancestor::matchup/#winner-id]/..",
name: ~x"./name/text()"
]
]
)
assert result == %{
matchups: [
%{name: 'Match One', winner: %{name: 'Team One'}},
%{name: 'Match Two', winner: %{name: 'Team Two'}},
%{name: 'Match Three', winner: %{name: 'Team One'}}
],
last_matchup: %{name: 'Match Three', winner: %{name: 'Team One'}}
}
Another option is to use https://github.com/willemdj/erlsom and do manual travers over tuple tree it emits from the simple_form call. Note you will have to handle XMLNS and attr/value problem anyways.

Related

Compare two nested json files and show user where exactly the change has occurred and which json file using Python?

I have two json files. I am validating the response is same or different. I need to show the user where there is an exact change. Some what like the particular key is added or removed or changed in this file.
file1.json
[
{
"Name": "Jack",
"region": "USA",
"tags": [
{
"name": "Name",
"value": "Assistant"
}
]
},
{
"Name": "MATHEW",
"region": "USA",
"tags": [
{
"name": "Name",
"value": "Worker"
}
]
}
]
file2.json
[
{
"Name": "Jack",
"region": "USA",
"tags": [
{
"name": "Name",
"value": "Manager"
}
]
},
{
"Name": "MATHEW",
"region": "US",
"tags": [
{
"name": "Name",
"value": "Assistant"
}
]
}
]
If you see Two JSON you can find the difference as a region in file2.json has changed US and Values changed from manager to assistant and worker. Now I want to show the user that file2.json has some changes like region :US and Manager changed to Assistant.
I have used deepdiff for validating purpose.
from deepdiff import DeepDiff
def difference(oldurl_resp,newurl_resp,file1):
ddiff = DeepDiff(oldurl_resp, newurl_resp,ignore_order=True)
if(ddiff == {}):
print("BOTH JSON FILES MATCH !!!")
return True
else:
print("FAILURE")
output = ddiff
if(output.keys().__contains__('iterable_item_added')):
test = output.get('iterable_item_added')
print('The Resource name are->')
i=[]
for k in test:
print("Name: ",test[k]['Name'])
print("Region: ",test[k]['region'])
msg= (" Name ->"+ test[k]['Name'] +" Region:"+test[k]['region'] +". ")
i.append(msg)
raise JsonCompareError("The json file has KEYS changed!. Please validate for below" +str(i) +"in "+file1)
elif(output.keys().__contains__('iterable_item_removed')):
test2 = output.get('iterable_item_removed')
print('The name are->')
i=[]
for k in test2:
print(test2[k]['Name'])
print(test2[k]['region'])
msg= (" Resource Name ->"+ test2[k]['Name'] +" Region:"+test2[k]['region'] +". ")
i.append(msg)
raise JsonCompareError("The json file has Keys Removed!!. Please validate for below" +str(i)+"in "+file1)
This code just shows the resource Name I want to show the tags also which got changed and added or removed.
Can anybody guide me
If you just print out the values of "test" variables, you will find out that "tag" variable changes are inside of it, test value of test in this example will be:
test = {'root[0]': {'region': 'USA', 'Name': 'Jack', 'tags': [{'name': 'Name', 'value': 'Manager'}]}, 'root[1]': {'region': 'US', 'Name': 'MATHEW', 'tags': [{'name': 'Name', 'value': 'Assistant'}]}}
and you can print test[k]['tags'] or add it your "msg" variable.
Suggestion:
Also, if your data has some primary key (for example they have "id", or their order is always fixed), you can compare their data 1 by 1 (instead of comparing whole lists) and you can have a better comparison. For example if you compare data of "Jack" together, you will have the following comparison:
{'iterable_item_removed': {"root['tags'][0]": {'name': 'Name', 'value': 'Assistant'}}, 'iterable_item_added': {"root['tags'][0]": {'name': 'Name', 'value': 'Manager'}}}
You should try the deepdiff library. It gives you the key where the difference occurs and the old and new value.
from deepdiff import DeepDiff
ddiff = DeepDiff(json_object1, json_object2)
# if you want to compare by ignoring order
ddiff = DeepDiff(json_object1, json_object2, ignore_order=True)

Trying to turn json array into a field issue

I have the following JSON which parses OK:
It is just some term code data that I want to send.
[{
"TermCode": 2164,
"ACAD_LEVEL_BOT": "GR",
"ACAD_LEVEL_EOT": "GR",
"ACAD_CAREER": "GRAD",
"UA_PRIM_MAJ_PLN": "DPMDP",
"UA_DEGR_DT": "1900-01-01",
"UA_PRIM_MJ_PLN_DES": "Development Practice",
"UA_PRIM_MIN_PLN_D": "-",
"CUM_GPA": 3.707,
"CUR_GPA": 4,
"TOT_CUMULATIVE": 300,
"UNT_PASSD_PRGRSS": 10,
"UNT_TAKEN_PRGRSS": 10,
"UA_PRIM_MJ_PLN_OWN": "School of Anthropology",
"ADMIT_TYPE": "GRD",
"ADMIT_TERM": "2144",
"UA_FA_ST_RES_DESCR": "Resident",
"VISA_PERMIT_TYPE": "-",
"UA_SALT_ROSTER": "-",
"UA_SALT_STATUS": "-",
"DEGR_CHKOUT_STAT": "-",
"EXP_GRAD_TERM": "-",
"UNT_TRNSFR": 0,
"ACAD_PROG_LD": "Graduate Degree Seeking",
"TOT_TRNSFR": 0,
"TUITION_RES": "RES",
"UA_CITIZEN_COUNTRY": "United States",
"UA_CITIZEN_ST_DESC": "Citizen",
"ACADEMIC_LOAD": "F",
"UNT_AUDIT": 0,
"CAMPUS": "MAIN",
"ACADEMIC_YEAR": "2016-2017",
"UA_DEGREE_LEVEL": "Masters",
"ACAD_PROG": "GDEG",
"UA_SEC_MAJ_PLN": "-",
"ACAD_ORG": "0410",
"EFF_START_DT": "2016-12-18 20:37:08",
"CURRENT_IND": "N",
"PROG_STATUS": "AC",
"PROG_ACTION": "MATR"
}]
I want to use this array as a field named term data so I try:
[{
termdata: [{
"TermCode": 2164,
"ACAD_LEVEL_BOT": "GR",
"ACAD_LEVEL_EOT": "GR",
"ACAD_CAREER": "GRAD",
"UA_PRIM_MAJ_PLN": "DPMDP",
"UA_DEGR_DT": "1900-01-01",
"UA_PRIM_MJ_PLN_DES": "Development Practice",
"UA_PRIM_MIN_PLN_D": "-",
"CUM_GPA": 3.707,
"CUR_GPA": 4,
"TOT_CUMULATIVE": 300,
"UNT_PASSD_PRGRSS": 10,
"UNT_TAKEN_PRGRSS": 10,
"UA_PRIM_MJ_PLN_OWN": "School of Anthropology",
"ADMIT_TYPE": "GRD",
"ADMIT_TERM": "2144",
"UA_FA_ST_RES_DESCR": "Resident",
"VISA_PERMIT_TYPE": "-",
"UA_SALT_ROSTER": "-",
"UA_SALT_STATUS": "-",
"DEGR_CHKOUT_STAT": "-",
"EXP_GRAD_TERM": "-",
"UNT_TRNSFR": 0,
"ACAD_PROG_LD": "Graduate Degree Seeking",
"TOT_TRNSFR": 0,
"TUITION_RES": "RES",
"UA_CITIZEN_COUNTRY": "United States",
"UA_CITIZEN_ST_DESC": "Citizen",
"ACADEMIC_LOAD": "F",
"UNT_AUDIT": 0,
"CAMPUS": "MAIN",
"ACADEMIC_YEAR": "2016-2017",
"UA_DEGREE_LEVEL": "Masters",
"ACAD_PROG": "GDEG",
"UA_SEC_MAJ_PLN": "-",
"ACAD_ORG": "0410",
"EFF_START_DT": "2016-12-18 20:37:08",
"CURRENT_IND": "N",
"PROG_STATUS": "AC",
"PROG_ACTION": "MATR"
}] }]
However, this doesn't parse. What is wrong with my syntax?
There is an error in Stack Overflow that is not letting me end my question here so I have to add more description please down down arrow me I didn't have a choice. So my next step would be to have json such as:
[{ termdata:jsonarray,coursedata:jsonarray,admitdata:jsonarray}]
Any thoughts on that would be appreciated also.
To debug small snippets such as yours, you could use https://jsonlint.com,
which gives helpful error messages making it easy to pinpoint the problem.
As an alternative that also works for large files, you might like to consider jq. With your snippet in the file "input.txt", the invocation:
jq empty input.txt
produces the error message:
parse error: Invalid literal at line 2, column 9
This amounts to saying that the first key was not properly quoted.
(Here, empty has the effect of suppressing normal output.)
Let jq do the walking
Better yet, you can save yourself the hassle by letting jq do the work:
jq '[{termdata:.}]' original.json
This will produce the desired output. Unless specifically instructed otherwise, jq takes JSON (or a JSON stream) and produces JSON (or a JSON stream).
You could also use jq to do more sophisticated transformations with multiple input files.

Unexpected symbol: COMMA error from json file

I'm using Talend ETL Tool and extracting data from json files and storing them in Mysql database.
But I get the error while reading in very first json. For reading json I'm using tExtractJSONFileds component.
I'm sure about the configuation set up in talend etl tool its right. I believe there is some problem in json file.
While extracting the component shows error like this
Exception in component tExtractJSONFields_1
javax.xml.stream.XMLStreamException: java.io.IOException: Unexpected symbol: COMMA
at de.odysseus.staxon.base.AbstractXMLStreamReader.initialize(AbstractXMLStreamReader.java:218)
at de.odysseus.staxon.json.JsonXMLStreamReader.<init>(JsonXMLStreamReader.java:65)
at de.odysseus.staxon.json.JsonXMLInputFactory.createXMLStreamReader(JsonXMLInputFactory.java:148)
at de.odysseus.staxon.json.JsonXMLInputFactory.createXMLStreamReader(JsonXMLInputFactory.java:44)
at de.odysseus.staxon.base.AbstractXMLInputFactory.createXMLEventReader(AbstractXMLInputFactory.java:118)
I dont know how to deal with JSONs, So Acc to this error can anyone help me where could be the error in JSON file ?
Is there any value passed as NULL or something else ?
Sample JSON
[
[, {
"tstamp": "123456",
"event": "tgegfght",
"is_duplicate": false,
"farm": "dyhetygdht",
"uid": "tutyvbrtyvtrvy",
"clientip": "52351365136",
"device_os_label": "MICROSOFT_WINDOWS_7",
"device_browser_label": "MOZILLA_FIREFOX",
"geo_country_code": "MA",
"geo_region_code": "55",
"geo_city_name_normalized": "agadir",
"referer": "www.abc.com",
"txn": "etvevv5r",
"txn_isnew": true,
"publisher_id": 126,
"adspot_id": 11179502,
"ad_spot": 5188,
"format_id": 1611,
"misc": {
"PUBLISHER_FOLDER": "retvrect",
"NO_PROMO": "rctrctrc",
"SECTION": "evtrevr",
"U_COMMON_ALLOW": "0",
"U_Auth": "0"
},
"handler": "uint"
}, , ]
Thanks in advance !!
You have extra empty commas in your sample json.
Your Sample Json should look like
[{
"tstamp": "123456",
"event": "tgegfght",
"is_duplicate": false,
"farm": "dyhetygdht",
"uid": "tutyvbrtyvtrvy",
"clientip": "52351365136",
"device_os_label": "MICROSOFT_WINDOWS_7",
"device_browser_label": "MOZILLA_FIREFOX",
"geo_country_code": "MA",
"geo_region_code": "55",
"geo_city_name_normalized": "agadir",
"referer": "www.abc.com",
"txn": "etvevv5r",
"txn_isnew": true,
"publisher_id": 126,
"adspot_id": 11179502,
"ad_spot": 5188,
"format_id": 1611,
"misc": {
"PUBLISHER_FOLDER": "retvrect",
"NO_PROMO": "rctrctrc",
"SECTION": "evtrevr",
"U_COMMON_ALLOW": "0",
"U_Auth": "0"
},
"handler": "uint"
}]
OR
[
{
"somethinghere": "its value"
},
"somethingelse": "its value"
]
Your sample json is not valid json, due to the spurious extra commas on the second and last lines. Json only allows commas BETWEEN elements of a vector or object, and empty elements are not allowed.

Parsing Different Json Objects in WP8

Please help me in parsing this Json sample as I'm not able to parse it because of the complexity of it as well as different objects inside it. I'm able to parse Json when a list of same objects & same structure but not like the one below.
[
{
"notificationBrowserHead":
{
"notificationId": 4,
"notificationType": "NEW_PRODUCT",
"creationTime": 1421933381000,
"notificationNormalUserId": 4,
"notificationViewed": false
},
"brandIdAndNameHolder":
{
"brandId": 1,
"name": "B1"
},
"brandLogo": null,
"productIdAndNameHolder":
{
"productId": 1,
"name": "JK product1"
}
},
{
"notificationBrowserHead":
{
"notificationId": 2,
"notificationType": "USER_INT_COMMENT",
"creationTime": 1421924403000,
"notificationNormalUserId": 2,
"notificationViewed": false
},
"uploadId": 22,
"uploadThumbnail": "/mediaUrl/location/thumbNail",
"uploadDescription": "upload 1 location desc",
"notificationCreator":
{
"normalUserId": 90,
"displayName": "amit"
},
"uploadRemoved": false
},
{
"notificationBrowserHead":
{
"notificationId": 1,
"notificationType": "NEW_LOCATION_VOTE",
"creationTime": 1421924403000,
"notificationNormalUserId": 1,
"notificationViewed": false
},
"locationIdAndNameHolder":
{
"locationId": 11,
"name": "Current King JK"
},
"locationLogo": null
}
]
Any help would be truly appreciated.
I presume that you receive different set of json properties when your NotificationType varies.
Solution 1:
Define all your members(the collection of all your properties that you receive for different types of notification) in a Class and use it for DeSerialization, so that the unwanted properties for your particular notification type will be null.
Solution 2:
Parser manually. Newtonsoft json documentation here
Make class "Notifications (or something)" and put inside everything you got back from json2csharp.com site, then use this framework http://www.newtonsoft.com/json to deserialize data as you download it from server and you should be able to get notificationType by Object.Notificationbrowserhead[x].notificationType or similar.

Smaller JSON Format

I'm going to develop a pushing server (HTML5 WebSocket/Polling) and in order to reduce the size of packets (that presents in JSON format) I want to do something like this with packets:
[["id", "username", "password"], [1, "afshin", "123"], [2, "barak", "meme"]]
Instead of clear JSON format like:
[{"id": 1, "username": "afshin", "password": "123"}, {"id": 2, "username": "barak", "password": "meme"}]
Exactly, I want to prevent sending contract properties in each object.
So, I want to know is there any library for doing this (or something like)? I have C# on server and JavaScript on clients.
JSON DB or RJSON should be exactly what you're looking for. You'll most likely have to implement serializers/deserializers yourself (RJSON is already implemented in JS though).
As for compressing pure JSON, I think you could bypass the "keys are needed" rule by wrapping all your data in a single object entry:
{"data" : [["id", "username", "password"], [1, "afshin", "123"], [2, "barak", "meme"]]}
So, besides all the arguments against manual compression, this would be a solution:
var input = [{"id": 1, "username": "afshin", "password": "123"}, {"id": 2, "username": "barak", "password": "meme"}];
var keys = {}
input.map ( function (e) { Object.keys(e).map( function (k) { keys[k] = 1; })});
var output = [ Object.keys(keys) ] .concat( input.map( function (e) {
return Object.keys(keys).map( function (k) { return e[k]; } );
} ) );
console.log(output);
and Node.js produces:
[ [ 'id', 'username', 'password' ],
[ 1, 'afshin', '123' ],
[ 2, 'barak', 'meme' ] ]
I really don't know if this works with every browser etc.
By removing the name of the name-value pair you'd be breaking a JSON syntax rule. Effectively, it wouldn't be JSON. You also might cause problems for JSON client deserialization. Could you consider reducing the length of your names:
[{"id": 1, "u": "afshin", "p": "123"}, {"id": 2, "u": "barak", "p": "meme"}]
This JSON document is the same size as the one you propose above.