Using ZappySys in SSIS to import JSON data from a 3rd party.
All working well, but sometimes the 3rd party doesn't include the same elements each time. Samples below:
The first works as it includes the expected Equipment and Catering cost elements that I have supplied in the Sample JSON string section of the "ZS JSON Parser Transform" screen
{
"RoomCost": {
"Net": 150,
"Tax": 0,
"Gross": 150
},
"CateringCost": {
"Net": 187.2,
"Tax": 0,
"Gross": 187.2
},
"EquipmentCost": {
"Net": 0,
"Tax": 0,
"Gross": 0
},
"Discount": {
"Net": 0,
"Tax": 0,
"Gross": 0
},
"Total": {
"Net": 337.2,
"Tax": 0,
"Gross": 337.2
}
}
The example below fails to import with NULL errors in the logs as the Equipment and Catering elements are not supplied. Error is
{
"RoomCost": {
"Net": 150,
"Tax": 0,
"Gross": 150
},
"Discount": {
"Net": 0,
"Tax": 0,
"Gross": 0
},
"Total": {
"Net": 337.2,
"Tax": 0,
"Gross": 337.2
}
}
Error: The type of the value (DBNull) being assigned to variable "User::mCateringCostGross" differs from the current variable type (Decimal). Variables may not change type during execution. Variable types are strict, except for variables of type Object.
What is the best way to handle this situation?
The fix for this was to include a Derived Column task between the ZappySys JSON source and the output, to convert nulls to default values.
Related
I have a JSON column (called "roi") which contains users' Instagram performance. This is the roi column:
{
"data": {
"campaignName": "Master Cosy",
"currency": "GBP",
"reportData": {
"AAAAAAAAAA": {
"id": "0f20d833-d0f-bdb7-19",
"name": "cornish_gregorys",
"thumbnail": "https://sstagram.com/v/t51.2885-19/s320x320/87244862_1017848048596",
"Name": "cornisorys",
"instagramCount": 2319,
"instagramEngagementFactor": 0,
"instagramAuthorised": true,
"hasPosts": true,
"budget": 0,
"derivedFee": 0,
"inventoryItems": [],
"trackedAssetsStats": {
"totalAssets": 9,
"facebook": {
"count": 0
},
"instagram": {
"total": 9,
"stories": 9,
"carousels": 0,
"videos": 0,
"images": 0,
"igtvs": 0
},
"BBBBBBBBBBBBB": {
"id": "d3d30db4-0b453dfc3ae2a09",
"name": "itssdha",
"thumbnail": "https://in9809609728_n.jpg?_nc_ht=instagram.fhel5-1.fna.fbcdn.net&_nc_ohc=Se3ySAoqnFwAX4f6&oeF1623",
"Name": "itsshdha",
"instagramCount": 26700,
"instagramEngagementFactor": 0,
"instagramAuthorised": true,
"hasPosts": true,
"budget": 0,
"derivedFee": 0,
"inventoryItems": [],
"trackedAssetsStats": {
"totalAssets": 5,
"facebook": {
"count": 0
},
"instagram": {
"total": 9,
"stories": 9,
"carousels": 0,
"videos": 0,
"images": 0,
"igtvs": 0}, etc.....
After "reportData" I have the specific names of the users (in this case AAAAAAAA and BBBBBBBBB) and within them the performance of their Instagram accounts. How can I access all the metrics within the object username without having to type the specific username (AAAAAAAA and BBBBBBBBB)
My query is this:
roi -> 'date' -> 'reportData' -> 'AAAAAAA' -> 'instagramCount' -> etc ....
But I need something to 'jump' this part -> 'AAAAAAA' -> and go straight to the metrics, in this case 'instagramCount', etc...
From what I have read I may need to use jsonb_each, does anyone know how to use it?
demos:db<>fiddle
You have several ways.
Use jsonb_each() to expand all users' data: You can create a record per user and than ask for the count afterwards
SELECT
users.value -> 'instagramCount'
FROM
mytable,
jsonb_each(mydata -> 'data' -> 'reportData') as users
Since Postgres 12 you can use JSONpath for that, to achieve the same:
SELECT
jsonb_path_query(mydata, '$.**.instagramCount')
FROM mytable
I have a JSON list with 253 entries and 7 records where the 7th record is a list with usually 2 entries.
I'm trying to convert this in Power Query to give me a table output. I've only really ever used the basic connection in Excel to do this automatically without problems.
My current error is:
Expression.Error: We cannot convert a value of type Record to type Text.
Details:
Value=Record
Type=Type
I'm not sure where to go and I can't see examples for my specific situation, although I'm sure there are plenty of examples - just that I'm not good enough with this to understans
[
{
"Id": "lorum-ipsum1",
"Description": "sitename 1",
"Latitude": 1.0,
"Longitude": -1.0,
"Postcode": "AB1 2CD",
"CountryCode": "GB",
"Connectors": [
{
"Id": "lorum-ipsum1-a",
"Number": 1,
"Status": 1,
"ErrorCode": "NoError",
"ChargepointName": "GP00000",
"Shape": 0,
"Mode": 0,
"Volts": 240,
"Amps": 0,
"Phase": 2,
"PricingInformationUrl": "hrefhere",
"UsageRestrictions": []
},
{
"Id": "lorum-ipsum1-b",
"Number": 2,
"Status": 1,
"ErrorCode": "NoError",
"ChargepointName": "GP0000",
"Shape": 0,
"Mode": 0,
"Volts": 240,
"Amps": 0,
"Phase": 2,
"PricingInformationUrl": "hrefhere",
"UsageRestrictions": []
}
]
},
{
"Id": "lorum-ipsum2",
"Description": "sitename 2",
"Latitude": 1.0,
"Longitude": -1.0,
"Postcode": "AB1 2CD",
"CountryCode": "GB",
"Connectors": [
{
"Id": "lorum-ipsum2-a",
"Number": 1,
"Status": 1,
"ErrorCode": "NoError",
"ChargepointName": "GP00000",
"Shape": 0,
"Mode": 0,
"Volts": 240,
"Amps": 0,
"Phase": 2,
"PricingInformationUrl": "hrefhere",
"UsageRestrictions": []
},
{
"Id": "lorum-ipsum2-b",
"Number": 2,
"Status": 1,
"ErrorCode": "NoError",
"ChargepointName": "GP0000",
"Shape": 0,
"Mode": 0,
"Volts": 240,
"Amps": 0,
"Phase": 2,
"PricingInformationUrl": "hrefhere",
"UsageRestrictions": []
}
]
},
I think you want to use Table.FromRecords function to transform the JSON into a table. However, since the Connectors field of each record is itself a list of records, I think you will need to call it a second time.
To give you an example:
let
serialised = "[{""Id"":""lorum-ipsum1"",""Description"":""sitename 1"",""Latitude"":1,""Longitude"":-1,""Postcode"":""AB1 2CD"",""CountryCode"":""GB"",""Connectors"":[{""Id"":""lorum-ipsum1-a"",""Number"":1,""Status"":1,""ErrorCode"":""NoError"",""ChargepointName"":""GP00000"",""Shape"":0,""Mode"":0,""Volts"":240,""Amps"":0,""Phase"":2,""PricingInformationUrl"":""hrefhere"",""UsageRestrictions"":[]},{""Id"":""lorum-ipsum1-b"",""Number"":2,""Status"":1,""ErrorCode"":""NoError"",""ChargepointName"":""GP0000"",""Shape"":0,""Mode"":0,""Volts"":240,""Amps"":0,""Phase"":2,""PricingInformationUrl"":""hrefhere"",""UsageRestrictions"":[]}]},{""Id"":""lorum-ipsum2"",""Description"":""sitename 2"",""Latitude"":1,""Longitude"":-1,""Postcode"":""AB1 2CD"",""CountryCode"":""GB"",""Connectors"":[{""Id"":""lorum-ipsum2-a"",""Number"":1,""Status"":1,""ErrorCode"":""NoError"",""ChargepointName"":""GP00000"",""Shape"":0,""Mode"":0,""Volts"":240,""Amps"":0,""Phase"":2,""PricingInformationUrl"":""hrefhere"",""UsageRestrictions"":[]},{""Id"":""lorum-ipsum2-b"",""Number"":2,""Status"":1,""ErrorCode"":""NoError"",""ChargepointName"":""GP0000"",""Shape"":0,""Mode"":0,""Volts"":240,""Amps"":0,""Phase"":2,""PricingInformationUrl"":""hrefhere"",""UsageRestrictions"":[]}]}]",
deserialised = Json.Document(serialised),
toTable = Table.FromRecords(deserialised),
transformConnectors = Table.TransformColumns(toTable, {{"Connectors", Table.FromRecords}})
in
transformConnectors
which gives me:
From there, you can explore/continue the rest of the transformation yourself (since you haven't specified what you want).
You can expand some/all nested columns inside of the Connectors column by either clicking the icon highlighted in the image above -- or writing any necessary M code.
Now I am using vibed JSON module, but I do not know how to get access to nested elements without iteration it with foreach.
Here is my JSON:
{
"hasMore": false,
"result": [{
"ip": "127.0.0.1",
"passedtests": "[firsttest8,firsttest8,firsttest8,firsttest8]",
"guid": ""
}],
"code": 201,
"extra": {
"stats": {
"writesIgnored": 0,
"scannedIndex": 0,
"scannedFull": 1,
"executionTime": 0,
"filtered": 0,
"writesExecuted": 0
},
"warnings": []
},
"error": false,
"cached": false
}
I would like to do something like: result.passedtests. But here result is an array.
I found perfect solution:
Json resultPassedtestsJson = visitorsInfo["result"][0]["passedtests"]; // "[firsttest8,firsttest8,firsttest8,firsttest8]"
[0] is because result is array, and we access to it's first element.
How do I approach writing a query to return all the records matching both match.id and player.name for the following collection?
{
"match": {
"id": 1,
"event": {
"timestamp": "2015-06-03 15:02:22",
"event": "round_stats",
"round": 1,
"player": {
"name": "Jim",
"userId": 45,
"uniqueId": "BOT",
"team": 2
},
"shots": 0,
"hits": 0,
"kills": 0,
"headshots": 0,
"tks": 0,
"damage": 0,
"assists": 0,
"assists_tk": 0,
"deaths": 0,
"head": 0,
"chest": 0,
"stomach": 0,
"leftArm": 0,
"rightArm": 0,
"leftLeg": 0,
"rightLeg": 0,
"generic": 0
}
}
}
I've attempted it with both the following query statements, but had no luck -- they both return no results:
db.warmod_events.find( { $and: [ { "match.id": 1}, { "player.name": 'Jim' } ] } )
db.warmod_events.find( { $and: [ { "match.id": 1}, { "event": { "player.name": "Jim" } } ] } )
I'm pretty new to Mongo and any guidance and explanation would help a bunch -- truthfully I've chosen to use Mongo for this project as the data I am working with is already presented in this form (the JSON) and, due to that, it seemed like a good opportunity to use and learn Mongo.
I am referring to the documentation on the Mongo site currently.
Thanks all
Try the following query:
db.warmod_events.find({ "match.id": 1, "match.event.player.name": 'Jim' })
which will match documents where the match id is the same as the embedded document player name.
I'm working with an API that is sending JSON responses like this:
[
- {
"id": 46843639,
"nickname": "JSON",
"registration_date": "2004-03-07T23:16:20.000-04:00",
"country_id": "BR",
"user_type": "normal",
"logo": null,
"points": 91,
"site_id": "JSN",
"permalink": "http://xxxxxx",
"seller_reputation": - {
"level_id": "2_orange",
"power_seller_status": null,
"transactions": - {
"period": "historic",
"total": 14,
"completed": 8,
"canceled": 6,
"ratings": - {
"positive": 1,
"negative": 0,
"neutral": 0,
},
},
},
"status": - {
"site_status": "active",
},
},
]
I couldn't find that minus '-' sign in any specification documents. Is this JSON standard?
No, it's not. The server is sending you broken JSON.
The only times - is valid are when it's in a string, when it immediately precedes a number, or when it immediately follows the e or E in a floating-point number.
Add to that, commas should separate name/value pairs and array elements, not follow them. Those commas following the last entry in each object / array, are invalid as well.