Snowflake - Get value of key in an object within another object - json

I'm working with Snowflake and loading json files from a Staging environment to an ODS environment.
The structure of the json data looks like this:
{"address": "921 Pearl St",
"attributes": {
"Alcohol": "'beer_and_wine'",
"Ambience": "{'touristy': False, 'hipster': False, 'romantic': False, 'divey': False, 'intimate': False, 'trendy': False, 'upscale': False, 'classy': False, 'casual': True}",
"BikeParking": "True",
"BusinessAcceptsBitcoin": "False",
"BusinessAcceptsCreditCards": "True",
"BusinessParking": "{'garage': False, 'street': True, 'validated': False, 'lot': False, 'valet': False}"
}
Now I'm trying to query my data in Snowflake like this:
SELECT json_data:attributes.BusinessParking.garage
FROM my_json_table;
but all I get are null values.
Any idea how do I get to the bottom levels in the "BusinessParking" field?
thanks!!!

Trying to parse the JSON in online tool I got:
Error: Parse error on line 9:
..., 'valet': False}"}
----------------------^
Expecting '}', ',', got 'EOF'
You could check if it is correct by running TRY_PARSE_JSON:
SELECT TRY_PARSE_JSON(json_data) from my_json_table

Firstly your object is missing a closing }
So if I rewrite you "JSON" to be valid then the select form you use works:
select parse_json('{
"address": "921 Pearl St",
"attributes": {
"Alcohol": "beer_and_wine",
"Ambience": {
"touristy": False,
"hipster": False,
"romantic": False,
"divey": False,
"intimate": False,
"trendy": False,
"upscale": False,
"classy": False,
"casual": True
},
"BikeParking": True,
"BusinessAcceptsBitcoin": "False",
"BusinessAcceptsCreditCards": "True",
"BusinessParking": {
"garage": False,
"street": True,
"validated": False,
"lot": False,
"valet": False
}
}
}') as json_data,
json_data:attributes.BusinessParking.garage,
json_data:attributes:BusinessParking:garage
;
but as that implies your JSON is not valid.
looking at your Ambience it a stringfied JSON sub-object, not an object, thus the embedded ' inside it. So escaping your '`' in the sub strings, to it parses correctly we will need to PARSE_JSON the sub-objects, thus:
select parse_json('{"address": "921 Pearl St",
"attributes": {
"Alcohol": "\'beer_and_wine\'",
"Ambience": "{\'touristy\': False, \'hipster\': False, \'romantic\': False, \'divey\': False, \'intimate\': False, \'trendy\': False, \'upscale\': False, \'classy\': False, \'casual\': True}",
"BikeParking": "True",
"BusinessAcceptsBitcoin": "False",
"BusinessAcceptsCreditCards": "True",
"BusinessParking": "{\'garage\': False, \'street\': True, \'validated\': False, \'lot\': False, \'valet\': False}"
}}') as json_data,
PARSE_JSON(json_data:attributes.BusinessParking):garage
gives the answer false that you are looking for.
EDIT
to add more details, you object has json objects nestsed which are Stringify'ed
to put replicate that the above (and below) SQL uses escape tokens \ to allow single quotes into the data. But after that we can select json_data:attributes.BusinessParking and see it presented like you have it shown in your example, which mean if we then PARSE_JSON on that, we get a valid JSON object again, which we can then access via :garage
select parse_json('{"address": "921 Pearl St",
"attributes": {
"Alcohol": "\'beer_and_wine\'",
"Ambience": "{\'touristy\': False, \'hipster\': False, \'romantic\': False, \'divey\': False, \'intimate\': False, \'trendy\': False, \'upscale\': False, \'classy\': False, \'casual\': True}",
"BikeParking": "True",
"BusinessAcceptsBitcoin": "False",
"BusinessAcceptsCreditCards": "True",
"BusinessParking": "{\'garage\': False, \'street\': True, \'validated\': False, \'lot\': False, \'valet\': False}"
}}') as json_data,
json_data:attributes.BusinessParking as it_looks_the_same,
PARSE_JSON(it_looks_the_same) as valid_json,
valid_json:garage
which gives:

After a lot of research I'm able to query the fields the way I needed to.
I found out that I need to use the "try_parse_json" functions with those fields as they are defined as strings not actual json objects.
After using the function I can parse specific fields just like this:
SELECT
TRY_PARSE_JSON(
jsondata: attributes.BusinessParking
): garage AS garage
FROM
"RESTAURANT"."STAGING"."JSON_TABLE_BUSINESS"
Snowflake's doc isn't that great but somehow it explains what you can do: https://docs.snowflake.com/en/sql-reference/functions/try_parse_json.html
This is another way you could transform string values to columns in case you need it: https://sqlkover.com/cool-stuff-in-snowflake-part-3-split-and-flatten/
Thanks for all the comments #simeon-pilgrim!

Related

Graph API - messages - invalid Json produced by API for eventMessageRequest

See attached screenshot, when processing the messages in a user's folder, when the message is an "eventMessageRequest" then the json is invalid: it contains the "type" key twice and the Json parsers can't deal with it.
screenshot of original json received from MS Graph
abbreviated json: note the "type" key used twice:
[
{
"type": "#microsoft.graph.eventMessageRequest",
"createdDateTime": "2020-07-01T18:21:32Z",
"lastModifiedDateTime": "2020-07-01T18:21:33Z",
"categories": [
],
"receivedDateTime": "2020-07-01T18:21:33Z",
"sentDateTime": "2020-07-01T18:21:29Z",
"hasAttachments": true,
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": true,
"isDraft": false,
"inferenceClassification": "focused",
"meetingMessageType": "meetingRequest",
"type": "singleInstance",
"isOutOfDate": false,
"isAllDay": false,
"isDelegated": false,
"responseRequested": true,
"allowNewTimeProposals": null,
"meetingRequestType": "informationalUpdate"
}
]

How can I parse nested JSON using Flutter?

I want to parse this JSON data but I have no idea how can I parse it.
{
"kind": "Testing",
"data": {
"modhash": "fffffghhhhhhbf0bb4601d6158db548b5296632215a0",
"dist": 75,
"children": [
{
"kind": "z4",
"data": {
"subreddit": "iOS",
"selftext": "I am iOS developer",
"author_fullname": "tyy_iif5q",
"saved": false,
"gilded": 0,
"clicked": false,
"title": "When is a good time to use docker?",
"subreddit_name_prefixed": "r/webdev",
"hidden": false,
"pwls": 6,
"downs": 0,
"hide_score": false,
"name": "t3_gwqan4",
"quarantine": false,
"link_flair_text_color": "dark",
"upvote_ratio": 0.96,
"subreddit_type": "public",
"ups": 277,
"total_awards_received": 0,
"is_original_content": false,
"is_reddit_media_domain": false,
"is_meta": false,
"can_mod_post": false,
"score": 277,
"author_premium": false,
"thumbnail": "self",
"edited": false,
"is_self": true,
"created": 1591333752,
"link_flair_type": "text",
"wls": 6,
"author_flair_type": "text",
"domain": "self.webdev",
"allow_live_comments": false,
"selftext_html": "<!-- SC_OFF --><div class=\"md\"><p>I just finished part 7 of the Full Stack Open course (<a href=\"https://www.fullstackopen.com/en\">www.fullstackopen.com/en</a>) and due to graphQL and typescript not being within the core content of the course, I thought I might try and learn docker. </p>\n\n<p>Turns out, trying to learn docker is just making me even more confused than when I didn&#39;t even know docker existed. </p>\n\n<p>&#x200B;</p>\n\n<p>Could someone explain to me, when is a good time to use docker? How one would go about using docker, and when in someone&#39;s learning path would one ideally learn docker?</p>\n\n<p>&#x200B;</p>\n\n<p>I am still learning react, databases, and overall front-end and back-end web development currently. The most I have done thus-far is hosting an app I made on heroku. I haven&#39;t setup a personal website, portfolio, or hosted an app on an independent website either.</p>\n</div><!-- SC_ON -->",
"archived": false,
"no_follow": false,
"is_crosspostable": true,
"pinned": false,
"over_18": false,
"media_only": false,
"can_gild": true,
"spoiler": false,
"locked": false,
"visited": false,
"subreddit_id": "t5_2qs0q",
"id": "gwqan4",
"is_robot_indexable": true,
"author": "Sensualities",
"num_comments": 127,
"send_replies": true,
"whitelist_status": "all_ads",
"contest_mode": false,
"author_patreon_flair": false,
"permalink": "/r/webdev/comments/gwqan4/when_is_a_good_time_to_use_docker/",
"parent_whitelist_status": "all_ads",
"stickied": false,
"url": "https://www.reddit.com/r/webdev/comments/gwqan4/when_is_a_good_time_to_use_docker/",
"subreddit_subscribers": 487697,
"created_utc": 1591304952,
"num_crossposts": 1,
"is_video": false
}
},
{
"kind": "t3",
"data": {
"subreddit": "iOS",
"selftext": "I am iOS developer",
"author_fullname": "tyy_iif5q",
"saved": false,
"gilded": 0,
"clicked": false,
"title": "When is a good time to use docker?",
"subreddit_name_prefixed": "r/webdev",
"hidden": false,
"pwls": 6,
"downs": 0,
"hide_score": false,
"name": "t3_gwqan4",
"quarantine": false,
"link_flair_text_color": "dark",
"upvote_ratio": 0.96,
"subreddit_type": "public",
"ups": 277,
"total_awards_received": 0,
"is_original_content": false,
"is_reddit_media_domain": false,
"is_meta": false,
"can_mod_post": false,
"score": 277,
"author_premium": false,
"thumbnail": "self",
"edited": false,
"is_self": true,
"created": 1591333752,
"link_flair_type": "text",
"wls": 6,
"author_flair_type": "text",
"domain": "self.webdev",
"allow_live_comments": false,
"selftext_html": "<!-- SC_OFF --><div class=\"md\"><p>I just finished part 7 of the Full Stack Open course (<a href=\"https://www.fullstackopen.com/en\">www.fullstackopen.com/en</a>) and due to graphQL and typescript not being within the core content of the course, I thought I might try and learn docker. </p>\n\n<p>Turns out, trying to learn docker is just making me even more confused than when I didn&#39;t even know docker existed. </p>\n\n<p>&#x200B;</p>\n\n<p>Could someone explain to me, when is a good time to use docker? How one would go about using docker, and when in someone&#39;s learning path would one ideally learn docker?</p>\n\n<p>&#x200B;</p>\n\n<p>I am still learning react, databases, and overall front-end and back-end web development currently. The most I have done thus-far is hosting an app I made on heroku. I haven&#39;t setup a personal website, portfolio, or hosted an app on an independent website either.</p>\n</div><!-- SC_ON -->",
"archived": false,
"no_follow": false,
"is_crosspostable": true,
"pinned": false,
"over_18": false,
"media_only": false,
"can_gild": true,
"spoiler": false,
"locked": false,
"visited": false,
"subreddit_id": "t5_2qs0q",
"id": "gwqan4",
"is_robot_indexable": true,
"author": "Sensualities",
"num_comments": 127,
"send_replies": true,
"whitelist_status": "all_ads",
"contest_mode": false,
"author_patreon_flair": false,
"permalink": "/r/webdev/comments/gwqan4/when_is_a_good_time_to_use_docker/",
"parent_whitelist_status": "all_ads",
"stickied": false,
"url": "https://www.reddit.com/r/webdev/comments/gwqan4/when_is_a_good_time_to_use_docker/",
"subreddit_subscribers": 487697,
"created_utc": 1591304952,
"num_crossposts": 1,
"is_video": false
}
}
],
"after": "ggggggg"
}
}
I want to parse this JSON data but I have no idea how can I parse it.I worked with simple json but I have no idea how can I parse nested json please help me. I want to parse this JSON data but I have no idea how can I parse it.I worked with simple json but I have no idea how can I parse nested json please help me.
Take a look at this library, it will help you ;)
https://pub.dev/packages/json_serializable
This lib will generate conversion code fromJson and toJson for each model annotated with #JsonSerializable
It take care of nested objects, type conversion and many other things ;)
Use this link to prepare model class of Json
https://javiercbk.github.io/json_to_dart/
HttpResponse response = callApi;
then after getting response just use json.decode;
then use fromjson Method of your Model generated through link and your whole json is parsed
Map userMap = jsonDecode(jsonString);
var user = User.fromJson(userMap);
User is Model created from site
At first you need a classes representing JSON with the method fromJson which parses JSON into your object.
Also you should use nested classes for nested JSON fields
Alternatively, you could generate necessary classes with this class generator
class JSONRepresentation {
final String kind;
final Data data;
JSONRepresentation({this.kind, this.data});
JSONRepresentation .fromJson(Map<String, dynamic> data)
: kind = data['kind'],
data = Data .fromJson(data['data']);
}
class Data {
final String modhash;
final int dist;
//final List<AnotherDataClass> children;
final String after;
Data({this.modhash, this.dist, this.after});
Data .fromJson(Map<String, dynamic> data)
: modhash = data['modhash'],
dist = data['dist'],
after = data['after'];
}
And then you can parse it using dart:convert and get data from objects representing JSON as below
import "dart:convert";
void main() {
var jsonData =
'{"kind": "Testing",
"data": {
"modhash": "fffffghhhhhhbf0bb4601d6158db548b5296632215a0",
"dist": 75,
//"children": [...],
"after": "ggggggg"}}';
var parsedJson = json.decode(jsonData);
var jsonRepresentationObject = JSONRepresentation.fromJson(parsedJson);
//Example of getting data from objects representing JSON
print("kind: "+ jsonRepresentationObject.kind +
"\nData modhash: " + jsonRepresentationObject.data.modhash);
}
You could test my example here - online dart compiler

Anyway to "format" this json file, I can't seem to use run or count to make it work in mongodb

This is a sample of the json file that I've imported into my mongodb, I want build a basic index on it to search fields for relationship.
{
"business_id":"93otbGHE0s0m-lU1osvg9w",
"name":"Rivertowne",
"neighborhood":"North Side",
"address":"337 N Shore Dr",
"city":"Pittsburgh",
"state":"PA",
"postal_code":"15212",
"latitude":40.4459861,
"longitude":-80.0108802,
"stars":3.0,
"review_count":102,
"is_open":1,
"attributes":[
"Alcohol: full_bar",
"Ambience: {'romantic': False, 'intimate': False, 'classy': False, 'hipster': False, 'divey': False, 'touristy': False, 'trendy': False, 'upscale': False, 'casual': True}",
"BestNights: {'monday': False, 'tuesday': False, 'friday': True, 'wednesday': False, 'thursday': False, 'sunday': True, 'saturday': True}",
"BikeParking: True",
"BusinessAcceptsCreditCards: True",
"BusinessParking: {'garage': False, 'street': True, 'validated': False, 'lot': False, 'valet': False}",
"Caters: False",
"CoatCheck: False",
"GoodForDancing: False",
"GoodForKids: True",
"GoodForMeal: {'dessert': False, 'latenight': False, 'lunch': True, 'dinner': True, 'breakfast': False, 'brunch': False}",
"HappyHour: True",
"HasTV: True",
"Music: {'dj': False, 'background_music': False, 'no_music': False, 'karaoke': False, 'live': False, 'video': False, 'jukebox': False}",
"NoiseLevel: loud",
"OutdoorSeating: True",
"RestaurantsAttire: casual",
"RestaurantsDelivery: False",
"RestaurantsGoodForGroups: True",
"RestaurantsPriceRange2: 2",
"RestaurantsReservations: False",
"RestaurantsTableService: True",
"RestaurantsTakeOut: True",
"Smoking: no",
"WheelchairAccessible: True",
"WiFi: free"
],
"categories":[
"American (New)",
"Pubs",
"Nightlife",
"Bars",
"Restaurants"
],
"hours":null,
"type":"business"
}
But after I imported into json, the count and find all turn up zero. An expert told me the json file was not formatted the right way, so it won't fit my mongdb requirement. I have to find a different version of this json file.
Is there way to fix this?
attribute
GoodForDancing: True
And attribute
RestaurantsAttire: casual
--------------------PART B.-----------------
GoodForDancing: True
And attribute
RestaurantsAttire: dressy
I want to compare some things, out of restaurants with the attribute "Good For Dancing" I want to see how many have casual vs. dressy attire. I want to do a query search for both part using key, but so far count and find showed up 0.
This is my import command, business is the collection name and yep is the dbname:
mongoimport --db yelp --collection business --file /home/dachen/Desktop/business.json
My find command turned up nothing, so obviously json file is something wrong with it
db.business.find({ "attributes.GoodForDancing" : { $exists : true } })
People say the syntax is wrong, "", anyway way to fix it?

xPath of JSON response in soapUI

can anyone help in finding xpath of 'tenantId' which is in JSON response given below
{
"statusCode": "200",
"errorParams": null,
"payLoad": {
"UGX:XCD": {
"id": 42802,
"fromCurrency": {
"tenantId": "2a610af6-2e42-4a78-8e04-98755e2c6113",
"code": "UGX",
"name": "Ugandan Shilling",
"isoCode": "UGX",
"active": true,
"enabled": true,
"flagModifiedName": false,
"erpId": null,
"createdOn": 1427155200000,
"fieldLibraryStatus": "DRAFT",
"userId": null
},
"toCurrency": {
"tenantId": "2a610af6-2e42-4a78-8e04-98755e2c6113",
"code": "XCD",
"name": "East Caribbean Dollar",
"isoCode": "XCD",
"active": true,
"enabled": true,
"flagModifiedName": true,
"erpId": null,
"createdOn": 1427155200000,
"fieldLibraryStatus": "DRAFT",
"userId": "c7f68b05-044d-44e1-8fc5-0f97a4ece91b"
},
"rate": 33,
"inverseRate": 0.0303,
"createdOn": 1434025926983,
"modifiedOn": 1434326400000,
"active": true,
"purpose": "cmd",
"enabled": true,
"tenantFlag": true,
"erpId": null
}
How to traverse till 'tenantId' ??
Thanks in Advance.
SoapUI internally converts almost anything to XML. You will have to use the ResponseAsXml property. In the GUI, if you click on the XML tab, you can see this representation.
Something like the following should work:
${test_step#ResponseAsXml#//*:fromCurrency/*:tenantId}
${test_step#ResponseAsXml#//*:toCurrency/*:tenantId}
If you want to handle this using groovy, then extend SOAPUI with GSON.jar library provided by Google.
this library specially written to handle the JSON objects.
With this you can rip every individual objects and its attribute-values easily how complex may the JSON object be, a nice add-on to work with.

typeahead prefetch is not working

I have searched for an answer but still cant find it. I guess I am doing something wrong but i can't figure out what it is. This is my typeahead code:
$('.zoek').typeahead({
name: 'items01',
prefetch: {
url: '/models/_global.json',
ttl: 1,
},
template: '<div id="breed">
<span>{{#lock}}<img src="/assets/img/lock-icon-small.png">{{/lock}}{{^lock}}<img src="/assets/img/empty-icon-small.png">{{/lock}}</span>
<span class="searchresult">{{value}}</span>
<span class="searchresult-type"> - {{type}} </span>
<span class="red">{{#popular}} - popular {{/popular}}</span>
<span class="red">{{#new}} - new {{/new}}</span>
</div>',
engine: Hogan
});
And this is my _global.json file:
{
"items": [
{"value":"making claims for ebook subscriptions",
"icon":"download-icon-small.png",
"type":"brochure",
"date":"12-10-2011",
"lock": false,
"popular": false,
"new": true,
"tag": ["ebook","claim"]},
{"value":"Claim discovery & Online Management Suite - English",
"icon":"download-icon-small.png",
"type":"case",
"date":"28-02-2010",
"lock": true,
"popular": false,
"new": false,
"tag": ["claim","online"]},
{"value":"Get access to all online ebook publishers ",
"icon":"download-icon-small.png",
"type":"factsheet",
"date":"17-08-2012",
"lock": false,
"popular": true,
"new": true,
"tag": ["online","ebook"]},
{"value":"Ebook catalogue in swetswise - howto",
"icon":"show-icon-small.png",
"type":"hint",
"date":"08-01-2009",
"lock": true,
"popular": false,
"new": false,
"tag": ["ebook","swetswise"]},
{"value":"Swetswise software improved: you can claim now!",
"icon":"download-icon-small.png",
"type":"press",
"date":"20-08-2002",
"lock": false,
"popular": true,
"new": false,
"tag": ["swetswise","claim"]}
]
}
The Json file can't have an other format because some other code is relying on this format. Prefetch is not working and I guess I have to use a filter. Tried all kind of filter examples i could find but nothing seems to work. What am I doing wrong.
p.s.
(When I leave out the top curly brackets and the key "items" in the json file it is all working, so there are no problems with the js. libraries and Hogan)
Hope someonecan help me.
regards. M.dekker
After a good sleep and some thinking and found the answer to my question.
adding the following code in the prefetch block solved my problem:
filter: function(data){
// filter the returned data
return data.items;
}