I have the following JSON array constructed out of ZFS pool information to which a node field as been added (which could have many elements of course as there are many nodes):
[
{
"NodeName": "node1",
"VolumeName": "vol1",
"Size": "50.0T",
"Used": "18.6T",
"Free": "31.4T",
"Fragmentation": "31%",
"Capacity": "37%",
"Health": "ONLINE"
},
{
"NodeName": "node1",
"VolumeName": "vol2",
"Size": "796G",
"Used": "28.5M",
"Free": "796G",
"Fragmentation": "0%",
"Capacity": "0%",
"Health": "ONLINE"
}
]
And I would like to map this into this type of object format, promoting a field to be the new 'key' and return an object with that field as the key to an array of the original data without that key: (Almost like grouping the data by a particular field)
{
"node1": [
{
"VolumeName": "vol1",
"Size": "50.0T",
"Used": "18.6T",
"Free": "31.4T",
"Fragmentation": "31%",
"Capacity": "37%",
"Health": "ONLINE"
},
{
"VolumeName": "vol2",
"Size": "796G",
"Used": "28.5M",
"Free": "796G",
"Fragmentation": "0%",
"Capacity": "0%",
"Health": "ONLINE"
}
]
}
I believe map or group-by are the way to go but not getting the required result. Any suggestions would be helpful. Thank.
Group by .NodeName, and index each group with that value in an object. Then remove NodeName field from all members.
group_by(.NodeName)
| INDEX(.[]; .[0].NodeName)
| del(.[][].NodeName)
Online demo
Related
I'm trying to limit the map in my view to a specific set of documents by either having the id "startsWith" a string or based on there being a specific node in the JSON> I can't seem to get a result set once I add an IF statement. The reduce is a simple _count:
function(doc, meta) {
if (doc.metricType == "Limit_Exceeded") {
emit([doc.ownedByCustomerNumber, doc.componentProduct.category], meta.id);
}
}
I've also tried if (doc.metricType) and also if(meta.id.startsWith("Turnaway:")
Example Doc:
{
"OvidUserId": 26105400,
"id": "Turnaway:00005792:10562440",
"ipAddress": "111187081038",
"journalTurnawayNumber": 10562440,
"metricType": "Limit_Exceeded",
"oaCode": "OA_Gold",
"orderNumber": 683980,
"ovidGroupID": 3113900,
"ovidGroupName": "tnu999",
"ovidUserName": "tnu999",
"ownedByCustomerNumber": 59310,
"platform": "Lippincott",
"samlString": "",
"serialName": "00005792",
"sessionID": "857616ee-dab7-43d0-a08b-abb2482297dd",
"soldProduct": {
"category": "Multidisciplinary Subjects",
"name": "Custom Collection For CALIS - LWW TA 2020",
"productCode": "CCFCCSI20",
"productNumber": 33410,
"subCategory": "",
"subject": "Multidisciplinary Subjects"
},
"soldToCustomer": {
"customerNumber": 59310,
"keyAccount": false,
"name": "Tongji University"
},
"turnawayDateTime": "2022-05-04T03:01:44.600",
"usedByCustomer": {
"customerNumber": 59310,
"keyAccount": false,
"name": "Tongji University"
},
"usedByCustomerNumber": 59310,
"yearMonth": "202205"
},
"id": "Turnaway:00005792:10562440"
}
Thanks,
Gerry
Found it (of course after posting the question) The second component of the Key in the emit has to exist. I entered doc.componentProduct.category instead of doc.soldProduct.Category.
I need some help querying this JSON file I've ingested into a temp table in Snowflake. So, I've created a JSON_DATA variant column and plan to query and do a COPY INTO another table, but my query isn't working yet... I feel I'm close (possibly?)
JSON layout:
{
"nextPage": "01",
"page": "0",
"status": "ok",
"transactions": [
{
"id": "65985",
"recordTp": "vendorbill",
"values": {
"account": [
{
"text": "14500 Deferred Expenses",
"value": "249"
}
],
"account.number": "1450",
"account.type": [
{
"text": "Deferred Expense",
"value": "DeferExpense"
}
],
"amount": "51733",
"classnohierarchy": [
{
"text": "901 Corporate",
"value": "139"
}
],
"currency": [
{
"text": "Canadian Dollar",
"value": "3"
}
],
"customer.altname": "V Sties expenses (Tor)",
"customer.custate": "12/31/2019",
"customer.custentient": "ada Inc.",
"customer.custendate": "1/1/2019",
"customer.entyid": "PR781",
"departmentnohierarchy": [
{
"text": "8rity",
"value": "37"
}
],
"fxamount": "689",
"location": [
{
"text": "Othad Projects",
"value": "48"
}
],
"postingperiod": [
{
"text": "Jan 2020",
"value": "1"
}
],
"subsidiary.custrecord_region": [
{
"text": "CANADA",
"value": "3"
}
],
"subsidiarynohierarchy": [
{
"text": "ada Inc.",
"value": "25"
}
]
}
},
I've been able to query the values that are not (deeply) nested but I need help getting, for example, the values from 'classnohierarchy', to get both the 'text' and 'value' I tried:
transactions.value:"values".classnohierarchy.text::string as class_txt,
transactions.value:"values".classnohierarchy.value::string as class_val,
but it's returning NULL values.
Below is my entire query:
SELECT
JSON_DATA:status::string as connection_status,
transactions.value:id::string as id,
transactions.value:recordType::string as record_type,
transactions.value:"values"::variant as trans_val,
transactions.value:"values".account as acc,
transactions.value:"values".account.text as text,
transactions.value:"values".account.value as val,
transactions.value:"values"."account.number"::string as acc_num,
transactions.value:"values"."account.type".text::string as acc_type_txt,
transactions.value:"values"."account.type".value::string as acc_type_val,
transactions.value:"values".amount::string as amount,
**transactions.value:"values".classnohierarchy.text::string as class_txt,
transactions.value:"values".classnohierarchy.value::string as class_val,**
transactions.value:"values".currency.text::string as currency_text,
transactions.value:"values".currency.value::string as currency_val,
transactions.value:"values"."customer.altname"::string as customer_project_name,
transactions.value:"values"."customer.custate"::string as customer_end_date,
transactions.value:"values"."customer.custentient"::string as customer_end_client,
transactions.value:"values"."customer.custendate"::string as customer_start_date,
transactions.value:"values"."customer.entyid"::string as customer_project_id,
transactions.value:"values".departmentnohierarchy.text::string as department_name,
transactions.value:"values".departmentnohierarchy.value::string as department_value,
transactions.value:"values".fxamount::string as fx_amount,
transactions.value:"values".location.text::string as product_name,
transactions.value:"values".postingperiod.text::string as postingperiod,
transactions.value:"values".postingperiod.value::string as postingperiod,
transactions.value:"values"."subsidiary.custrecord_region".text::string as region_name,
transactions.value:"values"."subsidiary.custrecord_region".value::string as region_value,
transactions.value:"values".subsidiarynohierarchy.text::string as entity_name,
transactions.value:"values".subsidiarynohierarchy.value::string as entity_value,
FROM MY_TABLE,
LATERAL FLATTEN (JSON_DATA:transactions) as transactions
and here's a picture of whats showing in Snowflake:
SNOWFLAKE_SCREENSHOT
departmentnohierarchy is an array. you need to mention the index as below.
select *,transactions.VALUE:"values".departmentnohierarchy[0].value::text as department_name
FROM jsont1,
LATERAL FLATTEN (JSON_DATA:transactions) as transactions
I have the following array of data inside the data bucket SITES in my Couchbase server
"siteMaster": [
{
"sitename": "HTS_SITE_001",
"sitelink": "http://facebook.com",
"address" : "19/2, Bellandur, Bangalore India",
"filename": "site1.json",
"persons": 1,
"status": "70%",
"contact": "max.smith#honeywell.com",
}, {
"sitename": "HTS_SITE_002",
"sitelink": "http://facebook.com",
"address": "5th Avenue, New York",
"filename": "site2.json",
"persons": 1,
"status": "70%",
"contact": "john.smith#facebook.com",
}, {
"sitename": "HTS_SITE_003",
"sitelink": "http://facebook.com",
"address": "Palo Alto, California",
"filename": "site3.json",
"persons": 1,
"status": "80%",
"contact": "steve.jobs#apple.com",
}, {
"sitename": "HTS_SITE_004",
"sitelink": "http://facebook.com",
"address": "Bellandur, Bangalore",
"filename": "site4.json",
"persons": 1,
"status": "80%",
"contact": "max.mustermann#deutsche.com",
}
]
The N1QL query for
select * from SITES where status = "70%" should return me two rows, but unfortunately it is not returning any rows.
Where am I going wrong with the query ?
Please use the following query:
SELECT *
FROM SITES
WHERE ANY sm IN siteMaster SATISFIES sm.status = "70%" END;
You can also create the following array index to speed up the query:
CREATE INDEX idx ON SITES( DISTINCT ARRAY sm.status FOR sm IN siteMaster END );
My response from backend is not in form which ember store. I am not able to serialize the response.
response.json
[{
"pk": 127,
"url": "http://example.com/api/galleries/127/",
"gallery_name": "Faces",
"thumbnail_url": "https://example.cloud.net/galleryThumbs/2656a05c-4ec7-3eea-8c5e-d8019454d443.jpg",
"time": "1 month ago",
"description": "Created by user",
"is_following": true,
"feedPhotos": [{
"pk": 624,
"url": "http://example.com/api/photos/624/",
"profilePic": "https://example.cloud.net/userDPs/50906ce2-394d-39c8-9261-8cf78e3611c2.jpg",
"userName": "Nabeela",
"userKarma": 915,
"caption": "Old woman spinning her 'chhos-khor' ...a rotation of which is equivalent to the recitation of a mantra.",
"numComments": 0,
"owner": "http://example.com/api/users/44/",
"time": "1 month ago",
"photo_url": "https://example.cloud.net/photos/9cbd6423-3bc5-36e0-b8b4-d725efb3249a.jpg",
"comments_url": "http://example.com/api/photos/624/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/1362/fivers/",
"fivers_pk": 1362,
"fullphoto_url": "http://example.com/api/photogalleries/1362/photo/",
"fullphoto_pk": 1362,
"is_fived": true,
"hiFiveKarma": 1,
"owner_pk": 44,
"userFirstName": "Nabeela",
"is_bookmarked": false
}, {
"pk": 574,
"url": "http://example.com/api/photos/574/",
"profilePic": "https://example.cloud.net/userDPs/b6f69e4e-980d-3cc3-8b3e-3eb1a7f21350.jpg",
"userName": "Rohini",
"userKarma": 194,
"caption": "Life # Myanmar!",
"numComments": 0,
"owner": "http://example.com/api/users/45/",
"time": "2 months ago",
"photo_url": "https://example.cloud.net/photos/eeae72d5-d6af-391e-a218-b442c0c7e34e.jpg",
"comments_url": "http://example.com/api/photos/574/comments/",
"numFives": 2,
"fivers_url": "http://example.com/api/photogalleries/1303/fivers/",
"fivers_pk": 1303,
"fullphoto_url": "http://example.com/api/photogalleries/1303/photo/",
"fullphoto_pk": 1303,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 45,
"userFirstName": "Rohini",
"is_bookmarked": false
}
]
}, {
"pk": 65,
"url": "http://example.com/api/galleries/65/",
"gallery_name": "Royal",
"thumbnail_url": "https://example.cloud.net/galleryThumbs/d8a900af-1f1d-3977-8cc8-b8bb36e32be5.jpg",
"time": "2 months ago",
"description": "This is a gallery about Royal",
"is_following": false,
"feedPhotos": [{
"pk": 347,
"url": "http://example.com/api/photos/347/",
"profilePic": "https://example.cloud.net/userDPs/50906ce2-394d-39c8-9261-8cf78e3611c2.jpg",
"userName": "Nabeela",
"userKarma": 915,
"caption": "I cannot forget the name of this palace - Moti Mahal (translation: Pearl Palace). Indescribably beautiful, ainnit! at Mehrangarh fort, Jodhp",
"numComments": 0,
"owner": "http://example.com/api/users/44/",
"time": "2 months ago",
"photo_url": "https://example.cloud.net/photos/958ed406-708e-3f01-a2f4-9467cd709fdd.jpg",
"comments_url": "http://example.com/api/photos/347/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/759/fivers/",
"fivers_pk": 759,
"fullphoto_url": "http://example.com/api/photogalleries/759/photo/",
"fullphoto_pk": 759,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 44,
"userFirstName": "Nabeela",
"is_bookmarked": false
}, {
"pk": 593,
"url": "http://example.com/api/photos/593/",
"profilePic": "https://example.cloud.net/userDPs/95ac6974-f7df-338c-ab84-99fa1df7514c.jpg",
"userName": "Vikanshu",
"userKarma": 932,
"caption": "Marvelous architecture!! in Florence, Italy",
"numComments": 0,
"owner": "http://example.com/api/users/48/",
"time": "1 month ago",
"photo_url": "https://example.cloud.net/photos/7a86eb37-6c68-3d6c-b6cf-2e3b74d330dd.jpg",
"comments_url": "http://example.com/api/photos/593/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/1363/fivers/",
"fivers_pk": 1363,
"fullphoto_url": "http://example.com/api/photogalleries/1363/photo/",
"fullphoto_pk": 1363,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 48,
"userFirstName": "Vikanshu",
"is_bookmarked": false
}]
}]
How do I serialize this using JSONPISerailizer or any other serializer in ember-cli so that it gets stored in ember store
Reference jsonapi.org
++++Top Level:
Root:
A JSON object must be root of every JSON API request response.
A document must contain at least one top-level members:
1. data: documents "primary data"
2. errors: an array of error objects (id,status,code,title....)
3. meta: a meta object that contains non-standard meta-information (copyright,author...)
member data and errors must not co-exist together.
"data"{}
+++++Resource Objects
1. A resource object MUST contain atleast following top-level member
*id
*type
```
//structure-1
//for galleries
{
"data": {
"type": "galleries",
"id": "1"
}
}
//for photos
{
"data": {
"type": "photos",
"id": "1"
}
}
```
In addition, a resource object may contain any of these top-level members
*attributes
*relationship
*links
*meta
//added attributes first
```
//structure-2
//for galleries
{
"data": {
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
}
}
}
//for photos
{
"data": {
"type": "photos",
"id": "1",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}
}
```
//Adding relationship
Relationship object must contain atleast one of the following
*links (containing atleast one of "self" or "related" resource link
*data
*meta
//link in relationship (minimum one required from link,data,meta).
//
```
//structure-3
//for galleries
{
"data":[{ //Array(square bracket as adding relationship one more item to data
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
}]
}
}
```
//data in relationship
```
//structure-4
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
}]
}
}
```
//Adding related resource "included"
```
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
}],
"included":[{
"type": "photos",
"id": "77",
"attributes":{
userName: "Nabeela",
userKarma: 915
},
{
"type": "photos",
"id": "78",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}]
}
}
```
For collections. I am not confident but try this
Now for collection of galleries.
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
},{
"type": "galleries",
"id": "2",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "79"
}
}],
"included":[{
"type": "photos",
"id": "77",
"attributes":{
userName: "Nabeela",
userKarma: 915
},{
"type": "photos",
"id": "78",
"attributes":{
userName: "Nabeela",
userKarma: 915
},{
"type": "photos",
"id": "79",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}]
}
}
============Implementation part =================================
JSONSerializer normalization process follows these steps
*normalizeResponse : entry method.
*normalizeCreateRecordResponse : a normalizeResponse for specific operation.
*normalizeSingleResponse|normalizeArrayResponse:
- for methods like createRecord. we expect a single record back.
- for methods like findAll we expect multiple records back.
+normalize =
normalizeArray iterates and calls normalize for each of it's records
normalizeSingle call its once.
+extractID | extractAttributes | extractRelationships
= normalize delegates to these method to turn record payload into jsonAPI format
Starting with normalizeResponse method. If you open and see normalizeResponse method
in json-serializer
link normalizeResponse: https://github.com/emberjs/data/blob/v2.2.1/packages/ember-
data/lib/serializers/json-serializer.js#L192
you with find a switch case switch(requestType). If requestType if
"findRecord" then "normalizeFindRecordResponse" is called
"queryRecord" then "normalizeQueryRecordResponse" is called
"findAll" then "normalizeFindAllResponse" is called
...so on and so forth.
if you notice the parameter passed to all the methods are same as that of normalize
(...arguments) :)
**Lets start for findAll
i.e normalizeResponse -> normalizeFindAllResponse -> normalizeArrayResponse
as normalizeFindAllResponse method has only one line that call
normalizeArrayResponse.
normalizeFindAllResponse
normalizeResponse -> normalizeFindAllResponse -> normalizeArrayResponse ->
_normalizeResponse{ extractMeta,normalize }
extractMeta [extract meta information like pagination and stuff ]
if single: normalize []
example of normalize method in emberjs docs
```
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
normalize: function(typeClass, hash) {
var fields = Ember.get(typeClass, 'fields');
fields.forEach(function(field) {
var payloadField = Ember.String.underscore(field);
if (field === payloadField) { return; }
hash[field] = hash[payloadField];
delete hash[payloadField];
});
return this._super.apply(this, arguments);
}
});
```
"normalizeArrayResponse calls `return this._normalizeResponse
(store,primaryModelClass,payload,id,requestType,false).
so isSingle is false for _normalizeResponse method. so we will have to push all the
related records of included array
in our case the photos which is done by below snippet from "_normalizeRespose"
method.
_normalizeResponse
```
else{
documentHash.data = payload.map((item) => {
let { data, included } = this.normalize(primaryModelClass,item);
if(included){
documentHash.included.push(...included);
}
return data;
});
return documentHash;
}
```
Things are still unclear in the context of our JSON reponse from server
but atleast we know the flow now.
Lets try to apply it for findAll ( as per the flow above).
run "ember g serializer application" //assuming you are using ember-cli and you
intend to make this serializer generic for application.
As of now I have no information how and when normalizeResponse is called. :(
I just scanned through and guess on recieving data from server the store calls
normalizeResponseHelpers which in turn calls normalizeResponse.
In any case "normalizeResponse" is going to send payload and other necessar
information to normalizeFindAllResponse(...arguments) which in turn will call
normalizeArrayResponse(...arguments) which in turn will call "_normalizeRespone".
Here is where we need to take action
for extraMeta and normalize.
+extraMeta
I am not sure if there is any meta information in you json response.
in case there is you can refer to the example mentioned in docs
extractMeta
So I guess you can directly user the normalize method from example in your application ;).
please try and check. Since i am learning ember myself I cannot guarantee it will work but it should. the lonngggg explation is my thought while i was learning the problem/solution
//app/serializers/application.js
+normalize
```
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
normalize: function(typeClass, hash) {
var fields = Ember.get(typeClass, 'fields');
fields.forEach(function(field) {
var payloadField = Ember.String.underscore(field);
if (field === payloadField) { return; }
hash[field] = hash[payloadField];
delete hash[payloadField];
});
return this._super.apply(this, arguments);
}
});
```
The primary key in the JSON from server is pk. You will have to mention that too
http://emberjs.com/api/data/classes/DS.JSONSerializer.html#property_primaryKey
app/serializers/application.js
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
primaryKey: 'pk'
});
I have a big query table that has a JSON object as one of the field in the table. How can I extract the data from the JSON object using app script. The object itself is nested. It looks like this
{
"uid": "124551",
"subjects": [
{
"tid": 37,
"title": "Algebra",
"html_id": "algebra",
"selected": true
},
{
"tid": 214853,
"title": "Trigonometry",
"html_id": "trigonometry",
"selected": true
},
{
"tid": 38,
"title": "Geometry",
"html_id": "geometry",
"selected": true
}
],
"cellphone": "09178854579",
"educations": [
{
"index": 0,
"schoolname": "University of the Philippines - Los Baños",
"degree": "BS Mathematics",
"major": "Mathematics",
"eduFrom": "2009-05-31T16:00:00.000Z",
"eduTo": "2013-04-26T16:00:00.000Z",
"eduFromTs": 1243785600,
"eduToTs": 1366992000
}
],
"info": {
"os": "Windows",
"internet": "ADSL",
"browser": "Chrome",
"network": "Wireless",
"speed": "",
"timezone": "Asia/Hong_Kong"
}
}
I want to extract all school names from education field. Any ideas?
Working with JSON objects is similar to working with XML, except that parsing or encoding a JSON object is much easier.
Once this string is retrieved, simply call JSON.parse() on the string to get a native object representation.
var data = JSON.parse(json);
Browser.msgBox(data.info.os);
Other sample code is at https://developers.google.com/apps-script/advanced/bigquery#run_query