JSON object extract from big query database using app script - json

I have a big query table that has a JSON object as one of the field in the table. How can I extract the data from the JSON object using app script. The object itself is nested. It looks like this
{
"uid": "124551",
"subjects": [
{
"tid": 37,
"title": "Algebra",
"html_id": "algebra",
"selected": true
},
{
"tid": 214853,
"title": "Trigonometry",
"html_id": "trigonometry",
"selected": true
},
{
"tid": 38,
"title": "Geometry",
"html_id": "geometry",
"selected": true
}
],
"cellphone": "09178854579",
"educations": [
{
"index": 0,
"schoolname": "University of the Philippines - Los Baños",
"degree": "BS Mathematics",
"major": "Mathematics",
"eduFrom": "2009-05-31T16:00:00.000Z",
"eduTo": "2013-04-26T16:00:00.000Z",
"eduFromTs": 1243785600,
"eduToTs": 1366992000
}
],
"info": {
"os": "Windows",
"internet": "ADSL",
"browser": "Chrome",
"network": "Wireless",
"speed": "",
"timezone": "Asia/Hong_Kong"
}
}
I want to extract all school names from education field. Any ideas?

Working with JSON objects is similar to working with XML, except that parsing or encoding a JSON object is much easier.
Once this string is retrieved, simply call JSON.parse() on the string to get a native object representation.
var data = JSON.parse(json);
Browser.msgBox(data.info.os);
Other sample code is at https://developers.google.com/apps-script/advanced/bigquery#run_query

Related

JMeter - Extract value from the variable having JSON data

I am getting following response in one GET API call.
{
"id": 78929,
"created": "04-12-2021 02:37:52",
"modified": "04-12-2021 02:37:52",
"name": "Applebee",
"url": "https://applebee-2.prashant.com",
"key1": "vp_9d254bb754ab6283afa9ecf0f0847b5a",
"interval2": 120,
"tenant_id": 60895,
"channel_id": 2,
"channel_type": "platform",
"warehouse_id": 54533,
"warehouse_name": "Bedfordview Warehouse",
"stock_policy": "CUSTOM",
"currency_id": 45396,
"order_from_date": "2021-10-03 02:36:48",
"inventory_update": true,
"order_update": true,
"price_update": true,
"active": true,
"resync": true,
"last_access_record_id": 89880,
"last_accessed_record": {
"id": 89880,
"last_order_to_date": "2021-12-07T00:16:39.000+0000",
"last_order_id": "0",
"last_order_page": 1
},
"store_config": "{\"isOrderUpdateProcessor\":false,\"auto_archive\":true,\"tagOrderRisks\":true,\"error_emails\":\"kyle#gmail.com\",\"autoMapDuplicateProduct\":true,\"enabledMultiCurrency\":false,\"disableSyncPIIData\":false}",
"last_inventory_sync": "04-12-2021 02:43:35",
"inventory_sync_interval": 1440,
"stock_policy_warehouse_ids": [
{
"id": 53543,
"created": "04-12-2021 02:37:52",
"modified": "04-12-2021 02:37:52",
"store_id": 78929,
"external_warehouse_id": "34639577171",
"warehouse_id": 54533,
"add_qty_value": "0%",
"add_incoming_qty_value": "0%",
"tenant_id": 60895
},
{
"id": 53544,
"created": "04-12-2021 02:37:52",
"modified": "04-12-2021 02:37:52",
"store_id": 78929,
"external_warehouse_id": "60921610323",
"warehouse_id": 54534,
"add_qty_value": "0%",
"add_incoming_qty_value": "0%",
"tenant_id": 60895
}
],
"old_shopify_locations": [
{
"id": "60921610323",
"name": "Supplier",
"warehouseId": 54534,
"warehouseName": "Lux"
},
{
"id": "34639577171",
"name": "Bedfordview Warehouse",
"warehouseId": 54533,
"warehouseName": "Bedfordview Warehouse"
}
]
}
I have added following JSON Extractor to fetch store_config value from the above API response.
So as per above example, I am getting following store_config value via JSON Extractor.
{
"isOrderUpdateProcessor": false,
"auto_archive": true,
"tagOrderRisks": true,
"error_emails": "kyle#gmail.com",
"autoMapDuplicateProduct": true,
"enabledMultiCurrency": false,
"disableSyncPIIData": false
}
So Now, I want to fetch "isOrderUpdateProcessor" value from the above store_config variable.
So please help me into this.
You can use another JSON Extractor to extract from your variable by choosing the radio button JMeter variable and add your expression as$.isOrderUpdateProcessor and save result in a new variable
If you want to extract isOrderUpdateProcessor from the result
Add a JSON Extractor just below you JSON extractor
Configure the JSON Extractor to extract the value isOrderUpdateProcessor from the JMeter variable store_config
Given you have groovy in your tags it would be something like:
vars.put('foo', new groovy.json.JsonSlurper().parseText(vars.get('StoreConfig')).isOrderUpdateProcessor as String)
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy - Why and How You Should Use It

How can I create an activity for data conversion in Design Automation API?

I'm prototyping a web service to convert data using Design Automation API in Autodesk Forge.
My approach is to invoke an activity that executes a script to import a target data file (such as STEP, IGES format).
As an example, I created an activity to convert a STEP file to DWG as follows:
{
"HostApplication": "",
"RequiredEngineVersion": "22.0",
"Parameters": {
"InputParameters": [{
"Name": "Source",
"LocalFileName": "input.stp"
}, {
"Name": "HostDwg",
"LocalFileName": "$(HostDwg)"
}],
"OutputParameters": [{
"Name": "Result",
"LocalFileName": "output.dwg"
}]
},
"Instruction": {
"CommandLineParameters": null,
"Script": "import\ninput.stp\nsaveas\n\noutput.dwg\n"
},
"Version": 1,
"Id": "Step2Dwg"
}
The workitem to invoke this activity was executed without errors, but the output file (output.dwg) had nothing imported from the input file (input.stp).
Perhaps this is because some fields (e.g., AllowedChildProcess) were missing in the definition of the activity "Step2Dwg", but I do not know how to fix it.
My questions are:
How to fix the definition of the activity "Step2Dwg" to convert data successfully?
Is there any other approach to create an activity to convert data successfully?
You can use the Activity “Translate-STEP2DWG". It takes a .stp file as input and generate result.dwg as output. This is a public activity that anybody can send workitems against to it.
The activity is defined like this:
{
"Id": "Translate-STEP2DWG",
"AppPackages": [],
"HostApplication": "AcTranslators.exe",
"RequiredEngineVersion": "22.0",
"Parameters": {
"InputParameters": [
{
"Name": "HostDwg",
"LocalFileName": "source.stp"
}
],
"OutputParameters": [
{
"Name": "Result",
"LocalFileName": "result.dwg"
}
]
},
"Instruction": {
"CommandLineParameters": "-i source.stp -o result.dwg",
"Script": ""
},
"AllowedChildProcesses": [
],
"IsPublic": true,
"Version": 1,
"Description": ""
}
Here is a sample workitem request body:
{
"ActivityId": "Translate-STEP2DWG",
"Arguments": {
"InputArguments": [
{
"Resource": "https://s3.amazonaws.com/AutoCAD-Core-Engine-Services/TestDwg/3DStep.stp",
"Name": "HostDwg"
}
],
"OutputArguments": [
{
"Name": "Result",
"HttpVerb": "POST"
}
]
}
}

Serialize the response from backend to store ember store

My response from backend is not in form which ember store. I am not able to serialize the response.
response.json
[{
"pk": 127,
"url": "http://example.com/api/galleries/127/",
"gallery_name": "Faces",
"thumbnail_url": "https://example.cloud.net/galleryThumbs/2656a05c-4ec7-3eea-8c5e-d8019454d443.jpg",
"time": "1 month ago",
"description": "Created by user",
"is_following": true,
"feedPhotos": [{
"pk": 624,
"url": "http://example.com/api/photos/624/",
"profilePic": "https://example.cloud.net/userDPs/50906ce2-394d-39c8-9261-8cf78e3611c2.jpg",
"userName": "Nabeela",
"userKarma": 915,
"caption": "Old woman spinning her 'chhos-khor' ...a rotation of which is equivalent to the recitation of a mantra.",
"numComments": 0,
"owner": "http://example.com/api/users/44/",
"time": "1 month ago",
"photo_url": "https://example.cloud.net/photos/9cbd6423-3bc5-36e0-b8b4-d725efb3249a.jpg",
"comments_url": "http://example.com/api/photos/624/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/1362/fivers/",
"fivers_pk": 1362,
"fullphoto_url": "http://example.com/api/photogalleries/1362/photo/",
"fullphoto_pk": 1362,
"is_fived": true,
"hiFiveKarma": 1,
"owner_pk": 44,
"userFirstName": "Nabeela",
"is_bookmarked": false
}, {
"pk": 574,
"url": "http://example.com/api/photos/574/",
"profilePic": "https://example.cloud.net/userDPs/b6f69e4e-980d-3cc3-8b3e-3eb1a7f21350.jpg",
"userName": "Rohini",
"userKarma": 194,
"caption": "Life # Myanmar!",
"numComments": 0,
"owner": "http://example.com/api/users/45/",
"time": "2 months ago",
"photo_url": "https://example.cloud.net/photos/eeae72d5-d6af-391e-a218-b442c0c7e34e.jpg",
"comments_url": "http://example.com/api/photos/574/comments/",
"numFives": 2,
"fivers_url": "http://example.com/api/photogalleries/1303/fivers/",
"fivers_pk": 1303,
"fullphoto_url": "http://example.com/api/photogalleries/1303/photo/",
"fullphoto_pk": 1303,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 45,
"userFirstName": "Rohini",
"is_bookmarked": false
}
]
}, {
"pk": 65,
"url": "http://example.com/api/galleries/65/",
"gallery_name": "Royal",
"thumbnail_url": "https://example.cloud.net/galleryThumbs/d8a900af-1f1d-3977-8cc8-b8bb36e32be5.jpg",
"time": "2 months ago",
"description": "This is a gallery about Royal",
"is_following": false,
"feedPhotos": [{
"pk": 347,
"url": "http://example.com/api/photos/347/",
"profilePic": "https://example.cloud.net/userDPs/50906ce2-394d-39c8-9261-8cf78e3611c2.jpg",
"userName": "Nabeela",
"userKarma": 915,
"caption": "I cannot forget the name of this palace - Moti Mahal (translation: Pearl Palace). Indescribably beautiful, ainnit! at Mehrangarh fort, Jodhp",
"numComments": 0,
"owner": "http://example.com/api/users/44/",
"time": "2 months ago",
"photo_url": "https://example.cloud.net/photos/958ed406-708e-3f01-a2f4-9467cd709fdd.jpg",
"comments_url": "http://example.com/api/photos/347/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/759/fivers/",
"fivers_pk": 759,
"fullphoto_url": "http://example.com/api/photogalleries/759/photo/",
"fullphoto_pk": 759,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 44,
"userFirstName": "Nabeela",
"is_bookmarked": false
}, {
"pk": 593,
"url": "http://example.com/api/photos/593/",
"profilePic": "https://example.cloud.net/userDPs/95ac6974-f7df-338c-ab84-99fa1df7514c.jpg",
"userName": "Vikanshu",
"userKarma": 932,
"caption": "Marvelous architecture!! in Florence, Italy",
"numComments": 0,
"owner": "http://example.com/api/users/48/",
"time": "1 month ago",
"photo_url": "https://example.cloud.net/photos/7a86eb37-6c68-3d6c-b6cf-2e3b74d330dd.jpg",
"comments_url": "http://example.com/api/photos/593/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/1363/fivers/",
"fivers_pk": 1363,
"fullphoto_url": "http://example.com/api/photogalleries/1363/photo/",
"fullphoto_pk": 1363,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 48,
"userFirstName": "Vikanshu",
"is_bookmarked": false
}]
}]
How do I serialize this using JSONPISerailizer or any other serializer in ember-cli so that it gets stored in ember store
Reference jsonapi.org
++++Top Level:
Root:
A JSON object must be root of every JSON API request response.
A document must contain at least one top-level members:
1. data: documents "primary data"
2. errors: an array of error objects (id,status,code,title....)
3. meta: a meta object that contains non-standard meta-information (copyright,author...)
member data and errors must not co-exist together.
"data"{}
+++++Resource Objects
1. A resource object MUST contain atleast following top-level member
*id
*type
```
//structure-1
//for galleries
{
"data": {
"type": "galleries",
"id": "1"
}
}
//for photos
{
"data": {
"type": "photos",
"id": "1"
}
}
```
In addition, a resource object may contain any of these top-level members
*attributes
*relationship
*links
*meta
//added attributes first
```
//structure-2
//for galleries
{
"data": {
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
}
}
}
//for photos
{
"data": {
"type": "photos",
"id": "1",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}
}
```
//Adding relationship
Relationship object must contain atleast one of the following
*links (containing atleast one of "self" or "related" resource link
*data
*meta
//link in relationship (minimum one required from link,data,meta).
//
```
//structure-3
//for galleries
{
"data":[{ //Array(square bracket as adding relationship one more item to data
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
}]
}
}
```
//data in relationship
```
//structure-4
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
}]
}
}
```
//Adding related resource "included"
```
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
}],
"included":[{
"type": "photos",
"id": "77",
"attributes":{
userName: "Nabeela",
userKarma: 915
},
{
"type": "photos",
"id": "78",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}]
}
}
```
For collections. I am not confident but try this
Now for collection of galleries.
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
},{
"type": "galleries",
"id": "2",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "79"
}
}],
"included":[{
"type": "photos",
"id": "77",
"attributes":{
userName: "Nabeela",
userKarma: 915
},{
"type": "photos",
"id": "78",
"attributes":{
userName: "Nabeela",
userKarma: 915
},{
"type": "photos",
"id": "79",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}]
}
}
============Implementation part =================================
JSONSerializer normalization process follows these steps
*normalizeResponse : entry method.
*normalizeCreateRecordResponse : a normalizeResponse for specific operation.
*normalizeSingleResponse|normalizeArrayResponse:
- for methods like createRecord. we expect a single record back.
- for methods like findAll we expect multiple records back.
+normalize =
normalizeArray iterates and calls normalize for each of it's records
normalizeSingle call its once.
+extractID | extractAttributes | extractRelationships
= normalize delegates to these method to turn record payload into jsonAPI format
Starting with normalizeResponse method. If you open and see normalizeResponse method
in json-serializer
link normalizeResponse: https://github.com/emberjs/data/blob/v2.2.1/packages/ember-
data/lib/serializers/json-serializer.js#L192
you with find a switch case switch(requestType). If requestType if
"findRecord" then "normalizeFindRecordResponse" is called
"queryRecord" then "normalizeQueryRecordResponse" is called
"findAll" then "normalizeFindAllResponse" is called
...so on and so forth.
if you notice the parameter passed to all the methods are same as that of normalize
(...arguments) :)
**Lets start for findAll
i.e normalizeResponse -> normalizeFindAllResponse -> normalizeArrayResponse
as normalizeFindAllResponse method has only one line that call
normalizeArrayResponse.
normalizeFindAllResponse
normalizeResponse -> normalizeFindAllResponse -> normalizeArrayResponse ->
_normalizeResponse{ extractMeta,normalize }
extractMeta [extract meta information like pagination and stuff ]
if single: normalize []
example of normalize method in emberjs docs
```
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
normalize: function(typeClass, hash) {
var fields = Ember.get(typeClass, 'fields');
fields.forEach(function(field) {
var payloadField = Ember.String.underscore(field);
if (field === payloadField) { return; }
hash[field] = hash[payloadField];
delete hash[payloadField];
});
return this._super.apply(this, arguments);
}
});
```
"normalizeArrayResponse calls `return this._normalizeResponse
(store,primaryModelClass,payload,id,requestType,false).
so isSingle is false for _normalizeResponse method. so we will have to push all the
related records of included array
in our case the photos which is done by below snippet from "_normalizeRespose"
method.
_normalizeResponse
```
else{
documentHash.data = payload.map((item) => {
let { data, included } = this.normalize(primaryModelClass,item);
if(included){
documentHash.included.push(...included);
}
return data;
});
return documentHash;
}
```
Things are still unclear in the context of our JSON reponse from server
but atleast we know the flow now.
Lets try to apply it for findAll ( as per the flow above).
run "ember g serializer application" //assuming you are using ember-cli and you
intend to make this serializer generic for application.
As of now I have no information how and when normalizeResponse is called. :(
I just scanned through and guess on recieving data from server the store calls
normalizeResponseHelpers which in turn calls normalizeResponse.
In any case "normalizeResponse" is going to send payload and other necessar
information to normalizeFindAllResponse(...arguments) which in turn will call
normalizeArrayResponse(...arguments) which in turn will call "_normalizeRespone".
Here is where we need to take action
for extraMeta and normalize.
+extraMeta
I am not sure if there is any meta information in you json response.
in case there is you can refer to the example mentioned in docs
extractMeta
So I guess you can directly user the normalize method from example in your application ;).
please try and check. Since i am learning ember myself I cannot guarantee it will work but it should. the lonngggg explation is my thought while i was learning the problem/solution
//app/serializers/application.js
+normalize
```
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
normalize: function(typeClass, hash) {
var fields = Ember.get(typeClass, 'fields');
fields.forEach(function(field) {
var payloadField = Ember.String.underscore(field);
if (field === payloadField) { return; }
hash[field] = hash[payloadField];
delete hash[payloadField];
});
return this._super.apply(this, arguments);
}
});
```
The primary key in the JSON from server is pk. You will have to mention that too
http://emberjs.com/api/data/classes/DS.JSONSerializer.html#property_primaryKey
app/serializers/application.js
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
primaryKey: 'pk'
});

creating JSON object without using a list

I want to be able to create a JSON object so that I can access it like this.
education.schools.UNCC.graduation
Currently, my JSON is like this:
var education = {
"schools": [
"UNCC": {
"graduation": 2015,
"city": "Charlotte, NC",
"major": ["CS", "Spanish"]
},
"UNC-CH": {
"graduation": 2012,
"city": "Chapel Hill, NC"
"major": ["Sociology", "Film"]
}
],
"online": {
"website": "Udacity",
"courses": ["python", "java", "data science"]
}
};
When I go to Lint my JSON, I get an error message.
I know I can reformat my object to access it like this (below), but I don't want to do it this way. I want to be able to call the school name, and not use an index number.
education.schools[1].graduation
Objects have named keys. Arrays are a list of members.
Replace the value of "schools" with an object. Change [] to {}.
This is your JSON corrected.
Your JSON is invalid.
{
"schools": [
{
"UNCC": {
"graduation": "2015",
"city": [
"CS",
"Spanish"
],
"major": [
"CS",
"Spanish"
]
}
},
{
"UNC-CH": {
"graduation": "2012",
"city": [
"Chapel Hill",
"NC"
],
"major": [
"Sociology",
"Film"
]
}
}
],
"online": {
"website": "Udacity",
"courses": [
"python",
"java",
"data science"
]
}
}
Explanation:
"city": "Chapel Hill, NC" -> this is a array with 2 values "Chapel Hill" and "HC", like you do with major and courses.
The Schools array, you need to use this sintaxe to construct a array [{
http://adobe.github.io/Spry/samples/data_region/JSONDataSetSample.html

Backbone.js Collections

I am working with backbone.js. I am trying to send a request to restful service i am getting the resultset as json object as shown
{
"Msgs": [
"Alert",
"Not"
],
"MessageStatus": [
"Active",
"Inactive"
],
"date": {
"From": "2013-04-25",
"To": "2013-06-25"
},
"Mlist": {
"Status": "PND",
"Role": "Admin,User",
"To": "2013-06-24",
"Id": 6,
"Datecreated": "2013-06-24",
"Title": "Title5",
"From": "2013-06-20"
}
}.
I am putting the json object extracting and setting it to collection but I am not able to get particular model from the collection with specific id.
If you want to make your model can be identified with id. You have to set the id in the attributes hash:
{
"id": 1001,
"Msgs": [
"Alert",
"Not"
],
"MessageStatus": [
"Active",
"Inactive"
],
"date": {
"From": "2013-04-25",
"To": "2013-06-25"
},
"Mlist": {
"Status": "PND",
"Role": "Admin,User",
"To": "2013-06-24",
"Id": 6,
"Datecreated": "2013-06-24",
"Title": "Title5",
"From": "2013-06-20"
}
}
Then using backbone collection "findWhere" method to get the specific id model.
//assume msgCollection is which you put the models
var model = msgCollection.findWhere({id: 1001});
Hope this is helpful for you.
I think you need to set your model and parse your input data. Checkout http://backbonejs.org/#Collection-model, http://backbonejs.org/#Model-parse and http://backbonejs.org/#Collection-parse.