Transform the payload by grouping the fields in mule 4 dataweave - json

The below JSON Response containing collection which needs to be iterate and group according to the field serviceType.
Response:
{
data:[
{
"jobCardNumber": "JC20026333",
"serviceType": "RUNNING REPAIR",
"partAmt": 1659.98,
"labourAmt": 727.48,
"totalAmt": 2387.46,
"estPartAmt": 0.0,
"estLabourAmt": 0.0,
"estTotalAmt": 0.0,
"partBasic": 1356.12,
"partDiscount": null,
"labourBasic": 616.5,
"variant": "MARUTI WAGON R LXI BS4",
"color": "SUPERIOR WHITE"
},
{
"jobCardNumber": "JC20026456",
"serviceType": "BODY REPAIR",
"partAmt": 1659.98,
"labourAmt": 727.48,
"totalAmt": 2387.46,
"estPartAmt": 0.0,
"estLabourAmt": 0.0,
"estTotalAmt": 0.0,
"partBasic": 1356.12,
"partDiscount": null,
"labourBasic": 616.5,
"variant": "MARUTI WAGON R LXI BS4",
"color": "SUPERIOR WHITE"
},
{
"jobCardNumber": "JC20026456",
"serviceType": "RUNNING REPAIR",
"partAmt": 1659.98,
"labourAmt": 727.48,
"totalAmt": 2387.46,
"estPartAmt": 0.0,
"estLabourAmt": 0.0,
"estTotalAmt": 0.0,
"partBasic": 1356.12,
"partDiscount": null,
"labourBasic": 616.5,
"variant": "MARUTI WAGON R LXI BS4",
"color": "SUPERIOR WHITE"
}
]
}
Each collection having field serviceType, I need to group and transform the data according to the type, meaning that, if service type is same then we need to group the related fields and create another collection of it called history.
The below JSON depicts the scenario, for service type RUNNING REPAIR, another collection has been created which groups all the properties related to RUNNING REPAIR
{
"service_records": [
{
"service_type": "RUNNING REPAIR",
totalAmt:2800"
"history": [
{
"service_date": "08-07-22 23:17",
"mileage": "23465",
"jc_number": "JC22212345",
"labour_amount": "600",
"part_amount": "800",
"total_amount": "1400",
},
{
"service_date": "08-07-22 23:17",
"mileage": "23465",
"jc_number": "JC22212398",
"labour_amount": "600",
"part_amount": "800",
"total_amount": "1400",
}
]
},
{
"service_type": "BODY REPAIR",
"total_amount": "1400",
"history": [
{
"service_date": "08-07-22 23:17",
"mileage": "23465",
"jc_number": "JC22212345",
"labour_amount": "600",
"part_amount": "800",
"total_amount": "1400",
}
],
"WORKSHOP":{ //exclude body repair all other services will be count in workshop
totalSumOfAllServices: 4774 //(running repair)
}
}
]
}
I did some workaround in dataweave playground but unable to group the fields according to the serviceType
%dw 2.0
output application/json
---
service_records: payload.data map (item, index) -> {
service_type: item.serviceType,
labour_amount: item.labourAmt,
part_amount: item.partAmt,
total_amount: item.totalAmt,
is_expanded: false,
history:payload.data map(item,v)->{
"service_date": "08-07-22 23:17",
"service_type": item.serviceType,
"model": item.model,
"mileage": item.mileage,
"jc_number": item.jobCardNumber,
"labour_amount": item.labourAmt,
"part_amount": item.partAmt,
"total_amount": item.totalAmt
}
}
https://developer.mulesoft.com/learn/dataweave/playground?projectMethod=GHRepo&repo=mulesoft%2Fdocs-dataweave&path=modules%2FROOT%2Fpages%2F_partials%2Fcookbook-dw%2Fmap-ex1

The groupBy function is what you need.
You can use it to group by the serviceType of the input and it will give you output which will look something like this:
{
"RUNNING REPAIR": [
{"jobCardNumber": "JC20026333", .....},
{"jobCardNumber": "JC20026456", .....},
],
"BODY REPAIR": [
{"jobCardNumber": "JC20026456",....}
]
}
The above is the result of payload.data groupBy $.serviceType Which you can pluck and map to desired output
%dw 2.0
output application/json
---
{
service_records: payload.data groupBy $.serviceType
pluck ((groupedValues, serviceType, index) -> {
service_type: serviceType,
model: groupedValues[0].variant, // mabe?
totalAmt: sum(groupedValues.totalAmt),
history: groupedValues map {
service_date: "", // not sure from where it is coming from it is not in the input that you shared
mileage: "", // not sure from where it is coming from it is not in the input that you shared
jc_number: $.jobCardNumber,
labour_amount: $.labourAmt,
part_amount: $.partAmt,
total_amount: $.totalAmt
}
})
}

Related

Filtering JSON data with filter function in React- Redux

I have a JSON file with the following format:
const data = [
{
"quantity": "200",
"prodType": "stock",
"symbol": "LOL",
"prodDesc": "Εθνική τράπεζα",
"market": "Greece",
"averageCost": "131,16",
"totalCost": "123,47",
"lastPrice": "121,123",
"value": "123,34",
"positionPercentage": "10",
"valueEUR": "113,23",
"pl": "1300",
"plPercentage": "12",
"plEuro": "1238",
"results": "12-01-2017",
"dividend": "12-03-2017",
"isin": "1234566"
}
]
I want to filter the data using their Product Type.
As a result, im creating an action
export function fetchSearchData(product_type) {
const filtered_data = data.filter(record=>
{
return record.prodType.match(product_type)
});
return {
type: FETCH_SEARCH_DATA,
payload: filtered_data
};
}
But it does not seem to work. Do you have any idea why?

parse json output for primary and secondary hosts from replSetGetStatus

I've used pymongo to connect to mongo replica set and print the status of replica set using json dump. I want to parse this output and display "name" and "stateStr" into a list or array for the user to be able to pick a particular host.Here is my json dump output:
{
{
"replSetGetStatus": {
"date": "2016-10-07T14:21:25",
"members": [
{
"_id": 0,
"health": 1.0,
"name": "xxxxxxxxxxx:27017",
"optime": null,
"optimeDate": "2016-10-07T13:50:11",
"self": true,
"state": 1,
"stateStr": "PRIMARY",
"uptime": 32521
},
{
"_id": 1,
"health": 1.0,
"lastHeartbeat": "2016-10-07T14:21:24",
"lastHeartbeatRecv": "2016-10-07T14:21:24",
"name": "xxxxxxxxxxxx:27017",
"optime": null,
"optimeDate": "2016-10-07T13:50:11",
"pingMs": 0,
"state": 2,
"stateStr": "SECONDARY",
"syncingTo": "xxxxxxxxxxxx:27017",
"uptime": 27297
},
{
"_id": 2,
"health": 1.0,
"lastHeartbeat": "2016-10-07T14:21:24",
"lastHeartbeatRecv": "2016-10-07T14:21:24",
"name": "xxxxxxxxxxxxx:27020",
"pingMs": 0,
"state": 7,
"stateStr": "ARBITER",
"uptime": 32517
}
],
"myState": 1,
"ok": 1.0,
"set": "replica1"
}
}
Please try below Javascript code. It worked for me.
use admin;
var result = rs.status();
var length = result.members.length;
for (var i=0;i<length;i++){
print ("Server Name-" +result.members[i].name);
print ("Server State-" +result.members[i].stateStr);
}

Serialize the response from backend to store ember store

My response from backend is not in form which ember store. I am not able to serialize the response.
response.json
[{
"pk": 127,
"url": "http://example.com/api/galleries/127/",
"gallery_name": "Faces",
"thumbnail_url": "https://example.cloud.net/galleryThumbs/2656a05c-4ec7-3eea-8c5e-d8019454d443.jpg",
"time": "1 month ago",
"description": "Created by user",
"is_following": true,
"feedPhotos": [{
"pk": 624,
"url": "http://example.com/api/photos/624/",
"profilePic": "https://example.cloud.net/userDPs/50906ce2-394d-39c8-9261-8cf78e3611c2.jpg",
"userName": "Nabeela",
"userKarma": 915,
"caption": "Old woman spinning her 'chhos-khor' ...a rotation of which is equivalent to the recitation of a mantra.",
"numComments": 0,
"owner": "http://example.com/api/users/44/",
"time": "1 month ago",
"photo_url": "https://example.cloud.net/photos/9cbd6423-3bc5-36e0-b8b4-d725efb3249a.jpg",
"comments_url": "http://example.com/api/photos/624/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/1362/fivers/",
"fivers_pk": 1362,
"fullphoto_url": "http://example.com/api/photogalleries/1362/photo/",
"fullphoto_pk": 1362,
"is_fived": true,
"hiFiveKarma": 1,
"owner_pk": 44,
"userFirstName": "Nabeela",
"is_bookmarked": false
}, {
"pk": 574,
"url": "http://example.com/api/photos/574/",
"profilePic": "https://example.cloud.net/userDPs/b6f69e4e-980d-3cc3-8b3e-3eb1a7f21350.jpg",
"userName": "Rohini",
"userKarma": 194,
"caption": "Life # Myanmar!",
"numComments": 0,
"owner": "http://example.com/api/users/45/",
"time": "2 months ago",
"photo_url": "https://example.cloud.net/photos/eeae72d5-d6af-391e-a218-b442c0c7e34e.jpg",
"comments_url": "http://example.com/api/photos/574/comments/",
"numFives": 2,
"fivers_url": "http://example.com/api/photogalleries/1303/fivers/",
"fivers_pk": 1303,
"fullphoto_url": "http://example.com/api/photogalleries/1303/photo/",
"fullphoto_pk": 1303,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 45,
"userFirstName": "Rohini",
"is_bookmarked": false
}
]
}, {
"pk": 65,
"url": "http://example.com/api/galleries/65/",
"gallery_name": "Royal",
"thumbnail_url": "https://example.cloud.net/galleryThumbs/d8a900af-1f1d-3977-8cc8-b8bb36e32be5.jpg",
"time": "2 months ago",
"description": "This is a gallery about Royal",
"is_following": false,
"feedPhotos": [{
"pk": 347,
"url": "http://example.com/api/photos/347/",
"profilePic": "https://example.cloud.net/userDPs/50906ce2-394d-39c8-9261-8cf78e3611c2.jpg",
"userName": "Nabeela",
"userKarma": 915,
"caption": "I cannot forget the name of this palace - Moti Mahal (translation: Pearl Palace). Indescribably beautiful, ainnit! at Mehrangarh fort, Jodhp",
"numComments": 0,
"owner": "http://example.com/api/users/44/",
"time": "2 months ago",
"photo_url": "https://example.cloud.net/photos/958ed406-708e-3f01-a2f4-9467cd709fdd.jpg",
"comments_url": "http://example.com/api/photos/347/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/759/fivers/",
"fivers_pk": 759,
"fullphoto_url": "http://example.com/api/photogalleries/759/photo/",
"fullphoto_pk": 759,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 44,
"userFirstName": "Nabeela",
"is_bookmarked": false
}, {
"pk": 593,
"url": "http://example.com/api/photos/593/",
"profilePic": "https://example.cloud.net/userDPs/95ac6974-f7df-338c-ab84-99fa1df7514c.jpg",
"userName": "Vikanshu",
"userKarma": 932,
"caption": "Marvelous architecture!! in Florence, Italy",
"numComments": 0,
"owner": "http://example.com/api/users/48/",
"time": "1 month ago",
"photo_url": "https://example.cloud.net/photos/7a86eb37-6c68-3d6c-b6cf-2e3b74d330dd.jpg",
"comments_url": "http://example.com/api/photos/593/comments/",
"numFives": 4,
"fivers_url": "http://example.com/api/photogalleries/1363/fivers/",
"fivers_pk": 1363,
"fullphoto_url": "http://example.com/api/photogalleries/1363/photo/",
"fullphoto_pk": 1363,
"is_fived": false,
"hiFiveKarma": 0,
"owner_pk": 48,
"userFirstName": "Vikanshu",
"is_bookmarked": false
}]
}]
How do I serialize this using JSONPISerailizer or any other serializer in ember-cli so that it gets stored in ember store
Reference jsonapi.org
++++Top Level:
Root:
A JSON object must be root of every JSON API request response.
A document must contain at least one top-level members:
1. data: documents "primary data"
2. errors: an array of error objects (id,status,code,title....)
3. meta: a meta object that contains non-standard meta-information (copyright,author...)
member data and errors must not co-exist together.
"data"{}
+++++Resource Objects
1. A resource object MUST contain atleast following top-level member
*id
*type
```
//structure-1
//for galleries
{
"data": {
"type": "galleries",
"id": "1"
}
}
//for photos
{
"data": {
"type": "photos",
"id": "1"
}
}
```
In addition, a resource object may contain any of these top-level members
*attributes
*relationship
*links
*meta
//added attributes first
```
//structure-2
//for galleries
{
"data": {
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
}
}
}
//for photos
{
"data": {
"type": "photos",
"id": "1",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}
}
```
//Adding relationship
Relationship object must contain atleast one of the following
*links (containing atleast one of "self" or "related" resource link
*data
*meta
//link in relationship (minimum one required from link,data,meta).
//
```
//structure-3
//for galleries
{
"data":[{ //Array(square bracket as adding relationship one more item to data
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
}]
}
}
```
//data in relationship
```
//structure-4
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
}]
}
}
```
//Adding related resource "included"
```
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
}],
"included":[{
"type": "photos",
"id": "77",
"attributes":{
userName: "Nabeela",
userKarma: 915
},
{
"type": "photos",
"id": "78",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}]
}
}
```
For collections. I am not confident but try this
Now for collection of galleries.
//for galleries
{
"data": [{
"type": "galleries",
"id": "1",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "77"
}
},{
"type": "galleries",
"id": "2",
"attributes": {
"galleryName": "Faces"
"thumbnailUrl:"https://example.cloud.net/galleryThumbs/2656a05c-4ec7.jpg",
"description": "Created by user",
},
"relationships":{ //This has all your photo stuff
"links":{
"self": "https://example.cloud.net/photos/9cbd6423.jpg //"photo_url" in your payload
},
"data": { //picked it up from structure-1
"type": "photos",
"id": "79"
}
}],
"included":[{
"type": "photos",
"id": "77",
"attributes":{
userName: "Nabeela",
userKarma: 915
},{
"type": "photos",
"id": "78",
"attributes":{
userName: "Nabeela",
userKarma: 915
},{
"type": "photos",
"id": "79",
"attributes":{
userName: "Nabeela",
userKarma: 915
}
}]
}
}
============Implementation part =================================
JSONSerializer normalization process follows these steps
*normalizeResponse : entry method.
*normalizeCreateRecordResponse : a normalizeResponse for specific operation.
*normalizeSingleResponse|normalizeArrayResponse:
- for methods like createRecord. we expect a single record back.
- for methods like findAll we expect multiple records back.
+normalize =
normalizeArray iterates and calls normalize for each of it's records
normalizeSingle call its once.
+extractID | extractAttributes | extractRelationships
= normalize delegates to these method to turn record payload into jsonAPI format
Starting with normalizeResponse method. If you open and see normalizeResponse method
in json-serializer
link normalizeResponse: https://github.com/emberjs/data/blob/v2.2.1/packages/ember-
data/lib/serializers/json-serializer.js#L192
you with find a switch case switch(requestType). If requestType if
"findRecord" then "normalizeFindRecordResponse" is called
"queryRecord" then "normalizeQueryRecordResponse" is called
"findAll" then "normalizeFindAllResponse" is called
...so on and so forth.
if you notice the parameter passed to all the methods are same as that of normalize
(...arguments) :)
**Lets start for findAll
i.e normalizeResponse -> normalizeFindAllResponse -> normalizeArrayResponse
as normalizeFindAllResponse method has only one line that call
normalizeArrayResponse.
normalizeFindAllResponse
normalizeResponse -> normalizeFindAllResponse -> normalizeArrayResponse ->
_normalizeResponse{ extractMeta,normalize }
extractMeta [extract meta information like pagination and stuff ]
if single: normalize []
example of normalize method in emberjs docs
```
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
normalize: function(typeClass, hash) {
var fields = Ember.get(typeClass, 'fields');
fields.forEach(function(field) {
var payloadField = Ember.String.underscore(field);
if (field === payloadField) { return; }
hash[field] = hash[payloadField];
delete hash[payloadField];
});
return this._super.apply(this, arguments);
}
});
```
"normalizeArrayResponse calls `return this._normalizeResponse
(store,primaryModelClass,payload,id,requestType,false).
so isSingle is false for _normalizeResponse method. so we will have to push all the
related records of included array
in our case the photos which is done by below snippet from "_normalizeRespose"
method.
_normalizeResponse
```
else{
documentHash.data = payload.map((item) => {
let { data, included } = this.normalize(primaryModelClass,item);
if(included){
documentHash.included.push(...included);
}
return data;
});
return documentHash;
}
```
Things are still unclear in the context of our JSON reponse from server
but atleast we know the flow now.
Lets try to apply it for findAll ( as per the flow above).
run "ember g serializer application" //assuming you are using ember-cli and you
intend to make this serializer generic for application.
As of now I have no information how and when normalizeResponse is called. :(
I just scanned through and guess on recieving data from server the store calls
normalizeResponseHelpers which in turn calls normalizeResponse.
In any case "normalizeResponse" is going to send payload and other necessar
information to normalizeFindAllResponse(...arguments) which in turn will call
normalizeArrayResponse(...arguments) which in turn will call "_normalizeRespone".
Here is where we need to take action
for extraMeta and normalize.
+extraMeta
I am not sure if there is any meta information in you json response.
in case there is you can refer to the example mentioned in docs
extractMeta
So I guess you can directly user the normalize method from example in your application ;).
please try and check. Since i am learning ember myself I cannot guarantee it will work but it should. the lonngggg explation is my thought while i was learning the problem/solution
//app/serializers/application.js
+normalize
```
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
normalize: function(typeClass, hash) {
var fields = Ember.get(typeClass, 'fields');
fields.forEach(function(field) {
var payloadField = Ember.String.underscore(field);
if (field === payloadField) { return; }
hash[field] = hash[payloadField];
delete hash[payloadField];
});
return this._super.apply(this, arguments);
}
});
```
The primary key in the JSON from server is pk. You will have to mention that too
http://emberjs.com/api/data/classes/DS.JSONSerializer.html#property_primaryKey
app/serializers/application.js
import DS from 'ember-data';
export default DS.JSONSerializer.extend({
primaryKey: 'pk'
});

How to convert a json file without the same length of json objects into csv

I have a json file and I want to convert it to csv format.
The problem I face is that every json object in the file has not the same length of the converted columns I have. For example the one object have 49 columnns and the next have 50.
I provide here an example of 2 data from which the first one has not the creator.slug but the next has it is and so there is the problem with data. The problem is that the process create all 50 columns but for the object which don't have the value creator.slug it takes the next price.
{
"id": 301852363,
"name": "Song of the Sea",
"blurb": "One evening, two shows: SIRENS and The Girl From Bare Cove. Building a community. Giving voice to survivors of sexual violence.",
"goal": 5000,
"pledged": 671,
"state": "live",
"slug": "song-of-the-sea",
"disable_communication": false,
"country": "US",
"currency": "USD",
"currency_symbol": "$",
"currency_trailing_code": true,
"deadline": 1399293386,
"state_changed_at": 1397133386,
"created_at": 1396672480,
"launched_at": 1397133386,
"backers_count": 20,
"photo": {
"full": "https://s3.amazonaws.com/ksr/projects/939387/photo-full.jpg?1397874930",
"ed": "https://s3.amazonaws.com/ksr/projects/939387/photo-ed.jpg?1397874930",
"med": "https://s3.amazonaws.com/ksr/projects/939387/photo-med.jpg?1397874930",
"little": "https://s3.amazonaws.com/ksr/projects/939387/photo-little.jpg?1397874930",
"small": "https://s3.amazonaws.com/ksr/projects/939387/photo-small.jpg?1397874930",
"thumb": "https://s3.amazonaws.com/ksr/projects/939387/photo-thumb.jpg?1397874930",
"1024x768": "https://s3.amazonaws.com/ksr/projects/939387/photo-1024x768.jpg?1397874930",
"1536x1152": "https://s3.amazonaws.com/ksr/projects/939387/photo-1536x1152.jpg?1397874930"
},
"creator": {
"id": 1714048992,
"name": "Maridee Slater",
"slug": "maridee",
"avatar": {
"thumb": "https://s3.amazonaws.com/ksr/avatars/996153/DSC_0310.thumb.jpg?1337713264",
"small": "https://s3.amazonaws.com/ksr/avatars/996153/DSC_0310.small.jpg?1337713264",
"medium": "https://s3.amazonaws.com/ksr/avatars/996153/DSC_0310.medium.jpg?1337713264"
},
"urls": {
"web": {
"user": "https://www.kickstarter.com/profile/maridee"
},
"api": {
"user": "https://api.kickstarter.com/v1/users/1714048992?signature=1398256877.e6d63adcca055cd041a5920368b197d40459f748"
}
}
},
"location": {
"id": 2459115,
"name": "New York",
"slug": "new-york-ny",
"short_name": "New York, NY",
"displayable_name": "New York, NY",
"country": "US",
"state": "NY",
"urls": {
"web": {
"discover": "https://www.kickstarter.com/discover/places/new-york-ny",
"location": "https://www.kickstarter.com/locations/new-york-ny"
},
"api": {
"nearby_projects": "https://api.kickstarter.com/v1/discover?signature=1398256786.89b2c4539aeab4ad25982694dd7e659e8c12028f&woe_id=2459115"
}
}
},
"category": {
"id": 17,
"name": "Theater",
"slug": "theater",
"position": 14,
"urls": {
"web": {
"discover": "http://www.kickstarter.com/discover/categories/theater"
}
}
},
"urls": {
"web": {
"project": "https://www.kickstarter.com/projects/maridee/song-of-the-sea"
}
}
},
{
"id": 967108708,
"name": "Good Bread Alley",
"blurb": "A play by April Yvette Thompson. A Gullah Healer Woman and an Afro-Cuban Priest forge a new world of magic & dreams in Jim Crow Miami.",
"goal": 100000,
"pledged": 33242,
"state": "live",
"slug": "good-bread-alley",
"disable_communication": false,
"country": "US",
"currency": "USD",
"currency_symbol": "$",
"currency_trailing_code": true,
"deadline": 1399271911,
"state_changed_at": 1396334313,
"created_at": 1393278556,
"launched_at": 1396334311,
"backers_count": 261,
"photo": {
"full": "https://s3.amazonaws.com/ksr/projects/883489/photo-full.jpg?1397869394",
"ed": "https://s3.amazonaws.com/ksr/projects/883489/photo-ed.jpg?1397869394",
"med": "https://s3.amazonaws.com/ksr/projects/883489/photo-med.jpg?1397869394",
"little": "https://s3.amazonaws.com/ksr/projects/883489/photo-little.jpg?1397869394",
"small": "https://s3.amazonaws.com/ksr/projects/883489/photo-small.jpg?1397869394",
"thumb": "https://s3.amazonaws.com/ksr/projects/883489/photo-thumb.jpg?1397869394",
"1024x768": "https://s3.amazonaws.com/ksr/projects/883489/photo-1024x768.jpg?1397869394",
"1536x1152": "https://s3.amazonaws.com/ksr/projects/883489/photo-1536x1152.jpg?1397869394"
},
"creator": {
"id": 749318998,
"name": "April Yvette Thompson",
"avatar": {
"thumb": "https://s3.amazonaws.com/ksr/avatars/9751919/kick_thumb.thumb.jpg?1396128151",
"small": "https://s3.amazonaws.com/ksr/avatars/9751919/kick_thumb.small.jpg?1396128151",
"medium": "https://s3.amazonaws.com/ksr/avatars/9751919/kick_thumb.medium.jpg?1396128151"
},
"urls": {
"web": {
"user": "https://www.kickstarter.com/profile/749318998"
},
"api": {
"user": "https://api.kickstarter.com/v1/users/749318998?signature=1398256877.af4db50c53f93339b05c7813f4534e833eaca270"
}
}
},
"location": {
"id": 2459115,
"name": "New York",
"slug": "new-york-ny",
"short_name": "New York, NY",
"displayable_name": "New York, NY",
"country": "US",
"state": "NY",
"urls": {
"web": {
"discover": "https://www.kickstarter.com/discover/places/new-york-ny",
"location": "https://www.kickstarter.com/locations/new-york-ny"
},
"api": {
"nearby_projects": "https://api.kickstarter.com/v1/discover?signature=1398256786.89b2c4539aeab4ad25982694dd7e659e8c12028f&woe_id=2459115"
}
}
},
"category": {
"id": 17,
"name": "Theater",
"slug": "theater",
"position": 14,
"urls": {
"web": {
"discover": "http://www.kickstarter.com/discover/categories/theater"
}
}
},
"urls": {
"web": {
"project": "https://www.kickstarter.com/projects/749318998/good-bread-alley"
}
}
}
Here is the code I run
#open the json file
require(RJSONIO)
require(rjson)
library("rjson")
filename2 <- "C:/Users/Desktop/in.json"
json_data <- fromJSON(file = filename2)
#unlist the json because it has a problem
unlisted <- unlist(unlist(json_data,recursive=FALSE),recursive=FALSE)
use to fill the NA but as I can understand now it is for already existed nulls http://stackoverflow.com/questions/16947643/getting-imported-json-data-into-a-data-frame-in-r/16948174#16948174
unlisted <- lapply(unlisted, function(x) {
x[sapply(x, is.null)] <- NA
unlist(x)
})
json <- do.call("rbind", unlisted)
Here is a full list with the columns of the output csv and after that I provide what I would like to keep from every object of json, less columns
id
name
blurb
goal
pledged
state
slug
disable_communication
country
currency
currency_symbol
currency_trailing_code
deadline
state_changed_at
created_at
launched_at
backers_count
photo.full
photo.ed
photo.med
photo.little
photo.small
photo.thumb
photo.1024x768
photo.1536x1152
creator.id
creator.name
creator.slug
creator.avatar.thumb
creator.avatar.small
creator.avatar.medium
creator.urls.web.user
creator.urls.api.user
location.id
location.name
location.slug
location.short_name
location.displayable_name
location.country
location.state
location.urls.web.discover
location.urls.web.location
location.urls.api.nearby_projects
category.id
category.name
category.slug
category.position
category.urls.web.discover
category.urls.web.project
category.urls.web.rewards
Here it is the list of columns I would try to have in the output csv:
id
name
blurb
goal
pledged
state
slug
disable_communication
country
currency
currency_symbol
currency_trailing_code
deadline
state_changed_at
created_at
launched_at
backers_count
creator.id
creator.name
creator.slug
location.id
location.name
location.slug
location.short_name
location.displayable_name
location.country
location.state
category.id
category.name
category.slug
category.position
Looks like there's a very similar question (with answer, though not pure R) here: convert json to csv format
However, since you do seem to want most, if not all, the JSON in a "wide CSV" format you can use fromJSON from jsonlite, rbindlist from data.table (which gets you the fill=TRUE parameter to handle uneven lists nicely) and unlist:
library(jsonlite)
library(data.table)
# tell fromJSON we want a list back
json_data <- fromJSON("in.json", simplifyDataFrame=FALSE)
# iterate over the list we have so we can "flatten" it then
# covert it back to a data.frame-like object
dat <- rbindlist(lapply(json_data, function(x) {
as.list(unlist(x))
}), fill=TRUE)
You may need to tweak column names, but I think this gets you what you're looking for.

Parsing Json In Golang

Have been trying to parse this json file using go to get the Minimum and Maximum Temperature
of a given city.
{
"data": {
"current_condition": [
{
"cloudcover": "25",
"humidity": "56",
"observation_time": "01:33 PM",
"precipMM": "0.0",
"pressure": "1016",
"temp_C": "20",
"temp_F": "68",
"visibility": "10",
"weatherCode": "116",
"weatherDesc": [
{
"value": "Partly Cloudy"
}
],
"weatherIconUrl": [
{
"value": "http://cdn.worldweatheronline.net/images/wsymbols01_png_64/wsymbol_0002_sunny_intervals.png"
}
],
"winddir16Point": "SSW",
"winddirDegree": "210",
"windspeedKmph": "7",
"windspeedMiles": "4"
}
],
"request": [
{
"query": "London, United Kingdom",
"type": "City"
}
],
"weather": [
{
"date": "2014-09-07",
"precipMM": "0.0",
"tempMaxC": "23",
"tempMaxF": "74",
"tempMinC": "10",
"tempMinF": "49",
"weatherCode": "119",
"weatherDesc": [
{
"value": "Cloudy"
}
],
"weatherIconUrl": [
{
"value": "http://cdn.worldweatheronline.net/images/wsymbols01_png_64/wsymbol_0003_white_cloud.png"
}
],
"winddir16Point": "N",
"winddirDegree": "355",
"winddirection": "N",
"windspeedKmph": "9",
"windspeedMiles": "6"
}
]
}
}
Have been successful using structures and decoding the json string.
Now i want to try using maps, such as map[string]interface{}
If u is of type map[string]interface{} and the json is parsed into u,
u["data"].(map[string]interface{})["weather"]
gives the value
http://api.worldweatheronline.com/free/v1/weather.ashx?q=london&format=json&num_of_days=1&key=8c52bb73c5f6160f5f3aa535d22184638372d22b [map[tempMaxC:23 tempMaxF:74 tempMinC:10 tempMinF:49 winddirection:N windspeedMiles:6 date:2014-09-07 precipMM:0.0 weatherCode:119 winddir16Point:N winddirDegree:355 weatherDesc:[map[value:Cloudy]] weatherIconUrl:[map[value:http://cdn.worldweatheronline.net/images/wsymbols01_png_64/wsymbol_0003_white_cloud.png]] windspeedKmph:9]]
whereas u["data"].(map[string]interface{})["weather"].(map[string]interface{})["tempMinC"],
gives me an panic: interface conversion: interface is []interface {}, not map[string]interface {}
Can someone explain what's happening?
u["data"].(map[string]interface{})["weather"] is a slice of maps, not a map, so you would have to do something like :
maps, ok := u["data"].(map[string]interface{})["weather"].([]interface{})
if !ok {
panic("bad json")
}
for _, m := range maps {
if m, ok := m.(map[string]interface{}); ok {
fmt.Println(m["tempMinC"])
}
}
From your JSON example, weather is an array of objects so this translates to a slice of maps ([]map[string]interface{})in Go.
It's a slice not a map.
range the u["data"].(map[string]interface{})["weather"]