Where to put current user context data in response JSON? - json

Consider a social network. It has posts. For feed, you request /feed and get the list of posts.
In the UI, there are things to show for a post, like if the user liked the post or not, if the user starred it or not, etc. These things don't look like they belong inside the post object.
Another case is when you fetch the likes. The frontend needs to know if the user in each 'like' object is being followed or not.
Where to put this info in the response JSON?

Its depends on your application and which data you want to show to the user. For ex,consider you are listing a user's feeds. In that feed,you want to show
Message
Liked by the current user or not(i don't know the difference between liked and stared)
Number of likes
List of liked users.
shared by the user or not
Shared count
List ofShared users.
etc..
In the above list,
Some data need two api fetch to get complete info and some not. For example,"List of liked users","List of Shared users". This is generally a dynamic data module. You have to get those details in a separate api for better performance of the server and also data integrity.
In some cases,some apps needs sneak peek of the liked shared users info in the listing page. In that case,you can include the some fixed small number of users details in the same list /feeds response itself and include the "See More(like Facebook)" option in the UI.
Some static singular data(single column data) can be list in the initial get /feeds itself.
I wonder why don't you follow the same twitter's list tweets style,
https://dev.twitter.com/rest/reference/get/search/tweets
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Fri Sep 21 23:40:54 +0000 2012",
"id_str": "249292149810667520",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "FreeBandNames",
"indices": [
20,
34
]
}
],
"user_mentions": [
]
},
"in_reply_to_user_id_str": null,
"contributors": null,
"text": "Thee Namaste Nerdz. #FreeBandNames",
"metadata": {
"iso_language_code": "pl",
"result_type": "recent"
},
"retweet_count": 0,
"in_reply_to_status_id_str": null,
"id": 249292149810667520,
"geo": null,
"retweeted": false,
"in_reply_to_user_id": null,
"place": null,
"user":
{
"profile_sidebar_fill_color": "DDFFCC",
"profile_sidebar_border_color": "BDDCAD",
"profile_background_tile": true,
"name": "Chaz Martenstein",
"profile_image_url": "http://a0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
"created_at": "Tue Apr 07 19:05:07 +0000 2009",
"location": "Durham, NC",
"follow_request_sent": null,
"profile_link_color": "0084B4",
"is_translator": false,
"id_str": "29516238",
"entities": {
"url": {
"urls": [
{
"expanded_url": null,
"url": "http://bullcityrecords.com/wnng/",
"indices": [
0,
32
]
}
]
},
"description": {
"urls": [
]
}
},
"default_profile": false,
"contributors_enabled": false,
"favourites_count": 8,
"url": "http://bullcityrecords.com/wnng/",
"profile_image_url_https": "https://si0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
"utc_offset": -18000,
"id": 29516238,
"profile_use_background_image": true,
"listed_count": 118,
"profile_text_color": "333333",
"lang": "en",
"followers_count": 2052,
"protected": false,
"notifications": null,
"profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/9423277/background_tile.bmp",
"profile_background_color": "9AE4E8",
"verified": false,
"geo_enabled": false,
"time_zone": "Eastern Time (US & Canada)",
"description": "You will come to Durham, North Carolina. I will sell you some records then, here in Durham, North Carolina. Fun will happen.",
"default_profile_image": false,
"profile_background_image_url": "http://a0.twimg.com/profile_background_images/9423277/background_tile.bmp",
"statuses_count": 7579,
"friends_count": 348,
"following": null,
"show_all_inline_media": true,
"screen_name": "bullcityrecords"
},
"in_reply_to_screen_name": null,
"source": "web",
"in_reply_to_status_id": null
}

You have two options:
Make a separate API method for getting information about user context data - /api/users/1/feeds/1 .Pay attention, that this option will force you to send request per feed. So, if you'll have 1000 feeds - you will have 1000 + 1 request (so called N+1 problem).
As for me - it's not a good idea.
You can store user data in json, for example is this way:
{
"feedName": "feed1",
...
"currentUser": {
"liked": true,
"starred": true
}
}
By using this option you will avoid N+1 requests problem in your RESTful service

For all the users, the post resource should be the same. Adding specific user context info inside it seems like polluting it
I can see where you're coming from and I quite agree.
Ivan's 1st solution should not be used as he already mentioned, his 2nd is better but then if you GET the posts JSON which should contain only post objects, there is also this currentUser that doesn't really belong there.
My suggestion is that for each post you keep track of which users have liked and/or starred it, etc. Then you keep a clean structure while still having the info you need available in the same request/response.
Example
GET /feed HTTP/1.1
[
{
"text": "hello world, im a post!",
"author": "Jack",
"likes": 3,
"likedBy": [
"John",
"James",
"Jessica"
],
"stars": 2,
"starredBy": [
"John",
"Mary"
]
},
{
"text": "hello world, im also a post! :D",
"author": "Mary",
"likes": 1,
"likedBy": [
"James"
],
"stars": 0,
"starredBy": [
]
},
]
Where each {} object represents a post object.
On the client side, you could then check if the likedBy list contains the currently logged in user and proceed with the result as you see fit. Same for stars and any other of these properties a post might have.

Related

How can I see logs of the JSON post bodies sent by zapier to my CRM (Current RMS) via the Webhook zap during setup and testing?

I'm trying to send new users / new customres of my WooCommerce store into the rental management app current-rms.com as new Organisations / new contacts. Since Current RMS does not have a native Zap, I am trying to use the generic Webhook zap that Zapier maintains.
Specifically, I'd like to see the sent JSON body in Zapier posts that I make during the setup and testing of the Zap after clicking "Make a Zap!". The Task History is not detailed enough nor does it show hits during test and setup, since it's not live yet.
My trigger is a WooCommerce New Customer. This is working with Zapier WooCommerce Plugin and webhooks OK.
My action is the generic Zapier "Webhooks" Zap. The label "instant" appears next to it in the list at /app/zaps and it is "off".
One version uses JSON PAYLOAD as the action.
Another version uses CUSTOM PAYLOAD as the action.
Wrap request in array is YES.
Unflatten is YES.
My API key and subdomain are in the app URL as query strings and working OK.
When I hit test I get:
We had trouble sending your test through.
The app returned "Invalid JSON - missing or invalid entry for 'member'". This usually happens when your Zap is missing a required field or a field value isn't in a recognized format.
We made a request to api.current-rms.com and received (400) Bad Request.
Official docs are at: https://api.current-rms.com/doc#members-members-post
Logging available at Current RMS side
Part of the authentication of Current RMS involves knowing the domain of the account you are trying to access, in my case its therockfactory due to it being an account for my company https://therockfactory.net/
https://api.current-rms.com/api/v1/members?apikey=APIKEYCENSORED&subdomain=therockfactory
which returns the following when I use the correct API key:
{"webhook_logs":[],"meta":{"total_row_count":0,"row_count":0,"page":1,"per_page":20}}
Maybe if I could see the actual hit that Zapier is posting to Current I could wrap my confused brain around it better? What me worry.
The hit should look somewhat similar to this example, but I've not been able to locate it so far... (in Zapier)
Headers
Content-Type: application/json
Body
{
"member": {
"name": "Chris Bralton",
"description": "Pictures and leaned back was strewn at one would rather more. People don't want of his own means of one hand! Unless it from our pioneer has he fallen tree but that ever stronger and a. Hid among us against the full of verdure through by my eyes.",
"active": true,
"bookable": false,
"location_type": 0,
"locale": "en-GB",
"membership_type": "Contact",
"lawful_basis_type_id": 10001,
"sale_tax_class_id": 1,
"purchase_tax_class_id": 1,
"tag_list": [
"[\"Red\", \"Blue\", \"Green\"]"
],
"custom_fields": {},
"membership": {},
"primary_address": {
"name": "Chris Branson",
"street": "16 The Triangle",
"postcode": "NG2 1AE",
"city": "Nottingham",
"county": "Nottinghamshire",
"country_id": "1",
"country_name": "United Kingdom",
"type_id": 3001,
"address_type_name": "Primary",
"created_at": "2015-06-29T10:00:00.000Z",
"updated_at": "2015-06-29T10:30:00.000Z"
},
"emails": [
{
"address": "abigail.parker#ggmail.co.uk",
"type_id": 4001,
"email_type_name": "Work",
"id": 1
}
],
"phones": [
{
"number": "+44 115 9793399",
"type_id": 6001,
"phone_type_name": "Work",
"id": 1
}
],
"links": [
{
"address": "www.facebook.com/profile.php?id=566828251",
"type_id": 5002,
"link_type_name": "Facebook",
"id": 1
}
],
"addresses": [
{
"name": "Chris Branson",
"street": "16 The Triangle",
"postcode": "NG2 1AE",
"city": "Nottingham",
"county": "Nottinghamshire",
"country_id": "1",
"country_name": "United Kingdom",
"type_id": 3002,
"address_type_name": "Billing",
"created_at": "2017-06-29T10:00:00.000Z",
"updated_at": "2017-06-29T10:30:00.000Z",
"id": 1
}
],
"service_stock_levels": [
{
"item_id": 10,
"store_id": 1,
"member_id": 1,
"asset_number": "Chris Bralton",
"serial_number": "",
"location": "",
"stock_type": 3,
"stock_category": 60,
"quantity_held": "1.0",
"quantity_allocated": "0.0",
"quantity_unavailable": "0.0",
"quantity_on_order": "0.0",
"starts_at": "",
"ends_at": "",
"icon": {
"iconable_id": 85,
"id": 1,
"image_file_name": "abigail.jpeg",
"url": "https://s3.amazonaws.com/current-rms-development/64a0ccd0-5fbd-012f-2201-60f847290680/icons/46/original/abigail.jpeg",
"thumb_url": "https://s3.amazonaws.com/current-rms-development/64a0ccd0-5fbd-012f-2201-60f847290680/icons/46/thumb/abigail.jpeg",
"created_at": "2015-06-29T10:00:00.000Z",
"updated_at": "2015-06-29T10:30:00.000Z",
"iconable_type": "StockLevel"
},
"custom_fields": {},
"id": 487,
"item_name": "Sound Engineer",
"store_name": "Nottingham",
"stock_type_name": "Service",
"stock_category_name": "Resource"
}
],
"day_cost": "",
"hour_cost": "",
"distance_cost": "",
"flat_rate_cost": "",
"icon": {
"image": ""
},
"child_members": [
{
"relatable_id": 317,
"relatable_type": "Member",
"related_id": 25,
"related_type": "Member"
}
],
"parent_members": [
{
"relatable_id": 317,
"relatable_type": "Member",
"related_id": 25,
"related_type": "Member"
}
]
}
}
UPDATE: After reading my chosen answer I was able to see what Zapier was sending:
[
{
"member[emails_attributes][0][address]": "test#test.co.nz",
"member[membership_type]": "Organisation",
"member[name]": "Testafari Testing"
}
]
You can send your webhook to a tool like this one to inspect the payloads that are being sent from anywhere on the internet: https://requestbin.com/
You can find more help in regards to using Webhooks by Zapier and other ideas on how you can troubleshoot issues stemming from its use: https://zapier.com/apps/webhook/help#inspect-the-requests

Retrieving a specific object from JSON

This is a snippet of a JSON file that was returned by TMDB and I'm trying to access the title of every object. I've tried using the following methods like from this post How to access specific value from a nested array within an object array.
"results": [
{
"vote_count": 2358,
"id": 283366,
"video": false,
"vote_average": 6.5,
"title": "Miss Peregrine's Home for Peculiar Children",
"popularity": 20.662756,
"poster_path": "/AvekzUdI8HZnImdQulmTTmAZXrC.jpg",
"original_language": "en",
"original_title": "Miss Peregrine's Home for Peculiar Children",
"genre_ids": [
18,
14,
12
],
"backdrop_path": "/9BVHn78oQcFCRd4M3u3NT7OrhTk.jpg",
"adult": false,
"overview": "A teenager finds himself transported to an island where he must help protect a group of orphans with special powers from creatures intent on destroying them.",
"release_date": "2016-09-28"
},
{
"vote_count": 3073,
"id": 381288,
"video": false,
"vote_average": 6.8,
"title": "Split",
"popularity": 17.488396,
"poster_path": "/rXMWOZiCt6eMX22jWuTOSdQ98bY.jpg",
"original_language": "en",
"original_title": "Split",
"genre_ids": [
27,
53
],
"backdrop_path": "/4G6FNNLSIVrwSRZyFs91hQ3lZtD.jpg",
"adult": false,
"overview": "Though Kevin has evidenced 23 personalities to his trusted psychiatrist, Dr. Fletcher, there remains one still submerged who is set to materialize and dominate all the others. Compelled to abduct three teenage girls led by the willful, observant Casey, Kevin reaches a war for survival among all of those contained within him — as well as everyone around him — as the walls between his compartments shatter apart.",
"release_date": "2016-11-15"
},
var titles = results.map(function extract(item){return item.title})
The map function iterates through the array and builds the resulting array by applying the extract function on each item.
for (var i =0; i < obj.results.length; i++) {
console.log(obj.results[i].title);
}
First, we get the results key, and then, iterate over it since it is an array.
Results is nothing more than an array of objects, there's nothing nested.
You could use .forEach()
results.forEach(function(item){
console.log(item.title);
});

Convert to dataframe from JSON in R

I am facing issues with conversion of JSON to dataframe. I tried using libraries: jsonlite, RJSONIO,rjson.
I keep getting 'invalid character in the string' or unclosed string.
I am getting this data from a standard API so should be able to parse this json. Also, JSON editors can parse this data just fine.
My question is:
Is there a standard way using which I can make sure that my dataframe gets created and ignore above errors?
My best guess was to convert this data to JSON format using toJSON function from either of the libraries but if I use
newdata <- fromJSON(toJSON(data))
it somehow never gets converted to dataframe. Why is that?
If I instead use
newdata <- fromJSON(data)
I get a valid dataframe but sometimes because of above errors, it doesn't work which is what I am trying to know. How do you deal with this?
I have tried using this too freshDeskTicketsToDF <-
jsonlite::fromJSON(paste(readLines(textConnection(freshDeskTickets)), collapse=""))
It seemed to solve the problem but somewhere I got unclosed string with this method which I otherwise did not.
Are there better ways to deal with this in R?
Also, why is it that using toJSON on data passed to fromJSON never gets converted to a dataframe?
If I decide to take off html tags from the values assisgned to keys in JSON data. How does that work? Can I do that?
Edit: It looks like I get this error when I have <html tags> in my "string data" but I have them all across my JSON data and I don't get it every time.
How to deal with problems like this?
Note: this issue not specific to the data that I have. What I am looking for is ways to deal with problems like these and not one specific solution to a single problem.
I just realized thattoJSON converts R objects to JSON and not JSON to valid JSON. Is there a way to do it instead?
Sample data:
[
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022205968,
"responder_id": 1018353725,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Order number-100403891",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174093,
"type": "Order Status query",
"due_by": "2016-09-02T08:57:30Z",
"fr_due_by": "2016-09-02T02:57:30Z",
"is_escalated": true,
"description": "<div dir=\"ltr\">Hi Team,<div><br></div>\n<div>I have ordered an item from your website, order number-100403891. I had called on August 30 2016 to postpone the delivery date. The guy i spoke from your end had confirmed that he will hold and push the delivery date to September 5 or 6 or 7 2016. And he confirmed the same.</div>\n<div>However, the guy I spoke to<b> did not do it</b>. </div>\n<div>I got to know it from ABHINAV from your customer care team who I spoke to on August 1st at 13:10. Hence I have put a request again and he said he will talk to some guys and give me the desired dates for delivery which is 5,6,7 of September 2016. </div>\n<div>Please let me know the concern on this and hope for a quick turn around.</div>\n<div><br></div>\n<div>Thank you,</div>\n<div>Hari,</div>\n<div>+91-9538199699.</div>\n</div>\n",
"description_text": "Hi Team,\r\n\r\nI have ordered an item from your website, order number-100403891. I had\r\ncalled on August 30 2016 to postpone the delivery date. The guy i spoke\r\nfrom your end had confirmed that he will hold and push the delivery date to\r\nSeptember 5 or 6 or 7 2016. And he confirmed the same.\r\nHowever, the guy I spoke to* did not do it*.\r\nI got to know it from ABHINAV from your customer care team who I spoke to\r\non August 1st at 13:10. Hence I have put a request again and he said he\r\nwill talk to some guys and give me the desired dates for delivery which is\r\n5,6,7 of September 2016.\r\nPlease let me know the concern on this and hope for a quick turn around.\r\n\r\nThank you,\r\nHari,\r\n+91-9538199699.\n",
"custom_fields": {
},
"created_at": "2016-09-01T07:51:18Z",
"updated_at": "2016-09-11T11:00:33Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022148025,
"responder_id": 1021145209,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Defect in d piece",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174092,
"type": "Return",
"due_by": "2016-09-01T15:51:00Z",
"fr_due_by": "2016-09-01T09:51:00Z",
"is_escalated": false,
"description": "<div><br></div>\n<div><br></div>\n<div><br></div>\n<div><div style=\"font-size:75%;color:#575757\">Sent from Samsung Mobile</div></div>",
"description_text": "\n\n\nSent from Samsung Mobile",
"custom_fields": {
},
"created_at": "2016-09-01T07:51:00Z",
"updated_at": "2016-09-06T09:00:14Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022205895,
"responder_id": 1018353725,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Re: StalkBuyLove Return Request for order: 100404435",
"to_emails": [
"StalkBuyLove <contact#stalkbuylove.com>"
],
"product_id": null,
"id": 174088,
"type": "Refund query",
"due_by": "2016-09-01T15:43:56Z",
"fr_due_by": "2016-09-01T09:43:56Z",
"is_escalated": true,
"description": "<div>Hi. Can u deposit the amount if i giv u my account number. Right away i cant choose any other product frim ur site. <br><br>Sent from my iPhone</div>\n<div>\n<br>On Sep 1, 2016, at 12:38 PM, StalkBuyLove <contact#stalkbuylove.com> wrote:<br><br>\n</div>\n<blockquote><div>\n<div><img title=\"StalkBuyLove\" alt=\"Stalkbuylove\" src=\"http://www.stalkbuylove.com/launcher_icons/Newlogo_Stalkbuylove_240x50.png\"></div>\n<div>Hello <b>Anamica Aggarwal</b>,</div>\n<div>We have initiated a return request for order: <b>100404435</b> with the following products:</div>\n<table style=\"width:80%\">\r\n <tbody>\n<tr style=\"background-color:#B0C4DE\">\r\n <th>Item Name</th>\r\n <th>Sku</th>\r\n </tr>\n<tr>\r\n <td style=\"text-align:center\">Articuno Top</td>\r\n <td style=\"text-align:center\">IN1627MTOTOPPCH-198-18</td>\r\n </tr>\n</tbody>\n</table>\n<div>Lots of love,</div>\n<div>Team SBL</div>\n<img src=\"http://mandrillapp.com/track/open.php?u=30069003&id=bff0a5daee4a47fe9c6b04d2680c3c39\" height=\"1\" width=\"1\">\r\n</div></blockquote>",
"description_text": "Hi. Can u deposit the amount if i giv u my account number. Right away i cant choose any other product frim ur site. \n\nSent from my iPhone\n\n> On Sep 1, 2016, at 12:38 PM, StalkBuyLove <contact#stalkbuylove.com> wrote:\n> \n> \n> Hello Anamica Aggarwal,\n> \n> We have initiated a return request for order: 100404435 with the following products:\n> \n> Item Name\tSku\n> Articuno Top\tIN1627MTOTOPPCH-198-18\n> Lots of love,\n> \n> Team SBL\n> \n",
"custom_fields": {
},
"created_at": "2016-09-01T07:43:56Z",
"updated_at": "2016-09-11T11:00:32Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022205881,
"responder_id": 1021145209,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Details for order",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174086,
"type": "Order Status query",
"due_by": "2016-09-01T15:42:50Z",
"fr_due_by": "2016-09-01T09:42:50Z",
"is_escalated": false,
"description": "<div><span></span></div>\n<div>\n<span>Hey can i get details of my order </span><br><span>How much more time will it take to get delivered? </span><br><span>Order no-</span><h2 style=\"font-weight: normal; margin: 0px;\"><font><span style=\"background-color: rgba(255, 255, 255, 0);\">100403837</span></font></h2>\n<span></span><br><span>Sent from my iPhone</span><br>\n</div>",
"description_text": "Hey can i get details of my order \r\nHow much more time will it take to get delivered? \r\nOrder no-\r\n100403837\r\n\r\nSent from my iPhone\n",
"custom_fields": {
},
"created_at": "2016-09-01T07:42:50Z",
"updated_at": "2016-09-06T09:00:13Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": true,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022204690,
"responder_id": 1021145209,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Refund",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174080,
"type": "Refund query",
"due_by": "2016-09-01T15:36:26Z",
"fr_due_by": "2016-09-01T09:36:26Z",
"is_escalated": true,
"description": "<div>\r<br>Bank statement as asked for refund! Please intiate the proccedings asap!<br>\n</div>",
"description_text": "\r\nBank statement as asked for refund! Please intiate the proccedings asap!\n",
"custom_fields": {
},
"created_at": "2016-09-01T07:36:26Z",
"updated_at": "2016-09-07T08:00:19Z"
}
]
library(jsonlite)
df <- stream_in(file("~/data/sample.json"))
This stream_in function directly convert into datafram

Get a list of live streaming users in Twitch

I want to retrieve a list of users who are live streaming code.
This endpoint will give a list of live streams:
curl -H 'Accept: application/vnd.twitchtv.v3+json' -X GET https://api.twitch.tv/kraken/streams?stream_type=live
A stream (one of the responses) has a channel which has a name:
{
"_id": 21413418816,
"_links": {
"self": "https://api.twitch.tv/kraken/streams/nervarien"
},
"average_fps": 60.0490196078,
"channel": {
"_id": 25452510,
"_links": {
"chat": "http://api.twitch.tv/kraken/chat/nervarien",
"commercial": "http://api.twitch.tv/kraken/channels/nervarien/commercial",
"editors": "http://api.twitch.tv/kraken/channels/nervarien/editors",
"features": "http://api.twitch.tv/kraken/channels/nervarien/features",
"follows": "http://api.twitch.tv/kraken/channels/nervarien/follows",
"self": "http://api.twitch.tv/kraken/channels/nervarien",
"stream_key": "http://api.twitch.tv/kraken/channels/nervarien/stream_key",
"subscriptions": "http://api.twitch.tv/kraken/channels/nervarien/subscriptions",
"teams": "http://api.twitch.tv/kraken/channels/nervarien/teams",
"videos": "http://api.twitch.tv/kraken/channels/nervarien/videos"
},
"background": null,
"banner": null,
"broadcaster_language": "pl",
"created_at": "2011-10-14T16:36:04Z",
"delay": null,
"display_name": "Nervarien",
"followers": 205381,
"game": "League of Legends",
"language": "pl",
"logo": "https://static-cdn.jtvnw.net/jtv_user_pictures/nervarien-profile_image-8a488c78bf3d3082-300x300.png",
"mature": false,
"name": "nervarien",
"partner": true,
"profile_banner": null,
"profile_banner_background_color": null,
"status": "PI\u0104TEK 12:00!",
"updated_at": "2016-05-20T11:02:47Z",
"url": "https://www.twitch.tv/nervarien",
"video_banner": "https://static-cdn.jtvnw.net/jtv_user_pictures/nervarien-channel_offline_image-8b8038246b36a5d6-1920x1080.jpeg",
"views": 22045728
},
"created_at": "2016-05-20T10:08:00Z",
"delay": 0,
"game": "League of Legends",
"is_playlist": false,
"preview": {
"large": "https://static-cdn.jtvnw.net/previews-ttv/live_user_nervarien-640x360.jpg",
"medium": "https://static-cdn.jtvnw.net/previews-ttv/live_user_nervarien-320x180.jpg",
"small": "https://static-cdn.jtvnw.net/previews-ttv/live_user_nervarien-80x45.jpg",
"template": "https://static-cdn.jtvnw.net/previews-ttv/live_user_nervarien-{width}x{height}.jpg"
},
"video_height": 720,
"viewers": 2776
}
Is that name a user's name?
What is the difference between channel and user?
How can I know whether they are streaming code or not?
Every user on twitch has their own channel, which in general (except in some circumstances) is the one they stream under. In the example given above, the user's display name, eg. the one that shows in the stream title and in the chat is "Nervarien", indexed under json['channel']['display-name'].
Officially, there is no difference between the channel and the user. They should both be identical.
That user is not streaming code, as seen under json['game'], they are streaming League of Legends. On Twitch, there is no official programming/code section, only a hashtag under the "game" Creative which is a catch all directory for pretty much all non-gaming content. The only way to parse if a user is streaming code on twitch is to check if json['game'] is "Creative" and if this is matched, check json['channel']['status'] and search it for the string #programming, as the tags are located under the stream title.
I hope this answers your question.

JSON to CSV: How to add filters (columns) in the final Excel table?

First, I apologize if my description is not accurate enough for you, I am a total newbie and I don't know a thing about programming, so don't hesitate to tell me if you need more detailed info, but I will try to be as precise as possible.
So I have downloaded a bunch of tweets thanks to Twitter's API and the Terminal (through Twurl). All the tweets are in a .json file (that I open with TextWrangler, I'm on a Mac) and the thing is that when I export my .json file to a .csv file in order to process and analyze the data more easily thanks to Excel (or at least the Excel version of LibreOffice), I don't have all the parameters I would require for my study, I lack the "bio" part of each Tweet info present in the .json file. In other words, in my final table I have a column for the tweet ID, one for the tweet author, one for the text of the tweet itself and so on... But I don't have a column for the bio of the tweet author, whereas this information is displayed in the .json file itself. So my question is: is there a code or anything which would enable me to have one more column displaying some more info present in the basic .json file in my final .csv table?
Again, this may not be clear, so don't hesitate to tell me if you need me to highlight a specific point.
Thanks in advance for any insight, I really need help on this one, this is for a research project I need to carry on for my PhD, so any help would be more than welcome!
EDIT: As an example, here is a sample of the data I have for one tweet in my original .json file:
{
"created_at": "Mon Apr 28 09:00:40 +0000 2014",
"id": 460705144846712800,
"id_str": "460705144846712832",
"text": "Work can suck a dick today",
"source": "Twitter for iPhone",
"truncated": false,
"in_reply_to_status_id": null,
"in_reply_to_status_id_str": null,
"in_reply_to_user_id": null,
"in_reply_to_user_id_str": null,
"in_reply_to_screen_name": null,
"user": {
"id": 253350311,
"id_str": "253350311",
"name": "JEEEZUS",
"screen_name": "Maxi_Flex",
"location": "Southchestershire",
"url": "http://www.soundcloud.com/maxi_flex",
"description": "Jazz Personality.G Mentality.",
"protected": false,
"followers_count": 457,
"friends_count": 400,
"listed_count": 1,
"created_at": "Thu Feb 17 02:08:57 +0000 2011",
"favourites_count": 1229,
"utc_offset": null,
"time_zone": null,
"geo_enabled": true,
"verified": false,
"statuses_count": 13661,
"lang": "en",
"contributors_enabled": false,
"is_translator": false,
"is_translation_enabled": false,
"profile_background_color": "08ABFC",
"profile_background_image_url": "http://pbs.twimg.com/profile_background_images/444297891977244672/Z1BkfCFB.jpeg",
"profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/444297891977244672/Z1BkfCFB.jpeg",
"profile_background_tile": true,
"profile_image_url": "http://pbs.twimg.com/profile_images/454073282778902529/gCGicDBH_normal.jpeg",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/454073282778902529/gCGicDBH_normal.jpeg",
"profile_banner_url": "https://pbs.twimg.com/profile_banners/253350311/1392339276",
"profile_link_color": "FA05F2",
"profile_sidebar_border_color": "FFFFFF",
"profile_sidebar_fill_color": "DDEEF6",
"profile_text_color": "333333",
"profile_use_background_image": true,
"default_profile": false,
"default_profile_image": false,
"following": null,
"follow_request_sent": null,
"notifications": null
},
"geo": null,
"coordinates": null,
"place": null,
"contributors": null,
"retweet_count": 0,
"favorite_count": 0,
"entities": {
"hashtags": [],
"symbols": [],
"urls": [],
"user_mentions": []
},
"favorited": false,
"retweeted": false,
"filter_level": "medium",
"lang": "en"
}
So in the final csv file, I have some of the info I mentionned above, but what I would need to add in the csv file is the "description" part (bold) of each string. Any help would be appreciated!
The problem is probably that JSON is hierarchical and CSV is not. I'm guessing that you are only getting the top level JSON elements and not the nested objects. For example if your JSON is:
{
'name': 'test',
'author': {
'id': 123,
'created': ''
}
}
you are only getting 'name' and not 'author.id'? If this is the case, check out other questions on SO related to flattening JSON out for CSV e.g. flattening json to csv format
Any good JSON to CSV converter will work, try this one. If there is somehting funky in the JSON we need an example of the input JSON and what is getting spit out.
If you just need that one field enter the following command on the command line:
cat test.json | sed -n 's/.*description\":\"\([^"]*\)\".*/Description, \1/p' > result.csv
Where test.json is the file with all the JSON entries in it.
Here is the output from an example I ran:
cat test.json | sed -n 's/.*description\":\"\([^"]*\)\".*/\1/p'
Jazz Personality.G Mentality.
Jazz Personality.G Mentality.
Jazz Personality.G Mentality.
Jazz Personality.G Mentality.
If the file is very large you may need to split in to parts:
split -l N test.json part
Where N is the number of lines per part.