Parsing Facebook Json results - json

I'm trying to use the following code to parse a response from the Facebook API, and I get a strange error... below is the code and error... Any thoughts are greatly appreciated!
Link to code: How to get most popular Facebook post in R
The error I get is:
Error in myposts[[1]]$paging$"next" : $ operator is invalid for atomic vectors
Here is what the data looks like:
tail(myposts[[1]])
results in:
$paging
previous
"https://graph.facebook.com/74133697733/posts?access_token=###token###&limit=25&since=1361736602&__previous=1"
next
"https://graph.facebook.com/74133697733/posts?access_token=###token###&limit=25&until=1359727199"


It's because the paging element is a named vector, not a list, so you can't use $ to get sub-elements.
The following should work :
myposts[[1]]$paging["next"]

Related

Getting error while loading JSON into Bigquery using UI

I have to load the JSON into Bigquery. As per the Bigquery documentation, I made my JSON in the correct formation i.e. Newline delimited with one JSON object each row.
Now, JSON file has around 10 million rows and while loading I get the below error:
Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 2165; errors: 1. Please look into the error stream for more details.
When I found the line 2165, it looks like as follows:
{"deviceId":"13231fd01222a28e","dow":"Wednesday","downloadFlag":"N","email":"clstone898#gmail.com","emailSha256":"1bdf11821f867799bde022ccb57a2e899f827c988b4275571ffd60279c863272","event":"streamStop","firebaseUID":"UDVC3hyQpBWLCnlhXhjAQBeI95Q2","halfHourFull":"08h1","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":1,"login":"google","minutesSinceMidnight":497,"quarterHourFull":"08q2","stationName":"Fox hit 101.9","streamListenMethod":"BluetoothA2DPOutput","timestampLocal":"2018-02-07T08:017:04.679+11:00","timestampUTC":"2018-02-06T21:17:04.679Z"}
When I load this single line then it gets loaded successfully. Kindly guide/suggest what is incorrect here.
I'm loading this json from Bigquery UI using schema Auto-Detect option.
Sample records are as follows:
{"deviceId":"3c7a345dafcff93f","dow":"Tuesday","downloadFlag":"N","email":"psloper.ps#gmail.com","emailSha256":"1cebae8c35db32edcd35e746863fc65a04ac68f2f5b3350f2df477a86bfaa07d","event":"streamStop","firebaseUID":"AMFYYjsvZjauhCktJ5lUzZj0d3D2","halfHourFull":"21h2","liveFlag":"Y","localDate":"2018-02-06","localHalfHour":2,"login":"google","minutesSinceMidnight":1311,"quarterHourFull":"21q4","stationName":"hit 105","streamListenMethod":"Headphones","timestampLocal":"2018-02-06T21:51:40.216+10:00","timestampUTC":"2018-02-06T11:51:40.216Z"}
{"deviceId":"2f1a8c84c738b752","dow":"Wednesday","downloadFlag":"N","email":"kory.maxwell#icloud.com","emailSha256":"13348786c15bff95e4afb4968a9bdbe883b70206a737c02c89fc8215f2a4e101","event":"streamStop","facebookId":"1784054201892593","firebaseUID":"Tx1bHjP6dhaDB2nl2c7yi2KZHsq2","halfHourFull":"06h1","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":1,"login":"facebook","minutesSinceMidnight":384,"quarterHourFull":"06q2","stationName":"hit 105","streamListenMethod":"BluetoothA2DPOutput","timestampLocal":"2018-02-07T06:24:44.533+10:00","timestampUTC":"2018-02-06T20:24:44.533Z"}
{"deviceId":"AA1D685F-6BF6-B0DC-0000-000000000000","dow":"Wednesday","email":"lozza073#bigpond.com","emailSha256":"525db286e9a35c9f9f55db0ce338762eee02c51955ede6b35afb7e808581664f","event":"streamStart","facebookId":"10215879897177171","firebaseUID":"f2efT61sW5gHTfgEbtNfyaUKWaF3","halfHourFull":"7h2","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":2,"login":"facebook","minutesSinceMidnight":463,"quarterHourFull":"7q3","stationName":"Fox hit 101.9","streamListenMethod":"Speaker","timestampLocal":"2018-02-07T07:43:00.39+11:00","timestampUTC":"2018-02-06T20:43:00.39Z"}
{"deviceId":"AEFD39FC-B116-4063-0000-000000000000","dow":"Wednesday","event":"catchUpPause","facebookId":"379907925802180","firebaseUID":"vQPh9tbO3Yge88fpMyNUFzJO7dl1","halfHourFull":"7h2","liveFlag":"N","localDate":"2018-02-07","localHalfHour":2,"login":"facebook","minutesSinceMidnight":465,"quarterHourFull":"7q4","stationName":"Fox hit 101.9","streamListenMethod":"USBAudio","timestampLocal":"2018-02-07T07:45:08.524+11:00","timestampUTC":"2018-02-06T20:45:08.524Z"}
{"deviceId":"AA1D685F-6BF6-B0DC-0000-000000000000","dow":"Wednesday","email":"lozza073#bigpond.com","emailSha256":"525db286e9a35c9f9f55db0ce338762eee02c51955ede6b35afb7e808581664f","event":"streamStop","facebookId":"10215879897177171","firebaseUID":"f2efT61sW5gHTfgEbtNfyaUKWaF3","halfHourFull":"7h2","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":2,"login":"facebook","minutesSinceMidnight":475,"quarterHourFull":"7q4","stationName":"Fox hit 101.9","streamListenMethod":"Speaker","timestampLocal":"2018-02-07T07:55:35.788+11:00","timestampUTC":"2018-02-06T20:55:35.788Z"}
{"deviceId":"AA1D685F-6BF6-B0DC-0000-000000000000","dow":"Wednesday","email":"lozza073#bigpond.com","emailSha256":"525db286e9a35c9f9f55db0ce338762eee02c51955ede6b35afb7e808581664f","event":"streamStart","facebookId":"10215879897177171","firebaseUID":"f2efT61sW5gHTfgEbtNfyaUKWaF3","halfHourFull":"7h2","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":2,"login":"facebook","minutesSinceMidnight":477,"quarterHourFull":"7q4","stationName":"Fox hit 101.9","streamListenMethod":"Speaker","timestampLocal":"2018-02-07T07:57:42.343+11:00","timestampUTC":"2018-02-06T20:57:42.343Z"}
{"deviceId":"13231fd01222a28e","dow":"Wednesday","downloadFlag":"N","email":"clstone898#gmail.com","emailSha256":"1bdf11821f867799bde022ccb57a2e899f827c988b4275571ffd60279c863272","event":"streamStop","firebaseUID":"UDVC3hyQpBWLCnlhXhjAQBeI95Q2","halfHourFull":"08h1","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":1,"login":"google","minutesSinceMidnight":497,"quarterHourFull":"08q2","stationName":"Fox hit 101.9","streamListenMethod":"BluetoothA2DPOutput","timestampLocal":"2018-02-07T08:017:04.679+11:00","timestampUTC":"2018-02-06T21:17:04.679Z"}
Any help is greatly appreciated.

Well, look to that specific 2165 line:
{"deviceId":"13231fd01222a28e","dow":"Wednesday","downloadFlag":"N","email":"clstone898#gmail.com","emailSha256":"1bdf11821f867799bde022ccb57a2e899f827c988b4275571ffd60279c863272","event":"streamStop","firebaseUID":"UDVC3hyQpBWLCnlhXhjAQBeI95Q2","halfHourFull":"08h1","liveFlag":"Y","localDate":"2018-02-07","localHalfHour":1,"login":"google","minutesSinceMidnight":497,"quarterHourFull":"08q2","stationName":"Fox hit 101.9","streamListenMethod":"BluetoothA2DPOutput","timestampLocal":"2018-02-07T08:017:04.679+11:00","timestampUTC":"2018-02-06T21:17:04.679Z"}
And specifically to:
"timestampLocal":"2018-02-07T08:017:04.679+11:00"
And the error message:
Couldn't convert value to timestamp: Could not parse
'2018-02-07T08:017:04.679+11:00' as a timestamp. Required format is
YYYY-MM-DD HH:MM[:SS[.SSSSSS]]
So, if you change "T08:017:04.679" to "T08:17:04.679" (17 minutes instead of 017) then it works. :)

How to know what is wrong in a JSON object?

I am trying to send the following JSON object and I get the error shown below,I checked #http://jsonviewer.stack.hu ,the JSON format seems to be correct,what am I missing and how to fix this?
{"component":{"name":"Company tech (New Bugs)", "version":"B"},"assignee":1234456,"milestone":"chiHW","priority":2,"state":"Analyze","substate":"Nominate","title":"[please ignore]CS\:4355C1\,4364B2\:WDI\:DHD\:HLK\(16299\)\-\>\"DF\ \-\ Sleep\ Tests\"\-\>Assert\-\>bcmpciedhd63\.sys\(dhd\_os\_ioctl\_resp\_wait\)\-\>dhd\_ndis\.c\#4449"}
Error:-
{"message":"An invalid JSON Object was passed in the request body. Please pass a valid JSON object.","help":"View documentation at http://bugs.company.com/","title":"Invalid Request","status":"400 Bad Request"}

You could have pasted your expression into https://jsonlint.com. It tells you where the problem is, and you can experiment until the JSON is no longer invalid. In your case, the problem is those backslashes in the last dictionary value (the one that starts "please ignore").

You could use jq to get a more specific parse error
watson:~$ cat >j
{"component":{"name":"Company tech (New Bugs)", "version":"B"},"assignee":1234456,"milestone":"chiHW","priority":2,"state":"Analyze","substate":"Nominate","title":"[please ignore]CS\:4355C1\,4364B2\:WDI\:DHD\:HLK\(16299\)\-\>\"DF\ \-\ Sleep\ Tests\"\-\>Assert\-\>bcmpciedhd63\.sys\(dhd\_os\_ioctl\_resp\_wait\)\-\>dhd\_ndis\.c\#4449"}
^D
watson:~$ jq <j
parse error: Invalid escape at line 1, column 333

Error with jsonlite package in R

Has anyone ever received this error when trying to web scrape a site:
Error in feed_push_parser(readBin(con, raw(), n), reset = TRUE) :
lexical error: invalid char in json text.
<!doctype html><html xmlns="htt
(right here) ------^
I do not understand why I am receiving this error when I scraped the first page of the site no problem with the same declaration on that first page. But the second page I get this error. Is there a way around this?
This works fine:
jsonlite::fromJSON("https://www.reddit.com/r/BestOfStreamingVideo/.json", flatten = TRUE)
Get the error here:
jsonlite::fromJSON("https://www.reddit.com/r/BestOfStreamingVideo/?count=25&after=t3_5fvgls/.json", flatten = TRUE)

The latter one does not return a JSON. It returns HTML. Enter both URL's in the browser and you'll see the difference.
I guess the URL you are looking for is:
https://www.reddit.com/r/BestOfStreamingVideo/.json?count=25&after=t3_5fvgls/
You need to put the ./json first and append the URL parameter after.

TypeError: string indices must be integers Search API Twitter

I have been fiddling with Twitter Search API, I have coded a Code like this using requests
import requests
from requests_oauthlib import OAuth1
url = "https://api.twitter.com/1.1/account/verify_credentials.json"
auth = OAuth1('redacted', 'redacted','redacted', 'redacted')
requests.get(url, auth=auth)
r = requests.get("https://api.twitter.com/1.1/search/tweets.json?q=%40twitterapi", auth=auth)
print r.json()
The output of this i Get Some JSON Blobs which contains information i wanted to access text part of the JSON data for that i used the following code
for tweet in r.json():
print tweet['text']
The error message is TypeError: string indices must be integers . I fiddled more and used this code
for tweet in r.json():
print tweet
That gave me output search_metadata & statuses. So how do i print only tweets which currently gives me TypeError. Help would be appreciated :(

I figured it out where i was doing mistake, With some comments out there telling me that Indexes must be in integer but to Query out any specific JSON having text the Code context of mine was appropriate.
The solution for this is
for tweet in r.json()['statuses']:
print tweet['text']
I don't know who downvoted, but seriously if you can't answer, than downvoting makes no sense.. if you know what i mean .Cheers, any other simple solution would be appreciateed

how to prevent \r\n and the likes in JSON string

I am getting text from MySQL via a laravel model and converting the content to JSON. Unfortunately the text contains new lines and carriage returns, is there any way to get the contents properly escaped?
Article::find(1)->toJSON();
Gives me an Ajax/JSON error in my view so I was wondering where the problem is. Am I either storing the content the wrong way or am I retrieving it the wrong way?
Thanks.
This is the JSON string I am getting for a test article:
{"id":22,"short_title":"Another test article","long_title":"Longer title for the test article mention in the short title","description":"This article describes a computer classroom where strange things happen and stuf","content":"This is a test article and I think this will generate the error I am looking for. <\/p>\r\n\r\n Maybe or maybe not. <\/p>\r\n","deleted_at":null,"created_at":"2014-04-25 09:10:45","updated_at":"2014-04-25 09:10:45","category_id":1,"zenra_link":"","published_on":null,"published":1,"source_url":"http:\/\/blog.livedoor.jp\/morisitakurumi\/archives\/51856280.html","source_title":"Some source article","source_date":null,"slug":"another-test-article","view_count":0}
The content portion is generated by a textarea that's running CKEditor and it gets saved to a MySQL MEDIUMTEXT field.
Next thing I do is that I inject this into my view to populate my backbone views like this:
<script>var initialData = JSON.parse('{{ $cJson }}');</script>
And that's where the console tells me: Uncaught SyntaxError: Unexpected Token. As suggested I tested the above string on jsonlint.com and it comes back as valid.

I figured out the problem:
<script>var initialData = JSON.parse('{{ $cJson }}');</script>
This is where the error happens. JSON.parse freaks out on the string I supply. If I remove JSON.parse it works like this:
<script>var initialData = {{ $cJson }};</script>
No more unexpected token errors. I was on the wrong track with the problem because I read on so many places that the carriage returns and new lines produce errors in the JSON format.