I am trying to import JSON data from an URL and extract the value of a specific key using python 2.7. I tried the following:
import urllib
import json
daily_stock = urllib.urlopen('http://www.bloomberg.com/markets/api/bulk-time-series/price/NFLX%3AUS?timeFrame=1_DAY')
stock_json = json.load(daily_stock)
print stock_json
The output is:
[{u'lastPrice': 95.9, u'lastUpdateDate': u'2016-04-22', u'price': [{u'value': 95.45, u'dateTime': u'2016-04-22T13:30:00Z'} ...
u'dateTimeRanges': {u'start': u'2016-04-22T13:30:00Z', u'end': u'2016-04-22T20:30:00Z'}}]
When i try to retrieve the value of 'lastPrice':
print stock_json["lastPrice"]
I get the following error:
TypeError: list indices must be integers, not str
Please help.
stock_json is a list with a single dictionary inside, get the dictionary by index:
print stock_json[0]["lastPrice"]
Related
I am stuck in my code. I am parsing 'Json' using pandas but while transforming there is one column that I am stuck with. This column below is "json" inside a list. I want to transform it to 4 columns and the the corresponding data in the rows. The column name is "TypeValues". If I use, pd.concat to transform, I get the error message: "DataFrame constructor not properly called!". I tried using the "pd.DataFrame(eval(data))" but that gives me " eval() arg 1 must be a string, bytes or code object".
There is another column ID. For each ID, i have this below type of data in the "TypeValues" column. I want to get the transformed data with each ID. Does someone have an idea, how can I achieve this? (moreover, some of the rows of the column "TypeValues" have entries [] & blanks)
The column is in the comments section as it didn't allow me to post directly
Thanks in advance
import numpy as np
import pandas as pd
import json from pathlib
import Path from pandas.io.json
import json_normalize import ast
#Reading the File data = pd.read_csv('Test.csv')
Data1 = (pd.concat({k: pd.DataFrame(v) for k, v in data['TypeValues'].pop('TypeValues').items()})).reset_index(level=1, drop=True)
This returns the error of incorrectly using the Dataframe
I am trying to get as many profile links as I can on khanacademy.org. I am using their api.
I am struggling navigating through the json file to get the desired data.
Here is my code :
from urllib.request import urlopen
import json
with urlopen("https://www.khanacademy.org/api/internal/discussions/video/what-are-algorithms/questions?casing=camel&limit=10&page=0&sort=1&lang=en&_=190422-1711-072ca2269550_1556031278137") as response:
source = response.read()
data= json.loads(source)
for item in data['feedback']:
print(item['authorKaid'])
profile_answers = item['answers']['authorKaid']
print(profile_answers)
My goal is to get as many authorKaid as possible en then store them (to create a database later).
When I run this code I get this error :
TypeError: list indices must be integers or slices, not str
I don't understand why, on this tutorial video : https://www.youtube.com/watch?v=9N6a-VLBa2I at 16:10 it is working.
the issue is item['answers'] are lists and you are trying to access by a string rather than an index value. So when you try to get item['answers']['authorKaid'] there is the error:
What you really want is
print (item['answers'][0]['authorKaid'])
print (item['answers'][1]['authorKaid'])
print (item['answers'][2]['authorKaid'])
etc...
So you're actually wanting to iterate through those lists. Try this:
from urllib.request import urlopen
import json
with urlopen("https://www.khanacademy.org/api/internal/discussions/video/what-are-algorithms/questions?casing=camel&limit=10&page=0&sort=1&lang=en&_=190422-1711-072ca2269550_1556031278137") as response:
source = response.read()
data= json.loads(source)
for item in data['feedback']:
print(item['authorKaid'])
for each in item['answers']:
profile_answers = each['authorKaid']
print(profile_answers)
I have a JSON file with some lines like:
"updatedAt" : ISODate("2018-11-20T09:32:16.732+0000"),
I tried json.loads but it has an error json.decoder.JSONDecodeError: Expecting value: line 2 column 13 (char 15).
I believe that the problem is at ISODate () but how could I handle that with Python?
Many thanks
This is not valid JSON, to begin with. I guess the ISODATE("...") is generated from MongoDB, maybe dumping the ISODate() helper directly instead of its string representation into the JSON?
In any case, you could use a regex on the whole JSON-string to get rid of the ISODate("..."), retrieve the date as a string and then use python-dateutil to parse the value to a datetime.datetime.
Something to the tune of
import json
import dateutil.parse
import re
json_str = ....
clean_json = re.compile('ISODate\(("[^"]+")\)').sub('\\1', json_str)
json_obj = json.loads(clean_json)
# use dateutil.parser.parse(s) to parse each date into a datetime.datetime
I am trying to develop an application to retrieve stock prices (in JSON) and then do some analysis on them. My problem is with getting the JSON response into a pandas DataFrame where I can work. Here is my code:
'''
References
http://stackoverflow.com/questions/6862770/python-3-let-json-object- accept-bytes-or-let-urlopen-output-strings
'''
import json
import pandas as pd
from urllib.request import urlopen
#set API call
url = "https://www.quandl.com/api/v3/datasets/WIKI/AAPL.json?start_date=2017-01-01&end_date=2017-01-31"
#make call and receive response
response = urlopen(url).read().decode('utf8')
dataresponse = json.loads(response)
#check incoming
#print(dataresponse)
df = pd.read_json(dataresponse)
print(df)
The application errors at df = pd.read_json... with error TypeError: Expected String or Unicode.
So I reckon this is the first hurdle.
The second is getting where I need to. The JSON response contains only two arrays I am interested in, column_names and data. How do I extract only these two and put into a pandas DataFrame?
To answer your first question, pd.read_json takes a JSON string directly, so you should be doing this:
pd.read_json(response)
But instead, considering how the data is structured, it's best to first convert the JSON string to a dictionary containing the data:
d = json.loads(response)
Then simply build the dataframe from d['dataset']['data'] and d['dataset']['column_names']:
pd.DataFrame(data=d['dataset']['data'], columns=d['dataset']['column_names'])
I am trying to get Python3 to PUT json info in string format to an API. And I want to do it without
import requests
Thus far I am stuck with this code:
import urllib.request
import urllib.parse
import json
url = "http://www.example.com"
DATA = json.dumps({'grades': {"math": "92", "chem": "39"}})
req = urllib.request.Request(url, data=DATA, method='PUT')
response = urllib.request.urlopen(req)
Naturally this raises the error:
raise TypeError(msg)
TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
To get rid of the error I can do:
DATA= str.encode(DATA)
But this turns my data into bytes format, instead of string that I want to put up. Is there a way to PUT up strings without importing "requests"?(importing anything that comes with python install is OK). Or can I PUT up a *.json file?
Essentially I am trying to do the opposite of this.