I'm having some difficulties with extracting a single value from a JSON dump. I'm trying to extract the single value of a stock from JSON output generated by using the GoogleFinance package, but I keep getting this error message:
expected string or buffer.
I've tried to find a solution on the forum but nothing seems to be working.
One thing I've tried is loading the JSON in to a string by using json.loads, but I keep running in to the same wall.
from googlefinance import getQuotes
import json
import sys
Apple = json.dumps(getQuotes('AAP'), indent=2) #gets the quote from Apple stock. I know I should use the json.loads but that doesn't seem to be working for the getQuotes.
#JSON output
[
{
"Index": "NYSE",
"LastTradeWithCurrency": "137.24",
"LastTradeDateTime": "2016-11-07T13:09:43Z",
"LastTradePrice": "137.24",
"LastTradeTime": "1:09PM EST",
"LastTradeDateTimeLong": "Nov 7, 1:09PM EST",
"StockSymbol": "AAP",
"ID": "668575"
}
]
#Trying to solve the issue by loading the json to a string
resp = json.loads(Apple)
#print the resp
print (resp)
#extract an element in the response
print (resp["LastTradeWithCurrency"])
change resp = json.loads(Apple) to resp = json.loads(Apple)[0]
Related
I am making a curl call to rest api visa curl in groovy. Response is coming fine but the response is very large, it is a 17MB of data, following is my script :
def converter = "curl.......'"
def initialSize = 4096
def out = new ByteArrayOutputStream(initialSize)
def err = new ByteArrayOutputStream(initialSize)
def process = [ 'bash', '-c', converter].execute()
process.consumeProcessOutput(out, err)
process.waitFor()
Curl response is coming fine, when I print response on console ,store in variable out, it gives response data where it is not neat json as I see some "/n" characters. When I write this to file then I dont see any new line and neat json, all I see data in one line in key value format.
{"key1":"value1","key2":"value2",} in one huge line only
This is when i view in sublime. Now I want to convert this to pretty json and write neatly into file.I tried following to approaches but both prints empty ({ }) in console and in file.
def json = JsonOutput.toJson(out)
println new JsonBuilder(out).toPrettyString()
What did I miss?
I am trying to use groovy libraries only.
UPDATE:
As i try to debug, i found that it may be because all JSON parsers expect string but my output is ByteArrayOutputStream. But now how can I convert the out to string ? I tried out.toString and out.text, it does not work.
Use StringWriter instead of ByteArrayOutputStream
Then JsonOutput.prettyPrint( stringWriter.toString() )
So I've been beating my head against a wall for days now and have been diving down the google/SO rabbit hole in search of answers. I've been debating on how to phrase this question as the API that I am pulling from, may or may not contain some sensitive information that gets uncomfortably close to HIPPA laws for my liking. For that reason I will not be providing the direct link/auth for the my code. That being said I will be providing a made up JSON script to help with the explaining.
import requests
import json
import urllib3
r = requests.get('https://madeup.url.com/api/vi/information here', auth=('123456789', '1111111111222222222223333333333444444455555555'))
payload = {'query': 'firstName'}
response = requests.get(r, params=payload)
json_response = response.json()
print(json.dumps(json_response))
The JSON file that I'm trying to parse looks in part like this:
"{\"id\": 123456789, \"firstName\": \"NAME\", \"lastName\": \"NAME\", \"phone\": \"NUMBER\", \"email\": \"EMAIL#gmail.com\", \"date\": \"December 16, 2021\", \"time\": \"9:50am\", \"endTime\": \"10:00am\",.....
When I run the code I am getting a "urllib3.exceptions.LocationParseError: Failed to parse: <Response [200]>" traceback and I can not for the life of me figure out what is going on. urllib3 is installed and updated according to the console.
Any help would be much appreciated. TIA
That is not a JSON file. It is a string containing escaped characters. It needs to be unescaped before parsing can work.
youre passing r to requests.get() (line 9) , but r is a response to another requests.get() (line 5)... shouldn't you be passing params=payload in line 5? then getting de response from there, in one single request
import requests
import json
import urllib3
payload = {'query': 'firstName'}
response = requests.get('{YOUR_URL}', auth=('{USER}', '{PASS}'), params=payload)
json_response = response.json()
print(json.dumps(json_response))
That is not a JSON file. It is a string containing escaped characters. It needs to be unescaped before parsing can work.
Well now I'm even more confused. I'm trying to self teach myself python and clearly struggling. To get the "JSON" I posted I used the following code:
r = requests.get('URL', 'auth = ('user', 'pass'))
Data = r.json()
packages_str = json.dumps(Data[0])
with open('Data.json', 'w') as f:
json.dump(packages_str, f)
So basically I'm even more lost now...
Okay, update: Good news! kinda... so my code now reads as follows;
import requests
import json
import urllib3
payload = {
'query1'= 'firstName',
'query2'= 'lastName'
}
response = requests.get("url", auth= ("user","pass"), params=payload)
Data = response.json()
packages_str = json.dumps(Data, ensure_ascii=False, indent=2)
with open('Data.json), 'w') as f:
json.dump(packages_str,f)
f.write(packages_str)
And when I then open the JOSN file, the first line of is the entire API in a string but below that, is a properly formatted JSON file. Unfortunately its the entire API and not a parsed JSON file looking for the the information That I need...
Continuing down the google/youtube/SO rabbit hole and will update at a later date if i find a work around.
I'm trying to convert an http JSON response into a DataFrame, then out to CSV file.
I'm struggling with the JSON into DF.
http line:
http://api.kraken.com/0/public/OHLC?pair=XXBTZEUR&interval=1440
JSON response (part of - 720 records in arrays):
[formatted using a JSON site does not post here apparently]
{
"error": [],
"result": {
"XXBTZEUR": [
[1486252800, "959.7", "959.7", "935.0", "943.6", "945.6", "4423.72544809", 5961],
[1486339200, "943.8", "959.7", "940.0", "952.9", "953.5", "4464.48492401", 7678],
[1486425600, "953.6", "990.0", "952.7", "988.5", "977.3", "8123.94462701", 10964],
[1486512000, "988.4", "1000.1", "963.3", "987.5", "983.7", "10989.31074845", 16741],
[1486598400, "987.4", "1007.4", "847.9", "926.4", "934.5", "22530.11626076", 52668],
[1486684800, "926.4", "949.0", "886.0", "939.7", "916.7", "11173.53504917", 12588],
],
"last": 1548288000
}
}
I get
KeyError: 'XXBTZEUR'
on the json_normalize line. Seems to indicate to me that json_normalize is trying to build the DF from the "XXBTZEUR" level, not lower down at the record level. How do I get json_normalize to read the records instead. ie How do I get it to reference deep enough?
I have read several other posts on this site without understanding what I'm doing wrong.
One post mentions that json.loads() must be used. Is json_string.json() also loading the JSON object or do I need the json.loads() instead?
Also tried variations of json_normalize:
BTCEUR_Daily_Table = json_normalize(json_data[[]])
TypeError: unhashable type: 'list'
Can normalize not load an array into a DF line?
code so far:
BTCEUR_Daily_URL = 'http://api.kraken.com/0/public/OHLC?pair=XXBTZEUR&interval=1440'
json_string = requests.get(BTCEUR_Daily_URL)
json_data = json_string.json()
BTCEUR_Daily_Table = json_normalize(json_data, record_path=["XXBTZEUR"])
What I need in result:
In my DF, I just want the arrayed records shown in the "body" of the JSON structure. None of the header & footer are needed.
The solution I found was:
BTCEUR_Daily_Table = json_normalize(data=json_data, record_path=[['result','XXBTZEUR']])
The 2nd parameter specifies the full "path" to the parent label of the records.
Apparently double brackets are needed to specify a full path, otherwise the 2 labels are taken to mean 2 top level names.
Without another post here, I would never have found the solution.
I am very new in python. I have an API link, username, password to read the data. I have to read the data from the API request and then store it into a JSON file. Later I will read it in pandas or R dataframe to do more analysis. So I did:
First approach:
import requests
import json
import urllib, base64
url = 'https://abcXXXXXXXX.com/'
username = 'kebXXXXXXXXX'
password = 'XXXb8thXXXXpb8thXX'
The following code gives me a byte string which I am not able to export as JSON probably.
print(requests.get(url, auth=(username, password)).content)
so I decode it as:
print(requests.get(url, auth=(username, password)).content.decode("utf-8"))
which give me a list but the child text are in a European language and it does not read the proper format. In bot file the byte and decode file the text comes like that format Dobr\u00fd den, where \u00fd should be a Latin word.
The Second Approach:
I try to use urllib library.
URL = 'https://abcXXXXXXXX.com/'
username = b'kebXXXXXXXXX'
password = b'XXXb8thXXXXpb8thXX'
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
but the base64 string gives me an error: expected bytes-like object, not str. I tried different sources could not find any solution. If someone can tell me What will be the best approach to do that. My final goal is to read the JSON data, store it in a file and read by pandas or data frame and do some text analysis. Python or R solution will be ok.
The data looks like(sample):
[
"messages": [
{
"type": "agent",
"name": "",
"from": null,
"content": "Hezký den, potřebujete poradit s výběrem eliptického trenažeru?",
"sentAt": 1509526170320,
"triggeredBy": "Eliptický trenažér",
"email": null
}
]
]
If you are not aware, requests result objects have a .json() method. This should automatically decode the content for you, assuming the results have returned in a json format. Some APIs require an extra header. The line that you wrote,
print(requests.get(url, auth=(username, password)).content)
would change to something like
print(requests.get(url, auth=(username, password)).json())
if headers are not necessary or
print(requests.get(url, auth=(username, password), headers={'Accept': 'application/json'}).json())
if they are.
I am trying to get some data from a rest api and save a JSON to a txt-file. Here is what I do:
#random rest api
a = 'https://thiswouldbemyurl.com'
#urllib3 + poolmanager for requests
import urllib3
http = urllib3.PoolManager()
import json
r = http.request('GET', a)
json.loads(r.data.decode('utf-8'))
with open('data.txt', 'w') as f:
json.dump(data, f, ensure_ascii=False)
I get an error already with json.load. What am I doing wrong?
EDIT: This is how the JSON looks like
{
"success":true,
"data":[
{
"id":26,
"name":"A",
"comment":"",
"start_time_plan":null,
"start_time_actual":"2016-09-13 00:00:00",
"start_time_delta":null,
"start_time_score":null,
"start_time_score_achievement":null,
"start_time_traffic_light":null,
"end_time_plan":null,
"end_time_actual":"2016-09-13 00:00:00",
"end_time_delta":null,
"end_time_score":null,
"end_time_score_achievement":null,
"end_time_traffic_light":null,
"status":0,
"measure_schedule_revision_id":63,
"responsible_user_id":3,
"created_time":"2016-09-13 11:29:14",
"created_user_id":3,
"modified_time":"2016-09-21 16:33:41",
"modified_user_id":3,
"model":"Activity"
}
It sounds like you're trying to json.load(...) something that is not actually JSON.
Looking at the URL you're using, https://jsonplaceholder.typicode.com/ returns HTML rather than JSON.
If you use something like https://jsonplaceholder.typicode.com/posts which does return JSON, then that particular error should go away.