I'm writing a script that will check the CVS COVID vaccine availability for cities in my state of VA. I have been successful getting the data I'm looking for, but my code is hard coded in some areas. I'm specifically asking for help improving my code in the areas number 1 & 2 below:
The JSON file can be found here:
https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo
I'm trying to access the data in the responsePayloadData key. The only way I could figure out how to do this is to make it the only key. For that reason, I deleted the other key responseMetaData:
#remove the key that we don't need
del obj['responseMetaData']
I'm also not sure how to dynamically loop through the VA items without hard coding the number of cities I know are there in the data:
for x, y in obj.items():
for a in range(34):
Here's the full code:
import requests
import json
import time
from datetime import datetime
import urllib2
try:
import indigo
except:
pass
strAvail = "False"
strAvailCity = "None"
try:
# download raw json object from CVS Virginia Website
url = "https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo"
data = urllib2.urlopen(url).read().decode()
except urllib2.HTTPError, err:
return {"error": err.reason, "error_code": err.code}
# parse json object
obj = json.loads(data)
# remove the key that we don't need
del obj['responseMetaData']
# loop through the JSON dictionary and check availability
# status options: {"Fully Booked", "Available"}
for x, y in obj.items():
for a in range(34):
# print('City: ' + y['data']['VA'][a]['city'])
# print('Total Available: ' + y['data']['VA'][a]['totalAvailable'])
# print('Percent Available: ' + y['data']['VA'][a]['pctAvailable'])
# print('Status: ' + y['data']['VA'][a]['status'])
# print("------------------------------")
# If there is availability anywhere in the state, take some action.
if y['data']['VA'][a]['status'] == "Available":
strAvail = True
strAvailCity = y['data']['VA'][a]['city']
# Log timestamp for this check to the JSON
now = datetime.now()
strDateTime = now.strftime("%m/%d/%Y %I:%M %p")
EDIT: Since the JSON is not available outside the US. I've pasted it below:
{"responsePayloadData":{"currentTime":"2021-02-11T14:55:00.470","data":{"VA":[{"totalAvailable":"1","city":"ABINGDON","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"0","city":"ALEXANDRIA","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ARLINGTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BEDFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BLACKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHARLOTTESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHATHAM","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHESAPEAKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"1","city":"DANVILLE","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"2","city":"DUBLIN","state":"VA","pctAvailable":"0.39%","status":"Fully Booked"},{"totalAvailable":"0","city":"FAIRFAX","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"FREDERICKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"GAINESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HAMPTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HARRISONBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LEESBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LYNCHBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MARTINSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MECHANICSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MIDLOTHIAN","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"NEWPORT NEWS","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"NORFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PETERSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PORTSMOUTH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"RICHMOND","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ROANOKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"ROCKY MOUNT","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"STAFFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"SUFFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"VIRGINIA BEACH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WARRENTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WILLIAMSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WINCHESTER","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WOODSTOCK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"}]}},"responseMetaData":{"statusDesc":"Success","conversationId":"Id-beb5f68730b34e6aa3bbc1fd927ea12b","refId":"Id-b4a7256078789eb59b8912b4","operation":"getInventorybyCity","statusCode":"0000"}}
Regarding problem 1, you can just access the data by key. You don't need to delete the other key:
payload = obj['responsePayloadData']
For the second problem, you can just iterate over the items in the list associated with obj['data']['VA']:
for city in payload['data']['VA']:
print(city)
{'city': 'ABINGDON',
'pctAvailable': '0.19%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '1'}
{'city': 'ALEXANDRIA',
'pctAvailable': '0.00%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '0'}
...
I've created a basic client and server that pass a string, which I've changed to JSON instead. But the JSON string is only parsable before it gets sent through TCP. After it's sent, the string version is identical (after a chomp), but on the server side it no longer processes the JSON correctly. Here is some of my code (with other bits trimmed)
Some of the client code
require 'json'
require 'socket'
foo = {'a' => 1, 'b' => 2, 'c' => 3}
puts foo.to_s + "......."
foo.to_json
puts foo['b'] # => outputs the correct '2' answer
client = TCPSocket.open('localhost', 2000)
client.puts json
client.close;
Some of the server code
require 'socket'
require 'json'
server = TCPServer.open(2000)
while true
client = server.accept # Accept client
response = client.gets
print response
response = response.chomp
response.to_json
puts response['b'] # => outputs 'b'
end
The output 'b' should be '2' instead. How do I fix this?
Thanks
In your server you wrote response.to_json. This turns a string to JSON, then throws it away. And I don't like the .chomp, either.
Try
response = client.gets
hash = JSON.parse(response)
Now hash is a Ruby Hash object with your data in it, and hash['b'] should work correctly.
The problem is that .to_json does not parse JSON inside a string and replace itself with the result. It is used to convert the string into a format that is an acceptable JSON value.
require 'json'
string = "abc"
puts string
puts string.to_json
This will output:
abc
"abc"
The method is added to the String class by the JSON generator and it uses it internally to generate the JSON document.
But why does your response['b'] return "b"?
Because Ruby strings have a method called [] that can be used to:
Return a substring: "abc"[0,2] => "ab"
Return a single character from index: "abc"[1] => "b"
Return a substring if the string contains it: "abc"["bc"] => "bc", "abc"["fg"] => nil
Return a regexp match: "abc"[/^a([a-z])c/, 1] => "b"
and possibly some other ways I can't think of right now.
So this happens because your response is a string that has the character "b" in it:
response = "something with a b"
puts response["b"]
# outputs b
puts response["x"]
# outputs a blank line because response does not contain "x".
Instead of .to_json your code has to call JSON.parse or JSON.load:
data = JSON.parse(response)
puts data['b']
I am using http.get() to get a JSON from an API I am using, but it's not getting the data. I have the suspicion that this JSON is too big for the NodeMCU. I only need the information in the subpart "stats:". Is it possible to only http.get() that part of the JSON?
EDIT:
This is my code
function getstats()
http.get("https://api.aeon-pool.com/v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit", nil, function(code, pool)
if (code < 0) then
print("can't get stats")
else
h = cjson.decode(pool)
hashrate = h[1]["hashrate"]
print(hashrate)
dofile('update_display.lua')
end
end)
end
I also have another function getting data from another api above getstats()
function getaeonrate()
http.get("https://api.coinmarketcap.com/v1/ticker/aeon/?convert=EUR", nil, function(code, dataaeon)
if (code < 0) then
print("can't get aeon")
else
-- Decode JSON data
m = cjson.decode(dataaeon)
-- Extract AEON/EUR price from decoded JSON
aeonrate = string.format("%f", m[1]["price_eur"]);
aeonchange = "24h " .. m[1]["percent_change_24h"] .. "% 7d " .. m[1]["percent_change_7d"] .. "%"
dofile('update_display.lua')
end
end)
end
But now the weird thing is, when I want to access 'pool' from getstats() I get the json data from getaeonrate(). So "hashrate" isn't even in the json because I am getting the json from another function.
I tried making a new project only with getstats() and that doesn't work at all I always get errors like this
HTTP client: Disconnected with error: -9
HTTP client: Connection timeout
HTTP client: Connection timeout
Yesterday I thought that the response was too big from api.aeon-pool.com, I if you look at the json in your webbrowser you can see that the top entry is 'stats:' and I only need that, none of the other stuff. So If the request is to big It would be nice to only http.get() that part of the json, hence my original question. At the moment I am not even sure what is not working correctly, I read that the nodemcu firmware generally had problems with http.get() and that it didn't work correctly for a long time, but getting data from api.coinmarketcap.com works fine in the original project.
The problems with the HTTP module are with near certainty related to https://github.com/nodemcu/nodemcu-firmware/issues/1707 (SSL and HTTP are problematic).
Therefore, I tried with the more bare-bone TLS module on the current master branch. This means you need to manually parse the HTTP response including all headers looking for the JSON content. Besides, you seem to be on an older NodeMCU version as you're still using CJSON - I used SJSON below:
Current NodeMCU master branch
function getstats()
buffer = nil
counter = 0
local srv = tls.createConnection()
srv:on("receive", function(sck, payload)
print("[stats] received data, " .. string.len(payload))
if buffer == nil then
buffer = payload
else
buffer = buffer .. payload
end
counter = counter + 1
-- not getting HTTP content-length header back -> poor man's checking for complete response
if counter == 2 then
print("[stats] done, processing payload")
local beginJsonString = buffer:find("{")
local jsonString = buffer:sub(beginJsonString)
local hashrate = sjson.decode(jsonString)["stats"]["hashrate"]
print("[stats] hashrate from aeon-pool.com: " .. hashrate)
end
end)
srv:on("connection", function(sck, c)
sck:send("GET /v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit HTTP/1.1\r\nHost: api.aeon-pool.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
end)
srv:connect(443, "api.aeon-pool.com")
end
Note that the receive event is fired for every network frame: https://nodemcu.readthedocs.io/en/latest/en/modules/net/#netsocketon
NodeMCU fails to establish a connection to api.coinmarketcap.com due to a TLS handshake failure. Not sure why that is. Otherwise your getaeonrate() could be implemented likewise.
Frozen 1.5.4 branch
With the old branch the net module can connect to coinmarketcap.com.
function getaeonrate()
local srv = net.createConnection(net.TCP, 1)
srv:on("receive", function(sck, payload)
print("[aeon rate] received data, " .. string.len(payload))
local beginJsonString = payload:find("%[")
local jsonString = payload:sub(beginJsonString)
local json = cjson.decode(jsonString)
local aeonrate = string.format("%f", json[1]["price_eur"]);
local aeonchange = "24h " .. json[1]["percent_change_24h"] .. "% 7d " .. json[1]["percent_change_7d"] .. "%"
print("[aeon rate] aeonrate from coinmarketcap.com: " .. aeonrate)
print("[aeon rate] aeonchange from coinmarketcap.com: " .. aeonchange)
end)
srv:on("connection", function(sck, c)
sck:send("GET /v1/ticker/aeon/?convert=EUR HTTP/1.1\r\nHost: api.coinmarketcap.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
end)
srv:connect(443, "api.coinmarketcap.com")
end
Conclusion
The HTTP module and TLS seem a no-go for your APIs due to a bug in the firmware (1707).
The net/TLS module of the current master branch manages to connect to api.aeon-pool.com but not to api.coinmarketcap.com.
With the old and frozen 1.5.4 branch it's exactly the other way around.
There may (also) be issues with cipher suits that don't match between the firmware and the API provider(s).
-> :( no fun like that
I need to extract only the value for 'admins' from this Json using Ruby :
JSON -
{
"Roles":[
{
"admins":[
"me"
],
"role":"cleanup"
},
{
"admins":[
"tester"
],
"role":"create a mess"
},
]
}
RUBY -
require 'json'
file = File.read('adminlist_Feb_2017.json')
thismonthlist=JSON.parse(file)
puts thismonthlist['admins']
Output - this gives me a blank output however if i change the last line to :
puts thismonthlist['Roles']
it gives me everything. I just want the list of admins.
Try something like this
thismonthlist[:Roles].flat_map { |role| role[:admins] }
=> ["me", "tester"]
admins = []
File.open('adminlist_Feb_2017.json', 'r') do |file|
json = JSON.parse(file.read)
admins = json["Roles"].flat_map{|role| role["admins"]}.uniq
end
admins
# => ["me", "tester"]
I open the file and process it in a block to ensure it's closed at the end. In the block I read the file content and parse the json string into a hash. Then I go through the "Roles" of the hash, grab the "admins" arrays and return it as one array only with Enumerable#flat_map. After I use Enumerable#uniq to return each admin only once.
ok I am trying to create a definition which will read a list of IDS from an external Json file, Which it is doing. Its even putting the data into the database on load of the program, my issue is this. I cant seem to match the list IDs to a comparison. Here is my current code:
def check(account):
global ID_account
import json, httplib
if not hasattr(BigWorld, 'iddata'):
UID_DB = account['databaseID']
UID = ID_account
try:
conn = httplib.HTTPConnection('URL')
conn.request('GET', '/ids.json')
conn.sock.settimeout(2)
resp = conn.getresponse()
qresp = resp.read()
BigWorld.iddata = json.loads(qresp)
LOG_NOTE('[ABRO] Request of URL data successful.')
conn.close()
except:
LOG_NOTE('[ABRO] Http request to URL problem. Loading local data.')
if UID_DB is not None:
list = BigWorld.iddata["ids"]
#print (len(list) - 1)
for n in range(0, (len(list) - 1)):
#print UID_DB
#print list[n]
if UID_DB == list[n]:
#print '[ABRO] userid located:'
#print UID_DB
UID = UID_DB
else:
LOG_NOTE('[ABRO] userid not set.')
if 'databaseID' in account and account['databaseID'] != UID:
print '[ABRO] Account not active in database, game closing...... '
BigWorld.quit()
now my json file looks like this:
{
"ids":[
"1001583757",
"500687699",
"000000000"
]
}
now when I run this with all the commented out prints it seems to execute perfectly fine up till it tries to do the match inside the for loop. Even when the print shows UID_DB and list[n] being the same values, it does not set my variable, it doesn't post any errors, its just simply acting as if there was no match. am I possibly missing a loop break? here is the python log starting with the print of the length of the table print:
INFO: 2
INFO: 1001583757
INFO: 1001583757
INFO: 1001583757
INFO: 500687699
INFO: [ABRO] Account not active, game closing......
as you can see from the log, its never printing the User located print, so it is not matching them. its just continuing with the loop and using the default ID I defined above the definition. Anyone with an idea would definitely help me out as ive been poking and prodding this thing for 3 days now.
the answer to this was found by #VikasNehaOjha it was missing simply a conversion to match types before the match comparison I did this by adding in
list[n] = int(list[n])
that resolved my issue and it finally matched comparisons.