I have just started learning ruby and trying to learn extracting web data. I am using samplewebapp code given in a sites API documentation
On executing the test.rb file as given below it generates a token.
The token.json file contains the following
{"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIzY2U1NjFhMC01ZmE5LTQ5NjUtYTQ2Mi01NmI2MzEzNjRlZGMiLCJqdGkiOiJmMjc0ZTgwM2FmZjVlMTFiMTAzNzIwMzAwNzRlNDRhZDE3ODg3ZjAzOGQ2ODk0OWIwNzVkMmQ5N2E2MjAwYzc5ZWRhNzU4MmQ3MTg1MTc1YiIsImlhdCI6MTY1NDU2OTc0MS41MjgxNTUsIm5iZiI6MTY1NDU2OTc0MS41MjgxNTcsImV4cCI6MTY1NDU3MzM0MS41MjgwMzksInN1YiI6IjE5NGFlNTU5LTU5OWMtNDQ2ZC1hOTRhLWM2MTIyYjZkMTA2ZiIsInNjb3BlcyI6W10sImNyZWRpdHNfcmVtYWluaW5nIjo0NzAwLCJyYXRlX2xpbWl0cyI6W3sicmF0ZSI6NSwiaW50ZXJ2YWwiOjYwfV19.XoN5A-H8Z7d8p_0e2rCcyV4yWO9MAF3TZlHk5VP5NjUEqtTXTVYKfKuzidYDs7K8ZzAtWzyt3aR5VSUG0_nw-uPd7Z3_Y75alQMqa2b5rHGn5JFWH7nuAriskr26WSCqAdj4cNjccPORryRr5sYKwpE4Y4kTG0IX_8frvwYxwp-8wFpLLj98K3axwN7CCpWdnPDSzuMqmH6tSpF0XhdYxB5LTVH0AyH7lN0S0_Lftq0b3sOLIvEaTfNkuRGNqwLkBYFkHFuPqwrKd8RJhC2W1QZhrmUw3eYnh-0iQABGk2V0skIqDlb6BbQ5GFX6MXgiXAc-h5Ndda7pZ5N5UCQU3g","expires_at":1654573341}
However it also generates a JSON parser error as under
/Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse': 809: unexpected token at '' (JSON::ParserError)
from /Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse'
from /Users/mm/Desktop/prokerala/client.rb:21:in `parseResponse'
from /Users/mm/Desktop/prokerala/client.rb:99:in `get'
I googled around for a while and some of the responses talk about the issues between double and single quotes.. I tried changing those also but it doesn't work.
I have also tried running it with system ruby - which gives a different error
Would appreciate very much if someone could explain what the error is / logic behind this.. Also I am not clear about the error message itself. The message talks about parse error 809:"unexpected token" on column no 809 of the token there is nothing.. Am i reading it wrong?
code used in test.rb file
client = ApiClient.new('YOUR CLIENT_ID', 'YOUR CLIENT_SECRET');
result = client.get('v2/astrology/thirumana-porutham/advanced', {
:girl_nakshatra => 4,
:girl_nakshatra_pada => 2,
:boy_nakshatra => 26,
:boy_nakshatra_pada => 3
})
puts JSON.pretty_generate(result)
code used in client.rb file
require 'net/http'
require 'json'
class ApiError < StandardError
end
class ApiClient
BASE_URL = "https://api.prokerala.com/"
# Make sure that the following file path is set to a location that is not publicly accessible
TOKEN_FILE = "./token.json"
def initialize(clientId, clientSecret)
# Instance variables
#clientId = clientId
#clientSecret = clientSecret
end
def parseResponse(response)
content = response.body
res = JSON.parse(content)
if res.key?('access_token')
return res
end
if res['status'] == "error"
raise ApiError, res['errors'].map {|e| e['detail']}.join("\n")
end
if res['status'] != "ok"
raise "HTTP request failed"
end
return res
end
def saveToken(token)
# Cache the token until it expires.
File.open(ApiClient::TOKEN_FILE,"w") do |f|
token = {
:access_token => token['access_token'],
:expires_at => Time.now.to_i + token['expires_in']
}
f.write(token.to_json)
end
end
def getTokenFromCache
if not File.file?(ApiClient::TOKEN_FILE)
return nil
end
begin
# Fetch the cached token, and return if not expired
text = File.read(ApiClient::TOKEN_FILE)
token = JSON.parse(text)
if token['expires_at'] < Time.now.to_i
return nil
end
return token['access_token']
rescue JSON::ParserError
return nil
end
end
def fetchNewToken
params = {
:grant_type => 'client_credentials',
:client_id => #clientId,
:client_secret => #clientSecret
}
res = Net::HTTP.post_form(URI(ApiClient::BASE_URL + 'token'), params)
token = parseResponse(res)
saveToken(token)
return token['access_token']
end
def get(endpoint, params)
# Try to fetch the access token from cache
token = getTokenFromCache
# If failed, request new token
token ||= fetchNewToken
uri = URI(ApiClient::BASE_URL + endpoint)
uri.query = URI.encode_www_form(params)
req = Net::HTTP::Get.new(uri.to_s, {'Authorization' => 'Bearer ' + token})
res = Net::HTTP.start(uri.hostname) do |http|
http.request(req)
end
return parseResponse(res)
end
end
running the test file using ruby 3.0.4 gives the a JSON parse error. Full error reproduced as below
/Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse': 809: unexpected token at '' (JSON::ParserError)
from /Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse'
from /Users/mm/Desktop/prokerala/client.rb:21:in `parseResponse'
from /Users/mm/Desktop/prokerala/client.rb:99:in `get'
from test.rb:11:in `<main>'
I executed the above code and from this, we are getting
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true> as response
and response body as empty string '', which results in error in parsing the body.
Since, API endpoint is secure with https, we need set use_ssl: true before starting the session. Try below code and it will fix the issue.
res = Net::HTTP.start(uri.hostname, use_ssl: true) do |http|
http.request(req)
end
I'm writing a script that will check the CVS COVID vaccine availability for cities in my state of VA. I have been successful getting the data I'm looking for, but my code is hard coded in some areas. I'm specifically asking for help improving my code in the areas number 1 & 2 below:
The JSON file can be found here:
https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo
I'm trying to access the data in the responsePayloadData key. The only way I could figure out how to do this is to make it the only key. For that reason, I deleted the other key responseMetaData:
#remove the key that we don't need
del obj['responseMetaData']
I'm also not sure how to dynamically loop through the VA items without hard coding the number of cities I know are there in the data:
for x, y in obj.items():
for a in range(34):
Here's the full code:
import requests
import json
import time
from datetime import datetime
import urllib2
try:
import indigo
except:
pass
strAvail = "False"
strAvailCity = "None"
try:
# download raw json object from CVS Virginia Website
url = "https://www.cvs.com//immunizations/covid-19-vaccine.vaccine-status.VA.json?vaccineinfo"
data = urllib2.urlopen(url).read().decode()
except urllib2.HTTPError, err:
return {"error": err.reason, "error_code": err.code}
# parse json object
obj = json.loads(data)
# remove the key that we don't need
del obj['responseMetaData']
# loop through the JSON dictionary and check availability
# status options: {"Fully Booked", "Available"}
for x, y in obj.items():
for a in range(34):
# print('City: ' + y['data']['VA'][a]['city'])
# print('Total Available: ' + y['data']['VA'][a]['totalAvailable'])
# print('Percent Available: ' + y['data']['VA'][a]['pctAvailable'])
# print('Status: ' + y['data']['VA'][a]['status'])
# print("------------------------------")
# If there is availability anywhere in the state, take some action.
if y['data']['VA'][a]['status'] == "Available":
strAvail = True
strAvailCity = y['data']['VA'][a]['city']
# Log timestamp for this check to the JSON
now = datetime.now()
strDateTime = now.strftime("%m/%d/%Y %I:%M %p")
EDIT: Since the JSON is not available outside the US. I've pasted it below:
{"responsePayloadData":{"currentTime":"2021-02-11T14:55:00.470","data":{"VA":[{"totalAvailable":"1","city":"ABINGDON","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"0","city":"ALEXANDRIA","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ARLINGTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BEDFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"BLACKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHARLOTTESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHATHAM","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"CHESAPEAKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"1","city":"DANVILLE","state":"VA","pctAvailable":"0.19%","status":"Fully Booked"},{"totalAvailable":"2","city":"DUBLIN","state":"VA","pctAvailable":"0.39%","status":"Fully Booked"},{"totalAvailable":"0","city":"FAIRFAX","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"FREDERICKSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"GAINESVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HAMPTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"HARRISONBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LEESBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"LYNCHBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MARTINSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MECHANICSVILLE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"MIDLOTHIAN","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"NEWPORT NEWS","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"NORFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PETERSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"PORTSMOUTH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"RICHMOND","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"ROANOKE","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"ROCKY MOUNT","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"STAFFORD","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"SUFFOLK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},
{"totalAvailable":"0","city":"VIRGINIA BEACH","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WARRENTON","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WILLIAMSBURG","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WINCHESTER","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"},{"totalAvailable":"0","city":"WOODSTOCK","state":"VA","pctAvailable":"0.00%","status":"Fully Booked"}]}},"responseMetaData":{"statusDesc":"Success","conversationId":"Id-beb5f68730b34e6aa3bbc1fd927ea12b","refId":"Id-b4a7256078789eb59b8912b4","operation":"getInventorybyCity","statusCode":"0000"}}
Regarding problem 1, you can just access the data by key. You don't need to delete the other key:
payload = obj['responsePayloadData']
For the second problem, you can just iterate over the items in the list associated with obj['data']['VA']:
for city in payload['data']['VA']:
print(city)
{'city': 'ABINGDON',
'pctAvailable': '0.19%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '1'}
{'city': 'ALEXANDRIA',
'pctAvailable': '0.00%',
'state': 'VA',
'status': 'Fully Booked',
'totalAvailable': '0'}
...
I am using boto3 with Python 3.6 to start a Step Function execution. The Step Function is designed to share my base AMI across all my accounts. I have 4 variables I need to pass to the input parameter to kick off the execution. These are the AMI ID, the account list of accounts I own, the source account, and the KMS key. The AMI ID and the account list are constructed in my code and are the variables that need to get passed dynamically. According to the documentation, input is a string that contains the JSON input data for the execution and gives the following example: "input": "{\"ami_id\" : \"ami_id\"}". My question is how do I pass the variables in question to this parameter as the value? My code is below with the traceback:
CODE:
import boto3
import json
# Get an STS token to assume roles into AWS accounts
def get_sts_token(**kwargs):
role_arn = kwargs['RoleArn']
region_name = kwargs['RegionName']
sts = boto3.client(
'sts',
region_name=region_name,
)
token = sts.assume_role(
RoleArn=role_arn,
RoleSessionName='GetInstances',
DurationSeconds=900,
)
return token["Credentials"]
def get_accounts():
role_arn = "arn:aws:iam::xxxxxxxxxx:role/list-accounts-role"
region_name = "us-east-1"
token = get_sts_token(RoleArn=role_arn, RegionName=region_name)
access_key = token['AccessKeyId']
secret_key = token['SecretAccessKey']
session_token = token['SessionToken']
client = boto3.client('organizations',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
aws_session_token=session_token)
moreAccounts=True
nexttoken=''
global accountList
accountList =[]
while moreAccounts:
if (len(nexttoken)>0):
accounts=client.list_accounts(NextToken=nexttoken)
else:
accounts=client.list_accounts()
if 'NextToken' in accounts:
nexttoken=accounts['NextToken']
else:
moreAccounts=False
for account in accounts['Accounts']:
if account['Status'] != 'SUSPENDED' and account['Status'] != 'CLOSED' :
account_id = account['Id']
accountList.append(account_id)
print(accountList)
def trigger_sfn():
ssm = boto3.client('ssm')
role_arn = "arn:aws:iam::xxxxxxxx:role/execute-sfn"
region_name = "us-east-1"
ami_id = ssm.get_parameter(Name='/BaseAMI/newest')['Parameter']['Value']
print(ami_id)
token = get_sts_token(RoleArn=role_arn, RegionName=region_name)
print(token)
access_key = token['AccessKeyId']
secret_key = token['SecretAccessKey']
session_token = token['SessionToken']
sfn = boto3.client('stepfunctions',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
aws_session_token=session_token)
response = sfn.start_execution(
stateMachineArn='arn:aws:states:us-east-1:xxxxxxxx:stateMachine:ami-share',
input="{\"ami_id\": ami_id, \"source_account_id\": \"112233445566\", \"accountList\": accountList, \"kms_key_arn\": \"alias/aws/ebs\"}"
)
print(response)
TRACE:
An error occurred (InvalidExecutionInput) when calling the
StartExecution operation: Invalid State Machine Execution Input:
'com.fasterxml.jackson.core.JsonParseException: Unrecognized token
'ami_id': was expecting ('true', 'false' or 'null')
at [Source: (String)"{"ami_id": ami_id, "source_account_id":
"112233445566", "accountList": accountList, "kms_key_arn":
"alias/aws/ebs"}"; line: 1, column: 18]': InvalidExecutionInput
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 6, in lambda_handler
trigger_sfn()
File "/var/task/lambda_function.py", line 96, in trigger_sfn
input="{\"ami_id\": ami_id, \"source_account_id\": \"112233445566\",
\"accountList\": accountList, \"kms_key_arn\": \"alias/aws/ebs\"}"
File "/var/runtime/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidExecutionInput: An error occurred
(InvalidExecutionInput) when calling the StartExecution operation:
Invalid State Machine Execution Input:
'com.fasterxml.jackson.core.JsonParseException: Unrecognized token
'ami_id': was expecting ('true', 'false' or 'null')
at [Source: (String)"{"ami_id": ami_id, "source_account_id":
"112233445566", "accountList": accountList, "kms_key_arn":
"alias/aws/ebs"}"; line: 1, column: 18]'
You can add the value of the variable to the string like this:
input="{\"ami_id\": \"" + ami_id + "\", \"source_account_id\": \"112233445566\", \"accountList\": accountList, \"kms_key_arn\": \"alias/aws/ebs\"}"
Your code currently sends the literal string "ami_id", instead of the variable value of ami_id
I am working on Instagram API in Django(python)
I am getting code from
'https://api.instagram.com/oauth/authorize/?client_id=%s&response_type=code&redirect_uri=%s' % (INSTAGRAM_APP_CONSUMER_KEY, redirect_uri)
but when i am exchanging code for access token code is failing
# All arguments are valid
def execute(self, code, redirect_uri, app_id, app_secret):
exchange_url = 'https://api.instagram.com/oauth/access_token'
try:
#Update : get request to post
r = requests.post(exchange_url, params={
'client_id': app_id,
'redirect_uri': redirect_uri,
'client_secret': app_secret,
'code': code,
'grant_type': 'authorization_code'
})
#print(r)
#print(json.loads(r))
print(r.json())
return r.json()
except Exception as e:
print(e)
r.json() gives simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1
Update 1 : r.json() works after request changed from get to post but error
message comming 'You must provide a client_id'
Please let me know what i am doing wrong
I think it requires data in post data, not in query params.
Try this :
your_post_data = {'client_id': '', ... }
r = requests.post('your_url', data=your_post_data)
Ref: Python Requests Docs