Best way to deal with "key error" when scraping (yahoo finance)? - json

Question

Hi I am making a scrape for yahoo finance and I am using JSON to get keys and then scraping the keys e.g ...
fwd_div_yield = data['context']['dispatcher']['stores']['QuoteSummaryStore']["summaryDetail"]['dividendYield']['raw']
The error is that if a company doesn't pay a dividend it will produce a key error as there is no key 'raw' instead of using raw = 0 they just don't have raw. But if a company does have a dividend it will return 'raw', 'fmt' etc.
I was wondering what the most efficient way of dealing with this is?
Another Question Is how would you access ...
[{'raw': 1595894400, 'fmt': '2020-07-28'}, {'raw': 1596412800, 'fmt': '2020-08-03'}]
my current soloution is...
earnings_dates = data['context']['dispatcher']['stores']['QuoteSummaryStore']['calendarEvents']['earnings']['earningsDate'][0]['fmt']
earnings_datee = data['context']['dispatcher']['stores']['QuoteSummaryStore']['calendarEvents']['earnings']['earningsDate'][1]['fmt']
earnings_date = earnings_dates+", "+earnings_datee

Answer 1

To extract the dividend yield from the raw key and not get a KeyError when it's not there, do the following:
fwd_div_yield = data['context']['dispatcher']['stores']['QuoteSummaryStore']["summaryDetail"]['dividendYield'].get('raw', 0)
In the event raw is not there, the fwd_div_yield will be 0.
Then to retrieve each date from the list of dictionaries, you can use a list comprehension:
earnings_dates = data['context']['dispatcher']['stores']['QuoteSummaryStore']['calendarEvents']['earnings']['earningsDate']
fmt_dates = [date['fmt'] for date in earnings_dates]
Also, this data is available via url: https://query2.finance.yahoo.com/v10/finance/quoteSummary/aapl?modules=summaryDetail. Just replace aapl with the symbol you're scraping.

Answer 2

I would wrap whatever code is checking if the company pays a dividend in a try/except block.
def paysDivivend(data):
try:
if 'raw' in data:
return True
except KeyError:
return False
Without seeing any example code this is a quick fix solution
For the second question...
IF you are asking to create [{'raw': 1234,'fmt':'2020-07-28'},...]:
Based on the compiled list of companies that pay a dividend.
Create a the list:
def dividendList(data):
dividend_list = []
for company in data:
dividend_list.append({'raw':compay['path']['to']['raw'],'fmt':company['path']['to'][fmt']})
return dividend_list
IF you are trying to access each one after you already created the list:
def accessDividend(dividend_data):
for dividend in dividend_data:
print(f"{dividend['raw']}, {dividend['fmt']}")

Answer 3

I created this method as a workaround.
def yfinanceDataframe(symbol, interval, _range):
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
data = requests.get(f'https://query1.finance.yahoo.com/v8/finance/chart/{symbol}?interval={interval}&range={_range}', headers=headers).json()
timestamp = data['chart']['result'][0]['timestamp']
data = data['chart']['result'][0]['indicators']['quote'][0]
df = pd.DataFrame(data)
df['Datetime'] = timestamp
df['Datetime'] = df['Datetime'].apply(lambda x: dt.fromtimestamp(x).strftime('%m/%d/%Y %H:%M'))
df.dropna(inplace=True)
df.reset_index(inplace=True)
df.rename(columns={'close': 'Close'}, inplace=True)
return df