AIOHTTP RetryClient on JSON Decode Errors - json

Question

I am trying to use the aiohttp_retry's RetryClient to address random JSON decoding errors, but it seems retries are not working. Can JSON decode errors be retried?
ERROR - message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8'
When I try to use debugging (latest community Pycharm), my app seems to get confused and errors out, but running straight through works, albeit with the decode error/exceptions still. Error rate is ~20 out of 3950 URI's in 20 minutes, but I want to alleviate manually fixing them afterwards.
aiohttp 3.8.3
aiohttp_retry 2.8.3
Pythton 3.10
from aiohttp import TCPConnector
from aiohttp_retry import RetryClient, ExponentialRetry
async def get_parcel_details(client, sem, url):
async with sem, client.get(url) as resp:
if resp.status == 200:
try:
parcel_details = await resp.json(encoding='UTF-8', content_type='application/json')
return parcel_details
except Exception as e:
logger.error(str(e))
await asyncio.sleep(2)
logger.warning(f"sleeping on {url} for 2 seconds, retrying?")
parcel_details = {'Owner': 'ERROR', 'Rental': False}
return parcel_details
else:
logger.error(resp.status)
async def async_main(APNs: list):
connector = TCPConnector(ssl=False, limit=15, limit_per_host=10, enable_cleanup_closed=True)
async with RetryClient(headers=API_HEADER, connector=connector, raise_for_status=True,
retry_options=ExponentialRetry(attempts=3)) as retry_client:
sem = asyncio.Semaphore(20)
tasks = []
for apn in APNs:
parcel_url = f'https://api_endpoint/parcel/{apn}'
tasks.append(asyncio.create_task(get_parcel_details(retry_client, sem, parcel_url)))
parcels = await asyncio.gather(*tasks, return_exceptions=True)
return parcels
I tried putting another get in the Exception, but made things worse.

Answer 1

After reading post: How to retry async requests upon ClientOSError: [Errno 104] Connection reset by peer?
it became apparent that I was trying/Excepting the wrong piece of code, I followed his example and no more errors!
async def get_parcel_details(client, sem, url):
""" Takes in an api client, semaphore, and url to get latest parcel data
Returns a dictionary
"""
try:
async with sem, client.get(url) as resp:
parcel_details = await resp.json(encoding='UTF-8', content_type='application/json')
return parcel_details
except (json.JSONDecodeError, aiohttp.client.ClientOSError, aiohttp.client.ContentTypeError,
aiohttp.ClientResponseError, TypeError) as e:
await asyncio.sleep(4)
async with sem, client.get(url) as resp:
parcel_details = await resp.json(encoding='UTF-8', content_type='application/json')
return parcel_details