How to get the href value (which changes based on session) from the "Inspect Element" - html

Question

Requirement:
To get the data from a URL into S3 and build a data pipeline. I am trying to get the dataset using python requests.get().
However, the URL changes every few minutes, and it becomes difficult to fetch the data using requests.get() without manual changes to the python script.
My Approach:
I am able to get the data to S3 and build the data pipeline. However, the URL is dynamic I am kinda stuck there.
Code:
import json
import requests
import boto3
import csv
import re
def lambda_handler(event, context):
response= requests.get("https://calcat.covid19.ca.gov/cacovidmodels/_w_d58f724d/session/d37479fcceb6b02080b6b7c23dd05046/download/dl.custom?w=d58f724d",verify=False)
#print(response.content)
s3_client = boto3.client('s3',region_name='us-west-2')
with open('/tmp/data.csv', 'w') as datacsvfile:
writer = csv.writer(datacsvfile, delimiter=",")
for line in response.iter_lines():
writer.writerows([line.strip().split(',')])
Query:
How can I get the href from the "Inspect element" section , store it in a variable and create a dynamic URL to make a REST call to. Below is the "Inspect element" code I am referring to :
This href only appears in the "inspect element" and not in the "View Source Code".