Loop through list of dictionaries and append to csv - json

Question

I'm currently trying to collect tweets with the Twitter API. I want to merge two list dictionaries to a csv. The ['data'] list consist of ID and tweet, the second list, ['includes']['users'], consist of username and location. I have tried with two for loops in order to merge these elements, one for ['data'] and one for ['includes']['users']. But I end up having the exact same tweet and ID for every user in my csv output.
print(json.dumps(json_response, indent=4, sort_keys=True))
My data looks like this (not real tweets):
{"data": [{"author_id": "1234","id": "9999","text": "This is tweet number 1"},{"author_id": "9876","id": "1111","text": "This is another tweet"},],"includes": {"users": [{"id": "9999","location": "Earth","name": "George Huston","username": "George_Huston"},{"id": "1111","name": "Adam Sandler,"username": "adam_sandler"}]
json_response['includes']['users']
[{'name': 'George Huston','location': 'Earth','id': '9876','username': 'George_Huston'},{'name': 'Adam Sandler', 'id': '9999', 'username': 'adam_sandler}]
Creating a csv:
# Create file
csvFile = open("data.csv", "a", newline="", encoding='utf-8')
csvWriter = csv.writer(csvFile)
#Create headers for the data you want to save, in this example, we only want save these columns in our dataset
csvWriter.writerow(['id', 'username', 'text', 'location'])
csvFile.close()
def append_to_csv(json_response, fileName):
#A counter variable
counter = 0
#Open OR create the target CSV file
csvFile = open(fileName, "a", newline="", encoding='utf-8')
csvWriter = csv.writer(csvFile)
#Loop through each tweet
for tweet in json_response['data']:
tweet_id = tweet['id']
text = tweet['text']
for element in json_response['includes']['users']:
username = element['username']
if ('location' in tweet):
location = element['location']
else:
location = " "
# Assemble all data in a list
res = [tweet_id,username,text,location]
# Append the result to the CSV file
csvWriter.writerow(res)
counter += 1
# When done, close the CSV file
csvFile.close()
# Print the number of tweets for this iteration
print("# of Tweets added from this response: ", counter)
append_to_csv(json_response, "data.csv")
But get this csv output:
id,username,text,location
9999,George_Huston,"This is tweet number 1",
9999,adam_sandler,"This is tweet number 1",
The id, text, location is always the same, while the username is different. How can I solve this problem?

Answer 1

In your for tweet in json_response['data'] loop you overwrite tweet_id and text as the loop goes on. The output you see is whatever they were set to in the last iteration of the loop.
It seems from the Twitter API that you can get usernames from the Tweet Objects as well, without json_response['includes']['users'] that you used.
Does this do what you want?
# Create file
fileName = 'data.csv'
csvFile = open("data.csv", "w", newline="", encoding='utf-8')
csvWriter = csv.writer(csvFile)
#Create headers for the data you want to save, in this example, we only want save these columns in our dataset
csvWriter.writerow(['id', 'username', 'text', 'location'])
csvFile.close()
def append_to_csv(json_response, fileName):
#A counter variable
counter = 0
#Open OR create the target CSV file
csvFile = open(fileName, "a", newline="", encoding='utf-8')
csvWriter = csv.writer(csvFile)
#Loop through each tweet
for tweet in json_response['data']:
tweet_id = tweet['id']
text = tweet['text']
username = tweet['username']
if ('location' in tweet):
location = element['location']
else:
location = " "
# Assemble all data in a list
res = [tweet_id,username,text,location]
# Append the result to the CSV file
csvWriter.writerow(res)
counter += 1
# When done, close the CSV file
csvFile.close()
# Print the number of tweets for this iteration
print("# of Tweets added from this response: ", counter)
append_to_csv(json_response, "data.csv")