Writing Encoded JSON data to a csv using tweepy - json

Question

I've been stuck on this one for a while. Right now this function writes date,latitude,longitude,userid,text of a live tweet to a csv file.
The problem is that text of the tweet often contains letters from other alphabets e.g. arabic. These letters show up in this form (\u0641\u064a).
Is it possible to encode the text to a utf-8 string and append it to the rest of the data, so that the csv file would correclty display all characters?
def on_data(self, data):
try:
tweets = json.loads(data)
with open('Data.csv','a',encoding = 'utf-8') as f:
if(tweets['coordinates'] is not None):
coordinates_string = json.dumps(tweets["coordinates"]["coordinates"])
val_lg = coordinates_string.split(',')[0].strip("[")
val_lt = coordinates_string.split(',')[1].strip("]")
else:
val_lg = "None"
val_lt = "None"
text = json.dumps(tweets["text"])
user_id = json.dumps(tweets["user"]["id_str"])
time = json.dumps(tweets["created_at"])
data_string = time + "," + val_lt + "," + val_lg + "," + user_id + "," + text + "\n"
print(data_string)
f.write(data_string)
except:
pass

Answer 1

You've got some overuse of json. Once you loads the tweet, group the data fields in a list and use the csv module to write it nicely.
import json
import csv
# A guess on the data format of the tweet that was parsable by the OP's original code.
D = {'coordinates' : {'coordinates' : [45.6,122.3]}, 'text' : u'some text\u0641\u064a',
'user' : {'id_str' : 'some id'}, 'created_at': 'some date'}
data = json.dumps(D)
tweets = json.loads(data)
# 'utf-8-sig' makes sure the output csv will open in Excel if that is a goal.
# newline='' is a requirement for csv.writer in Python 3.
with open('Data.csv','a',encoding = 'utf-8-sig', newline='') as f:
# This forces quoting of strings like the OP got from json.dumps
w = csv.writer(f,quoting=csv.QUOTE_NONNUMERIC)
if tweets['coordinates'] is not None:
val_lg = tweets['coordinates']['coordinates'][1]
val_lt = tweets['coordinates']['coordinates'][0]
else:
val_lg = "None"
val_lt = "None"
text = tweets["text"]
user_id = tweets["user"]["id_str"]
time = tweets["created_at"]
# group the fields in a list for writerow
data = [time,val_lt,val_lg,user_id,text]
print(data)
w.writerow(data)
Output (UTF-8 terminal):
['some date', 45.6, 122.3, 'some id', 'some textفي']
Output (Data.csv):
"some date",45.6,122.3,"some id","some textفي"