I want to do data analytics project on my emails in gmail. How can I get all the mail from gmail into a csv file format. I already have downloaded mailobox.mbox file o my mailbox but couldn't find any valide script to turn it into csv form.
Also can you answer is data scrapping can work on gmail.
I am getting option to use csv conertors tools online but I don't have trust on those websites as some of my mails.I also used this code bu it doesn't return any CSV file.
import mailbox
import csv
mbox_file = 'file_path.mbox'
with open('clean_mail_B.csv', 'w', newline='', encoding='utf-8') as f_output:
# Create a column for the first 30 message payload sections
fieldnames = {f'Part{part:02}' for part in range(1, 31)}
for message in mailbox.mbox(mbox_file):
fieldnames.update(message.keys())
csv_output = csv.DictWriter(f_output, fieldnames=sorted(fieldnames), restval='', extrasaction='ignore')
csv_output.writeheader()
for message in mailbox.mbox(mbox_file):
items = dict(message.items())
for part, payload in enumerate(message.get_payload(), start=1):
items[f'Part{part:02}'] = payload
csv_output.writerow(items)
Related
I'm working with a raspberry pi zero and Python to send and recieve sensor data with Azure IoT. I've already created an endpoint and message routing to the storage container. But when I check the JSON-Files in the container, I've got two problems:
The file include various general data which i don't need
My message body is in Base24-format
My message looks like this:
{"EnqueuedTimeUtc":"2021-06-25T13:03:25.7110000Z","Properties":{},"SystemProperties":{"connectionDeviceId":"RaspberryPi","connectionAuthMethod":"{"scope":"device","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}","connectionDeviceGenerationId":"637555519600003402","enqueuedTime":"2021-06-25T13:03:25.7110000Z"},"Body":"eyJ0ZW1wZXJhdHVyZSI6IDI4Ljk1LCAicHJlc3N1cmUiOiA5ODEuMDg2Njk1NDU5MzMyNiwgImh1bWlkaXR5IjogNDYuMjE0ODE3NjkyOTEyODgsICJ0aW1lIjogIjIwMjEtMDYtMjUgMTQ6MDM6MjUuNjMxNzk1In0="}
The body included my sensor data in Base64-format. I've already read about contentType = application/JSON and contentEncoding = UTF-8 so that Azure can work with correct JSON files. But where do i apply these settings? When I apply it to the routing query, I get the following error:
Routing Query Error (The server didn't understand your query. Check your query syntax and try again)
I just want to get the body-message in correct JSON Format.
Thank you all for any kind of help! Since it's my first experience with this kind of stuff, I'm a little helpless.
Zero clue if this helps, but here is my code for sending data from Raspberry Pi Python to AWS - Parse Server using base64/JSON. The only reason I use base64 is to send pictures. You should only have to use JSON to send your other data.
import requests
import random, time
import math
import json
import Adafruit_DHT
import base64
from Adafruit_CCS811 import Adafruit_CCS811
from picamera import PiCamera
from time import sleep
DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN =4
ccs = Adafruit_CCS811()
camera = PiCamera()
while True:
time.sleep(5)
camera.start_preview()
sleep(5)
camera.capture('/home/pi/Desktop/image.jpg')
camera.stop_preview()
with open('/home/pi/Desktop/image.jpg', 'rb') as binary_file:
binary_file_data = binary_file.read()
base64_encoded_data = base64.b64encode(binary_file_data)
base64_message = base64_encoded_data.decode('utf-8')
humidity, temperature = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN)
ccs.readData()
parseServer = {
"temp": temperature,
"humid": humidity,
"co2": ccs.geteCO2(),
"pic": base64_message
}
resultJSON = json.dumps(parseServer)
headers = {
'X-Parse-Application-Id': 'myappID',
'Content-Type': 'application/json',
}
data = resultJSON
response =
requests.put('http://1.11.111.1111/parse/classes/Gamefuck/TIuRnws3Ag',
headers=headers, data=data)
print(data)
If you're using the Python SDK for Azure IoT, sending the message as UTF-8 encoded JSON is as easy as setting two properties on your message object. There is a good example here
msg.content_encoding = "utf-8"
msg.content_type = "application/json"
Furthermore, you don't need to change anything in IoT Hub for this. This message setting is a prerequisite to be able to do message routing based on the body of the message.
I have a function that generates a CSV report for survey.surveys to my employer's liking and I need a way to download that data. The issue I'm having is to get a button on the page to actually download the data, I've tried adding a controller but when I go to it, it says 404.
I'm using Odoo 13
Here's the controller I tried, but when i go to it, it returns a 404. Yes i checked my __init__.py for both my module and controllers folder
class MyExport(http.Controller):
#http.route(['/my_module/export/certs'], type='http', auth='user', methods=['GET'], website=True)
def csv_download(self, **kw):
csv = http.request.env['survey.survey'].generate_cert_report()
filename = 'Certification Report - {}.csv'.format(
datetime.now().strftime("%m/%d/%Y"))
headers = [
('Content-Type', 'application/octet-stream'),
('Content-Disposition', 'attachment; filename="%s"' % (filename))
]
return request.make_response(csv, headers=headers)
I was successful in publishing (POST) a JSON file in Zapier and creating a Storage for it. However, I´d like to access the JSON in Zapier Storage using a Python code run locally. I am able to access the storage with Python3, see that is something written there, but I cannot access the JSON contents.
import urllib
import json
import codecs
reader = codecs.getreader("utf-8")
access_token = "password"
def GetStorage(page_id, access_token):
url = 'https://hooks.zapier.com/url/'
response = urllib.request.urlopen(url)
data = json.load(reader(response))
return data
a=GetStorage(url, access_token)
print(a)
All I get is:
{'attempt': '5a539a49-65eb-44f8-a30e-e171faf7a680',
'id': '1b38d21a-0150-46df-98c1-490a0d04b565',
'request_id': '5a539a49-65eb-44f8-a30e-e171faf7a680',
'status': 'success'}
When in fact I need:
{'Name':'value',
'Address': 'value'
}
Any ideas ?
David here, from the Zapier Platform team.
You're close! hooks.zapier.com is the url we use for incoming webhooks, so we always reply with a 200 and the response body you're seeing.
Instead, use store.zapier.com. You'll also want to make sure to include your secret. A full request URL will look like:
https://store.zapier.com/api/records?secret=test
which will return arbitrary json data:
{
"name": "david",
"job": "programmer"
}
The full docs are in json here: https://store.zapier.com/
I run an iPython Notebook server, and would like users to be able to download a pandas dataframe as a csv file so that they can use it in their own environment. There's no personal data, so if the solution involves writing the file at the server (which I can do) and then downloading that file, I'd be happy with that.
How about using the FileLinks class from IPython? I use this to provide access to data directly from Jupyter notebooks. Assuming your data is in pandas dataframe p_df:
from IPython.display import FileLink, FileLinks
p_df.to_csv('/path/to/data.csv', index=False)
p_df.to_excel('/path/to/data.xlsx', index=False)
FileLinks('/path/to/')
Run this as a notebook cell and the result will be a list of links to files downloadable directly from the notebook. '/path/to' needs to be accessible for the notebook user of course.
For not too large tables you can use the following code:
import base64
import pandas as pd
from IPython.display import HTML
def create_download_link( df, title = "Download CSV file", filename = "data.csv"):
csv = df.to_csv()
b64 = base64.b64encode(csv.encode())
payload = b64.decode()
html = '<a download="{filename}" href="data:text/csv;base64,{payload}" target="_blank">{title}</a>'
html = html.format(payload=payload,title=title,filename=filename)
return HTML(html)
df = pd.DataFrame(data = [[1,2],[3,4]], columns=['Col 1', 'Col 2'])
create_download_link(df)
If you want to avoid storing CSVs on the server, you can use this Javascript alternative that create the CSV on the client-side:
from IPython.display import Javascript
js_download = """
var csv = '%s';
var filename = 'results.csv';
var blob = new Blob([csv], { type: 'text/csv;charset=utf-8;' });
if (navigator.msSaveBlob) { // IE 10+
navigator.msSaveBlob(blob, filename);
} else {
var link = document.createElement("a");
if (link.download !== undefined) { // feature detection
// Browsers that support HTML5 download attribute
var url = URL.createObjectURL(blob);
link.setAttribute("href", url);
link.setAttribute("download", filename);
link.style.visibility = 'hidden';
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
}
}
""" % data_in_dataframes.to_csv(index=False).replace('\n','\\n').replace("'","\'")
Javascript(js_download)
Basically, it creates a CSV string in python from the pd dataframe and use it in a small js script that creates a CSV file on the client side and open a saving dialog to save it on the user computer. I tested in my iPython env and it works like a charm!
Note that I am escaping the \n. If I don't do so, the js script string will have the CSV variable written on multiple lines.
For example, print "var csv = '%s'" % industries_revenues.to_csv(index=False).replace('\n','\\n') results to this:
var csv = 'Industry,sum_Amount\nBanking,65892584.0\n(...)Finance,20211917.0\n'
Instead of print "var csv = '%s'" % industries_revenues.to_csv(index=False) without the \n escaping that results on a multiple lined and therefore errored javascript:
var csv = 'Industry,sum_Amount
Banking,65892584.0
(...)
Finance,20211917.0
'
I also escape the ' not to break the variable string in javascript.
A function that creates a csv download link, based on Coen Jonker's answer and similar to Yasin Zähringer's answer except that it uses IPython.display.FileLink so that there is no need to create html code.
The function has an optional delete prompt so you can delete the file after download to keep the notebook server clean.
# Import a module to create a data frame
import pandas
# Import a module to display a link to the file
from IPython.display import FileLink
# Import a module to delete the file
import os
# Create a download function
def csv_download_link(df, csv_file_name, delete_prompt=True):
"""Display a download link to load a data frame as csv within a Jupyter notebook
Parameters
----------
df : pandas data frame
csv_file_name : str
delete_prompt : bool
"""
df.to_csv(csv_file_name, index=False)
display(FileLink(csv_file_name))
if delete_prompt:
a = input('Press enter to delete the file after you have downloaded it.')
os.remove(csv_file_name)
# Create an example data frame
df = pandas.DataFrame({'x':[1,2,3],'y':['a','b','c']})
# Use the function to diplay a download link
csv_download_link(df, 'file_name.csv')
This is mostly for people who use jupyter notebooks on their own machine. On a shared machine, the use of os.remove might be problematic depending on how you set up file write permissions.
You can use the fact that the notebook can display html for objects, and data urls, to make the content of a csv downloadable:
import urllib
class CSV(object):
def _repr_html_(self):
html = []
html.append("{},{},{}".format(
"user",
"age",
"city"
)
)
html.append("{},{},{}".format(
"Alice",
"39",
"New York"
)
)
html.append("{},{},{}".format(
"Bob",
"30",
"Denver"
)
)
html.append("{},{},{}".format(
"Carol",
"27",
"Tulsa"
)
)
export = '\n'.join(html)
export = urllib.quote(export.encode("utf-8"))
csvData = 'data:application/csv;charset=utf-8,' + export
return "<a download='export.csv' href='{}' target='_blank'>csv file</a>".format(csvData)
CSV()
The simple method that I found was:
df.to_csv('~/Desktop/file_name.csv')
My simple approach to download all the files from the jupyter notebook would be by simply using this wonderful command
!tar cvfz my_compressed_file_name.tar.gz *
This will download all the files of the server including the notebooks.
In case if your server has multiple folders, you might be willing to use the following command. write ../ before the * for every step up the directory.
tar cvfz zipname.tar.gz ../../*
Hope it helps..
How can I upload a local CSV file to Google Drive using the Drive API v2 so that the uploaded file is in the native Google Spreadsheet format. Preferably in Python, but a raw HTTP request will suffice.
What I tried:
request body content-type: 'application/vnd.google-apps.spreadsheet', media_body content-type: 'text/csv'. --> 401 Bad Request
request body content-type: 'application/vnd.google-apps.spreadsheet', media_body content-type: 'application/vnd.google-apps.spreadsheet'. --> 400 Bad Request
... (a couple of others such as leaving a property out and similar, usually got 400 or Drive didn't recognise it as a native spreadsheet)
Your insert request should specify text/csv as the content-type.
The trick to get the file converted is to add the ?convert=true query parameter to the request url:
https://developers.google.com/drive/v2/reference/files/insert
(Mar 2017) Note, while the question specifically asks about Drive API v2, developers should know that the Google Drive API team released v3 at the end of 2015, and in that release, insert() changed names to create() so as to better reflect the file operation. There's also no more convert flag -- you just specify MIMEtypes... imagine that!
The documentation has also been improved: there's now a special guide devoted to uploads (simple, multipart, and resumable) that comes with sample code in Java, Python, PHP, C#/.NET, Ruby, JavaScript/Node.js, and iOS/Obj-C to upload a file and another that imports a CSV file as a Google Sheet.
Just to show how straightforward it is, below is one alternate Python solution (to the sample in the docs) for short files ("simple upload") where you don't need the apiclient.http.MediaFileUpload class. This snippet assumes your auth code works where your service endpoint is DRIVE with a minimum auth scope of https://www.googleapis.com/auth/drive.file.
# filenames & MIMEtypes
DST_FILENAME = 'inventory'
SRC_FILENAME = DST_FILENAME + '.csv'
SHT_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
CSV_MIMETYPE = 'text/csv'
# Import CSV file to Google Drive as a Google Sheets file
METADATA = {'name': DST_FILENAME, 'mimeType': SHT_MIMETYPE}
rsp = DRIVE.files().create(body=METADATA, media_body=SRC_FILENAME).execute()
if rsp:
print('Imported %r to %r (as %s)' % (SRC_FILENAME, DST_FILENAME, rsp['mimeType']))
Claudio Cherubino's answer is correct -- you have to add the parameter manually. Since you asked in Python though, here's a concrete example:
body = {
'mimeType':'text/csv',
'title': 'title'
}
# service: your authenticated service
# media: your apiclient.http.MediaFileUpload object, with 'text/csv' mimeType
req = service.files().insert(media_body=media, body=body)
# patch the uri to ensure conversion, as the documented kwarg seems to be borked.
# you may need to use '?convert=true' depending on the uri, not taking that into
# account here for sake of simplicity.
req.uri = req.uri + '&convert=true'
# now we can execute the response.
resp = req.execute()
# should be OK
assert resp['mimeType'] == u'application/vnd.google-apps.spreadsheet'
Java :
//Insert a file
File body = new File();
body.setTitle("CSV");
body.setDescription("A test document");
body.setMimeType("text/csv");
java.io.File fileContent = new java.io.File("document.csv");
FileContent mediaContent = new FileContent("text/csv", fileContent);
Insert insert = service.files().insert(body, mediaContent);
insert.setConvert(true);
File file = insert.execute();
System.out.println("File ID: " + file.getId());
The best way to get started is using the web form at
https://developers.google.com/drive/v2/reference/files/insert#try-it