I am pulling from a mysql database table using python3.4. I use the csv module to write the rows of data from the database into .CSV format. Now I am trying toros figure out how I can vet the rows of data by keywords that may show up in the fourth column of data (row[3]). I was thinking of using the re moduleas below but I keep getting errors. Is it not possible to search keywords in a field that is string type and to filter those results if they have those keywords? I keep getting an error. Please help
import re
import csv
userdate = input('What date do you want to look at?')
query = ("SELECT *FROM sometable WHERE timestamp LIKE %s", userdate)
keywords = 'apples', 'bananas', 'cocoa'
# Execute sql Query
cursor.execute(query)
result = cursor.fetchall()
#Reads a CSV file and return it as a list of rows
def read_csv_file(filename):
"""Reads a CSV file and return it as a list of rows."""
for row in csv.reader(open(filename)):
data.append(row)
return data
f = open(path_in + data_file)
read_it = read_csv_file(path_in + data_file)
with open('file.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
for row in data:
match = re.search('keywords, read_it)
if match:
spamwriter.writerow(row)
I gave up on the regular expressions and used
for row in data:
found_it = row.find(keywords)
if found_it != -1:
spamwriter.writerow(row)
Related
I have been trying to export a result table from a SQL query, but only the records inside the table is displayed in the Excel table. I am using MySQL and python(3.9.11) to run this code. Here is the code that I have been using to do so.
import pymysql
dbconn = pymysql.connect(<db details>)
cus = dbconn.cursor()
cus.execute('sql query')
res = cus.fetchall()
data = []
for i in res:
data += list(i)
var = open('main.csv','w')
for i in data:
var.write(str(i))
var.close()
Also, the result table's all columns data is written in a single column.
I tried and adding a var.write('/n'), but it will convert the rows into columns.
import pymysql
import csv
dbconn = pymysql.connect(DB Connection details)
cus = dbconn.cursor()
cus.execute("sql query here")
res = cus.fetchall()
column_names = [i[0] for i in cus.description]
fp = open('main.csv', 'w')
myFile = csv.writer(fp, lineterminator = '\n')
myFile.writerow(column_names)
myFile.writerows(res)
First of all, sorry for my lack of knowledge regarding databases, this is my first time working with them.
I am having some issues trying to get the data from an excel file and putting it into a data base.
Using answers from the site, I managed to kind of connect to the database by doing this.
import pandas as pd
import pyodbc
server = 'XXXXX'
db = 'XXXXXdb'
# create Connection and Cursor objects
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + server + ';DATABASE=' + db + ';Trusted_Connection=yes')
cursor = conn.cursor()
# read data from excel
data = pd.read_excel('data.csv')
But I dont really know what to do now.
I have 3 tables, which are connected by a 'productID', my excel file mimics the data base, meaning that all the columns in the excel file have a place to go in the DB.
My plan was to read the excel file and make lists with each column, then insert into the DB each column value but I have no idea how to create a query that can do this.
Once I get the query I think the data insertion can be done like this:
query = "xxxxxxxxxxxxxx"
for row in data:
#The following is not the real code
productID = productID
name = name
url = url
values = (productID, name, url)
cursor.execute(query,values)
conn.commit()
conn.close
Database looks like this.
https://prnt.sc/n2d2fm
http://prntscr.com/n2d3sh
http://prntscr.com/n2d3yj
EDIT:
Tried doing something like this, but i'm getting 'not all arguments converted during string formatting' Type error.
import pymysql
import pandas as pd
connStr = pymysql.connect(host = 'xx.xxx.xx.xx', port = xxxx, user = 'xxxx', password = 'xxxxxxxxxxx')
df = pd.read_csv('GenericProducts.csv')
cursor = connStr.cursor()
query = "INSERT INTO [Productos]([ItemID],[Nombre])) values (?,?)"
for index,row in df.iterrows():
#cursor.execute("INSERT INTO dbo.Productos([ItemID],[Nombre])) values (?,?,?)", row['codigoEspecificoProducto'], row['nombreProducto'])
codigoEspecificoProducto = row['codigoEspecificoProducto']
nombreProducto = row['nombreProducto']
values = (codigoEspecificoProducto,nombreProducto)
cursor.execute(query,values)
connStr.commit()
cursor.close()
connStr.close()
I think my problem is in how I'm defining the query, surely thats not the right way
Try this, you seem to have changed the library from pyodbc to mysql, it seems to expect %s instead of ?
import pymysql
import pandas as pd
connStr = pymysql.connect(host = 'xx.xxx.xx.xx', port = xxxx, user = 'xxxx', password = 'xxxxxxxxxxx')
df = pd.read_csv('GenericProducts.csv')
cursor = connStr.cursor()
query = "INSERT INTO [Productos]([ItemID],[Nombre]) values (%s,%s)"
for index,row in df.iterrows():
#cursor.execute("INSERT INTO dbo.Productos([ItemID],[Nombre]) values (%s,%s)", row['codigoEspecificoProducto'], row['nombreProducto'])
codigoEspecificoProducto = row['codigoEspecificoProducto']
nombreProducto = row['nombreProducto']
values = (codigoEspecificoProducto,nombreProducto)
cursor.execute(query,values)
connStr.commit()
cursor.close()
connStr.close()
I have a MySQL database with a few tables. I wrote a python script to dump some of the tables data into a JSON file. I am a bit confused with dumping the date and time stamp.
Here is the code sample, conversion.py:
import MySQLdb
import json
import collections
from datetime import date, datetime
#connect to database
conn = MySQLdb.connect(host= "localhost", user="root", passwd="root", db="testdb")
#Fetch rows
sql = "SELECT * from offices"
cursor = conn.cursor()
cursor.execute(sql)
rows = cursor.fetchall()
data = []
def json_serial(obj):
"""JSON serializer for objects not serializable by default json code"""
if isinstance(obj, (datetime, date)):
return obj.isoformat()
raise TypeError ("Type %s not serializable" % type(obj))
for row in rows:
d = collections.OrderedDict()
d['officeCode'] = row[0]
d['city'] = row[1]
d['phone'] = row[2]
d['eff_date'] = row[3]
d['lastupdatedby'] = row[4]
d['state'] = row[5]
d['country'] = row[6]
d['postalcode'] = row[7]
d['territory'] = row[8]
data.append(d)
with open('data.json', 'w') as outfile:
json.dump(data, outfile, default=json_serial)
conn.close()
When I execute this code, a JSON file is created which is fine. I have a problem with two fields, eff_date which is a date type in database and lastupdatedby is a timestamp type in the database.
"eff_date": "2015-09-23"
"lastupdatedby": "2016-08019T08:13:53"
So, in my JSON file, eff_time is created fine but lastupdatedby is getting a T in middle of date and time as shown above. But, in my actual database there is no T between the date and time. I would like to get rid of that T because I am planning to dump this file into a different database and I don't think it will accept that format.
Any help will be much appreciated.
The T between the date and time is per the ISO 8601 format.
And that's format returned by the datetime.isoformat function, found in the code here:
return obj.isoformat()
(That happens to be the format that Javascript is expecting.)
If we want to return a string different format, we probably need to use a different function, e.g. strftime function in place of isoformat.
If isoformat is working for the date objects, leave that alone. Just do the strftime for a datetime object.
The format string "%Y-%m-%d %H:%M:%S" might suit your needs.
https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
I'm having a problem with running a select query, using mysql-python, on an established database. The issue is that a number, what Python refers to as a long, is returned instead of the data queried- it should be noted that this number corresponds to the number of records which should be returned (I logged into the database and ran the query from MySQL to make sure).
Here is the code:
db = MySQLdb.connect(db = 'testdb', user='testuser', passwd='test', host='localhost', charset='utf8', use_unicode=True)
dbc = db.cursor()
result = dbc.execute("""SELECT %s FROM example_movie""", ('title',))
urls = [row[0] for row in result]
The last bit of code, urls = [row[0] for row in result] is to put everything into a list.
The error looks like this:
TypeError: 'long' object is not iterable
When I have python print result it returns:
('RESULT:', 52L)
When I enclose result like str(result) it just returns the number 52 (not long)
Any help and suggestions are greatly appreciated!
The return value from dbc.execute is not the results of the select; I believe it is the number of rows in the results. In order to get the actual results you need to call one of the fetch methods. See documentation here.
You should update your code to read:
db = MySQLdb.connect(db = 'testdb', user='testuser', passwd='test', host='localhost', charset='utf8', use_unicode=True)
dbc = db.cursor()
row_count = dbc.execute("""SELECT title FROM example_movie""")
results = dbc.fetchall()
urls = [row[0] for row in result]
Very similar to this question MySQL Dynamic Query Statement in Python
However what I am looking to do instead of two lists is to use a dictionary
Let's say i have this dictionary
instance_insert = {
# sql column variable value
'instance_id' : 'instnace.id',
'customer_id' : 'customer.id',
'os' : 'instance.platform',
}
And I want to populate a mysql database with an insert statement using sql column as the sql column name and the variable name as the variable that will hold the value that is to be inserted into the mysql table.
Kind of lost because I don't understand exactly what this statement does, but was pulled from the question that I posted where he was using two lists to do what he wanted.
sql = "INSERT INTO instance_info_test VALUES (%s);" % ', '.join('?' for _ in instance_insert)
cur.execute (sql, instance_insert)
Also I would like it to be dynamic in the sense that I can add/remove columns to the dictionary
Before you post, you might want to try searching for something more specific to your question. For instance, when I Googled "python mysqldb insert dictionary", I found a good answer on the first page, at http://mail.python.org/pipermail/tutor/2010-December/080701.html. Relevant part:
Here's what I came up with when I tried to make a generalized version
of the above:
def add_row(cursor, tablename, rowdict):
# XXX tablename not sanitized
# XXX test for allowed keys is case-sensitive
# filter out keys that are not column names
cursor.execute("describe %s" % tablename)
allowed_keys = set(row[0] for row in cursor.fetchall())
keys = allowed_keys.intersection(rowdict)
if len(rowdict) > len(keys):
unknown_keys = set(rowdict) - allowed_keys
print >> sys.stderr, "skipping keys:", ", ".join(unknown_keys)
columns = ", ".join(keys)
values_template = ", ".join(["%s"] * len(keys))
sql = "insert into %s (%s) values (%s)" % (
tablename, columns, values_template)
values = tuple(rowdict[key] for key in keys)
cursor.execute(sql, values)
filename = ...
tablename = ...
db = MySQLdb.connect(...)
cursor = db.cursor()
with open(filename) as instream:
row = json.load(instream)
add_row(cursor, tablename, row)
Peter
If you know your inputs will always be valid (table name is valid, columns are present in the table), and you're not importing from a JSON file as the example is, you can simplify this function. But it'll accomplish what you want to accomplish. While it may initially seem like DictCursor would be helpful, it looks like DictCursor is useful for returning a dictionary of values, but it can't execute from a dict.