Inserting data into a SQL server from an excel file - mysql

First of all, sorry for my lack of knowledge regarding databases, this is my first time working with them.
I am having some issues trying to get the data from an excel file and putting it into a data base.
Using answers from the site, I managed to kind of connect to the database by doing this.
import pandas as pd
import pyodbc
server = 'XXXXX'
db = 'XXXXXdb'
# create Connection and Cursor objects
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + server + ';DATABASE=' + db + ';Trusted_Connection=yes')
cursor = conn.cursor()
# read data from excel
data = pd.read_excel('data.csv')
But I dont really know what to do now.
I have 3 tables, which are connected by a 'productID', my excel file mimics the data base, meaning that all the columns in the excel file have a place to go in the DB.
My plan was to read the excel file and make lists with each column, then insert into the DB each column value but I have no idea how to create a query that can do this.
Once I get the query I think the data insertion can be done like this:
query = "xxxxxxxxxxxxxx"
for row in data:
#The following is not the real code
productID = productID
name = name
url = url
values = (productID, name, url)
cursor.execute(query,values)
conn.commit()
conn.close
Database looks like this.
https://prnt.sc/n2d2fm
http://prntscr.com/n2d3sh
http://prntscr.com/n2d3yj
EDIT:
Tried doing something like this, but i'm getting 'not all arguments converted during string formatting' Type error.
import pymysql
import pandas as pd
connStr = pymysql.connect(host = 'xx.xxx.xx.xx', port = xxxx, user = 'xxxx', password = 'xxxxxxxxxxx')
df = pd.read_csv('GenericProducts.csv')
cursor = connStr.cursor()
query = "INSERT INTO [Productos]([ItemID],[Nombre])) values (?,?)"
for index,row in df.iterrows():
#cursor.execute("INSERT INTO dbo.Productos([ItemID],[Nombre])) values (?,?,?)", row['codigoEspecificoProducto'], row['nombreProducto'])
codigoEspecificoProducto = row['codigoEspecificoProducto']
nombreProducto = row['nombreProducto']
values = (codigoEspecificoProducto,nombreProducto)
cursor.execute(query,values)
connStr.commit()
cursor.close()
connStr.close()
I think my problem is in how I'm defining the query, surely thats not the right way

Try this, you seem to have changed the library from pyodbc to mysql, it seems to expect %s instead of ?
import pymysql
import pandas as pd
connStr = pymysql.connect(host = 'xx.xxx.xx.xx', port = xxxx, user = 'xxxx', password = 'xxxxxxxxxxx')
df = pd.read_csv('GenericProducts.csv')
cursor = connStr.cursor()
query = "INSERT INTO [Productos]([ItemID],[Nombre]) values (%s,%s)"
for index,row in df.iterrows():
#cursor.execute("INSERT INTO dbo.Productos([ItemID],[Nombre]) values (%s,%s)", row['codigoEspecificoProducto'], row['nombreProducto'])
codigoEspecificoProducto = row['codigoEspecificoProducto']
nombreProducto = row['nombreProducto']
values = (codigoEspecificoProducto,nombreProducto)
cursor.execute(query,values)
connStr.commit()
cursor.close()
connStr.close()

Related

Why won't my extracted data from Spotify's API store in MySQL database?

I have connected to Spotify's API in Python to extract the top twenty tracks of a searched artist. I am trying to store the data in MySQL Workbench in a database named 'spotify_api', I created called 'spotify'. Before I added my code to connect to MySQL Workbench, my code worked correctly and was able to extract the list of tracks, but I have run into issues in getting my code to connect to my database. Below is the code I have written to both extract the data and store it into my database:
import spotipy
from spotipy.oauth2 import SpotifyClientCredentials
import mysql.connector
mydb = mysql.connector.connect(
host = "localhost",
user = "root",
password = "(removed for question)",
database = "spotify_api"
)
mycursor = mydb.cursor()
sql = 'DROP TABLE IF EXISTS spotify_api.spotify;'
mycursor.execute(sql)
sp = spotipy.Spotify(auth_manager=SpotifyClientCredentials(client_id="(removed for question)",
client_secret="(removed for question)"))
results = sp.search(q='sza', limit=20)
for idx, track in enumerate(results['tracks']['items']):
print(idx, track['name'])
sql = "INSERT INTO spotify_api.spotify (tracks, items) VALUES (" + \
str(idx) + ", '" + track['name'] + "');"
mycursor.execute(sql)
mydb.commit()
print(mycursor.rowcount, "record inserted.")
mycursor.execute("SELECT * FROM spotify_api.spotify;")
myresult = mycursor.fetchall()
for x in myresult:
print(x)
mycursor.close()
Every time I run my code in the VS Code terminal, I receive an error stating that my table doesn't exist. This is what it states:
"mysql.connector.errors.ProgrammingError: 1146 (42S02): Table 'spotify_api.spotify' doesn't exist"
I'm not sure what I need to fix in my code or in my database in order to eliminate this error and get my data stored into my table. In my table I have created two columns 'tracks' and 'items', but I'm not sure if my issues lie in my database or in my code.
Well, it seems pretty clear. You ran
DROP TABLE IF EXISTS spotify_api.spotify;
...
INSERT INTO spotify_api.spotify (tracks, items) VALUES ...
We won't even raise the spectre of the Chuck Berry
track titled little ol' Bobby Tables here.
You DROP'd it, then tried to INSERT into it.
That won't work.
You'll need to CREATE TABLE prior to the INSERT.

How to add column title to Excel while exporting result table from SQL query with Python

I have been trying to export a result table from a SQL query, but only the records inside the table is displayed in the Excel table. I am using MySQL and python(3.9.11) to run this code. Here is the code that I have been using to do so.
import pymysql
dbconn = pymysql.connect(<db details>)
cus = dbconn.cursor()
cus.execute('sql query')
res = cus.fetchall()
data = []
for i in res:
data += list(i)
var = open('main.csv','w')
for i in data:
var.write(str(i))
var.close()
Also, the result table's all columns data is written in a single column.
I tried and adding a var.write('/n'), but it will convert the rows into columns.
import pymysql
import csv
dbconn = pymysql.connect(DB Connection details)
cus = dbconn.cursor()
cus.execute("sql query here")
res = cus.fetchall()
column_names = [i[0] for i in cus.description]
fp = open('main.csv', 'w')
myFile = csv.writer(fp, lineterminator = '\n')
myFile.writerow(column_names)
myFile.writerows(res)

Migrating MySQL blob (image) to FileMaker container using PowerShell

In searching I've found a number of other people that have tried, but none that have been successful.
Here's the problem. I want to take a bunch of images I have stored on my MySQL server in blobs and move them into FileMaker containers.
The best lead I've got is the putas command. It looks something like putas ('$Image','JPEG').
My particular application is as follows. $DataSet.Image1 is a JPEG file stored as "0xFFD8....". The data being in this format may well be the issue, but I don't know what I'd need to convert it to first.
$cmd.CommandText = "update Checklists set Image1 = PutAs('$($DataSet.Image1)', 'JPEG')"
$cmd.ExecuteNonQuery();
All I keep getting is syntax error, but I've tried the syntax many different ways I can't get it to go no matter what I do.
I'd very much like to see someone having success with this to post their example. Any other ideas or workarounds are welcome as well.
Edit:
Here is some extra info. Greg Lane at Skeleton Key gives this example, but I'm not sure how to translate it to PowerShell.
import java.sql.*; import java.io.*;
def url = "jdbc:filemaker://localhost/fmserver_sample";
def driver = "com.filemaker.jdbc.Driver";
def user = "admin";
def password = "";
System.setProperty("jdbc.drivers", driver);
connection = DriverManager.getConnection (url, user, password);
filename = "/Users/Greg/Pictures/vacation/DSC_0202.jpg";
file = new File (filename);
inputstream = new FileInputStream (filename);
sql = "INSERT INTO english_nature (ID, img) VALUES (-1, PutAs(?, 'JPEG'))";
pstatement = connection.prepareStatement ( sql );
pstatement.setBinaryStream (1, inputstream, (int)file.length ());
pstatement.execute ();
//cleanup
pstatement = null;
inputstream = null;
file = null;
connection.close();
I figured it out. For anyone in the future here is how you do it.
$cmd.CommandText = "update Checklists set Image1 = PutAs(?, 'JPEG') where serial = '$($DataSet.serial)' AND ChecklistNumber = 1"
$cmd.Parameters.Add('?', $DataSet.Image1)
$cmd.Prepare()
$cmd.ExecuteNonQuery();

How to insert data into a table containing one single column?

I'm currently learning Python and MySQL and have an issue inserting data if my table has one single column (actually one auto-incremented id and a column).
I tried several syntaxes, "playing" with quotes and parenthesis, several ways to implement execute() method, but nothing worked.
Here is my statement :
import mysql.connector
db_name = "purbeurre"
list_categories = ['Drinks', 'Meat', 'Bread']
cnx = mysql.connector.connect(user='toto', password='toto', host='123.456.0.78')
cursor = cnx.cursor()
cursor.execute("USE {}".format(db_name))
insert_categories = ("INSERT INTO Categories (name) VALUES (%s)")
cursor.executemany(insert_categories, list_categories)
The error is : "ValueError: Could not process parameters"
If I add a column, the statement becomes this one and works fine :
import mysql.connector
db_name = "purbeurre"
list_categories = [('Drinks', 'Liquid products'), ('Meat', 'All kind of meat', ('Bread', 'Bakery products')]
cnx = mysql.connector.connect(user='toto', password='toto', host='123.456.0.78')
cursor = cnx.cursor()
cursor.execute("USE {}".format(db_name))
insert_categories = ("INSERT INTO Categories (name, description) VALUES (%s)")
cursor.executemany(insert_categories, list_categories)
As you can see, the only difference is the number of columns.
Any idea of what happens?
I received the answer.
The list of data was not correctly defined, here is the correct syntax :
list_categories = [('Drinks',), ('Meat',), ('Bread',)]
Beware of the comma before the parenthesis is closed to ensure each element in the list are tuples.

keyword search in string from mysql using python?

I am pulling from a mysql database table using python3.4. I use the csv module to write the rows of data from the database into .CSV format. Now I am trying toros figure out how I can vet the rows of data by keywords that may show up in the fourth column of data (row[3]). I was thinking of using the re moduleas below but I keep getting errors. Is it not possible to search keywords in a field that is string type and to filter those results if they have those keywords? I keep getting an error. Please help
import re
import csv
userdate = input('What date do you want to look at?')
query = ("SELECT *FROM sometable WHERE timestamp LIKE %s", userdate)
keywords = 'apples', 'bananas', 'cocoa'
# Execute sql Query
cursor.execute(query)
result = cursor.fetchall()
#Reads a CSV file and return it as a list of rows
def read_csv_file(filename):
"""Reads a CSV file and return it as a list of rows."""
for row in csv.reader(open(filename)):
data.append(row)
return data
f = open(path_in + data_file)
read_it = read_csv_file(path_in + data_file)
with open('file.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
for row in data:
match = re.search('keywords, read_it)
if match:
spamwriter.writerow(row)
I gave up on the regular expressions and used
for row in data:
found_it = row.find(keywords)
if found_it != -1:
spamwriter.writerow(row)