import pandas as pd
from sqlalchemy import create_engine
db_instrumentos = pd.DataFrame(columns=('Market','Ticker','cficode'))
db_instrumentos = db_instrumentos.append({'Market':'NYSE'],
'Ticker':'MMM',
'cficode':'EAEWD25A'},
ignore_index=True)
db = db_obtenerInstrumentos
sql_engine = create_engine('mysql+pymysql://root:#localhost')
sql_conn = sql_engine.connect()
sql_conn.execute(f"CREATE database IF NOT EXISTS proof_Rofex")
sql_conn.execute(f"USE proof_Rofex")
db.to_sql(con=sql_conn, name='proof_table2', if_exists="replace")
sql_conn.close()
I am wanting to use this line of code db.to_sql (with = sql_conn, name = 'proof_table2', if_exists = "replace") and it throws me the following error
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1050, "Table 'proof_table2' already exists") [SQL: CREATE TABLE proof_table2 (index BIGINT, Market TEXT, Ticker TEXT, cficode TEXT)
] (Background on this error at: http://sqlalche.me/e/14/e3q8)
the proof_table2 table already exists but precisely that line should allow me to make a drop and replace ... that error that throws me is precisely on that line
I was able to solve it by adding schema, for your example it should be as below:
db.to_sql(con=sql_conn, name='proof_table2', if_exists='replace',
schema='proof_Rofex')
Related
i want to put a Panda Dataframe to a IBM i Series / AS400. I already researched a much, but now I am stuck.
I already made a lot of queries, where I use pyodbc. For df.to_sql() I should use, as readed on other stacks, sqlalchemy with the ibm_db_sa dialect.
My actual code is:
CONNECTION_STRING = (
"driver={iSeries Access ODBC Driver};"
"System=111.111.111.111;"
"database=TESTDB;"
"uid=USER;"
"pwd=PASSW;"
)
quoted = urllib.parse.quote_plus(CONNECTION_STRING)
engine = create_engine('ibm_db_sa+pyodbc:///?odbc_connect={}'.format(quoted))
create_statement = df.to_sql("TABLETEST", engine, if_exists="append")
the following packages are installed
python 3.9
ibm-db 3.1.3
ibm-db-sa 0.3.7
ibm-db-sa-py3 0.3.1.post1
pandas 1.3.5
pip 22.0.4
setuptools 57.0.0
SQLAlchemy 1.4.39
when I run, i get the following error:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42S02', '[42S02] [IBM][System i Access ODBC Driver][DB2 for i5/OS]SQL0204 - COLUMNS in SYSCAT type *FILE not found. (-204) (SQLPrepare)')
[SQL: SELECT "SYSCAT"."COLUMNS"."COLNAME", "SYSCAT"."COLUMNS"."TYPENAME", "SYSCAT"."COLUMNS"."DEFAULT", "SYSCAT"."COLUMNS"."NULLS", "SYSCAT"."COLUMNS"."LENGTH", "SYSCAT"."COLUMNS"."SCALE", "SYSCAT"."COLUMNS"."IDENTITY", "SYSCAT"."COLUMNS"."GENERATED"
FROM "SYSCAT"."COLUMNS"
WHERE "SYSCAT"."COLUMNS"."TABSCHEMA" = ? AND "SYSCAT"."COLUMNS"."TABNAME" = ? ORDER BY "SYSCAT"."COLUMNS"."COLNO"]
[parameters: ('USER', 'TABLETEST')]
(Background on this error at: https://sqlalche.me/e/14/f405)
I think, the dialect could be wrong, because the parameters are the username and the table for the ODBC connection?
AND: I am not really sure, whats the difference between ibm_db_sa and ibm_db?
I tried a few days again, before someone is trying to do this via sqlalchemy should do it via pyodbc.
Here is my working example
refering the df_to_sql_bulk_insert function to this
(and now I am currently using my system-DSN):
def df_to_sql_bulk_insert(df: pd.DataFrame, table: str, **kwargs) -> str:
df = df.copy().assign(**kwargs)
columns = ", ".join(df.columns)
tuples = map(str, df.itertuples(index=False, name=None))
values = re.sub(r"(?<=\W)(nan|None)(?=\W)", "NULL", (",\n" + " " * 7).join(tuples))
return f"INSERT INTO {table} ({columns})\nVALUES {values}"
cnxn = pyodbc.connect("DSN=XXX")
cursor = cnxn.cursor()
sqlstr = df_to_sql_bulk_insert(df,"DBXXX.TBLXXX")
cursor.execute(sqlstr)
cnxn.commit()
How to upload whole text of a text file in a row in database, the text gets divided and is stored in subsequent rows.
This is the code of my SQL file, database name is info containing table named info having two columns des1 and des2 having field VARCHAR(3000):
use info;
INSERT INTO info (des1) VALUES (LOAD_FILE('eng.txt'));
select * from(info);
I am getting following output:
des1 des2
NULL NULL
I also attatched an image showing output
I expect all of the text in the file to be in a single row of the database, this has to be done in terminal
There might be a problem with how you form the string that is to be inserted. You can do it easily by the help of a programming language.
I am providing a simple solution that would work in Python.
import mysql.connector
mydb = mysql.connector.connect(
host="localhost",
user="root",
passwd="password",
database="db"
)
def fileReadToString(filename):
result = ""
with open(filename, "r") as ins:
for line in ins:
result +=(line)
return result
file = fileReadToString('eng.txt')
mycursor = mydb.cursor()
sql = "INSERT INTO info VALUES (%s)"
val = (file)
mycursor.execute(sql, val)
mydb.commit()
print(mycursor.rowcount, "done")
LOAD DATA INFILE is for structured data. For unstructured data use LOAD_FILE():
INSERT INTO info (des1) VALUES (LOAD_FILE('eng.txt'))
I'm looking for a clean way to add multiple values on one row which corresponds to a list of columns in mysql.
Essentially I have two lists:
cols = ['Col_A', 'Col_B','Col_E', 'Col_H, 'Col_n'....]
vals = ['1','56','HEX 00 A0 DB 00', 'Pass', '87'....]
The lists lengths can be 100+ items. Both cols and vals lists will be the same length, so each cols item has a corresponding vals item.
I am using pymysql to connect to an SQL database on a network storage device running MariaDB.
Here's a snippet of my non-working function attempt at passing the two lists:
def add_to_database(cols, vals):
connection = pymysql.connect(host='11.22.33.44',
user='usr',
password='pass',
db='my_db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor,
autocommit=True)
cursor = connection.cursor()
try:
cursor.execute("CREATE TABLE data_tbl (%s)" % 'id INTEGER PRIMARY KEY')
except:
pass
# Add Column and Values lists to database here
for item in cols:
try:
# First add columns if not already present
cursor.execute("ALTER TABLE data_tbl ADD COLUMN " + str(item))
except:
# Pass column add if already present
pass
cursor.execute("INSERT INTO data_tbl '{0}' VALUES '{1}';".format(cols, vals,))
conn.close()
return
I'm still new to SQL and I've also been playing around with the SQL syntax, so apologies if the code looks a bit odd now.
The common error I get is below:
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''('Col_A', 'Col_B', 'Col_E', 'Col_H', 'Col_...' at line 1")
I am running an R script that adds data to mySQL database. I usually format the data and add it as as it comes in after couple of hours (data string is not continuous). My first set of data was added properly in MySQL database. The second string of data can not be added properly.
con = dbConnect(MySQL(), user='root', password='xxxxxx', dbname='test', host='localhost')
dbWriteTable(con, 'Tables', value = parseTweets(filterStream(file.name= "", track=c("lebron"), timeout=10, oauth=my_oauth)))
When I rerun the last code (dbWriteTable) again, it gives me following error
Error: Error in .local(conn, statement, ...) :
could not run statement: Table 'tables' already exists
I also used
dbWriteTable(con, 'Tables', value = parseTweets(filterStream(file.name= "", track=c("lebron"), timeout=10, oauth=my_oauth)), append = TRUE)
but it provides the same error
For some reason giving setting append to TRUE is not working. Instead give it a number. See the code below for better understanding
dbWriteTable(con, 'Tables', value = parseTweets(filterStream(file.name= "", track=c("lebron"), timeout=100, oauth=my_oauth)), overwrite = 0, row.names = 0, append = 1)
Can you do such a thing? I have the following but cursor.execute does not like the syntax of selectSQL. Ultimately I'm looking to iterate through all tables in a .accdb and insert records from each table into a another .accdb with the same tables and fields. Reason being, bringing over new records from field data collection on TabletPCs to master database on server.
import pyodbc
connOtherDB = pyodbc.connect("Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ='path to my dbase;")
otherDBtbls = connOtherDB.cursor().tables()
for t in otherDBtbls:
if t.table_name.startswith("tbl"): #ignores MS sys tables
cursor = connOtherDB.cursor()
#selectSQL = '"SELECT * from '+ str(t.table_name) + '"'
cursor.execute("SELECT * from tblDatabaseComments") #this works if I refer to a table by name
cursor.execute(selectSQL) #this does not work. throws SQL syntax error
row = cursor.fetchone()
if row:
print t.table_name
print row
Use str.format() to ease building of SQL statements:
import pyodbc
connOtherDB = pyodbc.connect("Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ='path to my dbase;")
otherDBtbls = connOtherDB.cursor().tables()
for t in otherDBtbls:
if t.table_name.startswith("tbl"): #ignores MS sys tables
cursor = connOtherDB.cursor()
selectSQL = 'SELECT * FROM {}'.format(t.table_name)
cursor.execute(selectSQL)
row = cursor.fetchone()
if row:
print t.table_name
print row
As an aside, take a look a PEP 8 -- Style Guide for Python Code for guidance on maximum line length and variable naming, among other coding conventions.