I am trying to export the output of a query in a MYSQL database to a CSV file in the local system using Python. There are 2 issues. First of all using fetchall() I am not getting any data( The same query in database produces more than 5000 rows of data), though I got data output initially. Secondly I would like to know the code to put the username and password in a separate file which the user cannot access but will be imported to this file when the script runs.
import os
import csv
import pymysql
import pymysql.cursors
d=open('c:/Users/dasa17/Desktop/pylearn/Roster.csv', 'w')
c=csv.writer(d)
Connection = pymysql.connect(host='xxxxx', user='xxxxx', password='xxxx',
db='xxxx',charset='utf8mb4',cursorclass=pymysql.cursors.DictCursor )
a=Connection.cursor()
a.execute("select statement")
data=a.fetchall()
for item in data:
c.writerow(item)
a.close()
d.close()
Connection.close()
Related
So I have a MySQL database, let's call it "MySQLDB". When trying to create a new table (let's call it datatable) and insert data from a pandas dataframe, my code keeps adding rows to the SQL table, and I'm not sure if they are duplicates or not. For reference, there are around 50,000 rows in my pandas dataframe, but after running my code, the SQL table contains over 1 million rows. Note that I am using XAMPP to run a local MySQL server on which the database "MYSQLDB" is stored. Below is a simplified/generic version of what I am running. Note I have removed the port number and replaced it with generic [port] in this post.
import pandas as pd
from sqlalchemy import create_engine
import mysql.connector
pandas_db = pd.read_csv('filename.csv', index_col = [0])
engine = create_engine('mysql+mysqlconnector://root:#localhost:[port]/MySQLDB', echo=False)
pandas_db.to_sql(name='datatable', con=engine, if_exists = 'replace', chunksize = 100, index=False)
Is something wrong with the code? Or could it be something to do with XAMPP or the way I set up my database? If there is anything I could improve, please let me know.
I haven't found any other good posts that describe having the same issue.
***import pandas as pd
employees = {'Name of Machining': ['milling','Drilling','Drilling','Chamfering'],
'Speed': [275,275,275,275],
'Feed': [0.28,0.03,0.03,0.28],
'Tool': ['EndMill','TwistDrill','TwistDrill','EndMill']
}
df = pd.DataFrame(employees, columns= ['Name of Machining','Speed','Feed','Tool'])***
i have used panda to contruct a table here. I am working on the project to save the table in
a database how can i save the table so constructed in a database)
print (df)
If it's a MySQL database, you can use the SQLAlchemy toolkit.
Make sure to run this in your terminal (for Windows) to install the packages :
pip install sqlalchemy pymysql
And then you can use pandas.DataFrame.to_sql to create, append or overwrite a table :
import pandas as pd
from sqlalchemy import create_engine
# - Creating the dataframe
employees = {'Name of Machining': ['milling','Drilling','Drilling','Chamfering'],
'Speed': [275,275,275,275],
'Feed': [0.28,0.03,0.03,0.28],
'Tool': ['EndMill','TwistDrill','TwistDrill','EndMill']
}
df = pd.DataFrame(employees)
# - Creating an engine to connect to MySQL Database
host, db, user, pw = "localhost", "mydb_name", "my_user_name", "my_password"
engine = create_engine(f"mysql+pymysql://{user}:{pw}#{host}/{db}")
# - Inseting the dataframe to the MySQL table
df.to_sql('employees', engine, index=False)
Read here in the official documentation to understand more.
I am using exporting neo4j all db to json using apoc APIs & again importing with same. Import query executes successfully but cannot find any data in neo4j.
Export query:
CALL apoc.export.json.all('complete-db.json',{useTypes:true, storeNodeIds:false})
Import query:
CALL apoc.load.json('complete-db.json')
When I execute:
MATCH (n) RETURN n
It shows no results found.
This is a little bit confusing but apoc.load.json just reads(loads) data from the JSON File/URL.
It doesn't import the data or create the graph. You need to create the graph(nodes and/or relationships) using the Cypher statements.
In this case, you just read the file and didn't do anything with it so statement executed successfully. Your query isn't an import query, it's a JSON load query.
Refer the following example for import using apoc.load.json:
CALL apoc.load.json('complete-db.json') YIELD value
UNWIND value.items AS item
CREATE (i:Item(name:item.name, id:item.id)
apoc.import.json does what you need.
The export-import process:
Export:
CALL apoc.export.json.all('file:///complete-db.json', {useTypes:true, storeNodeIds:false})
Import:
CALL apoc.import.json("file:///complete-db.json")
(#rajendra-kadam explains why your version does not work, and this is the complementary API call to apoc.export.json.all you were expecting. )
I work as a Business Analyst and new to Python.
In one of my project, I want to extract data from .csv file and load that data into my MySQL DB (Staging).
Can anyone guide me with a sample code and frameworks I should use?
Simple program to create sqllite. You can read the CSV file and use dynamic_entry to insert into your desired target table.
import sqlite3
import time
import datetime
import random
conn = sqlite3.connect('test.db')
c = conn.cursor()
def create_table():
c.execute('create table if not exists stuffToPlot(unix REAL, datestamp TEXT, keyword TEXT, value REAL)')
def data_entry():
c.execute("INSERT INTO stuffToPlot VALUES(1452549219,'2016-01-11 13:53:39','Python',6)")
conn.commit()
c.close()
conn.close()
def dynamic_data_entry():
unix = time.time();
date = str(datetime.datetime.fromtimestamp(unix).strftime('%Y-%m-%d %H:%M:%S'))
keyword = 'python'
value = random.randrange(0,10)
c.execute("INSERT INTO stuffToPlot(unix,datestamp,keyword,value) values(?,?,?,?)",
(unix,date,keyword,value))
conn.commit()
def read_from_db():
c.execute('select * from stuffToPlot')
#data = c.fetchall()
#print(data)
for row in c.fetchall():
print(row)
read_from_db()
c.close()
conn.close()
You can iterate through the data in CSV and load into sqllite3. Please refer below link as well.
Quick easy way to migrate SQLite3 to MySQL?
If that's a properly formatted CSV file you can use the LOAD DATA INFILE MySQL command and you won't need any python. Then after it is loaded in the staging area (without processing) you can continue transforming it using sql/etl tool of choice.
https://dev.mysql.com/doc/refman/8.0/en/load-data.html
A problem with that is that you need to add all columns but still even if you have data you don't need you might prefer to load everything in the staging.
I'm trying to upload a csv file with a lot of posible clients (15000) on a MySQL table. I want to keep on a table for later retrieve info, complete forms and make users.
Now i'm on the beginning, trying to import the csv to the MySQL.
I read some solutions that takes smarterCSV gem, so i must do a migration with the db structure and then execute the rake task or not needed to execute the migration for this?
The kind of code i want to use for import the csv is something like i read on before posts like Ruby on Rails - Import Data from a CSV file
require 'smarter_csv'
options = {}
SmarterCSV.process('input_file.csv', options) do |chunk|
chunk.each do |data_hash|
Moulding.create!( data_hash )
end
end
You may use my importer gem. It also uses SmarterCSV.
https://github.com/michaelnera/active_record_importer