I'm trying to insert JSON data into an MySQL database:
def mapClients():
for d in devices:
clientMap = d['dot11.device']['dot11.device.associated_client_map'].keys()
for item in clientMap:
clientList = kr.device_by_mac(item)
times = kr.device_summary_since()
for c in clientList:
sqlMac = c['kismet.device.base.macaddr'],
sqlType = c['kismet.device.base.type'],
sqlManuf = c['kismet.device.base.manuf'],
ktime = c['kismet.device.base.last_time'],
for t in ktime:
sqlTime = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(t))
cur.execute("INSERT INTO devices(apID,mac,type,manuf,last_seen) VALUES(1,'" + str(sqlMac) + "','" + str(sqlType) + "','" + str(sqlManuf) + "','" + sqlTime + "');")
conn.commit()
mapClients()
This returns the following error:
pymysql.err.ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'VALUES(1,'(u'58:E2:8F:CF:20:B3',)','(u'Wi-Fi Client',)','(u'Apple',)','20-10-201' at line 1")
I can see from the error that the various values are being suffixed with a 'u'. I understand through a lot of searching and learning that (I think) this means the data is unicode.
What I want to do is find a way of converting/decoding the data so the INSERT statements work. Some of the variables are tuples, some strings. Any help much appreciated.
You are inserting tuples, not strings; remove the trailing commas:
sqlMac = c['kismet.device.base.macaddr']
sqlType = c['kismet.device.base.type']
sqlManuf = c['kismet.device.base.manuf']
ktime = c['kismet.device.base.last_time']
sqlTime = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(ktime))
It’s the trailing comma that turns those expressions into tuples, and str() on a tuple gives you the container Unicode string as the u'....' representation that then clashes with the ' quoting you are adding.
Note that removes the need to loop over ktime!
Next, you really want to use SQL parameters, not string concatenation. Use placeholders instead of '" + str(...) + "', and leave handling of quoting to the database adapter:
cur.execute("""
INSERT INTO devices (apID, mac, type, manuf, last_seen)
VALUES (1, %s, %s, %s, %s)
""", (sqlMac, sqlType, sqlManuf, sqlTime))
The %s are the placeholders; depending on your exact MySQL Python library, you may need to use ? questionmarks instead.
Not only would this let you avoid having to think about quoting, it also removes a serious security issue: the JSON you load could contain a SQL injection attack and SQL parameters are the best way to neutralise that attack vector.
For the error message you posted, you have forgotten to place the closing parentheses before the VALUES in your SQL Query. The query should be like:
cur.execute("INSERT INTO devices(apID,mac,type,manuf,last_seen) VALUES(1,'" + str(sqlMac) + "','" + str(sqlType) + "','" + str(sqlManuf) + "','" + str(sqlTime) + "');")
Related
Following SQL statement is creating an error with message :
"Message: Fatal error encountered during command execution."
"Inner exception: Parameter '#LastUserID' must be defined."
If I directly use LAST_INSERT_ID() instead of LastUserID, it always returns zero (hence fails at second insert) when executed like this.
I don't see my syntax is different than in mySQL document.
Could some one help me ?
string Query = #"INSERT INTO login (" +
"LOGIN_EMAIL," +
"LOGIN_PASSWORD," +
"LOGIN_SALT," +
"LOGIN_LAST_LOGIN_DATE," +
// "LOGIN_LAST_LOGIN_LOCATION," +
"LOGIN_ACCOUNT_STATUS," +
"LOGIN_LOGIN_ATTEMPTS," +
"LOGIN_CREATED_DATE) " +
"VALUES (" +
"#Parameter2," +
"#Parameter3," +
"#Parameter4," +
"#Parameter5," +
// "#Parameter6," +
"#Parameter6," +
"#Parameter7," +
"#Parameter8); " +
"SET #LastUserID = LAST_INSERT_ID(); " +
"INSERT INTO user_role (" +
"USER_ROLE_USER_ID," +
"USER_ROLE_ROLE," +
"USER_ROLE_STATUS," +
"USER_ROLE_CREATED_DATE) " +
"SELECT " +
"#LastUserID," +
"#Parameter9," +
"#Parameter10," +
"#Parameter11 " +
"FROM dual WHERE NOT EXISTS (SELECT USER_ROLE_USER_ID FROM user_role " +
"WHERE USER_ROLE_USER_ID = #LastUserID AND USER_ROLE_ROLE = #Parameter9)";
MySqlCommand oCommand = new MySqlCommand(Query, oMySQLConnecion);
oCommand.Transaction = tr;
Create a procedure in which you first do your insert, cache the last inserted id, do the other insert and let it print out your parameters with a bool if your last insert worked or not. That way you can debug it properly.
In general you should avoid concatinating strings to generate sql-commands or you might get troubles with parameters containing unexpected characters or be hit by a injection.
Simple fix : Replace "$LastUserID" with "$'LastUserID'". The ephostophy makes the difference.
Hi all im having issues with this section of script;
linkString = "Info "
SQL_query ="INSERT INTO search (Title,Program,Open) VALUES ('"& txtTitle.value &"','"& txtProgram.value &"','"& linkString.value &"')"
I basically need the code to input the "linkString" value into the section of my SQL Query however any way I try I recieve either object errors or string errors if I attempt to set the value?
Thanks,
James
linkString is a string, not an object. So there is no linkString.value to concatenate. Just use plain linkString.
Update wrt comments:
Assuming the objects and their properties are decent, your concatenations - though disgusting - are correct. Evidence:
txtTitle_value = "AAA"
txtProgram_value = "BBB"
linkString = "CCC"
txtOpen_value = "111"
linkString = "Info"
WScript.Echo linkString
SQL_query ="INSERT INTO search (Title,Program,Open) VALUES ('"& txtTitle_value &"','"& txtProgram_value &"','"& linkString &"')"
WScript.Echo SQL_query
output:
cscript 26403056.vbs
<a href="#" onclick="TINY.box.show({iframe:'nodes/111',boxid:'frameless',width:750,height:450,fixed:false,mask
opacity:0})">Info</a>
INSERT INTO search (Title,Program,Open) VALUES ('AAA','BBB','<a href="#" onclick="TINY.box.show({iframe:'nodes
/111',boxid:'frameless',width:750,height:450,fixed:false,maskopacity:0})">Info</a>')
If you get "missing operator" errors you removed those operators yourself.
Update wrt comments II:
If you get an SQL error, escape the single quotes in linkString; check your SQL docs to learn how ('', \', ...).
In the syntax of load infile data i saw that the fields and line clauses are optional. So I used only character set clause for utf8
Here my sql:
cmd = new MySqlCommand("LOAD DATA INFILE " + filename + " INTO TABLE " + tblname + " CHARACTER SET 'UTF8'", conn);
filename is the addresse it's format is: "E:\Macdata\20131228\atelier.sql"
table name is directly taken from database is as : "atelier"
But I get the error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'E:\Macdata\20131228\atelier.sql INTO TABLE atelier CHARACTER SET 'UTF8'' at line 1
What is the mistake in my query command ?
MYSQLversion is 5.0.10 with XAMPP
After changing the query I begin to receive fatal error number 0 (enclosed filename with ')
cmd = new MySqlCommand("LOAD DATA LOCAL INFILE '" + filename + "' IGNORE INTO TABLE " + tblname + " CHARACTER SET UTF8", conn);
My data file has this form which works on phpmyadmin
INSERT INTO `atelier` VALUES(1, 'Chateau Carbonnieux -1', '2013-12-26', 23, 10, 0, '4 macarons differents', 'mamie', '2013-12-15 11:09:14', 'sabrina', '2013-12-18 05:29:26');
As the error says, your statements is wrong. Quotes are missing in your first statement (see second statement). Check the syntax here:
http://dev.mysql.com/doc/refman/5.6/en/load-data.html
Some sparse notes:
0 is not a fatal error, it's the code for success.
IGNORE handles duplicate rows, not syntax errors.
I am writing a script to read data from a file by lines and insert each line into MySQL database. I use mysql.connector to do this. Here is a piece of script.
def insert_query(data):
return ("INSERT INTO " + tblname + " (`log`) " + "VALUES " + "(" + "'" + data + "'" + ")")
with open('data.txt', 'r') as f:
lines = f.readlines()
for line in lines:
add_line = insert_query(line)
cursor.execute(add_line)
cursor.commit()
File data.txt has size is 5Mbyte, but it has about 10000 lines.
tblname has 2 field: ID - INT (11) (auto-increment) , log - TEXT
When i run this script, it add to database about 100 lines and crashed. It report a error:
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an
error in your SQL syntax; check the manual that corresponds to your
MySQL server version for ')'
MySQL version : 5.5.27
How to solve this problem ? Thanks.
Your insert statement is incorrect: 'tblname' is undefined in your function, hence the syntax error. However, there are bigger issues with how you solve it: what if a line in the log has quotes and brackets?
The following code show how to read data from a file and insert it line by line:
stmt = "INSERT INTO {table} (c1) VALUES (%s)".format(table=table_name)
with open('data.txt', 'r') as fp:
for line in fp:
data = (line.strip(),)
cur.execute(stmt, (line.strip(),))
cnx.commit()
Or using executemany(), which would be faster:
with open('data.txt', 'r') as fp:
cur.executemany(stmt, [(line.strip(),) for line in fp])
cnx.commit()
Does anyone know what could be causing this error? I'm trying to convert a MySQL site to Postgres so I can host on Heroku. I'm new to database syntax, and this problem has been bugging me for days.
PG::Error: ERROR: syntax error at or near "ON"
LINE 1: ...tores ("key", "value") VALUES ('traffic:hits', 0) ON DUPLICA...
^
Here's the github page for the site I'm trying to convert. https://github.com/jcs/lobsters
This is the query. I added the backslash double quotes in replace of `.
if Rails.env == "test"
Keystore.connection.execute("INSERT OR IGNORE INTO " <<
"#{Keystore.table_name} (\"key\", \"value\") VALUES " <<
"(#{q(key)}, 0)")
Keystore.connection.execute("UPDATE #{Keystore.table_name} " <<
"SET \"value\" = \"value\" + #{q(amount)} WHERE \"key\" = #{q(key)}")
else
Keystore.connection.execute("INSERT INTO #{Keystore.table_name} (" +
"\"key\", \"value\") VALUES (#{q(key)}, #{q(amount)}) ON DUPLICATE KEY " +
"UPDATE \"value\" = \"value\" + #{q(amount)}")
end
Postgres' INSERT doesn't support MySQL's variant INSERT ... ON DUPLICATE KEY UPDATE.
For alternatives see the answers to this question.
I was working on this exact code last night, here's an initial take at how I fixed it, following this answer :
def self.put(key, value)
key_column = Keystore.connection.quote_column_name("key")
value_column = Keystore.connection.quote_column_name("value")
if Keystore.connection.adapter_name == "SQLite"
Keystore.connection.execute("INSERT OR REPLACE INTO " <<
"#{Keystore.table_name} (#{key_column}, #{value_column}) VALUES " <<
"(#{q(key)}, #{q(value)})")
elsif Keystore.connection.adapter_name == "PostgreSQL"
Keystore.connection.execute("UPDATE #{Keystore.table_name} " +
"SET #{value_column} =#{q(value)} WHERE #{key_column} =#{q(key)}")
Keystore.connection.execute("INSERT INTO #{Keystore.table_name} (#{key_column}, #{value_column}) " +
"SELECT #{q(key)}, #{q(value)} " +
"WHERE NOT EXISTS (SELECT 1 FROM #{Keystore.table_name} WHERE #{key_column} = #{q(key)}) "
)
elsif Keystore.connection.adapter_name == "MySQL" || Keystore.connection.adapter_name == "Mysql2"
Keystore.connection.execute("INSERT INTO #{Keystore.table_name} (" +
"#{key_column}, #{value_column}) VALUES (#{q(key)}, #{q(value)}) ON DUPLICATE KEY " +
"UPDATE #{value_column} = #{q(value)}")
else
raise "Error: keystore requires db-specific put method."
end
true
end
There's a number of things to be fixed in the lobsters codebase beyond just this for postgres compatability - found mysql specific things in other controller files. I'm currently working on them at my own lobsters fork at https://github.com/seltzered/journaltalk - postgres fixes should be commited on there in the coming day or two.