I am trying to run the following code to populate a table in parallel for a certain application. First the following function is defined which is supposed to connect to my db and execute the sql command with the values given (to insert into table).
def dbWriter(sql, rows) :
# load cnf file
MYSQL_CNF = os.path.abspath('.') + '/mysql.cnf'
conn = MySQLdb.connect(db='dedupe',
charset='utf8',
read_default_file = MYSQL_CNF)
cursor = conn.cursor()
cursor.executemany(sql, rows)
conn.commit()
cursor.close()
conn.close()
And then there is this piece:
pool = dedupe.backport.Pool(processes=2)
done = False
while not done :
chunks = (list(itertools.islice(b_data, step)) for step in
[step_size]*100)
results = []
for chunk in chunks :
print len(chunk)
results.append(pool.apply_async(dbWriter,
("INSERT INTO blocking_map VALUES (%s, %s)",
chunk)))
for r in results :
r.wait()
if len(chunk) < step_size :
done = True
pool.close()
Everything works and there are no errors. But at the end, my table is empty, meaning somehow the insertions were not successful. I have tried so many things to fix this (including adding column names for insertion) after many google searches and have not been successful. Any suggestions would be appreciated. (running code in python2.7, gcloud (ubuntu). note that indents may be a bit messed up after pasting here)
Please also note that "chunk" follows exactly the required data format.
Note. This is part of this example
Please note that the only thing I am changing in the above example (linked) is that I am separating the steps for creation of and inserting into the tables since I am running my code on gcloud platform and it enforces GTID standards.
Solution was changing dbwriter function to:
conn = MySQLdb.connect(host = # host ip,
user = # username,
passwd = # password,
db = 'dedupe')
cursor = conn.cursor()
cursor.executemany(sql, rows)
cursor.close()
conn.commit()
conn.close()
We are using production and staging databases in our application.
Our requirement is to insert all the records to staging database when ever a record is added in production database, so that both the servers are consistent and same data.
I have used Mysql2 client pool to connect to staging server and insert the record that is added to production.
here is my code:
def create
#aperson = Person.new
#person = #aperson.save
if #person && Rails.env == "production"
#add_new_person_to_staging
client = Mysql2::Client.new(:host => dbconfig[:host], :username => dbconfig[:username], :password => dbconfig[:password], :database => dbconfig[:database])
#person_result = client.query('INSERT INTO user_types(user_name, regex, code) Values ("myname" , "\.myregex\." , "ns" );')
end
end
Here "#person_result" record is inserted to mysql table but the "regex" column eliminates "\" slashes.
like : user_name = myname, regex = .myregex., code = ns
when I manually execute the "Insert" query in mysql command line it inserts as it is along with \ slash. but not through "client.query"
Why does \ slash is eliminated. please help me here.
Thanks.
\ is likely being removed by the MySQL2 client as part of a SQL injection protection preprocessor.
Have you looked at trying either a double backslash or using the escape method to properly escape the string?
Try using this
#person_result = client.query('INSERT INTO user_types(user_name, regex, code) Values (myname , "\."+myregx+".\" , ns )')
I'm trying to get data from MySQL DB into Rstudio-server. My actions are like
mydb = dbConnect(MySQL(), user='user', password='password', dbname='dbname', host='localhost')
query <- stri_paste('select sellings.updated_at AS Up_Date, concat(item_parameters.title, " ", ad_attributes.int_value) AS Class, CONCAT(geos.name, " ", geos.kind) AS place, geos.lon, geos.lat, sellings.price AS price, ((geo_routes.distance*2/1000 + 100)) AS delivery_cost FROM sellings, users, item_parameters, ad_attributes, geos, geo_routes WHERE users.encrypted_password!="" && item_parameters.title="Класс" && sellings.price IS NOT NULL && ad_attributes.int_value IS NOT NULL AND users.id=sellings.user_id AND item_parameters.id=ad_attributes.item_parameter_id AND sellings.id = ad_attributes.ad_id AND sellings.geo_guid = geos.guid AND geos.routable_guid = geo_routes.src_guid AND geo_routes.distance = (SELECT geo_routes.distance FROM geo_routes, geos WHERE geos.guid = sellings.geo_guid AND geo_routes.src_guid = geos.routable_guid AND geo_routes.dst_guid = (SELECT geos.routable_guid FROM geos WHERE geos.name = "Воронеж" && geos.kind = "г")) ORDER BY Up_Date;')
rs = dbGetQuery(mydb, query)
And I get an empty dataframe. But when I do the same with my local DB everything is OK. The query takes a pretty long time, about 3 minutes, but it works properly. Moreover the same query works right from the command line in MySQL. On the server, it takes about 4 seconds. OS of server is Debian 7, OS of local machine is Win 8. Any idea?
Sometimes when querying from the command line the default schema has been set in a previous command. This command doesn't carry over to R so the exact same query from a command line to a R session might not work. Maybe check the dbname.
Insert the below statements in your SQL query
SET NOCOUNT ON
SET ANSI_WARNINGS OFF
It worked for me
I am having trouble with my server.
It is a CentOS RedHat Linux server and runs "Dashing" a Ruby/Sinatra-based dashboard.
I am trying to close the active connections as defined by my MySQL database "SHOW PROCESSLIST;"
Example.rb File
require 'mysql2'
SCHEDULER.every '10s'do
db = Mysql.new('host_name', 'database_name', 'password', 'table')
mysql1 = "SELECT `VAR` from `TABLE` ORDER BY `VAR` DESC LIMIT 1"
result1 = db.query(mysql1)
result1.each do |row|
strrow1 = row[0]
$num1 = strrow1.to_i
end
...
db.close
LINK[0] = { label: 'LABEL', value: $num1}
...
send_event('LABEL FOR HTML', { items: LINK.values })
end
However, after a few clicks back and forth, it is clear that the database does not drop the connections, but instead keeps them. This causes the browser to slow down to the point that loading a page becomes impossible and the output of the log reads:
"max_user_connections" reached
Can anyone think of a way to fix this?
It is a best practice for DB/File/handle stuff to be in a begin/rescue/ensure block. It could be that something is happening and Rufus/Dashing is just being quiet about the error since they trap exceptions and go on their merry way. This would prevent your db connection from closing. The symptoms you are having could be from a similar problem, either way it's a good idea.
SCHEDULER.every '10s'do
begin
db = Mysql.new('host_name', 'database_name', 'password', 'table')
# .... stuff ....
rescue
# what happens if an error happens? log it, toss it, ignore it?
ensure
db.close
end
# ... more stuff if you want ...
end
I know that there is general_log that logs all queries, but I want to find out which query has an error, and get the error message. I have tried running an error query on purpose, but it logs as a normal query and doesn't report it with error. Any ideas?
There is no functionality in MySQL to do this.
You will have to stick with application-layer logging.
I know this is ancient now, but for anyone having the same issue that lands here from Google, here's my two cents.
If you're using the cli client, you can simply redirect your sterr to a file, and then parse through that.
mysql -u user -p 2> errors.log
It is possible to log error queries with the MariaDB Audit Plugin.
The MariaDB Audit Plugin works for MariaDB, MySQL and Percona Server.
For example, for these queries
select now();
select now()+();
select 9+();
select 'hello';
log seems like this:
20150807 23:00:36,mnv-Satellite-L300D,root,localhost,82,377,QUERY,`test`,'select now()
LIMIT 0, 1000',0
20150807 23:00:37,mnv-Satellite-L300D,root,localhost,82,379,QUERY,`test`,'select now()+()',1064
20150807 23:00:37,mnv-Satellite-L300D,root,localhost,82,382,QUERY,`test`,'select 9+()',1064
20150807 23:00:38,mnv-Satellite-L300D,root,localhost,82,383,QUERY,`test`,'select \'hello\'
LIMIT 0, 1000',0
Last column is return code. 0 is Ok. Else - error.
Even though this question is quite old I hope it will be useful to someone who searched for mysql log error queries or similar terms.
Not too long ago I also required mysqld to log only erroneous queries. I found that mysql-proxy enables you to do that and wrote a small LUA script:
local err_flag = false
function read_query( packet )
if packet:byte() == proxy.COM_QUERY then
local user = proxy.connection.client.username
local host = proxy.connection.client.src.name
if user:lower() == 'someuser' then -- change this to any condition where errors should be logged
proxy.queries:append(1, packet, {resultset_is_needed = true})
proxy.queries:append(2, string.char(proxy.COM_QUERY) .. "SET #last_query = '" .. string.sub(packet, 2) .. "'", {resultset_is_needed = true} )
proxy.queries:append(3, string.char(proxy.COM_QUERY) .. "SHOW WARNINGS", {resultset_is_needed = true} )
end
return proxy.PROXY_SEND_QUERY
end
end
function insert_query(err_t, err_n, err_m)
local query = "INSERT INTO `somedb`.`mysql_error` " .. -- change log destination
"(`date`, `err_num`,`err_type`, `err_message`, `problem_query`, `conn_id`)" ..
" VALUES ( NOW(), " ..
err_n .. "," .. "\"" ..
err_t .."\"" .. "," .. "\"" ..
err_m .. "\"" .. "," ..
"#last_query" .. "," ..
proxy.connection.server.thread_id .. ")"
proxy.queries:append(4, string.char(proxy.COM_QUERY) .. query, {resultset_is_needed = true})
return proxy.PROXY_SEND_QUERY
end
function read_query_result(inj)
local res = assert(inj.resultset)
if inj.id == 1 then
err_flag = false
if res.query_status == proxy.MYSQLD_PACKET_ERR then
err_flag = true
return proxy.PROXY_IGNORE_RESULT
end
elseif inj.id == 2 then
return proxy.PROXY_IGNORE_RESULT
elseif inj.id == 3 then
if err_flag == true then
for row in res.rows do
proxy.response.type = proxy.MYSQLD_PACKET_ERR
proxy.response.errmsg = row[3]
insert_query(row[1], row[2], row[3])
end
return proxy.PROXY_SEND_RESULT
end
return proxy.PROXY_IGNORE_RESULT
elseif inj.id == 4 then
return proxy.PROXY_IGNORE_RESULT
end
end
DDL needed for logging table, adjust somedb.mysql_error to liking, but don't forget to do so in the above LUA script also.
CREATE TABLE `somedb`.`mysql_error` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` datetime NOT NULL,
`err_num` smallint(6) NOT NULL,
`err_type` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
`err_message` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`problem_query` varchar(8000) COLLATE utf8_unicode_ci NOT NULL,
`conn_id` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
To use the script, run
/path/to/mysql-proxy --proxy-lua-script=/path/to/mysql-proxy-log-error-queries.lua
or if that fails (>=v0.9)
/path/to/mysql-proxy --proxy-lua-script=/path/to/mysql-proxy-log-error-queries.lua --plugins=proxy
The proxy runs on port 4040 by default, to test:
mysql -u username -p --host=127.0.0.1 --port=4040
and run some faulty sql.
When everything seems to be in order set the port in your application to 4040 instead of the actual mysqld port and you have mysql error logging on the database level.
Final note: mysql-proxy is beta. Use with caution I guess. Been running here for almost half a year now without problems, however YMMV.
I have tried running an error query on
purpose, but it logs as a normal query
and doesn't report it with error. Any
ideas?
so, you did it wrong. No other idea without the code.
in PHP I'm doing it this way (assume you're using mysql driver):
$res=mysql_query($sql) or trigger_error(mysql_error().$sql);
it will log all erroneous queries if you have log_errors setting on (and you have to)
EDIT:
I see now, you want global level logging,m not application level.
But may be application level will suit you as well?
MariaDB is capable to do this with a plugin https://mariadb.com/kb/en/sql-error-log-plugin/ which is distributed with mariadb.
I tried this yesterday and it works as advertised.
One need to just run:
install plugin SQL_ERROR_LOG soname 'sql_errlog';
and the queries with errors will go to $datadir/sql_errors.log that is on most linux install at /var/lib/mysql/sql_errors.log.