Why mysql program hangs(deadlock)? - mysql

I am struggling more than one day on dealing with a mysql hangs(deadlock). In below testcase, I will try to create the database first if it doesn't exist and try to create a table if it doesn't exist too. Then I do a query on the table. Each time I execute the SQL command, I strictly close the cursor. But the program still hangs. I have found two workarounds. 1) close the connection after creating the database and create a new connection. 2) call commit() after the query.
The two workarounds works good but they make me more confused. As my understanding, it's ok to keep connection if the cursors are closed in time and commit() are called after each change. And also, there is no reason to call commit() after query.
So my two workarounds even destroyed my understanding of database operation. I do need some help to point out what's wrong with the program basically.... Just give me some light...
Thanks very much!
#!/usr/bin/python2
import MySQLdb
def NewConnectToMySQL():
conn = MySQLdb.Connect("localhost", "root", "mypassword")
return conn
def CreateDBIfNotExists(conn):
sql = "CREATE DATABASE IF NOT EXISTS testdb"
cur = conn.cursor()
cur.execute(sql)
cur.close()
conn.select_db("testdb")
conn.commit()
"""workaround-1"""
#conn.close()
#conn = NewConnectToMySQL()
#conn.select_db("testdb")
return conn
def CreateTableIfNotExists(conn):
sql = "CREATE TABLE IF NOT EXISTS mytable (id INTEGER, name TEXT)"
cur = conn.cursor()
cur.execute(sql)
cur.close()
conn.commit()
def QueryName(conn, name):
sql = "SELECT * FROM mytable WHERE name = '%s'" % name
cur = conn.cursor()
cur.execute(sql)
info = cur.fetchall()
cur.close()
"""workaround-2"""
#conn.commit()
return info
conn1 = NewConnectToMySQL()
CreateDBIfNotExists(conn1)
CreateTableIfNotExists(conn1)
QueryName(conn1, "tom")
conn2 = NewConnectToMySQL()
CreateDBIfNotExists(conn2)
CreateTableIfNotExists(conn2) #hangs here!!!!!!!!!!
Here is the output of SHOW FULL PROCESSLIST when hangs.
mysql> SHOW FULL PROCESSLIST
-> ;
+-----+------+-----------+--------+---------+------+---------------------------------+------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------+--------+---------+------+---------------------------------+------------------------------------------------------------+
| 720 | root | localhost | testdb | Sleep | 96 | | NULL |
| 721 | root | localhost | testdb | Query | 96 | Waiting for table metadata lock | CREATE TABLE IF NOT EXISTS mytable (id INTEGER, name TEXT) |
| 727 | root | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST |
+-----+------+-----------+--------+---------+------+---------------------------------+------------------------------------------------------------+
3 rows in set (0.00 sec)

Related

Bulk insert operation are going too slow

Here is the Structure of my 'venprices' table.
+-------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| conid | int(10) | NO | PRI | NULL | |
| rate | double | YES | | NULL | |
| venid | varchar(50) | NO | PRI | | |
+-------+-------------+------+-----+---------+-------+
JAVA Code :
new Thread(){
public void run(){
XSSFWorkbook myWorkBook;
XSSFSheet mySheet = null;
Iterator<Row> rowIterator = null;
String venid = null, errorlog = null;
int code;
double rate;
int rows, maxcount;
PreparedStatement ps = null;
Connection con;
ProgressMonitor pm;
try {
myWorkBook = new XSSFWorkbook(new FileInputStream(new File(jTextField1.getText())));
mySheet = myWorkBook.getSheetAt(0);
rowIterator = mySheet.iterator();
rowIterator.next();
venid = jComboBox1.getItemAt(jComboBox1.getSelectedIndex());
con = Mycon.getConnection();
ps = con.prepareStatement("DELETE FROM venprices WHERE venid = ?");
ps.setString(1, venid);
ps.executeUpdate();
ps.clearBatch();
ps = con.prepareStatement("insert into venprices values (?,?,?)");
} catch(Exception ioe) {
JOptionPane.showMessageDialog(null, ioe.getMessage());
}
rows = 1;maxcount = mySheet.getLastRowNum();
// Traversing over each row of XLSX file
while (rowIterator.hasNext())
{
try{
Row row = rowIterator.next();
Iterator<Cell> cellIterator = row.cellIterator();
Cell cell = cellIterator.next();
code = (int) cell.getNumericCellValue();
cell = cellIterator.next();
rate = cell.getNumericCellValue();
ps.setInt(1,code);
ps.setDouble(2,rate);
ps.setString(3, venid);
ps.addBatch();
rows++;
}catch(Exception e){errorlog = errorlog + "\n" +rows+ e.getMessage();}
}
try{
System.gc();
ps.executeBatch();
}catch(Exception e){e.printStackTrace();}
if(errorlog == null)
JOptionPane.showMessageDialog(null, "Import Successful. " + rows + " Records Imported.");
else
JOptionPane.showMessageDialog(null, "Error Log :\n"+errorlog);
}
}.start();
The user is expected to insert around 50,000 records in a single shot with an Excel File. But the query takes around 6-7 minutes.
Can anyone please help me in reducing the insert operation time or tell me some tweaks in the insert query?
Thanks in Advance!
Edit 1:
As Requested, Here is the result of show create table venprices
mysql> show create table venprices;
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| venprices | CREATE TABLE `venprices` (
`conid` int(10) NOT NULL,
`rate` double DEFAULT NULL,
`venid` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`conid`,`venid`),
KEY `vepr` (`conid`,`rate`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I don't know the Java syntax, but add a MySQL START TRANSACTION (BEGIN) at the beginning of your code. And add COMMIT to the end.
Why?
50K rows inserted in 6-7 minutes smells very much like inserting each row in its own transaction (a la autocommit=ON). This is slow because of all the transactional activity on the disk.
By turning the 50K transactions into 1 transaction, you will have a lot less I/O, hence it will run faster.
Secondly... By turning the 50K 1-row INSERTs into 50 1000-row INSERTs, you can get another 10x speedup. This is because of avoiding most of the roundtrip time between client and server and most of the parsing time. (Again, I don't know if Java has a special class to facilitate such; it might.)
These two changes will perhaps be competitive with Drew's LOAD_DATA INFILE approach.

Perl and MySQL user input to select a row?

Suppose I have a simple database table that doesn't have an ID_KEY but has a name column. I want to display the output like this
+----+---------+
| | name |
+----+---------+
| 1 | dog |
| 2 | cat |
| 3 | penguin |
| 4 | lax |
| 5 | whale |
| 6 | ostrich |
+----+---------+
Then have a <STDIN> for like, say, 3 to select penguin. 3 is just the line number that appears when you do the select call.
Is there any way to do this, or is it only possible with an id key associated and then a subsequent select statement matching that id key?
I misunderstood you at first but I've caught on. But it doesn't make much sense, as when you're entering a number into a Perl program you won't be working with the MySQL command-line tool, and won't be able to see what numbers to enter..
What you need to do is to write your Perl program so that it prints all the name fields from the table together with a line number. Then your program can translate from an input animal number to its name because it knows what it printed.
Something like this would work. Of course you will have to set the name, IP address and credentials correctly so that DBI can connect to the database.
use strict;
use warnings;
use DBI;
my $dbh = DBI->connect(
'DBI:mysql:database=animal_test',
'username',
'password',
{ RaiseError => 1 },
);
my $names = map #$_, $dbh->selectall_arrayref('SELECT name FROM animals');
for my $i ( 0 .. $#$names ) {
printf "%3d: %s\n", $i+1, $names->[$i];
}
print "\n";
print "Enter animal number: ";
my $animal = <>;
chomp $animal;
my $name = $names->[$animal-1];
printf "Animal chosen is %s\n", $name;
Option 1 - You would have put a id field in the DB if you want to find by integer 3 because row 3 will not always be penguin from an SQL query.
Option 2 - Dump the data into and array or hash and use the index of that to find the item from with in the variable and not the DB when 3 is captured from STIN.
Just use query:
my $select = $dbh->prepare('
SET #id:=0;
SELECT name,
#id = #id+1
FROM table
');

sqlalchemy FetchedValue and primary_key

I'm trying to create a table that uses a UUID_SHORT() as a primary key. I have a trigger that inserts a value when you do an insert. I'm having trouble making sqlalchemy recognize a column as a primary_key without complaining about not providing a default. If I do include a default value, it will use that default value even after flush despite declaring server_default=FetchedValue(). The only way I can seem to get things to work properly is if the column is not a primary key.
I'm using Pyramid, SQLAlchemy ORM, and MySQL.
Here's the model object:
Base = declarative_base()
class Patient(Base):
__tablename__ = 'patient'
patient_id = Column(BigInteger(unsigned=True), server_default=FetchedValue(), primary_key=True, autoincrement=False)
details = Column(Binary(10000))
in initializedb.py I have:
with transaction.manager:
patient1 = Patient(details = None)
DBSession.add(patient1)
DBSession.flush()
print(patient1.patient_id)
running ../bin/initialize_mainserver_db development.ini gives me the following error:
2012-11-01 20:17:22,168 INFO [sqlalchemy.engine.base.Engine][MainThread] BEGIN (implicit)
2012-11-01 20:17:22,169 INFO [sqlalchemy.engine.base.Engine][MainThread] INSERT INTO patient (details) VALUES (%(details)s)
2012-11-01 20:17:22,169 INFO [sqlalchemy.engine.base.Engine][MainThread] {'details': None}
2012-11-01 20:17:22,170 INFO [sqlalchemy.engine.base.Engine][MainThread] ROLLBACK
Traceback (most recent call last):
File "/sites/metrics_dev/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
context)
File "/sites/metrics_dev/lib/python3.3/site-packages/sqlalchemy/engine/default.py", line 333, in do_execute
cursor.execute(statement, parameters)
File "/sites/metrics_dev/lib/python3.3/site-packages/mysql/connector/cursor.py", line 418, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/sites/metrics_dev/lib/python3.3/site-packages/mysql/connector/cursor.py", line 345, in _handle_result
self._handle_noresultset(result)
File "/sites/metrics_dev/lib/python3.3/site-packages/mysql/connector/cursor.py", line 321, in _handle_noresultset
self._warnings = self._fetch_warnings()
File "/sites/metrics_dev/lib/python3.3/site-packages/mysql/connector/cursor.py", line 608, in _fetch_warnings
raise errors.get_mysql_exception(res[0][1],res[0][2])
mysql.connector.errors.DatabaseError: 1364: Field 'patient_id' doesn't have a default value
Running a manual insert using the mysql client results in the everything working fine, so the problem seems to be with SQLAlchemy.
mysql> insert into patient(details) values (null);
Query OK, 1 row affected, 1 warning (0.00 sec)
mysql> select * from patient;
+-------------------+---------+
| patient_id | details |
+-------------------+---------+
| 94732327996882980 | NULL |
+-------------------+---------+
1 row in set (0.00 sec)
mysql> show triggers;
+-----------------------+--------+---------+-------------------------------------+--------+---------+----------+----------------+----------------------+----------------------+--------------------+
| Trigger | Event | Table | Statement | Timing | Created | sql_mode | Definer | character_set_client | collation_connection | Database Collation |
+-----------------------+--------+---------+-------------------------------------+--------+---------+----------+----------------+----------------------+----------------------+--------------------+
| before_insert_patient | INSERT | patient | SET new.`patient_id` = UUID_SHORT() | BEFORE | NULL | | root#localhost | utf8 | utf8_general_ci | latin1_swedish_ci |
+-----------------------+--------+---------+-------------------------------------+--------+---------+----------+----------------+----------------------+----------------------+--------------------+
1 row in set (0.00 sec)
Here's what I did as a work-around...
DBSession.execute(
"""CREATE TRIGGER before_insert_patient BEFORE INSERT ON `patient`
FOR EACH ROW BEGIN
IF (NEW.patient_id IS NULL OR NEW.patient_id = 0) THEN
SET NEW.patient_id = UUID_SHORT();
END IF;
END""")
and in the patient class:
patient_id = Column(BigInteger(unsigned=True), default=text("uuid_short()"), primary_key=True, autoincrement=False, server_default="0")
So, the trigger only does something if someone accesses the database directly and not through the python code. And hopefully no one does patient1 = Patient(patient_id=0, details = None) as SQLAlchemy will use the '0' value instead of what the trigger produces
For completeness, here are two additional possible solutions for your question (also available here), based on your answer. They are slightly simpler than your solution (omitting passing parameters with correct default values) and using SQLAlchemy constructs for defining the triggers.
#!/usr/bin/env python3
from sqlalchemy import BigInteger, Column, create_engine, DDL, event
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.schema import FetchedValue
from sqlalchemy.sql.expression import func
Base = declarative_base()
class PatientOutputMixin(object):
'''
Mixin to output human readable representations of models.
'''
def __str__(self):
return '{}'.format(self.patient_id)
def __repr__(self):
return str(self)
class Patient1(Base, PatientOutputMixin):
'''
First version of ``Patient`` model.
'''
__tablename__ = 'patient_1'
patient_id = Column(BigInteger, primary_key=True,
default=func.uuid_short())
# the following trigger is only required if columns are inserted in the table
# not using the above model/table definition, otherwise it is redundant
create_before_insert_trigger = DDL('''
CREATE TRIGGER before_insert_%(table)s BEFORE INSERT ON %(table)s
FOR EACH ROW BEGIN
IF NEW.patient_id IS NULL THEN
SET NEW.patient_id = UUID_SHORT();
END IF;
END
''')
event.listen(Patient1.__table__, 'after_create',
create_before_insert_trigger.execute_if(dialect='mysql'))
# end of optional trigger definition
class Patient2(Base, PatientOutputMixin):
'''
Second version of ``Patient`` model.
'''
__tablename__ = 'patient_2'
patient_id = Column(BigInteger, primary_key=True,
default=0, server_default=FetchedValue())
create_before_insert_trigger = DDL('''
CREATE TRIGGER before_insert_%(table)s BEFORE INSERT ON %(table)s
FOR EACH ROW BEGIN
SET NEW.patient_id = UUID_SHORT();
END
''')
event.listen(Patient2.__table__, 'after_create',
create_before_insert_trigger.execute_if(dialect='mysql'))
# test models
engine = create_engine('mysql+oursql://test:test#localhost/test?charset=utf8')
Base.metadata.bind = engine
Base.metadata.drop_all()
Base.metadata.create_all()
Session = sessionmaker(bind=engine)
session = Session()
for patient_model in [Patient1, Patient2]:
session.add(patient_model())
session.add(patient_model())
session.commit()
print('{} instances: {}'.format(patient_model.__name__,
session.query(patient_model).all()))
Running the above script produces the following (sample) output:
Patient1 instances: [22681783426351145, 22681783426351146]
Patient2 instances: [22681783426351147, 22681783426351148]

SQL query to remove certain text from each field in a specific column?

I recently recoded one of my sites, and the database structure is a little bit different.
I'm trying to convert the following:
*----*----------------------------*
| id | file_name |
*----*----------------------------*
| 1 | 1288044935741310953434.jpg |
*----*----------------------------*
| 2 | 1288044935741310352357.rar |
*----*----------------------------*
Into the following:
*----*----------------------------*
| id | file_name |
*----*----------------------------*
| 1 | 1288044935741310953434 |
*----*----------------------------*
| 2 | 1288044935741310352357 |
*----*----------------------------*
I know that I could do a foreach loop with PHP, and explode the file extension off the end, and update each row that way, but that seems like way too many queries for the task.
Is there any SQL query that I could run that would allow me to remove the file exentision from each field in the file_name column?
You can use the REPLACE() function in native MySQL to do a simple string replacement.
UPDATE tbl SET file_name = REPLACE(file_name, '.jpg', '');
UPDATE tbl SET file_name = REPLACE(file_name, '.rar', '');
This should work:
UPDATE MyTable
SET file_name = SUBSTRING(file_name,1, CHAR_LENGTH(file_name)-4)
This will strip off the final extension, if any, from file_name each time it is run. It is agnostic with respect to extension (so you can have ".foo" some day) and won't harm extensionless records.
UPDATE tbl
SET file_name = TRIM(TRAILING CONCAT('.', SUBSTRING_INDEX(file_name, '.', -1) FROM file_name);
You can use SUBSTRING_INDEX function
SUBSTRING_INDEX(str,delim,count)
Where str is the string, delim is the delimiter (from which you want a substring to the left or right of), and count specifies which delimiter (in the event there are multiple occurrences of the delimiter in the string)
Example:
UPDATE table SET file_name = SUBSTRING_INDEX(file_name , '.' , 1);

JavaBean SQL query to MySQL db, shows twice the results only in JSF

I'm having a weird problem when querying a MySQL database from a Javabean and showing it in JSF. (Java code at the bottom)
The table contains the following registers:
mysql> select * from trescols;
+----+---------+---------+
| id | camp1 | camp2 |
+----+---------+---------+
| 1 | fila1A | fila1B |
| 2 | fila2A | fila2B |
| 3 | fila3A | fila3B |
| 4 | fila4A | fila4B |
...
| 20 | fila20A | fila20B |
+----+---------+---------+
Ok! Let's go with the process done until it failed:
First of all I queryed the db with a large query, and it didn't work, so I began with a simple query and added stuff step-by-step.
First of all, I queryed the db with the following query:
SELECT * FROM trescols;
Everything OK, it was showed well in the resultpage with JSF.
Next I queryed the following:
SELECT * FROM tabla WHERE id%2=1;
Everything OK, it just showed the records with odd id.
The problem came when I queryed:
SELECT * FROM tabla WHERE id%2=1 AND campo1 LIKE '%7%';
I expected it to show registers with odd id and containing the string '7' somewhere in "camp1" column.
The result with JSF has been:
ID Camp1 Camp2
7 fila7A fila7B
17 fila17A fila17B
7 fila7A fila7B
17 fila17A fila17B
While the same query from mysql cli returns:
mysql> select * from trescols where id%2=1 and camp1 LIKE '%7%';
+----+---------+---------+
| id | camp1 | camp2 |
+----+---------+---------+
| 7 | fila7A | fila7B |
| 17 | fila17A | fila17B |
+----+---------+---------+
2 rows in set (0.10 sec)
I added LIMIT 1, 1 to the query string and removed the id%2=1 AND, and returns:
ID Camp1 Camp2
17 fila17A fila17B
17 fila17A fila17B
The expected result was just to show once the second register, not twice. Removing the first condition was just to verify that having double condition wasn't the reason for the duplicated results and exclude a wrong SQL query.
Then I made a stored procedure as follows:
mysql> CREATE PROCEDURE getTrescols3()
-> BEGIN
-> SELECT * FROM trescols WHERE camp1 LIKE '%7%';
-> END
-> //
Calling this from MySQL CLI works fine:
mysql> call gettrescols3();
+----+---------+---------+
| id | camp1 | camp2 |
+----+---------+---------+
| 7 | fila7A | fila7B |
| 17 | fila17A | fila17B |
+----+---------+---------+
2 rows in set (0.00 sec)
Query OK, 0 rows affected (0.01 sec)
But calling it from JSF shows it twice again!!!
ID Camp1 Camp2
7 fila7A fila7B
17 fila17A fila17B
7 fila7A fila7B
17 fila17A fila17B
And this is where I'm lost... I could understand that the query could -by some reason altough it shouldn't- show duplicated results, then I started by a simple query adding sentences to the string until it showed duplicated results. With just one condition it worked fine, with two it showed duplicated results. Then I removed the first of the two conditions with which it didn't had any problem and left just the second condition, it showed duplicated results. Then I though that maybe java had any trouble with the condition <field> LIKE '%7%', so I created a stored procedure in MySQL so Java shouldn't process the querystring and just call the SP and get the result, called it from CLI and worked fine, called it from java bean, and showed duplicated results.
I don't know what more tests to do...I discarded a wrong SQL query because in CLI everything worked as expected, and it just failed when adding the condition <field> LIKE '%7%' and just failed in JSF not with mysql cli, also I ruled out wrong query again because calling a SP from java bean and from cli, it worked in cli but not from java bean...
I was trying to get all results from table "trescols" where id%2=1 and where the field "camp1" contains a 7.
I'm using mysql jdbc connector to connect to db.
Could you help me, please.
Kind regards,
Carles
Java code:
package beans;
import java.sql.*;
import java.util.*;
public class beanNomesRetorna {
Connection con;
Statement ps;
ResultSet rs;
private List llista = new ArrayList();
public List getLlista() {
int i=0;
try {
Class.forName("com.mysql.jdbc.Driver");
con = DriverManager.getConnection("jdbc:mysql://localhost:3306/provesjsf","root","pa$$word");
ps = con.createStatement();
rs = ps.executeQuery("SELECT * FROM trescols WHERE id%2 = 1 AND camp1 LIKE '%7%' ORDER BY id LIMIT 0,2");
while(rs.next()) {
System.out.println(rs.getString(1));
llista.add(i, new agafaInfo(Integer.parseInt(rs.getString("id")), rs.getString("camp1"), rs.getString("camp2")));
i++;
} // while
} catch(Exception e) {
System.out.println(e);
} // try ... catch
return llista;
} // getLlista
package beans;
public class agafaInfo {
int id;
String camp1;
String camp2;
agafaInfo(int parid, String parcamp1, String parcamp2) {
this.id = parid;
this.camp1 = parcamp1;
this.camp2 = parcamp2;
}
public int getParid() {
return id;
}
public String getCamp1() {
return camp1;
}
public String getCamp2() {
return camp2;
}
} // agafaInfo
Don't do the DB operations in the getter of a bean. Do it in the constructor of a bean. The symptoms indicate that you've a List property and are filling it inside the getter instead of the constructor. A getter can be called multiple times during bean's life. Its sole task should be returning the data, not doing some business stuff.
Update as I predicted, you're doing business stuff inside the getter. That's not how it works. Move that code into the constructor.
public class Bean {
public List list;
public Bean() {
// Fill the list here.
}
public List getList() {
return list; // Do nothing else in the getter!
}
}
See also:
Why JSF calls getters multiple times
Unrelated to the concrete problem, you've got serious problems in your JDBC code approach as well. The resources are declared as class fields instead of method local fields and they are not closed properly in the finally block. It is sensitive to resource leaks. I'd suggest to get yourself through this article to learn by example how to do the JDBC stuff properly.