Bulk insert operation are going too slow - mysql

Question

Here is the Structure of my 'venprices' table.
+-------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| conid | int(10) | NO | PRI | NULL | |
| rate | double | YES | | NULL | |
| venid | varchar(50) | NO | PRI | | |
+-------+-------------+------+-----+---------+-------+
JAVA Code :
new Thread(){
public void run(){
XSSFWorkbook myWorkBook;
XSSFSheet mySheet = null;
Iterator<Row> rowIterator = null;
String venid = null, errorlog = null;
int code;
double rate;
int rows, maxcount;
PreparedStatement ps = null;
Connection con;
ProgressMonitor pm;
try {
myWorkBook = new XSSFWorkbook(new FileInputStream(new File(jTextField1.getText())));
mySheet = myWorkBook.getSheetAt(0);
rowIterator = mySheet.iterator();
rowIterator.next();
venid = jComboBox1.getItemAt(jComboBox1.getSelectedIndex());
con = Mycon.getConnection();
ps = con.prepareStatement("DELETE FROM venprices WHERE venid = ?");
ps.setString(1, venid);
ps.executeUpdate();
ps.clearBatch();
ps = con.prepareStatement("insert into venprices values (?,?,?)");
} catch(Exception ioe) {
JOptionPane.showMessageDialog(null, ioe.getMessage());
}
rows = 1;maxcount = mySheet.getLastRowNum();
// Traversing over each row of XLSX file
while (rowIterator.hasNext())
{
try{
Row row = rowIterator.next();
Iterator<Cell> cellIterator = row.cellIterator();
Cell cell = cellIterator.next();
code = (int) cell.getNumericCellValue();
cell = cellIterator.next();
rate = cell.getNumericCellValue();
ps.setInt(1,code);
ps.setDouble(2,rate);
ps.setString(3, venid);
ps.addBatch();
rows++;
}catch(Exception e){errorlog = errorlog + "\n" +rows+ e.getMessage();}
}
try{
System.gc();
ps.executeBatch();
}catch(Exception e){e.printStackTrace();}
if(errorlog == null)
JOptionPane.showMessageDialog(null, "Import Successful. " + rows + " Records Imported.");
else
JOptionPane.showMessageDialog(null, "Error Log :\n"+errorlog);
}
}.start();
The user is expected to insert around 50,000 records in a single shot with an Excel File. But the query takes around 6-7 minutes.
Can anyone please help me in reducing the insert operation time or tell me some tweaks in the insert query?
Thanks in Advance!
Edit 1:
As Requested, Here is the result of show create table venprices
mysql> show create table venprices;
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| venprices | CREATE TABLE `venprices` (
`conid` int(10) NOT NULL,
`rate` double DEFAULT NULL,
`venid` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`conid`,`venid`),
KEY `vepr` (`conid`,`rate`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Answer 1

I don't know the Java syntax, but add a MySQL START TRANSACTION (BEGIN) at the beginning of your code. And add COMMIT to the end.
Why?
50K rows inserted in 6-7 minutes smells very much like inserting each row in its own transaction (a la autocommit=ON). This is slow because of all the transactional activity on the disk.
By turning the 50K transactions into 1 transaction, you will have a lot less I/O, hence it will run faster.
Secondly... By turning the 50K 1-row INSERTs into 50 1000-row INSERTs, you can get another 10x speedup. This is because of avoiding most of the roundtrip time between client and server and most of the parsing time. (Again, I don't know if Java has a special class to facilitate such; it might.)
These two changes will perhaps be competitive with Drew's LOAD_DATA INFILE approach.