I am posting this thread in order to have some advices regarding the performance of my SQL query.
I have actually 2 tables, one which called HGVS_SNP with about 44657169 rows and another on run table which has an average of 2000 rows.
When I try to update field Comment of my run table it takes lot's of time to perform the query. I was wondering if there is any method to increase my SQL query.
Structure of HGVS_SNP Table:
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| snp_id | int(11) | YES | MUL | NULL | |
| hgvs_name | text | YES | | NULL | |
| source | varchar(8) | NO | | NULL | |
| upd_time | varchar(32) | NO | | NULL | |
+-----------+-------------+------+-----+---------+-------+
My run table has the following structure:
+----------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+--------------+------+-----+---------+-------+
| ID | varchar(7) | YES | | NULL | |
| Reference | varchar(7) | YES | MUL | NULL | |
| HGVSvar2 | varchar(120) | YES | MUL | NULL | |
| Comment | varchar(120) | YES | | NULL | |
| Compute | varchar(20) | YES | | NULL | |
+----------------------+--------------+------+-----+---------+-------+
Here's my query:
UPDATE run
INNER JOIN SNP_HGVS
ON run.HGVSvar2=SNP_HGVS.hgvs_name
SET run.Comment=concat('rs',SNP_HGVS.snp_id) WHERE run.Compute not like 'tron'
I`m guessing since you JOIN a text column with a VARCHAR(120) column that you don`t really need a text column. Make it a VARCHAR so you can index it
ALTER TABLE `HGVS_SNP` modify hgvs_name VARCHAR(120);
ALTER TABLE `HGVS_SNP` ADD KEY idx_hgvs_name (hgvs_name);
This will take a while on large tables
Now your JOIN should be much faster,also add an index on compute column
ALTER TABLE `run` ADD KEY idx_compute (compute);
And the LIKE is unnecessary,change it to
WHERE run.Compute != 'tron'
Related
When trying to describe a table I get a table with missing information (see first table); what would be the best way to update that table in a way that looks like the second one? My SQL background is not so strong, so I'd love to hear ideas on how to do this
First table (how it is actually)
+-----------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+-------+
| config_id | int(10) unsigned | NO | | 0 | |
| scope | varchar(8) | NO | | default | |
| scope_id | int(11) | NO | | 0 | |
| path | varchar(255) | NO | | general | |
| value | text | YES | | NULL | |
+-----------+------------------+------+-----+---------+-------+
Second table (how it should be)
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| config_id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| scope | varchar(8) | NO | MUL | default | |
| scope_id | int(11) | NO | | 0 | |
| path | varchar(255) | NO | | general | |
| value | text | YES | | NULL | |
+-----------+------------------+------+-----+---------+----------------+
I see following differences between the two tables :
column config_id should be auto_increment and also primary key of the table
a (non-unique) index is missing on column scope
The following statement should change the table as required :
ALTER TABLE mytable
MODIFY COLUMN config_id INT auto_increment,
ADD PRIMARY KEY (config_id),
ADD INDEX idx_scope(scope)
;
PS : DEFAULT NULL does not make sense for config_id : since it is a primary key, your RDBMS will never allow it to be set to NULL.
Please note that this answer is based on the information you provided only. Running this statement will not necessarily make the table structures strictly equivalent, since there could be other differences that can not be seen in the representation that you provided. You can get a complete DDL statement describing the table using the SHOW CREATE TABLE mytable syntax.
I have a doubt with the following query.
update src_woz_waardeklasse_2011 a use index(waardeklasse_main,woz_val)
inner join waardeklasse_average b use index (waardeklasse)
on b.waardeklasse_new = a.waardeklasse
set a.woz_value = b.average
where b.waardeklasse_new = a.waardeklasse;
I am trying to update a new column 'woz_val' in the 'src_woz_waardeklasse_2011' table using the 'average' values from the 'waardeklasse_average' table. I am joining using the 'waaderklasse' numbers in both the tables. But the 'src_woz_waardeklasse_2011' table is nearly 7 million records and the 'waardeklasee_average' table is 46 records. So the query is taking a really long time. 25 minutes and counting.
Is there a way to optimize it? I am sure it's taking a long time as I am trying to compare values between a large table and a small table. I have included the table structure of both the tables below.
src_woz_waardeklasse_2011
+----------------------+---------------------------+------+-----+---------+-
---------------+
| Field | Type | Null | Key | Default |
Extra |
+----------------------+---------------------------+------+-----+---------+-
---------------+
| id | int(11) unsigned | NO | PRI | NULL |
auto_increment |
| postcode | varchar(150) | YES | MUL | NULL |
|
| huisnummeraanduiding | varchar(150) | YES | | NULL |
|
| huisletter | varchar(150) | YES | | NULL |
|
soort_woonobject | varchar(150) | YES | | NULL |
|
| bouwjaar | varchar(150) | YES | | NULL |
|
| bouwjaarsklasse | varchar(150) | YES | | NULL |
|
| inhoud | varchar(150) | YES | | NULL |
|
| reg_oppervlak | varchar(150) | YES | | NULL |
|
| woz_value | int(15) unsigned zerofill | YES | UNI | NULL |
|
| reg_oppervlak_bn | varchar(150) | YES | | NULL |
|
| waardeklasse | int(10) | NO | PRI | NULL |
|
| waardepeildatum | varchar(150) | YES | | NULL |
|
| zipandnumber | varchar(150) | YES | | NULL |
|
+----------------------+---------------------------+------+-----+---------+-
---------------+
waardeklasse_average
+-------------------------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------------+---------+------+-----+---------+-------+
| waardeklasse_average_id | int(11) | NO | | NULL | |
| waardeklasse_new | int(10) | NO | PRI | NULL | |
| lower | int(11) | NO | | NULL | |
| higher | int(11) | NO | | NULL | |
| average | int(11) | NO | PRI | NULL | |
+-------------------------+---------+------+-----+---------+-------+
update src_woz_waardeklasse_2011 a use index(waardeklasse_main,woz_val)
Left join waardeklasse_average b
on b.waardeklasse_new = a.waardeklasse
set a.woz_value = b.average
where b.waardeklasse_new = a.waardeklasse;
Use the above query - in this you can use the left join and where condition to speed up the query.
Also don't use the index for the second table which has 46 record this will affect the performance - if the table has less records then index will create more burden in this case full table scan without index is lot better. Also if you have a cluster index in the first table(7 million) remove them and run the query, because for each and every update you perform default table structure And page alignment will be updated in the clustered index. Hope this helps..!
First, you don't need the where clause. Second, remove the index hints:
update src_woz_waardeklasse_2011 ww inner join
waardeklasse_average wa
on wa.waardeklasse_new = ww.waardeklasse
set ww.woz_value = wb.average ;
Then, try an index on src_woz_waardeklasse_2011(waardeklasse). This should improve the execution plan.
You might also check that the 46 records in the second table actually correspond to 46 updates. If the join conditions are wrong, you could be updating all the records.
EDIT:
Indexes have nothing to do with your problem. Updating 7 million records takes a long time. You might consider creating the data you want in a temporary table and then truncating the original table and inserting the new rows into it. Otherwise, batch the updates a few rows at a time.
update src_woz_waardeklasse_2011 a use index(waardeklasse_main,woz_val)
inner join waardeklasse_average b use index (waardeklasse)
on b.waardeklasse_new = a.waardeklasse
set a.woz_value = b.average
Where condition is not needed. As you had already mention that condition in On clause.
Try above query.
Here's my table.
+-------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| Postcode | varchar(255) | YES | | NULL | |
| Town | varchar(255) | YES | | NULL | |
| Region | varchar(255) | YES | | NULL | |
| Company Name | varchar(255) | YES | | NULL | |
| Fee | double | YES | | NULL | |
| Company Benefits | varchar(255) | YES | | NULL | |
| Date Updated | date | YES | | NULL | |
| Website | mediumtext | YES | | NULL | |
| Updated By | varchar(255) | YES | | NULL | |
| Notes | varchar(255) | YES | | NULL | |
| LNG | varchar(255) | YES | | NULL | |
| LAT | varchar(255) | YES | | NULL | |
+-------------------+--------------+------+-----+---------+----------------+
You can see we have an "Updated by" column.
How can I make it so that, when a user updates the row, the "Updated By" column automatically updates (or inserts if it's a new row they're adding) with the currently logged-in users name?
Many Thanks
You will have to explicitly make sure about that and whenever an UPDATE is happening then you need to update that column as well saying below. Best way to assure it, have your application logic fill in the column whenever an UPDATE to the record is happening from current logged-in user principle or claim
update tbl1
set ...,
Updated By = <logged in user name>
where Id = <some val>
You can use USER() or CURRENT_USER() in Update or Insert statements to achieve needed effect.
From my side - the only one secure way is to create stored procedures, providing inserts or updates to desired table.
Indeed, this problem was discussed here:
mysql Set default value of a column as the current logged in user
Something like this !
CREATE TRIGGER `updater`.`tableName_BEFORE_INSERT` BEFORE INSERT ON `tableName`
FOR EACH ROW
BEGIN
Set New.Updated_By = current_user();
END
I have a webapp that I'm building. This webapp will take as input some products (cars, motos, boats, houses, etc...) and each product will have one or more photos associated with it. The id of each of photo is generated by the uniqid() function of php.
My problem is:I can't seem to fit more than two id_photos into the same column
+-----------+------------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------------------------------+------+-----+---------+----------------+
| carid | int(11) | NO | MUL | NULL | auto_increment |
| brand | enum('Alfa Romeo','Aston Martin','Audi') | NO | | NULL | |
| color | varchar(20) | NO | | NULL | |
| type | enum('gasoline','diesel','eletric') | YES | | NULL | |
| price | mediumint(8) unsigned | YES | | NULL | |
| mileage | mediumint(8) unsigned | YES | | NULL | |
| model | text | YES | | NULL | |
| year | year(4) | YES | | NULL | |
| id_photos | varchar(30) | YES | | NULL | |
+-----------+------------------------------------------+------+-----+---------+----------------+
What I would like to happen is something like this: INSERT INTO cars(id_photos) values ('id_1st_photo', 'id_2nd_photo')
Ending up having something like this:
| 60 | Audi | Yellow | diesel | 252352 | 1234112 | R8 | 1990 | id_1st_photo id_2nd_photo |
Eventually I would have to grab those photos from the folders they are in which is something like this: /var/www/website/$login/photos/id_of_photo with the query select id_photos from cars where carid=$id.
You may found some data types that is not proprelly good for the data that the server will receive but I'm one week into mysql and I'll worry about data types later on.
First of all I don't know if that is possible, if it's not how can I design something to work like that?
I have found this question that is quite the same of mine but I can't seem to implement something like this: add multiple values in one column
You can insert the concatenated values into a field. But it is not a good practice. You can create another table with foreign key having the id of the parent table.
You can easily adapt the approach in the linked question and even remove one table needed:
You first table stays almost the same, but has the id_photos column removed:
+-----------+------------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------------------------------+------+-----+---------+----------------+
| carid | int(11) | NO | MUL | NULL | auto_increment |
| brand | enum('Alfa Romeo','Aston Martin','Audi') | NO | | NULL | |
| color | varchar(20) | NO | | NULL | |
| type | enum('gasoline','diesel','eletric') | YES | | NULL | |
| price | mediumint(8) unsigned | YES | | NULL | |
| mileage | mediumint(8) unsigned | YES | | NULL | |
| model | text | YES | | NULL | |
| year | year(4) | YES | | NULL | |
+-----------+------------------------------------------+------+-----+---------+----------------+
Then you'll add a second table to store the links to the photo ids:
+-----------+------------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------------------------------+------+-----+---------+----------------+
| carid | int(11) | NO | MUL | NULL | |
| id_photos | varchar(30) | NO | | NULL | |
+-----------+------------------------------------------+------+-----+---------+----------------+
Both tables are linked by the field carid (You should even make carid in the second table a foreign key pointing to the one in the first table).
Each id_photos then results in a new row in the second table.
To query the data you probably need a JOIN between both tables and maybe a GROUP BY to reduce the result to one row per carid again, but this depends on the other usecases.
You can insert the string formatted woth multiple photo name
INSERT INTO cars(id_photos) values ('id_1st_photo, id_2nd_photo')
In this way you don'have a well normalized database structure so you have problem when retrive the singole foto name ..
i suggest you of normalize the id_photo column in a separata table with reference to the master table and in this way store each single photo in one row
I am getting this error
javax.servlet.ServletException: com.mysql.jdbc.NotUpdatable: Result
Set not updatable.
I know this error is regarding the primary key but for all my tables I initially insert a primary key.So for this table also I have a primary key.I am posting part of my code.
Statement st=con.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_UPDATABLE);
ResultSet rs=st.executeQuery("Select * from test3 order by rand() limit 5");
List arrlist = new ArrayList();
while(rs.next()){
String xa =rs.getString("display");
if(xa.equals("1")){
arrlist.add(rs.getString("question_text"));
}
rs.updateString("display", "0");
rs.updateRow();
Just tell me if something is going wrong in this code.please help.
This is my database
+----------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+---------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| index_question | varchar(45) | YES | | NULL | |
| question_no | varchar(10) | YES | | NULL | |
| question_text | varchar(1000) | YES | | NULL | |
| file_name | varchar(128) | YES | | NULL | |
| attachment | mediumblob | YES | | NULL | |
| display | varchar(10) | YES | | NULL | |
+----------------+---------------+------+-----+---------+----------------+
You have to update the row immediately after you have fetched it (FOR UPDATE and rs.updateRow(),
OR
you have to write an UPDATE tablename set = where statement to update a row at any time
The query can not use functions. Try removing the "rand()" from the SQL query string.
See the JDBC 2.1 API Specification, section 5.6 for more details.