Is there way to realize this algorithm with mysql without 100500 queries and lots of resources?
if (exists %name% in table.name) {
num = 2;
while(exists %newname%+(num) in table.name) num++;
%name% = newname+(num);
}
Thanks
I don't know how much better you can do with a stored procedure in MySql, but you can definitely do better than 100500 queries:
SELECT name FROM table WHERE name LIKE 'somename%' ORDER BY name DESC LIMIT 1
At that point, you know that you can increment the number at the end of name and the result will be unused.
I 'm glossing over some fine print (this approach will never find and fill any "holes" in the naming scheme that may exist, and it's still not guaranteed that the name will be available due to race conditions), but in practice it can be made to work quite easily.
The simpliest way I can see of doing it is to create a table of sequential numbers
then cross join on to it....
SELECT a.name,b.id
FROM table a
WHERE a.name = 'somename'
CROSS JOIN atableofsequentialnumbers b
WHERE NOT EXISTS (SELECT 1 FROM table x WHERE x.name = CONCAT(a.name,b.id))
LIMIT 10
This will return the first 10 available numbers/names
Related
Now I have a Query that runs 50 minutes on Mysql database and I can't accept that...
I want this process can running under 15 minutes....
insert into appianData.IC_DeletedRecords
(sourcetableid,
concatkey,
sourcetablecode)
select mstid,
concatkey,
'icmstlocationheader' as sourcetablecode
from appianData.IC_MST_LocationHeader
where concatkey not in(select concatkey
from appianData.IC_PURGE_LocationHeader)
The "sourcetableid" and "mstid" are unique.
So what is the best way to add INDEX or optimize on this?
Thank you
I would write the select as:
select mstid, concatkey, 'icmstlocationheader' as sourcetablecode
from appianData.IC_MST_LocationHeader lh
where not exists (select 1
from appianData.IC_PURGE_LocationHeader lhp
where lhp.concatkey = lh.concatkey
);
Then you want an index on IC_PURGE_LocationHeader(concatkey).
Since it is a NOT IN condition you should be able to use a "LEFT JOIN ... WHERE rightTable has no match" without concern for multiple matches inflating the results.
INSERT INTO appianData.IC_DeletedRecords (sourcetableid, concatkey, sourcetablecode)
SELECT m.mstid, m.concatkey, 'icmstlocationheader' as sourcetablecode
FROM appianData.IC_MST_LocationHeader AS m
LEFT JOIN appianData.IC_PURGE_LocationHeader AS p
ON m.concatkey = p.concatkey
WHERE p.concatkey IS NULL
;
With this version query, or the one you presented in the question, indexes on concatkey in both source tables should help significantly.
The left table is a result of my query. And I need to sort it as the right table.
I need to order by p_id, if level >= 2. The blue box of right table is a target of order by.
Is it possible? Of course it is an example. Actual data is hundreds and really need to be sorted.
I searched a lot, but coudln't find the same case.
edit : this table will be returned as java.util.ArrayList. If this kind of 'order by' is not possilbe, is it possible in java.util.ArrayList?
I'm sure it's not possible in one query in MySQL.
In your diagram on the right, the ordering has been done in two separate steps:
Sort by id
Sort each block by p_id if level >= 2
That's quite difficult to do in MySQL as you would need to identify the blocks and then iterate over them, sorting each block separately.
I've done something similar where ordering within blocks was required and then selecting from those ordered blocks. You can view that here but as I said, I think that that SQL code is horribly complicated involving 5 temporary tables. You would probably need fewer temp tables, but it would still be a very complicated procedure, quite slow and hard to maintain.
"Actual data is hundreds and really need to be sorted."
Is there any reason why you can't just sort it as you want in code?
$blockStart = FALSE;
$count = 0;
foreach($dataArray as $data){
if($blockStart === FALSE){
$blockStart = $count;
}
if($data['level'] < 2){ //Block has finished
sortBlock($dataArray, $blockStart, $count);
$blockStart = $count;
}
$count++;
}
sortBlock($dataArray, $blockStart, $count - 1);
function sortBlock($dataArray, $indexStart, $indexEnd){
//Sort the elements of $dataArray, between $indexStart and $indexEnd inclusive
//by the value of p_id
}
Trying to solve a general programming problem in MySQL when you could solve it in 1/10th of the programmer time (and probably have it perform faster as well) in Java is not a good path to follow.
It is possible to do this in SQL, but it would be a very, very complicated query in MySQL. Here is the approach.
(1) Create a subquery that has the original ids and an indicator of whether something is in level 2 or not. The ids in this table are going to define the final order.
(2) Next, create a separate counter for each group in this above table. In other databases, you would use row_number(). In my SQL, this requires a correlated subquery. This provides the mapping from id to the new ordering.
(3) Next, create a counter for each group, but this time with the needed order (by id for the non-level2 group, by your rules for ordering).
(4) Join the tables together to get the matching.
(5) Order by the original id.
Here is an attempt:
select altord.*
from (select t.*,
(select count(*) from t t2 where t2.id <= t.id and ((t2.level = 2 and t1.level = 2) or (t2.level <> 2 and t1.level <> 2))
) as seqnum
from t
) ord join
(select t.*,
(select count(*) from t t2 where (t2.id <= t.id and t2.level <> 2 and t.level <> 2) or (t2.level = 2 and t.level = 2 and (t2.pid < t.pid or t2.pid = t.pid and t2.id < t.id)))
) as seqnum
) altord
on ord.seqnum = altord.seqnum
order by ord.id
I'm not sure if this SQL is correct, but the idea can be implemented in a single query.
I can find sequenced record gaps where sequenced weeks with same numbers using following query.
SELECT * FROM pointed_numbers A WHERE EXISTS (
SELECT * FROM pointed_numbers B WHERE A.number = B.number AND (A.week = B.week + 1 XOR A.week = B.week - 1)
) ORDER BY A.number, A.week;
How can I identify each gaps without stored procedure. I have tried with user-defined variable but I had no success.
Take a look at http://www.artfulsoftware.com/infotree/queries.php and look at the stuff under the "sequences" section. This is a super super helpful site with recipes for how to do complicated things in mysql!
I'm using this kind of queries with different parameters :
EXPLAIN SELECT SQL_NO_CACHE `ilan_genel`.`id` , `ilan_genel`.`durum` , `ilan_genel`.`kategori` , `ilan_genel`.`tip` , `ilan_genel`.`ozellik` , `ilan_genel`.`m2` , `ilan_genel`.`fiyat` , `ilan_genel`.`baslik` , `ilan_genel`.`ilce` , `ilan_genel`.`parabirimi` , `ilan_genel`.`tarih` , `kgsim_mahalleler`.`isim` AS mahalle, `kgsim_ilceler`.`isim` AS ilce, (
SELECT `ilanresimler`.`resimlink`
FROM `ilanresimler`
WHERE `ilanresimler`.`ilanid` = `ilan_genel`.`id`
LIMIT 1
) AS resim
FROM (
`ilan_genel`
)
LEFT JOIN `kgsim_ilceler` ON `kgsim_ilceler`.`id` = `ilan_genel`.`ilce`
LEFT JOIN `kgsim_mahalleler` ON `kgsim_mahalleler`.`id` = `ilan_genel`.`mahalle`
WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
ORDER BY `ilan_genel`.`id` DESC
LIMIT 225 , 15
and this is what i get in explain section:
these are the indexes that i already tried to use:
any help will be deeply appreciated what kind of index will be the best option or should i use another table structure ?
You should first simplify your query to understand your problem better. As it appears your problem is constrained to the ilan_gen1 table, the following query would also show you the same symptoms.:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
So the first thing to do is check that this is the case. If so, the simpler question is simply why does this query require a file sort on 3661 rows. Now the 'hepsi' index sort order is:
ilce->mahelle->durum->kategori->tip->ozelik
I've written it that way to emphasise that it is first sorted on 'ilce', then 'mahelle', then 'durum', etc. Note that your query does not specify the 'mahelle' value. So the best the index can do is lookup on 'ilce'. Now I don't know the heuristics of your data, but the next logical step in debugging this would be:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'`
Does this return 3661 rows?
If so, you should be able to see what is happening. The database is using the hepsi index, to the best of it's ability, getting 3661 rows back then sorting those rows in order to eliminate values according to the other criteria (i.e. 'durum', 'kategori', 'tip').
The key point here is that if data is sorted by A, B, C in that order and B is not specified, then the best logical thing that can be done is: first a look up on A then a filter on the remaining values against C. In this case, that filter is performed via a file sort.
Possible solutions
Supply 'mahelle' (B) in your query.
Add a new index on 'ilan_gene1' that doesn't require 'mahelle', i.e. A->C->D...
Another tip
In case I have misdiagnosed your problem (easy to do when I don't have your system to test against), the important thing here is the approach to solving the problem. In particular, how to break a complicated query into a simpler query that produces the same behaviour, until you get to a very simple SELECT statement that demonstrates the problem. At this point, the answer is usually much clearer.
I am importing data to a table structured: content_id|user_id|count - all integers all comprise the composite primary key
The table I want to select it from is structured: content_id|user_id
For reasons quite specific to my use case, I will need to fire quite a lot of data into this regularly enough to want a pure MySQL solution
insert into new_db.table
select content_id,user_id,xxx from old_db.table
I want each row to go in with xxx set to 0, unless this would create a duplicate key, in which case I wish to increment the number, for the current user_id/content_id combination
Not being a MySQL expert, I tried a few options like trying to populate xxx by selecting from the target table during insert, with no luck. Also tried using ON DUPLICATE KEY to increment counters instead of the usual UPDATE. But it all seemed a bit daft so I thought I would come here!
Any ideas anyone? I have a backup option of wrapping this in PHP, but it would drastically raise the overall running time of the script in which this would be the only non-pure MySQL part
Any help really appreciated. thanks in advance!
--edit
this may sound really awful in principle. but id settle for a way to do it in an update after entering random numbers (i have sent in random numbers to allow me to continue other work at the moment) - and this is a purely dev setup
--edit again
12|234
51|45
51|45
51|45
23|67
would ideally insert
12|234|0
51|45|0
51|45|1
51|45|2
23|67|0
INSERT INTO new_db.table (content_id, user_id, cnt)
SELECT old.content_id, old.user_id, COUNT(old.*) - 1 FROM old_db.table old
GROUP BY old.content_id, old.user_id
this would be the way I would go, so if 1 entry it would put 0 on cnt, for more it would just put 1-2-3 etc.
Edit:
Your correct answer would be somewhat complicated but I tested it and it works:
INSERT INTO newtable(user_id,content_id,cnt)
SELECT o1.user_id, o1.content_id,
CASE
WHEN COALESCE(#rownum, 0) = 0
THEN #rownum:=c-1
ELSE #rownum:=#rownum-1
END as cnt
FROM
(SELECT user_id, content_id, COUNT(*) as c FROM oldtable
GROUP BY user_id, content_id ) as grpd
LEFT JOIN
(SELECT oldtable.* FROM oldtable) o1 ON
(o1.user_id = grpd.user_id AND o1.content_id = grpd.content_id)
;
Assuming that in the old db table (source), you will not have the same (content_id, user_id) combination, then you can import using this query
insert newdbtable
select o.content_id, o.user_id, ifnull(max(n.`count`),-1)+1
from olddbtable o
left join newdbtable n on n.content_id=o.content_id and n.user_id=o.user_id
group by o.content_id, o.user_id;