Unable to find Comma Separated Values from Table in MySQL [duplicate] - mysql

I have a table say, ITEM, in MySQL that stores data as follows:
ID FEATURES
--------------------
1 AB,CD,EF,XY
2 PQ,AC,A3,B3
3 AB,CDE
4 AB1,BC3
--------------------
As an input, I will get a CSV string, something like "AB,PQ". I want to get the records that contain AB or PQ. I realized that we've to write a MySQL function to achieve this. So, if we have this magical function MATCH_ANY defined in MySQL that does this, I would then simply execute an SQL as follows:
select * from ITEM where MATCH_ANY(FEAURES, "AB,PQ") = 0
The above query would return the records 1, 2 and 3.
But I'm running into all sorts of problems while implementing this function as I realized that MySQL doesn't support arrays and there's no simple way to split strings based on a delimiter.
Remodeling the table is the last option for me as it involves lot of issues.
I might also want to execute queries containing multiple MATCH_ANY functions such as:
select * from ITEM where MATCH_ANY(FEATURES, "AB,PQ") = 0 and MATCH_ANY(FEATURES, "CDE")
In the above case, we would get an intersection of records (1, 2, 3) and (3) which would be just 3.
Any help is deeply appreciated.
Thanks

First of all, the database should of course not contain comma separated values, but you are hopefully aware of this already. If the table was normalised, you could easily get the items using a query like:
select distinct i.Itemid
from Item i
inner join ItemFeature f on f.ItemId = i.ItemId
where f.Feature in ('AB', 'PQ')
You can match the strings in the comma separated values, but it's not very efficient:
select Id
from Item
where
instr(concat(',', Features, ','), ',AB,') <> 0 or
instr(concat(',', Features, ','), ',PQ,') <> 0

For all you REGEXP lovers out there, I thought I would add this as a solution:
SELECT * FROM ITEM WHERE FEATURES REGEXP '[[:<:]]AB|PQ[[:>:]]';
and for case sensitivity:
SELECT * FROM ITEM WHERE FEATURES REGEXP BINARY '[[:<:]]AB|PQ[[:>:]]';
For the second query:
SELECT * FROM ITEM WHERE FEATURES REGEXP '[[:<:]]AB|PQ[[:>:]]' AND FEATURES REGEXP '[[:<:]]CDE[[:>:]];
Cheers!

select *
from ITEM where
where CONCAT(',',FEAURES,',') LIKE '%,AB,%'
or CONCAT(',',FEAURES,',') LIKE '%,PQ,%'
or create a custom function to do your MATCH_ANY

Alternatively, consider using RLIKE()
select *
from ITEM
where ','+FEATURES+',' RLIKE ',AB,|,PQ,';

Just a thought:
Does it have to be done in SQL? This is the kind of thing you might normally expect to write in PHP or Python or whatever language you're using to interface with the database.
This approach means you can build your query string using whatever complex logic you need and then just submit a vanilla SQL query, rather than trying to build a procedure in SQL.
Ben

Related

Select records from table where column value is contained in string (mysql with ActiveRecord)

Say I have a table called books with a column in it called keyword.
Say I also have a string called words.
I want to select all records from books where the column keyword is included in words.
I can do this using LIKE directly in mysql.
I'm trying to translate this into an ActiveRecord query for a rails app if anyone could give me a hand with the syntax.
The tricky part is that I want to find records where the column value is included in my string, not the other way around. Very similar to this question, except I need to translate it to ActiveRecord.
SQL - Query to find if a string contains part of the value in Column
Here is a working sql query that I am trying to translate.
SELECT * FROM books WHERE "new science fiction" LIKE CONCAT('%',keyword,'%');
In the above example "new science fiction" is the string words.
Say I have 3 book records. book1 has keyword 'science', book2 has keyword 'fiction' and book3 has keyword 'other'. My above query would return the records book1 and book2. Because 'science' and 'fiction' are included in the words string 'new science fiction'.
My SQL query works but I can't figure out how to do it with a Book.where statement in ActiveRecord.
I think your best option is:
Book.where(keyword: words.split)
which will result in
SELECT * FROM books WHERE books.keyword IN ('new', 'science', 'fiction')
this will return the same records.
Note: Depending on your RDBMS IN() may be case sensitive. To avoid this we can change the above to
Book.where(Book.arel_table[:keyword].lower.in(words.downcase.split))
which will result in
SELECT * FROM books WHERE LOWER(books.keyword) IN ('new', 'science', 'fiction')
If you really want to go the with the way you have it now we can hack this together as follows:
Book.where(
Arel::Nodes::UnaryOperation.new(nil,Arel::Nodes.build_quoted(words))
.matches(
Arel::Nodes::NamedFunction.new(
'CONCAT',
[Arel.sql("'%'"),Book.arel_table[:keyword],Arel.sql("'%'")])
))
This will result in the desired SQL while still taking advantage of the escaping provided by Arel and the connection adapter.
Case sensitivity is not an issue here because Arel::Predications#matches uses case_sensitive = false by default meaning Postgres will use ILIKE instead of LIKE

how to pass multiple variables in WHERE ... IN in stored procedure? [duplicate]

I have a column in one of my table where I store multiple ids seperated by comma's.
Is there a way in which I can use this column's value in the "IN" clause of a query.
The column(city) has values like 6,7,8,16,21,2
I need to use as
select * from table where e_ID in (Select city from locations where e_Id=?)
I am satisfied with Crozin's answer, but I am open to suggestions, views and options.
Feel free to share your views.
Building on the FIND_IN_SET() example from #Jeremy Smith, you can do it with a join so you don't have to run a subquery.
SELECT * FROM table t
JOIN locations l ON FIND_IN_SET(t.e_ID, l.city) > 0
WHERE l.e_ID = ?
This is known to perform very poorly, since it has to do table-scans, evaluating the FIND_IN_SET() function for every combination of rows in table and locations. It cannot make use of an index, and there's no way to improve it.
I know you said you are trying to make the best of a bad database design, but you must understand just how drastically bad this is.
Explanation: Suppose I were to ask you to look up everyone in a telephone book whose first, middle, or last initial is "J." There's no way the sorted order of the book helps in this case, since you have to scan every single page anyway.
The LIKE solution given by #fthiella has a similar problem with regards to performance. It cannot be indexed.
Also see my answer to Is storing a delimited list in a database column really that bad? for other pitfalls of this way of storing denormalized data.
If you can create a supplementary table to store an index, you can map the locations to each entry in the city list:
CREATE TABLE location2city (
location INT,
city INT,
PRIMARY KEY (location, city)
);
Assuming you have a lookup table for all possible cities (not just those mentioned in the table) you can bear the inefficiency one time to produce the mapping:
INSERT INTO location2city (location, city)
SELECT l.e_ID, c.e_ID FROM cities c JOIN locations l
ON FIND_IN_SET(c.e_ID, l.city) > 0;
Now you can run a much more efficient query to find entries in your table:
SELECT * FROM location2city l
JOIN table t ON t.e_ID = l.city
WHERE l.e_ID = ?;
This can make use of an index. Now you just need to take care that any INSERT/UPDATE/DELETE of rows in locations also inserts the corresponding mapping rows in location2city.
From MySQL's point of view you're not storing multiple ids separated by comma - you're storing a text value, which has the exact same meaing as "Hello World" or "I like cakes!" - i.e. it doesn't have any meaing.
What you have to do is to create a separated table that will link two objects from the database together. Read more about many-to-many or one-to-many (depending on your requirements) relationships in SQL-based databases.
Rather than use IN on your query, use FIND_IN_SET (docs):
SELECT * FROM table
WHERE 0 < FIND_IN_SET(e_ID, (
SELECT city FROM locations WHERE e_ID=?))
The usual caveats about first form normalization apply (the database shouldn't store multiple values in a single column), but if you're stuck with it, then the above statement should help.
This does not use IN clause, but it should do what you need:
Select *
from table
where
CONCAT(',', (Select city from locations where e_Id=?), ',')
LIKE
CONCAT('%,', e_ID, ',%')
but you have to make sure that e_ID does not contain any commas or any jolly character.
e.g.
CONCAT(',', '6,7,8,16,21,2', ',') returns ',6,7,8,16,21,2,'
e_ID=1 --> ',6,7,8,16,21,2,' LIKE '%,1,%' ? FALSE
e_ID=6 --> ',6,7,8,16,21,2,' LIKE '%,6,%' ? TRUE
e_ID=21 --> ',6,7,8,16,21,2,' LIKE '%,21,%' ? TRUE
e_ID=2 --> ',6,7,8,16,21,2,' LIKE '%,2,%' ? TRUE
e_ID=3 --> ',6,7,8,16,21,2,' LIKE '%,3,%' ? FALSE
etc.
Don't know if this is what you want to accomplish. With MySQL there is feature to concatenate values from a group GROUP_CONCAT
You can try something like this:
select * from table where e_ID in (Select GROUP_CONCAT(city SEPARATOR ',') from locations where e_Id=?)
this one in for oracle ..here string concatenation is done by wm_concat
select * from table where e_ID in (Select wm_concat(city) from locations where e_Id=?)
yes i agree with raheel shan .. in order put this "in" clause we need to make that column into row below code one do that job.
select * from table where to_char(e_ID)
in (
select substr(city,instr(city,',',1,rownum)+1,instr(city,',',1,rownum+1)-instr(city,',',1,rownum)-1) from
(
select ','||WM_CONCAT(city)||',' city,length(WM_CONCAT(city))-length(replace(WM_CONCAT(city),','))+1 CNT from locations where e_Id=? ) TST
,ALL_OBJECTS OBJ where TST.CNT>=rownum
) ;
you should use
FIND_IN_SET Returns position of value in string of comma-separated values
mysql> SELECT FIND_IN_SET('b','a,b,c,d');
-> 2
You need to "SPLIT" the city column values. It will be like:
SELECT *
FROM table
WHERE e_ID IN (SELECT TO_NUMBER(
SPLIT_STR(city /*string*/
, ',' /*delimiter*/
, 1 /*start_position*/
)
)
FROM locations);
You can read more about the MySQL split_str function here: http://blog.fedecarg.com/2009/02/22/mysql-split-string-function/
Also, I have used the TO_NUMBER function of Oracle here. Please replace it with a proper MySQL function.
IN takes rows so taking comma seperated column for search will not do what you want but if you provide data like this ('1','2','3') this will work but you can not save data like this in your field whatever you insert in the column it will take the whole thing as a string.
You can create a prepared statement dynamically like this
set #sql = concat('select * from city where city_id in (',
(select cities from location where location_id = 3),
')');
prepare in_stmt from #sql;
execute in_stmt;
deallocate prepare in_stmt;
Ref: Use a comma-separated string in an IN () in MySQL
Recently I faced the same problem and this is how I resolved it.
It worked for me, hope this is what you were looking for.
select * from table_name t where (select (CONCAT(',',(Select city from locations l where l.e_Id=?),',')) as city_string) LIKE CONCAT('%,',t.e_ID,',%');
Example: It will look like this
select * from table_name t where ',6,7,8,16,21,2,' LIKE '%,2,%';

MYSQL REGEX search many words with no order condition

I try to use a regex with mysql that search boundary words in a json array string but I don't want the regex match words order because I don't know them.
So I started firstly to write my regex on regex101 (https://regex101.com/r/wNVyaZ/1) and then try to convert this one for mysql.
WHERE `Wish`.`services` REGEXP '^([^>].*[[:<:]]Hygiène[[:>:]])([^>].*[[:<:]]Radiothérapie[[:>:]]).+';
WHERE `Wish`.`services` REGEXP '^([^>].*[[:<:]]Hygiène[[:>:]])([^>].*[[:<:]]Andrologie[[:>:]]).+';
In the first query I get result, cause "Hygiène" is before "Radiothérapie" but in the second query "Andrologie" is before "Hygiène" and not after like it written in the query. The problem is that the query is generated automatically with a list of services that are choosen with no order importance and I want to match only boundary words if they exists no matter the order they have.
You can search for words in JSON like the following (I tested on MySQL 5.7):
select * from wish
where json_search(services, 'one', 'Hygiène') is not null
and json_search(services, 'one', 'Andrologie') is not null;
+------------------------------------------------------------+
| services |
+------------------------------------------------------------+
| ["Andrologie", "Angiologie", "Hygiène", "Radiothérapie"] |
+------------------------------------------------------------+
See https://dev.mysql.com/doc/refman/5.7/en/json-search-functions.html#function_json-search
If you can, use the JSON search queries (you need a MySQL with JSON support).
If it's advisable, consider changing the database structure and enter the various "words" as a related table. This would allow you much more powerful (and faster) queries.
JOIN has_service AS hh ON (hh.row_id = id)
JOIN services AS ss ON (hh.service_id = ss.id
AND ss.name IN ('Hygiène', 'Angiologie', ...)
Otherwise, in this context, consider that you're not really doing a regexp search, and you're doing a full table scan anyway (unless MySQL 8.0+ or PerconaDB 5.7+ (not sure) and an index on the full extent of the 'services' column), and several LIKE queries will actually cost you less:
WHERE (services LIKE '%"Hygiène"%'
OR services LIKE '%"Angiologie"%'
...)
or
IF(services LIKE '%"Hygiène"%', 1, 0)
+IF(services LIKE '%"Angiologie"%', 1, 0)
+ ... AS score
HAVING score > 0 -- or score=5 if you want only matches on all full five
ORDER BY score DESC;

Query MySQL field for LIKE string

I have a field called 'areasCovered' in a MySQL database, which contains a string list of postcodes.
There are 2 rows that have similar data e.g:
Row 1: 'B*,PO*,WA*'
Row 2: 'BB*, SO*, DE*'
Note - The strings are not in any particular order and could change depending on the user
Now, if I was to use a query like:
SELECT * FROM technicians WHERE areasCovered LIKE '%B*%'
I'd like it to return JUST Row 1. However, it's returning Row 2 aswell, because of the BB* in the string.
How could I prevent it from doing this?
The key to using like in this case is to include delimiters, so you can look for delimited values:
SELECT *
FROM technicians
WHERE concat(', ', areasCovered, ', ') LIKE '%, B*, %'
In MySQL, you can also use find_in_set(), but the space can cause you problems so you need to get rid of it:
SELECT *
FROM technicians
WHERE find_in_set('B', replace(areasCovered, ', ', ',') > 0
Finally, though, you should not be storing these types of lists as strings. You should be storing them in a separate table, a junction table, with one row per technician and per area covered. That makes these types of queries easier to express and they have better performance.
You are searching wild cards at the start as well as end.
You need only at end.
SELECT * FROM technicians WHERE areasCovered LIKE 'B*%'
Reference:
Normally I hate REGEXP. But ho hum:
SELECT * FROM technicians
WHERE concat(",",replace(areasCovered,", ",",")) regexp ',B{1}\\*';
To explain a bit:
Get rid of the pesky space:
select replace("B*,PO*,WA*",", ",",");
Bolt a comma on the front
select concat(",",replace("B*,PO*,WA*",", ",","));
Use a REGEX to match "comma B once followed by an asterix":
select concat(",",replace("B*,PO*,WA*",", ",",")) regexp ',B{1}\\*';
I could not check it on my machine, but it's should work:
SELECT * FROM technicians WHERE areasCovered <> replace(areaCovered,',B*','whatever')
In case the 'B*' does not exist, the areasCovered will be equal to replace(areaCovered,',B*','whatever'), and it will reject that row.
In case the 'B*' exists, the areCovered will NOT be eqaul to replace(areaCovered,',B*','whatever'), and it will accept that row.
You can Do it the way Programming Student suggested
SELECT * FROM technicians WHERE areasCovered LIKE 'B*%'
Or you can also use limit on query
SELECT * FROM technicians WHERE areasCovered LIKE '%B*%' LIMIT 1
%B*% contains % on each side which makes it to return all the rows where value contains B* at any position of the text however your requirement is to find all the rows which contains values starting with B* so following query should do the work.
SELECT * FROM technicians WHERE areasCovered LIKE 'B*%'

Searching for multiple keywords using SQL Server stored procedure

I'm going to search my database (SQL Server 2008) using a stored procedure. My users can enter keyword(s) in a textbox (keywords can be separated using , for instance).
Currently I'm using something like this:
keyword like N"%'+#SearchQuery%'%"
(keyword is a nvarchar column in my table, and #SearchQuery is the input to my stored procedure)
It works fine but what if user types several keywords: apple,orange, banana
Should I limit number of my keywords? How should I write my stored procedure if I have more than one keyword? How should I pass my user input to the stored procedure? I should pass apple, orange, banana as a whole phrase and then I should parse them in my stored procedure, or I should separate my keywords and send 3 keywords? How can I query these 3 keywords? A for loop?
What are best practices for performing such queries?
thanks
Do the parsing of the keywords in your application. SQL is not the best place for string manipulation.
Send the keywords as a table valued parameter (ie : http://www.mssqltips.com/sqlservertip/2112/table-value-parameters-in-sql-server-2008-and-net-c/ ) then you aren't limited to a fixed number of keywords.
Add the wildcards to the parameter in the stored procedure
update #keywords set keyword = '%'+keyword+'%'
filter your results by joining your source data to this table
eg:
SELECT result
FROM source
INNER JOIN #keywords keywords
ON source.keyword LIKE keywords.keyword
It depends on:
* How big it's your database.
* How often users will search for something.
* How precise results users except.
LIKE is not performance daemon, especially starting with %.
Maybe you should try full search text?
If you would like stay with LIKE (it will works only for small tables) I would try something like:
Split intput by , character (insert them into table as podiluska suggested is a good idea).
Build query for each token and UNION all results. Or run it in loop for each token and insert results to temporary table.
If you need some precise results (i.e. only records matches all 3 words) you can select most matching results from temporary results built above.
You could use CTE to split the string of keywords in a temporary table and then use it as you like. The keyword list can even have numbers or any characters, like %$<> or what you want, just remember comma is the string separator
DECLARE #CommaSeparatorString VARCHAR(MAX),
#CommaSeparatorXML XML
DECLARE #handle INT
SELECT #CommaSeparatorString = 'apple,orange,banana'
SELECT #CommaSeparatorString = REPLACE(REPLACE(#CommaSeparatorString,'<','$^%'),'>','%^$')
SELECT #CommaSeparatorXML = CAST('<ROOT><i>' + REPLACE(#CommaSeparatorString, ',', '</i><i>') + '</i></ROOT>' AS XML)
SELECT REPLACE(REPLACE(c.value('.', 'VARCHAR(100)'),'$^%','<'),'%^$','>') AS ID
FROM (SELECT #CommaSeparatorXML AS CommaXML) a
CROSS APPLY CommaXML.nodes('//i') x(c)
Result:
ID
------
apple
orange
banana