MySQL version = 5.7.29
I want to do a MySQL search on a column which has delimited data. For example:
'field_black:1, field_white:2, field1_black:5, field_green:3'
I want a function which takes input the color and returns only the delimited values which do not have the input color.
func(input, color, delimiter)
func('field_black:1, field_white:2, field1_black:5, field1_green:3', 'black', ',') = 'field_white:2, field1_green:3'
This is pretty easy to implement in python using string split on delimiter and returning result set where the color is not in the given input.
def func(inp, col, delim):
inp = inp.split(delim)
res = []
for data in inp:
if col not in data:
res.append(data)
return (','.join(res))
Can anyone help me with an equivalent implementation in MySQL.
Thank you for the help!
CREATE FUNCTION func (input TEXT, color TEXT, delimiter CHAR(1))
RETURNS TEXT
DETERMINISTIC
BEGIN
DECLARE piece TEXT;
DECLARE result TEXT DEFAULT '';
/* SET color = CONCAT('field_', color); */ /* uncomment if needed */
REPEAT
SET piece = SUBSTRING_INDEX(input, delimiter, 1);
SET input = SUBSTRING(input FROM 2 + LENGTH(piece) FOR LENGTH(input));
IF NOT LOCATE(color, piece) THEN
SET result = CONCAT(result, delimiter, TRIM(piece));
END IF;
UNTIL input = ''
END REPEAT;
RETURN TRIM(BOTH delimiter FROM result);
END
fiddle
PS. Of course you may use multi-char delimiter if needed - alter input parameter type simply.
Just cracked this after a few iterations due to unfamiliarity with MySQL syntax. This is unnecessarily complicated though.
Answer by Akina is more simple and elegant: mysql delimit and search column data
CREATE FUNCTION `new_function`(input longtext, col TEXT, delim CHAR(1)) RETURNS longtext CHARSET utf8
DETERMINISTIC
BEGIN
declare result longtext default '';
declare piece longtext default '';
declare inptext longtext default '';
set inptext = input;
while (substring_index(inptext,delim,1) = '') = 0 DO
set piece = substring_index(inptext,delim,1);
IF NOT LOCATE(col, piece) THEN
set result = concat(result, piece, delim);
END IF;
set inptext = substr(inptext, length(SUBSTRING_INDEX(inptext, '|', 1) ) + 2);
END WHILE;
set result = left(result, length(result) -1);
RETURN result;
END
Im trying to remove newline characters when the next character is also a newline
using the Function I wrote below, and im running into a problem. When I input this string :
"Line1
Line2
Line3
Line4"
It removes the newlines after line 1 and 2 , but the lines between 3 and 4 remain ? Any ideas as to why the function doesn't work then ?
DELIMITER $$
DROP FUNCTION IF EXISTS removeLines;
CREATE FUNCTION test_dev.removeLines(address varchar(255))
RETURNS varchar(255)
DETERMINISTIC
BEGIN
DECLARE i int;
SET i = 0;
WHILE i < length(address)-1 DO
IF(((SUBSTR(address,i,1) in (CHAR(10) ,CHAR(13))) AND (SUBSTR(address,i+1,1) in (CHAR(10) ,CHAR(13))) ) )
THEN set address = INSERT(address,i,1,'');
SET i= i+1;
ELSE set i= i+1;
END IF;
END WHILE;
RETURN address;
END$$
DELIMITER ;
An alternate way to write your function so you don't have to manually keep track of character indexing.
while instr(address, '\r\n\r\n') do
address = replace(address, '\r\n\r\n', '\r\n');
end while;
I didn't have a chance to test this in a database so the syntax may not be 100% correct.
The problem is that you increment i after you replace a newline. Consider the following simple string:
1\n\n\n2
01 2 3 4 -- indexes
When i is 1, you see that there are newlines at indexes i=1 and i+1=2, so you remove the first one with INSERT(). Now the string is:
1\n\n2
01 2 3
and you do SET i = i + 1. Now i is 2, but indexes i=2 and i+1=3 don't have newlines, so you don't remove the next newline.
Change it so that you only increment i when you don't find a pair of newlines.
IF(((SUBSTR(address,i,1) in (CHAR(10) ,CHAR(13))) AND (SUBSTR(address,i+1,1) in (CHAR(10) ,CHAR(13))) ) )
THEN set address = INSERT(address,i,1,'');
ELSE set i= i+1;
END IF;
I have a table with ~500k rows; varchar(255) UTF8 column filename contains a file name;
I'm trying to strip out various strange characters out of the filename - thought I'd use a character class: [^a-zA-Z0-9()_ .\-]
Now, is there a function in MySQL that lets you replace through a regular expression? I'm looking for a similar functionality to REPLACE() function - simplified example follows:
SELECT REPLACE('stackowerflow', 'ower', 'over');
Output: "stackoverflow"
/* does something like this exist? */
SELECT X_REG_REPLACE('Stackoverflow','/[A-Zf]/','-');
Output: "-tackover-low"
I know about REGEXP/RLIKE, but those only check if there is a match, not what the match is.
(I could do a "SELECT pkey_id,filename FROM foo WHERE filename RLIKE '[^a-zA-Z0-9()_ .\-]'" from a PHP script, do a preg_replace and then "UPDATE foo ... WHERE pkey_id=...", but that looks like a last-resort slow & ugly hack)
MySQL 8.0+:
You can use the native REGEXP_REPLACE function.
Older versions:
You can use a user-defined function (UDF) like mysql-udf-regexp.
If you are using MariaDB or MySQL 8.0, they have a function
REGEXP_REPLACE(col, regexp, replace)
See MariaDB docs and PCRE Regular expression enhancements
Note that you can use regexp grouping as well (I found that very useful):
SELECT REGEXP_REPLACE("stackoverflow", "(stack)(over)(flow)", '\\2 - \\1 - \\3')
returns
over - stack - flow
My brute force method to get this to work was just:
Dump the table - mysqldump -u user -p database table > dump.sql
Find and replace a couple patterns - find /path/to/dump.sql -type f -exec sed -i 's/old_string/new_string/g' {} \;, There are obviously other perl regeular expressions you could perform on the file as well.
Import the table - mysqlimport -u user -p database table < dump.sql
If you want to make sure the string isn't elsewhere in your dataset, run a few regular expressions to make sure they all occur in a similar environment. It's also not that tough to create a backup before you run a replace, in case you accidentally destroy something that loses depth of information.
With MySQL 8.0+ you could use natively REGEXP_REPLACE function.
12.5.2 Regular Expressions:
REGEXP_REPLACE(expr, pat, repl[, pos[, occurrence[, match_type]]])
Replaces occurrences in the string expr that match the regular expression specified by the pattern pat with the replacement string repl, and returns the resulting string. If expr, pat, or repl is NULL, the return value is NULL.
and Regular expression support:
Previously, MySQL used the Henry Spencer regular expression library to support regular expression operators (REGEXP, RLIKE).
Regular expression support has been reimplemented using International Components for Unicode (ICU), which provides full Unicode support and is multibyte safe. The REGEXP_LIKE() function performs regular expression matching in the manner of the REGEXP and RLIKE operators, which now are synonyms for that function. In addition, the REGEXP_INSTR(), REGEXP_REPLACE(), and REGEXP_SUBSTR() functions are available to find match positions and perform substring substitution and extraction, respectively.
SELECT REGEXP_REPLACE('Stackoverflow','[A-Zf]','-',1,0,'c');
-- Output:
-tackover-low
DBFiddle Demo
I recently wrote a MySQL function to replace strings using regular expressions. You could find my post at the following location:
http://techras.wordpress.com/2011/06/02/regex-replace-for-mysql/
Here is the function code:
DELIMITER $$
CREATE FUNCTION `regex_replace`(pattern VARCHAR(1000),replacement VARCHAR(1000),original VARCHAR(1000))
RETURNS VARCHAR(1000)
DETERMINISTIC
BEGIN
DECLARE temp VARCHAR(1000);
DECLARE ch VARCHAR(1);
DECLARE i INT;
SET i = 1;
SET temp = '';
IF original REGEXP pattern THEN
loop_label: LOOP
IF i>CHAR_LENGTH(original) THEN
LEAVE loop_label;
END IF;
SET ch = SUBSTRING(original,i,1);
IF NOT ch REGEXP pattern THEN
SET temp = CONCAT(temp,ch);
ELSE
SET temp = CONCAT(temp,replacement);
END IF;
SET i=i+1;
END LOOP;
ELSE
SET temp = original;
END IF;
RETURN temp;
END$$
DELIMITER ;
Example execution:
mysql> select regex_replace('[^a-zA-Z0-9\-]','','2my test3_text-to. check \\ my- sql (regular) ,expressions ._,');
we solve this problem without using regex
this query replace only exact match string.
update employee set
employee_firstname =
trim(REPLACE(concat(" ",employee_firstname," "),' jay ',' abc '))
Example:
emp_id employee_firstname
1 jay
2 jay ajay
3 jay
After executing query result:
emp_id employee_firstname
1 abc
2 abc ajay
3 abc
UPDATE 2: A useful set of regex functions including REGEXP_REPLACE have now been provided in MySQL 8.0. This renders reading on unnecessary unless you're constrained to using an earlier version.
UPDATE 1: Have now made this into a blog post: http://stevettt.blogspot.co.uk/2018/02/a-mysql-regular-expression-replace.html
The following expands upon the function provided by Rasika Godawatte but trawls through all necessary substrings rather than just testing single characters:
-- ------------------------------------------------------------------------------------
-- USAGE
-- ------------------------------------------------------------------------------------
-- SELECT reg_replace(<subject>,
-- <pattern>,
-- <replacement>,
-- <greedy>,
-- <minMatchLen>,
-- <maxMatchLen>);
-- where:
-- <subject> is the string to look in for doing the replacements
-- <pattern> is the regular expression to match against
-- <replacement> is the replacement string
-- <greedy> is TRUE for greedy matching or FALSE for non-greedy matching
-- <minMatchLen> specifies the minimum match length
-- <maxMatchLen> specifies the maximum match length
-- (minMatchLen and maxMatchLen are used to improve efficiency but are
-- optional and can be set to 0 or NULL if not known/required)
-- Example:
-- SELECT reg_replace(txt, '^[Tt][^ ]* ', 'a', TRUE, 2, 0) FROM tbl;
DROP FUNCTION IF EXISTS reg_replace;
DELIMITER //
CREATE FUNCTION reg_replace(subject VARCHAR(21845), pattern VARCHAR(21845),
replacement VARCHAR(21845), greedy BOOLEAN, minMatchLen INT, maxMatchLen INT)
RETURNS VARCHAR(21845) DETERMINISTIC BEGIN
DECLARE result, subStr, usePattern VARCHAR(21845);
DECLARE startPos, prevStartPos, startInc, len, lenInc INT;
IF subject REGEXP pattern THEN
SET result = '';
-- Sanitize input parameter values
SET minMatchLen = IF(minMatchLen IS NULL OR minMatchLen < 1, 1, minMatchLen);
SET maxMatchLen = IF(maxMatchLen IS NULL OR maxMatchLen < 1
OR maxMatchLen > CHAR_LENGTH(subject),
CHAR_LENGTH(subject), maxMatchLen);
-- Set the pattern to use to match an entire string rather than part of a string
SET usePattern = IF (LEFT(pattern, 1) = '^', pattern, CONCAT('^', pattern));
SET usePattern = IF (RIGHT(pattern, 1) = '$', usePattern, CONCAT(usePattern, '$'));
-- Set start position to 1 if pattern starts with ^ or doesn't end with $.
IF LEFT(pattern, 1) = '^' OR RIGHT(pattern, 1) <> '$' THEN
SET startPos = 1, startInc = 1;
-- Otherwise (i.e. pattern ends with $ but doesn't start with ^): Set start pos
-- to the min or max match length from the end (depending on "greedy" flag).
ELSEIF greedy THEN
SET startPos = CHAR_LENGTH(subject) - maxMatchLen + 1, startInc = 1;
ELSE
SET startPos = CHAR_LENGTH(subject) - minMatchLen + 1, startInc = -1;
END IF;
WHILE startPos >= 1 AND startPos <= CHAR_LENGTH(subject)
AND startPos + minMatchLen - 1 <= CHAR_LENGTH(subject)
AND !(LEFT(pattern, 1) = '^' AND startPos <> 1)
AND !(RIGHT(pattern, 1) = '$'
AND startPos + maxMatchLen - 1 < CHAR_LENGTH(subject)) DO
-- Set start length to maximum if matching greedily or pattern ends with $.
-- Otherwise set starting length to the minimum match length.
IF greedy OR RIGHT(pattern, 1) = '$' THEN
SET len = LEAST(CHAR_LENGTH(subject) - startPos + 1, maxMatchLen), lenInc = -1;
ELSE
SET len = minMatchLen, lenInc = 1;
END IF;
SET prevStartPos = startPos;
lenLoop: WHILE len >= 1 AND len <= maxMatchLen
AND startPos + len - 1 <= CHAR_LENGTH(subject)
AND !(RIGHT(pattern, 1) = '$'
AND startPos + len - 1 <> CHAR_LENGTH(subject)) DO
SET subStr = SUBSTRING(subject, startPos, len);
IF subStr REGEXP usePattern THEN
SET result = IF(startInc = 1,
CONCAT(result, replacement), CONCAT(replacement, result));
SET startPos = startPos + startInc * len;
LEAVE lenLoop;
END IF;
SET len = len + lenInc;
END WHILE;
IF (startPos = prevStartPos) THEN
SET result = IF(startInc = 1, CONCAT(result, SUBSTRING(subject, startPos, 1)),
CONCAT(SUBSTRING(subject, startPos, 1), result));
SET startPos = startPos + startInc;
END IF;
END WHILE;
IF startInc = 1 AND startPos <= CHAR_LENGTH(subject) THEN
SET result = CONCAT(result, RIGHT(subject, CHAR_LENGTH(subject) + 1 - startPos));
ELSEIF startInc = -1 AND startPos >= 1 THEN
SET result = CONCAT(LEFT(subject, startPos), result);
END IF;
ELSE
SET result = subject;
END IF;
RETURN result;
END//
DELIMITER ;
Demo
Rextester Demo
Limitations
This method is of course going to take a while when the subject
string is large. Update: Have now added minimum and maximum match length parameters for improved efficiency when these are known (zero = unknown/unlimited).
It won't allow substitution of backreferences (e.g. \1, \2
etc.) to replace capturing groups. If this functionality is needed, please see this answer which attempts to provide a workaround by updating the function to allow a secondary find and replace within each found match (at the expense of increased complexity).
If ^and/or $ is used in the pattern, they must be at the very start and very end respectively - e.g. patterns such as (^start|end$) are not supported.
There is a "greedy" flag to specify whether the overall matching should be greedy or non-greedy. Combining greedy and lazy matching within a single regular expression (e.g. a.*?b.*) is not supported.
Usage Examples
The function has been used to answer the following StackOverflow questions:
How to count words in MySQL / regular expression
replacer?
How to extract the nth word and count word occurrences in a MySQL
string?
How to extract two consecutive digits from a text field in
MySQL?
How to remove all non-alpha numeric characters from a string in
MySQL?
How to replace every other instance of a particular character in a MySQL
string?
How to get all distinct words of a specified minimum length from multiple columns in a MySQL table?
I'm happy to report that since this question was asked, now there is a satisfactory answer! Take a look at this terrific package:
https://github.com/mysqludf/lib_mysqludf_preg
Sample SQL:
SELECT PREG_REPLACE('/(.*?)(fox)/' , 'dog' , 'the quick brown fox' ) AS demo;
I found the package from this blog post as linked on this question.
You 'can' do it ... but it's not very wise ... this is about as daring as I'll try ... as far as full RegEx support your much better off using perl or the like.
UPDATE db.tbl
SET column =
CASE
WHEN column REGEXP '[[:<:]]WORD_TO_REPLACE[[:>:]]'
THEN REPLACE(column,'WORD_TO_REPLACE','REPLACEMENT')
END
WHERE column REGEXP '[[:<:]]WORD_TO_REPLACE[[:>:]]'
I think there is an easy way to achieve this and It's working fine for me.
To SELECT rows using REGEX
SELECT * FROM `table_name` WHERE `column_name_to_find` REGEXP 'string-to-find'
To UPDATE rows using REGEX
UPDATE `table_name` SET column_name_to_find=REGEXP_REPLACE(column_name_to_find, 'string-to-find', 'string-to-replace') WHERE column_name_to_find REGEXP 'string-to-find'
REGEXP Reference:
https://www.geeksforgeeks.org/mysql-regular-expressions-regexp/
We can use IF condition in SELECT query as below:
Suppose that for anything with "ABC","ABC1","ABC2","ABC3",..., we want to replace with "ABC" then using REGEXP and IF() condition in the SELECT query, we can achieve this.
Syntax:
SELECT IF(column_name REGEXP 'ABC[0-9]$','ABC',column_name)
FROM table1
WHERE column_name LIKE 'ABC%';
Example:
SELECT IF('ABC1' REGEXP 'ABC[0-9]$','ABC','ABC1');
The one below basically finds the first match from the left and then replaces all occurences of it (tested in mysql-5.6).
Usage:
SELECT REGEX_REPLACE('dis ambiguity', 'dis[[:space:]]*ambiguity', 'disambiguity');
Implementation:
DELIMITER $$
CREATE FUNCTION REGEX_REPLACE(
var_original VARCHAR(1000),
var_pattern VARCHAR(1000),
var_replacement VARCHAR(1000)
) RETURNS
VARCHAR(1000)
COMMENT 'Based on https://techras.wordpress.com/2011/06/02/regex-replace-for-mysql/'
BEGIN
DECLARE var_replaced VARCHAR(1000) DEFAULT var_original;
DECLARE var_leftmost_match VARCHAR(1000) DEFAULT
REGEX_CAPTURE_LEFTMOST(var_original, var_pattern);
WHILE var_leftmost_match IS NOT NULL DO
IF var_replacement <> var_leftmost_match THEN
SET var_replaced = REPLACE(var_replaced, var_leftmost_match, var_replacement);
SET var_leftmost_match = REGEX_CAPTURE_LEFTMOST(var_replaced, var_pattern);
ELSE
SET var_leftmost_match = NULL;
END IF;
END WHILE;
RETURN var_replaced;
END $$
DELIMITER ;
DELIMITER $$
CREATE FUNCTION REGEX_CAPTURE_LEFTMOST(
var_original VARCHAR(1000),
var_pattern VARCHAR(1000)
) RETURNS
VARCHAR(1000)
COMMENT '
Captures the leftmost substring that matches the [var_pattern]
IN [var_original], OR NULL if no match.
'
BEGIN
DECLARE var_temp_l VARCHAR(1000);
DECLARE var_temp_r VARCHAR(1000);
DECLARE var_left_trim_index INT;
DECLARE var_right_trim_index INT;
SET var_left_trim_index = 1;
SET var_right_trim_index = 1;
SET var_temp_l = '';
SET var_temp_r = '';
WHILE (CHAR_LENGTH(var_original) >= var_left_trim_index) DO
SET var_temp_l = LEFT(var_original, var_left_trim_index);
IF var_temp_l REGEXP var_pattern THEN
WHILE (CHAR_LENGTH(var_temp_l) >= var_right_trim_index) DO
SET var_temp_r = RIGHT(var_temp_l, var_right_trim_index);
IF var_temp_r REGEXP var_pattern THEN
RETURN var_temp_r;
END IF;
SET var_right_trim_index = var_right_trim_index + 1;
END WHILE;
END IF;
SET var_left_trim_index = var_left_trim_index + 1;
END WHILE;
RETURN NULL;
END $$
DELIMITER ;
Yes, you can.
UPDATE table_name
SET column_name = 'seach_str_name'
WHERE column_name REGEXP '[^a-zA-Z0-9()_ .\-]';
for example,
I have a dataset as below:
asdf1234asdf
1235asdfasdf
asdfasdfef489
How could I select such that I can get a result as below?
1234
1235
489
If your MySQL version supports it, create a function and use such as regex, to extract the first digits:
DELIMITER $$
CREATE FUNCTION first_digits(str TEXT)
RETURNS TEXT
BEGIN
DECLARE ret TEXT DEFAULT '';
DECLARE chr TEXT DEFAULT '';
DECLARE i INT DEFAULT 1;
WHILE i < (LENGTH(str) + 1) DO
SET chr = SUBSTRING(str, i, 1);
IF chr REGEXP '[0-9]'
THEN SET ret = CONCAT(ret, chr);
ELSEIF ret != ''
THEN RETURN ret;
END IF;
SET i = i + 1;
END WHILE;
RETURN ret;
END;
$$
DELIMITER ;
Then just SELECT desired column with newly created function first_digits();
SELECT first_digits('asdf1234asdf') num;
num
1234
To extract all digits (not only first), drop the ELSEIF part and rename the function such as digits.
To drop the function:
DROP FUNCTION IF EXISTS first_digits;
Hope this helps!
MySQL, unfortunately, doesn't have a good way of replacing regexes, but if you have only one numerical sequence, you could use trim:
SELECT TRIM(BOTH 'abcdefghijklmnopqrstuvwxyz' FROM word)
FROM (SELECT 'asdf1234asdf' AS word
UNION ALL
SELECT '1235asdfasdf'
UNION ALL
SELECT 'asdfasdfef489') t
How do I convert data like that
\u0441\u043e\u0432\u0440\u0435\u043c\u0435\u043d
back to normal UTF8 inside MySQL? Is it even possible?
You can convert between charsets in mysql by reading the data as binary and convert it to a target charset, like this (if you have a table called example and a column called data:
UPDATE `example` SET data=CONVERT(CONVERT(`data` USING binary) USING utf8);
The JSON \uXXXX entities are in hexadecimal UTF-16 encoding, so if you have a mysql version with utf-16 support, you can convert them to utf-8. The function below shows you how to do it. First, you unhex() those values and then convert from UTF-16 to UTF-8:
DELIMITER ##
CREATE FUNCTION Unjson (instring TEXT CHARACTER SET utf8)
RETURNS TEXT CHARACTER SET utf8
BEGIN
DECLARE i INT DEFAULT 0;
DECLARE c VARCHAR(1);
DECLARE utfstr TEXT CHARACTER SET utf16 DEFAULT "";
DECLARE outstring TEXT CHARACTER SET utf8 DEFAULT "";
WHILE i < CHAR_LENGTH(instring) DO
SET i = i + 1;
SET c = SUBSTRING(instring, i, 1);
IF c = "\\" THEN
SET c = SUBSTRING(instring, i + 1, 1);
IF c = "u" THEN
SET utfstr = CONCAT(utfstr, UNHEX(SUBSTRING(instring, i + 2, 4)));
SET i = i + 5;
END IF;
ELSE
IF utfstr != "" THEN
SET outstring = CONCAT(outstring, CONVERT(utfstr USING utf8));
SET utfstr = "";
END IF;
SET outstring = CONCAT(outstring, c);
END IF;
END WHILE;
IF utfstr != "" THEN
SET outstring = CONCAT(outstring, CONVERT(utfstr USING utf8));
END IF;
RETURN outstring;
END##
DELIMITER ;
With this MySQL function in place, you can convert your tables with:
UPDATE `table_name` SET `column_name`=Unjson(`column_name`);
I'm using MySQL 5.5, but I don't think 5.0 has utf-16 support, so you might have to check your MySQL version...
(And yeah, I recommend you to make a backup before running this in your production environment...) ;)