In not a database guy but: I have mixed up data in a mySql database that I inherited.
Some Phone numbers are formatted (512) 555-1212 (call it dirty)
Others 5125551212 (Call it clean)
I need a sqlstamet that says
UPDATE table_name
SET Phone="clean'(Some sort of cleaning code - regex?)
WHERE Phone='Dirty'
Unfortunately there's no regex replace/update in MySQL. If it's just parentheses and dashes and spaces then some nested REPLACE calls will do the trick:
UPDATE table_name
SET Phone = REPLACE(REPLACE(REPLACE(REPLACE(Phone, '-', ''), ')', ''), '(', ''), ' ', '')
To my knowledge you can't run a regexp to replace data during the update process. Only during the SELECT statement.
Your best bet is to use a scripting language that you're familiar with and read the table and change it that way. Basically by retrieving all the entries. Then using a string replace to match a simple regexp such as [^\d]* and remove those characters. Then update the table with the new value.
Also, see this answer:
How to do a regular expression replace in MySQL?
I use a CONVERT function in my SELECT statement in order to avert utf8 errors, but MySQL leaves question marks behind. Is there a way to convert the unconvertable characters to blank or space characters?
SELECT MeetId,
ResId,
Special,
CONVERT(proposal USING ascii) as Proposal,
Analysis,
Vote,
Vote_for,
Oppose,
Discret,
Abstain,
gpVote %s
FROM RESO
WHERE RESO.MeetId = %s
As an example a typical result may have this in a field: 'The current issue ?A? is on the table '
What about just using REPLACE:
SELECT
REPLACE(CONVERT('§123' USING ascii), '?', '')
And the Fiddle.
Good luck.
Be careful, sgeddes solution also removes all question marks(if exist) from your string!
For example :
SELECT REPLACE(CONVERT('§How are you?' USING ascii), '?', '')
Output will be : How are you
I am using MySQL. My data has a column called text, which uses the TEXT data type.
There are several newlines for each record in this column. I want to remove all new lines with a sql query. How can I do that?
Try this one -
CREATE TABLE table1(column1 TEXT);
INSERT INTO table1 VALUES ('text1\r\ntext2
text3');
SELECT * FROM table1;
--------
text1
text2
text3
UPDATE table1 SET column1 = REPLACE(column1, '\r\n', '');
SELECT * FROM table1;
--------
text1text2text3
The previous suggestions did not work for me. It only seems to work if I had actually typed the \r and \n text in as text. I found the following to work well -
replace(replace([MyFieldName],char(13),''),char(10),'')
I also created a calculated field in my table which uses this formula. That way I can just reference that field in areas of my program that were breaking with the original field contents.
Given suggestion i.e. REPLACE(REPLACE(DBField, '\n', ''), '\r', '') won't work if there are invisible html code like \n \r. For that you have to use char code.
Example:
REPLACE(REPLACE(REPLACE(DBField, CHAR(10), ''), CHAR(13), ''), CHAR(9), '')
Try this
REPLACE(REPLACE(FIELD, '\n', ''), '\r', '')
You can use REPLACE(text,'\n\r',' ') for it.
i've tried all said here but neither worked for me, my DB is IBM Informix,
however i managed to solve the problem like this:
UPDATE personas SET foto_path = SUBSTRING(foto_path FROM 1 FOR LENGTH(foto_path) - 1);
Hope it helps other in similar situation.
The above version with single backslash cleared up some for me but also had to run this to clear up the rest.
REPLACE(REPLACE(FIELD, '\\n', ''), '\\r', '')
I have the following value in my table:
When I Don\'t Know What To Do
Notice the apostrophe is already escaped. Now I want to select this value from this table using the following SQL string:
SELECT * FROM wp_wpsc_productmeta WHERE meta_value = 'When I Don\'t Know What To Do'
I am getting 0 rows returned. When I change the SQL string to this:
SELECT * FROM wp_wpsc_productmeta WHERE meta_value LIKE 'When I Don%'
I get my result set correctly. So, my question is, how do I write the SQL string so that I can select a value from a table when that value has an already-escaped apostrophe in it?
Please do not answer that I should use parameters. I'd very much like to figure out the answer using the method that I'm trying. Thanks for your help!
If the string in the database is actually "When I Don\'t Know What To Do", with the backslash included then to search for it you would need to use the statement.
SELECT * FROM wp_wpsc_productmeta WHERE meta_value = 'When I Don\\''t Know What To Do'
Or
SELECT * FROM wp_wpsc_productmeta WHERE meta_value = 'When I Don\\\'t Know What To Do'
You'll need to escape both the backslash and the quote:
'I don\\\'t know';
Why?
The problem is that the slash and the quote are special characters in your programming language as well. So when you're typing:
'I don\'t know';
What the database sees is 'I don't know', which does not exist. If on the other hand you double-escape your string like so:
'I don\\\'t know';
...the result sent to the database is 'I don\'t know', which is what you want.
What language are you using? I think the language is probably interpreting the escape backslash as an escape for its benefit, so you may need to do it twice to ensure the database sees it too:
'I don\\\'t know';
I'm importing some data from a CSV file, and numbers that are larger than 1000 get turned into 1,100 etc.
What's a good way to remove both the quotes and the comma from this so I can put it into an int field?
Edit:
The data is actually already in a MySQL table, so I need to be able to this using SQL. Sorry for the mixup.
My guess here is that because the data was able to import that the field is actually a varchar or some character field, because importing to a numeric field might have failed. Here was a test case I ran purely a MySQL, SQL solution.
The table is just a single column (alpha) that is a varchar.
mysql> desc t;
+-------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| alpha | varchar(15) | YES | | NULL | |
+-------+-------------+------+-----+---------+-------+
Add a record
mysql> insert into t values('"1,000,000"');
Query OK, 1 row affected (0.00 sec)
mysql> select * from t;
+-------------+
| alpha |
+-------------+
| "1,000,000" |
+-------------+
Update statement.
mysql> update t set alpha = replace( replace(alpha, ',', ''), '"', '' );
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from t;
+---------+
| alpha |
+---------+
| 1000000 |
+---------+
So in the end the statement I used was:
UPDATE table
SET field_name = replace( replace(field_name, ',', ''), '"', '' );
I looked at the MySQL Documentation and it didn't look like I could do the regular expressions find and replace. Although you could, like Eldila, use a regular expression for a find and then an alternative solution for replace.
Also be careful with s/"(\d+),(\d+)"/$1$2/ because what if the number has more then just a single comma, for instance "1,000,000" you're going to want to do a global replace (in perl that is s///g). But even with a global replace the replacement starts where you last left off (unless perl is different), and would miss the every other comma separated group. A possible solution would be to make the first (\d+) optional like so s/(\d+)?,(\d+)/$1$2/g and in this case I would need a second find and replace to strip the quotes.
Here are some ruby examples of the regular expressions acting on just the string "1,000,000", notice there are NOT double quote inside the string, this is just a string of the number itself.
>> "1,000,000".sub( /(\d+),(\d+)/, '\1\2' )
# => "1000,000"
>> "1,000,000".gsub( /(\d+),(\d+)/, '\1\2' )
# => "1000,000"
>> "1,000,000".gsub( /(\d+)?,(\d+)/, '\1\2' )
# => "1000000"
>> "1,000,000".gsub( /[,"]/, '' )
# => "1000000"
>> "1,000,000".gsub( /[^0-9]/, '' )
# => "1000000"
Here is a good case for regular expressions. You can run a find and replace on the data either before you import (easier) or later on if the SQL import accepted those characters (not nearly as easy). But in either case, you have any number of methods to do a find and replace, be it editors, scripting languages, GUI programs, etc. Remember that you're going to want to find and replace all of the bad characters.
A typical regular expression to find the comma and quotes (assuming just double quotes) is: (Blacklist)
/[,"]/
Or, if you find something might change in the future, this regular expression, matches anything except a number or decimal point. (Whitelist)
/[^0-9\.]/
What has been discussed by the people above is that we don't know all of the data in your CSV file. It sounds like you want to remove the commas and quotes from all of the numbers in the CSV file. But because we don't know what else is in the CSV file we want to make sure that we don't corrupt other data. Just blindly doing a find/replace could affect other portions of the file.
You could use this perl command.
Perl -lne 's/[,|"]//; print' file.txt > newfile.txt
You may need to play around with it a bit, but it should do the trick.
Here's the PHP way:
$stripped = str_replace(array(',', '"'), '', $value);
Link to W3Schools page
Actually nlucaroni, your case isn't quite right. Your example doesn't include double-quotes, so
id,age,name,...
1,23,phil,
won't match my regex. It requires the format "XXX,XXX". I can't think of an example of when it will match incorrectly.
All the following example won't include the deliminator in the regex:
"111,111",234
234,"111,111"
"111,111","111,111"
Please let me know if you can think of a counter-example.
Cheers!
The solution to the changed question is basically the same.
You will have to run select query with the regex where clause.
Somthing like
Select *
FROM SOMETABLE
WHERE SOMEFIELD REGEXP '"(\d+),(\d+)"'
Foreach of these rows, you want to do the following regex substitution s/"(\d+),(\d+)"/$1$2/ and then update the field with the new value.
Please Joseph Pecoraro seriously and have a backup before doing mass changes to any files or databases. Because whenever you do regex, you can seriously mess up data if there are cases that you have missed.
My command does remove all ',' and '"'.
In order to convert the sting "1,000" more strictly, you will need the following command.
Perl -lne 's/"(\d+),(\d+)"/$1$2/; print' file.txt > newfile.txt
Daniel's and Eldila's answer have one problem: They remove all quotes and commas in the whole file.
What I usually do when I have to do something like this is to first replace all separating quotes and (usually) semicolons by tabs.
Search: ";"
Replace: \t
Since I know in which column my affected values will be I then do another search and replace:
Search: ^([\t]+)\t([\t]+)\t([0-9]+),([0-9]+)\t
Replace: \1\t\2\t\3\4\t
... given the value with the comma is in the third column.
You need to start with an "^" to make sure that it starts at the beginning of a line. Then you repeat ([0-9]+)\t as often as there are columns that you just want to leave as they are.
([0-9]+),([0-9]+) searches for values where there is a number, then a comma and then another number.
In the replace string we use \1 and \2 to just keep the values from the edited line, separating them with \t (tab). Then we put \3\4 (no tab between) to put the two components of the number without the comma right after each other. All values after that will be left alone.
If you need your file to have semicolon to separate the elements, you then can go on and replace the tabs with semicolons. However then - if you leave out the quotes - you'll have to make sure that the text values do not contain any semicolons themselves. That's why I prefer to use TAB as column separator.
I usually do that in an ordinary text editor (EditPlus) that supports RegExp, but the same regexps can be used in any programming language.