I have created a database table in mySQL of which two column names are "landPhone" and "mobilePhone" to store phone numbers (in the format of: 123-456-8000 for land and 098-765-6601 for mobile). These two columns' data type are set to VARCHAR(30). The data have been inserted in the table. But after SQL query, I found the phone numbers have been truncated. It shows (above two data for example) only first 3 digits (123) for landPhone and only first 2 digits after removing the leading '0' (98) for mobilePhone.
Why this is happening ?
Phone numbers are not actually numbers; they are strings that happen to contain digits (and, in your case, dashes). If you try to interpret one as a number, two things typically happen:
Leading zeros are forgotten.
Everything from the first non-digit to the end of the string is stripped off.
That sounds exactly like the result you're describing. Even if you end up stuffing the result into a string field, it's too late -- the data has already been corrupted.
Make sure you're not treating phone numbers as integers at any point in the process.
You must use
insert into sample values('123-456-8000', '098-765-6601' )
instead of
insert into sample values(123-456-8000, 098-765-6601 )
see this SQLFiddle.
Thanks all for your solution. As cHao suspected, it was me who did the mistake. When I first time created the table, I declared the datatype of the phone columns as INT, later I corrected them to VARCHAR().
When I dropped the table and inserted the same data to the new table, it is working fine.
That sounds exactly like the result you're describing. Even if you end up stuffing the result into a string field, it's too late -- the data has already been corrupted. ..cHao
Question to understand: Why mySQL doesn't override the previous datatype with the new one ?
Related
In my table, I've got a column call mobile and I need this mobile field value to be normalized and save to another column call formatted_phone. For this purpose, I am using the below MySQL query and unfortunately, it is not working. I am putting my query here, please someone correct it. Thank you.
UPDATE hiring_detail
SET formatted_phone = replace(replace(
replace(replace(replace(replace(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.','')
WHERE mobile IS NOT NULL;
Error what it throws:
SQL Error (1265):Data truncated for column 'formatted_phone' at row 3
mobile column: varchar 50
formatted_phone: bigint 15
Usually, beofre blindly executing UPDATE commands, we do a simply What If analysis first, just run the query as a SELECT so you can inspect the output and importantly, you can compare it to the existing values:
http://sqlfiddle.com/#!9/47723a/2
SELECT mobile, formatted_phone
, REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.','') as test
FROM hiring_detail
WHERE mobile IS NOT NULL;
If that logic works for you (and it does in my tests) then there should be no issue with your UPDATE logic.
UPDATE:
If your error is
SQL Error (1265):Data truncated for column 'formatted_phone' at row 3
Then that means that your formatted phone numbers are longer than the column width for formatted_phone. If you know what the length is, you can truncate your formatted numbers, but with phone numbers, if we remove the actual number, this usually results in phone numbers that cannot be called.
I would recommend instead that you increase the width of the formatted_phone field.
This is an example of a forced truncation:
UPDATE hiring_detail
SET formatted_phone = RIGHT(replace(replace(
replace(replace(replace(replace(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.',''),10)
WHERE mobile IS NOT NULL;
Update #2
given that the column is an int, we need to convert the value into an integer.
UPDATE hiring_detail
SET formatted_phone = CAST(replace(replace(
replace(replace(replace(replace(mobile,'-',''),'+',''),')',''),'(',''),' ',''),'.','') as UNSIGNED INT)
WHERE mobile IS NOT NULL;
Warning: It is NOT advisable to store phone numbers as integers, the leading zeros can be significant in area codes in many localities, by storing as a numeric value this can have significant effects and can result in los of data. It also makes it hard to search for partial matches on the numbers. Almost all operations that you can think of (including sorting) on phone numbers will involve string manipulations, not mathematical or numerical.
Firstly I want this to be purely done with MySQL query.
I have a series of Invoice numbers
invoice_number
INV001
INV002
INV003
INV004
INV005
001
002
003
006
007
009
010
INVOICE333
INVOICE334
INVOICE335
INVOICE337
INVOICE338
INVOICE339
001INV
002INV
005INV
009INV
I want to output something like this
from_invoice_no to_invoice_no total_invoices
INV001 INV005 5
001 010 7
INVOICE333 INVOICE339 6
001INV 009INV 4
The invoice number pattern cannot be fixed. They can change in future
Please help me to achieve this.
Thanks in advance.
I will first show a general idea how to solve this problem and provide some code which will be ugly, but easily understandable. Then I'll explain what the issues are and how to remedy them.
STEP 1: Deriving the grouping criterion
For the first step, I assume you have the right (privilege) to create an additional column in your table. Let us name it invoice_text. Now, the general idea is to remove all digits from the invoice number so that only the "text pattern" remains. Then we can group by the text pattern.
Assuming that you have already created the column mentioned above, you could do the following:
UPDATE Invoices SET invoice_text = REPLACE(invoice_number, '0', '');
UPDATE Invoices SET invoice_text = REPLACE(invoice_text, '1', '');
UPDATE Invoices SET invoice_text = REPLACE(invoice_text, '2', '');
...
UPDATE Invoices SET invoice_text = REPLACE(invoice_text, '9', '');
After having done that, you will have the pure text pattern without digits in invoice_text and can use that for grouping:
SELECT COUNT(invoice_number) AS total_invoices FROM Invoices
GROUP BY invoice_text
This is nice, but it is not yet what you wanted. It does not show the first and last invoice number for each group.
STEP 2: Deriving the first and last invoice for each group
For this step, create one more column in your table. Let us name it invoice_digits. As the name implies, it is meant to take only the pure invoice number without the "pattern text".
Assuming you have that column, you could do the following:
UPDATE Invoices SET invoice_digits = REPLACE(invoice_number, 'A', '');
UPDATE Invoices SET invoice_digits = REPLACE(invoice_digits, 'B', '');
UPDATE Invoices SET invoice_digits = REPLACE(invoice_digits, 'C', '');
...
UPDATE Invoices SET invoice_digits = REPLACE(invoice_digits, 'Z', '');
Now, you can use that column to get the minimum and maximum invoice number (without "pattern text"):
SELECT
MIN(invoice_digits) AS from_invoice_no,
MAX(invoice_digits) AS to_invoice_no,
COUNT(invoice_number) AS total_invoices
FROM Invoices
GROUP BY invoice_text
Problems and how to solve them
1) According to your question, you want to get the minimum and maximum full invoice number text. The solution above will show only the minimum and maximum invoice number text without the text parts, i.e. only the digits.
We could remedy this by doing a further JOIN, but since I can very well imagine that you won't insist on this :-), and since it won't make the general idea more clear, I am leaving this to you. If you are interested, let us know.
2) It might be difficult to decide what a digit (i.e. what the actual invoice number) is. For example, if you have invoice numbers like INV001, INV002, this will be no problem, but what if you have INV001/001, INV001/002, INV002/003 and so on? In this example, my code would would yield 001001, 001002, 002003 as actual invoice numbers and use that to decide what the minimum and maximum numbers are.
This might not be what you want to do in that case. The only way around this is that you thoroughly think about what you should consider a digit and what not, and to adapt my code accordingly.
3) My code currently uses string comparisons to get the minimum and maximum invoice numbers. This may yield other results than comparing the values as numbers. If you are wondering what that means: Compare '19' to '9' as string, and compare 19 to 9 as number.
If this is a problem, then use MySQL's CAST to convert the text to a number before feeding it to MAX or MIN. But please be aware that this has its own caveats:
If you have very long invoice numbers with so many digits that they don't fit into MySQL's numeric data types, this method will fail. It will also fail if you have defined a character like / to be digits (due to the issues described in 2)) since MySQL can't convert this into a number.
Instead of converting to numbers, you can also pad the values in invoice_digits with leading zeroes, for example using MySQL's LPAD function. This will avoid the problems described above and sort the numbers as expected, even if they include non-digits like /, but you will have to know the maximum length of the digit string in advance.
4) The code is ugly! Do you really have to remove all possible characters from A to Z one by one by doing UPDATE statements to get the digit string?
Actually, it is even worse. I just have assumed that you only have the "text characters" A to Z in your invoices. But there could be any character Unicode defines: Russian or Chinese ones, special characters, in other words: thousands of different characters.
Unfortunately, AFAIK, MySQL still does not provide a REGEX-REPLACE function. I don't see any chance to get this problem solved unless you extend MySQL with an appropriate UDF (user defined function). There are some cool guys out there who have recognized the problem and have added such functions to MySQL. Since recommending libraries seems to be discouraged on SO, just google for "mysql regex replace".
When having extended MySQL that way, you can replace the ugly bunch of UPDATE statements which remove the digits / the text from the invoice number by a single one (using a REGEX, you can replace all digits or all non-digits at once).
For the sake of completeness, you could avoid the many UPDATE statements by doing UPDATE ... SET ... = REPLACE(REPLACE(REPLACE(...))) and thus applying all updates with one statement. But this is even more ugly and error prone, so if you are serious about your problem, you'll really have to extend MySQL by a REGEX-REPLACE.
5) The solution will only work if you have the privilege to create new columns in the table.
This is true for the solution as-is. But I have chosen to go that way solely because it makes the general idea clear and understandable. Instead of adding columns to your original table, you could also create a new table where you store the pure text / digits (this table might be a temporary one).
Furthermore, since MySQL supports grouping by computed values, you don't need additional columns / tables at all. You should decide by yourself what is the best way to go.
I am trying to save some value prefixed with the currency symbol like in €10. Yet when I manually enter them in the DB the euro sign gets turned now and then in ?. When then I query the line I get sometimes again the question mark with the value and some other times NaN for the full value.
The issue changes if I query the line by using the email field or the unique identifier. Using $ or ® instead of € presents no problems; even ™ is turned to ?, though.
What is strange is that if I try to replace the question mark with the original character, MariaDB complains that there is no change in the line, like if that character were in fact present even if not shown!
I tried restarting MariaDB, just in case, but the problem remained.
I am using UTF32 for encodage and utf32_unicode_ci for collation.
I am testing the thing with Sequel_pro without even touching php not to stack things.
At any rate if I execute the query from a php script and parse the result with JSON I get null for the value.
What could be the issue with those special characters?
Plan A: Store the amount as a string and do not try to get the value out of it. This requires, as already mentioned, "utf8 all the way through".
Plan B: Store only the amount in a numeric field. Either store the 'currency' in another field as 'EUR' or 'USD' or ... Or simply assume that all amounts are Euros. Then put the Euro sign in front of the amount when you print it.
Do not use DOUBLE or FLOAT, you get an undesirable extra rounding. Instead, consider DECIMAL(11,2). That will exactly handle amounts in most countries. (A few countries need 4 decimal places; some can live with 0.)
Do not use utf32; use utf8 (or utf8mb4).
A database is a repository of data, not a formatting tool. Keeping this distinction will help avoid problem like this.
I'm using RODBC to interface R with a MySQL database and have encountered a problem. I need to join two tables based on unique ID numbers (IDNUM below). The issue is that the ID numbers are 20 digit integers and R wants to round them. OK, no problem, I'll just pull these IDs as character strings instead of numeric using CAST(blah AS CHAR).
But R sees the incoming character strings as numbers and thinks "hey, I know these are character strings... but these character strings are just numbers, so I'm pretty sure this guy wants me to store this as numeric, let me fix that for him" then converts them back into numeric and rounds them. I need to force R to take the input as given and can't figure out how to make this happen.
Here's the code I'm using (Interval is a vector that contains a beginning and an ending timestamp, so this code is meant to only pull data from a chosen timeperiod):
test = sqlQuery(channel, paste("SELECT CAST(table1.IDNUM AS CHAR),PartyA,PartyB FROM
table1, table2 WHERE table1.IDNUM=table2.IDNUM AND table1.Timestamp>=",Interval[1],"
AND table2.Timestamp<",Interval[2],sep=""))
You will most likely want to read the documentation for the function you are using at ?sqlQuery, which includes notes about the following two relevant arguments:
as.is which (if any) columns returned as character should be
converted to another type? Allowed values are as for read.table. See
‘Details’.
and
stringsAsFactors logical: should columns returned as character and
not excluded by as.is and not converted to anything else be converted
to factors?
In all likelihood you want to specify the columns in questions in as.is.
I ran into a problem with SQL Server Integration Services 2012's new string function in the Expression Editor called TOKEN().
This is supposed to help you parse a delimited record. If the record comes out of a flat file, you can do this with the Flat File Source. In this case, I am dealing with old delimited import records that were stored as strings in a database VARCHAR field. Now they need to be extracted, massaged, and re-exported as delimited strings. For example:
1^Apple^0001^01/01/2010^Anteater^A1
2^Banana^0002^03/15/2010^Bear^B2
3^Cranberry^0003^4/15/2010^Crow^C3
If these strings are in a column called OldImportRecord, the delimiter is a caret (as shown), and we wish to put the fifth field into a Derived Column, we would use an expression like:
TOKEN(OldImportRecord,"^",5)
This returns Anteater, Bear, Crow, etc. In fact, we can create Derived Columns for each of the fields in this record (note that the index is one-based), change them as needed, and then build another delimited record for export.
Here's the problem. What if some of our data includes some empty strings (or Nulls rendered as empty strings)?
4^^0004^6/15/2010^Duck^D4
The TOKEN() fails to count the adjacent column delimiters, which throws off the column count. Now it only sees five columns instead of six columns. Our TOKEN(OldImportRecord,"^",5) returns "D4" instead of the intended "Duck". When we extract the fourth column, we wind up trying to put "Duck" into a Date column, and all sorts of fun ensues.
Here's a partial workaround:
TOKEN(REPLACE(OldImportRecord,"^^","^ ^"),"^",5)
Notice this misses every second delimiter pair, so it will fail for a string like "5^^^^Emu^E5", which looks like"5^ ^^ ^Emu^E5" after the REPLACE(). The column count is still wrong.
So here's my full workaround. This includes two nested REPLACE statements(), an RTRIM() to remove the superfluous spaces, and a DT_STR cast because I would like to keep the result in VARCHAR:
(DT_STR,255,1252)RTRIM(TOKEN(REPLACE(REPLACE(OldImportRecord,"^^","^ ^"),"^^","^ ^"),"^",5))
I am posting this for information, since others may also run into this problem.
Does anyone have a better workaround, or even a real solution?
Reason for the issue:
TOKEN method in SSIS uses the implementation of strtok function in C++. I gathered this information while reading the book Microsoft® SQL Server® 2012 Integration Services. It is mentioned as note on page 113 (I like this book! Lots of nice information.).
I searched for the implementation of strtok function and I found the following links.
INFO: strtok(): C Function -- Documentation Supplement - The code sample in this link shows that the function does ignore consecutive delimiter characters.
The answers to the following SO questions point out that strtok function is designed to ignore consecutive delimiters.
Need to know when no data appears between two token separators using strtok()
strtok_s behaviour with consecutive delimiters
I think that the TOKEN and TOKENCOUNT functions are working as per design but whether that is how SSIS should behave might be a question for the Microsoft SSIS team.
Original Post - Above section is an update:
I created a simple package in SSIS 2012 based on your data inputs. As you had described in your question, the TOKEN function does not behave as intended. I agree with you that the function doesn't seem to work. This post is not an answer to your original issue.
Here is an alternative way to write the expression in a relatively simpler fashion. This will only work if the last segment in your input record will always have a value (say A1, B2, C3 etc.).
Expression can be rewritten as:
This statement will take the input record as the parameter, the delimiter caret (^) as the second parameter. The third parameter calculates the total number segments in the records when split by the delimiter. If you have data in the last segment, you are guaranteed to have two segments. You can then subtract 1 to fetch the penultimate segment.
(DT_STR,50,1252)TOKEN(OldImportRecord,"^",TOKENCOUNT(OldImportRecord,"^") - 1)
I created a simple package with data flow task. OLE DB source retrieves the data and the derived transformation parses and splits the data as per the screenshot below. The output is then inserted into the destination table. You can see the source and destination tables in the last screenshot. Destination table has two columns. The first column stores the penultimate segment data and the segments count based on the delimiter (which again isn't correct). You can notice that the last record didn't fetch the correct results. If the last record didn't have the value 8, then the above expression will fail because the expression will evaluate to zero index.
Hope that helps to simplify your expression.
If you don't hear from anyone else, I would recommend logging this issue in Microsoft Connect website.
Create table and populate scripts:
CREATE TABLE [dbo].[SourceTable](
[OldImportRecord] [varchar](50) NOT NULL
) ON [PRIMARY]
GO
CREATE TABLE [dbo].[DestinationTable](
[NewImportRecord] [varchar](50) NOT NULL,
[CaretCount] [int] NOT NULL
) ON [PRIMARY]
GO
INSERT INTO dbo.SourceTable (OldImportRecord) VALUES
('1^Apple^0001^01/01/2010^Anteater^A1'),
('2^Banana^0002^03/15/2010^Bear^B2'),
('3^Cranberry^0003^4/15/2010^Crow^C3'),
('4^^0004^6/15/2010^Duck^D4'),
('5^^^^Emu^E5'),
('6^^^^Geese^F6'),
('^^^^Pheasant^G7'),
('8^^^^Sparrow^');
GO
Derived column transformation inside data flow task:
Data in source and destination tables:
Not only does TOKEN skip adjacent delimiters, it also skips leading and trailing delimiters as well. So, using your example, if you had a field "good" field that looks like this:
1^Apple^0001^01/01/2010^Anteater^A1
Followed by one with adjacent and leading delimiters like this:
^^^0004^6/15/2010^Duck^
TOKENCOUNT would only find two delimiters and you'd end up with 0004 assigned to Token1, 6/15/2010 for Token2, and Duck for Token3.
I used a different kind of replace. Rather than placing spaces between adjacent delimiters, which wouldn't help with leading or training, I used replace to surround the delimiters with characters I absolutely wouldn't find in my text. The following Expression works well for me. It's wordy, but it is what it is.
(DT_STR,255,1252)REPLACE(TOKEN(REPLACE(OldImportRecord,"^","~^~"),"^",1),"~","")
Of course, you'd replace the number 1 with whatever Token you wanted and adjust the cast according to your needs. Hope that helps.