I am trying a simple conversion - a mysql varchar column which contains longitude values (eg "-1.234" to -1.234) to enhance optimisation.
I have read and read on this and nothing works, last tried was:
UPDATE latlng_data SET lng_copy = CAST(lng AS DECIMAL(6,4));
met with:
Error Code: 1366. Incorrect decimal value: '' for column '' at row -1
Target column, created to recover from errors is:
FLOAT(6,4)
and null allowed.
Why is mysql error messages so useless.
Following 'dognose' advice (below) I have used:
UPDATE latlng_data SET lng='999.9999' where lng='';-- to give invalid lng in this case
then increase the copy data field (with the aim to rename it orig) and then:
UPDATE latlng_data SET lng_copy = CAST(lng AS DECIMAL(7,4));
this appears to be required in mysql cmd as Workbench timesout but using:
show full processlist;
still shows it as running - so here the best monitor is cmd.
I think the easiest way to convert numbers in MySQL is to use silent conversion. That is, use the string in an arithmetic operation and any conversion errors are ignored. A string with no numbers is interpreted as 0:
UPDATE latlng_data
SET lng_copy = lng + 0.0;
Here is some relevant documentation.
EDIT:
If you are concerned about matching only correct values, then check the format of the string
WHERE lng rlike '[0-9]+[.]?[0-9]*'
Checked this with:
SELECT * FROM t31data_happyenergy.latlng_data where lng_copy <> CAST(lng AS DECIMAL(7,4));
which now returns no results and can be better optimised as all are numerical, with bad data fully valued for a longitude.
I also recommend using a mysql command window as most apps timeout when extensive queries are used with this large data.
So basically the process went [ALWAYS CHECK BETWEEN EACH QUERY AS I GOT LAZY AND HAD TO START AGAIN AS ONE QUERY WAS WRONG):
-- convert all mostly read only tables to myisam:
-- dropped old lat and lng after checking same data by
SELECT count(*) FROM latlng_postcode where CAST(lat AS DECIMAL(7,4))!=latcopy;
SELECT count(*) FROM latlng_postcode where CAST(lng AS DECIMAL(7,4))!=lngcopy;
ALTER TABLE `latlng_postcode`
CHANGE COLUMN `latcopy` `lat` FLOAT(7,4) NULL DEFAULT NULL,
CHANGE COLUMN `lngcopy` `lng` FLOAT(7,4) NULL DEFAULT NULL;
-- then index (are these the best settings ??
ALTER TABLE `latlng_postcode`
DROP INDEX `pcode` ,
ADD INDEX `pcode` USING BTREE (`postcode`(5) ASC),
ADD INDEX `lat` USING BTREE (`lat`(4) ASC),
ADD INDEX `lng` USING BTREE (`lng`(4) ASC);
Related
I was trying to convert an existing varchar column with a unique index on it to a case sensitive column. So to do this, I updated the collation of the particular column.
Previous value: utf8mb4_unicode_ci
Current value: utf8mb4_bin
Now I have a row in my table TEST_TABLE with test_column value is abcd.
When I try to run a simple query like SELECT * FROM TEST_TABLE WHERE test_column = 'abcd'; it returns no result.
However when I try SELECT * FROM TEST_TABLE WHERE test_column LIKE 'abcd'; it returns the data correctly.
Also when I try SELECT * FROM TEST_TABLE WHERE BINARY test_column = 'abcd'; it returns the data correctly.
One more thing I tried was creating a duplicate of the table with column collation set as utf8mb4_bin while creating itself and then copy all data from original table. Then the query SELECT * FROM TEST_TABLE WHERE test_column = 'abcd'; is working alright.
So this seems to be a problem with BINARY conversion. Is there any solution to this or Am I doing something wrong ?
This seems to be an issue with MySQL. The steps I followed to resolve this is as follows:
dropped the unique index on the column
change the collation of the column
created the unique index again
Now it is working as expected. It seems MySQL didn't rebuild unique index when collation was changed. However the above steps solved my issue.
How did you change the collation? There are about 4 ways that you might think to do it. Most do something different.
Probably ALTER TABLE ... CONVERT TO COLLATION utf8mb4_bin was what you needed.
Why "bin"? You want to match case and accents? That is "abcd" != "Abcd"?
I have a table that looks like this:
CREATE TABLE t1 (
id BIGINT AUTO_INCREMENT NOT NULL PRIMARY KEY,
col1 VARCHAR(256),
UNIQUE INDEX t1_col1_index (col1)
)
I'm trying to modify the col1 type using the following query:
ALTER TABLE t1 MODIFY COLUMN col1 varchar(191) COLLATE utf8mb4_unicode_ci;
However, I run into this duplication error:
error: ("1062", "QMYSQL3: Unable to execute statement", "Duplicate entry '+123456789' for key 't1_col1_index'")
I initially thought it could be because two or more rows might 'contain' similar value for col1 and on changing varchar length the data gets truncated but then I found out that data truncation wouldn't even allow the query to go through. Any pointers on what could be causing this?
EDIT (Resolved): Truncation does happen when ##sql_mode is not set with STRICT_TRANS_TABLES. This was causing the error.
You are reducing the length of a varchar column that is controlled by a UNIQUE constraint.
This is risky business. Oversize data will be silently trimed (unless you have the ##sql_mode set to STRICT_TRANS_TABLES in which case an error will be raised). This probably generates duplicates, which cause the error to be raised by your UNIQUE constraint.
You can check the max length of the values in your column with :
SELECT MAX(CHAR_LENGTH(col1)) FROM t1:
I am not sure if this is work.
Try to check the table t1.
select count(1) from t1 where col1 = 123456789
Now if count is greater than one then try to remove the other one and leave only one record.
Then try to run your statement again.
Reminder:
Do back up first before removing.
I am working in a PHP + MySQL application. The application is working fine for me. But when I hosted it in another server, I got a MySQL error:
Error Code: 1364. Field 'field' doesn't have a default value
I know this is a problem with the MySQL version and we should setup default values for all columns. But currently I have more than 100 tables. So I need to set default value to NULL for all columns in all tables that has no default value yet.
I can't make use of the strict mode option, because the server is a shared one. Is it possible to setup in a single step rather than setting for each and every table ? If not possible tell me the easiest way to setup it.
Any help would be greatly appreciated
For anyone else with this problem, it will take a bit of coding to perform automatically, but the following would be how you would do so:
First run the following query:
SELECT table_schema,table_name,column_name,data_type FROM information_schema.columns WHERE IS_NULLABLE='NO' AND column_default is null AND column_key=''
Next, for each row returned from the above query perform the following:
If data_type contains 'int' set default to 0
else if data_type='datetime' set default to '0000-00-00 00:00:00'
else if data_type='date' set default to '0000-00-00'
else if data_type='time' set default to '00:00:00'
else set default to ''
create and run the following query with all [[...]] variables replaced with their proper values:
ALTER TABLE `[[table_schema]]`.`[[table_name]]` ALTER COLUMN `[[column_name]]` SET DEFAULT '[[default]]'
This should replace the default values for all databases, all tables, all columns that are set to be NOT NULL and are not primary keys and have no default value set.
Another solution that i found is like:-
Get all column name put it in array...
Now push values in column array for inserting -- with ZERO value for all those arrays we do not have values.
FOR EXAMPLE:
in a table we have COLUMN
NAME LASTNAME COMPNAME PHONO EMAIL ADDRESS ALTERPERSON ALTERPHONE ALTEREMAIL
Now after migration we see the eeror
Error Code: 1364. Field 'field' doesn't have a default value
if we run a INSERT QUERY LIKE
mysqli_query($con,'insert into table
(NAME,LASTNAME,COMPNAME,PHONO,EMAIL,ADDRESS) values
(NAME,LASTNAME,COMPNAME,PHONO,EMAIL,ADDRESS)')
now it will give error...
So just turn the table
get all the column value from DB.TABLE
put it in an array or do it like one by one using while loop or for loop....
check insert values for each column
put condition if insert value is equal to ZERO or NULL then insert ZERO it will solve all issues.
WHY ZERO --
because it will work for VARCHAR,TEXT,INT,BIGINT and in many Data Types except time or date function and DATE/TIME data type got ZERO values by default...
=============================== Another option...
run a PHP code
get all TABLE NAME
then for each TABLE NAME
get all COLUMN NAME
and run this command as in function under loop
ALTER TABLE DB.TABLEnAME CHANGE columnNAME_A columnNAME_A
VARCHAR(100) CHARACTER SET latin1 COLLATE latin1_swedish_ci NULL
DEFAULT NULL;
=======================
And its DONE
I have the following sql table:
id|email|fbid
When I perform the query
INSERT INTO users(email,fbid) VALUES('randomvalue','otherrandomvalue')
I want to get the id of the inserted row. To do so, I've tried to edit the query like this:
INSERT INTO users(email,fbid) VALUES('randomvalue','otherrandomvalue') OUTPUT Inserted.id
But I'm getting:
1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use
near 'OUTPUT Inserted.id' at line 1
What could be the problem?
Unfortunately (as far as I can tell) mysql does not support output as sql-server does.
You do have an option for what you're trying to accomplish in a single row insert (assuming auto_increment primary key):
SELECT LAST_INSERT_ID();
This unfortunately would not work in the case of a batch insert - though in your case you are not (at least not in your example), so this should be fine.
I'm going to use the process i describe below to handle the same situation with a private at home (non-enterprise) application that i wrote for personal use. I know this question is a year old right now but there doesn't seem to be an adequate answer for batch processing. I can't find an adequate answer. MySQL doesn't seem to have the facilities built into it to handle this type of thing.
I had concerns about the reliability of this solution, when put into a production environment where multiple different users/jobs could access the same procedure at the same time to do the insert. I believe I have resolved these concerns by adding the connection id to the #by variable assignment. Doing this makes it so that the by has a: the connection id for the session and b: the name of the program/job/procedure doing the insert. Combined with the date AND time of the insert, I believe these three values provide a very secure key to retrieve the correct set of inserted rows. If absolute certainty is required for this, you could possibly add a third column of a GUID type (or varchar) generate a GUID variable to insert into that, then use the GUID variable along with #by and #now as your key. I feel it's unnecessary for my purpose because the process I'm going to use it in is an event (job) script that runs on the server rather than in PHP. So I am not going to exemplify it unless someone asks for that.
WARNING
If you are doing this in PHP, consider using a GUID column in your process rather than the CreatedBy. It's important that you do that in PHP because your connection can be lost in between inserting the records and trying to retrive the IDS and your CreatedBy with the connection ID will be rendered useless. If you have a GUID that you create in PHP, however, you can loop until your connection succeeds or recover using the GUID that you saved off somewhere in a file. The need for this level of connection security is not necessary for my purposes so I will not be doing this.
The key to this solution is that CreatedBy is the connection id combined with the name of the job or procedure that is doing the insert and CreatedDate is a CURRENT_TIMESTAMP that is held inside a variable that is used through the below code. Let's say you have a table named "TestTable". It has the following structure:
Test "Insert Into" table
CREATE TABLE TestTable (
TestTableID INT NOT NULL AUTO_INCREMENT
, Name VARCHAR(100) NOT NULL
, CreatedBy VARCHAR(50) NOT NULL
, CreatedDate DATETIME NOT NULL
, PRIMARY KEY (TestTableID)
);
Temp table to store inserted ids
This temporary table will hold the primary key ids of the rows inserted into TestTable. It has a simple structure of just one field that is both the primary key of the temp table and the primary key of the inserted table (TestTable)
DROP TEMPORARY TABLE IF EXISTS tTestTablesInserted;
CREATE TEMPORARY TABLE tTestTablesInserted(
TestTableID INT NOT NULL
, PRIMARY KEY (TestTableID)
);
Variables
This is important. You need to store the CreatedBy and CreatedDate in a variable. CreatedBy is stored for consistency/coding practices, CreatedDate is very important because you are going to use this as a key to retrieve the inserted rows.
An example of what #by will look like: CONID(576) BuildTestTableData
Note that it's important to encapsulate the connection id with something that indicates what it is since it's being used as a "composite" with other information in one field
An example of what #now will look like: '2016-03-11 09:51:10'
Note that it's important to encapsulate #by with a LEFT(50) to avoid tripping a truncation error upon insert into the CreatedBy VARCHAR(50) column. I know this happens in sql server, not so sure about mysql. If mysql does not throw an exception when truncating data, a silent error could persist where you insert a truncated value into the field and then matches for the retrieval fail because you're trying to match a non-truncated version of the string to a truncated version of the string. If mysql doesn't truncate upon insert (i.e. it does not enforce type value restrictions) then this is not a real concern. I do it out of standard practice from my sql server experience.
SET #by = LEFT(CONCAT('CONID(', CONNECTION_ID(), ') BuildTestTableData'), 50);
SET #now = CURRENT_TIMESTAMP;
Insert into TestTable
Do your insert into test table, specifying a CreatedBy and CreatedDate of #by and #now
INSERT INTO TestTable (
Name
, CreatedBy
, CreatedDate
)
SELECT Name
, #by
, #now
FROM SomeDataSource
WHERE BusinessRulesMatch = 1
;
Retrieve inserted ids
Now, use #by and #now to retrieve the ids of the inserted rows in test table
INSERT INTO tTestTablesInserted (TestTableID)
SELECT TestTableID
FROM TestTable
WHERE CreatedBy = #by
AND CreatedDate = #now
;
Do whatever with retreived information
/*DO SOME STUFF HERE*/
SELECT *
FROM tTestTablesInserted tti
JOIN TestTable tt
ON tt.TestTableID = tti.TestTableID
;
if You are using php then it is better to use following code :
if ($conn->query($sql) === TRUE) {
$last_id = $conn->insert_id;
echo "New record created successfully. Last inserted ID is: " . $last_id;
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
where $conn is connection variable.
CREATE TABLE dummy (
id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
name VARCHAR( 30 ) NOT NULL
) ENGINE = MYISAM ;
and running this query:
SELECT GROUP_CONCAT(`name` SEPARATOR "||") FROM `dummy`
This query joins name column in all rows with || in a single column. BUT, the result is truncated with mysql configuration as explained in mysql manual :
"... result is truncated to the maximum length that is given by the group_concat_max_len system variable, which has a default value of 1024 ..."
Also in manual it is said that this setting can be changed in run time by the following syntax:
SET [GLOBAL | SESSION] group_concat_max_len = val;
Does changing this configuration works in all mysql server environments? If not, how can I achieve the same result without GROUP_CONCAT without limits?
Also, I think that changing the configuration will not solve my problem because I don't know what to set the value of group_concat_max_len because number of rows in dummy table can be any number.
Thanks.
Have you tried using stored procedure to accomplish your task? You can create a temporary table with a single row/column and append to it while fetching rows from your table. In the end just SELECT the single value from the temporary table.
You can find information about stored routines in mysql manual and other places.