Is it possible to LOAD DATA a csv into mysql without having to add empty values for non existing columns at the end?
All my optional columns are sorted at the end of the schema:
CREATE TABLE `person` (
id int(20) NOT NULL AUTO_INCREMENT,
firstname varchar(30) NOT NULL,
lastname varchar(30) NOT NULL,
optional1 varchar DEFAULT NULL,
optional... varchar DEFAULT NULL,
optional50 varchar DEFAULT NULL,
PRIMARY KEY (`id`)
) engine=innodb AUTO_INCREMENT=0;
sample.csv:
1;john;doe
2;jabe;doe;;;opt val3;;;;;;opt val9;;;;;;...
Important: I don't want to explicit list all the columns in my LOAD DATA INFILE sql statement (I know that this would work by using a combination of IFNULL and #var).
But can't I just load into the table, telling mysql to ignore any missing fields at the end of each line?
The documentation of MySQL LOAD DATA syntax provides the following information:
By default, when no column list is provided at the end of the LOAD DATA statement, input lines are expected to contain a field for each table column. If you want to load only some of a table's columns, specify a column list.
[...]
If an input line has too few fields, the table columns for which input fields are missing are set to their default values. For numeric types, the column is set to 0.
[...]
An empty field value is interpreted different from a missing field: for string types, the column is set to the empty string.
So given your sample data:
1;john;doe
2;jabe;doe;;;opt val3;;;;;;opt val9;;;;;;...
Record with id 1 will have all optional columns set to NULL (ie their default). For id2, optional string columns will be set to the empty string. .
I cannot tell whether this would be OK for your use case or not. If you do want consistent values in the optional columns, available options would be:
input pre-processing: use SET to set to NULL columns that contains an empty string
LOAD DATA INFILE 'file.txt' INTO TABLE t1
SET
optional1 = NULLIF(optional1, ''),
optional2 = NULLIF(optional1, ''),
...
set up a BEFORE INSERT trigger on the table that sets to NULL empty values
run an update on the table after it was populated
UPDATE t1 SET optional1 = NULLIF(option1, ''), optional2 = NULLIF(optional1, '')
WHERE '' IN (optional1, optional2, ...)
I found out it works as expected if adding the IGNORE keyword to the LOAD DATA statement:
LOAD DATA INFILE 'sample.csv' IGNORE INTO TABLE persons
Thereby, I can define all my optional columns as DEFAULT NULL, and if values are missing, they are set to NULL during import.
Related
I have a mysql database with a TINYINT(4) column that previously had no value set to default, and I assume null was being set as the default value. I had to manually push '0' into the database saving logic every time in my code, so I have now updated the column to instead have an explicit default value of '0'. Now when I save my data, instead null is being set, and throwing off my code. When I set not null for the column, then it throws an error due to my not passing in a value in my database saving code. I've checked all of my other entries in the table, and do not see any other 'null' values that may be throwing this off - all are either '0' or '1'.
There must be some quirk here with my database. I have the same exact column with the same logic and that is saving as '0' perfectly fine. Looking for any insight into what could be causing this.
Are you using a framework where objects are automatically converted into SQL statements for saving? If the value of the property in your PHP class is not set and the column is nullable then it will insert null instead of 0.
Consider:
class Foo{
protected $propertyName;
}
That is equivalent in PHP to
class Foo{
protected $propertyName = null;
}
If the TINYINT(4) column for propertyName is nullable when it builds the query to save the data it will save as null in the database. If you are using a design pattern like this you need set the default value in the PHP class itself. Something like
class Foo{
protected $propertyName = 0;
}
Note, if the column is not null-able then saving the object would throw an error in this scenario. If you wanted to get fancy, you can fetch the default values for a column using
DESC tableName;
That will return information about the table, there will be a column NULL which will be YES or NO (describing if the column is null-able). And a column Default which will be the default value (or NULL if there is none). You could then populate null fields in your class based on the default values from the database. You would want to be careful here as there are likely some columns that should be NULL. Ideally, you would make any fields that can't accept null non-nullable and then key your logic for handling defaults only fire when the Null column is NO.
When inserting a new row, the default value for a column with an expression default can be inserted either by omitting the column name or by specifying the column as DEFAULT (just as for columns with literal defaults).
source: https://dev.mysql.com/doc/refman/8.0/en/data-type-defaults.html
In case you specify NULL for the column on INSERT MySQL try to set the NULL for the column (and fails in case of NOT NULL). You have to ommit the column on INSERT or using DEFAULT as value on the INSERT statement.
See the following example:
CREATE TABLE test (
col1 INT DEFAULT 0 NOT NULL,
col2 INT DEFAULT 0 NULL,
col3 INT NULL
);
-- doesn't work since col1 can't be NULL
INSERT INTO test VALUES (NULL, NULL, NULL)
-- is working: col1 is 0 after INSERT, col2 IS NULL
INSERT INTO test (col2, col3) VALUES (NULL, NULL)
-- is working: col1 is 0 after INSERT, col2 IS 0 - because using DEFAULT instead of NULL.
INSERT INTO test (col2, col3) VALUES (DEFAULT, NULL)
In phpMyAdmin, as we create table there is not null constraints by default for all fields...and as per my knowledge when we set the constraint to not null...it doesn't allow user to remain field empty which are not null as per this link.....
http://www.techopedia.com/definition/27370/not-null-constraint
now my question is..according to this link, not null means every row of data must contain a value - it cannot be left blank during insert or update operations.....but when i insert data programatically like insert into, i am able to insert data in just two fields and other remains blank although there is not null constraints on that fields ...and still not generates any error....so i don't understand how not null works???
for example, i create table with lets say 5 fields...
create table myTable
(
Column1 int not null,
Column2 int not null,
Column3 int not null,
Column4 int not null,
Column5 int not null,
)
and insert values in just two fields like
"INSERT INTO myTable (column1,column2) VALUES(10,20)";
but other fields i don't give any '' so it takes 0 as value....and still i am able to insert data with no error...how is that possible??
Everything that has the NOT NULL constraint set needs to contain data. If you insert data programmatically and you do not insert data for a NOT NULL cell, then you will get an SQL Error.
e.g. you have this table:
CREATE TABLE test (
id INTEGER PRIMARY_KEY AUTO_INCREMENT,
some_value INTEGER NOT NULL,
some_other_value INTEGER);
Then some_value will contain data in every data set returned, some_other_value may or may not contain data in every data set returned. The only thing to work around this would be this:
CREATE TABLE test (
id INTEGER PRIMARY_KEY AUTO_INCREMENT,
some_value INTEGER NOT NULL DEFAULT 0,
some_other_value INTEGER);
If you now set data programatically and do not set data for some_value, some_value will default to 0 (or to whatever data you set the default to on table creation).
Maybe you can refer to this link:
For multiple-row INSERT statements or INSERT INTO ... SELECT
statements, the column is set to the implicit default value for the
column data type. This is 0 for numeric types, the empty string ('')
for string types, and the “zero” value for date and time types. INSERT
INTO ... SELECT statements are handled the same way as multiple-row
inserts because the server does not examine the result set from the
SELECT to see whether it returns a single row. (For a single-row
INSERT, no warning occurs when NULL is inserted into a NOT NULL
column. Instead, the statement fails with an error.)
If a column definition includes no explicit DEFAULT value and it is defined as "Not Null" then Mysql will automatically assign default value to the column based on datatype. e.g. 0 for integer and "" for varchar
If you create a unique index on a column, the default value will be accepted with the first row but will give an error with subsequent inserts.
I set default value in a field NOT NULL.
The sql is as follows.
insert into table_name () values ()
I expected this sql should be rejected by MySQL as a field restricts NULL.
But, the field had a value ""(null character string).
Then, I tried another sql.
insert into table_name (name) values (NULL)
As this sql was rejected, no value was inserted.
MySQL seems to distinguish NULL and "". Do I have to avoid first sql such as "values ()"?
For a good practice always make the column as not null and set a default value whenever you create a table.
At insert time if you do not provide a value, the column will then be assigned the default value.
Do I have to set default value in a field that is set as NOT NULL?
YES if you are doing an INSERT where in you will not set any values to the NOT NULL columns
and
NO not needed anymore, if you are providing a value on the columns that are NOT NULL on INSERT.
You can refer to this
http://dev.mysql.com/doc/refman/5.0/en/data-type-defaults.html
I hope this isn't a dumb question. You can set a default value for all variables or a function for when it is inserted. but if the field is not required to insert and you don't allow null values, what is the "blank" value that you see in phpMyAdmin? in a query is it returned as empty string, etc?
just trying to figure it out, I want to query for all records such that the value for a specific column in that record is not "empty" or blank or whatever.
thanks.
Referring to the manual,
For data entry for a NOT NULL column that has no explicit DEFAULT
clause, if an INSERT or REPLACE statement includes no value for the
column, or an UPDATE statement sets the column to NULL, MySQL handles
the column according to the SQL mode in effect at the time:
If strict SQL mode is not enabled, MySQL sets the column to the implicit default value for the column data type.
If strict mode is enabled, an error occurs for transactional tables and the statement is rolled back. For nontransactional tables, an
error occurs, but if this happens for the second or subsequent row of
a multiple-row statement, the preceding rows will have been inserted.
So your question now may be, what are the implicit default values for the various column data types? Here you go:
Implicit defaults are defined as follows:
For numeric types, the default is 0, with the exception that for integer or floating-point types declared with the AUTO_INCREMENT
attribute, the default is the next value in the sequence.
For date and time types other than TIMESTAMP, the default is the appropriate “zero” value for the type. For the first TIMESTAMP column
in a table, the default value is the current date and time. See Section 10.3, “Date and Time Types”.
For string types other than ENUM, the default value is the empty string. For ENUM, the default is the first enumeration value.
There IS no default value unless you specify one (i.e. unless you define a "default constraint" for the column in question).
Here's an example for adding a default on an existing column:
ALTER TABLE dbo.customer ALTER COLUMN contactname SET DEFAULT 'Unknown'
Here's an example creating the table with a default:
CREATE TABLE Books (
ID SMALLINT NOT NULL,
Name VARCHAR(40) NOT NULL,
PubID SMALLINT NOT NULL DEFAULT 0
)
It's good practice to declare ALL columns "not null", and provide default constraints as appropriate.
In the "books" example above, if you "insert" without specifying PubID, the PubID will be zero.
In the same example, if you "insert" without specifying ID or Name ... you'll get an error.
If you want MySQL to auto-assign an ID, use this syntax instead:
CREATE TABLE Books (
ID SMALLINT NOT NULL AUTO_INCREMENT,
Name VARCHAR(40) NOT NULL,
PubID SMALLINT NOT NULL DEFAULT 0
)
If you want to disallow null :-
alter table YOUR_TABLE modify column COLUMN varchar(255) not null default '';
The above query will disallow null and assign an empty string when the value is not supplied.
In phpmysqladmin, blank = empty.
Via PHP mysqli function or mysql function, null value is returned as null still.
Once you have apply the query, you can easily filter that by using
select ... from YOUR_TABLE
where COLUMN != ""; <-- no need to check is null
<-- because the first query already enforce not null
However, is best for you do this before perform the alter :-
update YOUR_TABLE set COLUMN = ""
where COLUMN is null;
I have a php form with three text boxes (webmeasurementsuiteId, webmeasurementsId, Id) and the values in the text boxes are retrieved from other tables of the database. Now my task is to submit the retrieved values in this php form named (mapping) to the database. I have created the table with the following syntax:
CREATE TABLE `mapping` (
`webmeasurementsuiteId` INT NOT NULL,
`webmeasurementsId` INT NOT NULL,
`Id` INT NOT NULL,
PRIMARY KEY (Id)
);
But I am getting an sql error as follows:
INSERT INTO mapping(webmeasurementsuiteId,webmeasurementsId,Id) values ('','','7')
ERROR: Incorrect integer value: '' for column 'webmeasurementsuiteId' at row 1
Can anyone correct my error?
all your columns are INT that means numbers while your insert statement is inserting STRINGs (text) remove the ' around the values in the INSERT-statement and it should work
example:
INSERT INTO mapping(webmeasurementsuiteId,webmeasurementsId,Id) values (0,0,7)
So you used '' for webmeasurementsuiteId which indicates it as a string. just leave the integers to 0 without the parantheses to indicate them as an integer. values (0,0,7) should probably do it.
you cannot insert blank in the values field .
try 0 instead of ''
INSERT INTO mapping(webmeasurementsuiteId,webmeasurementsId,Id) values (0,0,'7')
or alter the columns to take the null values