I am using LOAD DATA INTO command to read data from a txt file into my Database. In my input, ı need to take '-' characters as null (not the string null). For example;the input
stack - overflow
have to be
1st column = stack
2nd column = null
3rd column = overflow.
How can I do that??
A simple approach would be to first use LOAD DATA without any conversions, then afterwards run some updates to correct the values that you want changing:
UPDATE yourtable
SET col1 = NULL
WHERE col1 = '-'
The LOAD DATA syntax also allows you to specify transformations to your data:
[SET col_name = expr,...]
The column list can contain either column names or user variables. With user variables, the SET clause enables you to perform transformations on their values before assigning the result to columns.
An example would be:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, #var1)
SET column2 = CASE WHEN #var1 = '-' THEN NULL ELSE #var1 END;
Related
I imported a CSV file using LOAD DATA INFILE and have this column called Context.
The entire column appears empty - which it shouldn't be, because I already set the DEFAULT to be NULL. So I expected NULL to appear in the entire column.
I have tried if it is empty or NULL but the code below returns 0
SELECT COUNT(*) FROM table
WHERE Context IS NULL OR Context = '';
So now I do not know where the problem is.
FYI: This column is in TEXT datatype and I could change it to VARCHAR, but NOT INT - I don't know if it's related to the problem.
You can manipulate the empty values while importing the data using the SET col_name={expr | DEFAULT}, ... statement while importing the data combined with the function NULLIF(expr1,expr2).
So your query may looks like this:
LOAD DATA INFILE 'file.txt'
INTO table t1
(column1, #var1)
SET Context = NULLIF(#var1, '');
Or you can manipulate all records after the import
UPDATE `table` SET Context2 = NULL WHERE Context2 IS NULL;
LOAD DATA
NULLIF
Based on this thread, I modified my query to the following and was able to solve the problem.
UPDATE table SET Context = NULLIF(Context,' ');
UPDATE table SET Context = NULLIF(Context, '\t');
UPDATE table SET Context = NULLIF(Context, '\n');
UPDATE table SET Context = NULLIF(Context, '\r');
So the original problem could be caused by either \t , \n or \r.
I have a very large database and for testing, I want to set a certain amount of data to NULL.
As an example, I have 57 columns across 3 tables, all of which need to be nullified. I can't delete the rows, I just need to know that if the row exists and there's no data in those fields, that everything still works.
To clarify, all the data in those fields has been moved to anther table, and the old data was not wiped in the migration. To test my reports I need to know that the reports are pulling from the new location, not the old, since as new data is added, it will only go to the new location. Our plan is to generate each report from the old database, migrate, and then generate them again and compare. But to ensure that they are pulling from the right place, we want to wipe the old data so it doesn't provide a false positive.
Is there a way for me to do this in bulk or should I resign myself to writing one comma separated SET statement after another?
You can create the statements using the data from the internal information_schema.COLUMNS table.
Assuming you have this table:
CREATE TABLE my_table (
keep1 INT,
keep2 INT,
set_null1 INT,
set_null2 INT,
set_null3 INT
);
and you want to set all columns to NULL except of keep1 and keep2. Execute the following script:
set #db_name = 'test';
set #table_name = 'my_table';
set #exclude_columns = 'keep1,keep2';
select concat(
'UPDATE `', #table_name, '` SET\n',
group_concat('`', COLUMN_NAME, '` = NULL' separator ',\n'),
';'
)
from information_schema.COLUMNS c
where c.TABLE_SCHEMA = #db_name
and c.TABLE_NAME = #table_name
and find_in_set(c.COLUMN_NAME, #exclude_columns) = 0;
This will generate the following statement:
UPDATE `my_table` SET
`set_null1` = NULL,
`set_null2` = NULL,
`set_null3` = NULL;
Copy the result and paste it into your UPDATE script. Do it for all 12 tables adjusting the variables #db_name, #table_name and #exclude_columns.
See demo on db-fiddle.
This is a very unusual task for an SQL database, so it's not surprising that it's a bit awkward.
As you know, to set multiple columns to NULL in an UPDATE statement, you'd have to set each column individually.
UPDATE mytable
SET col1 = NULL, col2 = NULL, ... col57 = NULL
WHERE id = ?;
That could be quite a bit of typing. Or it could be a task to write code to loop over the column names in your table, and concatenate the terms for UPDATE statement. Up to you.
An alternative that might be easier is to delete the row and then re-insert it with no values specified except the primary key.
DELETE FROM mytable WHERE id = ?;
INSERT INTO mytable SET id = ?;
By omitting the other columns, they'll be NULL or else take a DEFAULT value defined in your table. If you want those columns with defaults to be NULL too, you'll have to specify that.
INSERT INTO mytable SET id = ?, col23 = NULL;
I have created a table in mysql called person and am trying to load data into it using a text file. However, I have a restriction, I must load the data while checking if the SSN should be trimmed to an appropriate amount of characters. If the value is empty I also want to set SSN to NULL.
However, none of the commands I use want to work. For example, I have no idea why the line below isn't working.
LOAD DATA LOCAL INFILE '/home/ugrad19/jflickin/cs482/PERSON.txt' INTO TABLE person
FIELDS TERMINATED BY ','
IGNORE 1 ROWS
(PIDM, FNAME, LNAME, #SSN, #DATE, GENDER)
SET SSN = LEFT(TRIM(#SSN), 8) --this line messes it up.
SET DOB = STR_TO_DATE(#DATE, '%m/%d/%Y');
Does anyone have any idea what I'm doing wrong?
Thanks!!
Please note: I set the data type of SSN to Varchar(10). Please tell me if this is not the most appropriate datatype for this field.
The SET keyword appears once, multiple col = expr are separated by commas. For example:
SET SSN = LEFT(TRIM(#SSN), 9)
, DOB = STR_TO_DATE(#DATE, '%m/%d/%Y');
^
And social security numbers are generally nine digit characters, not eight.
SET SSN = CASE WHEN #SSN = '' THEN NULL ELSE LEFT(TRIM(#SSN), 8) END
I have CSV file having two columns id_a and id_b, but I need to insert 4 more columns; ie. emp_sal_a, emp_sal_b, emp_dept_a, emp_dept_b using sqlldr. So my current control file looks like:
load data
infile '/home/.../employee.txt'
into table employee
fields terminated by ","
( id_a, id_b,
emp_sal_a ":id_a+1000", emp_sal_b "id_b+1000", emp_dept_a "10", emp_dept_b "20")
But I am getting error:
invalid binding variables
From MySQL Load Data Ref
note: search for the "(" character and it's the 35th instance of it on the page
User variables in the SET clause can be used in several ways. The following example uses the first input column directly for the value of t1.column1, and assigns the second input column to a user variable that is subjected to a division operation before being used for the value of t1.column2:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, #var1)
SET column2 = #var1/100;
#var1 is the name of a variable you want to run an operation on, and what you're doing is calling SET on column2 to be equal to #var1/100.
Say, you got a table of 100 records. And field age contains a some integers. And you want all those integers to be incremented by 1.
or you got a textfield called name and a bunch of names in there. And you want all those names to be prefixed as Mr..
Is there a way to achieve this in one SQL command?
The alternative would be to compile a recordset of these 100 recs and going thru a loop and then running an individual update statement.
Use the update command
update yourtable
set age=age +1
update yourtable
set name = 'Mr. ' + name
where gender='M'
UPDATE mytable SET age = age+1
UPDATE mytable SET name = CONCAT('Mr. ', name)
If MySQL is in ANSI mode – specifically, PIPES_AS_CONCAT, you can use 'Mr. ' || name instead.