mysql> select * from friends limit 10;
+----+------------------------+-------------------------+
| id | user | fri |
+----+------------------------+-------------------------+
|MHL_bKFgQ | NULL
| 2 | MWhR9LvOdRbqtu1I_DRFBg | 8Y2EN4XNNhnwssuPb31sJg
| 3 | MWhR9LvOdRbqtu1I_DRFBg | A1jPleJ99kXZ3t9wQ3np-g
| 4 | MWhR9LvOdRbqtu1I_DRFBg | resYiOoGkQg6q0qgtj_1GA
| 5 | MWhR9LvOdRbqtu1I_DRFBg | skl1OnkjMqD4GdFpVhU88Q
| 6 | MWhR9LvOdRbqtu1I_DRFBg | 3Ss2aqrSoO7WbxE2GcLlfQ
| 7 | MWhR9LvOdRbqtu1I_DRFBg | SGy3JDbhtzDTTBO7unqQxg
| 8 | MWhR9LvOdRbqtu1I_DRFBg | Aez4Y1F0m2ucIfmfzPZfjw
| 9 | MWhR9LvOdRbqtu1I_DRFBg | fT4jx_9cRWyKFROxv_MIrw
|10 | MWhR9LvOdRbqtu1I_DRFBg | QmZOAYM7ITbTdkTn6ay_ag
+----+------------------------+-------------------------+
10 rows in set (0.00 sec)
Command:
mysql> select * from friends where fri="3Ss2aqrSoO7WbxE2GcLlfQ";
Result:
Empty set (0.37 sec)
This is beyond my understanding. There is a valid data in the above table but when i try to retrieve it, it shows no rows.
Details of the table:
CREATE TABLE friends (
id INT NOT NULL,
user varchar(100) NOT NULL ,
fri varchar(100) ,
PRIMARY KEY (id)
)ENGINE=MyISAM;
I got the issue. I have loaded my file with following command:
LOAD DATA LOCAL INFILE 'file.txt' INTO TABLE friends fields terminated by ',' lines terminated by '\n';
But the last column has ending '\r\n'. So, \r was a garbage which was invisible. When i clean the file by removing \r using python, the SQL works as expected. Thank you guys for all your support.
Related
I have two tables, one that holds potential items, the other holds completed items.
The potential item table currently contains the records that have also been added to the completed items table. I want to remove (either by deleting or selecting new results) the already completed items from the list of potential items.
In both tables, items may appear multiple times, and I only want to remove the number of items that are completed, not all that match.
The real data set is more larger of course, but here are samples.
Potential items:
mysql> select * from stack;
+----------+------+------+
| stack_id | type | name |
+----------+------+------+
| 3 | a | aa |
| 4 | b | bb |
| 5 | c | cc |
| 6 | d | dd |
| 7 | a | aa |
| 8 | b | bb |
+----------+------+------+
6 rows in set (0.00 sec)
Completed items
mysql> select * from temp;
+----------+------+------+
| item_id | type | name |
+----------+------+------+
| 1 | a | aa |
| 2 | b | bb |
| 6 | b | bb |
+----------+------+------+
3 rows in set (0.00 sec)
The IDs between tables do not correlate, so they should be ignored as far as finding matches.
I want to omit 1 instance of a/aa and 2 of b/bb since those have been completed and exist in the other table.
when I try this:
mysql> select stack.* from stack where (type,name) not in (select type,name from temp);
I get this:
+----------+------+------+
| stack_id | type | name |
+----------+------+------+
| 5 | c | cc |
| 6 | d | dd |
+----------+------+------+
2 rows in set (0.03 sec)
But this omitted both instances of type="a" and name="aa" and I want to only omit one of them (since it only exists once in the completed items table)
How do I get this?
+----------+------+------+
| stack_id | type | name |
+----------+------+------+
| 5 | c | cc |
| 6 | d | dd |
| 7 | a | aa |
+----------+------+------+
I don't care which instance of a/aa is deleted (whether id=7 or id=3)
The best I've been able to come up with is to use PHP rather than MySQL to loop through each record in temp and delete with a LIMIT 1 from stack.
But I'd rather not have to run code for this, I'd like to do it in queries, it works better that way in my workflow
Thanks!
CREATE TABLE `test`.`stack` (
`stack_id` INT UNSIGNED NOT NULL,
`type` VARCHAR(45) NULL,
`name` VARCHAR(45) NULL,
PRIMARY KEY (`stack_id`));
CREATE TABLE `test`.`temp` (
`item_id` INT UNSIGNED NOT NULL,
`type` VARCHAR(45) NULL,
`name` VARCHAR(45) NULL,
PRIMARY KEY (`item_id`));
and than something like this:
select
min(stack_id), type, name
from stack _s
inner join
(
select min(item_id) item_id, type, name
from temp
group by type, name
) _t using(type, name)
group by _s.type, _s.name
will give you only one the first item in temp:
stack_id
type
name
3
a
aa
4
b
bb
When I'm trying to import csv into MySQL table, I'm getting an error
Data too long for column 'incident' at row 1
I'm sure the values are not higher than varchar(12). But, still I'm getting the error.
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
REPLACE INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
While trying with REPLACE, the data is uploading only one column(which set on primary key)
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
IGNORE INTO TABLE incidents
ignore 1 lines;
Query OK, 246 rows affected, 1968 warnings (0.015 sec)
Records: 246 Deleted: 0 Skipped: 0 Warnings: 1968
**Columns:**
+----------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| incident | varchar(12) | NO | PRI | NULL | |
| description | varchar(300) | YES | | NULL | |
| status | varchar(12) | YES | | NULL | |
| urgency | varchar(7) | YES | | NULL | |
| service | varchar(27) | YES | | NULL | |
| trigger | varchar(25) | YES | | NULL | |
| team | varchar(20) | YES | | NULL | |
| incident_start | datetime(6) | YES | | NULL | |
| incident_end | datetime(6) | YES | | NULL | |
| resolved_by | varchar(20) | YES | | NULL | |
+----------------+--------------+------+-----+---------+-------+
10 rows in set (0.003 sec)
By default, MySQL looks for a TAB character to separate values. Your file is using a comma, so MySQL reads the entire line and assumes it is the value for the first column only.
You need to tell MySQL that the column terminator is a comma, and while you're at it, tell it about the enclosing double quotes.
Try this:
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv' REPLACE INTO TABLE incidents
columns terminated by ','
optionally enclosed by '"'
ignore 1 lines;
Reference
If you THINK your data appears ok, and its still nagging about the too long data, how about creating a new temporary table structure and set your first column incident to a varchar( 100 ) just for grins... maybe even a few others if they too might be causing a problem.
Import the data to THAT table to see if same error or not.
If no error, then check the maximum trimmed length of the data in respective columns and analyze the data itself... bad format, longer than expected, etc.
Once resolved, then you can pull into production once you have figured it out. You could also always PRE-LOAD the data into this larger table structure, truncate it before each load so no dups via primary key on a fresh load.
I have had to do that in the past, also was efficient for pre-qualifying lookup table IDs for new incoming data. Additionally could apply data cleansing in the temp table before pulling into production.
In the other question, I asked about why non-empty values become NULL. But in this question, I asked about how I could make empty values become NULL but not zero.
I found that missing float values are always represented as 0, but not NULL. How could I change that?
The following are the codes with which I created the table and loaded data.
CREATE TABLE Products(sku INTEGER, name VARCHAR(255), description TEXT,
regularPrice FLOAT,
customerReviewAverage FLOAT default NULL );
LOAD DATA LOCAL INFILE 'product.csv'
INTO TABLE Products
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(sku, name, #description, regularPrice, customerReviewAverage)
SET description = IF(#description='',NULL,#description);
This is a sample of data in product.csv.
19658847,Glanzlichter - CD,,12.99,5.0
19658856,Glanzlichter - CD,,6.99,
19658865,Glanzlichter - CD,,8.99,
1965886,Beach Boys '69 - CASSETTE,,6.99,4.5
The issue is how MySQL interprets an empty field vs a missing field. From the docs for LOAD DATA INFILE...
If an input line has too few fields, the table columns for which input fields are missing are set to their default values.
An empty field value is interpreted different from a missing field. For string types, the column is set to the empty string. For numeric types, the column is set to 0.
In this case it seems MySQL considers it to be empty. You can see this from show warnings.
mysql> show warnings;
+---------+------+------------------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------------------------+
| Warning | 1265 | Data truncated for column 'customerReviewAverage' at row 2 |
| Warning | 1265 | Data truncated for column 'customerReviewAverage' at row 3 |
+---------+------+------------------------------------------------------------+
2 rows in set (0.00 sec)
Whereas if we remove the trailing commas so the field is missing...
19658847,Glanzlichter - CD,,12.99,5.0
19658856,Glanzlichter - CD,,6.99
19658865,Glanzlichter - CD,,8.99
1965886,Beach Boys '69 - CASSETTE,,6.99,4.5
Then the data is set to null.
mysql> show warnings;
+---------+------+--------------------------------------------+
| Level | Code | Message |
+---------+------+--------------------------------------------+
| Warning | 1261 | Row 2 doesn't contain data for all columns |
| Warning | 1261 | Row 3 doesn't contain data for all columns |
+---------+------+--------------------------------------------+
mysql> select * from products;
+----------+---------------------------+-------------+--------------+-----------------------+
| sku | name | description | regularPrice | customerReviewAverage |
+----------+---------------------------+-------------+--------------+-----------------------+
| 19658847 | Glanzlichter - CD | NULL | 12.99 | 5 |
| 19658856 | Glanzlichter - CD | NULL | 6.99 | NULL |
| 19658865 | Glanzlichter - CD | NULL | 8.99 | NULL |
| 1965886 | Beach Boys '69 - CASSETTE | NULL | 6.99 | 4.5 |
+----------+---------------------------+-------------+--------------+-----------------------+
4 rows in set (0.00 sec)
Doing the same thing for #customerReviewAverage as for #description worked for me in MySQL 5.7.
LOAD DATA
LOCAL INFILE 'product.csv'
INTO TABLE Products
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(sku, name, #description, regularPrice, #customerReviewAverage)
SET description = IF(#description='',NULL,#description),
customerReviewAverage = IF(#customerReviewAverage='',NULL,#customerReviewAverage);
mysql> select * from products;
+----------+---------------------------+-------------+--------------+-----------------------+
| sku | name | description | regularPrice | customerReviewAverage |
+----------+---------------------------+-------------+--------------+-----------------------+
| 19658847 | Glanzlichter - CD | NULL | 12.99 | 5 |
| 19658856 | Glanzlichter - CD | NULL | 6.99 | NULL |
| 19658865 | Glanzlichter - CD | NULL | 8.99 | NULL |
| 1965886 | Beach Boys '69 - CASSETTE | NULL | 6.99 | 4.5 |
+----------+---------------------------+-------------+--------------+-----------------------+
4 rows in set (0.00 sec)
I want to import data from excel to a mysql database using the command line client.
This is an example of how my csv-file is built:
Name 1 | 1 | 2 | 3 |
Name 2 | 1 | 2 | 3 |
Name 3 | 1 | 2 | 3 |
I'm using the code:
LOAD DATA LOCAL INFILE 'path to file.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
I get the "Query OK" and with this code the formatting on the table should be almost exactly as the csv-file but I get this result:
| NULL | NULL | NULL |
| NULL | NULL | NULL |
| NULL | NULL | NULL |
What is wrong?
seems u have used '|' in your csv file as delimiters instead of
comma, try the code as
`LOAD DATA LOCAL INFILE 'path to file.csv' INTO TABLE table_name FIELDS TERMINATED BY '|'LINES TERMINATED BY '\n';`
Assume the following table structure:
CREATE TABLE `table_name` (
`name` VARCHAR(20) CHARACTER SET utf8 DEFAULT NULL,
`value1` INT(11) DEFAULT NULL,
`value2` INT(11) DEFAULT NULL,
`value3` INT(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and the file, csv-file.csv:
Name 1,1,2,3
Name 2,1,2,3
Name 3,1,2,3
when I run the statement:
mysql> LOAD DATA INFILE '/path/csv-file.csv'
-> INTO TABLE `table_name`
-> FIELDS TERMINATED BY ','
-> LINES TERMINATED BY '\n';
mysql> SELECT `name`, `value1`, `value2`, `value3`
FROM `table_name`;
get the following result:
+--------+--------+--------+--------+
| name | value1 | value2 | value3 |
+--------+--------+--------+--------+
| Name 1 | 1 | 2 | 3 |
| Name 2 | 1 | 2 | 3 |
| Name 3 | 1 | 2 | 3 |
+--------+--------+--------+--------+
3 rows in set (0.00 sec)
For a preferences module I have "system defaults", and "user preferences".
If there is no personal/user preference stored, then use the system default values instead.
Here is my system preferences table:
mysql> desc rbl;
+-------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+-------+
| id | varchar(3) | NO | PRI | | |
| rbl_url | varchar(100) | NO | | | |
| description | varchar(100) | NO | | | |
| is_default | tinyint(1) unsigned | YES | | 1 | |
+-------------+---------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
Example data from system prefs:
mysql> select * from rbl;
+----+----------------------+------------------------------+------------+
| id | rbl_url | description | is_default |
+----+----------------------+------------------------------+------------+
| 1 | sbl-xbl.spamhaus.org | Spamhaus SBL-XBL | 1 |
| 2 | pbl.spamhaus.org | Spamhaus PBL | 1 |
| 3 | bl.spamcop.net | Spamcop Blacklist | 1 |
| 4 | rbl.example.com | Example RBL - not functional | 0 |
+----+----------------------+------------------------------+------------+
... and Query for system defaults:
mysql> SELECT rbl_url FROM rbl WHERE is_default='1';
+----------------------+
| rbl_url |
+----------------------+
| sbl-xbl.spamhaus.org |
| pbl.spamhaus.org |
| bl.spamcop.net |
+----------------------+
3 rows in set (0.01 sec)
So far so good.
OK. Now I need a user preferences table, and I came up with this:
mysql> desc rbl_pref;
+-----------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| domain_id | mediumint(8) unsigned | NO | | NULL | |
| rbl_id | tinyint(1) unsigned | NO | | NULL | |
+-----------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
(FYI - A "user" is represented by "domain_id". )
Let's view the preferences of a specific user who has personalized preferences saved:
mysql> select * from rbl_pref where domain_id='2277';
+----+-----------+--------+
| id | domain_id | rbl_id |
+----+-----------+--------+
| 4 | 2277 | 1 |
| 5 | 2277 | 2 |
| 6 | 2277 | 4 |
+----+-----------+--------+
3 rows in set (0.00 sec)
... again, but in a simpler format:
mysql> SELECT rbl.rbl_url FROM rbl_pref,rbl
WHERE rbl_pref.rbl_id=rbl.id AND domain_id='2277';
+----------------------+
| rbl_url |
+----------------------+
| sbl-xbl.spamhaus.org |
| pbl.spamhaus.org |
| rbl.example.com |
+----------------------+
3 rows in set (0.00 sec)
.. so far so good. If a user has stored a preference, a result is found.
The problem example now is, user 1999 has no custom preferences.
In place of the "Empty set" result, I want the system defaults.
mysql> SELECT rbl.rbl_url FROM rbl_pref,rbl
WHERE rbl_pref.rbl_id=rbl.id AND domain_id='1999';
Empty set (0.00 sec)
I was excited to find a very similar question:
mysql if row doesn't exist, grab default value
However after a couple of days trial and error and documentation review, I could not translate that answer over to here.
Like the above question, this must be done as a single MySQL query. I am not actually making this query from PHP, but from Exim macros (and it is a very picky language... best to feed it "one liners" as variable assignments, as I try to do here.. )
UPDATE: Tried one type of a UNION query suggested by #Biff McGriff, below. The table did not display in my comment reply, so here it is again:
mysql> SELECT rbl.rbl_url FROM rbl_pref,rbl
WHERE rbl_pref.rbl_id=rbl.id AND domain_id='2277'
UNION SELECT rbl_url FROM rbl WHERE is_default='1';
+----------------------+
| rbl_url |
+----------------------+
| sbl-xbl.spamhaus.org |
| pbl.spamhaus.org |
| rbl.example.com |
| bl.spamcop.net |
+----------------------+
4 rows in set (0.00 sec)
As you can see above, user 2277 did not opt in to rbl_id 3 (bl.spamcop.net), but that's showing up anyways.
What my UNION query seems to be doing is combining the result set. So user_pref acts as "in addition to" global defaults, and I was assuming/expecting I would get a result set matching either half of the query.
So my question now is, is it better (or possible, how) to solve this as "either result set" (either subquery on either side of the UNION)? OR do I really need a new field on rbl_pref, called for example "enabled". The latter seems to be more correct - that I need something in rbl_pref to explicitly designate opt-in or opt-out (other than the implicit "that pref is not here - no rbl_id=3 - in the over ridden user result SET")
UPDATE: All set, thanks #Imre L, and everyone else. I learned something through this example.
You should be able to use a left join and then coalesce the user's field with the default field.
NOTE: you have to enter the domain_id in two places.
SELECT rbl.rbl_url FROM rbl
JOIN rbl_pref ON rbl_pref.rbl_id=rbl.id AND domain_id=2277
UNION
SELECT rbl.rbl_url FROM rbl
WHERE rbl.is_default
AND NOT EXISTS (SELECT 1 FROM rbl_pref WHERE domain_id=2277 LIMIT 1)
;
Now one or the other side of UNION will be optimized away with impossible where
You also should not use varchar(3) for rbl.id but some sort of integer
and preferable same type as rbl_pref.rbl_id for which tinyint is too tiny
and when you compare integers fields in sql code domain_id='2277' you should not use ' or " around constants integers.
You can get away whith it mostly but sometimes it may confuse mysql optimizer.
Also for optimal performance and consistency i suggest you the add the index:
ALTER TABLE rbl_pref
ADD UNIQUE INDEX ux_domain_rbl (domain_id, rbl_id);