I'm dealing with this problem in my MYSQL database for several hours now. I work with OS X 10.8.4 and use the tool Sequel Pro to work with my database. The table I have troubles with looks like this:
CREATE TABLE `Descriptions` (
`id` int(11) unsigned zerofill NOT NULL AUTO_INCREMENT,
`company` varchar(200) DEFAULT NULL,
`overview` mediumtext,
`trade` mediumtext,
PRIMARY KEY (`id`))
ENGINE=InnoDB AUTO_INCREMENT=1703911 DEFAULT CHARSET=utf8;
I imported a csv file like this:
LOAD DATA LOCAL INFILE 'users/marc/desktop/descriptions kopie.txt'
INTO TABLE descriptions
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
(#dummy, company, overview, trade)
When I look at the data in my table now, everything looks the way I expect (SELECT * Syntax). But I can't work with the data. When I try to select the company 'SISTERS', from which I know that it exists, it gives me no results. Also the fields "overview" and "trade" are not NULL when there's no data, it is just an empty string. The other tables in the database works just fine with the imported data. Somehow MySQL just doesn't see the values as something to work with, it doesn't bothers to read them.
What I tried so far:
- I used text wrangler to convert the csv to txt (utf-8) and loadet it into the database, did not work
- I changed the fields into BLOB and back to varchar/mediumtext to force mysql to do something with the data, did not work
- I tried to use the Sequel Pro Import function, did not change anything
- I tried to make a new table and copy the old one into it, did not change anything
- I tried to force mysql to change the values by using the concat syntax (just adding random variables which I could delete later again)
Could it have something to do with the collation settings? Could it has something to do with my regional settings (Switzerland) on my OS X) Any other ideas? I would appreciate any help very much.
Kind Regards,
Marc
I could solve the problems. When I opened the csv in Text Wrangler and let the invisible characters show, it was full of red reversed question marks. Those sneaky bastards, they messed up everything. I don't now what they are, but they were the problem. I removed them with the "Zap Gremlins..." option.
Related
I am currently migrating data from MySQL 5.6.41 on Windows, to MySQL 8.0.21 on Windows. Overall, a pretty smooth migration, with a couple of very frustrating hiccups. There's one table that looks like this:
CREATE TABLE `domains` (
`intDomainID` int(11) NOT NULL AUTO_INCREMENT,
`txtDomain` varchar(100) NOT NULL,
`dtDateTime` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`blnTor` int(1) NOT NULL DEFAULT '0',
`txtTLD` varchar(100) NOT NULL,
PRIMARY KEY (`intDomainID`),
UNIQUE KEY `txtDomain` (`txtDomain`)
ENGINE=InnoDB AUTO_INCREMENT=10127897 DEFAULT CHARSET=utf8mb4;
The CREATE SCHEMA is complete, and was created by Workbench's "Copy to Clipboard" --> "Create Schema" function.
When I used the built in Workbench export/import, the import always failed with "Duplicate value in txtDomain" (paraphrasing here) error, which is weird because the original table has a UNIQUE KEY constraint on that field, so there cannot be duplicates, and I confirmed, the values it was finding as duplicates were NOT duplicates in the original database.
I then dumped the table using SELECT ... INTO OUTFILE, moved the file over to the new server, and did a LOAD DATE INFILE. This also failed with the same "Duplicate value in txtDomain" error.
I then removed the UNIQUE KEY constraint, and redid the LOAD DATE INFILE. This worked, the data is there. However, I cannot add back the UNIQUE KEY constraint due to "duplicates". I investigated and found this:
Query result on MySQL 5.6.41:
Query result on MySQL 8.0.21:
Now, what is going on? The table definition, the database, table and field charset/collations are identical. I need that UNIQUE KEY constraint back...
Why is http://deepdot35wvmeyd5.onion:80 == http://dee-p---dot35w-vmeyd5.onion:80 ??
In case it helps, my export command:
SELECT * INTO OUTFILE 'S:\\temp\\domains.txt'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
FROM domains;
My import command:
LOAD DATA INFILE 'E:\\DB Backup\\ServerMigration\\domains.txt'
INTO TABLE domains
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
COLLATIONS:
Collations Old Server: utf8_general_ci [I don't remember touching this value]
New Server: utf8mb4_0900_ai_ci [I didn't touch this value]
DB old/new are the same: utf8mb4_0900_ai_ci
Table old/new are the same: utf8mb4_0900_ai_ci
This is how the raw TXT file looks like on the file-system:
Note, if I paste in one of the URLs from the screenshot into here, it magically turns into the "correct" value, without the dashes:
i.e.: http://deepdot35wvmeyd5.onion:80
Note2: Using Notepad++, if I convert a regular "dash" to HEX I get "2D". However, if I convert one from the URLs that's causing trouble, I get HEX "C2AD". So it seems that I'm dealing with a weird unicode character and not a dash?
Note3: If anyone wants a small sample file, here it is:
https://www.dropbox.com/s/1ssbl95t2jgn2xy/domains_small.zip
The character in question is U+00AD "SOFT HYPHEN" - a non-printable character that is used to signal a possible hyphenation point inside a word.
It seems that the COLLATION used handles these characters differently on the new setup (MySQL 8.0 with default collation settings) than it did on the old setup (MySQL 5.7 with default collation settings):
These nonprintable characters are now ignored in a comparison.
You can test the difference with this simple fiddle. The comparison is "0" in 5.6, but "1" in MySQL 8.0 -> https://dbfiddle.uk/?rdbms=mysql_5.7&fiddle=a9bd0bf7de815dc14a886c5069bd1a0f
Note that the SQL fiddle also uses a default collation configuration when it's not specified explicitly.
You might fix that by setting a binary UTF-8 collation for the txtDomain column, which is actually what you want for technical strings anyway:
CREATE TABLE `domains` (
`intDomainID` int(11) NOT NULL AUTO_INCREMENT,
`txtDomain` varchar(100) NOT NULL
CHARACTER SET utf8mb4
COLLATE utf8mb4_binary_ci,
`dtDateTime` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`blnTor` int(1) NOT NULL DEFAULT '0',
`txtTLD` varchar(100) NOT NULL,
PRIMARY KEY (`intDomainID`),
UNIQUE KEY `txtDomain` (`txtDomain`)
) ENGINE=InnoDB AUTO_INCREMENT=10127897 DEFAULT CHARSET=utf8mb4;
UPDATE: As it turns out, the COLLATION must have been different between the old (5.6) and new (8.0) setup, as utf8mb4_0900_ai_ci was introduced with MySQL 8.0. The old collation must have been utf8mb4_general_ci, which when applied shows the desired behaviour in MySQL 8.0, too.
But still, you should use binary collation for technical strings like URLs anyways.
I am loading a csv file into an empty table with success and no errors. When comparing the number of original rows (from viewing .csv in external application and from the Output Response) with the number of rows in my resulting table (from Table Inspector) seems like not all rows are importing. Table Inspector reports that the table has 1,416,824 rows while the original csv has 1,419,910 rows. There should be no replicated primary keys in the data set though it should have error'd out on those lines in my mind if that were the case.
Table structure:
CREATE TABLE `table1` (
`pkfield` varchar(10) NOT NULL,
`field1` varchar(3) DEFAULT NULL,
`field2` varchar(1) DEFAULT NULL,
`field3` varchar(1) DEFAULT NULL,
PRIMARY KEY (`pkfield`),
UNIQUE KEY `pkfield_UNIQUE` (`pkfield`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Import command:
LOAD DATA INFILE 'c:/table1.csv'
INTO TABLE myschema.table1
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
MySQL Workbench Response:
1419910 row(s) affected Records: 1419910 Deleted: 0 Skipped: 0 Warnings: 0
Sample from csv file (data mocked up):
6623950258,XYZ,A,B
5377103432,XYZ,A,B
9131144416,XYZ,A,A
1326703267,XYZ,A,B
7847786312,XYZ,A,B
1119927042,XYZ,B,B
4144055385,CDE,A,B
4563489252,XYZ,A,B
5733611912,XYZ,A,B
3309418377,XYZ,A,B
6928148128,XYZ,A,B
1152657670,XYZ,A,B
8143082292,CDE,A,B
9373340750,XYZ,A,A
3318949288,XYZ,A,B
1166427684,XYZ,A,B
5062296807,XYZ,B,A
4624323293,XYZ,A,B
3088992643,CDE,A,B
6477504847,XYZ,A,B
Any suggestions or explanations would be greatly appreciated. Kind regards.
Honestly, I'm myself not sure why the number of rows isn't accurate after a fresh import of a table. I think the Table Inspector fetches the data from some statistics table and to my understanding that gets updated only when the table changes by more than 10 %. Perhaps this is the reason. However, the accurate amount of rows can always be fetched with traditional
select count(*) from myschema.table1;
As #nbayly told, this gives the expected result of 1419910 rows which matches the number LOAD DATA had reported.
Honestly, for now, I didn't know why this happened. But I knew a solution to this.
At first, I thought It might be because of special characters present in the csv records. but even after removing those still, I was receiving not all records. I also noticed that records don't need to be in thousand to be missed by load data. It even happens in a few hundred records.
So for now the only reliable solution is to import using phpMyAdmin. Please remove special characters etc before importing, and also remove new lines from fields header etc.
phpMyAdmin seems to have some sort of parser that tokenized the csv file and then create SQL from those tokens. So it does not use at all load data command and hence it did import corrects.
The downside is it is through GUI and all those slowness that comes with point and click etc.
I'm using Coldfusion8 and MySQL 5.0.91 and am trying to create a temporary table, import some data from CSV, validate, store in database and drop the temp table.
I can do all steps in MySQL directly, where it works without a problem. However when I try to create the temp table in Coldfusion I keep getting a MySQL error.
This is what I have:
<cfset variables.tt = CreateUUID()>
<cfquery datasource="#session.datasource#">
DROP TEMPORARY TABLE IF EXISTS ##variables.tt;
CREATE TEMPORARY TABLE ##variables.tt (
tmp_kto VARCHAR(7) DEFAULT ''
, tmp_kz VARCHAR(3) DEFAULT ''
, tmp_preisliste VARCHAR(1) DEFAULT ''
, tmp_eg VARCHAR(3) DEFAULT ''
) ENGINE=MEMORY;
</cfquery>
which does not work. I also tried #variables.tt# and plain variables.tt but I keep getting the same error, while it works fine when I run the code in MySQL directly (with table name "test")
Question:
I have no clue what I'm doing wrong. Is there a max length for table names perhaps?
Thanks for some insights.
EDIT:
I get the usual error:
you have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right syntax to use near
'CREATE TEMPORARY TABLE #variables.tt (tmp_sa VARCHAR(3) DEFAULT ''
The most likely source of your problem is that the uuid is not suitable as a tablename. For one thing, it contains hyphens. I don't know if MySQL allows this, but I can't remember ever seeing hyphens in a table name.
We have a similar situation, but with redbrick. We want to create unique names for temporary tables. Our approach, written in a custom tag, is to use the string "temp" followed by a random number. We also have some try/catch stuff going on in case we get really unlucky with the random number.
I wrote that tag several years ago. If I were to do it again, I would probably replace the random number with the cftoken, cfid, and tickcount.
Also, you can't drop a table before you create it.
It looks to me you are missing the hash marks around your variable to output.
CREATE TEMPORARY TABLE ##variables.tt (
should be
CREATE TEMPORARY TABLE #variables.tt# (
The way you have it, MySQL is attempting to use the string #variables.tt (the doubled hash escapes the character in ColdFusion) when what you want is to create a temp table with the name of the value of variables.tt.
What is best practice for storing data in a database which ever only requires a single entry. An example would be configuration data which relates to the entire application/website. Is it common to create a table for this which has only a single entry?
I'm asking under the context of a MongoDB database though I think the question is also valid for SQL databases.
An example of an auxiliary table commonly found in databases would be called Constants and may hold such values of pi, the idea begin that all applications using the database are required to use the same scale and precision. In standard SQL, to ensure they is at most one row e.g. (from Joe Celko):
CREATE TABLE Constants
(
lock CHAR(1) DEFAULT 'X' NOT NULL PRIMARY KEY,
CHECK (lock = 'X'),
pi FLOAT DEFAULT 3.142592653 NOT NULL,
e FLOAT DEFAULT 2.71828182 NOT NULL,
phi FLOAT DEFAULT 1.6180339887 NOT NULL,
...
);
Because mySQL doesn't support CHECK constraint then a trigger is required to achieve the same.
A table would be fine, no reason why not to use it just because it will have only one row.
I just had the weirdest idea (I wouldn't implement it but for some reason I thought of that). You can create a hard-coded view like this:
create view myConfigView
as
select 'myConfigvalue1' as configValue1, 'myConfigvalue2' as configValue2
and do select * from myConfigView :)
but again, no reason why not to use a table just because it will have only one row
If you are using a SQL DB, you will probably have columns like key name, and value and each attribute will be stored as a row.
In MongoDB, you can store all related configuration as a single JSON document
I use a config table with a name (config_name) and a value (config_value). I even add a help field so that users can see what the name/value pair is intended for, or where it is used.
CREATE TABLE config (
config_id bigint unsigned NOT NULL auto_increment,
config_name varchar(128) NOT NULL,
config_value text NOT NULL,
config_help text COMMENT 'help',
PRIMARY KEY (config_id),
UNIQUE KEY ix_config_name (config_name),
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
Then following php code recovers the value for a key, or returns an empty string. Assumes $db is an open database connection. All entries are forced to lower case.
function getConfigValue($name) {
$retval='';
$db = $this->db;
$sql = 'select config_value from config where LOWER(config_name)="'.strtolower($name).'"';
$result = $db->Query($sql);
if ($result) {
$row = $db->FetchAssoc($result);
$retval = $row['config_value'];
}
return $retval;
}
All mysql/php in this instance, but the general principle remains.
For MongoDB databases, I usually just make a new "table", but, for SQL databases, that entails a lot more (especially when others are also working on the same database; SQL isn't as malleable), so, you might want to be a bit more careful with it.
I would just create table for configurations, as rainecc told, and use cache then to take that all table to memory :) and use it from there (cache). It will be the best.
I have a file with large number of columns and I want to input this file in mysql table.
The thing is if we have file with, say, 8 columns then we will first create table by -
CREATE TABLE `input` (
`idInput` varchar(45) DEFAULT NULL,
`row2` varchar(45) DEFAULT NULL,
`col3` varchar(45) DEFAULT NULL,
`col4` varchar(45) DEFAULT NULL,
`col5` varchar(45) DEFAULT NULL,
`col6` varchar(45) DEFAULT NULL,
`col7` varchar(45) DEFAULT NULL,
`col8` varchar(45) DEFAULT NULL
);
then we will input the file by -
LOAD DATA INFILE "FILE" INTO TABLE input;
But the thing is, I have file with 150 columns and I want to insert this file in mysql table automatically (so that I should not have to create table first). The first row of my file is header and it should be as column names in table and also each column and each row has different datatype.
So is there any easy way to do this so that after that I can do different things with this table?
I am using mysql command line client version 5.5.20 (windows 7).
You can try using SequelPro mysql client.
With this tool you can use the option "File->Import", and in the window "CSV Import Field Mapping", instead of selecting to import into an existing table, you can choose the button "New".
It's better if your CSV have a header line describing the column names, so it gives the right column names. The tool also is good at guessing the types of the columns according to the content.
You can eventually experience problems if VARCHAR(255) is setted as default type for fields of text type. If it is the case, change the type of those fields to TEXT type.
use phpmyadmin. It has the ability to create table base on the first line of the file and guess the table structure. Click "Import" link and select your file. Don't forget to select the Format to fit your file format, mostly CSV.
If the file is too big to fit into phpmyadmin, sometimes I "head" the file and use the head file in phpmyadmin to create the table, then import the file using the LOAD DATA command.
It makes my life easier.
I don't think this is possible using straight-up MySQL. Somehow the column definitions would have to be guessed. You'll prob have to go with a 2ndary language to read out the first row, make the table and then import.
You can do this using mysqldump though.
As I understand it, you have a generated text file with different data types ready to load from the command line. Here are instructions from MySQL.
to create:
https://dev.mysql.com/doc/refman/5.7/en/mysql-batch-commands.html
to alter:
https://dev.mysql.com/doc/refman/5.7/en/alter-table.html
all command lines which I also use. (if someone has a handy video that describes every step of how to use one of those MySQL Developer Environments though, that might be kinda of nice, one that it doesn't take 20 steps to load a table, though always probably be faster to type one in by hand and use one step or edit a dump.).