We have a large MySql table having around 400 million records, it is running on google cloud sql version 1 (MySql version 5.5).
Now google has come up with the new version of cloud sql (MySql version 5.6) ,so we want to migrate our database into this new sql version.This new version is available only on new instance. That means version 1 is on machine 1 and version 2 is on machine 2.
In this process we also made few schema changes, please check the following.
Old Schema
CREATE TABLE page_views_old (
domain varchar(50) DEFAULT NULL,
guid varchar(100) DEFAULT NULL,
sid varchar(100) DEFAULT NULL,
url varchar(2500) DEFAULT NULL,
ip varchar(20) DEFAULT NULL,
is_new varchar(20) DEFAULT NULL,
ref varchar(2500) DEFAULT NULL,
user_agent varchar(255) DEFAULT NULL,
stats_time datetime DEFAULT NULL,
country varchar(50) DEFAULT NULL,
region varchar(50) DEFAULT NULL,
city varchar(50) DEFAULT NULL,
city_lat_long varchar(50) DEFAULT NULL,
email varchar(100) DEFAULT NULL
)
New schema
CREATE TABLE page_views_new (
domain varchar(50) DEFAULT NULL,
guid binary(16) NOT NULL,
sid binary(16) NOT NULL,
url varchar(2500) DEFAULT NULL,
ip varbinary(16) NOT NULL,
is_new tinyint(4) NOT NULL,
ref varchar(2500) DEFAULT NULL,
user_agent varchar(255) DEFAULT NULL,
stats_time datetime DEFAULT NULL,
country char(2) NOT NULL,
region varchar(6) NOT NULL,
city varchar(50) DEFAULT NULL,
city_lat_long varchar(50) DEFAULT NULL,
email varchar(100) DEFAULT NULL,
id int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (id)
)
As mentioned above we changed datatype varchar to binary, varchar to varbinary, varchar to tinyint, ip to binary and others. So my question is what is best way to load this database into new instance. I didn't mention here we need to add few indexes also.
My approach for this problem
1.Take a dump of complete Database.
2.Import into new instance.
3.Modify columns one by one using alter table statements.
4.Make it live.
Can you guys please specify is there any issue with this approach and suggest me best approaches. Please mention if any issues with schema also.
Thanks
Related
I am working on my database that used to work fine until recently, here I keep getting the following error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CREATE TABLE user_bans ( id INT PRIMARY KEY AUTO_INCREMENT, user INT NOT' at line 34, the code is below. I did my best to fix it and find answers, but none of them helped.
-- Create 'users' table
CREATE TABLE users (
id INT PRIMARY KEY AUTO_INCREMENT,
username VARCHAR(255) NOT NULL,
nickname VARCHAR(255) NOT NULL, -- Unlike the 'username' field, this one is NOT unique
email VARCHAR(255) DEFAULT NULL,
email_verified TINYINT(1) DEFAULT 0 NOT NULL,
email_verified_time INT DEFAULT NULL,
password VARCHAR(255) NOT NULL,
register_ip VARCHAR(255) NOT NULL,
last_ip VARCHAR(255) NOT NULL,
description TEXT DEFAULT NULL,
status VARCHAR(255) DEFAULT NULL,
profile_image VARCHAR(255) NOT NULL, -- Set a random one when creating
-- Ranks
admin TINYINT(1) DEFAULT 0 NOT NULL,
beta_tester TINYINT(1) DEFAULT 0 NOT NULL,
-- Options
profile_nsfw TINYINT(1) DEFAULT 0 NOT NULL,
profile_private TINYINT(1) DEFAULT 0 NOT NULL,
-- Moderation
deleted TINYINT(1) DEFAULT 0 NOT NULL,
banned TINYINT(1) DEFAULT 0 NOT NULL,
-- Time fields
time_registered INT NOT NULL, -- Set to current UNIX timestamp when creating
time_last_seen INT NOT NULL -- Set to current UNIX timestamp when going offline
);
-- Create 'user_bans' table
CREATE TABLE user_bans (
id INT PRIMARY KEY AUTO_INCREMENT,
user INT NOT NULL,
banned_by INT NOT NULL,
time_ban INT NOT NULL, -- Set to current UNIX timestamp when creating
time_unban INT NOT NULL,
ban_category VARCHAR(255) NOT NULL,
content TEXT DEFAULT NULL,
mod_note TEXT DEFAULT NULL,
status VARCHAR(255) DEFAULT 'banned' NOT NULL,
unbanned_by INT DEFAULT NULL,
FOREIGN KEY (user)
REFERENCES users (id)
ON DELETE CASCADE,
FOREIGN KEY (banned_by)
REFERENCES users (id)
ON DELETE CASCADE
);
This continues to happen even if the tables consist of just a single field.
According to the font color, "user" looks like a reserved word in your system,
try using back ticks `` to escape it.
I'm doing some very basic database exercises in phpMyAdmin like creating tables using SQL queries and observing it in designer mode.
But I realized there is an error shown there but when I was typing the commands to create the tables, there are no mistakes. But in designer mode, there is one line in each of my tables that is in red. The notice in the designer mode says "trying to access array offset on value of type bool".
When I ignored it and proceeded to create relations to form an ERD, I can't because the line that is in red was part of the constraint. I wonder what could be the problem. I've double-checked my queries and cross-checked the queries with my friends, drop all tables and re-create them and even re-installing xampp and phpMyAdmin. I'll add a query for one of my tables. For this particular table, the line customerName appears red in designer mode.
DROP TABLE IF EXISTS customers;
CREATE TABLE customers(
`customerNumber` int(11) NOT NULL,
`customerName` varchar(50) NOT NULL,
`contactLastName` varchar(50) NOT NULL,
`contactFirstName` varchar(50) NOT NULL,
`phone` varchar(50) NOT NULL,
`addressLine1` varchar(50) NOT NULL,
`addressLine2` varchar(50) DEFAULT NULL,
`city` varchar(50) NOT NULL,
`state` varchar(50) DEFAULT NULL,
`postalCode` varchar(15) DEFAULT NULL,
`country` varchar(50) NOT NULL,
`salesRepEmployeeNumber` int(11) DEFAULT NULL,
`creditLimit` decimal(10,2) DEFAULT NULL,
PRIMARY KEY (`customerNumber`),
KEY `salesRepEmployeeNumber` (`salesRepEmployeeNumber`))
ENGINE=InnoDB DEFAULT CHARSET=latin1;
I have created one table and inserting values in the table from a csv file using a python code .The createdatetime and updatedatetime columns are set to default CURRENT_TIMESTAMP.But when I am populating the data updatedatetime is populating the correct value but createdatetime is populating as 0000-00-00 00:00:00.
Here is my table definition:
CREATE TABLE `fico_details` (
`adf_contact_id` varchar(100) NOT NULL,
`sf_contact_id` varchar(100) DEFAULT NULL,
`Name` varchar(100) NOT NULL,
`BirthDate` date DEFAULT NULL,
`Address1` varchar(100) DEFAULT NULL,
`City` varchar(100) DEFAULT NULL,
`State` varchar(2) DEFAULT NULL,
`Zipcode` varchar(10) DEFAULT NULL,
`SSN` varchar(10) DEFAULT NULL,
`Address2` varchar(100) DEFAULT NULL,
`City2` varchar(100) DEFAULT NULL,
`State2` varchar(2) DEFAULT NULL,
`Zipcode2` varchar(10) DEFAULT NULL,
`Customerinput` varchar(10) DEFAULT NULL,
`AddrDiscrepancyFlg` varchar(10) DEFAULT NULL,
`Permid` varchar(10) DEFAULT NULL,
`score_date` varchar(10) DEFAULT NULL,
`reason_code_1` varchar(10) DEFAULT NULL,
`reason_code_2` varchar(10) DEFAULT NULL,
`reason_code_3` varchar(10) DEFAULT NULL,
`reason_code_4` varchar(10) DEFAULT NULL,
`CreateDateTime` TIMESTAMP not null DEFAULT CURRENT_TIMESTAMP,
`UpdateDateTime` TIMESTAMP not null DEFAULT CURRENT_TIMESTAMP,
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `SSN` (`SSN`)
) ENGINE=InnoDB AUTO_INCREMENT=876800 DEFAULT CHARSET=utf8;
please help me to solve it.
I am working on same thing these days.
First of all I use timestamp as the datatype as apart from having datetime capabilities, it has MANY usable functions as a string as well as timestamp datatype.
I got the SAME issue as yours, what I did was alter my table was created, I would click on ALTER table using any Db tool (SQLYog in my case) and then Delete the 0000-00-00 00:00:00 value and uncheck the Not NULL check box.
This issue gets resolved EVERYTIME (whenever I create same type of table/columns)after this simple one step.
Hope this helps you too, let me know if anything still bothers you....!!
I made my ER diagram via MySQL WorkBench and exported it as a sql. Below is code for one table.
CREATE TABLE IF NOT EXISTS `xxx`.`Agent` (
`idAgent` INT NOT NULL AUTO_INCREMENT,
`Name` VARCHAR(100) NOT NULL,
`RegistrationNumber` VARCHAR(45) NOT NULL,
`RegistrationDate` DATE NULL,
`DateOfDealStart` DATE NULL,
`AddressLine1` VARCHAR(100) NULL,
`AddressLine2` VARCHAR(100) NULL,
`Country` VARCHAR(45) NULL,
`CurrentStatus` TINYINT(1) NOT NULL,
`DateCreated` TIMESTAMP NOT NULL,
`LastUpdated` TIMESTAMP NOT NULL,
PRIMARY KEY (`idAgent`))
ENGINE = InnoDB;
However when I am trying to import the script into MySQL via PhpMyAdmin, it says below.
#1067 - Invalid default value for 'LastUpdated'
I have not provided any default value, so how can it could be "Invalid"?
hi every one i have a problem in mysql
my table is
CREATE TABLE IF NOT EXISTS `contactform` (
`contact_id` int(11) NOT NULL AUTO_INCREMENT,
`first_name` varchar(50) NOT NULL,
`addition` varchar(50) NOT NULL,
`surname` varchar(50) NOT NULL,
`Address` varchar(200) NOT NULL,
`postalcode` varchar(20) NOT NULL,
`city` varchar(50) NOT NULL,
`phone` varchar(20) NOT NULL,
`emailaddress` varchar(30) NOT NULL,
`dob` varchar(50) NOT NULL,
`howtoknow` varchar(50) NOT NULL,
`othersource` varchar(50) NOT NULL,
`orientationsession` varchar(20) NOT NULL,
`othersession` varchar(20) NOT NULL,
`organisation` int(11) NOT NULL,
`newsletter` int(2) NOT NULL,
`iscomplete` int(11) NOT NULL,
`registrationdate` date NOT NULL,
PRIMARY KEY (`contact_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=39 ;
mysql>insert into contactform values('','abhi','sir','shukla','vbxcvb','342342','asdfasd','234234234','abhi#gmail.com','1999/5/16','via vrienden of familie','','19','20','6','1','1','2010-03-29')
i get following error.
#1366 - Incorrect integer value: '' for column 'contact_id' at row 1
this query work fine on my local machine but give error on server
Had the same problem with some legacy project. I didn't want to turn off strict mode globally, but also didn't want to go through all the code to fix it, so I disabled it only within that application by executing the following query once after the connection to the db is made:
SET sql_mode = ""
Try using NULL instead of '' for contact_id in
insert into contactform values(NULL,......
Try to find following line in my.cnf/my.ini:
sql-mode=”STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION”
Now comment it out with # and restart your mysql server.
'' is an empty string, you want a integer, but this integer is created by the database. Drop the columnname in the INSERT and don't insert any value.
'' is not an integer, is it?
Also, that is some seriously weird indentation.
In mysql console, try this:
SET GLOBAL sql_mode=(SELECT REPLACE(##sql_mode,'ONLY_FULL_GROUP_BY',''));
If you are importing - make sure your CSV/TXT file is UTF-8 WITHOUT the BOM - the BOM will pass characters that are not intgers and MySQL will think you are importing that into the first field...