I'm trying to create a data warehouse.
Is it possible to populate a table in db1, from data in db2.
For example
Corporate Database Table Route
CREATE TABLE ROUTE (
RouteID INTEGER(4) PRIMARY KEY,
RouteName VARCHAR (50) NOT NULL,
BoardingStop VARCHAR (50) NOT NULL,
AlightingStop VARCHAR (50) NOT NULL
);
Insert Information
INSERT INTO `ROUTE` (`RouteID`,`RouteName`,`BoardingStop`,`AlightingStop`)
VALUES (1,"ab","B","C")
Data warehouse table dimRoute
CREATE TABLE DimROUTE (
RouteID INTEGER(4),
RouteName VARCHAR (50) NOT NULL,
BoardingStop VARCHAR (50) NOT NULL,
AlightingStop VARCHAR (50) NOT NULL,
PRIMARY KEY(RouteID)
);
Populate the above table with data from the first table.
You can copy from one table into another table with INSERT INTO ... SELECT. See docs here: http://dev.mysql.com/doc/refman/5.7/en/insert-select.html
You can copy between tables in different databases on the same MySQL instance, provided you have privileges to both database. Just use databasename.tablename syntax:
INSERT INTO warehouse.DimRoute
SELECT * FROM corporate.Route;
If the databases are hosted on different MySQL instances, you can dump data from the corporate instance and import to the data warehouse instance using mysqldump. Since your table is named differently in the data warehouse, this is a little bit tricky.
You could restore the data to its original table name, and then rename the table:
$ mysqldump --host=corporate corp_dbname ROUTE > route-dump.sql
$ mysql --host=datawarehouse dw_dbname < route-dump.sql
$ mysql --host=datawarehouse -e "RENAME TABLE ROUTE TO DimROUTE" dw_dbname
(I'm leaving out user/password options for brevity, but I suggest you use the config file for those.)
You just need a couple of queries to clone a table (with its indexes and keys) then populate it with the records:
CREATE TABLE DimROUTE LIKE ROUTE;
INSERT DimROUTE SELECT * FROM ROUTE;
Demo SQL Fiddle
Yes, you can. The technique you want is called Extract, Transform and Load (ETL). There are a number of tools you can use, which will help you automate and organise the process. Or you can roll your own solution.
It is quite common for reporting databases to be feed by other databases in this fashion.
Related
I am able to create a table manually in SQL Server by checking columns from MySQl table. And then able to move data from MySQL to SQL Server table.
Example: MySQL table = Employee
Describe Employee;
Output
Field, Type, Null, Key, Default, Extra
EmpId int(10) NO PRI 0
Name varchar(100) YES
Age int(10) YES 18
EmailId varchar(100) NO
Using this I am creating same table in SQL Server
Drop table MsSQLdb..Employee;
CREATE TABLE Employee (
EmpId int NOT NULL PRIMARY KEY DEFAULT 0,
Name varchar(100),
Age int de,
Name varchar(100) NOT NULL DEFAULT 18
);
Now using OpenQuery to copy the data from MySQL to SQL Server :
select EmpId,Name,Age,EmailId into MsSQLdb..Employee
from
OPENQUERY(LinkedServer, 'SELECT EmpId,Name,Age,EmailId FROM mySQL_db.Employee')
But, daily my mySQL_db.Employee table gets more columns or less. So daily I need to manually Map this columns in the above queries to redo the task. Is there any dynamic way for this task?
I have around 40-tables and each table have around 30+ columns. so looking for any a dynamic way.
You can go for SELECT * INTO for loading data into tables. More on SELECT
I assume that you are fine for DROP & RECREATE and simple scenario of loading data. You might not get right datatype as datatype is automatically decided by SQL Server based on initial set of rows.
DROP TABLE dbo.Employee;
SELECT * INTO dbo.Employee
from
OPENQUERY(LinkedServer, 'SELECT * FROM mySQL_db.Employee')
Indexes, constraints, and triggers defined in the source table are not
transferred to the new table, nor can they be specified in the
SELECT...INTO statement. If these objects are required, you can create
them after executing the SELECT...INTO statement.
I have googled this a lot, and I have not found anything matching my problem.
I have a lot of Time Series, containing different sensors readings. Each Time Series is stored into a .csv file, so each file contains a single column.
I have to populate this MySQL table:
CREATE TABLE scheme.sensor_readings (
id int unsigned not null auto_increment,
sensor_id int unsigned not null,
date_created datetime,
reading_value double,
PRIMARY KEY(id),
FOREIGN KEY (sensor_id) REFERENCES scheme.sensors (id) ON DELETE CASCADE
) ENGINE = InnoDB;
while the sensors table is:
CREATE TABLE scheme.sensors (
id int unsigned not null auto_increment,
sensor_title varchar(255) not null,
description varchar(255) not null,
date_created datetime,
PRIMARY KEY(id)
) ENGINE = InnoDB;
Now, I should fill the reading_value field with values contained in the above descripted .csv files. An example of this kind of file:
START INFO
Recording Time *timestamp*
Oil Pressure dt: 1,000000 sec
STOP INFO
0,445328
0,429459
0,4245
0,445099
0,432434
0,433426
...
EOF
What I need is to design an SQL query in which I populate this table while reading values from a .csv file.
I cannot figure out how to proceed: should I use some sort of temporary table as a buffer?
I use HeidiSQL as Client.
The kind of tool you looking for is called an ETL (Extract, transform, Load).
You can extract data form csv files (among other), transfrom them by adding the info from the sensor db-table (among other), and load it into the sensor_reading db-table.
There are plenty of ETL on the market. Although, I should be agnostic, a free, easy to learn and covering all your future needs, you may start evaluating PDI (Pentaho Data Integrator, nicknamed Kettle). Go there, download the latest Data Integrator, unzip and press the spoon.bat / spoon.sh. A nice getting started is there. And the StackOverFlow flag Pentaho Data Integration, respond usually quite quickly.
Alternatively you may try Talend or plenty others.
I need to create a table with a file or anything, everything needs to be done as a sql script.
Can someone help me create a table without a csv file,
The name of the table is "videos"
The rows will be:
unique id
title
minutes
URL
When creating table to database via script you need to create a file where you will define structure of table with DDL (Data definition language).
For example
Create file table.sql. Open file and use CREATE TABLE statement for table creation.
CREATE TABLE videos(
id INT NOT NULL AUTO_INCREMENT,
minutes INT NOT NULL,
title VARCHAR(255) NOT NULL,
url VARCHAR(255) NOT NULL,
PRIMARY KEY ( id )
);
Afterwards you can run this script in many different ways. For example in Linux based operating systems you can run script with command mysql in form mysql -u user -p database_name < table.sql, where user is your username and database_name is name of database for which you want to create table.
Is there a proper place to store a high level description of a database? Something along the lines of "This database is used to store XYZ for use by ABC". It's not necessarily information one would need to query, but something that would useful for someone administering the system (i.e. me in a few months when I'm trying to remember what I was trying to accomplish a few months ago.).
This seems like something that someone would have asked before (or information that is readily findable), but none of my searching came up with anything relevant. Most of what I found was for displaying the structure of the database itself.
Comment metadata is not available for MySQL databases, but you can create a table to store some comments: (I have this table in a generic tools database)
-- Create a table to store db name and fields as need.
create table dbinfo(
id int not null primary key auto_increment,
db_name varchar(64) not null collate utf8_bin,
db_comment varchar(255),
unique (db_name)
) default charset utf8;
To fill/update dbinfo table with current databases:
insert into dbinfo (db_name)
select SCHEMA_NAME
from INFORMATION_SCHEMA.SCHEMATA
where SCHEMA_NAME not in (select db_name from dbinfo);
You will only need to maintain dbinfo table until MySQL enables database comments.
i'm sure there is such question at stackoverflow, but i just c't find it :(
I have 2 databases with same data ("developer" database and "production" database).
"production" database is "Live" database - sitve visitors see this data
"developer" database is database where i create new functions at my local server.
I have situation when i add to "developer" database some new tables and some new fields in old tables.
And now i have to copy this new created fields and tables to "production" database (but only structure, data should not be copied and no data at "production" database must be changed).
UPD: Maybe there is solution where i can make database structure dump from developing database and when i import it to production database, it automatically add all new fields from all tables
What functions should i use?
Thanks.
You want alter table: http://dev.mysql.com/doc/refman/5.1/en/alter-table.html
To alter an existing table:
alter table tablename add column newcolumn tinyint(1) default 1 AFTER othercolumn
To create a new table:
CREATE TABLE `newtablename` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`newcolumn` int(10) unsigned NOT NULL,
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
to copy ONLY the structure you can use like:
jcho360> create table t3 like t2;
Query OK, 0 rows affected (0.05 sec).
If you want to copy the content after create the structure you can use :
INSERT INTO t3 SELECT * FROM t2;
if you want to copy with data you can use (without structure, I mean, PK, FK, etc)
mysql> create table t4 as select * from t1;
of course you are able to use backup to restore tables too
if you want to add new column you can use
alter table tablename add column newcolumn vartype;
if you want to use a table from another database you can use queries calling first the database and later the name, like
select * from database.tablename;
remember the user need permission to the other db too.