I have a large db that I am chopping into smaller databases based on time intervals. This will reduce query time dramatically. In a query can I copy a resultset from one database to another with an identical schema?
Basically a select followed by an update conducted in the same code block?
Thanks,
slothishtype
Copying data from one database into another should be almost as simple as #slotishtype describes except you'll need to qualify it with the OTHER database you want it replicated into.
create table OtherDatabase.Student Select * from FirstDatabase.student
However, as you mention about copying same schema, that is something else. If you want all your R/I rules, triggers, etc, you may have to dump the database schema from your first (where it has all the create tables, indexes, etc) and run in a new database. However, that might post an issue where you have auto-incrementing columns. You can't write to a read-only auto-increment column -- the database controls that. However, if such case existed, you would have to just make those columns as integer datatypes (or similar) and do a
insert into OtherDatabase.Student ( field1, field2, etc )
select field1, field2, etc from FirstDatabase.student
If it is not necessary to add it to a new database, this will do fine:
CREATE TABLE student1 SELECT * FROM student
EDIT: for the record: This will not coopy over indices etc.
This, however, will:
CREATE TABLE student_new LIKE student;
INSERT recipes_new SELECT * FROM student;
slotishtype
Related
So I have this problem where I need to create an exact copy of another table (schema, indexes) but not the data. We need to temporarily store data before we move it to the table it mirrors.
I can create a mirror table like CREATE TABLE foo_mirror LIKE foo; But I want to automatically update the schema when the schema of the table foo changes'.
I have considered temporary tables as well instead of mirror tables but we will have 2 different processes copying data to the mirror table foo_mirror and copying data to the actual table foo. So I can't use a single session.
So I can compare the schema using the following queries:
-- for_mirror schema
SELECT DISTINCT *
FROM `information_schema`.`columns`
WHERE `TABLE_NAME` LIKE 'foo_mirror'
-- foo schema
SELECT DISTINCT *
FROM `information_schema`.`columns`
WHERE `TABLE_NAME` LIKE 'foo'
I could write some code that will apply updates in a job, or perhaps a trigger. I'm wondering if there is a better way to keep tables in sync or even a tool where I don't need to manage this myself.
I also see a potential problem e.g. Changing the type for a column may fail if data hasn't been updated correctly in the mirror table (string to integer - "one" -> 1).
I'm rather new at database management, so this might not be feasible, but I got a handful of SQL select queries, rather long ones at that. What I'd like is to get the table column names and structure, without access to the actual database, so as to get a map of all this queries.
context: All we have are the queries used to output tables that will be given to us latter.
This need not be done with actual SQL code, maybe a short script in other language or a utility somebody knows of (but I do have MySQL workbench)
You can add a CREATE TABLE statement in front of your select queries to get the column names.
You cannot infer data types or keys from select queries.
For column names do something like:
drop table if exists your_table_name;
create table your_table_name
select *
from ...
where the select * portion is replaced by the select queries you have.
Then to see the column names in a friendlier way you can do:
show create table your_table_name;
or
desc your_table_name;
I'm experiencing huge performance problem in one legacy application.
There is a search form where user can search records with given value.
A result row contains 10 columns. Then a SP returns any row which contains in any column that value.
This SP uses 8 Tables and some of them have about million records. Every minute I get a new record. This SP conducts paging as well.
Execution of this SP takes sometimes around 40 seconds.
What I did was, I created a new table and put there all records by using a query from this SP, but without conditions.
When there is a new update or update in one of source table I use a trigger and update this new "cache" table.
Now waiting for results from this new table takes only 1-3 seconds.
Has someone experience with something like this?
One of my colleagues said I better use view, but then every time I will be making JOINS.
What do you think? Is there another way?
Often times temporary tables can help you resolve performance issues. One approach might be to collect only the records that you need to consider into temporary tables and then create your final select statement from the temporary tables joined to any other tables that you're not filtering.
As an example, let's say one of the fields you are searching for is field1 in table1. Start by inserting into table #table1 only records that have the value of field1 you are looking for:
select PrimaryKeyTable1, Field1, Field2, Field3, etc...
into #table1
from table1
where Field1 = 'Whatever you are looking for'
This should be pretty fast even for a big tables, especially if you have an index on Field1. You do this for every table with search fields to collect all the records that have relevant records you are searching.
Then you also need to be sure to insert any records into your temporary tables that might have foreign key references to any of your other temporary tables. So let's say you also built a table #table2 with the above method that has a foreign key to table1 called PrimaryKeyTable1. You would insert those records like:
Insert into #table1
(PrimaryKeyTable1, Field1, Field2, Field3, etc...)
select table1.PrimaryKeyTable1, table1.Field1, table1.Field2, table1.Field3, etc...
from table1
join #table2
on table1.PrimaryKeyTable1 = table2.PrimaryKeyTable1
where table1.PrimaryKeyTable1 not in
(Select PrimaryKeyTable1 from #table1)
Now you will also have any records in #table1 that match to a record in #table2 that contain records that match the search criteria. You do this for all your temporary tables that have relevant foreign keys. The order that you do the inserts matters; be sure that you don't reference any temporary tables until after the last insert statement while collecting the foreign key referenced records.
Then you can simply do your final select statement, replacing the actual tables with the temporary tables you have built and eliminating all the filters that search your field data. Depending on the structure of your query there might be other optimizations, but that is the general idea.
If you've already explored all of your indexing options and this still doesn't help, MS SQL Server has "Change Tracking" features that maybe be of use to you in building your cache table. You enable the database for change tracking and configure which tables you wish to track. SQL Server then creates change records on every update, insert, delete on a table and then lets you query for changes to records that have been made since the last time you checked. This is very useful for syncing changes and is more efficient than using triggers. It's also easier to manage than making your own tracking tables. This has been a feature since SQL Server 2005.
How to: Use SQL Server Change Tracking
Change tracking only captures the primary keys of the tables and let's you query which fields might have been modified. Then you can query the tables join on those keys to get the current data. If you want it to capture the data also you can use Change Capture, but it requires more overhead and at least SQL Server 2008 enterprise edition.
Change Data Capture
Your solution is a robust way of doing what is called in Microsoft SQL Server "an indexed view" or "materialized view" in Oracle.
Basically you are correct - it's faster to navigate single indexed table then a dozen ones which are updated constantly.
You should really try creating an indexed view (some start here https://technet.microsoft.com/en-us/library/dd171921(v=sql.100).aspx) and it will probably solve all your performance issues.
You can use schema binding View and create cluster index on view.it will store your view data physically.but after creating schema binding view you can not alter your table.
I was wondering is there a performance difference between:
query 1: CREATE TEMPORARY TABLE temp_table1 AS SELECT * FROM lookup_table JOIN ...
then
INSERT INTO dest_table SELECT * FROM temp_table1
vs
query 2: INSERT INTO dest_table SELECT * FROM lookup_table JOIN ...
My concern was, the lookup_table is accessed very often by different users and when I run query 2, most of the users need to wait longer to be able to retrieve their result. What I was thinking was to write the data into a temporary table then write it to dest_table afterwards . Im just not sure if writing into a temp table with give a difference performance compared to writing it directly to the destination table. Im using mysql 5.6.
The reason why I need to write data from lookup_table to dest_table is because I need to create a report from it. Seeing how complex the query from lookup_table is makes it very difficult to create a report so I decided to move those data to a single table then just make a report from it.
You're concerned about the lockout time that's taken by the SELECT query that populates this temporary table.
The tables are implemented the same way, so the cost of creating will be very close to the same in either case.
You might be able to get it to go a little faster by creating your temporary table in the MEMORY access method, but I suspect the difference will be minimal; the work involved here is the SELECT / JOIN stuff.
You might be able to get it to go faster by making sure your target table has no indexes when you create it. CREATE ... AS SELECT will do that.
You will be able to make it cheaper to create by getting rid of SELECT * (which yields redundant columns on JOINs anywhow), and instead specify the columns you really need.
But, your best bet is to figure out why you're creating this table, and see if you can deliver on those requirements by writing queries against the source tables instead. If you make those query operations efficient, you've saved yourself lots of data shuffling.
I have a huge table in a database and I want to split that into several parts physically, maintaining the database scheme.
For example, the table name is TableName and has 2 000 000 rows.
I would like to split that table into four parts, but I want to work in the same way with the table, so
select [Column List] from TableName where [Filter]
insert into TableName ([Column List]) values([Values])
update TableName [Updates] where [Filter]
delete from TableName where [filter]
would work in the same way after splitting the table as before. Basically I want my database to handle in different threads my queries. How can I achieve this?
Thanks in advance.
Perhaps you should look at partitioning.
If you are looking into copying data to a separate slave / duplicate; consider implementing that by binary logging, so the replica will read the binary logs to do the replication, rather than doing it manually or programmatically.