I have enabled CDC on few tables in my SQL server 2008 database. I want to change the number of days I can keep the change history.
I have read that by default change logs are kept for 3 days, before they are deleted by sys.sp_cdc_cleanup_change_table stored proc.
Does anyone know how I can change this default value, so that I can keep the logs for longer.
Thanks
You need to update the cdc_jobs.retention field for your database. The record in the cdc_jobs table won't exist until at least one table has been enabled for CDC.
-- modify msdb.dbo.cdc_jobs.retention value (in minutes) to be the length of time to keep change-tracked data
update
j
set
[retention] = 3679200 -- 7 years
from
sys.databases d
inner join
msdb.dbo.cdc_jobs j
on j.database_id = d.database_id
and j.job_type = 'cleanup'
and d.name = '<Database Name, sysname, DatabaseName>';
Replace <Database Name, sysname, DatabaseName> with your database name.
Two alternative solutions:
Drop the cleanup job:
EXEC sys.sp_cdc_drop_job #job_type = N'cleanup';
Change the job via sp:
EXEC sys.sp_cdc_change_job
#job_type = N'cleanup',
#retention = 2880;
Retention time in minutes, max 52494800 (100 years). But if you drop the job, data is never cleaned up, the job isn't even looking, if there is data to clean up. In case of wanting to keep data indefinitely, I'd prefer dropping the job.
Related
Started to get following error when executing certain SP. Code related to this error is pretty simple, joining #temp table to real table
Full text of error:
Msg 605, Level 21, State 3, Procedure spSSRSRPTIncorrectRevenue, Line 123
Attempt to fetch logical page (1:558552) in database 2 failed. It belongs to allocation unit 2089673263876079616 not to 4179358581172469760.
Here is what I found:
https://support.microsoft.com/en-us/kb/2015739
This suggests some kind of issue with database. I run DBCC CHECKDB on user database and on temp database - all passes.
Second thing I'm doing - trying to find which table those allocation units belong
SELECT au.allocation_unit_id, OBJECT_NAME(p.object_id) AS table_name, fg.name AS filegroup_name,
au.type_desc AS allocation_type, au.data_pages, partition_number
FROM sys.allocation_units AS au
JOIN sys.partitions AS p ON au.container_id = p.partition_id
JOIN sys.filegroups AS fg ON fg.data_space_id = au.data_space_id
WHERE au.allocation_unit_id in(2089673263876079616, 4179358581172469760)
ORDER BY au.allocation_unit_id
This returns 2 objects in tempdb, not in user db. So, it makes me think it's some kind of data corruption in tempdb? I'm developer, not DBA. Any suggestions on what I should check next?
Also, when I run query above, how can I tell REAL object name that I understand? Like #myTempTable______... instead of #07C650CE
I was able to resolve this by clearing the SQL caches:
DBCC FREEPROCCACHE
GO
DBCC DROPCLEANBUFFERS
GO
Apparently restarting the SQL service would have had the same affect.
(via Made By SQL, reproduced here to help others!)
I have like your get errors too.
firstly you must backing up to table or object for dont panic more after. I tryed below steps on my Database.
step 1:
Backing up table (data movement to other table as manuel or vs..how can you do)
I used to below codes to my table move other table
--CODE-
set nocount on;
DECLARE #Counter INT = 1;
DECLARE #LastRecord INT = 10000000; --your table_count
WHILE #Counter < #LastRecord
BEGIN
BEGIN TRY
BEGIN
insert into your_table_new SELECT * FROM your_table WHERE your_column= #Counter --dont forget! create your_table_new before
END
END TRY
BEGIN CATCH
BEGIN
insert into error_code select #Counter,'error_number' --dont forget the create error_code table before.
END
END CATCH
SET #Counter += 1;
END;
step 2:
-DBCC CHECKTABLE(your_table , REPAIR_REBUILD )
GO
check your table. if you have an error go to other step_3.
step 3:
!!attention!! you can lost some data/datas on your table. but dont worry. so you backed-up your table in step_1.
-DBCC CHECKTABLE(your_table , REPAIR_ALLOW_DATA_LOSS)
GO
Good luck!
~~pektas
In my case, truncating and re-populating data in the concerned tables was the solution.
Most probably the data inside tables was corrupted.
Database ID 2 means your tempdb is corrupted. Fixing tempdp is easy. Restart sqlserver service and you are good to go.
This could be an instance of a bug Microsoft fixed on SQL Server 2008 with queries on temporary tables that self reference (for example we have experienced it when loading data from a real table to a temporary table while filtering any rows we already have populated in the temp table in a previous step).
It seems that it only happens on temporary tables with no identity/primary key, so a workaround is to add one, although if you patch CU3 or later you also can enable the hotfix via turning a trace flag on.
For more details on the bug/fixes: https://support.microsoft.com/en-us/help/960770/fix-you-receive-error-605-and-error-824-when-you-run-a-query-that-inse
How can I get 'sliced' data and remove the data from database?
I was using two database MySQL and SQL SERVER.
Example:
Yesterday I transfered data from MySql to Sql Server about 1000 rows,
then today I simply deleted 5 rows in MySql then did the transfer again.
So, how can I know which ID was deleted from MySQL then remove it in SQL Server?
I was transferring data using Stored Procedures that check every ID in every loop inserting.
foreach($data AS $key => $value){ $this->MsSQL->Exec('exec sp_pendaftar {ID} {NAME}'); }
I have stored procedure like this:
CREATE PROCEDURE [dbo].[sp_pendaftar] #id INT,#name varchar(45) AS DECLARE #id nvarchar(40); SET NOCOUNT ON; SET #id = (SELECT TOP 1 id FROM t_pendaftar WHERE id = #id);
IF #id IS NULL BEGIN
INSERT INTO t_pendaftar VALUES(#id,#name);
END;
ELSE
BEGIN
UPDATE t_pendaftar SET name = #name WHERE id = #id;
END;
GO
Please help.
I do not understand anything of the SQL-Server code, but I would suggest to make a part of the replication via application. So, for the bigger and more dynamical data tables you could define a delete_pendafter and a insert_pendafer table (and also a change_pendafter if needed).
Before you delete from t_pendafter, you select just those rows into delete_pendafter. This is even possible with triggers, if that does not slow down the application too much.
On SQL-Server-side I hope you have a delete-join, so you just remove the deleted rows. In MySQL this would look like
DELETE orig
FROM t_pendafter AS orig
INNER JOIN delete_pendafter AS del ON del.id = orig.id
This solution can be extended to INSERT and CHANGE, but must be done with some care.
Every now and then you should make a full copy, and you should write some checks to ensure the data is consistent.
Just got some answer from my partner.
First, grab array id from MySQL DB and then grab array id from SQL Server and compare which id that not present in SQL Server using array_diff().
$MySQL : array[id] => [11,12,13,15,16,19,25]
$SQL_Server : array[id] => [11,12,13,14,15,16,17,18,19,20,21,23,24,25 ]
$array_to_be_deleted : array($SQL_Server , $MySQL)
print_r($array_to_be_deleted);
result would be:
array => [14,17,18,20,21,23,24]
Hope anyone can try to correct me.
I have a query that takes a very long time to complete. I am wondering if there is anyway to optimize it so it would run quicker. Currently the table has around 15 million row and has indexing on expirationDate. This query has been running for 8000+ seconds.
UPDATE LOW_PRIORITY items
SET expire = 1
WHERE expirationdate < Curdate()
AND expirationdate != '000-00-00'
AND expire = '0'
AND ( `submit_id` != '742457'
OR submit_id IS NULL )
Never mind your table structure (I requested it before this edit), I have a solution for you. I've done this before with my DBAs several times on our production mySQL databases even during peak hours of the day. The approach you need to take is taking your one massive UPDATE query (that will lock the table while executing) and turn it into several individual updates. Those individual updates will not have a material impact on your database, especially if you run them in smaller batches and break it down.
To generate those individual queries, run this (assuming the items table's primary key is named "id"):
SELECT CONCAT(
‘UPDATE items SET expire = 1 WHERE id = ‘,
items.id,
‘;’) AS ''
FROM items
WHERE expirationdate < Curdate()
AND expirationdate != '000-00-00'
AND expire = '0'
AND ( `submit_id` != ‘742457'
OR submit_id IS NULL )
This will generate SQL that looks like this:
UPDATE items SET expire = 1 WHERE id = 1234;
UPDATE items SET expire = 1 WHERE id = 2345;
.....
What I personally like to do is put the above query in a text file and run this from the command line:
cat theQuery.sql | mysql -u yourusername [+ command line args with db host, etc] -p > outputSQLCommands.sql
This will pass the query to mySQL and dump the results to an output file. If you notice, we named the column in our output to '' (nothing). That makes the output very nicely formatted in the text file so it can be easily dumped back into mySQL to run the commands. In fact you could do it all in one fell swoop like this:
cat theQuery.sql | mysql -u yourusername [+ command line args with db host, etc] -p | mysql -u yourusername [+ command line args with db host, etc] -p
Then you are definitely ballin' SQL style.
After you have the output SQL file though, if it's too many queries to run one after another, you can use a command line tool like "split" to split the file into a bunch of smaller files.
I would recommend testing to find the optimum batch size based on your DB's hardware/performance/etc. For example, try running 1000 updates first, if that works with no material impact try 5000, etc. Then you can split the rest of your file into chunks you are comfortable with and run them through.
Good luck!
How I can find out the changes happened in database like modifying functions, table indexes, procedures and adding or removing columns.
Here in this query
select * from sys.objects
where type IS NOT NULL
and modify_date between '2013-07-21' and '2013-07-29'
but here I am getting created objects list and modifying list, but if I deleted any object it is not showing anything.
How can I get the all the changes happened in database between specific dates?
Try a source control solution for SQL. I've used RedGate's SQL Source Control before, and it records a history of changes like this, including who made the change, and what was changed.
http://www.red-gate.com/products/sql-development/sql-source-control/
It's a bit expensive, but it's good. I don't know if there's a way to do it (especially deletions) just with SQL itself.
Many of these incidents are recorded in the default trace.
DECLARE #path NVARCHAR(260);
SELECT
#path = REVERSE(SUBSTRING(REVERSE([path]),
CHARINDEX(CHAR(92), REVERSE([path])), 260)) + N'log.trc'
FROM sys.traces
WHERE is_default = 1;
SELECT
LoginName,
HostName,
StartTime,
ObjectName,
TextData -- may or may not be populated
FROM sys.fn_trace_gettable(#path, DEFAULT)
WHERE EventClass IN
(
164, -- object:altered
46, -- object:created
47 -- object:deleted
)
AND StartTime >= '20130721' AND StartTime < '20130730';
Why you should never use BETWEEN for date range queries.
My application is very database intensive. Currently, I'm running MySQL 5.5.19 and using MyISAM, but I'm in the process of migrating to InnoDB. The only problem left is checksum performance.
My application does about 500-1000 "CHECKSUM TABLE" statements per second in peak times, because the clients GUI is polling the database constantly for changes (it is a monitoring system, so must be very responsive and fast).
With MyISAM, there are Live checksums that are precalculated on table modification and are VERY fast. However, there is no such thing in InnoDB. So, CHECKSUM TABLE is very slow...
I hoped to be able to check the last update time of the table, Unfortunately, this is not available in InnoDB either. I'm stuck now, because tests have shownn that the performance of the application drops drastically...
There are simply too much lines of code that update the tables, so implementing logic in the application to log table changes is out of the question...
The Database ecosystem consists of one master na 3 slaves, so local file checks is not an option.
I thought of a method to mimic a checksum cache - a lookup table with two columns - table_name, checksum, and update that table with triggers when changes in a table occurs, but i have around 100 tables to monitor and this means 3 triggers per table = 300 triggers. Hard to maintain, and i'm not sure that this wont be a performance hog again.
So is there any FAST method to detect changes in InnoDB tables?
Thanks!
The simplest way is to add a nullable column with type TIMESTAMP, with the trigger: ON UPDATE CURRENT_TIMESTAMP.
Therefore, the inserts will not change because the column accepts nulls, and you can select only new and changed columns by saying:
SELECT * FROM `table` WHERE `mdate` > '2011-12-21 12:31:22'
Every time you update a row this column will change automatically.
Here are some more informations: http://dev.mysql.com/doc/refman/5.0/en/timestamp.html
To see deleted rows simply create a trigger which is going to log every deletion to another table:
DELIMITER $$
CREATE TRIGGER MyTable_Trigger
AFTER DELETE ON MyTable
FOR EACH ROW
BEGIN
INSERT INTO MyTable_Deleted VALUES(OLD.id, NOW());
END$$
I think I've found the solution. For some time I was looking at Percona Server to replace my MySQL servers, and now i think there is a good reason for this.
Percona server introduces many new INFORMATION_SCHEMA tables like INNODB_TABLE_STATS, which isn't available in standard MySQL server.
When you do:
SELECT rows, modified FROM information_schema.innodb_table_stats WHERE table_schema='db' AND table_name='table'
You get actual row count and a counter. The Official documentation says the following about this field:
If the value of modified column exceeds “rows / 16” or 2000000000, the
statistics recalculation is done when innodb_stats_auto_update == 1.
We can estimate the oldness of the statistics by this value.
So this counter wraps every once in a while, but you can make a checksum of the number of rows and the counter, and then with every modification of the table you get a unique checksum. E.g.:
SELECT MD5(CONCAT(rows,'_',modified)) AS checksum FROM information_schema.innodb_table_stats WHERE table_schema='db' AND table_name='table';
I was going do upgrade my servers to Percona server anyway so this bounding is not an issue for me. Managing hundreds of triggers and adding fields to tables is a major pain for this application, because it's very late in development.
This is the PHP function I've come up with to make sure that tables can be checksummed whatever engine and server is used:
function checksum_table($input_tables){
if(!$input_tables) return false; // Sanity check
$tables = (is_array($input_tables)) ? $input_tables : array($input_tables); // Make $tables always an array
$where = "";
$checksum = "";
$found_tables = array();
$tables_indexed = array();
foreach($tables as $table_name){
$tables_indexed[$table_name] = true; // Indexed array for faster searching
if(strstr($table_name,".")){ // If we are passing db.table_name
$table_name_split = explode(".",$table_name);
$where .= "(table_schema='".$table_name_split[0]."' AND table_name='".$table_name_split[1]."') OR ";
}else{
$where .= "(table_schema=DATABASE() AND table_name='".$table_name."') OR ";
}
}
if($where != ""){ // Sanity check
$where = substr($where,0,-4); // Remove the last "OR"
$get_chksum = mysql_query("SELECT table_schema, table_name, rows, modified FROM information_schema.innodb_table_stats WHERE ".$where);
while($row = mysql_fetch_assoc($get_chksum)){
if($tables_indexed[$row[table_name]]){ // Not entirely foolproof, but saves some queries like "SELECT DATABASE()" to find out the current database
$found_tables[$row[table_name]] = true;
}elseif($tables_indexed[$row[table_schema].".".$row[table_name]]){
$found_tables[$row[table_schema].".".$row[table_name]] = true;
}
$checksum .= "_".$row[rows]."_".$row[modified]."_";
}
}
foreach($tables as $table_name){
if(!$found_tables[$table_name]){ // Table is not found in information_schema.innodb_table_stats (Probably not InnoDB table or not using Percona Server)
$get_chksum = mysql_query("CHECKSUM TABLE ".$table_name); // Checksuming the old-fashioned way
$chksum = mysql_fetch_assoc($get_chksum);
$checksum .= "_".$chksum[Checksum]."_";
}
}
$checksum = sprintf("%s",crc32($checksum)); // Using crc32 because it's faster than md5(). Must be returned as string to prevent PHPs signed integer problems.
return $checksum;
}
You can use it like this:
// checksum a signle table in the current db
$checksum = checksum_table("test_table");
// checksum a signle table in db other than the current
$checksum = checksum_table("other_db.test_table");
// checksum multiple tables at once. It's faster when using Percona server, because all tables are checksummed via one select.
$checksum = checksum_table(array("test_table, "other_db.test_table"));
I hope this saves some trouble to other people having the same problem.