MS SQL Server 2008 Standard, ShadowProtect Server Edition 4.0.0.5885 --
On Friday, our client discovered that records were missing from the database. I discovered that the Thursday night SQL backup contained all the missing records. User error is ruled out for multiple reasons.
All missing records fall within an 8-day range
The date range began 22 days before Friday and ended 14 days before Friday
All adds and all changes made during the 8-day range are missing from 14 separate tables
All missing records are present in the Thursday 11pm backup
The application logs show no unusual incidents as far as I can see.
I find nothing unusual in the Applications list in the MS SQL Server Event Viewer.
We are running ShadowProtect Server to make image backups of the 2 server drives every hour. The same sort of incident occurred 4 months ago.
ShadowProtect runs an hourly backup of the database.
One theory is that the ShadowProtect Server 4 disk image software, which runs hourly differential backups, somehow caused the data loss during its 9:00 am Friday backup. I am not aware of any other activity. other than normal user accesses, between the normal 11 pm Thursday database backup and the discovery of missing records on Friday.
Thank you for you help. As you can imagine, the client is very concerned.
If you want to know what is deleting the records or when they got deleted, the database should have audit tables set up that include the usernames and dates of changes. Then you can look at the audit logs to see when reords were deleted and by whom or what process. All databases that contain business critical information should have auditing. Unfortunately, after the event has happened is too late to find out who did it this time through auditing. You might be able to find someone third party product to look through the transaction logs and at least might find out what time the deletes happened if not who. You also should be doing transaction log backups every 15 minutes or so.
I'm not familar with the ShadowProtect server, but the missing data sounds exactly like a script was run (and cacade delete was turned on) and seems unlikely to be the ShadowProtect server. If it was interfering, I would expect a more random kind of change that one that can easily be done by a sql query. Do you allow direct access to your tables? You could have someone trying to harm the data or hide fradulent activities. Threats to the data are not always from outside sources or applications that would be in the event log. Who has access to the delete the data in the database on production? I would suspect a disgruntled employee.
We never did find out the cause of the lost records. We reinstalled the database in another MS SQL Server instance, upgraded the database to a new release, and migrated the data from the old database to the new one. That seems to have fixed the issue.
I had similar issue on VM. The error was was pointing to the database but SQL Server wasn't actually running. For some unknown reason SQL Server has stopped. It seems that VM was restarted and the service didn't start automatically.
After starting the service with Windows Administrative Tools, server was back on and the database was there
Related
I'll try to keep this as succinct as possible.
I was asked by my boss to make a duplicate of one of our apps with an empty DB but the same schema/structure for a venture he is doing with another company.
So like any good developer, I backed everything up on the server and made a snapshot of the DB in the RDS dashboard. This was at approx 10 pm.
Then I spun up a new ec2 and MySQL instance for the new venture and got everything networked and running there.
That's when everything went wrong.
Somehow my production DB for our app appeared to be a mirror of the new one that I made for the other company and it had no data. (Im still looking into how this happened but suspect that MySQL Workbench is to blame)
At this time I am feeling fine and dandy and just go grab the snapshot I made a few minutes earlier and restore it.
Low and behold the data is old and outdated. Now I'm freaking out but I knew that I had a manual backup I had done earlier in the day before running some pretty large insert scripts (10 am) so I just ran an import from that and the data seemed to be correct for the time that I backed it up, 10am that day.
So not all was lost. Only a day worth of updates and inserts but that is a lot of money for the company.
To get to the main question, How is it remotely even possible that a snapshot taken at 10 pm has older data than a manual backup done 12 hours earlier?
I have hosted my MySql database on Amazon RDS, and It has last automated snapshot. (e.g. yesterday midnight). Now situation is, I have deleted some of records from a very important table accidentally, and would like to recovery it. I have no additional backup since yesterday midnight. (as mention earlier). Now How should I recover data without taking any downtime?
How do I use point-in-time data recovery?
If someone need more information let me know and sorry for my poor explanation.
Point in time recovery allows you to create an additional RDS instance, based on the data as it existed on your instance at any specific point in time you choose between the oldest available automated backup and approximately 5 minutes ago. All you have to do is select what date and time you need.
There is no disruption or change to your running instance.
The process creates a new instance, which you connect to, collect the data you need in order to get your production system back the way it should be, and then destroy the new instance. Or, depending on what went wrong for you, you could also switch your application to this new instance and destroy the old one, though it seems unlikely that this is what you would want to do. But you can do either.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html
I have no additional backup since yesterday midnight.
Point in time recovery doesn't care. RDS preserves the snapshots as well as a complete, timestamped log of everything that changed between the snapshots. These logs are archived in an area that is not accessible to you... but they are there. RDS will automatically load the most recent snapshot that is earlier than the point-in-time you select, and then use the logs to roll the new instance's data forward in time to the target time. When the process is complete, your new instance will contain exactly the data that was present on the old instance at the point in time that you selected.
lets assume I have one table contain 10 records at midnight. which is exists in backup/snapshot.
Stop thinking about what is in the snapshot. It doesn't matter.
Next day morning I have added 5 another records at 10:00pm. After half an hour (at 10:30) I have deleted 2 records from them antecedently.
Perform a point in time recovery, selecting any point between 10:00 and 10:30 -- a point in time when the records were in your database.
Point in time recovery creates a new instance, which contains all your data exactly as it existed at the time you selected. Connect to that new instance manually, fetch the missing rows, insert them back into your live/main/production database, and then the newly-created instance can be destroyed because it is no longer needed.
Do not assume this process is complicated or difficult.
I have 61 websites on a live Magento instance. 10 or so are no longer used but we get about a thousand orders per week and have been running on Magento for over 2 years.
I went to delete one of the websites in the Manage Stores, and there is a little drop down for creating a DB backup that I left set to yes, thinking originally that it meant backup the reference in the DB at first glance.
It has been about an hour now and our sites are virtually down as no one can place orders, and the admin is unaccessible.
Is there a safe way to stop this process?
Is there a way to at least trace progress to see if there is some sort of SQL lock? I do not want to wait for hours only to find out that it will never finish. Ive read around here that the built in Magento backup tools are not good at all.
I'm facing a very complex situation with my software databases on a specific server. This is the second time this problem happend on this server.
Basically while user where working on the system, with no apparently reason, all the data from three user databases where lost.
The 3 databases where put on a previuos state from 1.5 month ago where all the databases changes made to this databases are lost, yes LOST! I had to restore a backup for all the databases but the data from passed two dates after the last backup where lost. The first time this happend, was with SQL Server Express 2008. Now happends on SQL Server 2008 R2 Standard.
I suspect a hardware / OS (Hard drive, RAM, etc) problem with the server itself, but no other data where lost (data files, etc) on SQL Server data.
The only lead i have to discover what happend is the Sql Server LOG, where i see some strange entrys:
1) All the log previous to 06/08/2015 11:55 (the exact moment where the problem apear) is lost.
2) An error during decryption, which don't know what it means.
3) A "Recovery complete" message
4) After that I see on the log constant message from CHECKDB over my users database (a lot of them) wich no one execute.
Does anyone have any idea of what could be hapenning here?
Is this a Hardware issue like I suspect?
Thanks!
We have a SQL Server 2008 R2 database that backs up transaction logs every now and then. Today there was a big error in the database caused at around 12am... I have transaction logs up to 8am and then 12am - 16pm - etc.
My question is: can I sort of reverse-merge those transaction logs into database, so that I return to the database state at 8am?
Or is my best chance to recover an older full backup and restore all transaction logs up to 8am?
The first option is preferable since full backup has been performed a bit of a while ago and I am afraid to f*ck things up restoring from there and applying trn logs. Am I falsely alarmed about that? Is it actually possible for anything bad to happen if going by that scenario (restoring the full backup and applying trn logs)?
The fact that you don’t create regular transaction log backups doesn’t affect the success of the recovery process. As long as your database is in the Full recovery model, the transactions are stored in the online transaction log and kept in it until a transaction log backup is made. If you make a transaction log backup later than usual, it only means that the online transaction log may grow and that the backup might be bigger. It will not cause any transaction history to be lost.
With a complete chain of transaction log backups back to 8 AM, you can successfully roll back the whole database to a point in time.
As for restoring the full backup and applying trn logs – nothing should go wrong, but it’s always recommended to test the scenario on a test server first, and not directly in production
To restore to a point in time:
In SSMS expand Databases
Right-click the database, select Tasks | Restore| Database
In the General tab, in the Backup sets the available backups will be listed. Click Timeline
Select Specific date and time, change the Time interval to show a wider time range, and move the slider to the time you want to roll back to
You can find more detailed instructions here: How to: Restore to a Point in Time (SQL Server Management Studio)
Keep in mind that this process will roll back all changes made to the database. If you want to roll back only specific changes (e.g. only recover some deleted data, or reverse wrong updates), I suggest a third party tool, such as ApexSQL Log
Reverting your SQL Server database back to a specific point in time
Restore a database to a point in time
Disclaimer: I work for ApexSQL as a Support Engineer