I have "full-load-and-cdc" migrations tasks set up to track changes made in MySQL database and replicate changes to AWS S3. The cluster is InnoDB and I connect to one of the replicas. Sometimes, it happens that instance has to be detached from cluster and then added back as clone. In this situation all replication tasks fail with errors like this: "Could not find first log file name in binary log index file".
So, it looks like BinLog file with a specific name doesn't exist anymore. What could I do in a situation like this and be sure I continue replication from last checkpoint?
Related
I just wanted to know use of this type of file "DESKTOP-7JFP5MF-bin.000001" in data folder mysql server in windows 10?
have a look on below files.
what is the use of these files?
what if we delete these files?
Thanks
These files are the binary log, which is a sequential log of changes to your database.
It is used primarily for replication. A replica MySQL Server instance downloads these files and applies the same changes in the same order to the replica instance. The replica instance may be restarted or go offline temporarily, and when it reconnects to the source it will resume downloading where it left off. So it can be handy to keep the binary log files around for some time.
Typically they are set to expire automatically after a day (this is configurable). If a given log file is expired and deleted on the source instance before the replica has downloaded it, the replica will miss some updates. It needs all the logs to be a true replica, so if it misses some, it's essentially a failed replica and needs to be scrapped and reinitialized with a fresh copy from the source.
So if you delete these files, you'll spoil the replica.
Read https://dev.mysql.com/doc/refman/8.0/en/binary-log.html and its subsections for more information on the binary log.
Using AWS RDS you can restore your database using Point-in-time recovery to get it back to a certain time.
My question is, if someone deleted a bunch of data or dropped a table and you need to recovery to the bin log statement just before the deletion occurred, how do you go about doing that using AWS RDS PITR?
Using MySQL installed on an EC2 instance, you would be able to export the bin log files and find the position just before the deletion happened, and restore to that point in time, however I don't see that functionality available within AWS RDS.
From what I can see, you have to just keep guessing at the time you think the deletion happened?
I'm trying to create a DataBase for a very little program that just I and 2 more friends use, so the database will be little.
What I want to do is to share this database with these 2 friends, so I thought about storing the data in a OneDrive shared folder.
At the moment, I'm using a txt file as a "database". Which is placed in a shared OneDrive folder, so when my friends execute the program, it can read the data from there and make it sort of real time "online".
The thing is that I can't find the my.ini file in my
C:\Program Files\MySQL\MySQL Server 8.0\
directory, so I can't change the data folder.
Another trouble I have is that I also don't have the Data folder in that directory, instead I've found it in this one:
C:\Program Files\MySQL\MySQL Workbench 8.0 CE
Is it possible to do what I want to do? And how should I proceed?
Do you think I should be using MySQL 5.x version?
Thanks
If you want to share a database with a couple of other users, putting the datadir in OneDrive isn't going to work.
MySQL stores data in the files under the datadir, that's true. They're stored in files called tablespaces which have the .ibd extension.
But as you do INSERT/UPDATE/DELETE operations, data is temporarily stored in memory and in the transaction logs (ib_logfile*. MySQL has to do a delicate coordination between these, to save data in a durable way, while ensuring good performance. It works well, but only if the MySQL Server is the only process writing to the files.
OneDrive is not coordinating with MySQL at all. It will periodically check for files that have changed since it last did a sync. The interval OneDrive checks files is about every 10 minutes, and this is not configurable.
OneDrive may choose to sync the files at a moment after you've done some INSERT/UPDATE/DELETE operations, and the data is modified in RAM, but hasn't yet been updated in the tablespace files on disk.
Once you've committed a change, it must also be safely stored in the transaction log, even if it isn't updated in the tablespace. But if your friends receive the files in this state (transaction log contains changes which aren't present in the tablespace), they can reconstruct the data from your RAM that they didn't get. This is called InnoDB crash recovery. MySQL Server does if automatically if it starts up and finds the transaction log contains changes that aren't in the tablespace. It assumes you had a sudden reboot and lost what was in RAM.
If your friends try to just keep their MySQL Server running continuously, reading a datadir that is simultaneously being updated by their one OneDrive, it'll basically overwrite their files and MySQL will become confused. It only checks to see if it should do crash recovery as MySQL Server starts. So if the files change in unexpected ways while MySQL Server is already running, it'll just conclude that your hard drive has gotten corrupted. It'll probably report a fatal error and shut down MySQL.
Also if your friends try to make changes of their own to the database, their changes would conflict with the updates from OneDrive. Then their attempts to overwrite files would eventually be synchronized in the opposite direction via OneDrive, and would eventually corrupt your database too. This would happen without warning at 10 minute intervals, whenever OneDrive chose to do its file sync.
So I'm afraid OneDrive is not the solution to share your database.
Alternatives that do have a chance of working include:
Hosting a single instance of MySQL Server on a website that you all share, and giving each of you a client that can use the database. A popular free client is phpMyAdmin. That way there'd be only one instance of MySQL Server, one datadir, even though you each would be concurrent clients reading and writing data. This is the simplest solution, most likely to work.
Exporting the data from your MySQL Server instance using mysqldump periodically, and putting the export file on OneDrive, or sending it to your friends through email or any other means. Then they would have to import that data to their MySQL Server manually. This would overwrite any changes they had done to their database, but it wouldn't appear as corruption. If they want to send changes back to you, they could do a similar operation: export their database, put the dump file on OneDrive, then you'd get their dump file and import it to your MySQL Server instance, overwriting any changes you had done locally since the last time you sent your export to your friends.
Use a MySQL add-on for multi-server synchronous replication, such as InnoDB Group Replication or Percona XtraDB Cluster. But this is probably too complex for you to set up if you're a newbie with MySQL.
Every night the MySQL database is saved by a cronjob which uses mysqldump.
During the day, when the CakePHP application is running, I would like to have a logfile working, that could be used as backup in case of a damage happening during the day.
For recovery, it would be necessary to first run the recovery from the mysqldump that was established at night. And secondly, run the recovery from the logfile to get the database changes from the current day.
Does there exist such a logfile possibility and where or how could I get it?
Or are there any other ways to get a proper backup?
It's built into MySQL, you wouldn't do this in your applictation typically.
enter link description here
The binlogs do what you say - it is a binary file that contains every transaction that hits the database. So, if you had a nightly back up and the server crashed, you would recover the bin logs, export the transaction logs from the datetime of the last transaction in the database and essentially re-run or re-play every transaction from that point.
And having used it >10 times before ... it is awesome... but you have to turn in on in your my.conf file (google that) and manage the binlog files as they do get big on busy servers.
Objective: to be able to synchronize 2 linux server realtime.
My concern is after using Rsync to mirror the mysql server. The only thing it wasnt able to synchronize is the entries (ie. inserting data to the database using the insert query). How will I be able to solve this?
Things I've done:
scp the keys of the 2 server so that password wont be asked for each transaction
I used
rsync -avc /var/lib/mysql/ root#10.1.99.XXX:/var/lib/mysql/
to sync the database/tables, but wasn't able to sync the entries.
Isubaki,
It's not quite as simple as just using rsync, as mysql may have the files open at the time you are pushing them across. Linux will do the file copy ok, but using this technique, the table is locked in memory until the database is restarted.
I do have a script that will do the sync part, but it does require a database restart, which may not be what you want (you mention realtime sync)