I have a scenario, where I need to capture the ongoing data changes (CDC) from MySQL and push it to the AWS data lake (S3). The challenge here is that the MYSQL binlog is disabled. We don't have control over the database configuration as the MySQL database server is in the vendor's scope and we cannot enable the binlog.
Hence we are not able to use services like Debezium and AWS DMS for which binlog is mandatory.
So, I'm looking for a low-code opensource tool or AWS services to push ongoing changes from MySQL to AWS datalake without relying on the MYSQL binlog file.
Note: Full data load is working with AWS DMS, however the problem is only with the CDC
Related
I have a running RDS Aurora MySQL 8.0.23 cluster running in production. The database is unencrypted and I need to enable encryption for it. As far as I understand, this is not possible to do directly. The procedure I am evaluating is:
Create a read replica on the current cluster.
Stop replication on replica and annotate binlog filename and
position.
Promote the read replica to a new encrypted cluster (maybe it
requires to do a snapshot before).
Set up back replication with the original cluster using binlog file
and position annotated before.
Wait until replication lag is zero.
Redirect production traffic to the new cluster.
Stop replication.
[Optional] Delete old cluster.
I have two issues with the above procedure:
Once created the replica, running commands like SHOW SLAVE STATUS
or SHOW REPLICA STATUS return empty set, so I can't annotate
binlog file and position. Please note that replication is enabled on
the original cluster (binlog_format is set to ROW).
It seems I can't promote the Aurora read replica to a new cluster,
the option is missing on the available actions. But according to the documentation it should be possible.
Has anyone have feedback about the issues above? What is the current up-to-date procedure to encrypt an Aurora MySQL cluster with minimum downtime and no data loss?
From my understanding, Aws RDS facilitate backup for the mysql database, but it is not cheap.
While using docker image for mysql may save us more in terms of cost? Because we only need to download the docker image for dockerhub and directly use it for free(e.g. create an instance and run the container).
Is there another reason of using RDS other than facilitating backup for the database?
I list several features of RDS which may warrant using it over self-managed MySQL docker container on an EC2 insistence or ECS:
RDS is managed service, so all OS updates, MySQL patches are managed by AWS and you don't have to worry about them.
RDS supports storage auto-scaling - you can start with small db, and RDS will extend storage automatically as needed.
Point-in-time recovery allowing you to "rewind" your recent db changes.
Read replicas - you can create up to 5 read replicas of your database to off-load read intensive applications from your primary db instance.
Cross-region read replica - you can have your replica in different region which is good for disaster recovery (entire AWS region goes down)
Automated and manual backups, including backups to a different region.
IAM authentication to your db instead of regular username/password.
Multi-AZ - RDS can keep a stand-by replica of your primary database instance in different availability zone, for quick recovery if it fails.
CloudWatch integrated db metrics and logs.
RDS event notifications allow you for straight-forward development of automations e.g. invoke lambda automatically for every backup, or if something fails.
Easier integration with other services, e.g. use of RDS Proxy in Lambda functions.
All these and other features of RDS make it much more expensive then hosting a self-managed MySQL docker container. But if MySQL in docker container meets all your requirements, then there is no need to use RDS. You can always start with the docker, and if your data and requirements grow, you can migrate to RDS.
I am looking for a tool to sync data online from one MySQL server to another MySQL server, the binlog format of source MySQL server is MIXED.
The tools as I know, such as tidb-syncer, pt-table-sync, tidb-syncer need the source binlog format be ROW, pt-table-sync need the source binlog format be STATEMENT.
tidb-syncer is used to sync data from MySQL to TiDB, if you want to sync data from one MySQL to another MySQL why not just use mysql replication itself.
And you can change the binlog format if need.
I have read that Maxscale(BinLog Server) of MariaDB can be used for relaying the bin logs from a MySQL Cluster to a single BinLog Server, however i wanted to know if its possible to collect all the bin logs from different MySQL Cluster and persist on a single BinLog Server and no mysql slaves would be reading from it. In case its possible how are the conflicts like same database name in different MySQL Cluster, etc are handled?
The binlogrouter module in MaxScale is a replication proxy. It stores the binary logs of the master on the MaxScale server. Unlike a normal replication slave, the binlogrouter will not write its own binary log. This means that the binary logs on the MaxScale server will be identical to those on the original master server.
To collect binlogs from multiple clusters, you need to configure multiple binlogrouter services and point each one of them to a master server. The binary logs are stored separately for each configured service.
Here's an example configuration of a binlogrouter service and a listener:
[Replication-Router]
type=service
router=binlogrouter
version_string=10.0.17-log
router_options=server_id=4000,binlogdir=/var/lib/maxscale/,filestem=mysql-bin
user=maxuser
passwd=maxpwd
[Replication-Listener]
type=listener
service=Replication-Router
protocol=MySQLClient
port=3306
Read the Binlogrouter documentation for more information about the various options.
I've been playing with AWS EC2 and really like it. There is one drawback though, the instance could disappear due to hardware failure or whatever reason. This happened to me in my first week of operation. I was wondering whether there are good solutions to backup a MySQL database so that I don't lose my customer credentials?
You can transfer mysql database directly from EC2 machine to S3bucket but you will consume more cost for bandwidth and storage. You go for a third party application (which is safe) to backup your mysql or any plugins. Because they compress your data & encrypt and then save in S3 storage. Also, you can enable snap shot and take snap shots for volumes (hard drives)
I suggest you to use 'StoreGrid' backup software to backup your mysql database in EC2 machine. check this following link to know more about Online Backup Service on Amazon EC2/S3 http://storegrid.vembu.com/online-backup/amazon-ec2-s3-cloud-online-backup.php
Check this following link to configure MySQL database BACKUP http://storegrid.vembu.com/online-backup/mysql-backup.php?ct=1
Note: You have mentioned Hardware failure occurs often ! --- you can backup entire hard drives too using the above software.
I hope, now your MySQL data base is backed up from EC2 instance and stored in S3 storage safely.
Cheers !
Amazon now offers Relational Database Storage, that is, pre-configured EC2 instances, without any OS access to host MySQL (or Oracle, or T-SQL for real) for you, but aim to solve much of the availability, reliability and durability issues one faces when trying to host transactional data store yourself on a bare EC2 instance.
http://aws.amazon.com/rds/
"automated backups, DB snapshots, automatic host replacement, and Multi-AZ deployments"