I have read MongoDB's replication docs and MySQL's Cluster page however I cannot figure out how to configure my web apps to connect to database.
My apps will have connection information, database host, username, password etc, however, even if there is multi server function, should I need a big master that has a fixed ip that distirbutes the load to servers? Then, how can I prevent single-point of failure? Is there any common approaches to that problem?
Features such as MongoDB's replica sets are designed to enable automatic failover and recovery. These will help avoid single points of failure at the database level if properly configured. You don't need a separate "big master" to distribute the load; that is the gist of what replica sets provide. Your application connects using a database driver and generally does not need to be aware of the status of individual replicas. For critical writes in MongoDB you can request that the driver does a "safe" commit which requires data to be confirmed written to a minimum number of replicas.
To be comprehensively insulated from server failures, you still have to consider other factors such as physical failure of disks, machines, or networking equipment and provision with appropriate redundancy. For example, your replica sets should be distributed across more than one server or instance. If all of those instances are in the same physical colocation facility, your single point of failure could still be the hopefully unlikely (but possible) case where the colocation facility loses power or network.
Related
I want to test Node and Deno and try to redirect users via proxy to one MySQL DB.
How will it impact the database?
Can some timestamp conflicts via CRUD operations arise or does MySQL have some mechanism to cope with connections from multiple servers?
What about performance or memory footprint of the database in RAM? Will it be occupying the same amount of space as if there was only one server requesting the database to CRUD something?
What would happen if I added another server that will connect to the DB, for example, java or Go server?
It will virtually have no impact on the database other than having any other concurrent processes connecting to it.
This is not a deno issue but rather a database issue.
The exact same problems can happen even with your current single Node.js instance, because the nature of all systems these days is concurrent/parallel.
You might as well replace the Deno app with another Node.js instance, Java, etc. Or even your current Node.js app.
Data in a database can change once you loaded it to the client, and it is up to you to implement the code that will handle such scenarios.
The fact that MySQL is not "ACID" is neither negative nor relevant in and of itself because it is doesn't have context.
If you need complete absolute integrity on a registry make sure you lock it when you select it, but there will be a trade off.
We are building a small advertising platform that will be used on several client sites. We want to setup multiple servers and load balancing (using Amazon Elastic Load Balancer) to help prevent downtime.
Our basic functions include rendering HTML for ads, recording click information (IP, user-agent, location, etc.), redirecting traffic with their click ID as a tracking variable (?click_id=XX), and basic data analysis for clients. It is very important that two different clicks don't end up with the same click ID.
I understand the basics of load balancing, but I'm not sure how to setup the MySQL server(s).
It seems there are a lot of options out there: master-master, master-slave, clusters, shards.
I'm trying to figure out what is best for us. The most important aspects we are looking for are:
Uptime - if one server goes down, automatically get picked up by another server.
Load sharing - keep CPU and RAM usage spread out.
From what I've read, it sounds like my best option might be a Master with 2 or more slaves. The Master would not be responsible for any HTTP traffic, that would go to the slaves only. The Master server would therefore only be responsible for database writes.
Would this slow down our click script? Since we have to insert first to get a click ID before redirecting, the Slave would have to contact the Master and then return with the data. Right now our click script is lightning fast and I'd like to keep it that way.
Also, what happens if the Master goes down? Would a slave be able to serve as the Master until the Master was back online?
If you use Amazon's managed database service, RDS, this will take a lot of the pain out of managing your database.
You can select the multi-AZ option on your master database instance to provide a redundant, synchronously replicated slave in another availability zone. In the event of a failure of the instance or the entire availability zone Amazon will automatically flip the A record pointing to your master instance to the slave in the backup AZ. This process, on vanilla MySQL or MariaDB, can take a couple of minutes during which time your application will be unable to write to the database.
You can also provision up to 5 read replicas for a MySQL or MariaDB instance that will replicate from the master asynchronously. You could then use an Elastic Load Balancer (or other TCP load balancer such as HAProxy or MariaDB's MaxScale for a more MySQL aware option) to distribute traffic across the read replicas. By default each of these read replicas will have a full copy of the master's data set but if you wanted to you could attempt to manually shard the data across these. You'd have to have some more complicated logic in your application or the load balancer to work out where to find the relevant shard of the data set though.
You can choose to promote a read replica into a stand alone master which will break replication to the master and give you a stand alone cluster which can then be reconfigured as to your previous setup (or something different if you want and just using the data set you had at the point of promotion). It doesn't sound like something you need to do here though.
Another option would be to use Amazon's own flavour of MySQL, Aurora, on RDS. Aurora is completely MySQL over the wire compatible so you can use whatever MySQL driver your application already uses to talk to it. Aurora will allow up to 15 read replicas and completely transparent load balancing. You simply provide your application with the Aurora cluster endpoint and then any writes will happen on the master and any reads will be balanced across however many read replicas you have in the cluster. In my limited testing, Aurora's failover between instances is pretty much instant too so that minimises down time in the event of a failure.
I am developing a CI app for a client with MySQL as back end.
The client has 8 shops. For each shop, there is a local server, and additionally, there is one central server, which is placed at the Head Quarters (HQ).
The problem i am facing is,
At the time of network failure at a shop, the billing and other
processes should work; without central server. Once the network is
back, they need it to sync with HQ Server.
Those who clicking on too board on close can you please say what all details you need? I am not getting that part thats why, please add that as comment, i will do it
This is a common problem in shop environment, you should cope to this requirements having basic data into the single store (eg. items, promotions, parameters) and setting up a database synchronization between local stores and center db ...
If you have MySQL in each store and as central DB, you can set up a MySQL replication, otherwise take a look at SymmetricDS that is in short the missing component that can perfectly fit your scenario, since :
SymmetricDS is open source software for both file and database
synchronization with support for multi-master replication, filtered
synchronization, and transformation across the network in a
heterogeneous environment. It supports multiple subscribers with one
direction or bi-directional, asynchronous data replication. It uses
web and database technologies to replicate data as a scheduled or near
real-time operation. The software was designed to scale for a large
number of nodes, work across low-bandwidth connections, and withstand
periods of network outage. It works with most operating systems, file
systems, and databases, including Oracle, MySQL, MariaDB, PostgreSQL,
MS SQL Server (including Azure), IBM DB2, H2, HSQLDB, Derby, Firebird,
Interbase, Informix, Greenplum, SQLite (including Android), Sybase
ASE, and Sybase ASA (SQL Anywhere) databases.
I have an application that runs on server A and the database is on the same server
there is a backup server B which I use in case the server A is down
the application will remain unchanged but the data in the DB is changing constantly
Is there a way to synchronize those 2 databases real-time automatically?
currently I wait till all the users are gone so I can manually backup and restore in the backup server.
Edit: When I said real-time I didn't mean it literally, I can handle up to one hour delay but the faster sync the better.
My databases are located on 2 servers on the same local network.
2 of them are SQL Server 2008, the main DB is on windows server 2008
the backup is on windows server 2003
A web application (intranet) is using the DB
I can use sql agent (if that can help)
I don't know what kind of details could be useful to solve this, kindly tell me what can help. Thanks.
Edit: I need to sync all the tables and table only.
the second database is writable not read-only
I think what you want is Peer to Peer Transactional Replication.
From the link:
Peer-to-peer replication provides a scale-out and high-availability
solution by maintaining copies of data across multiple server
instances, also referred to as nodes. Built on the foundation of
transactional replication, peer-to-peer replication propagates
transactionally consistent changes in near real-time. This enables
applications that require scale-out of read operations to distribute
the reads from clients across multiple nodes. Because data is
maintained across the nodes in near real-time, peer-to-peer
replication provides data redundancy, which increases the availability
of data.
I am studying the new Amazon RDS product and it seems it can be scaled only vertically (i.e. put a stronger server).
Did anyone see a possibility to configure multiple instances so that one is master and the other/s is/are replication slaves?
Same question asked (and answered) here http://developer.amazonwebservices.com/connect/thread.jspa?threadID=37823
Looks like there are plans for Master-Master HA or similar but that's not the same a replicated scale-out offering.
According to the FAQ it is possible now, see http://aws.amazon.com/rds/faqs/#86 :
Q: What types of replication does
Amazon RDS support and when should I
use each?
Amazon RDS provides two distinct replication options to serve different
purposes.
If you are looking to use replication to increase database
availability while protecting your
latest database updates against
unplanned outages, consider running
your DB Instance as a Multi-AZ
deployment. When you create or modify
your DB Instance to run as a Multi-AZ
deployment, Amazon RDS will
automatically provision and manage a
“standby” replica in a different
Availability Zone (independent
infrastructure in a physically
separate location). In the event of
planned database maintenance, DB
Instance failure, or an Availability
Zone failure, Amazon RDS will
automatically failover to the standby
so that database operations can resume
quickly without administrative
intervention. Multi-AZ deployments
utilize synchronous replication,
making database writes concurrently on
both the primary and standby so that
the standby will be up-to-date in the
event a failover occurs. While our
technological implementation for
Multi-AZ DB Instances maximizes data
durability in failure scenarios, it
precludes the standby from being
accessed directly or used for read
operations. The fault tolerance
offered by Multi-AZ deployments make
them a natural fit for production
environments; to learn more about
Multi-AZ deployments, please visit
this FAQ section.
If you are looking to take advantage of MySQL 5.1’s built-in
replication to scale beyond the
capacity constraints of a single DB
Instance for read-heavy database
workloads, Amazon RDS makes it easier
with Read Replicas. You can create a
Read Replica of a given “source” DB
Instance using the AWS Management
Console or CreateDBInstanceReadReplica
API. Once the Read Replica is created,
database updates on the source DB
Instance will be propagated to the
Read Replica. You can create multiple
Read Replicas for a given source DB
Instance and distribute your
application’s read traffic amongst
them. Unlike Multi-AZ deployments,
Read Replicas use MySQL 5.1’s built-in
replication and are subject to its
strengths and limitations. In
particular, updates are applied to
your Read Replica(s) after they occur
on the source DB Instance
(“asynchronous” replication), and
replication lag can vary
significantly. This means recent
database updates made to a standard
(non Multi-AZ) source DB Instance may
not be present on associated Read
Replicas in the event of an unplanned
outage on the source DB Instance. As
such, Read Replicas do not offer the
same data durability benefits as
Multi-AZ deployments. While Read
Replicas can provide some read
availability benefits, they and are
not designed to improve write
availability.
With Amazon RDS, you can use Multi-AZ deployments and Read Replicas
in conjunction to enjoy the
complementary benefits of each. You
can simply specify that a given
Multi-AZ deployment is the source DB
Instance for your Read Replica(s).
That way you gain both the data
durability and availability benefits
of Multi-AZ deployments and the read
scaling benefits of Read Replicas.