I am writing a db logging ruby gem which will simply take out a job from a Beanstalk queue and write it in the DB.
That is one process on Server A puts a job (that it wants to log) in the Beanstalk queue on Server B, and my logging process on Server B takes it out and writes it to the mysql DB on Server B.
I want to know if this is worth it?
Is putting a job in the Beanstalk queue faster than writing to the DB. Or can my process that wants to log to DB directly write it to DB instead of using the logging process.
Note that both the beanstalk server and DB are on another server.
Beanstalk internally makes a socket call from Server A to Server B.
I believe mysql would need to do the same as well?
So therefore is mysql to another server going to be slower than putting in the beanstalk queue.
It'll be much faster, primarily because Beanstalkd jobs, by default, are stored in-memory and are lost if, for example, you lose power on your server, whereas MySQL is a strongly ACID-compliant relational database, and hence will go to a lot of effort and flush each of your logs to disk.
I think you'll find that, after your do some benchmarking with a lot of logs being made by your system, that disk I/O will be your limiting factor, rather than the speed of TCP/IP sockets. Your current system's advantage is that when server A files a log on Server B's beanstalkd instance it takes up very little of Server A's time, and Server B can periodically flush our many logs at once from beanstalkd to MySQL, making the process more efficient. The disadvantage is that, the more you batch up the logs, the more logs you will lose in the event of a software / power failure, unless you use beanstalkd's "-b" parameter which makes jobs durable by writing them to disk (and hence making the process slower).
Of course, the only way to truly settle this question is to benchmark!
Related
I want to test Node and Deno and try to redirect users via proxy to one MySQL DB.
How will it impact the database?
Can some timestamp conflicts via CRUD operations arise or does MySQL have some mechanism to cope with connections from multiple servers?
What about performance or memory footprint of the database in RAM? Will it be occupying the same amount of space as if there was only one server requesting the database to CRUD something?
What would happen if I added another server that will connect to the DB, for example, java or Go server?
It will virtually have no impact on the database other than having any other concurrent processes connecting to it.
This is not a deno issue but rather a database issue.
The exact same problems can happen even with your current single Node.js instance, because the nature of all systems these days is concurrent/parallel.
You might as well replace the Deno app with another Node.js instance, Java, etc. Or even your current Node.js app.
Data in a database can change once you loaded it to the client, and it is up to you to implement the code that will handle such scenarios.
The fact that MySQL is not "ACID" is neither negative nor relevant in and of itself because it is doesn't have context.
If you need complete absolute integrity on a registry make sure you lock it when you select it, but there will be a trade off.
I have been working on the server migration of a legacy ecommerce application using PHP 5.6.
The switch involved two Dedicated 32 servers from Linode.
One server is for NginX + PHP and the other is for MySQL only.
The legacy application leverages memcached.
After the switch, I can see a heavy internal traffic caused due to private inbound and outbound connections.
So far this element didn't cause any problem on performance.
However, I was under the impression that the queries would be cached on the local machine, and not on the remote.
Because if the query is cached on the remote host, it sill has to transmit the result set over the private network, instead of retrieving from RAM or the local SSD.
Am I assuming this wrong?
It may be that I am missing the point where the private inbound traffic is more beneficial for overall performance when compared to a local cache.
MySQL has a feature called the Query Cache, but this caches query result sets in the mysqld server process, not on the client. If you run the exact same query again after the result has been cached in the Query Cache, it will copy the result from the Query Cache and avoid the cost of running the query again. But this will not avoid the time to transfer the result across the network from mysqld to your PHP application.
Also keep in mind that the MySQL Query Cache is being deprecated and retired.
Alternatively, your application may store data from query results in memcached, but typically this would be done by the application code (I know there are UDF's to read and write memcached from MySQL triggers, but this is a bad idea).
If your memcached service is not on the same host as your PHP code, it would result in network transfer twice: Once when querying the data from MySQL the first time, then again transferring the data into memcached, then later every time you fetch the cached data out of memcached.
PHP also has some features to do in-memory caching, such as APCu. I don't have any experience with this, and it's not clear from a brief scan of the documentation where it stores cached data.
PHP is designed to be a "shared nothing" language. Every PHP request has its own data, and data doesn't normally last until the next request. This is why a cache is typically not kept in PHP memory. Applications rely on either memcached or the database itself, because those will hold data longer than a single PHP request.
If you have a fast enough network, it shouldn't be a high cost to fetch items out of a cache over a network. The performance architects at a past job of mine developed this wisdom:
"Remote memory is faster than local storage."
They meant that if the data is in RAM on a server, then reading it from RAM even with the additional overhead of transferring it across a network is usually better than reading the data from persistent (disk) storage on the local host.
i have a local xampp, with php application runing on php5.6 and data base mysql are remote.
the issue is my application open very slow on my browser, is there an way to improve speed ?
mysql is on ssd server (fast)...
thanks
It's always going to be slower over a remote connection. How much really depends on a number of factors related to your connection, not really to MySql.
Few things to consider - do you really need to be working off the remote db? Can you use a local copy and sync changes later? Might even be faster to have your code remote and suffer the slightly longer save times when updating your code. Highly situational though depending on your dev set-up.
Another option which would be a bit more complicated is to set up your local db as a read-only slave and at least that way you get more of async update -- i.e., your local db may lag behind remote master a bit but any reads you do locally will be back to your "native" local performance. You would have the additional complexity of setting up Master/Slave replication with different connections for read/write but that may be something you want to do for production anyway. (You can do master/master but I wouldn't recommend it over a remote connection.)
I have a Rails 4.x application running on server A and MySQL on server B.
Using ab to do a load test of my API calls I notice that the MySQL server is showing CPU activity. So I go back to the code and check, but no SQL statements are triggered, to be sure I also deactivate all before filters, but still the MySQL server shows CPU load.
I went to MySQL and run
show processlist;
but that also shows no active SQL statements
Why would there be load on the DB server?
A Rails application initializes connection pools to the configured database on app load and also loads basic schema data for each ActiveModel defined to populate runtime mappings from the DB to instances of that model.
These connections/queries will happen as soon as you have loaded the application and running traffic.
If this is not what is responsible for the activity on your database server, you will need to use other tools to see what is responsible. For example, NewRelic's system monitoring tools are great for snapshotting CPU/memory usage over time correlated to what processes were running. This will help you rule out MySQL itself using resources vs. other things running on the DB server.
According to this article, storage engines like Innodb may have their own per thread and/or global memory allocations which is probably accounting for the CPU overhead. If this is a stock (non-tuned) MySql install, you're probably just seeing baseline CPU activity. The article mentions a number of places to look that might indicate areas that can be tuned to reduce this footprint.
I have an application that runs on server A and the database is on the same server
there is a backup server B which I use in case the server A is down
the application will remain unchanged but the data in the DB is changing constantly
Is there a way to synchronize those 2 databases real-time automatically?
currently I wait till all the users are gone so I can manually backup and restore in the backup server.
Edit: When I said real-time I didn't mean it literally, I can handle up to one hour delay but the faster sync the better.
My databases are located on 2 servers on the same local network.
2 of them are SQL Server 2008, the main DB is on windows server 2008
the backup is on windows server 2003
A web application (intranet) is using the DB
I can use sql agent (if that can help)
I don't know what kind of details could be useful to solve this, kindly tell me what can help. Thanks.
Edit: I need to sync all the tables and table only.
the second database is writable not read-only
I think what you want is Peer to Peer Transactional Replication.
From the link:
Peer-to-peer replication provides a scale-out and high-availability
solution by maintaining copies of data across multiple server
instances, also referred to as nodes. Built on the foundation of
transactional replication, peer-to-peer replication propagates
transactionally consistent changes in near real-time. This enables
applications that require scale-out of read operations to distribute
the reads from clients across multiple nodes. Because data is
maintained across the nodes in near real-time, peer-to-peer
replication provides data redundancy, which increases the availability
of data.