Here is the senario:
We have a site running on NodeJS. Periodically, we pull some data from internet, analyze it, and update a MySQL Database.
My questions are:
What is the best practice to create a Linux daemon? gcc? Can I do it in PHP or other languages?
Since NodeJs will be accessing to the same Database, how can we create mutex?
How can we manage the daemon? For example If the daemon crashes, we want to restart it automatically.
You can use forever.js ... see How does one start a node.js server as a daemon process?. It answers your 1st and 3rd question. I guess you should have searched stack overflow or just have googled a bit !!
You can code a daemon in any language: C, C++, Ocaml, Haskell, ... (but I won't code it in PHP).
The most important in coding a daemon is to be sure the code is robust and fault-detecting.
Concurrent access to the database should be handled by the MySQL server.
If you only share resources by a shared database, you can use its transaction isolation guarantees to stop other processes seeing incomplete data.
This means that you need to either do your operation atomically in SQL (a single statement) or use a transaction.
In any case, it means you need to use a transactional engine in MySQL (probably InnoDB) and your application needs to be aware of and handle deadlocks correctly.
Related
I want to test Node and Deno and try to redirect users via proxy to one MySQL DB.
How will it impact the database?
Can some timestamp conflicts via CRUD operations arise or does MySQL have some mechanism to cope with connections from multiple servers?
What about performance or memory footprint of the database in RAM? Will it be occupying the same amount of space as if there was only one server requesting the database to CRUD something?
What would happen if I added another server that will connect to the DB, for example, java or Go server?
It will virtually have no impact on the database other than having any other concurrent processes connecting to it.
This is not a deno issue but rather a database issue.
The exact same problems can happen even with your current single Node.js instance, because the nature of all systems these days is concurrent/parallel.
You might as well replace the Deno app with another Node.js instance, Java, etc. Or even your current Node.js app.
Data in a database can change once you loaded it to the client, and it is up to you to implement the code that will handle such scenarios.
The fact that MySQL is not "ACID" is neither negative nor relevant in and of itself because it is doesn't have context.
If you need complete absolute integrity on a registry make sure you lock it when you select it, but there will be a trade off.
I'm building a background service which boils down to a very complicated queue system. The idea is to use Redis as non-persistent storage, and have a sub/pub scheme which runs on an interval.
All of the subscribers will be behind a load balancer. This removes the complicated problem of maintaining state between all the servers behind the load balancer.
But, this introduces a new problem...how can I ensure that the non-persistent (Redis) and persistent (MySQL) databases are both updated by my application(s)?
It seems like I'm forced to prioritize one, and if I HAVE to prioritize one, I will prioritize persistence. But, in that scenario, what happens if MySQL is updated, Redis is not, and for some reason I have lost the connection to MySQL and cannot undo my last write?
There are two possible solutions to your problem:
Following these steps:
a. Start MySQL transaction with START TRANSACTION
b. Run your MySQL query INSERT INTO ...
c. Run your Redis command
d. Finish your MySQL transaction with COMMIT statement in case if Redis command succeeded or ROLLBACK if command failed
Using transctions ensures that data is consistent in both storages.
Write LUA script for Redis using LuaSQL library (https://realtimelogic.com/ba/doc/en/lua/luasql.html), where you will connect to MySQL, insert your data and then send commands to Redis as well. Then this LUA script can be called from client side with just one command EVAL or EVALSHA
You can try the mysql udf plugin (https://github.com/Ideonella-sakaiensis/lib_mysqludf_redis)
See the post: how to move data from mysql to redis
I have a small app running on a production server. In the next update the db schema will change; this means the production database schema will need to change and there will need to be some data manipulation.
What's the best way to do this? I.E run a one off script to complete these tasks when I deploy to the production server?
Stack:
Nodejs
Expressjs
MySQL using node mysql
Codeship
Elasticbeanstalk
Thanks!
"The best way" depends on your circumstances. Is this a rather seldom occurrence, or is it likely to happen on a regular basis? How many production servers are there? Are there other environments, e.g. for integration tests, staging etc.? Do your developers have an own DB environment on their machines? Does your process involve continuous integration?
The more complex your landscape is, the better it is to use solutions like Todd R suggested (Liquibase, Flywaydb).
If you just have one production server and it can be down for maintenance for a few hours, the it could be sufficient to
Schedule a maintenance downtime with your stakeholders and users
Shutdown the server
Create a backup
Update the database structure and contents as necessary
Deploy software updates
Restart the server
Test the result (manually or automatically)
Inform your stakeholders and users
If anything goes wrong, rollback to a backed up version of the database and your software.
Having database update scripts is advisable. Having tested them once or more is advisable even more. Creating a backup in advance is essential.
http://www.liquibase.org/ or http://flywaydb.org/ - pretty "heavy" for one time use, but if you'll need to change the schema again in the future, probably worth investing the time to learn one of these.
I'm trying to implement a proxy layer in front of MySQL server, that will catch redundant SQL queries and send them only once to the server. In other words, I have many clients (in PHP, Perl, on different web nodes) that talk to the MySQL and very often repeat the same SELECT queries. When traffic goes up MySQL, very often, goes down.
The question is - are you aware of any open source (or commercial) tool that can help? I tried MySQL Proxy, but looks like it can't help.
Two suggestions:
MySQL Proxy
This is a front end proxy from MySQL which does what you want as far as I know
vtocc
From the vitess project, used in the YouTube mysql environment, also does a similar thing. Query consolidation: The ability to reuse the results of an in-flight query to any subsequent requests that were received while the query was still executing.
You may want to look into HAProxy and how it works.
Here two additional suggestions
SUGGESTION #1 Setup a Cluster
If your data is all InnoDB, you should try Percona XtraDB Cluster and use HAProxy in conjunction with it. You can load balance across all server in the Cluster including the Write Master.
SUGGESTION #2 Setup a Cluster via MySQL Replication to 1 or more DB Servers
Use HAProxy to load balance your reads across the Read Slaves
If you are on a budget and your data is relatively small, setup multiple MySQL Instances on one server
I am Using a transaction scope. Within that transactionscope, i have a mysql database connection. However, I will need to open up a new [yet another] MySQL connection withing the previous transaction scope.
When I do so, i get the following error Multiple simultaneous connections or connections with different connection strings inside the same transaction are not currently supported..
Is this because MySQL Server does not support Multiple Distributed Transactions, Is there anything that I should change in code or anything that i should change in the server?.
Will there be a support if i am using the above scenario with both connections being made to SQL Server instead of mysql server
You might want to read about Two-phase commit protocol and it looks like MySQL is supporting it with MySQL XA Transactions
Hope this helps
Although I have never tried this myself for MySql, I would expect this to be possible provided you can create an XA capable datasource.
This question seems to imply that MySql does support XA.
One issue that you may then encounter is that you need to be running in an environment capable of dealing with XA transactions. (in effect you are doing a distributed transaction across two databases because you are using two separate connections) I'm used to working in a Java EE App Server and so the server acts as the transaction coordinator, I'm not sure what would happen in a stand-alone Java program.