I'm atm working to create a huge mySQL database by parsing XML files released on a FTP.
On a single computer, it takes ages, because of the huge amount of SQL INSERT INTO to make.
Thus, I modified my code to build it on AWS by creating a cluster, launching a database, build everything and download back the dump.
However, I got a question. Is there a "queue" for SQL requests sent ? I mean, if every of my nodes are sending requests at the same time to the database, what's going to happen ?
Thanks
On MySQL you can use SHOW FULL PROCESSLIST to see the open connections and what query they are running at the moment.
There is no queue of requests but some requests waits for others to complete, before starting, because they attempt to use rows or tables that are locked by the requests that are currently running.
Only one request is executed at a time for each connection.
Related
I have 2 servers.
Server A is where I have scripts that download html files from websites and then insert those texts into a MySQL server that its on server B.
Server B, at least for now, is just for Writing to MySQL, we dont read (select) to that server.
Data is downloaded using a PHP Script and another PHP Scrips opens a connection to server B a writes to MySQL using mysqli library.
By the way, the only service (that matters) running on server B is MySQL.
My problem is that server B bandwidth consumption is almost Symmetric, every data that comes in is the same that comes out, I asume its going back for some reason to server A, but I cant understand why o what is going back.
If you are doing only INSERTS, the Received data should go UP and the SENT data should stay low. but this is no happeing.
(See attached image showing the transfered data for MySQL)
I have use VNStat and Nethogs to try debugging but I cant figure it out. For some reason MySQL is transferring some data back to server A which is costing me a lot of BW.
Any ideas of why this could be happening?
Best Regards.
After a long night analyzing my PHP code I found a Loop with a Mysql query inside that at first I thought it was a light query, which it was but if you add up every cicle of the loop you get a lot of data being sent to the MySQL server.
I created a better way to do that and now the BW consumption is at least half.
I have a LabView data acquisition system that is writing data to a MySQL Database. It is writing data every second. The LabView system recently froze around the time when I was playing with the SQL queries.
I have a client computer, which is supposed to send queries to that MySQL Database every hour. This client computer set up a cron job to send the command to query the database every hour.
I recently added an index to my time_stamp, in order to optimize my query.
This may be a shot in the dark, but could there be any deleterious interaction between the fact that I had created an index on our time_stamp (to optimize the query), and set up a cron job to send the query every hour? Around that time, I think I may have also sent a query and aborted quickly before it completed, so I was wondering if something like that may cause the LabView system to freeze?
It doesn't appear to be an issue on the MySQL side, because the server was still running.
Check if you currently have deadlocks at your MySQL server, the query from your LabVIEW application might be suspended and is waiting for completion forever. Hint, look to the query execution time. Why it is happened should be investigated separately, but there is a good chance that if you just kill that suspended query the system will unfreeze and keeps running normally.
I just used mySQL workbench to connect to my clearDB account which is connected to an azure web app. The problem is even thought I ran a query that drops/creates tables in the newly made schema that mirrors exactly the tables and data in my previous live server, I go to mysite.azurewebsites.com/wp-admin and the error is in establishing data connection. Site could not be found. Check if your database contains the following pages: wp_blogs, ..........
What could be the problem? Does this process just need a bit of time to propagate all the data?
EDIT: something to note, which might be a factor, when I ran the last query, it also included dropping/adding the table "wp_users" so all previous data was wiped and replaced with the info from a previous live server.
Normally you will see any changes made immediately. But because your database is hosted on a geoseparated cluster in circular replication there are some rare circumstances where this might not be true.
Specifically, if your delete/write went to one master and your read query went to another. Data propagation is normally immediate but if one of the nodes is offline or the system is unusually busy there can be a delay.
currently we have one master mysql server that connect every 1 hour to 100 remote mobile devices [vehicles] over 3G connection [not very reliable: get disconnect daily while sync in progress for few cars]. the sync done through .net windows service tool. after checking the remote mysql status the master start perform the sync. sometimes the sync payload data is about 6-8 MB. the sync performed for one table only using non-transactional approach.
mysql server version in use is: 4.1.22
Questions:
is it useful to make the sync transactional knowing that only one table getting sync? or no value added!
the sync data loaded to remote machine using mysql statement:
LOAD DATA LOCAL INFILE
the file format is CSV. how i can send the data in compressed format? without developing tool that reside on the remote device.
is it good practice or architecture in the sync domain to deploy remote application that will perform the sync after sending the data or it should be done directly by the master? i mean the development of tool that will reside on remote machine will be difficult to update or fix in case new requirements appear. but it will save a lot of bandwidth for the sync operation and it will eliminate the errors that could raise from the live master sync in case disconnection occur while the sync is in-progress. so if this is recommend then only compressed data will be sent, then by using some sort of check-sum I'll verify that the whole data sent otherwise the request will be initiated again.
please share your thoughts and experience.
thanks,
Firstly, I would change the approach to a client inited sync vs a server inited sync. A many to one vs one to many approach will expand much easier than your current setup. My above comments give a few good examples of a required client to server syncing.
Secondly, Turn on transactional record entry. There is no reason not to have it. This will guarentee that the information gets entered in a timely fashion and will be able to possibly provide even more 'meta-data' (such as which clients are slow to update, etc...).
Lastly, you can 'enhance' this uploading by taking a different look at it. If you were to implement a sort of service at the server side that takes in a response via a POST from the client, you'd be able to send the data to the server side with no issues. It would be just like 'uploading' a file to a server. Once your 6-8 MB file is 'uploaded' it is then put into the database. The great thing about this is if your server is an APACHE (or even in your case an IIS server), you'd be able to have every single client uploading data at the same time without much of an issue. At that point, uploading to the mysql server via an insert would take virtually no time and your process would continue on without a problem.
This is the way I'd handle your situation...
I'm using R to read some data from a MySQL database using the RODBC package. The data is then processed and some results are sent back to the database. The problem is that the server closes the connection after about a minute due to inactivity, which is the time needed to process the data locally. It's a shared server, so the host won't bump up the timeout time.
I think there are two possibilities to get around this:
Open a connection before every database transaction and close it immediately after
Send some small 'ping' command to the server every 30 seconds or so to let the server know that I'm still there.
I can implement the first fairly easily, but it seems pretty slow to constantly open and close connections. Does anyone know an efficient command for the second? Or is a better way altogether?
The first solution is the one I prefer. It's really hard to do the latter with a single threaded program like R. If R is busy running analysis there's no way for it to handle the ping. Unless you are doing hundreds of reads/writes the method of opening and closing the connection should not introduce an extreme amount of overhead.