Hey guys, i had posted this question in another question but i didnt get a helpful response. Partly because i dont think i explained myself properly. So im going to try again.
I've been programming a back end server in vb.net and its been using a mysql database to store information. I was up until a couple of days ago using a webhost's mysql server to do this.
I did not care to renew my webhost so ive moved everything to a home server to continue work on my program. I've got mysql 5.5 installed (which is a newer version then the one on my previous webhost) and everything is working perfectly except for one thing so far.
This program when starting up for the first time sends a query about a million table inserts large. the query looks something like "INSERT into blah VALUES(1,1,1,1,1);INSERT into...." and so on. This used to take about 5-10 mins or so on my webhost (which i had my server program running on my home server machine and it was sending the info via the net to the webhost's mysql.)
Now doing everything locally i was hoping for this to be faster, but it didnt really matter i just needed it to work. So when i send this query it just locks up for a minute or so and then returns a timeout. When i check the table in the database it has loaded exactly 1000 items everytime i try this.
Now im assuming this is some sort of setting issue, however i've played with my "my.ini" to see if it would help, it didnt. I had tried switching to some of the pre-packaged my-huge or my-innodb etc and those did not help either. I would assume that if anything it would just take longer or something, not just timeout immediately after 1000 inserts.
And just for some background info the server machine im using has a quad core core i7, 8gb ram, and a 1tb hd in it running windows 7.
Any help would be great thank you.
Probably your query is too large, you can tune MySQL behavior via max_allowed_packet option.
You can also save some bytes by combining several insert queries into one, like this: INSERT INTO blah VALUES(1,1,1,1,1),VALUES(2,2,2,2,2),VALUES(3,3,3,3,3). But if this large combined query fails, then data in that insert will fail too.
But in my opinion it's not the smartest way to do it. If your application should import huge sql dump on start, it can possible use mysql executable like this mysql -uroot -ppassword db_name < dump.sql and you're done. It will be possibly the most effective way to accomplish this task.
Related
I have a project with node-red and I am using node-red-node-mysql to manipulate data inside my Mysql Workbench.
I do the next actions: insert data, update data, query for some rows to display them on the screen.
When I start my project, the PC's disk is almost instantly at 99% by "mysql.exe"
Below you can see 2 of my flows that use the connection to my database.
If I stop my server, my Disk usage goes to 1%
I refer to Disk like in the image below
Could somebody help me out here with a suggestion?
Could it be something from the configuration of mysql?
P.S: I am jut a n00b trying to understand node-red and use it :)
I investigated a bit the pc and decided to use another station.
After moving everything to another server, it works just fine. I think that the problem came after a windows update — my HDD runs slower, much slower.
I'm trying to figure out what could account for the very large performance difference between my dev environment (5 year old laptop) and our stage server (azure cloud). The table in question is a log table of web service requests for a service that processes XML. One of the columns in the table is the XML passed to the web service.
On my local computer it basically doesn't matter how many rows are in the table; performance is great. On the deployed server if there are more than a couple hundred rows then performance starts tanking quickly. A "select count(*)" on this table when it has 2000 rows in it will take 0.0017 seconds locally but close to 20 on the server. Even a simple insert of a new row takes a significant amount of time; on the order of whole seconds.
I found this article while researching the problem explanation of MySQL block performance. That makes sense to me and I'd be happy to implement the 1-to-1 solution but I don't want to do it until I understand why it's working fine locally and tanking on the server.
Are there some MySQL setting variables I can check to find the differences? I'd really like to get my local computer to have the same performance issue as the deployed so I can validate that the fix will work.
EDIT:
The create table queries are identical. MySQL versions are 5.7.23 and 5.7.22. I did notice that the buffer is 16x bigger on my local. Gonna try and get the server updated to the setting my local has and see if that resolves the issue.
The solution was updating the buffer pool size like Rick suggested.
I have installed syslog-ng on an ubuntu 18.04.4lts. but it looks like mysql is missing some logs. when I run syslog-ng -d I can see logs are coming in real time. But in phpmyadmin they are at least 25-30min behind. Furthermore, it is happening everyday. I have also made the changes so that there is no limit of mysql to store data. Any idea what can be wrong?
Please adjust spelling, reading this in all caps sounds like you're screaming.
In phpMyAdmin, as long as you refresh the page, you're seeing data as MySQL/MariaDB stores it; there's no delay or caching on the MySQL <--> phpMyAdmin connection. Any delay you're seeing as described here would have to be explained by syslog-ng; is there some sort of cache involved where it delays writing events to the database? That would make the most sense about why syslog-ng -d shows you the relevant logs but it takes some time to propagate to MySQL.
Really, I don't know much about using syslog-ng in this way, but based on your explanation of events, the answer has to lie there somewhere.
I give you some technical details before going forward to the issue:
MySQL Version: 5.3.1
Engine: Innodb
Server running under Centos
I'm experiencing some troubles in the user connections when running certain queries. For this example to show you my problem I will make a query slow enough to provoke this scenario to happen.
Lets say I have my user "testuser1" and I use it to run a 2 minutes long query (long in purpose). While this query is running I try to run another query with the same user (and different tables) but it doesn't actually run until the first query finish (Kind of sequential).
First thing I thought is... well, maybe those tables are locked? (even tho I cannot even connect to the database at all).
So I tried to use a new user "testuser2" while this query was running. In this case I didn't have any problem at all on connecting to the database and running queries.
After this I thought the user might be blocked, so I tried to connect
my "testuser1" to a different database with no issue.
The next thing I tried was to use "testuser1" on a different IP while the first one was running. However I didn't experience any issue on running my queries.
Does anyone know why I cannot run two queries at a time on the same IP and how to fix this problem?
Thanks a lot!
I'm new to MYSQL and there is something really weird happened and I can't figure out why.
Recently, the INSERT query to some of the table become extremely slow. Weirdly enough, the query time all around 60 secs.
The tables are all with the only 10k to 35k entries, so I think they are not that big.(But indeed they are the biggest one in the database, though.)
And the slowness is only with INSERT query, DELETE, UPDATE, SELECT are all executed with 0.000x sec.
Can some help me find out why is this happening?
UPDATE: So I turned on the general log and noticed all my INSERT queries are followed with 'DO sleep(60)'. It seems my server got hacked?
Where can I find this malicious script inject the sleep() command after INSERT?
If you use code to build the queries, copy the code base off the server to your machine (ideally in a VM, just in case) and search for the changes within the code. Alternatively, you could restore the code base from source control (you use source control, right?!).
If it's store procedures you use, you'll need to change them back to a working version without the sleep. Check previous backups to try and find out when this happened, which might help a wider investigation as to how they got in and did what they did.
You'll also need to think about the wider implications of this. Do you store user data? If so, then you'll need to inform them that you've had your database compromised and therefore they should assume their accounts are and change their passwords.
Finally, wipe the server. A hacked server is no longer in your control (or that's how you should look at it). Wipe it, reinstall everything, and put in changes to help prevent the same hack happening again.