I have just reloaded my laptop and am now trying to setup my localhost again.
Currently I am trying to re-setup the database.
the issue is,
The script is 169,328 KB.
This keeps crashing whatever I use to run the query and I get the error: The mysql Server has gone away.
Seems everyone is suggesting splitting the script. Instead as this is only me setting my localhost back up, I have temporarily increased the max_packet_size.
This would explain the error.
Perhaps you should open the script and see if you can chunk it into smaller, more manageable pieces.
I don't know what you're doing about transactions, but perhaps the rollback segment (or its MySQL equivalent) is getting too large. If that's the case, break the script into several transactions that you can safely commit individually.
If you're looking to avoid the err message, consider one of these remedies:
ensure that your environment or commands aren't causing this issue. Causes for MySQL gone-away.
split your large script into smaller scripts. You could then run those in sequence.
Related
I'm new to MYSQL and there is something really weird happened and I can't figure out why.
Recently, the INSERT query to some of the table become extremely slow. Weirdly enough, the query time all around 60 secs.
The tables are all with the only 10k to 35k entries, so I think they are not that big.(But indeed they are the biggest one in the database, though.)
And the slowness is only with INSERT query, DELETE, UPDATE, SELECT are all executed with 0.000x sec.
Can some help me find out why is this happening?
UPDATE: So I turned on the general log and noticed all my INSERT queries are followed with 'DO sleep(60)'. It seems my server got hacked?
Where can I find this malicious script inject the sleep() command after INSERT?
If you use code to build the queries, copy the code base off the server to your machine (ideally in a VM, just in case) and search for the changes within the code. Alternatively, you could restore the code base from source control (you use source control, right?!).
If it's store procedures you use, you'll need to change them back to a working version without the sleep. Check previous backups to try and find out when this happened, which might help a wider investigation as to how they got in and did what they did.
You'll also need to think about the wider implications of this. Do you store user data? If so, then you'll need to inform them that you've had your database compromised and therefore they should assume their accounts are and change their passwords.
Finally, wipe the server. A hacked server is no longer in your control (or that's how you should look at it). Wipe it, reinstall everything, and put in changes to help prevent the same hack happening again.
On several occasions, I've tried to do a query on Google Cloud SQL that involves an order by statement, only to have the query fail with the error
Incorrect key file for table '/cloudsqltmp/#sql_44f4_1.MYI'; try to repair it
This sounds like the /cloudsqltmp/ partition is filling up with my temporary table. The result set isn't that big, however, and the program doing that query has done so on several other occasions, so I suspect that the space is actually filling up with someone else's temporary table. I was able to clear this by restarting the instance several times (I assume it finally gave me a new machine, or the space cleared up), but this seems very inelegant.
Is there a better way to handle this if it happens again?
Is there a better way to prevent this from happening?
If my assumption of what happened is wrong - what actually happened?
What I would recommend doing is:
Creating a Google Compute Engine instance and run MySQL on it. This way if you run into this problem again, you can use the follow solution below.
1) Modify the /tmp partition to have more space. There is a very useful link here on how to do it.
2) Create a CronJob to clear the /tmp. If you are not comfortable with CronJob, I found another tool called tmpreaper. To install it, run sudo apt-get install tmpreaper. Also use InnoDB instead of MyISIAM. Google recommends using it.
3) You are correct to assume that /tmp is getting full since restarting the instance resolves the issue
I hope my recommendations were helpful.
I have a rather complex SQL script that does several things:
Drops an existing database
Reloads the database from a backup
Sets permissions
A few other miscellaneous db-specific tasks
I've saved this as a .sql file for mobility and ease of use, but I'd like to incorporate this into a Powershell script to make it even simpler to run (and to encapsulate a few other tasks that need to be done around the same process). Here are the bits of code I have in Powershell for the script:
add-pssnapin sqlservercmdletsnapin100
invoke-sqlcmd -inputfile "D:\Scripts\loaddatabase31_Loadtest.sql" -serverinstance "PerfSQL02" -hostname "PerfSQL02"
Assume that the serverinstance, inputfile, and hostname exist and are put in correctly.
The database I am restoring is a couple hundred gigabytes, so the actual drop and restore process takes around 20 minutes. Right now, I'm getting the following error when I try to run that script (it's being run from a workstation within the same network but on a different domain):
invoke-sqlcmd : Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
At line:2 char:1
I've tried using the -connectiontimeout switch on invoke-sqlcmd, but that doesn't seem to work to resolve this issue (so it's probably not timing out on connecting). Most of the examples I've seen of running SQL via Powershell don't use a file, but instead define the SQL within the script. I'd prefer not to do this due to the complexity of the SQL needed to run (and since it's dropping and restoring a database, I can't really wrap all of this up in a SP unless I use one of the System DBs).
Anyone have some ideas as to what could be causing that timeout? Does invoke-sqlcmd not work gracefully with files?
You don't want the -connectiontimeout, you likely want the -QueryTimeout
http://technet.microsoft.com/en-us/library/cc281720.aspx
Ok, so basically I've got a script in CakePHP where I'm putting over 7 million records into a database. Seeing as how there are that many records, I'm running into some issues with timeouts. This is on a personal server so the memory limit is set to 2000MB so that's not really an issue with how I'm wanting to do it.
The database rows are coming from a huge file. The file was too big for the memory limit, I've split it up into 101 pieces at 10000 lines in each file.
I want the page to refresh after 10 records, and when it comes back, restart inserting records where it left off.
Any ideas?
I've tried the $this->redirect() route, but it's created never-ending scripts that had to be stopped by manually restarting the server.
Why are you not using a shell for that?
To avoid the redirect loop you could try to redirect between two actions or try to attach a timestamp to the url. I'm not sure if that will work, the shell would be the much better approach anyways.
There is an action in the admin section of a client's site, say Admin::Analytics (that I did not build but have to maintain) that compiles site usage analytics by performing a couple dozen, rather intensive database queries. This functionality has always been a bottleneck to application performance whenever the analytics report is being compiled. But, the bottleneck has become so bad lately that, when accessed, the site comes to a screeching halt and hangs indefinitely. Until yesterday I never had a reason to run the "top" command on the server, but doing so I realized that Admin::Analytics#index causes mysqld to spin at upwards of 350+% CPU power on the quad-core, production VPS.
I have downloaded fresh copies of production data and the production log. However, when I access Admin::Analytics#index locally on my development box, while using the production data, it loads in about 10 - 12 seconds (and utilizes ~ 150+% of my dual-core CPU), which sadly is normal. I suppose there could be a discrepancy in mysql settings that has suddenly come into play. Also, a mysqldump of the database is now 531 MB, when it was only 336 MB 28 days ago. Anyway, I do not have root access on the VPS, so tweaking mysqld performance would be cumbersome, and I would really like to get to the exact cause of this problem. However, the production logs don't contain info. on the queries; they merely report the length that these requests took, which average out to a few minutes apiece (although they seemed to have caused mysqld to stall for much longer than this and prompting me to request our host to reboot mysqld just to get our site back up in one instance).
I suppose I can try upping the log level in production to solicit info. on the database queries being performed by Admin::Analytics#index, but at the same time I'm afraid to replicate this behavior in production because I don't feel like calling our host up to restart mysqld again! This action contains a single database request in its controller, and a couple dozen prepared statements embedded in its view!
How would you proceed to benchmark/diagnose and optimize/fix this action?!
(Aside: Obviously I would like to completely replace this functionality with Google Analytics or a similar solution, but I need fix this problem before proceeding.)
I'd recommend taking a look at this article:
http://axonflux.com/building-and-scaling-a-startup
Particularly, query_reviewer and newrelic have been a life-saver for me.
I appreciate all the help with this, but what turned out to be the fix for this was to implement a couple of indexes on the Analytics table to cater to the queries in this action. A simple Rails migration to add the indexes and the action now loads in less than a second both on my dev box and on prod!