I am executing a mysql insert query having very large amount of linestring data. but it's taking infinite time to execute the query. (insert query is so big that it takes more than 5 minutes to paste on the terminal).
My content-length is 10 mb as per request header. so i thought it must be due to mysql query size but that is 34mb. i also increase connection timeout to infinite but still it's not executing.
Insert into table (id,data) value (1,LinestringFromText('LINESTRING(0 9,8 0... so on and it goes on)'))
If i try the same query with the low amount of data it execute smoothly but with large amount of data it stucks.
You shouldn't use an App Engine application to hold and run a mysql server, reasons being that you shouldn't rely on an App Engine instance to hold and serve dynamic data (see how instances are managed). However, if having your mysql server in App engine is a must, you can try to change the instance class of your application in the app.yaml configuration file, to one of the following. A machine with more resources, should translate to a better performance.
However, I would recommend you to use Cloud SQL to hold and serve your data, and connect to it from your application. See this tutorial for an overview on how can you do this, which is based in this source code. This example also uses the mysql package that you linked.
Related
I have a simple pentaho transformation: One input from SQL Server, and two outputs, one to local MySQL and another to Google cloud MySQL. The total rows from input are 3000 with six columns
My problem is that in Google cloud MySQL ouput is too slow, the performance is 6 minutes in to insert 3000 rows!, howerever in local MySQL output the performance is 1 second.
is there any reason for this problema?, Can I fix it?.
Pentaho Transformation image
Thanks!
EDIT:
BEFORE INSERT
Screenshot before insert
AFTER INSERT
Screenshot after insert
Here are a few things to verify in terms of Cloud SQL performance in this case:
Enable lock monitoring which can be used for performance tuning. Here is how
If using a First Generation instance, using asynchronous mode might be much faster. Here is how
Check the locations of the writer and database; sending data a long distance introduces latency.
Caching is extremely important for read performance, which may come into play depending on the query for the insert. The entire data set should fit in 70% of the instance’s RAM.
If the query is CPU intensive and constitutes the majority of the workload, the instance might be throttled; increase the tier.
Since the connection to the database seems to be done from an external application, here is how to set this up properly. More information about the connection configuration and the insert statement is required to determine if indeed connections are established for each requests as suspected by others.
If using InnoDB (mandatory for 2nd generation instances and recommended for 1st generation ones), here are some best practices to follow to optimize performance, from the official MySQL documentation.
You should add this parameter to your connection parameter:
rewriteBatchedStatements=true
This will drastically improve the inserts speed.
Check this for more info: https://anonymousbi.wordpress.com/2014/02/11/increase-mysql-output-to-80k-rowssecond-in-pentaho-data-integration/
I am trying to implement a function that uploads around 40 million records to a MySQL database that is hosted on AWS. However, my write statement gets stuck at 94% for an infinitely long time.
This is the command I'm using to upload df_intermediate.write.mode("append").jdbc(jdbcUrl, "user", connectionProperties) with rewriteBatchedStatements and useServerPrepStmts enabled in the connection properties.
This statement works for small number of points(50000) but is unable to handle this large amount. I've also increased the maximum number of connections on the MySQL side.
EDIT: I'm running this on GCP n1-standard-16 machines.
Why could be the reasons that write is stuck at 94%?
I don't think this has anything to do with Scala really, you are just saying you want to add many many rows into a DB. The quick answer would be to not have all these in the one transaction, and to commit these lets say 100 at a time. Try this on a non-production sql database first to see if that works.
I'm atm working to create a huge mySQL database by parsing XML files released on a FTP.
On a single computer, it takes ages, because of the huge amount of SQL INSERT INTO to make.
Thus, I modified my code to build it on AWS by creating a cluster, launching a database, build everything and download back the dump.
However, I got a question. Is there a "queue" for SQL requests sent ? I mean, if every of my nodes are sending requests at the same time to the database, what's going to happen ?
Thanks
On MySQL you can use SHOW FULL PROCESSLIST to see the open connections and what query they are running at the moment.
There is no queue of requests but some requests waits for others to complete, before starting, because they attempt to use rows or tables that are locked by the requests that are currently running.
Only one request is executed at a time for each connection.
I am currently importing a huge CSV file from my iPhone to a rails server. In this case, the server will parse the data and then start inserting rows of data into the database. The CSV file is fairly large and would take a lot time for the operation to end.
Since I am doing this asynchronously, my iPhone is then able to go to other views and do other stuff.
However, when it requests another query in another table.. this will HANG because the first operation is still trying to insert the CSV's information into the database.
Is there a way to resolve this type of issue?
As long as the phone doesn't care when the database insert is complete, you might want to try storing the CSV file in a tmp directory on your server and then have a script write from that file to the database. Or simply store it in memory. That way, once the phone has posted the CSV file, it can move on to other things while the script handles the database inserts asynchronously. And yes, #Barmar is right about using an InnoDB engine rather than MyISAM (which may be default in some configurations).
Or, you might want to consider enabling "low-priority updates" which will delay write calls until all pending read calls have finished. See this article about MySQL table locking. (I'm not sure what exactly you say is hanging: the update, or reads while performing the update…)
Regardless, if you are posting the data asynchronously from your phone (i.e., not from the UI thread), it shouldn't be an issue as long as you don't try to use more than the maximum number of concurrent HTTP connections.
I have a large MySQL table, with proper indices etc. I run a simple select * query from a remote machine and I expect a large result set.
My problem is that when I run the query, the result set returns at a maximum data transfer speed of ~300 KBytes/sec.
I created the same table, and run the same query on SQLServer Express 2008 R2, and the results returned at a transfer speed of 2MBytes/second (my line limit).
The server machine is Windows Server 2008 R2 x64, Quad core, 4GB RAM and the the MySQL version is 5.6.2 m5 64-bit. I tried disabling the compession in the communication protocol but the results where the same.
Does anyone have an idea as to why this is happening ?
--theodore
You might be comparing apples to oranges.
I'd run SELECT * on the MySQL server, and see what kind of data rate you get for retrieving data on the server locally -- without the additional constraint of a network.
If that's slow also -- then it isn't the network's fault.
When the MySQL setup program runs, it asks the person setting up MySQL what role MySQL is going to play on the hardware -- i.e., Development Server, Shared Server, Dedicated.
The difference in all of these is how much memory MySQL will seek to consume on the Server.
The slowest setting is Development (use the least memory), and the fastest one is Dedicated (attempt to use a lot of memory). You can tinker with the my.ini file to change how much memory MySQL will allocate for itself, and/or google 'my.ini memory' for more detailed instructions.
The memory that MySQL is using (or isn't, as the case may be), will make a huge difference on performance.
First, check to see what the speed is retrieving data locally on the MySQL server is. If it's slow, the network isn't the problem -- check MySQL's memory usage -- ideally give it as much as possible. And of course, if it's fast, then either the network and/or some piece of database middleware (ODBC?) or tool-used-to-display-the-data -- is slow...
One more thing -- try the SELECT * TWICE... why? The second time some or all of the results (again, depending on memory) should be cached... the second time it should be faster...
Also, don't forget to restart MySQL when changing the my.ini file (and create a backup before you make any changes...)