I want to check whether kernel memory parameters i.e. shmmax or shmall is required or not for MYSQL database and what are the pros & cons if we set this parameter.
Because in my case MySQL working fine without setting them, However they are mandatory for Oracle database
Thanks
Related
I'm running into a issue with a spring boot application, in which I am getting the below error,
(conn=1126) Can't create more than max_prepared_stmt_count statements (current value: 16382)
Which seems to be hitting the ceiling of max_prepared_stmt_count in mysql. Increasing it as much as we want could be problematic as well as it could result in OOM killer issues like this.
I'm exploring if there are any ways to limit the creation of PreparedStatements in Spring boot.
One possible option that I can think of,
Avoiding lazy loading whenever possible would force Hibernate to fetch the data with lesser number of prepared statements thus avoiding the problem.
Cache the PreparedStatements created by hibernate.
If anyone solved this problem or with a deeper insight, please share your wisdom on solving this.
MySQL has no problem with many prepared statements if they are over time. Obviously some MySQL databases stay up and running for months at a time, serving prepared statements. The total count of prepared statements over those months can be limitless.
The problem is how many prepared statements are cached in the MySQL Server at any given moment.
It has long been a source of trouble for MySQL Server that it allocates a small amount of RAM in the server for each prepared statement, to store the compiled version of the prepared statement and allow it to be executed subsequently. But the server doesn't know if the client will execute it again, so it has to keep that memory allocation indefinitely. The server depends on the client to explicitly deallocate the prepared statement. This can become a problem if the client neglects to "close" its prepared statements. They become long-lived, and eventually all those accumulated data structures take too much RAM in the MySQL Server.
So the variable really should be named max_prepared_stmts_allocated, not max_prepared_stmt_count. Likewise the status variable Prepared_stmt_count should be Prepared_stmts_open or something like that.
To fix this in your case, I would make sure in the client code that you deallocate prepared statements promptly when you have no more need of them.
If you open a prepared statement using a try-with-resources block, it should automatically be closed at the end of the block.
Otherwise you should call stmt.close() explicitly when you're done with it.
I'm trying to run a few JDBC extracts in parallel, but this fails on: java.lang.OutOfMemoryError: Java heap space.
How does Data Connection memory usage work, and how do I resolve this problem?
The Data Connection Agent's memory usage here actually depends mostly on the value of the fetchSize parameter. Per the Oracle JDBC driver docs, fetchSize:
Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed for this ResultSet object.
So, the agent's memory usage should roughly be:
number of JDBC extracts running in parallel x fetchSize x size of each row
Unfortunately, the default value of fetchSize varies vastly among different JDBC drivers. For instance, certain versions of the Hive JDBC driver have it set to 50, while other, newer versions have a default value of 1000. Oracle JDBC drivers have a default of 10. Postgres by default will try to get the entire ResultSet at once.
Hence, Data Connection allows you to configure the fetchSize value. This is configurable both per-source and per-extract.
OOM errors aside, tuning fetchSize can improve performance significantly in general. There isn't a one-size-fits-all solution, though, and you'll have to experiment to figure out the best parameter value for your extracts. It usually lies somewhere in the 500–5000 range.
Which is better to use for query 1 or 2 give me some tips. I have a gridview that has a lot of info like hundreds. Which is advisable to use for fast querying the result without lagging
QUERY = "SELECT * FROM exdb.sample"
OR
DA = New MySqlDataAdapter("SELECT * FROM exdb.sample", dataCNN)
If I am right you are talking about Connected and Disconnected Architecture,
Connected Architecture : you have to declare the connection explicitly by using Open(), and close the connection using Close(), and u cn execute commands using different methods like.. ExecuteNonQuery,ExecuteScalar,ExecuteReader etc..
Disconnected Architecture: you dont need to define the connections explicitly..SqlDataAdatpter itself can open and closes the connection..and u can use dataset for storing the info. temporarily and Fill method was used to execute the commands give in adapter...
When you use DataAdapter, there's no requirement of opening and closing connection. It's done automatically.
By keeping connections open for only a minimum period of time, ADO .NET conserves system resources and provides maximum security for databases and also has less impact on system performance.
Only one operation can be performed at a time in
connected environment while in dissconnected multiple
operations can be performed.
But as you said there are hudreds of records in your application I'll suggest to use DataReader (i.e. Connected Architecture) as it is faster than DataAdapter (i.e. Disconnected Architecture) beacause it is read-only, forward-only stream of data.
I'll suggest you go through this for more information about DataReader and DataAdapter.
First one is a connected architecture where you will loop through all the rows, populate them to a DataTable and attach to your gridview.
Whereas, second one is disconnected architecture using a data adapter; you maintain the structure in memory.
I would go for the second approach, fill the dataset/datatable and bind the same source to gridview.
There is a limit of 2,100 parameters which can be passed to a Sql Server query i.e. via ADO.Net, but what are the documented limits for other common databases used by .Net developers - in particular I'm interested in:
Oracle 10g/11g
MySql
PostgreSql
Sqlite
Does anyone know?
Oracle: 64,000. Source
MySQL:
By default, there is no limit. The MySQL "text protocol" requires that the .NET client library substitute all parameters before sending the command text to the server; there is no server-side limit that can be enforced, and the client has no limit (other than available memory).
If using "prepared statements" by calling MySqlCommand.Prepare() (and specifying IgnorePrepare=false in the connection string), then there is a limit of 65,535 parameters (because num_params has to fit in two bytes).
PostgreSql: EDIT: 34464 for a query and 100 for a function as per Magnus Hagander's answer (Answer copied here to provide a single point of reference)
SqlLite: 999 (SQLITE_MAX_VARIABLE_NUMBER, which defaults to 999, but can be lowered at runtime) - And for functions default is 100 parameters. See section 9 Of Run-time limits documentation
In jOOQ, we've worked around these limitations by inlining bind values once we reach the relevant number per vendor. The numbers are documented here. Not all numbers are necessarily the correct ones according to vendor documentation, we've discovered them empirically by trial and error through JDBC. They are (without tying them to a specific version):
Ingres : 1024
Microsoft Access : 768
Oracle : 32767
PostgreSQL : 32767
SQLite : 999
SQL Server : 2100 (depending on the version)
Sybase ASE : 2000
Other databases do not seem to have any limitations - at least we've not discovered them yet (haven't been looking far beyond 100000, though).
The correct answer for PostgreSQL appears to be 34464, when talking about bound parameters to a query. The response 100 is still correct for number of parameters to a function.
The PostgreSQL wire protocol uses 16-bit integers for count of parameters in the bind message (https://www.postgresql.org/docs/current/protocol-message-formats.html).
Thus the PostgreSQL protocol doesn't allow over 65535 parameters for a single statement. This is, OK to send a single ado.net command with two statements, each of which has 65535 parameters.
In my view, the MySQL question actually has two answers. The prepared statement protocol defines a signed 2 byte short to describe the number of parameters that will be retrieved from the server. The client firstly calls COM_STMT_PREPARE, for which it receives a COM_STMT_PREPARE response if successful.
The documentation for the response states:
If num_params > 0 more packets will follow:
Parameter Definition Block
num_params * Protocol::ColumnDefinition
EOF_Packet
Given that num_params can only be a maximum of 2^16 (signed short), it would follow that this is the limit of parameters and as my company has a custom MySQL driver we chose to follow this rule when implementing it and an exception is thrown if the limit is exceeded.
However, COM_STMT_PREPARE does not actually return an error if you send more than this number of parameters. The value of num_params is actually just 2^16 and more parameters will follow afterwards. I'm not sure if this is a bug but the protocol documentation does not describe this behaviour.
So long as you have a way on your client-side to know the number of parameters (client_num_params if you will), you could implement your MySQL client in such a way that it expects to see client_num_params x Protocol::ColumnDefinition. You could also watch for EOF_Packet but that's only actually sent if CLIENT_DEPRECATE_EOF is not enabled.
It's also interesting to note that there's a reserved byte after num_params, indicating that the protocol designers probably wanted the option to make this a 24-bit number, allowing about 8.3 million parameters. This would also require an extra client capability flag.
To summarise:
The client/server protocol documentation seems to indicate that the maximum number of parameters could be 32768
The server doesn't seem to care if you send more but this doesn't appear to be documented and it might not be supported in future releases. I very much doubt this would happen though as this would break multiple drivers including Oracle's own ADO.NET Connector.
I have a sproc that generates an 80,000-row temp table which is passed as a table-valued parameter to 32 different other sprocs (each sproc the TVP as an input parameter).
Should I be concerned that I am going to get a balloon of memory I can't manage?
What is a good way to monitor (PerfMon?) how the memory is being used/tracked?
Thanks.
1)
According to this question:
Performance of bcp/BULK INSERT vs. Table-Valued Parameters
TVP's will underperform using bulkcopy on datasets that large.
On the other hand... figure out the max datasize of your 80,000 rows and determine if you're ok with that size object floating around in RAM (Personally I wouldn't have a problem with it... we could store our entire DB in RAM three times over)
2)
Here is a good thread on ServerFault for monitoring SQL Server's memory usage:
https://serverfault.com/questions/115957/viewing-sqls-cache-ram-usage