I am developing a web application with database access using Spring, JDBCTemplate and c3p0.
I often have a server freeze, and I am pretty sure it comes from the number of busy database connections. If I watch the application behavior, using jconsole, I can see that the maxPoolSize of the ComboPooledDataSource is reached, and the server doesn't load a page anymore.
Here is the useful code:
DataSource definition:
<Resource auth="Container" description="GDLWeb DB Connection"
driverClass="org.postgresql.Driver"
maxPoolSize="16"
minPoolSize="1"
acquireIncrement="1"
maxIdleTime="60"
maxStatements="0"
idleConnectionTestPeriod="1800"
acquireRetryAttempts="30"
breakAfterAcquireFailure="true"
name="jdbc/gdlweb"
user="gdlweb"
password=""
factory="org.apache.naming.factory.BeanFactory"
type="com.mchange.v2.c3p0.ComboPooledDataSource"
jdbcUrl="jdbc:postgresql://localhost:5432/postgres"
/>
Typical access method (in DAO class):
protected T getPersistentObject(
final String tableName,
final List<WhereClause> whereParams,
final RowMapper<T> rowMapper) {
try {
log.debug(this, "get " + tableName + " " + whereParams);
return (T) getTemplate().queryForObject(
generateSelectStar(tableName, whereParams),
extractValueMap(whereParams),
rowMapper);
} catch (final EmptyResultDataAccessException e) {
log.warning(this, "No " + tableName + " found with " + whereParams + " in the DB!");
return null;
}
}
I tried to increase the maxPoolSize to 100, which is the maxConnections defined in my postgresql server. This way, I could see that there were 43 busy connections currently openned, just before the postgresql server crashes.
I am probably using JDBCTemplate the wrong way, but I don't know where.
Thanks.
The problem may be with the Mysql Connector/J version you are using.
I had the same issue, updating to the new Mysql Connector v5.1.15 solved it for me. v5.1.13 has a bug which results in the problems you are seeing
Change-log for the version which fixes the bug: http://dev.mysql.com/doc/refman/5.1/en/cj-news-5-1-14.html
Thanks
Related
Since upgrading (described below) the Facet search throws this exception.
HSEARCH000268: Facet request 'groupArchiv' tries to facet on field
'facetfieldarchiv' which either does not exists or is not configured
for faceting (via #Facet). Check your configuration.
Migrating from hibernate.search.version 4.4.4 to hibernate.search.version 5.5.2
lucene-queryparser 5.3.1
jdk 1.8xx
All the Indexing is via a ClassBridge.
The field facetfieldarchiv is in the index.
All other searches are working fine.
protected List<FacetBean> searchFacets(String searchQuery, String defaultField,
String onField, String facetGroupName)
{
List<FacetBean> results = new ArrayList<FacetBean>();
FullTextSession ftSession = getHibernateFulltextSession();
org.apache.lucene.analysis.Analyzer analyzer = getAnalyzer(Archiv.class);
QueryParser parser = new QueryParser(defaultField, analyzer);
try
{
Query query = parser.parse(searchQuery);
QueryBuilder builder = ftSession.getSearchFactory().buildQueryBuilder().forEntity(Item.class).get();
FacetingRequest gruppeFacetingRequest = builder.facet()
.name(facetGroupName)
.onField(onField).discrete()
.orderedBy(FacetSortOrder.COUNT_DESC)
.includeZeroCounts(false)
.maxFacetCount(99999)
.createFacetingRequest();
org.hibernate.search.FullTextQuery hibQuery = ftSession.createFullTextQuery(query, Item.class);
FacetManager facetManager = hibQuery.getFacetManager();
facetManager.enableFaceting(gruppeFacetingRequest);
Iterator<Facet> itf1 = facetManager.getFacets(facetGroupName).iterator();
**// The error occurs here,**
while (itf1.hasNext())
{
FacetBean bean = new FacetBean();
Facet facetgruppe = itf1.next();
bean.setFacetName(facetgruppe.getFacetingName());
bean.setFacetFieldName(facetgruppe.getFieldName());
bean.setFacetValue(facetgruppe.getValue());
bean.setFacetCount(facetgruppe.getCount());
results.add(bean);
}
} catch (Exception e)
{
logger.error(" Fehler FacetSuche: " + e);
}
return results;
}
The faceting API went through an overhaul between Hibernate Search 4 and 5. In the 4.x series one could facet on any (single valued) field without special configuration. The implementation was based on a custom Collector.
In Hibernate Search 5.x the implementation has changed and native Lucene faceting support is used. For this to work though, the faceted fields need to be known at index time. For this the annotation #Facet got introduced which needs to be places on fields used for faceting. You find more information in the Hibernate Search online docs or check this blog post which gives you a short summary of the changes.
Thank you for answering.
I didn't catch that change since 5.x
My facets are made up of several fields.
Is there a possibility to build the facets in a ClassBridge using pur Lucene?
like
FacetField f = new FacetField(fieldName, fieldValue);
document.add(f);
indexWriter.addDocument(document);
Thank you
pe
I am getting an exception using PreparedStatement to select.
Got an exception accessing TestCase data! null
Problem to connect.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Too many connections
Here is my code:
public Integer getTypeByInputAndProblemId(String inputTestCase, Long problemId) {
String sql = "SELECT type FROM test_case where problem_id= ? and input= ?";
Integer type = 0;
try {
PreparedStatement ps = connection.prepareStatement(sql);
ps.setLong(1, problemId);
ps.setString(2, inputTestCase);
ResultSet rs = ps.executeQuery();
if (rs.next()) {
type = new Integer(rs.getInt("type"));
}
rs.close();
ps.close();
} catch (Exception e) {
System.err.println("Got an exception accessing TestCase data! " + e.getMessage());
}
return type;
}
In line PreparedStatement ps = connection.prepareStatement(sql);
my problem is because connection sometimes is Null (the debug shows this).
I'm guessing this is because of many connections, but I don't know why this is happening.
I would like some help, please!
Yes, the issue happens because your server is reaching the max number of multiple connections accepted by your MySQL Server.
First, you need to see if you have a proper number configured in MySQL for multiple connections: max_connections. If this look low to you, you can increase this number in order to "fix" this issue.
Secondly, if the number makes sense, you're probably using more connections than you think you are. Probably because you're opening connections in your application and not closing them.
Check how many multiple connections your server have used so far.
show status like 'Max_used_connections';
This number is reset when you restart your database service.
I wonder how come this error is thrown while hosting my project in APP ENGINE, I have added lots of logging just for analysis sake. When I use the com.mysql.jdbc.Driver using ip from my local it works. Kindly help !!
String name = "Vinodh";
String url = null;
try {
Class.forName("com.mysql.jdbc.GoogleDriver");
url = "jdbc:google:mysql://xxxxxxx:xxxxxx/vinodh?user=root&password=xxxxxx";
// Statements allow to issue SQL queries to the database
log.info("Initiate Connection");
Connection conn = DriverManager.getConnection(url);
log.info("Got Connection");
Statement statement = conn.createStatement();
// Result set get the result of the SQL query
ResultSet resultSet = statement
.executeQuery("select * from Family");
log.info("Entering While");
while(resultSet.next()){
log.info("Entered While");
String test = resultSet.getString("Name");
System.out.println(test);
name = test+test+test;
}
As shwown in this tutorial, during development you should use the normal mysql driver and only appengine use the Google mysql driver
if (SystemProperty.environment.value() ==
SystemProperty.Environment.Value.Production) {
// Load the class that provides the new "jdbc:google:mysql://" prefix.
Class.forName("com.mysql.jdbc.GoogleDriver");
url = "jdbc:google:mysql://your-project-id:your-instance-name/guestbook?user=root";
} else {
// Local MySQL instance to use during development.
Class.forName("com.mysql.jdbc.Driver");
url = "jdbc:mysql://127.0.0.1:3306/guestbook?user=root";
}
Also double check that you have enabled MySQL Connector/J for your application (it's not done by default)
https://developers.google.com/appengine/docs/java/cloud-sql/#enable_connector_j
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
...
<use-google-connector-j>true</use-google-connector-j>
</appengine-web-app>
Appearently they removed this quietly. It's not even in the docs of appengine-web.xml anymore.
Use the standard com.mysql.jdbc.Driver but update your JDBC url for:
jdbc:mysql://google/[your-db-schema]
?user=root
&password=[your-db-passord]
&socketFactory=com.google.cloud.sql.mysql.SocketFactory
&cloudSqlInstance=[your-db-project-id]:[your-db-region]:[your-db-intance]
Add also to your gradle:
dependencies {
...
implementation("mysql:mysql-connector-java:8.0.29")
implementation("com.google.cloud.sql:mysql-socket-factory-connector-j-8:1.5.0")
}
Note the "google" in the URI. There's no place in the docs saying that you need it, but you have to.
Github official page guide
I'm building a REST WebService with JAX-RS and Tomcat to consume a MySQL Database.
I'm following this model:
#Path("/login")
public class Login {
String username;
String password;
// This method is called if POST is requested
#POST
#Produces(MediaType.APPLICATION_XML)
public String loginResponseXML(#FormParam("username") String user, #FormParam("password") String pass) {
//Connection to MySQL Database
try {
Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection("jdbc:mysql://localhost/sakila", "root","larcom");
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery("Select first_name, last_name From actor where first_name='" +
user + "' and last_name='" + pass + "'");
while (rs.next()){
System.out.println(rs.getString("first_name") + " " + rs.getString("last_name"));
username = rs.getString("first_name");
password = rs.getString("last_name");
}
}
catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
}
if (user.equals(username) && pass.equals(password)) {
return ("<?xml version=\"1.0\"?>" + "<auth>200" + "</auth>"); //Success
//return "Success!";
} else {
return ("<?xml version=\"1.0\"?>" + "<auth>404" + "</auth>"); //Damn
//return "Damn!";
}
}
}
I call this method with:
HttpPost httppost = new HttpPost("http://192.168.15.245:8080/org.jersey.andre/rest/login");
Now, my question is:
If I want to query the DB for another table I have to create a new class like Login and make the JDBC connection again?
A new class and a new JDBC connection for each class that make a query to the DB? Performance issues?
Hope you can understand.
Thanks in advance.
A few tips are in order here: Please isolate the DB based code to a "data layer" so to speak...only perform dispatching/business logic within your resource classes.
Now If you are querying a different table, you WILL have a different query! You could either use the same connection (bad) or create a new one and fire a different query(s).
Now whether each resource hits a different table or the same table with a different query depends on your choice of 'representation' for that resource. There is a reason a RDB schema has multiple tables and it's quite common you'll have a different query involving multiple tables or to mutually independent tables.
Performance issues: For 'fresh data' you ARE always going to hit the DB so to speak. If you want to optimize that either develop your own cache (extremely hard) or use approaches like memcached or ehcache to boost performance - before you decide to do that make sure you verify if it's worth it.
Are you going to be having about 1000 DB hits per second? You probably need some performance boosting/handling. Per day...maybe not. Per 2-3 days...YAGNI (You ain't gonna need it, so don't worry for now)
So, for every 'resource' that you design in your application (Login is NOT a resource: See related post: Why is form based authentication NOT considered RESTful?) choose the representation. It may involve different queries etc., for you to return json/xml/xhtml (whatever you choose). Each 'DB related call' should be isolated into it's own 'data layer' - I suggest go with Spring JDBC to make your life easier. It'll take the burden of JDBC plumbing off your shoulders so you can focus on creating your DAOs (Data Access Objects - a patter for Data Access classes. All DAOs logically belong in the data layer)
Hope this helps
I have a little problem with my MysqlConnection objets. I have those two connection strings in my app.config :
<connectionStrings>
<add name="bdpvcLocalhost" connectionString="Persist Security Info=False;server=localhost;Uid=root;Password=***;database=bdpvc" providerName="Mysql.Data.MySqlClient"/>
<add name="bdpvcProduction" connectionString="server=pvcserver.pvcservprod.local;user id=pvc_app;Pwd=***;database=bdpvc" providerName="Mysql.Data.MySqlClient"/>
</connectionStrings>
The first connection is for a local test database, the second one is for the real production database. Since i am still early in the developpement, I use my local database. I then use the following code (I cut some of the Parameters definition cause there is really a lot)
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
using (MySqlTransaction transaction = connection.BeginTransaction())
{
const string sequenceQuery = "INSERT INTO sequence " +
"(No_Sequence, No_Client, No_Produit, No_Version_Produit, " +
" No_Soumission, No_Commande, No_Reference, Date_Livraison, " +
"Date_Commande, Date_Soumission, Quantite, Status, Contact, " +
"Notes, No_Production_Ancien, Erreur_Production, No_Fichier_Log, " +
"No_Utilisateur, Date_Enregistrement, Extension_Fichier_Fax, Uuid) " +
"VALUES(?No_Sequence, ?No_Client, ?No_Produit, ?No_Version_Produit, " +
"?No_Soumission, ?No_Commande, ?No_Reference, ?Date_Livraison, ?Date_Commande, " +
"?Date_Soumission, ?Quantite, ?Status, ?Contact, ?Notes, ?No_Production_Ancien, " +
"?Erreur_Production, ?No_Fichier_Log, ?No_Utilisateur, ?Date_Enregistrement, " +
"?Extension_Fichier_Fax, ?Uuid)";
const string selectSequenceNoQuery =
"SELECT MAX(No_Sequence) AS Valeur_No_Increment FROM sequence";
const string updateSequenceNoQuery =
"UPDATE tables SET Valeur_No_Increment = ?Valeur_No_Increment WHERE Nom_Tables = 'sequence'";
int currentIncrement = 0;
MySqlCommand selectNoSequenceCommand = new MySqlCommand(selectSequenceNoQuery, connection,
transaction);
MySqlCommand updateNoSequenceCommand = new MySqlCommand(updateSequenceNoQuery, connection,
transaction);
MySqlCommand insertSequenceCommand = new MySqlCommand(sequenceQuery, connection, transaction);
//------------------------------
//This query execute perfectly!
currentIncrement = Int32.Parse(selectNoSequenceCommand.ExecuteScalar().ToString());
insertSequenceCommand.Parameters.Add("?No_Sequence", MySqlDbType.String);
//Lots and lot of parameters definition
foreach (Sequence sequence in _sequences)
{
currentIncrement++;
sequence.No_Sequence = currentIncrement.ToString();
insertSequenceCommand.Parameters["?No_Sequence"].Value = sequence.No_Sequence;
//Lots and lots of value assignement
//---------------------------------
//But this one doesn't use the right user!
insertSequenceCommand.ExecuteNonQuery();
Console.WriteLine("Inserted new sequence with Uuid " + sequence.Uuid + " and increment " +
sequence.No_Sequence);
}
updateNoSequenceCommand.Parameters.AddWithValue("?Valeur_No_Increment", currentIncrement);
updateNoSequenceCommand.ExecuteNonQuery();
transaction.Commit();
}
}
The first query execute just fine. However, the second query isn't able to execute, because "the user pvc_app doesn't exists". But I am connecting to my local database as root! And never in my code do I switch connection string! I tried deleting the second connection string in app.config, but it kept trying to connect as pvc_app. I rebuild the application multiple time, rebooted my computer and even uninstalled/reinstalled the ADO.NET connector, but it kept trying to connect as pvc_app! Am I doing something wrong here?
Now, where is located this mysterious "connection string cache" so I can kill it with my bare hand? Cause it's really making my life a misery right now.
A workaround of this problem was to create an another user with a default password. The connection was then working fine. It seem to be a bug with ADO.NET connector.