karaf + pax-jdbc the connection pool had reached the limit - mysql

I have a problem with the pool connections of pax-jdbc in karaf, I'm trying to inject a Mysql DataSource (DS) through
blueprint.xml into my project, for test it, I have built a karaf command where injects the DS into karaf command class
and execute a query with that connection. That it's OK, but the problem is when I execute the command a lot times, for
each execution a new instance of the DS is created and the pool connection cannot open new connections to MySQL, because
the pool had reached the limit.
I have uploaded my code to github in this link: https://github.com/christmo/karaf-pax-jdbc , you can give a pull request
if you find an error in this project.
For test this project you can do:
1. Download karaf 4.0.4 or apache-karaf-4.1.0-SNAPSHOT
2. Copy the file karaf-pax-jdbc/etc/org.ops4j.datasource-my-ds.cfg to ${karaf}/etc, this file have the mysql
configuration change with your mysql configuration data.
4. Start mysql database engine
3. Start karaf -> cd ${karaf}/bin/; ./karaf
4. Add the repo of this project with this karaf command: feature:repo-add mvn:pax/features/1.0-SNAPSHOT/xml/features
5. Install the feature created for this project: feature:install mysql-test
6. Execute the command for test this problem: mysql-connection, this command only execute "Select 1" in mysql
If you execute 9 times this command "mysql-connection", it will freeze the prompt of karaf and if you interrupt the
execution you can get this exception:
java.sql.SQLException: Cannot get a connection, general error at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:146)
at com.twim.OrmCommand.execute(OrmCommand.java:53) at
org.apache.karaf.shell.impl.action.command.ActionCommand.execute(ActionCommand.java:83)
at
org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:67)
at
org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:87)
at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
at
org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108) at
org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182) at
org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119) at
org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
at
org.apache.karaf.shell.impl.console.ConsoleSessionImpl.run(ConsoleSessionImpl.java:270)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.lang.InterruptedException at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
... 12 more

The problem in your code is in the line System.out.println("--DS--: " + ds.getConnection());.
There you create a connection but never close it. So with every call you drain the pool.

Related

Getting logs/more information during start-build command execution

Jenkins pipeline is building Docker images. OpenShift plugin(s) are used for the same.
An example command:
openshift.selector(BUILD_CONFIG_NAME, "${appBcName}").startBuild("--from-dir=${artifactPath}", '--wait','--follow')
While this works smoothly most of the time, whenever this command fails due to some underlying platform issues, almost no information is seen in the Jenkins build job console:
[Pipeline] }
[start-build:buildconfig/amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd] ............................................................
[start-build:buildconfig/amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd] Uploading finished
[start-build:buildconfig/amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd] Error from server (BadRequest): unable to wait for build amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd-857 to run: timed out waiting for the condition
[Pipeline] }
ERROR: Error running start-build on at least one item: [buildconfig/amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd];
{err=, verb=start-build, cmd=oc --server=https://api.scp-west-zone02-z01.net:6443 --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --namespace=sb-1166-amld5-car-service-se --token=XXXXX start-build buildconfig/amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd --from-dir=./build/libs --wait --follow -o=name , out=Uploading directory "build/libs" as binary input for the build ...
............................................................
Uploading finished
Error from server (BadRequest): unable to wait for build amld5-car-reporting-spacetime-ubi-openshift-java-runtimejd-857 to run: timed out waiting for the condition
, status=1}
[Pipeline] // catchError
I need more verbosity, detailed error information. I checked the start-build command reference, and I thought --build-loglevel [0-5] might help here. When I used it, I got a warning that since I am using source type as 'Binary' in the BuildConfig, logging isn't supported(seriously???)
NOTE: the selector returned when -F/--follow is supplied to startBuild() will be inoperative for the various selector operations.
Consider removing those options from startBuild and using the logs() command to follow the build output.
[start-build:buildconfig/casc-docs-spacetime-ubi-openshift-java-runtimeadoptopenjdk] WARNING: Specifying --build-loglevel with binary builds is not supported.
[start-build:buildconfig/casc-docs-spacetime-ubi-openshift-java-runtimeadoptopenjdk] WARNING: Specifying environment variables with binary builds is not supported.
[start-build:buildconfig/casc-docs-spacetime-ubi-openshift-java-runtimeadoptopenjdk] Uploading directory "build/libs" as binary input for the build ...
[start-build:buildconfig/casc-docs-spacetime-ubi-openshift-java-runtimeadoptopenjdk] ..
How do I get more logs, info. while executing the start-build command?
I was facing the same problem, I just used something like:
def build = openshift.selector(BUILD_CONFIG_NAME, "${appBcName}").startBuild("--from-dir=${artifactPath}", '--wait','--follow')
build.logs('-f')
And so far it seems to work, I got the logs from my openshift build in my Jenkins pipeline. Now I'll try to get the logs only if build does not Complete, to reduce the overall logs.
(for future searchers like me ^^)

Cannot set configuration in Elastic Beanstalk

I have 4 Elastic Beanstalk deployments: 3 are Corretto 8 and the other one is Corretto 11.
On the Corretto 8 deployments, I can set new configuration without issue. On the Corretto 11 instance, however, any attempt to set a new configuration fails and causes a rollback.
The Corretto versions might not be the problem, but it's the only difference I can see. All 4 apps are Spring Boot apps that run as web servers (i.e embedded tomcat with exposed web ports). I am trying to set the exact same configuration name and value, and it only fails on the one instance.
The configuration I'm trying to set is pretty simple:
VALIDATE_RENEWALS = true
Even just trying to set DEBUG = true causes a failure and rollback.
I don't see a lot of information from the console about what's failing. Here is the event log:
2020-03-16 13:55:17 UTC-0600 INFO The environment was reverted to the previous configuration setting.
2020-03-16 13:54:45 UTC-0600 ERROR During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
2020-03-16 13:54:45 UTC-0600 ERROR Failed to deploy configuration.
2020-03-16 13:54:45 UTC-0600 ERROR Unsuccessful command execution on instance id(s) 'i-00553f4ac36afd327'. Aborting the operation.
2020-03-16 13:54:45 UTC-0600 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-03-16 13:54:45 UTC-0600 ERROR [Instance: i-00553f4ac36afd327] Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].
2020-03-16 13:54:20 UTC-0600 INFO Updating environment XXX's configuration settings.
2020-03-16 13:54:15 UTC-0600 INFO Environment update is starting.
I've also downloaded the full set of logs for the instance and don't see anything obvious. The app stdout doesn't have any errors or exceptions, it just starts normally and then gets terminated. None of the other log files have messages around the times above, so I'm really not sure what else I can look at.
Edit
The times don't line up but I do see this in eb-engine.log file:
2020/03/16 17:54:38.508634 [INFO] checking whether command is applicable to this instance...
2020/03/16 17:54:38.508658 [INFO] this command is applicable to the instance, thus instance should execute command
2020/03/16 17:54:38.508665 [INFO] check whether this is an enhanced env...
2020/03/16 17:54:38.508794 [INFO] Executing instruction: StageJavaApplication
2020/03/16 17:54:38.508858 [ERROR] GetArchivedFileType with file /opt/elasticbeanstalk/deployment/app_source_bundle failed with error open /opt/elasticbeanstalk/deployment/app_source_bundle: no such file or directory
2020/03/16 17:54:38.508868 [ERROR] An error occurred during execution of command [config-deploy] - [StageJavaApplication]. Stop running the command. Error: staging java app failed with error GetArchivedFileType with file /opt/elasticbeanstalk/deployment/app_source_bundle failed with error open /opt/elasticbeanstalk/deployment/app_source_bundle: no such file or directory

How to debug the cartridge scripts in Openshift?

I want to develop a new cartridge for my own use. I use OpenShift Cartridge Development Kit to start my work. My building script is written in .openshift/action_hooks/build and it can be successfully executed.
But when I tried to use the command displayed on the homepage of the CDK project - "rhc create-app mynewcart http://##YOUR-DOMAIN##/manifest/##YOUR-COMMIT-ID##" - to create an app, I got the following error: "Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ex-std-node161.prod.rhcloud.com. This indicates an unexpected problem during the
execution of your request."
How can I trace the progress of deploying and find out where is the code that caused the problem? Is there any log file available for me to analyse?
Use the --debug option when running the command and you'll get much more detailed output:
rhc create-app mynewcart http://##YOUR-DOMAIN##/manifest/##YOUR-COMMIT-ID## --debug

Oozie - Got exception running sqoop: Could not load db driver class: com.mysql.jdbc.Driver

I am trying to perform an sqoop export on HDP sandbox 2.1 via Oozie. When I run the Oozie job I get the following java runtime exception.
'>>> Invoking Sqoop command line now >>>
7598 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR
has not been set in the environment. Cannot check for additional
configuration.
7714 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
1.4.4.2.1.1.0-385
7760 [main] WARN org.apache.sqoop.SqoopOptions - Character argument
'\t' has multiple characters; only the first will be used.
7791 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has
not been set in the environment. Cannot check for additional
configuration.
7904 [main] INFO org.apache.sqoop.manager.MySQLManager - Preparing
to use a MySQL streaming resultset.
7905 [main] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code
generation
7946 [main] ERROR org.apache.sqoop.Sqoop - Got exception running
Sqoop: java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver Intercepting System.exit(1)
I have copied jdbc driver file "mysql-connector-java.jar" to Oozie's shared library folder which I believe is "/usr/lib/oozie/share/lib/sqoop/". I have restarted my sandbox and tried to perform the export with Oozie again and I still get the same error.
The export works perfectly fine when I try performing it only via sqoop, so I presume Oozie needs its own set of drivers.
My question is, which Oozie directory am I suppose to copy my jdbc drivers to?
If you guys think I'm doing something wrong or you need further information, please let me know.
Thank you for your time.
Normally for Oozie the sharelib directory is /user/oozie/share/lib/ on HDFS where "oozie" would be the name of the user which is used to start the Oozie Server. I don't know what that is in case of HDP sandbox 2.1 , but you can use ps command to figure that out.
And for jars needed for sqoop action, I think you should copy the jar to /user/oozie/share/lib/sqoop/ folder.

How to debug SQL Server Agent Jobs?

I two SSIS packages that both run fine on my local machine. I can import them to SSIS on the server and execute them both with no problems. However, I need to run them in series every night at 3AM. Thus, I created a SQL Server Agent Job with two steps. The first step runs the first package successfully. The second step should run the second package but it fails with this error message:
Message
Executed as user: IT-DEV\itdev. Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Copyright (C)
Microsoft Corporation. All rights reserved. Started: 1:27:21 PM
Error: 2013-05-30 13:27:21.53 Code: 0x00000001 Source: Create
text files on IT DEV Description: Exception has been thrown by
the target of an invocation. End Error DTExec: The package execution
returned DTSER_FAILURE (1). Started: 1:27:21 PM Finished: 1:27:21
PM Elapsed: 0.375 seconds. The package execution failed. The step
failed.
Both packages move some files from one directory to another and do some work. They both run in the same security context using the same proxy. I can see the work completed successfully from step 1, but again step 2 fails. I have looked through all possible step settings for each and see no differences. Thus, how can I get a more descriptive message so I know what the real issue is. I'm guessing its permissions but again both packages do similar work in similar locations requiring the same permissions. One runs, the other doesn't. Please help. Thanks in advance.
FYI I've also turned SSIS logging on and write everything to a file but the message below is the only error I see.
UPDATE
I ended up writing to a text file all over inside the script task to try to identify the problem. I seem to be getting stuck here:
string connStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + myPath + ";Extended Properties=\"text;HDR=NO;FMT=TabDelimited\"";
OleDbDataAdapter adapter = new OleDbDataAdapter("SELECT * FROM MYFILE.txt", connStr);
outfile.WriteLine("Adapter Created");
DataTable dtFiles = new DataTable();
adapter.Fill(dtFiles);
outfile.WriteLine("Filled DataTable");
I never see "Filled DataTable" in the file I'm writing to... the last entry says "Adapter Created".
Turns out I needed to replace this: Microsoft.Jet.OLEDB.4.0 with this: Microsoft.ACE.OLEDB.12.0 because I'm running the agent job on Windows Server 2008 64 bit.