Facing out of memory error on initializing Couchbase Java Client. The issue happens in the context of running Test cases in the Gradle build. It doesn't seem to be happening when running individual test cases. It seems to be happening on running all test cases in the build. The error is occurring on MacOS and not on Linux build machine
Environment
JVM = 16 (OpenJDK)
OS = MacOS Monterey
task = Gradle build
jvm memory settings = -Xmx8000m" "-Xms512m" "-XX:MaxDirectMemorySize=2000m"
StackTrace -
Caused by: java.lang.OutOfMemoryError: Cannot reserve 16384 bytes of direct buffer memory (allocated: 536861104, limit: 536870912)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:330)
at com.couchbase.client.core.deps.io.netty.channel.unix.Buffer.allocateDirectWithNativeOrder(Buffer.java:40)
at com.couchbase.client.core.deps.io.netty.channel.unix.IovArray.<init>(IovArray.java:72)
at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoop.<init>(KQueueEventLoop.java:62)
at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoopGroup.newChild(KQueueEventLoopGroup.java:151)
at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoopGroup.newChild(KQueueEventLoopGroup.java:32)
at com.couchbase.client.core.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at com.couchbase.client.core.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
at com.couchbase.client.core.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:49)
at com.couchbase.client.core.deps.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoopGroup.<init>(KQueueEventLoopGroup.java:110)
at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoopGroup.<init>(KQueueEventLoopGroup.java:97)
at com.couchbase.client.core.deps.io.netty.channel.kqueue.KQueueEventLoopGroup.<init>(KQueueEventLoopGroup.java:73)
at com.couchbase.client.core.env.IoEnvironment.createEventLoopGroup(IoEnvironment.java:476)
at com.couchbase.client.core.env.IoEnvironment.<init>(IoEnvironment.java:285)
at com.couchbase.client.core.env.IoEnvironment.<init>(IoEnvironment.java:66)
at com.couchbase.client.core.env.IoEnvironment$Builder.build(IoEnvironment.java:674)
at com.couchbase.client.core.env.CoreEnvironment.<init>(CoreEnvironment.java:153)
at com.couchbase.client.java.env.ClusterEnvironment.<init>(ClusterEnvironment.java:53)
at com.couchbase.client.java.env.ClusterEnvironment.<init>(ClusterEnvironment.java:46)
at com.couchbase.client.java.env.ClusterEnvironment$Builder.build(ClusterEnvironment.java:213)
Related
I have 4 Elastic Beanstalk deployments: 3 are Corretto 8 and the other one is Corretto 11.
On the Corretto 8 deployments, I can set new configuration without issue. On the Corretto 11 instance, however, any attempt to set a new configuration fails and causes a rollback.
The Corretto versions might not be the problem, but it's the only difference I can see. All 4 apps are Spring Boot apps that run as web servers (i.e embedded tomcat with exposed web ports). I am trying to set the exact same configuration name and value, and it only fails on the one instance.
The configuration I'm trying to set is pretty simple:
VALIDATE_RENEWALS = true
Even just trying to set DEBUG = true causes a failure and rollback.
I don't see a lot of information from the console about what's failing. Here is the event log:
2020-03-16 13:55:17 UTC-0600 INFO The environment was reverted to the previous configuration setting.
2020-03-16 13:54:45 UTC-0600 ERROR During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
2020-03-16 13:54:45 UTC-0600 ERROR Failed to deploy configuration.
2020-03-16 13:54:45 UTC-0600 ERROR Unsuccessful command execution on instance id(s) 'i-00553f4ac36afd327'. Aborting the operation.
2020-03-16 13:54:45 UTC-0600 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-03-16 13:54:45 UTC-0600 ERROR [Instance: i-00553f4ac36afd327] Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].
2020-03-16 13:54:20 UTC-0600 INFO Updating environment XXX's configuration settings.
2020-03-16 13:54:15 UTC-0600 INFO Environment update is starting.
I've also downloaded the full set of logs for the instance and don't see anything obvious. The app stdout doesn't have any errors or exceptions, it just starts normally and then gets terminated. None of the other log files have messages around the times above, so I'm really not sure what else I can look at.
Edit
The times don't line up but I do see this in eb-engine.log file:
2020/03/16 17:54:38.508634 [INFO] checking whether command is applicable to this instance...
2020/03/16 17:54:38.508658 [INFO] this command is applicable to the instance, thus instance should execute command
2020/03/16 17:54:38.508665 [INFO] check whether this is an enhanced env...
2020/03/16 17:54:38.508794 [INFO] Executing instruction: StageJavaApplication
2020/03/16 17:54:38.508858 [ERROR] GetArchivedFileType with file /opt/elasticbeanstalk/deployment/app_source_bundle failed with error open /opt/elasticbeanstalk/deployment/app_source_bundle: no such file or directory
2020/03/16 17:54:38.508868 [ERROR] An error occurred during execution of command [config-deploy] - [StageJavaApplication]. Stop running the command. Error: staging java app failed with error GetArchivedFileType with file /opt/elasticbeanstalk/deployment/app_source_bundle failed with error open /opt/elasticbeanstalk/deployment/app_source_bundle: no such file or directory
Am running a set of test cases from Jenkins. All the test cases are failing with org.openqa.selenium.TimedoutException on all the functions. I have tested the same code using eclipse in local and its executing fine. Please check on the below error and help on resolving this issue
Configurations:
Selenium- standalone - jar - 2.53
Chromedriver - 2.36
Chrome Browser - 65
testng jar = 6.8
Code Sample:
WebDriverWait wait = new WebDriverWait(driver, 1000);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(ORCollection.get("BtnMDUEligibilityClick").toString())));
WebElement el = driver.findElement(By.xpath(ORCollection.get("BtnMDUEligibilityClick").toString()));
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", el);
WebElementClick(" ",ORCollection.get("BtnMDUEligibilityClick").toString());
Thread.sleep(10000);
Logs:
org.openqa.selenium.TimeoutException: Timed out after 40 seconds waiting for visibility of element located by By.xpath: //button[contains(#class,'js-submit')]
I have increased webdriver.wait value in code from 40 seconds to 1000 but still we are getting the same error as Timedout.
I assume that your XPath expressions are correct and you can manually open the browser on the Jenkins slave machine.
The issue could be connected with the lack of interactive desktop. This could happen if you have installed Jenkins on the slave machine as windows service. The recommended for Windows way to control a Windows slave is by using Launch slave agents via Java Web Start. More info here and here.
After building the app, the deployment process works. The SpringBoot (mysql) application is deployed and marked as the usual "Starting the application".
After a few seconds, there is a message saying it "kills" the application. Notice, I am just within the limits of my Openshift free quota.
What is the mesage? .../wildfly/bin/standalone.sh: line 307: 183 Killed "java" -D"[Standalone]"
See the line in bold.
On another time I see is that the application is started, I can even login, but then after a minute is is killed.
14:47:15,334 INFO [nl.xyz.Application] (ServerService Thread Pool --
66) Started Application in 26.49 seconds (JVM running for 57.227)
14:47:15,723 INFO [javax.enterprise.resource.webcontainer.jsf.config]
(ServerService Thread Pool -- 66) Initializing Mojarra 2.2.13.SP1
20160303-1204 for context '' /wildfly/bin/standalone.sh: line 307: 183
Killed "java" -D"[Standalone]" -server -XX:+UseParallelGC -Xms40m
-Xmx250m -XX:+AggressiveOpts -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dorg.apache.tomcat.util.LOW_MEMORY=true -DOPENSHIFT_APP_UUID= -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djboss.node.name=solvedcaches-5-80fj7 -Djgroups.bind_addr=0.0.0.0 -Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on "-Dorg.jboss.boot.log.file=/wildfly/standalone/log/server.log"
"-Dlogging.configuration=file:/wildfly/standalone/configuration/logging.properties"
-jar "/wildfly/jboss-modules.jar" -mp "/wildfly/provided_modules:/wildfly/modules" org.jboss.as.standalone
-Djboss.home.dir="/wildfly" -Djboss.server.base.dir="/wildfly/standalone" '-b' '0.0.0.0' '-bmanagement' '0.0.0.0'
The last 2 build failed due to a generic build failture. Strange, the code was hardly changed. It looks like rebuilding takes memory and is competing with the running application and vice versa. The last time I stopped the POD, then deployment was ok.
Thank you #Jiri Fialka for all your help, online / offline.
As you suggested, after configuring the memory over the 2 POD's the application is working fine. Now I use 410 Mb for Mysql and 600 Mb for SpringBoot was WAR on a Wildfly pod. Even SSH was added within minutes!.
I am enthusiastic about the new Openshift v3 facilities. Also the GUI and oc.
I have a Dell Precision Rack running Ubuntu Precise and featuring two Tesla C2075 plus a Quadro 600 which is the display device. I have recently finished some tests on my desktop-computer and now tried to port stuff to the workstation.
Since CUDA was not present I installed it according to this guide and adapted the SDK Makefiles according to this suggestions.
What I am now facing is that not a single sample (I did test like 10 different ones) is running. Those are the errors I am getting:
[deviceQuery] starting...
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 10
-> invalid device ordinal
[deviceQuery] test results...
FAILED
> exiting in 3 seconds: 3...2...1...done!
[MonteCarloMultiGPU] starting...
CUDA error at MonteCarloMultiGPU.cpp:235 code=23510 (cudaErrorInvalidDevice) "cudaGetDeviceCount(&GPU_N)"MonteCarloMultiGPU
==================
Parallelization method = threaded
Problem scaling = weak
Number of GPUs = 0
Total number of options = 0
Number of paths = 262144
main(): generating input data...
main(): starting 0 host threads...
Floating point exception (core dumped)
[reduction] starting...
reduction.cpp(124) : cudaSafeCallNoSync() Runtime API error 10 : invalid device ordinal.
[simplePrintf] starting...
simplePrintf.cu(193) : CUDA Runtime API error 10: invalid device ordinal.
As you can see most of the errors are pointing towards a problem with the cudaGetDeviceCount call which return error code 10. According to the manual the problem is:
cudaErrorInvalidDevice: This indicates that the device ordinal supplied by the user does not correspond to a valid CUDA device.
Unfortunately, the only solution I was able to find suggested to check the devices power plugs. I did that and there was nothing wrong with it. Restarting the workstation does not help either.
I'd be happy to supply more details on my configuration. Just leave a comment!
Due to the comments to my original question I was able to find a solution. I followed this guide to learn how to set up the rc.local correctly (don't forget to chmod your script).
Im getting the following error while running pmd, findbugs in hudson.
FATAL: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.(Unknown Source)
at hudson.plugins.analysis.util.model.AnnotationContainer.initialize(AnnotationContainer.java:163)
at hudson.plugins.analysis.util.model.AnnotationContainer.(AnnotationContainer.java:119)
at hudson.plugins.analysis.util.model.WorkspaceFile.(WorkspaceFile.java:25)
at hudson.plugins.analysis.util.model.AnnotationContainer.addFile(AnnotationContainer.java:303)
at hudson.plugins.analysis.util.model.AnnotationContainer.updateMappings(AnnotationContainer.java:212)
at hudson.plugins.analysis.util.model.AnnotationContainer.addAnnotation(AnnotationContainer.java:317)
at hudson.plugins.analysis.util.model.AnnotationContainer.addAnnotations(AnnotationContainer.java:327)
at hudson.plugins.analysis.core.BuildResult.initialize(BuildResult.java:266)
at hudson.plugins.analysis.core.BuildResult.(BuildResult.java:206)
at hudson.plugins.findbugs.FindBugsResult.(FindBugsResult.java:40)
at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:132)
at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:280)
at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:560)
at hudson.model.Build$RunnerImpl.post2(Build.java:156)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:529)
at hudson.model.Run.run(Run.java:1349)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:139)
I have specified jvmargs for pmd, findbugs and ant_opts is set to 1024M.
If you are in windows you must edit the hudson.xml file to include the java argument
-Xmx1024m, however if you're on a *nix environment you must run the command
export JAVA_OPTS=-Xmx1024m
Or a more reasonable size for memory.