How to see my application's exception in hdinsight?
I created an hadoop stream job, when I run my job, it fails with
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
As far as I know, this is because my code has some bug and throws an exception out and then crash, how can I get the exception infomation? Is there a application log or something for hdinsight?
RDP to the head node (you will have to enable remote to the hdinsight cluster) and click on the yarn UI shortcut on the desktop. This will show the task logs.
Related
I am using apache drill in window 10 having latest version (1.9).
I want to start my drill in distributed mode.
I have configure zookeeper zoo.cfg file:-
tickTime=2000
initLimit=10
syncLimit=5
dataDir=F:/zookeepertest/data
clientPort=2181
server.1=192.589.XX.01:2888:3888
server.1=192.565.XX.02:2888:3888
And Drill folder inside drill-override.conf
drill.exec: {
cluster-id: "test",
zk.connect: "192.589.XX.01:2181,192.565.XX.02:2181"
}
And my zookeeper is running..
Now When i trying to start my drill using this command:-
sqlline.bat -u "jdbc:drill:zk=192.589.XX.01:2181"
Its thoughing following error:-
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. (state=,code=0)
java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client.
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:161)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:70)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client.
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:245)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:154)
... 18 more
Caused by: java.io.IOException: Failure to connect to the zookeeper cluster service within the allotted time of 10000 milliseconds.
at org.apache.drill.exec.coord.zk.ZKClusterCoordinator.start(ZKClusterCoordinator.java:123)
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:243)
... 19 more
Can anyone tell how to start drill in distributed mode in window.??
From Drill documentation
To start drill in distributed mode, use drillbit.sh not sqlline i.e.
drillbit.sh start
Since drillbit.sh is a shell script (not a windows batch job) you'll need a 3rd party shell scripting tool such as Cygwin or since you're using Windows 10, you can also enable Bash on Ubuntu.
Trying to run a simple hdfs query failed with:
[ms#cosmosmaster-gi ~]$ hadoop fs -ls /user/ms/def_serv/def_servpath
Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file:
/tmp/hsperfdata_ms/21066
Try using the -Djava.io.tmpdir= option to select an alternate temp location.
Exception in thread "main" java.lang.NoClassDefFoundError: ___/tmp/hsperfdata_ms/21078
Caused by: java.lang.ClassNotFoundException: ___.tmp.hsperfdata_ms.21078
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: ___/tmp/hsperfdata_ms/21078. Program will exit.
Any idea how to fix that or increase quota?
Thanks!
ms
Your quota has not been exceeded (see command below), but this was a problem with the cluster. It should be fixed now.
$ hadoop fs -dus /user/ms
hdfs://cosmosmaster-gi/user/ms 90731
Quick question about sonar/jenkins integration.
First, a little background - we're working on implementing the build pipeline plugin, and last night we had an issue where one of our pipeline jobs lost its history. This took out all of our executors (even on the slaves). I tried renaming and I tried bouncing and that didn't work. Finally, I brought down the master, cleared everything from temp and work in Tomcat, and brought it back up. This fixed the issues with my executors.
So this morning I run a build that runs Sonar as a post-build step. Now I am seeing this error:
[ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:sonar (default-cli) on project VendorProduct: Can not execute Sonar: The current batch process and the configured remote server do not share the same DB configuration.
[ERROR] - Batch side: jdbc:oracle:thin:#dansrzl105si.wellsfargo.com:3203/LBLDFRI1 (QMTDO / *****)
[ERROR] - Server side: check the configuration at http://lpwva3279:9000/sonar/system
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:sonar (default-cli) on project VendorProduct: Can not execute Sonar
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: Can not execute Sonar
at org.codehaus.mojo.sonar.Bootstraper.executeMojo(Bootstraper.java:118)
at org.codehaus.mojo.sonar.Bootstraper.start(Bootstraper.java:65)
at org.codehaus.mojo.sonar.SonarMojo.execute(SonarMojo.java:90)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: org.sonar.core.persistence.BadDatabaseVersion: The current batch process and the configured remote server do not share the same DB configuration.
- Batch side: jdbc:oracle:thin:#dansrzl105si.wellsfargo.com:3203/LBLDFRI1 (QMTDO / *****)
- Server side: check the configuration at http://lpwva3279:9000/sonar/system
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
I've verified the configuration against the sonar configuration file, deleted and recreated the post-build step, deleted and recreated the sonar instance on the Jenkins configuration page. I'm at a loss here. Any one have any suggestions?
Thanks!
This message is printed out when a Sonar batch connects to a database which is not the same as the one configured on the Sonar Web server.
In your case, if you go to http://lpwva3279:9000/sonar/system, there are chances that the DB config settings that you'll find are not the same as jdbc:oracle:thin:#dansrzl105si.wellsfargo.com:3203/LBLDFRI1 (QMTDO / *****). You should then go to your Jenkins settings and update the information relating to Sonar to match the one found on the Sonar Web server.
Go to http://host:port/sonar/system:
Copy value "Database URL" into Jenkins configuration ("Database URL")
Copy value "Database Login" into Jenkins configuration ("Database Login")
Copy value "Database Driver" into Jenkins configuration ("Database Driver")
And go to SONAR_HOME/conf/sonar.properties:
Copy value "Database password" into Jenkins configuration ("Database password")
I've been dealing with this error message for a while now and nobody has been able to help me...
I am trying to probe a java application using the standalone TPTP probekit agent with the following linux command:
java '-agentlib:JPIBootLoader=JPIAgent:server=standalone;ProbekitAgent:ext-pk-BCILibraryName=BCIEngProbe,ext-pk-probescript=ome/anis/qf-t/core/probe/testProbe.probescript' junit.textui.TestRunner quickfix.test.acceptance.AcceptanceTestSuite
My application is actually a junit testcase from the Quickfixj package (quickfixj.org)...
Running the above command, the probing starts and I am able to see the runtime traces. However, at some point during execution I am getting the runtime exception:
java.lang.NoClassDefFoundError: testProbe_probe$Probe_0
25-Jan-2012 10:25:11 AM quickfix.test.acceptance.ATServer run
SEVERE: error in AT server
java.lang.NoClassDefFoundError: testProbe_probe$Probe_0
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.<init>(DocumentBuilderFactoryImpl.java)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:147)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:233)
at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:123)
at quickfix.DataDictionary.load(DataDictionary.java:906)
at quickfix.DataDictionary.read(DataDictionary.java:893)
at quickfix.DataDictionary.<init>(DataDictionary.java:109)
at quickfix.DefaultSessionFactory.getDataDictionary(DefaultSessionFactory.java:325)
at quickfix.DefaultSessionFactory.createDataDictionary(DefaultSessionFactory.java:219)
at quickfix.DefaultSessionFactory.processFixtDataDictionaries(DefaultSessionFactory.java:258)
at quickfix.DefaultSessionFactory.create(DefaultSessionFactory.java:113)
at quickfix.mina.acceptor.AbstractSocketAcceptor.createSessions(AbstractSocketAcceptor.java:242)
at quickfix.mina.acceptor.AbstractSocketAcceptor.startAcceptingConnections(AbstractSocketAcceptor.java:99)
at quickfix.SocketAcceptor.initialize(SocketAcceptor.java:66)
at quickfix.SocketAcceptor.start(SocketAcceptor.java:59)
at quickfix.test.acceptance.ATServer.run(ATServer.java:193)
at java.lang.Thread.run(Thread.java:662)**
which states that my probe file is missing. But this file has been needed in the first place to print the traces!!???.....
Well, here are some facts:
I am able to probe other applications, including a simple junit test:
java '-agentlib:JPIBootLoader=JPIAgent:server=standalone;ProbekitAgent:ext-pk-BCILibraryName=BCIEngProbe,ext-pk-probescript=ome/anis/qf-t/core/probe/testProbe.probescript' junit.textui.TestRunner quickfix.test.acceptance.AnisJUnitTestExample
I am able to run the quickfix.test.acceptance.AcceptanceTestSuite using junit (no probing):
java junit.textui.TestRunner quickfix.test.acceptance.AcceptanceTestSuite
I have set the path to testProbe_probe$Probe_0 in CLASSPATH
I have called the System.getenv("CLASSPATH") from within my testcase and the CLASSPATH seems to be fine
My testcase deals with threads, sockets, etc
And my system configuration is:
-os: ubuntu 10.04-64bit (on VirtualBox!)
-java version "1.6.0_24"
-junit: junit-4.10.jar
-TPTP agent controller: agntctrl.linux_em64t-TPTP-4.7.2
Any ideas??
Thanks,
Anis
hibernate 3.6.8 final
c3p0 jar that came with hibernate 3.6.8 package -> c3p0-0.9.1.jar
1
15
40
0
5
2
The app seems to be working fine, however I get massive log calls with the following stacktrace:
org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2411)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor38.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:152)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1074)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1061)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1796)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:620)
Any information on how to remove that INFO log would be very much helpful thanks!
UPDATE: Is this a critical error? Or can should I just ignore it?
After a search on the web regarding this issue, I found some similar issues reported by several people. All of them point to a common problem: Threads. Basically, if you start new threads in your application (either in your code or by using a third party tool like Quartz, you have to make sure that all of the threads are stopped appropriately when the application is undeployed from the server. Here are some quotes from the searches:
Mikolaj Rydzewski wrote:
It looks like after webapp's instance has been undeployed, background quartz thread wants to do something and then exception occurs.
Another (and better explanation) on jspwiki.org:
It is possible that this is caused by Tomcat unsuccessfully reloading the web application. The app is unloaded, but all threads don't get shut down properly. As a result, when the threads try to run, they get clobbered by the fact that Tomcat has shut down its classloader, and an error is logged.
So, in order to solve this issue you have to make sure all threads started by your application will be stopped at application undeployment (or redeployment, it's the same). You can do this by registering a ServletContextListener to your application server and stopping your threads inside contextDestroyed(ServletContextEvent) method.
If you are using log4j, change your logging settings to something like this (the word ERROR replace INFO) :
log4j.rootLogger=ERROR, file, stdout
log4j.logger.org.hibernate=ERROR
They are located in the the log4j.properties file in yr project.
OK I switched to boneCP, c3p0 does not really seem to work for java6!!!