Gradle UnitTest | javax.xml.bind.PropertyException: name: eclipselink.media-type value: application/json - json

Need guidance on how to solve the issue occurring while executing a gradle unit-test involving marshaling of a JPA Entity using EclipseLink MOXy jaxb library. The same unit test runs successfully when ran as Junit Test from eclipse. Here is the exception stack trace. My suspect is gradle classpath issue.
javax.xml.bind.PropertyException: name: eclipselink.media-type value: application/json
at javax.xml.bind.helpers.AbstractMarshallerImpl.setProperty(AbstractMarshallerImpl.java:358)
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.setProperty(MarshallerImpl.java:527)
I ensured that a jaxb.index file available in com.dams.foo package and listing all the Entities. Here is code sinnpet involved in json marshalling
public static String toJson(Object objectToSeralize) throws Exception {
JAXBContext jaxbContext = JAXBContext.newInstance("com.dams.foo");
StringWriter stringWriter = new StringWriter();
Marshaller marshaller = jaxbContext.createMarshaller();
marshaller.setProperty("eclipselink.media-type", "application/json");
marshaller.marshal(objectToSeralize, stringWriter);
return stringWriter.toString();
}
I see the required jars are being listed in gradle debug under application classpath
15:37:43.196 [DEBUG] [org.gradle.process.internal.worker.DefaultWorkerProcessBuilder] Creating Gradle Test Executor 1
15:37:43.196 [DEBUG] [org.gradle.process.internal.worker.DefaultWorkerProcessBuilder] Using application classpath [C:\dev\build\classes\test, C:\dev\build\resources\test, C:\dev\build\classes\main, C:\dev\build\resources\main, ..... C:\Users\dams\.gradle\caches\modules-2\files-2.1\org.eclipse.persistence\eclipselink\2.6.4\526cc0ddb69c01784e7e9b0a048f39dc313403cb\eclipselink-2.6.4.jar .......]
But during test execution I don't see the application classpath being used, but just gradle-worker.jar file only as shown below
15:37:43.212 [INFO] [org.gradle.process.internal.DefaultExecHandle] Starting process 'Gradle Test Executor 1'. Working directory: C:\dev Command: C:\Program Files\Java\jdk1.8.0_102\bin\java.exe -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dfile.encoding=windows-1252 -Duser.country=IN -Duser.language=en -Duser.variant -ea -cp C:\Users\dams\.gradle\caches\3.3\workerMain\gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'
Please help to resolve this issue?
Here is dependencies section of build.gradle file
dependencies {
compile 'com.google.guava:guava:20.0'
compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.25'
compile group: 'org.slf4j', name: 'slf4j-log4j12', version: '1.7.25'
compile group: 'com.fasterxml.uuid', name: 'java-uuid-generator', version: '3.1.4'
compile group: 'org.eclipse.persistence', name: 'eclipselink', version: '2.6.4'
compile group: 'com.zaxxer', name: 'HikariCP', version: '2.6.0'
compile group: 'com.google.code.gson', name: 'gson', version: '2.8.0'
compile group: 'mysql', name: 'mysql-connector-java', version: '6.0.5'
compile group: 'com.googlecode.genericdao', name: 'dao', version: '1.2.0'
testCompile 'junit:junit:4.12'
}
stack trace from gradle debug mode
com.ct.test.pm.TestJsonSerializer > testJsonSerilalizer STANDARD_ERROR
javax.xml.bind.PropertyException: name: eclipselink.media-type value: application/json
at javax.xml.bind.helpers.AbstractMarshallerImpl.setProperty(AbstractMarshallerImpl.java:358)
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.setProperty(MarshallerImpl.java:527)
at com.ct.db.util.CustomJsonSerializerUtility.toJson(CustomJsonSerializerUtility.java:41)
at com.ct.test.pm.TestJsonSerializer.testJsonSerilalizer(TestJsonSerializer.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy3.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
and the output of gradle --stacktrace test is :
gradle --stacktrace test
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test
com.ct.test.pm.TestJsonSerializer > testJsonSerilalizer FAILED
java.lang.AssertionError at TestJsonSerializer.java:52
2 tests completed, 1 failed
:test FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///C:/dev/build/reports/tests/test/index.html
* Try:
Run with --info or --debug option to get more log output.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':test'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:84)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:62)
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:88)
at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker$1.execute(DefaultTaskGraphExecuter.java:236)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker$1.execute(DefaultTaskGraphExecuter.java:228)
at org.gradle.internal.Transformers$4.transform(Transformers.java:169)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:106)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:61)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:228)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:215)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:77)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:58)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:32)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:113)
at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
at org.gradle.initialization.DefaultGradleLauncher$3.execute(DefaultGradleLauncher.java:196)
at org.gradle.initialization.DefaultGradleLauncher$3.execute(DefaultGradleLauncher.java:193)
at org.gradle.internal.Transformers$4.transform(Transformers.java:169)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:106)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:56)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:193)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:119)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:102)
at org.gradle.launcher.exec.GradleBuildController.run(GradleBuildController.java:71)
at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:41)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:75)
at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:49)
at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:44)
at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:29)
at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:47)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
at org.gradle.util.Swapper.swap(Swapper.java:38)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:72)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
Caused by: org.gradle.api.GradleException: There were failing tests. See the report at: file:///C:/dev/build/reports/tests/test/index.html
at org.gradle.api.tasks.testing.Test.handleTestFailures(Test.java:1250)
at org.gradle.api.tasks.testing.Test.executeTests(Test.java:662)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.doExecute(DefaultTaskClassInfoStore.java:141)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.execute(DefaultTaskClassInfoStore.java:134)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.execute(DefaultTaskClassInfoStore.java:123)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:632)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:615)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:95)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:76)
... 70 more
BUILD FAILED

Related

apache drill java.lang.NullPointerException

I am trying to use drill in embedded mode under windows. But I am having the following error:
> sqlline.bat -u "jdbc:drill:zk=local"
DRILL_ARGS - " -u jdbc:drill:zk=local"
Calculating HADOOP_CLASSPATH ...
HBASE_HOME not detected...
Calculating Drill classpath...
java.lang.NullPointerException
apache drill 1.13.0
"a drill is a terrible thing to waste"
0: jdbc:drill:zk=local>
0: jdbc:drill:zk=local>
0: jdbc:drill:zk=local>
0: jdbc:drill:zk=local> show tables;
No current connection
0: jdbc:drill:zk=local>
You can see there is no current connection and also the web console does not work either. Thanks
Edit: Add log file content.
2018-05-31 03:44:27,981 [main] INFO
o.a.drill.common.config.DrillConfig - Configuration and plugin file(s)
identified in 115ms. Base Configuration:
- jar:file:/C:/apache-drill-1.13.0/jars/drill-common-1.13.0.jar!/drill-default.conf
Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/C:/apache-drill-1.13.0/jars/drill-storage-kafka-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-opentsdb-storage-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-hive-exec-shaded-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-java-exec-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-storage-hbase-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-kudu-storage-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-logical-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-jdbc-storage-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-mongo-storage-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-gis-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-memory-base-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-common-1.13.0.jar!/drill-module.conf
- jar:file:/C:/apache-drill-1.13.0/jars/drill-storage-hive-core-1.13.0.jar!/drill-module.conf
Override File: file:/C:/apache-drill-1.13.0/conf/drill-override.conf
Overridden Properties: -user = -password = -zk = local
2018-05-31 03:44:28,278 [main] WARN o.a.drill.exec.util.GuavaPatcher
- Unable to patch Guava classes. javassist.CannotCompileException: by java.lang.LinkageError: loader (instance of
sun/misc/Launcher$AppClassLoader): attempted duplicate class
definition for name: "com/google/common/base/Stopwatch" at
javassist.ClassPool.toClass(ClassPool.java:1099)
~[javassist-3.16.1-GA.jar:na] at
javassist.ClassPool.toClass(ClassPool.java:1042)
~[javassist-3.16.1-GA.jar:na] at
javassist.ClassPool.toClass(ClassPool.java:1000)
~[javassist-3.16.1-GA.jar:na] at
javassist.CtClass.toClass(CtClass.java:1140)
~[javassist-3.16.1-GA.jar:na] at
org.apache.drill.exec.util.GuavaPatcher.patchStopwatch(GuavaPatcher.java:66)
~[drill-java-exec-1.13.0.jar:1.13.0] at
org.apache.drill.exec.util.GuavaPatcher.patch(GuavaPatcher.java:36)
~[drill-java-exec-1.13.0.jar:1.13.0] at
org.apache.drill.exec.server.Drillbit.(Drillbit.java:69)
[drill-java-exec-1.13.0.jar:1.13.0] at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:134)
[drill-jdbc-1.13.0.jar:1.13.0] at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
[drill-jdbc-1.13.0.jar:1.13.0] at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
[drill-jdbc-1.13.0.jar:1.13.0] at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
[avatica-1.10.0.jar:1.10.0] at
org.apache.drill.jdbc.Driver.connect(Driver.java:72)
[drill-jdbc-1.13.0.jar:1.13.0] at
sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.Commands.connect(Commands.java:1083)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.Commands.connect(Commands.java:1015)
[sqlline-1.1.9-drill-r7.jar:na] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_171] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_171] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498)
~[na:1.8.0_171] at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.SqlLine.dispatch(SqlLine.java:742)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.SqlLine.initArgs(SqlLine.java:528)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.SqlLine.begin(SqlLine.java:596)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.SqlLine.start(SqlLine.java:375)
[sqlline-1.1.9-drill-r7.jar:na] at
sqlline.SqlLine.main(SqlLine.java:268) [sqlline-1.1.9-drill-r7.jar:na]
Caused by: java.lang.LinkageError: loader (instance of
sun/misc/Launcher$AppClassLoader): attempted duplicate class
definition for name: "com/google/common/base/Stopwatch" at
java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_171] at
java.lang.ClassLoader.defineClass(ClassLoader.java:763)
~[na:1.8.0_171] at
java.lang.ClassLoader.defineClass(ClassLoader.java:642)
~[na:1.8.0_171] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_171] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_171] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498)
~[na:1.8.0_171] at javassist.ClassPool.toClass2(ClassPool.java:1112)
~[javassist-3.16.1-GA.jar:na] at
javassist.ClassPool.toClass(ClassPool.java:1093)
~[javassist-3.16.1-GA.jar:na]
You have incompatible Guava version with Drill's one in your HADOOP_CLASSPATH.
You can find similar issues here: DRILL-4931,
DRILL-4928
Try to exclude Guava jars from your HADOOP_CLASSPATH or just unset it by:
~$unset HADOOP_CLASSPATH
This issue should be resolved after merging PR for DRILL-6422

spark throws exception when querying large amount of data in mysql

## when i submit my task to do some query from mysql to yarn by using spark's cluster mode like below: ##
./spark-submit --class org.com.scala.test.ScalaTestFile --master yarn --deploy-mode cluster --driver-memory 8g --executor-memory 5g --jars /usr/local/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/local/spark/lib/datanucleus-core-3.2.10.jar,/usr/local/spark/lib/datanucleus-rdbms-3.2.9.jar,/usr/local/spark/lib/mysql-connector-java-5.1.26-bin.jar /data/tmp/snodawn/svn/scalaScript/scalaMavenTest/out/artifacts/scalaMavenTest_jar/scalaMavenTest.jar
- org.com.scala.test.ScalaTestFile is to query large amount of data in mysql(which for about 1 billion lines), and save it to hive :
val conf = new SparkConf().setAppName("ScalaTestFile")
val spark = new SparkContext(conf)
val sqlContext = new SQLContext(spark)
val hiveContext = new HiveContext(spark);
val reader = hiveContext.read.format("jdbc")
val url="jdbc:mysql://xx.xx.xx.xx:3307/databases"
reader.option("url",url)
reader.option("driver","com.mysql.jdbc.Driver")
reader.option("user","admin")
reader.option("password","admin")
reader.option("dbtable","(select * from gold) as a")
val df = reader.load()
val nTable = df.toDF();
val nWrite = nTable.write
hiveContext.sql("use testment")
nWrite.saveAsTable("gold_test")
- the task will be failed after running with such error:
16/02/24 18:46:43 INFO DAGScheduler: Job 0 failed: saveAsTable at ScalaTestFile.scala:88, took 1697.970350 s
16/02/24 18:46:43 ERROR InsertIntoHadoopFsRelation: Aborting job.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256)
at org.apache.spark.sql.hive.execution.CreateMetastoreDataSourceAsSelect.run(commands.scala:258)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:251)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:221)
at org.com.scala.test.ScalaTestFile$.main(ScalaTestFile.scala:88)
at org.com.scala.test.ScalaTestFile.main(ScalaTestFile.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
16/02/24 18:46:44 ERROR DefaultWriterContainer: Job job_201602241818_0000 aborted.
16/02/24 18:46:44 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted.
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:156)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at org.apache.spark.sql.hive.execution.CreateMetastoreDataSourceAsSelect.run(commands.scala:258)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:251)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:221)
at org.com.scala.test.ScalaTestFile$.main(ScalaTestFile.scala:88)
at org.com.scala.test.ScalaTestFile.main(ScalaTestFile.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, slave4.4399data.com): ExecutorLos
tFailure (executor 4 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 122713 ms
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:150)
... 33 more
16/02/24 18:46:44 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.SparkException: Job aborted.)
it seems that because it needs a log of time for querying from mysql and there returns no responses for a long time, this application finishes with failed status.
so, how can i solve my problem of querying from mysql for getting large amount of data by using spark?
ps, when i use java to do such query, i will do it like this:
Connection conn = DriverManager.getConnection(hiveConnectString, username, password);
com.mysql.jdbc.Statement statement = (com.mysql.jdbc.Statement)conn.createStatement();
statement.enableStreamingResults();
statement.executeUpdate("select * from gold")
so, is there a solution in spark to handling big data querying?

How to set a bucket password with spring-data couchbase

I have followed the tutorial for spring-data couchbase and have a succesfull example project with unit tests for persisting a number of custom entities with a range of views implemented to query the entities.
This works correctly in both a local dev environment and a ci environment when using the "default" bucket name and no password as the authentication.
Moving beyond the example, I want to make use of a different bucket and ultimately make use of a password.
When I create a new bucket (named "test_bucket"), and update the property injected into the CouchbaseConfig (extends AbstractCouchbaseConfiguration) to use this new bucket inplace of the "default" I get the following exception when running the unit tests.
I also tried adding a password to the creation script and adding the same password ("psswd" string in both cases) to the properties used in the CouchbaseConfig but get the same exception shown below.
So is it possible to use another bucket than "default" (and its no-authorisation required) and how do I configure a password for use on this bucket ?
I have verified that the bucket(s) and the expected views have been created correctly in couchbase from the Admin GUI.
2015-06-09 16:41:40 INFO ClasspathLoggingApplicationListener:55 - Application failed to start with classpath: [file:/C:/tools/cmd/cygwin64/home/akirby/workspaces/repos/blackjack/persistence/target/surefire/surefirebooter7615727324811258159.jar]
2015-06-09 16:41:40 INFO AutoConfigurationReportLoggingInitializer:107 -
Error starting ApplicationContext. To display the auto-configuration report enabled debug logging (start with --debug)
2015-06-09 16:41:40 ERROR SpringApplication:338 - Application startup failed
java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeBase64String([B)Ljava/lang/String;
at com.couchbase.client.http.HttpUtil.buildAuthHeader(HttpUtil.java:55)
at com.couchbase.client.ViewConnection.addOp(ViewConnection.java:205)
at com.couchbase.client.CouchbaseClient.addOp(CouchbaseClient.java:803)
at com.couchbase.client.CouchbaseClient.asyncGetView(CouchbaseClient.java:342)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:430)
at org.springframework.data.couchbase.core.CouchbaseTemplate$2.doInBucket(CouchbaseTemplate.java:223)
at org.springframework.data.couchbase.core.CouchbaseTemplate$2.doInBucket(CouchbaseTemplate.java:220)
at org.springframework.data.couchbase.core.CouchbaseTemplate.execute(CouchbaseTemplate.java:244)
at org.springframework.data.couchbase.core.CouchbaseTemplate.queryView(CouchbaseTemplate.java:220)
at org.springframework.data.couchbase.repository.support.SimpleCouchbaseRepository.deleteAll(SimpleCouchbaseRepository.java:168)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:416)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:401)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:373)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$DefaultMethodInvokingMethodInterceptor.invoke(RepositoryFactorySupport.java:486)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.couchbase.repository.support.ViewPostProcessor$ViewInterceptor.invoke(ViewPostProcessor.java:87)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy50.deleteAll(Unknown Source)
at com.pubtech.cms.persistence.RepositoryService.doWork(RepositoryService.java:47)
at com.pubtech.cms.persistence.ApplicationRepository.lambda$commandLineRunner$0(ApplicationRepository.java:83)
at com.pubtech.cms.persistence.ApplicationRepository$$Lambda$9/594916129.run(Unknown Source)
at org.springframework.boot.SpringApplication.runCommandLineRunners(SpringApplication.java:672)
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:690)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
at org.springframework.boot.test.SpringApplicationContextLoader.loadContext(SpringApplicationContextLoader.java:101)
at org.springframework.test.context.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:68)
at org.springframework.test.context.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:86)
at org.springframework.test.context.DefaultTestContext.getApplicationContext(DefaultTestContext.java:72)
at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:170)
at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:110)
at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:212)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:200)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:259)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:261)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:219)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:83)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:68)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:163)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2015-06-09 16:41:40 INFO GenericWebApplicationContext:862 - Closing org.springframework.web.context.support.GenericWebApplicationContext#6302bbb1: startup date [Tue Jun 09 16:41:33 BST 2015]; root of context hierarchy
2015-06-09 16:41:40 INFO CouchbaseConnection:87 - Shut down Couchbase client
2015-06-09 16:41:40 INFO ViewConnection:87 - I/O reactor terminated
when using a bucket name that requires a password (bucket "t1", password "pswd") I see this authentication error in the logs, is there some format. other than plain text that the passsord should be encoded with ?
2015-06-10 10:55:58 INFO DefaultListableBeanFactory:822 - Overriding bean definition for bean 'beanNameViewResolver': replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter.class]]
2015-06-10 10:55:59 INFO Version:27 - HV000001: Hibernate Validator 5.1.3.Final
2015-06-10 10:56:00 ERROR SASLStepOperationImpl:93 - Error: Auth failure
2015-06-10 10:56:00 WARN BinaryMemcachedNodeImpl:90 - Discarding partially completed op: SASL steps operation
2015-06-10 10:56:00 WARN AuthThread:90 - Authentication failed to localhost/127.0.0.1:11210, Status: {OperationStatus success=false: cancelled}
2015-06-10 10:56:02 WARN AuthThread:90 - Authentication failed to localhost/127.0.0.1:11210, Status: {OperationStatus success=false: Invalid arguments}
2015-06-10 10:56:02 WARN AuthThread:90 - Authentication failed to localhost/127.0.0.1:11210, Status: {OperationStatus success=false: Invalid arguments}
I use the couchbase-cli to create the buckets from a script, using the same script to create the working "default" and not working "test_bucket", (properties are correctly injected using mvn filter) :
# Create Bucket
couchbase-cli bucket-create -c $COUCHBASE_HOST:$COUCHBASE_PORT -u $CB_REST_USERNAME -p $CB_REST_PASSWORD \
--bucket=$BUCKET_NAME \
--bucket-type=couchbase \
--bucket-ramsize=200 \
--bucket-replica=1 \
--wait
CouchbaseConfig class:
..
#Configuration
#EnableCouchbaseRepositories(basePackages = {"com.persistence.db"})
#EnableAutoConfiguration
public class CouchbaseConfig extends AbstractCouchbaseConfiguration {
#Value("${couchbase.bucket:boris}")
private String bucketName;
#Value("${couchbase.bucket.password:nopwd}")
private String password;
#Value("${couchbase.host:127.0.0.1}")
private String ip;
..
I think you got a similar issue to what i was experiencing, the issue for me was using #Value in an #Configuration class has a slight special requirement. I was using YAML for my properties file if that matters at all.
add this to your class (must be static as well)
/**
* this is required for some reason: https://jira.spring.io/browse/SPR-11773
*
* #return
*/
#Bean
public static PropertySourcesPlaceholderConfigurer propertyPlaceholderConfigurer() {
return new PropertySourcesPlaceholderConfigurer();

nutch2.3+mysql+solr ClassNotFoundException

I wrote mysql config to gora.properties and update sql lib in ivy.xml.
but when I run the command:
bin/crawl urls/seed.txt Test http://127.0.0.1:8080/solr 2
it throws exception:
[root#li257-46 local]# bin/crawl urls/seed.txt Test http://127.0.0.1:8080/solr 2
Injecting seed URLs
/home/work/nutch2/runtime/local/bin/nutch inject urls/seed.txt -crawlId Test
InjectorJob: starting at 2015-05-13 03:25:35
InjectorJob: Injecting urlDir: urls/seed.txt
**
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/ipc/ByteBufferOutputStream
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:259)
at org.apache.nutch.storage.StorageUtils.getDataStoreClass(StorageUtils.java:93)
at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:77)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.ipc.ByteBufferOutputStream
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 more
Error running:
/home/work/nutch2/runtime/local/bin/nutch inject urls/seed.txt -crawlId Test
Failed with exit value 1.**
First, I think the avro jar not add, but I found the jar in lib:
/home/work/nutch2/runtime/local/lib
[root#li257-46 lib]# ls
avro-1.7.6.jar
avro-compiler-1.7.6.jar
avro-ipc-1.7.6.jar
avro-mapred-1.7.6.jar
Can you help me?

Slick 2.1 Code generator for MySQL : ClassNotFoundException on driver

I am trying to generate the Table classes from an existing SQL tables in MySQL.
I am using Slick 2.1 (slick_2.11-2.1.0-M2), mysql-connector-java-5.1.31-bin
I have created a simple Scala file:
object MySQLPlayground {
def main(args: Array[String]) {
scala.slick.model.codegen.SourceCodeGenerator.main(
Array("scala.slick.driver.MySQLDriver", "com.mysql.jdbc.Driver",
"jdbc:mysql://localhost/fannuaire", "src/main/scala", "modelGene", "user", "password")
)
}
}
But it seems the driver path is wrong. I have the following exception:
Exception in thread "main" java.lang.NoClassDefFoundError: scala/reflect/runtime/package$
at scala.slick.model.codegen.SourceCodeGenerator$.main(SourceCodeGenerator.scala:60)
at com.scala.mysql.MySQLPlayground$.main(MySQLPlayground.scala:10)
at com.scala.mysql.MySQLPlayground.main(MySQLPlayground.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.ClassNotFoundException: scala.reflect.runtime.package$
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 8 more
Since the doc says :
Slick’s code generator comes with a default runner that can be used
from the command line or from Java/Scala.
I was expecting it to work out of the box.
Is there any other set up I have to do? Is the driver's path correct?
Thanks
Please add scala-reflect to your code generator project dependencies.
libraryDependencies += "org.scala-lang" % "scala-reflect" % scalaVersion.value