How to change from derby to MySQL in Spark - mysql

I have installed a spark 2.4.7 using "spark-2.4.7-bin-hadoop2.6.tgz"
I could see the metastore_db got created automatically under the usr home directory and it contains derby database. Also when I developer runs a query it throws error like below
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1#6cde460, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
we are not planning to use derby database, I have MySQL database ready on a different host, can some one please help me understand how can I change the config to point my spark to MySQL instead of the derby??
Thanks in advance
Suresh

Related

install4j throw java.lang.NoClassDefFoundError: com.install4j.runtime.beans.screens.SystemFormScreen error

I try to install JRuby - which uses install4j installer - on Windows 10 64 bit OS. Java runtime version: 1.8.0_171-b11.
I get following error message in log file during installation process and install is terminated.
java.lang.RuntimeException: java.lang.NoClassDefFoundError: com.install4j.runtime.beans.screens.SystemFormScreen
at com.install4j.runtime.installer.frontend.GUIHelper.invokeOnEDT(GUIHelper.java:711)
at com.install4j.runtime.installer.config.AbstractBeanConfig.finishBatch(AbstractBeanConfig.java:51)
at com.install4j.runtime.installer.ContextImpl.registerScreens(ContextImpl.java:694)
at com.install4j.runtime.installer.controller.Controller.start(Controller.java:75)
at com.install4j.runtime.installer.Installer.runInProcess(Installer.java:58)
at com.install4j.runtime.installer.Installer.main(Installer.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:85)
at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:94)
at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:25)
Caused by: java.lang.NoClassDefFoundError: com.install4j.runtime.beans.screens.SystemFormScreen
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.privateGetPublicMethods(Unknown Source)
at java.lang.Class.getMethods(Unknown Source)
at com.sun.beans.finder.MethodFinder$1.create(Unknown Source)
at com.sun.beans.finder.MethodFinder$1.create(Unknown Source)
at com.sun.beans.util.Cache.get(Unknown Source)
at com.sun.beans.finder.MethodFinder.findMethod(Unknown Source)
at java.beans.Statement.getMethod(Unknown Source)
at java.beans.Statement.invokeInternal(Unknown Source)
at java.beans.Statement.access$000(Unknown Source)
at java.beans.Statement$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.beans.Statement.invoke(Unknown Source)
at java.beans.Expression.getValue(Unknown Source)
at com.sun.beans.decoder.ObjectElementHandler.getValueObject(Unknown Source)
In your case JVM uses options which cannot be used by Ruby/SoapUI correctly.
JAVA_OPTIONS
IBM_JAVA_OPTIONS
JAVA_TOOL_OPTIONS
To work around the issue, you can do the following:
Use Task Manager to completely close the HP UFT and/or Oracle Single
Sign-On software or service.
Remove the JAVA_TOOL_OPTIONS, IBM_JAVA_OPTIONS and _JAVA_OPTIONS OS environment variables.
Also before removing the "JAVA_TOOL_OPTIONS, IBM_JAVA_OPTIONS and _JAVA_OPTIONS" kindly save the information in Notepad. Once after installing, again add"JAVA_TOOL_OPTIONS, IBM_JAVA_OPTIONS and _JAVA_OPTIONS" to your Environmental variables else HPUFT or Oracle SSO will have issue while launching.
else create a bat file and call your installer in the below content and execute the bat file [Jrubyinstall.bat (any prefered filename)] place the batfile in same folder as your installer is placed.
set IBM_JAVA_OPTIONS=
set _JAVA_OPTIONS=
set JAVA_TOOL_OPTIONS=
jruby_windows_9_2_8_0.exe
[Optional --- only if you required to launch it
The above solution is to Install the application,
But to execute it, create a bat file in place of installation [mostly in bin folder]
inside the Bat file:
set IBM_JAVA_OPTIONS=
set _JAVA_OPTIONS=
set JAVA_TOOL_OPTIONS=
[your executable file] xxx.exe/jar/...
and save it. every time you can launch the application via BAT file. ]

Error connecting to database: (using class org.gjt.mm.mysql.Driver) The statement (1) has no open cursor

I am working on Pentaho Data Integration tool version 5.2. I am trying to connect to data base using 'Table Input' step. but when I test the connection in this step it is giving me the below error message. I am able to connect to DB using the MySQL workbench. I have placed the 'mysql-connector-java-5.1.38-bin' file in the libext forlder.
do I need to downgrade my connector jar file? if yes my SQL version is 5.6 so can some one let me know the compatible connector jar version for this SQL Server version?
Error connecting to database [myLedger] : org.pentaho.di.core.exception.KettleDatabaseException:
Error occured while trying to connect to the database
Error connecting to database: (using class org.gjt.mm.mysql.Driver)
The statement (1) has no open cursor.
org.pentaho.di.core.exception.KettleDatabaseException:
Error occured while trying to connect to the database
Error connecting to database: (using class org.gjt.mm.mysql.Driver)
The statement (1) has no open cursor.
at org.pentaho.di.core.database.Database.normalConnect(Database.java:427)
at org.pentaho.di.core.database.Database.connect(Database.java:361)
at org.pentaho.di.core.database.Database.connect(Database.java:314)
at org.pentaho.di.core.database.Database.connect(Database.java:302)
at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:80)
at org.pentaho.di.core.database.DatabaseMeta.testConnection(DatabaseMeta.java:2685)
at org.pentaho.ui.database.event.DataHandler.testDatabaseConnection(DataHandler.java:546)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43)
at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:138)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)
at org.eclipse.jface.window.Window.open(Window.java:796)
at org.pentaho.ui.xul.swt.tags.SwtDialog.show(SwtDialog.java:389)
at org.pentaho.ui.xul.swt.tags.SwtDialog.show(SwtDialog.java:318)
at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:116)
at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:59)
at org.pentaho.di.ui.trans.step.BaseStepDialog$4.widgetSelected(BaseStepDialog.java:740)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.trans.steps.tableinput.TableInputDialog.open(TableInputDialog.java:435)
at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:124)
at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8720)
at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3027)
at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:744)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1310)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7931)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9202)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Error connecting to database: (using class org.gjt.mm.mysql.Driver)
The statement (1) has no open cursor.
at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:572)
at org.pentaho.di.core.database.Database.normalConnect(Database.java:410)
... 51 more
Caused by: java.sql.SQLException: The statement (1) has no open cursor.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2822)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:777)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1260)
at com.mysql.jdbc.MysqlIO.fetchRowsViaCursor(MysqlIO.java:3838)
at com.mysql.jdbc.CursorRowProvider.fetchMoreRows(CursorRowProvider.java:383)
at com.mysql.jdbc.CursorRowProvider.hasNext(CursorRowProvider.java:303)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:6459)
at com.mysql.jdbc.Connection.loadServerVariables(Connection.java:4059)
at com.mysql.jdbc.Connection.initializePropsFromServer(Connection.java:3778)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2608)
at com.mysql.jdbc.Connection.<init>(Connection.java:1509)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:554)
... 52 more
I have fixed the issue. the useCursorFetch was set to true which I have changed to false in the Options menu in the left side panel of Data Base connection.

Configuring embedded Derby in Sqoop2

I want to set up Sqoop2. But there is some problem in configuring it. When I start the server by sqoop2-server start it does not give any errors on console but it fails to start and it shows errors in the log.
Console output:
Sqoop home directory: /opt/sqoop-1.99.6
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE: /opt/sqoop-1.99.6/server
Using CATALINA_HOME: /opt/sqoop-1.99.6/server
Using CATALINA_TMPDIR: /opt/sqoop-1.99.6/server/temp
Using JRE_HOME: /opt/jdk1.8.0_51/jre
Using CLASSPATH: /opt/sqoop-1.99.6/server/bin/bootstrap.jar
The main error seems to be:
ERROR XSLAN: Database at /opt/sqoop-1.99.6/repository/db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.10
Here is the complete log:
2015-08-18 12:00:44,932 INFO authentication.SimpleAuthenticationHandler [org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:40)] Using simple/pseudo authentication, principal [root]
2015-08-18 12:00:44,932 INFO security.AuthenticationManager [org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:102)] Authentication loaded.
2015-08-18 12:00:44,937 INFO security.AuthorizationManager [org.apache.sqoop.security.AuthorizationManager.initialize(AuthorizationManager.java:117)] Authorization loaded.
2015-08-18 12:00:44,938 INFO audit.AuditLoggerManager [org.apache.sqoop.audit.AuditLoggerManager.initialize(AuditLoggerManager.java:80)] Begin audit logger manager initialization
2015-08-18 12:00:44,939 WARN audit.FileAuditLogger [org.apache.sqoop.audit.FileAuditLogger.initialize(FileAuditLogger.java:68)] appender: log4j.appender.defaultAppender
2015-08-18 12:00:44,940 INFO audit.AuditLoggerManager [org.apache.sqoop.audit.AuditLoggerManager.initializeLoggers(AuditLoggerManager.java:127)] Audit Logger has been initialized: default
2015-08-18 12:00:44,941 INFO audit.AuditLoggerManager [org.apache.sqoop.audit.AuditLoggerManager.initialize(AuditLoggerManager.java:86)] Audit logger manager initialized: OK
2015-08-18 12:00:44,942 INFO repository.RepositoryManager [org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:86)] Setting system properties: {derby.stream.error.file=/var/log/sqoop/derbyrepo.log}
2015-08-18 12:00:44,948 INFO repository.JdbcRepositoryContext [org.apache.sqoop.repository.JdbcRepositoryContext.<init>(JdbcRepositoryContext.java:154)] [repo-ctx] handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler, conn-url=jdbc:derby:/opt/sqoop-1.99.6/repository/db;create=true, driver=org.apache.derby.jdbc.EmbeddedDriver, user=sa, password=*****, jdbc-props={}, tx-isolation=READ_COMMITTED, max-conn=10
2015-08-18 12:00:45,260 INFO derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.initialize(DerbyRepositoryHandler.java:96)] DerbyRepositoryHandler initialized.
2015-08-18 12:00:45,263 INFO repository.JdbcRepositoryProvider [org.apache.sqoop.repository.JdbcRepositoryProvider.initializeRepositoryHandler(JdbcRepositoryProvider.java:160)] JdbcRepositoryProvider initialized
2015-08-18 12:00:45,263 INFO repository.JdbcRepositoryProvider [org.apache.sqoop.repository.JdbcRepositoryProvider.initialize(JdbcRepositoryProvider.java:67)] JdbcRepository initialized.
2015-08-18 12:00:45,263 INFO repository.RepositoryManager [org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:122)] Creating or updating respository at bootup
2015-08-18 12:00:45,341 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: -1, rollback: false
2015-08-18 12:00:45,343 ERROR core.SqoopServer [org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:62)] Server startup failure
org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:87)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63)
at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:129)
at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:77)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:56)
at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.sql.SQLException: Failed to start database '/opt/sqoop-1.99.6/repository/db' with class loader WebappClassLoader
context: /sqoop
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader#4783da3f
, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:61)
... 31 more
Caused by: java.sql.SQLException: Failed to start database '/opt/sqoop-1.99.6/repository/db' with class loader WebappClassLoader
context: /sqoop
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader#4783da3f
, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 48 more
Caused by: java.sql.SQLException: Database at /opt/sqoop-1.99.6/repository/db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.10.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 45 more
Caused by: ERROR XSLAN: Database at /opt/sqoop-1.99.6/repository/db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.10.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
... 45 more
2015-08-18 12:00:45,346 ERROR server.ServerInitializer [org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:46)] Sqoop server failed to start
java.lang.RuntimeException: Failure in server initialization
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:63)
at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:87)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63)
at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:129)
at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:77)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:56)
... 25 more
Caused by: java.sql.SQLException: Failed to start database '/opt/sqoop-1.99.6/repository/db' with class loader WebappClassLoader
context: /sqoop
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader#4783da3f
, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:61)
... 31 more
Caused by: java.sql.SQLException: Failed to start database '/opt/sqoop-1.99.6/repository/db' with class loader WebappClassLoader
context: /sqoop
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader#4783da3f
, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 48 more
Caused by: java.sql.SQLException: Database at /opt/sqoop-1.99.6/repository/db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.10.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 45 more
Caused by: ERROR XSLAN: Database at /opt/sqoop-1.99.6/repository/db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.10.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
... 45 more
sqoop.properties:
# Logging Configuration
# Any property that starts with the prefix
# org.apache.sqoop.log4j is parsed out by the configuration
# system and passed to the log4j subsystem. This allows you
# to specify log4j configuration properties from within the
# Sqoop configuration.
#
org.apache.sqoop.log4j.appender.file=org.apache.log4j.RollingFileAppender
org.apache.sqoop.log4j.appender.file.File=/var/log/sqoop/sqoop.log
org.apache.sqoop.log4j.appender.file.MaxFileSize=25MB
org.apache.sqoop.log4j.appender.file.MaxBackupIndex=5
org.apache.sqoop.log4j.appender.file.layout=org.apache.log4j.PatternLayout
org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} [%l] %m%n
org.apache.sqoop.log4j.debug=true
org.apache.sqoop.log4j.rootCategory=WARN, file
org.apache.sqoop.log4j.category.org.apache.sqoop=DEBUG
org.apache.sqoop.log4j.category.org.apache.derby=INFO
#
# Audit Loggers Configuration
# Multiple audit loggers could be given here. To specify an
# audit logger, you should at least add org.apache.sqoop.
# auditlogger.[LoggerName].class. You could also provide
# more configuration options by using org.apache.sqoop.
# auditlogger.[LoggerName] prefix, then all these options
# are parsed to the logger class.
#
org.apache.sqoop.auditlogger.default.class=org.apache.sqoop.audit.FileAuditLogger
org.apache.sqoop.auditlogger.default.file=/var/log/sqoop/default.audit
#
# Repository configuration
# The Repository subsystem provides the special prefix which
# is "org.apache.sqoop.repository.sysprop". Any property that
# is specified with this prefix is parsed out and set as a
# system property. For example, if the built in Derby repository
# is being used, the sysprop prefixed properties can be used
# to affect Derby configuration at startup time by setting
# the appropriate system properties.
#
# Repository provider
org.apache.sqoop.repository.provider=org.apache.sqoop.repository.JdbcRepositoryProvider
# Repository upgrade
# If set to true, it will not upgrade the sqoop respository schema, by default it will iniate the upgrade on server start-up
org.apache.sqoop.repository.schema.immutable=false
# JDBC repository provider configuration
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:/opt/sqoop-1.99.6/repository/db;create=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=
# System properties for embedded Derby configuration
org.apache.sqoop.repository.sysprop.derby.stream.error.file=/var/log/sqoop/derbyrepo.log
#
# Sqoop Connector configuration
# If set to true will initiate Connectors config upgrade during server startup
#
org.apache.sqoop.connector.autoupgrade=false
#
# Sqoop Driver configuration
# If set to true will initiate the Driver config upgrade during server startup
#
org.apache.sqoop.driver.autoupgrade=false
# Sleeping period for reloading configuration file (once a minute)
org.apache.sqoop.core.configuration.provider.properties.sleep=60000
#
# Submission engine configuration
#
# Submission engine class
org.apache.sqoop.submission.engine=org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine
# Number of milliseconds, submissions created before this limit will be removed, default is one day
#org.apache.sqoop.submission.purge.threshold=
# Number of milliseconds for purge thread to sleep, by default one day
#org.apache.sqoop.submission.purge.sleep=
# Number of milliseconds for update thread to sleep, by default 5 minutes
#org.apache.sqoop.submission.update.sleep=
#
# Configuration for Mapreduce submission engine (applicable if it's configured)
#
# Hadoop configuration directory
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop-2.7.1/etc/hadoop/
#
# Execution engine configuration
#
org.apache.sqoop.execution.engine=org.apache.sqoop.execution.mapreduce.MapreduceExecutionEngine
#
# Authentication configuration
#
#org.apache.sqoop.security.authentication.type=SIMPLE
#org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
#org.apache.sqoop.security.authentication.anonymous=true
#org.apache.sqoop.security.authentication.type=KERBEROS
#org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.KerberosAuthenticationHandler
#org.apache.sqoop.security.authentication.kerberos.principal=sqoop/_HOST#NOVALOCAL
#org.apache.sqoop.security.authentication.kerberos.keytab=/home/kerberos/sqoop.keytab
#org.apache.sqoop.security.authentication.kerberos.http.principal=HTTP/_HOST#NOVALOCAL
#org.apache.sqoop.security.authentication.kerberos.http.keytab=/home/kerberos/sqoop.keytab
#org.apache.sqoop.security.authentication.enable.doAs=true
#org.apache.sqoop.security.authentication.proxyuser.#USER#.users=*
#org.apache.sqoop.security.authentication.proxyuser.#USER#.groups=*
#org.apache.sqoop.security.authentication.proxyuser.#USER#.hosts=*
#
# Authorization configuration
#
#org.apache.sqoop.security.authorization.handler=org.apache.sqoop.security.authorization.DefaultAuthorizationHandler
#org.apache.sqoop.security.authorization.access_controller=org.apache.sqoop.security.authorization.DefaultAuthorizationAccessController
#org.apache.sqoop.security.authorization.validator=org.apache.sqoop.security.authorization.DefaultAuthorizationValidator
#org.apache.sqoop.security.authorization.authentication_provider=org.apache.sqoop.security.authorization.DefaultAuthenticationProvider
#org.apache.sqoop.security.authorization.server_name=SqoopServer1
# External connectors load path
# "/path/to/external/connectors/": Add all the connector JARs in the specified folder
#
org.apache.sqoop.connector.external.loadpath=
I think the problem has some thing to do with embedded derby. Can someone help me find where the problem is?
I deleted the #BASEDIR#/repository/db which in my case is /opt/sqoop-1.99.6/repository/db and again started the server. I am not sure if this is a good method but this worked for me.

Cant connect to MySQL via Squirrel - Unable to create connection. Check your URL

I can't connect to our database using Squirrel. I am certain my URL is correct as I'm using it via my java code to access the database.
I got the drivers from here: http://dev.mysql.com/downloads/file.php?id=454395
What is interesting is when I add the driver to MySQL in squirrel it shows it as:
com.mysql.fabric.jdbc.FabricMySQLDriver
In the past when I've used squirrel it would just show com.mysql.jdbc.Driver and no "fabric" stuff.
Stack trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.sql.SQLException: Unable to create connection. Check your URL.
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.awaitConnection(OpenConnectionCommand.java:132)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.access$100(OpenConnectionCommand.java:45)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand$2.run(OpenConnectionCommand.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Unable to create connection. Check your URL.
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.executeConnect(OpenConnectionCommand.java:171)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.access$000(OpenConnectionCommand.java:45)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand$1.run(OpenConnectionCommand.java:104)
... 5 more
Caused by: java.sql.SQLException: Unable to create connection. Check your URL.
at net.sourceforge.squirrel_sql.fw.sql.SQLDriverManager.getConnection(SQLDriverManager.java:136)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.executeConnect(OpenConnectionCommand.java:167)
... 7 more
Double-click on the MySQL driver in the Drivers window. Take a look at the class name dropdown at the bottom of the newly opened window. Click on the dropdown arrow and see if you can select the original com.mysql.jdbc.Driver class name.

How can I get Hudson to update through Proxy

We run a corporate proxy with authentication and we're running Hudson as our CI platform. Hudson can get to the outside world to tell which plugins need updating, but when we tell Hudson to download the updates to the plugins, we get the following error.
java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$6.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:603)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:812)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )"
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(Unknown Source)
at java.net.URLConnection.getHeaderFieldInt(Unknown Source)
at java.net.URLConnection.getContentLength(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLength(Unknown Source)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:602)
... 7 more
Does anyone know how I can configure Hudson or our JRE or whatever needs configuring, to allow Hudson to update via the GUI? Sure we can download updates manually but that gets annoying after a while.
Currently we have Hudson installed as a service on windows server 2008 and its using the built in server that came with Hudson to host it. Also, I've set the proxy configuration via the "Advanced" tab
Two things to do
1. Pop over into the plugin manager and select advanced, enter the settings in there
2. Set the properties of the JRE http://java.sun.com/javase/6/docs/technotes/guides/net/properties.html
You can set properties using the -Dproperty.name=some.value syntax when starting up hudson.