I'm using a Liquibase changeset to update null values of the customId column. The sql is a modified version of this answer.
The problem is that it seems to use the same UUID for all rows. My changeset looks like this:
<changeSet id="v3.0.3" author="msp" runAlways="false">
<preConditions onFail="MARK_RAN">
<not>
<sqlCheck expectedResult="0">select count(*) from FOOBAR where customId is null</sqlCheck>
</not>
</preConditions>
<sql>update FOOBAR set customId = concat('cu:', (select uuid())) where customId is null</sql>
</changeSet>
Note that if I'm running this slightly modified version of the query (adding a limit of 2) in the MySQL console, then it works:
update FOOBAR set customId = concat('cu:', (select uuid())) where customId is null limit 2
But if I try to run the very same query (including the limit) with liquibase, I get the following Exception:
Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set changelog/db.changelog-v3.0.3.xml::v3.0.3::msp:
Reason: liquibase.exception.DatabaseException: Error executing SQL update FOOBAR set customId = concat('cu:', (select uuid())) where customId is null limit 2: Duplicate entry 'cu:cde325da-f71f-11e5-95f8-d15a50829a7d' for key 'customId'
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:554)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:43)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:70)
at liquibase.Liquibase.update(Liquibase.java:195)
at liquibase.Liquibase.update(Liquibase.java:174)
at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:330)
at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:287)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1625)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1562)
... 77 more
Caused by: liquibase.exception.DatabaseException: Error executing SQL update FOOBAR set customId = concat('cu:', (select uuid())) where customId is null limit 2: Duplicate entry 'cu:cde325da-f71f-11e5-95f8-d15a50829a7d' for key 'customId'
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:62)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:122)
at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1206)
at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1189)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:518)
... 85 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'cu:cde325da-f71f-11e5-95f8-d15a50829a7d' for key 'customId'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1039)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4096)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4028)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2627)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:841)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:681)
at com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:1006)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:310)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:55)
... 89 more
Note that there's of course a UNIQUE constraint on customId - but shouldn't select uuid() (also tried uuid() without select) generate a unique value, even when using Liquibase?
The problem can be solved with a small workaround. I split the update into two queries:
<changeSet id="v3.0.3" author="msp" runAlways="false">
<preConditions onFail="MARK_RAN">
<not>
<sqlCheck expectedResult="0">select count(*) from FOOBAR where customId is null</sqlCheck>
</not>
</preConditions>
<sql>update FOOBAR set customId = (select uuid())) where customId is null</sql>
<sql>update FOOBAR set customId = concat('cu:', customId) where length(customId) = 36
</changeSet>
This of course only works for my case where I can be sure that all rows of the first query can be identified by length(customId) = 36.
Related
I use Hive 3.1.0.3.1.0.0-78.
Create source table with statement:
CREATE TABLE IF NOT EXISTS core.source_jira
(
pk_jira BIGINT DEFAULT SURROGATE_KEY(),
test_1 STRING,
test_2 STRING,
valid BOOLEAN,
PRIMARY KEY (pk_jira) DISABLE NOVALIDATE
)
STORED AS ORC
LOCATION '/warehouse/tablespace/managed/hive/core.db/source_jira'
TBLPROPERTIES (
"orc.compress" = "SNAPPY",
"transactional" = "true",
"NO_AUTO_COMPACTION"="TRUE"
)
;
INSERT INTO TABLE core.source_jira(test_1, test_2, valid)
SELECT "text_1", "text_2", TRUE;
Check table data with select:
SELECT * from core.source_jira
pk_jira
test_1
test_2
valid
1099511627776
text_1
text_2
true
Create destination table with statement:
CREATE TABLE IF NOT EXISTS core.destination_jira
(
pk_jira BIGINT DEFAULT SURROGATE_KEY(),
test_1 STRING,
test_2 STRING,
valid BOOLEAN,
PRIMARY KEY (pk_jira) DISABLE NOVALIDATE
)
STORED AS ORC
LOCATION '/warehouse/tablespace/managed/hive/core.db/destination_jira'
TBLPROPERTIES (
"orc.compress" = "SNAPPY",
"transactional" = "true",
"NO_AUTO_COMPACTION"="TRUE"
)
;
Add column:
ALTER TABLE core.destination_jira ADD COLUMNS (jira_type STRING);
Insert data statement:
insert into table core.destination_jira (test_1, test_2, valid) select test_1, test_2, valid from core.source_jira
I got an error:
SQL Error [40000] [42000]: Error while compiling statement: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [40000] [42000]: Error while compiling statement: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:133)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeStatement(SQLQueryJob.java:577)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.lambda$1(SQLQueryJob.java:486)
at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:172)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeSingleQuery(SQLQueryJob.java:493)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.extractData(SQLQueryJob.java:894)
at org.jkiss.dbeaver.ui.editors.sql.SQLEditor$QueryResultsContainer.readData(SQLEditor.java:3645)
at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:123)
at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:172)
at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:121)
at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:4949)
at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:279)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:265)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:303)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244)
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:329)
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.lambda$0(JDBCStatementImpl.java:131)
at org.jkiss.dbeaver.utils.SecurityManagerUtils.wrapDriverActions(SecurityManagerUtils.java:94)
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)
... 12 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy47.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getColNameToDefaultValueMap(SemanticAnalyzer.java:750)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getColForInsertStmtSpec(SemanticAnalyzer.java:4620)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.handleInsertStatementSpec(SemanticAnalyzer.java:4735)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4580)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4326)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:10662)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10601)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11530)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11400)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12202)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:620)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12292)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1816)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1811)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
... 26 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.getEnabledDefaultConstraints(Hive.java:5389)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getColNameToDefaultValueMap(SemanticAnalyzer.java:744)
... 45 more
Caused by: java.lang.NullPointerException: null
I have the following code table A has a check constraint on column Denial.
CREATE TABLE Table a
(
[ID] int IDENTITY(1,1) NOT NULL ,
[EntityID] int ,
Denial nVarchar(20)
CONSTRAINT Chk_Denial CHECK (Denial IN ('Y', 'N')),
)
Merge statement
MERGE INTO Table a WITH (HOLDLOCK) AS tgt
USING (SELECT DISTINCT
JSON_VALUE(DocumentJSON, '$.EntityID') AS EntityID,
JSON_VALUE(DocumentJSON, '$.Denial') AS Denial
FROM Table1 bd
INNER JOIN table2 bf ON bf.FileUID = bd.FileUID
WHERE bf.Type = 'Payment') AS src ON tgt.[ID] = src.[ID]
WHEN MATCHED
)) THEN
UPDATE SET tgt.ID = src.ID,
tgt.EntityID = src.EntityID,
tgt.Denial = src.Denial,
WHEN NOT MATCHED BY TARGET
THEN INSERT (ID, EntityID, Denial)
VALUES (src.ID, src.EntityID, src.Denial)
THEN DELETE
I get this error when running my MERGE statement:
Error Message Msg 547, Level 16, State 0, Procedure storproctest1, Line 40 [Batch Start Line 0]
The MERGE statement conflicted with the CHECK constraint "Chk_Column". The conflict occurred in the database "Test", table "Table1", and column 'Denial'. The statement has been terminated.
This is due to the source files having "Yes" and "No" instead of 'Y' and 'N'. Hence, I'm getting the above error.
How can I use a Case statement in merge statement to handle the above Check constraints error? or Any alternative solutions.
You can turn Yes to Y and No to N before merging your data. That would belong to the using clause of the merge query:
USING (
SELECT Distinct
JSON_VALUE(DocumentJSON, '$.EntityID') AS EntityID,
CASE JSON_VALUE(DocumentJSON, '$.Denial')
WHEN 'Yes' THEN 'Y'
WHEN 'No' THEN 'N'
ELSE JSON_VALUE(DocumentJSON, '$.Denial')
END AS Denial
FROM Table1 bd
INNER JOIN table2 bf ON bf.FileUID = bd.FileUID
WHERE bf.Type = 'Payment'
) AS src
The case expression translates Y and N values, and leaves other values untouched. Since this applies to the source dataset, the whole rest of the query benefits (ie both the update and insert branches).
I want to put a constraint on a table to limit the input of values in the table.
I want to make sure that any combination of values can only be in once.
I.e (1,2) and (2,1) can not be in the same table.
For example I have a table with two columns (c1 and c2):
The following has to be followed.
C1 C2
1 2 OK
2 1 NOT OK
3 1 OK
1 2 NOT OK
1 4 OK
1 3 NOT OK
Is there any way to do this in SQLALchemy?
I used UNIQUE(c1, c2) but that only says (1,2) and (1,2) can't be in the same table but as mentioned I also want to include that (2,1) can't be in the table either.
Thanks
Probably the easiest solution is to add a check constraint for c1 < c2 (or c1 <= c2 if they're allowed to be the same) so that (c1, c2) will always be in "ascending order":
import sqlalchemy as sa
connection_uri = (
"mssql+pyodbc://#localhost:49242/myDb?driver=ODBC+Driver+17+for+SQL+Server"
)
engine = sa.create_engine(connection_uri)
Base = declarative_base()
class So64232358(Base):
__tablename__ = "so64232358"
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
c1 = sa.Column(sa.Integer, nullable=False)
c2 = sa.Column(sa.Integer, nullable=False)
comment = sa.Column(sa.String(50))
sa.CheckConstraint(c1 < c2)
sa.UniqueConstraint(c1, c2)
Base.metadata.drop_all(engine, checkfirst=True)
Base.metadata.create_all(engine)
"""SQL rendered:
CREATE TABLE so64232358 (
id INTEGER NOT NULL IDENTITY,
c1 INTEGER NOT NULL,
c2 INTEGER NOT NULL,
comment VARCHAR(50) NULL,
PRIMARY KEY (id),
CHECK (c1 < c2),
UNIQUE (c1, c2)
)
"""
Session = sessionmaker(bind=engine)
session = Session()
obj = So64232358(c1=2, c2=1, comment="no es bueno")
session.add(obj)
try:
session.commit()
except sa.exc.IntegrityError as ie:
print(ie)
"""console output:
(pyodbc.IntegrityError) ('23000', '[23000] [Microsoft]
[ODBC Driver 17 for SQL Server][SQL Server]The INSERT statement conflicted
with the CHECK constraint "CK__so64232358__429B0397". The conflict
occurred in database "myDb", table "dbo.so64232358".
(547) (SQLExecDirectW);
[23000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
The statement has been terminated. (3621)')
"""
session.rollback()
obj = So64232358(c1=1, c2=2, comment="bueno")
session.add(obj)
session.commit() # no error
obj = So64232358(c1=1, c2=2, comment="duplicado")
session.add(obj)
try:
session.commit()
except sa.exc.IntegrityError as ie:
print(ie)
"""console output:
(pyodbc.IntegrityError) ('23000', "[23000] [Microsoft]
[ODBC Driver 17 for SQL Server][SQL Server]Violation of UNIQUE KEY
constraint 'UQ__so642323__E13250592117193A'. Cannot insert duplicate key
in object 'dbo.so64232358'. The duplicate key value is (1, 2).
(2627)(SQLExecDirectW);
[23000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
The statement has been terminated. (3621)")
"""
session.rollback()
I get a deadlock exception, but when I want to see some info about that from "SHOW ENGINE INNODB STATUS", it contains deadlock information from longer time ago (long before the exception), so it looks like it is just "not catched" by that command.
From the stacktrace I can see, that the last my application code executed was the $Proxy174.attemptToAcquireJobExecution , which is realization of the spring data interface:
#Transactional(TxType.REQUIRES_NEW)
#Modifying
#Query("UPDATE Job SET last_run = NOW(),
last_run_by=:runner
WHERE name = :name
AND (last_run + :interval ) < NOW()")
int attemptToAcquireJobExecution(
#Param("name") String var1,
#Param("interval") int var2,
#Param("runner") String var3);
It operates on such a simple table:
CREATE TABLE `jobs` (
`name` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL,
`last_run_by` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_run` datetime(3) DEFAULT NULL,
`context_data` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Scheduled tasks, that are executing that command runs every few minutes (on 3 app instances), so there is really minimal risk of collisions, yet deadlock occurs very often... There is no #Transactional anywhere before that operation, so there should be no hanging transaction really. Anyway, pinging those commands yields empty results:
SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCKS ;
SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCK_WAITS;
Original stacktrace, my classes marked as "xxx", at the bottom:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:951)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2497)
at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1566)
... 2 frames excluded
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)
at com.sun.proxy.$Proxy121.commit(Unknown Source)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:80)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:232)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:65)
at org.hibernate.jpa.internal.TransactionImpl.commit(TransactionImpl.java:61)
... 49 common frames omitted
Wrapped by: javax.persistence.PersistenceException: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at org.hibernate.jpa.internal.TransactionImpl.commit(TransactionImpl.java:75)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:517)
... 48 common frames omitted
Wrapped by: org.springframework.orm.jpa.JpaSystemException: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction; nested exception is javax.persistence.PersistenceException: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:418)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:246)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:521)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:504)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:292)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:133)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy174.attemptToAcquireJobExecution(Unknown Source)
at xxx.lib.scheduler.JobsService.checkIfJobShouldBeExecuted(JobsService.java:32)
at xxx.lib.scheduler.StandardBatchStarter.startScheduled(StandardBatchStarter.java:38)
at xxx.scheduled.ImportProductDescriptionJob.startScheduled(ImportProductDescriptionJob.java:46)
at xxx.scheduled.ImportProductDescriptionJob$$FastClassBySpringCGLIB$$494978af.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at org.springframework.cloud.sleuth.instrument.scheduling.TraceSchedulingAspect.traceBackgroundThread(TraceSchedulingAspect.java:72)
... 2 frames excluded
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
... 21 frames truncated
AND (last_run + :interval ) < NOW()
Probably leads to a syntax error -- or a data error.
Please provide the generated SQL. I suspect it is generating
AND (last_run + 2) < NOW()
But it needs to be something like
AND (last_run + INTERVAL 2 DAY) < NOW()
Also, please provide SHOW CREATE Job.
I am trying to use LOAD HADOOP command to copy data from teradata source.
LOAD HADOOP USING JDBC CONNECTION URL 'jdbc:teradata://<<ip_address>>/database=EDW01_V_LV_BASE' WITH PARAMETERS ('user' = 'USR1','password'='pass1') FROM SQL QUERY 'SELECT COL1, COL2, COL3, COL4, COL5, COL6, COl7, COl8, COl9, COL10 FROM POS_RTL_STR WHERE TBL_TMST > '2015-12-15 03:49:33' AND TBL_TMST <= '2015-12-15 14:54:05' AND $CONDITIONS' SPLIT COLUMN COL1 INTO TABLE SCHEMA1.TABLE1 APPEND;.
Now load is failing with the error
2015-12-17 04:02:54,219 ERROR com.ibm.biginsights.ie.db.DBImportImpl [Thread-3] : Error during import
java.lang.RuntimeException: java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 5628] [SQLState HY000] Column 2015-12-15 03:49:33 not found in EDW01_V_LV_BASE.POS_RTL_STR.
at com.ibm.biginsights.ie.sqoop.td.TeradataFastManager.getColumnTypesForRawQuery(TeradataFastManager.java:181)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:234)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:304)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.manager.BIImportTool.importTable(BIImportTool.java:97)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at com.ibm.biginsights.ie.db.SqoopUtils.runSqoopTool(SqoopUtils.java:146)
at com.ibm.biginsights.ie.db.DBImportImpl.importData(DBImportImpl.java:159)
at com.ibm.biginsights.ie.impl.ImporterImpl.executeImport(ImporterImpl.java:504)
at com.ibm.biginsights.ie.impl.ImporterImpl.executePerformImport(ImporterImpl.java:417)
at com.ibm.biginsights.ie.impl.ImporterImpl.performImport(ImporterImpl.java:264)
at com.ibm.biginsights.biga.udf.LoadTool.performImport(LoadTool.java:214)
at com.ibm.biginsights.biga.udf.BIGSQL_DDL.doLoadStatement(BIGSQL_DDL.java:644)
at com.ibm.biginsights.biga.udf.BIGSQL_DDL.processDDL(BIGSQL_DDL.java:207)
Caused by: java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 5628] [SQLState HY000] Column 2015-12-15 03:49:33 not found in EDW01_V_LV_BASE.POS_RTL_STR.
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:308)
at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:307)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:196)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:385)
at com.teradata.jdbc.jdbc_4.TDStatement.prepareRequest(TDStatement.java:569)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.<init>(TDPreparedStatement.java:117)
at com.teradata.jdbc.jdk6.JDK6_SQL_PreparedStatement.<init>(JDK6_SQL_PreparedStatement.java:29)
at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.constructPreparedStatement(JDK6_SQL_Connection.java:81)
at com.teradata.jdbc.jdbc_4.TDSession.prepareStatement(TDSession.java:1357)
at com.teradata.jdbc.jdbc_4.TDSession.prepareStatement(TDSession.java:1401)
at com.teradata.jdbc.jdbc.ManagerConnectionBase.prepareStatement(ManagerConnectionBase.java:302)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:752)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:775)
at com.ibm.biginsights.ie.sqoop.td.TeradataFastManager.executeStmt(TeradataFastManager.java:191)
at com.ibm.biginsights.ie.sqoop.td.TeradataFastManager.getColumnTypesForRawQuery(TeradataFastManager.java:177)
... 18 more
2015-12-17 04:02:54,219 ERROR com.ibm.biginsights.ie.db.DBImportImpl [Thread-3] : [BSL-0-1ae1ada8b]: Error during import (Job Id = ):java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 5628] [SQLState HY000] Column 2015-12-15 03:49:33 not found in EDW01_V_LV_BASE.POS_RTL_STR.
I think it is somewhere wrong in the where clause where I have used value within single quote in where clause. How should it be specified? Or there is any other error?
Please visit: https://www.ibm.com/support/knowledgecenter/?lang=en#!/SSPT3X_4.1.0/com.ibm.swg.im.infosphere.biginsights.db2biga.doc/doc/biga_load_from.html?cp=SSPT3X_4.1.0%2F4-4-0
In the section FROM SQL QUERYselect-statement:
Surround the value for QUERY with single quotes. If a single quote is needed inside the value then use two single quotes.
… FROM SQL QUERY 'SELECT ID, NAME,
STATE, CITY FROM CITIES
WHERE CITY=''San Jose'' AND $CONDITIONS'
...