I am trying to use LOAD HADOOP command to copy data from teradata source.
LOAD HADOOP USING JDBC CONNECTION URL 'jdbc:teradata://<<ip_address>>/database=EDW01_V_LV_BASE' WITH PARAMETERS ('user' = 'USR1','password'='pass1') FROM SQL QUERY 'SELECT COL1, COL2, COL3, COL4, COL5, COL6, COl7, COl8, COl9, COL10 FROM POS_RTL_STR WHERE TBL_TMST > '2015-12-15 03:49:33' AND TBL_TMST <= '2015-12-15 14:54:05' AND $CONDITIONS' SPLIT COLUMN COL1 INTO TABLE SCHEMA1.TABLE1 APPEND;.
Now load is failing with the error
2015-12-17 04:02:54,219 ERROR com.ibm.biginsights.ie.db.DBImportImpl [Thread-3] : Error during import
java.lang.RuntimeException: java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 5628] [SQLState HY000] Column 2015-12-15 03:49:33 not found in EDW01_V_LV_BASE.POS_RTL_STR.
at com.ibm.biginsights.ie.sqoop.td.TeradataFastManager.getColumnTypesForRawQuery(TeradataFastManager.java:181)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:234)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:304)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.manager.BIImportTool.importTable(BIImportTool.java:97)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at com.ibm.biginsights.ie.db.SqoopUtils.runSqoopTool(SqoopUtils.java:146)
at com.ibm.biginsights.ie.db.DBImportImpl.importData(DBImportImpl.java:159)
at com.ibm.biginsights.ie.impl.ImporterImpl.executeImport(ImporterImpl.java:504)
at com.ibm.biginsights.ie.impl.ImporterImpl.executePerformImport(ImporterImpl.java:417)
at com.ibm.biginsights.ie.impl.ImporterImpl.performImport(ImporterImpl.java:264)
at com.ibm.biginsights.biga.udf.LoadTool.performImport(LoadTool.java:214)
at com.ibm.biginsights.biga.udf.BIGSQL_DDL.doLoadStatement(BIGSQL_DDL.java:644)
at com.ibm.biginsights.biga.udf.BIGSQL_DDL.processDDL(BIGSQL_DDL.java:207)
Caused by: java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 5628] [SQLState HY000] Column 2015-12-15 03:49:33 not found in EDW01_V_LV_BASE.POS_RTL_STR.
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:308)
at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:307)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:196)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:385)
at com.teradata.jdbc.jdbc_4.TDStatement.prepareRequest(TDStatement.java:569)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.<init>(TDPreparedStatement.java:117)
at com.teradata.jdbc.jdk6.JDK6_SQL_PreparedStatement.<init>(JDK6_SQL_PreparedStatement.java:29)
at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.constructPreparedStatement(JDK6_SQL_Connection.java:81)
at com.teradata.jdbc.jdbc_4.TDSession.prepareStatement(TDSession.java:1357)
at com.teradata.jdbc.jdbc_4.TDSession.prepareStatement(TDSession.java:1401)
at com.teradata.jdbc.jdbc.ManagerConnectionBase.prepareStatement(ManagerConnectionBase.java:302)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:752)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:775)
at com.ibm.biginsights.ie.sqoop.td.TeradataFastManager.executeStmt(TeradataFastManager.java:191)
at com.ibm.biginsights.ie.sqoop.td.TeradataFastManager.getColumnTypesForRawQuery(TeradataFastManager.java:177)
... 18 more
2015-12-17 04:02:54,219 ERROR com.ibm.biginsights.ie.db.DBImportImpl [Thread-3] : [BSL-0-1ae1ada8b]: Error during import (Job Id = ):java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 5628] [SQLState HY000] Column 2015-12-15 03:49:33 not found in EDW01_V_LV_BASE.POS_RTL_STR.
I think it is somewhere wrong in the where clause where I have used value within single quote in where clause. How should it be specified? Or there is any other error?
Please visit: https://www.ibm.com/support/knowledgecenter/?lang=en#!/SSPT3X_4.1.0/com.ibm.swg.im.infosphere.biginsights.db2biga.doc/doc/biga_load_from.html?cp=SSPT3X_4.1.0%2F4-4-0
In the section FROM SQL QUERYselect-statement:
Surround the value for QUERY with single quotes. If a single quote is needed inside the value then use two single quotes.
… FROM SQL QUERY 'SELECT ID, NAME,
STATE, CITY FROM CITIES
WHERE CITY=''San Jose'' AND $CONDITIONS'
...
Related
I'm querying a MySQL table using PySpark JDBC. The problem I'm facing is that said table has column names with spaces.
In MySQL Workbench I would do the query like this to get the data and that works fine.
select
`some id`
from
table
But when I do it via PySpark like this:
query = 'select `some id` from table'
df = sqlContext.read.format('jdbc') \
.options(
driver = 'com.mysql.jdbc.Driver'
, url = some_connection_url
, dbtable = query
, user = user
, password = password
).load()
I get the following error message:
Py4JJavaError: An error occurred while calling o497.load.
: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'select
`some id`
from
some_table WHERE 1=0' at line 1
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:61)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:210)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
I've tried quoting the column names with [] and "" but I get the same error message.
I'm also using the latest MySQL jars: mysql-connector-java-8.0.21.jar and Spark 2.4.
Any insights are highly appreciated.
Enclose your query in brackets () then try to pass to read.format("jdbc").
Example:
query = '(select `some id` from table)e'
df = sqlContext.read.format('jdbc') \
.options(
driver = 'com.mysql.jdbc.Driver'
, url = some_connection_url
, dbtable = query
, user = user
, password = password
).load()
I am trying to write some data to a database but only when its not already in the table. Im using the Variables data , data1 ,timestamp, and name.My Python MySql Connector looks like this
mycursor.execute("""INSERT INTO Messung (Date,capacity,Cpu,HostID) VALUES (%s, %s, %s, %s)WHERE NOT EXISTS (SELECT * FROM Messung WHERE Date = 'timestamp' AND HostID = 'name')""", (timestamp ,data ,data1 ,name))
I get the Following Error
mycursor.execute("""INSERT INTO Messung (Date,capacity,Cpu,HostID) VALUES (%s, %s, %s, %s)WHERE NOT EXISTS (SELECT * FROM Messung WHERE Date = 'timestamp' AND HostID = 'name')""", (timestamp ,data ,data1 ,name))
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 559, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/usr/local/lib/python3.6/site-packages/mysql/connector/connection.py", line 494, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/usr/local/lib/python3.6/site-packages/mysql/connector/connection.py", line 396, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE NOT EXISTS (SELECT * FROM Messung WHERE Date = 'timestamp' AND HostID = 'n' at line 1
I dont get whats wrong with the Syntax
I also dont want to update an Existing entry just to skip them
SAMPLE DATA:
timestamp=2019-11-20 00:00:00
name =testpc
data= 18.14
data1= 58.53
I am Splitting strings like this
2019-11-18 00:00:00 testpc data 11.58 data1 29.20
MySQL Insert Syntax doesn't support where clause in it. May be you can try
INSERT ON DUPLICATE KEY UPDATE.
mycursor.execute("""INSERT INTO Messung (Date,capacity,Cpu,HostID) VALUES (%s, %s, %s, %s) ON DUPLICATE KEY UPDATE Date = 'timestamp' """, (timestamp ,data ,data1 ,name))
https://www.mysqltutorial.org/mysql-insert-or-update-on-duplicate-key-update/
i am trying to connect to mariaDB v10.2.14 from my spring JPA hibernate application. i am using
driverClassName: com.mysql.jdbc.Driver
and
setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5Dialect");
i have a query method to find the top first row, by using "findTop1" keywords... when the query is actually formed , it uses 'offset 0 rows fetch next 1 rows only' instead of LIMIT 0,1 ... because of this the maria DB server throws an error...
how can i change my driver / dialect file to form a query using LIMIT clause instead of offset-fetch..
note: i have tried all combinations of mariadbdialect and mariadb driverclass names like , org.hibernate.dialect.MariaDB53Dialect and org.mariadb.jdbc.Driver..
below is the exception i am getting
2018-06-27 11:31:19.936 INFO 10484 --- [nio-8080-exec-2] o.h.h.i.QueryTranslatorFactoryInitiator : HHH000397: Using ASTQueryTranslatorFactory
Hibernate: select aboutpage0_.id as id1_0_, aboutpage0_.APPLICATION_NAME as APPLICAT2_0_, aboutpage0_.createdDate as createdD3_0_, aboutpage0_.effectiveDate as effectiv4_0_, aboutpage0_.LAST_UPD_TS as LAST_UPD5_0_, aboutpage0_.LAST_UPD_SYSUSR_ID as LAST_UPD6_0_, aboutpage0_.version as version7_0_ from XXXXX aboutpage0_ where aboutpage0_.APPLICATION_NAME=? and aboutpage0_.effectiveDate<=? order by aboutpage0_.LAST_UPD_TS desc, aboutpage0_.effectiveDate desc offset 0 rows fetch next ? rows only
2018-06-27 11:31:20.112 WARN 10484 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 1064, SQLState: 42000
2018-06-27 11:31:20.112 ERROR 10484 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'offset 0 rows fetch next 1 rows only' at line 1
2018-06-27 11:31:20.142 ERROR 10484 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet] with root cause
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'offset 0 rows fetch next 1 rows only' at line 1
Prettyprinting the query:
select aboutpage0_.id as id1_0_, aboutpage0_.APPLICATION_NAME as APPLICAT2_0_,
aboutpage0_.createdDate as createdD3_0_, aboutpage0_.effectiveDate as effectiv4_0_,
aboutpage0_.LAST_UPD_TS as LAST_UPD5_0_, aboutpage0_.LAST_UPD_SYSUSR_ID as LAST_UPD6_0_,
aboutpage0_.version as version7_0_
from XXXXX aboutpage0_
where aboutpage0_.APPLICATION_NAME=?
and aboutpage0_.effectiveDate<=?
order by aboutpage0_.LAST_UPD_TS desc,
aboutpage0_.effectiveDate desc
offset 0 rows fetch next ? rows only 2018-06-27
should make it obvious that something went wrong in creating the SQL from Hibernate. It was probably headed for saying limit 1 offset 0, but failed. (Note: LIMIT comes before OFFSET.)
I have a website build with Symfony3 and MySQL.
Now I must migrate from MySQL to Postgres for specials needs.
I've found this tool for migrate it automatically (nmig), and all it was all done, no problem found! YEAH!
I've changed my app/config/parameters.yml as symfony documentation tells.
Cache cleared!
But when I try to execute a simple query from symfony form index, I get this error:
An exception occurred while executing
SELECT DISTINCT id_0, MIN(sclr_25) AS dctrn_minrownum
FROM (
SELECT c0_.ID AS id_0,
c0_.RAG_SOCIALE AS rag_sociale_1,
c0_.INDIRIZZO AS indirizzo_2,
c0_.CIVICO AS civico_3,
c0_.CITTA AS citta_4,
c0_.PROVINCIA AS provincia_5,
c0_.NAZIONE AS nazione_6,
c0_.SETTORE AS settore_7,
c0_.REFERENTE1 AS referente1_8,
c0_.REFERENTE2 AS referente2_9,
c0_.CAP AS cap_10,
c0_.EMAIL1 AS email1_11,
c0_.EMAIL2 AS email2_12,
c0_.PIVA AS piva_13,
c0_.PATHLOGO AS pathlogo_14,
c0_.TELEFONO AS telefono_15,
c0_.NOTE AS note_16,
c0_.VALORE_OFFERTE_TOTALI AS valore_offerte_totali_17,
c0_.VALORE_OFF_ORD_ATTIVE AS valore_off_ord_attive_18,
c0_.VALORE_FATTURE_TOTALI AS valore_fatture_totali_19,
c0_.BANCA_APPOGGIO AS banca_appoggio_20,
c0_.CONDIZIONI_PAGAMENTO AS condizioni_pagamento_21,
c0_.RELATIVEID AS relativeid_22,
c0_.created_at AS created_at_23,
c0_.updated_at AS updated_at_24,
ROW_NUMBER() OVER(ORDER BY c0_.ID DESC) AS sclr_25
FROM clienti c0_
) dctrn_result
GROUP BY id_0
ORDER BY dctrn_minrownum ASC
LIMIT 10
OFFSET 0
**SQLSTATE[42703]: Undefined column: 7 ERRORE: la colonna c0_.id non esiste
LINE 1: ...d_0, MIN(sclr_25) AS dctrn_minrownum FROM (SELECT c0_.ID AS ...
^
500 Internal Server Error - InvalidFieldNameException**
Same error if I try this query into sqleditor: same error!
What's wrong on my configuration??
Thx
When I execute this query...
UPDATE tbl a,
(SELECT id, EXTRACTVALUE(content, '//a[contains(text(), "View")]/#href') AS url FROM tbl) b
SET tbl.`url` = b.`url`
...I see this error:
Error Code: 1525
Incorrect XML value: 'parse error at line 57 pos 195: '</div>' unexpected (END-OF-INPUT wanted)'
But when I execute this query...
SELECT id, EXTRACTVALUE(content, '//a[contains(text(), "View")]/#href') AS url FROM tbl
...the query succeeds.
Why does the UPDATE query fail where the standalone SELECT query succeeds?