SnappyData v.0.5.2
I am using this SnappyData version to get a fix for SNAP-961.
However, now I am unable to load data from a CSV anymore, after moving from the preview release v0.5 to v0.5.2.
ERROR IS:
ERROR 38000: (SQLState=38000 Severity=-1)
(Server=ip-10-0-18-66.us-west-2.compute.internal[1528],Thread[DRDAConnThread_28,5,gemfirexd.daemons])
The exception 'Failed to find a builtin provider
com.databricks.spark.csv;' was thrown while evaluating an expression.
Here is what I am executing:
-- creates in-memory table from csv
CREATE TABLE STAGING_ROAD (road_id string, name string) USING com.databricks.spark.csv OPTIONS(path 'roads.csv', header 'true', inferSchema 'false');
There has been an alignment of SQL and Spark APIs so now only builtin datasources (column, row, streaming/AQP ones) can use "CREATE TABLE" while others have to use "CREATE EXTERNAL TABLE". Similar was the case with SnappyContext where createTable API could be used only for builtin sources while for others createExternalTable was required. The following should work with both older releases and newer ones:
CREATE EXTERNAL TABLE STAGING_ROAD (road_id string, name string) USING com.databricks.spark.csv OPTIONS(path 'roads.csv', header 'true', inferSchema 'false')
Related
I'm trying to use alembic with snowflake to version control the schema I use for both PostgreSQL and snowflake. I keep running into this Unexpected 'UNIQUE' error. I know this is being caused as it is trying to create an index, something that snowflake does not support. This is strange to me as I thought the purpose of the dialect system in SQLAlchemy was to manage differences between implementations and stop it trying to create this index when it's not supported.
I followed the guide on the snowflake website to add the dialect to alembic, calling the upgrade function like this:
engine = create_engine(SNOWFLAKE_DATABASE_URI)
with engine.begin() as con:
logger.info("Starting db upgrade.")
cfg = Config("migrations/alembic.ini")
cfg.attributes["connection"] = con
cfg.attributes["configure_logger"] = False
command.upgrade(cfg, "head")
Is the connector not working properly or am I not calling this in the correct way?
We are trying to save JSON string to the database column defined as CLOB. We get the below error:
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-02290: check constraint (MYSCHEMA.MY_JSON_CHK) violated
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:466)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:407)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1113)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:546)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:603)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:234)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:55)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1006)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1316)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:5010)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:5136)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1519)
at weblogic.jdbc.wrapper.PreparedStatement.executeUpdate(PreparedStatement.java:170)
Version: Weblogic 12.2.1.2
Attempted two data types at application ORM layer for this column
java.sql.Clob & oracle.jbo.domain.ClobDomain
Direct update via SQL works good
Enabled & Disabled wrapping of datatype in weblogic
JDBC datasource.
I attempted to switch my EO (ORM) attribute from oracle.jbo.ClobDomain back to a plain String and this worked.
I have some implementation that is best served by pickling a pandas dataframe and storing it in a DB.
This works fine if the database is sqlite but fails with a load error when it is MySQL
I have found other people with similar issues on stackoverflow and google but it seems that everybodys solution is to use sql to store the dataframe.
As a last resort I would go down that route but it would be a shame for this use case to do that.
Anybody got a solution to get the same behaviour from mysql as sqlite here?
I simply dump the dataframe with
pickledframe = pickle.dumps(frame)
and store pickledframe as a BinaryField
pickledframe = models.BinaryField(null=True)
I load it in with
unpickled = pickle.loads(pickledframe)
with sqlite it works fine, with mysql I get
Exception Type: UnpicklingError
Exception Value: invalid load key, ','.
upon trying to load it.
Thanks
I am using eclipselink 2.6 with spring-boot JPA for persistance in postgres.
I am persisting a List of objects as a JSON column in database.Acording to this solution: eclipselink + #convert(json) + postgres + list property
I am able to save the data in postgres.
When the column is null, I get this exception:
Caused by: org.postgresql.util.PSQLException: ERROR: column "sample_column" is of type json but expression is of type character varying
Hint: You will need to rewrite or cast the expression.
I can solve this issue by this answer:
Writing to JSON column of Postgres database using Spring / JPA
Q1: Is there an alternate solution other than setting this property stringtype=unspecified int url spring.datasource.url=jdbc:postgresql://localhost:5432/dbname?stringtype=unspecified
Q2: If not, How can I set stringtype=unspecified in application.prooerties of spring-boot rather than embedding it in the spring.datasource.url
The answer is yes, but is implementation-specific.
For example, in Tomcat, this attribute is called connectionProperties, you would therefore write:
spring.datasource.tomcat.connection-properties: stringtype=unspecified
From the Spring-Boot documentation.
It is also possible to fine-tune implementation-specific settings using their respective prefix (spring.datasource.tomcat., spring.datasource.hikari., spring.datasource.dbcp.* and spring.datasource.dbcp2.*). Refer to the documentation of the connection pool implementation you are using for more details.
If you are using spring-boot 1.4.1 or above,
add a data.sql file in resources folder,
-- IN H2 database, create a column as 'OTHER' data type,
-- if H2 fails to create a column as 'JSON' data type.
CREATE DOMAIN IF NOT EXISTS JSON AS OTHER;
This .sql file will be executed during startup of your application and will create a column with data type others in table for the 'json' data type columns in H2 database.
I'm following these instructions and I've got to running Hive. I ran the following commands:
ADD JAR /home/cloudera/Downloads/hive-serdes-1.0-SNAPSHOT.jar
CREATE EXTERNAL TABLE tweets (
id BIGINT,
created_at STRING,
source STRING,
favorited BOOLEAN,
retweeted_status STRUCT<
text:STRING,
user:STRUCT<screen_name:STRING,name:STRING>,
retweet_count:INT>,
entities STRUCT<
urls:ARRAY<STRUCT<expanded_url:STRING>>,
user_mentions:ARRAY<STRUCT<screen_name:STRING,name:STRING>>,
hashtags:ARRAY<STRUCT<text:STRING>>>,
text STRING,
user STRUCT<
screen_name:STRING,
name:STRING,
friends_count:INT,
followers_count:INT,
statuses_count:INT,
verified:BOOLEAN,
utc_offset:INT,
time_zone:STRING>,
in_reply_to_screen_name STRING
)
PARTITIONED BY (datehour INT)
ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe'
LOCATION '/home/cloudera/flume/tweets';
and then I encountered an error:
CREATE does not exist
Query returned non-zero code: 1, cause: CREATE does not exist.
As I'm new to Hive, I might be missing something obvious.
What might be causing such an error?
I was getting similar error on my Hive console while runing hive commands:
create does not exist
Query returned non-zero code: 1, cause: create does not exist
I resolved this problem by setting the Hive run as user setting.
I changed it from "Run as end user instead of Hive user" from True to False and restarted Hive server/clients.
with this setting my hive commands started running with hive user and started working.
before making this setting the default user id the root user where hive was running from.
This is hive setting issue please restart your hive console and check your hive-jdbc version and hadoop version compatability. Hope this will solve your issue as i can see the query is fine.
The problem is that you didn't put ; in the end of the first statement.
You need to change this:
ADD JAR /home/cloudera/Downloads/hive-serdes-1.0-SNAPSHOT.jar
Into this:
ADD JAR /home/cloudera/Downloads/hive-serdes-1.0-SNAPSHOT.jar;