I'm following these instructions and I've got to running Hive. I ran the following commands:
ADD JAR /home/cloudera/Downloads/hive-serdes-1.0-SNAPSHOT.jar
CREATE EXTERNAL TABLE tweets (
id BIGINT,
created_at STRING,
source STRING,
favorited BOOLEAN,
retweeted_status STRUCT<
text:STRING,
user:STRUCT<screen_name:STRING,name:STRING>,
retweet_count:INT>,
entities STRUCT<
urls:ARRAY<STRUCT<expanded_url:STRING>>,
user_mentions:ARRAY<STRUCT<screen_name:STRING,name:STRING>>,
hashtags:ARRAY<STRUCT<text:STRING>>>,
text STRING,
user STRUCT<
screen_name:STRING,
name:STRING,
friends_count:INT,
followers_count:INT,
statuses_count:INT,
verified:BOOLEAN,
utc_offset:INT,
time_zone:STRING>,
in_reply_to_screen_name STRING
)
PARTITIONED BY (datehour INT)
ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe'
LOCATION '/home/cloudera/flume/tweets';
and then I encountered an error:
CREATE does not exist
Query returned non-zero code: 1, cause: CREATE does not exist.
As I'm new to Hive, I might be missing something obvious.
What might be causing such an error?
I was getting similar error on my Hive console while runing hive commands:
create does not exist
Query returned non-zero code: 1, cause: create does not exist
I resolved this problem by setting the Hive run as user setting.
I changed it from "Run as end user instead of Hive user" from True to False and restarted Hive server/clients.
with this setting my hive commands started running with hive user and started working.
before making this setting the default user id the root user where hive was running from.
This is hive setting issue please restart your hive console and check your hive-jdbc version and hadoop version compatability. Hope this will solve your issue as i can see the query is fine.
The problem is that you didn't put ; in the end of the first statement.
You need to change this:
ADD JAR /home/cloudera/Downloads/hive-serdes-1.0-SNAPSHOT.jar
Into this:
ADD JAR /home/cloudera/Downloads/hive-serdes-1.0-SNAPSHOT.jar;
Related
I have a table in hive created from many json files using hive-json-serde method, WITH SERDEPROPERTIES ('dots.in.keys' = 'true'), as some keys there have a dot in, like `aaa.bbb`. I create external table and use backticks for these keys. Now I have a problem dropping this table from pyspark script, using sqlContext.sql("DROP TABLE IF EXISTS "+table_name), I'm getting this error message:
An error occurred while calling o63.sql.
: org.apache.spark.SparkException: Cannot recognize hive type string: struct<associations:struct<aaa.bbb:array<string> ...
Caused by: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '.' expecting ':'(line 1, pos 33)
== SQL ==
struct<associations:struct<aaa.bbb:array<string>,...
---------------------------------^^^
In HUE i can drop this table without any problem. Am I doing it wrong, or may be there is better way to do it?
It looks like it is not possible to work with Hive tables created with the hive-json-serde method, with dot in keys , using sqlContext.sql("...") from pyspark script, as I want. There is always the same error, if I want to drop such Hive table, or create it (haven't tried other things yet). So my workaround is to use python os.system() and execute required query through hive itself:
q='hive -e "DROP TABLE IF EXISTS '+ table_name+';"'
os.system(q)
It's more complicated with CREATE TABLE query, as we need to escape backticks with '\':
statement = "CREATE TABLE test111 (testA struct<\`aa.bb\`:string>)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3a://bucket/test111';"
q='hive -e "'+ statement+'"'
It outputs some additional hive related info, but works!
i work on cloudera quickstart with docker, I'm trying to create a table into hive interface.
This is my code.
add jar hdfs:///user/cloudera/hive-serdes-1.0-SNAPSHOT.jar
drop table if exists tweets;
CREATE EXTERNAL TABLE tweets (
id BIGINT,
created_at STRING,
source STRING,
favorited BOOLEAN,
retweeted_status STRUCT<
text:STRING,
user1:STRUCT<screen_name:STRING,name:STRING>,
retweet_count:INT>,
entities STRUCT<
urls:ARRAY<STRUCT<expanded_url:STRING>>,
user_mentions:ARRAY<STRUCT<screen_name:STRING,name:STRING>>,
hashtags:ARRAY<STRUCT<text:STRING>>>,
text STRING,
user1 STRUCT<
screen_name:STRING,
name:STRING,
friends_count:INT,
followers_count:INT,
statuses_count:INT,
verified:BOOLEAN,
utc_offset:INT,
time_zone:STRING>,
in_reply_to_screen_name STRING
)
ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe'
LOCATION '/user/cloudera/';
load data inpath '/user/cloudera/search.json' into table tweets;
when I run "select * from tweets;", I get this error
Fetching results ran into the following error(s):
Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\xf2e\xcc\xb6v\x8eC"\xae^x\x89*\xd6j\xa7', guid='h\xce\xacgmZIP\x8d\xcc\xc0\xe8C\t\x1a\x0c')), orientation=4, maxRows=100): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage='java.io.IOException: java.io.IOException: Not a file: hdfs://quickstart.cloudera:8020/user/cloudera/2015_11_18', sqlState=None, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:java.io.IOException: java.io.IOException: Not a file: hdfs://quickstart.cloudera:8020/user/cloudera/2015_11_18:25:24', 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:366', 'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:275', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:752', 'sun.reflect.GeneratedMethodAccessor19:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606',
Don't use your user folder as a Hive table location. A user folder is meant for general file storage, such as that 2015_11_18 directory it's trying to read, not an entire Hive structure.
Do LOCATION '/user/cloudera/tweets';, for example instead.
You could also just make a regular managed table if you don't care if things are deleted when you drop the table.
I have transfered my project from MySQL to PostgreSQL and tried to drop the column as result of previous issue, because after I removed the problematic column from models.py and saved. error didn't even disappear. Integer error transferring from MySQL to PostgreSQL
Tried both with and without quotes.
ALTER TABLE "UserProfile" DROP COLUMN how_many_new_notifications;
Or:
ALTER TABLE UserProfile DROP COLUMN how_many_new_notifications;
Getting the following:
ERROR: relation "UserProfile" does not exist
Here's a model, if helps:
class UserProfile(models.Model):
user = models.OneToOneField(User)
how_many_new_notifications = models.IntegerField(null=True,default=0)
User.profile = property(lambda u: UserProfile.objects.get_or_create(user=u)[0])
I supposed it might have something to do with mixed-case but I have found no solution through all similar questions.
Yes, Postgresql is a case aware database but django is smart enough to know that. It converts all field and it generally converts the model name to a lower case table name. However the real problem here is that your model name will be prefixed by the app name. generally django table names are like:
<appname>_<modelname>
You can find out what exactly it is by:
from myapp.models import UserProfile
print (UserProfile._meta.db_table)
Obviously this needs to be typed into the django shell, which is invoked by ./manage.py shell the result of this print statement is what you should use in your query.
Client: DataGrip
Database engine: PostgreSQL
For me this worked opening a new console, because apparently from the IDE cache it was not recognizing the table I had created.
Steps to operate with the tables of a database:
Database (Left side panel of the IDE) >
Double Click on PostgreSQL - #localhost >
Double Click on the name of the database >
Right click on public schema >
New > Console
GL
SnappyData v.0.5.2
I am using this SnappyData version to get a fix for SNAP-961.
However, now I am unable to load data from a CSV anymore, after moving from the preview release v0.5 to v0.5.2.
ERROR IS:
ERROR 38000: (SQLState=38000 Severity=-1)
(Server=ip-10-0-18-66.us-west-2.compute.internal[1528],Thread[DRDAConnThread_28,5,gemfirexd.daemons])
The exception 'Failed to find a builtin provider
com.databricks.spark.csv;' was thrown while evaluating an expression.
Here is what I am executing:
-- creates in-memory table from csv
CREATE TABLE STAGING_ROAD (road_id string, name string) USING com.databricks.spark.csv OPTIONS(path 'roads.csv', header 'true', inferSchema 'false');
There has been an alignment of SQL and Spark APIs so now only builtin datasources (column, row, streaming/AQP ones) can use "CREATE TABLE" while others have to use "CREATE EXTERNAL TABLE". Similar was the case with SnappyContext where createTable API could be used only for builtin sources while for others createExternalTable was required. The following should work with both older releases and newer ones:
CREATE EXTERNAL TABLE STAGING_ROAD (road_id string, name string) USING com.databricks.spark.csv OPTIONS(path 'roads.csv', header 'true', inferSchema 'false')
hi there i made track star application and everything works fine except for one issue that i face error
CDbCommand failed to execute the SQL statement: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'devnmark_root.AuthAssignment' doesn't exist. The SQL statement executed was: SELECT *
FROM 'AuthAssignment'
WHERE userid=:userid
now this was automatically generated by Yii when i checked for if(Yii::app()->user->checkAccess('createUser',array('project'=>$model)))
18 {
19 $this->menu[] = array('label'=>'Add User To Project','url'=>array('adduser', 'id'=>$model->id));
20 }
then i went to phpmyadmin and executed this query manually
SELECT * FROMAuthAssignmentWHERE userid=4 and there is error which says same that table does not exist.
if i use small case letter for table name then no error.
i executed same query on local wamp 's phpmyadmin same query does not show any error there so this is clear that there is error with my sql .any idea what can i do to solve?
I suggest you hitting up SQL_MODE documentation for setting your final options. http://dev.mysql.com/doc/refman/5.0/en/server-sql-mode.html
For testing you can just do a:
SET sql_mode = '';
Or adjust your command line:
--sql-mode=""
You may try to set the system variable *lower_case_table_names* to 1.
My local application was developed under Windows, howerver production is under Linux and I caused the same problem.
For me it happened because of case sensitivity - table AuthAssignment in database was actually authassignment.
I have found two options to solve it:
rename database table from authassignment to AuthAssignment
or
edit config file component section:
'components'=>array(
...
'user'=>array(...
),
'authManager'=>array(
...
'assignmentTable'=>'authassignment' //THIS LINE SOLVED MY PROBLEM
),
I had the same problem of non-existent authassignment table, I had forgotten running the rights installation which in my case was by following URL i.e.
/index.php/rights/install
and then the setup was like a breeze and self explanatory :)
For me this answer helped the most: original source.
In the authManager in config/main.php you can add lowercased names of the tables like this:
'components'=>array(
'authManager'=>array(
'defaultRoles'=>array('guest'),
'class'=>'RDbAuthManager',
'assignmentTable'=>'authassignment',
'itemTable'=>'authitem',
'rightsTable'=>'rights',
'itemChildTable'=>'authitemchild',
),