Here I am doing Incremental import using Java. But I am not getting how to use already existing job.
I mean
sqoop job --create moviesJob1 -- import --connect jdbc:mysql://localhost/mydb --username root -P --table movies --m 1 --target-dir /movies --incremental append --check-column movieId --last-val 0
SqoopOptions options = new SqoopOptions();
options.setJobName(tableDTO.getTableName()+"Job");
options.setDriverClassName(driver);
options.setHadoopHome(HADOOP_HOME);
options.setConnectString(MYSQL_CONNECTION_STRING);
options.setUsername(USERNAME);
options.setPassword(PASSWORD);
options.setTableName(tableDTO.getTableName());
options.setNumMappers(Integer.parseInt(tableDTO.getNoOfMappers()));
options.setTargetDir(HDFS_DIRECTORY_PATH+"/"+tableDTO.getTableName());
options.setFieldsTerminatedBy('|');
options.setAppendMode(true);
options.setIncrementalMode(IncrementalMode.AppendRows);
options.setIncrementalLastValue("0");
I have written code for this,
now I have to execute:
sqoop job --exec moviesJob1
To execute this:
options.getJobName(); // getting JobName but not finding method to run job
Can anyone give suggestion?
Thanks in advance.
Try something like this:
new ExportTool().run(options);
or:
new com.cloudera.sqoop.tool.ImportTool().run(options);
Related
I have a table (regularly updated) in Hive that I want to have in one of my tool that has a MySQL database. I can't just connect my application to the Hive database, so I want to export those data directly in the MySQL database.
I've searched a bit and found out that it was possible with Sqoop, and I've been told to use Oozie since I want to regularly update the table and export it.
I've looked around for a while and tried some stuff but so far I can't succeed, and I just don't understand what I'm doing.
So far, the only code I understand but doesn't work looks like that :
export --connect jdbc:mysql://myserver
--username username
--password password
--table theMySqlTable
--hive-table cluster.hiveTable
I've seen people using temporary table and export it on a txt file to then export it, but I'm not sure I can do it.
Should Oozie have specific parameters too ? I'm not the administrator so I'm not sure if I'm able to do it...
Thank you !
Try this.
sqoop export \
--connect "jdbc:sqlserver://servername:1433;databaseName=EMP;" \
--connection-manager org.apache.sqoop.manager.SQLServerManager \
--username userid \
-P \
--table theMySqlTable\
--input-fields-terminated-by '|' \
--export-dir /hdfs path location of file/part-m-00000 \
--num-mappers 1 \
I am new to cassandra. Here I am tring to transfer whole my MYSQL database to cassandra using sqoop. But after all setup, when i execute following command.
bin/dse sqoop import-all-tables -m 1 --connect jdbc:mysql://127.0.0.1:3306/ABCDatabase --username root --password root --cassandra-thrift-host localhost --cassandra-create-schema --direct
I have received following error.
Sqoop functionality has been removed from DSE.
It said that sqoop functionality is removed from datastax. can you please if it removed then is there any other way to do that?
Thanks
You can use Spark to transfer data - it should be easy, something like:
val table = spark.read.jdbc(jdbcUrl, "table", connectionProperties)
table.write.format("org.apache.spark.sql.cassandra").options(
Map("table" -> "TBL", "keyspace" -> "KS")).save()
Examples of jdbc URLs, options, etc. are described in Databrick's documentation as they could be different for different databases.
I'm training myself for the HDPCD exam, so I'm testing all possible import and exports using MySQL to Hive. In this example, I would like to import a table from MySQL and create from scratch the same table in hive using the parameter --create-hive-table. Although in the [documentation][1] it's included I have found a right example to do it. I have tried this but it doesn't work
sqoop import --connect jdbc:mysql://master/poc --username root --table dept --where 'id_dept > 2' --hive-import --hive-database poc --hive-table deptv2 --create-hive-table true -m 1 --split-by id_dept
Please, guys if someone of you knows how to use it let me know it. I appreciate thanks so much
I'm back cause I have just to try again just putting the parameter without any input and it worked that's it. Anyways I'll leave an example. Probably it's going to help someone.
sqoop import --connect jdbc:mysql://master/poc --username root \
--table dept --where 'id_dept > 2' --hive-import \
--hive-database poc --hive-table deptv2 --create-hive-table -m 1 --split-by id_dept
Thanks.
I'm using dse 5.0.3 with have sqoop version 1.4.5.15.1 while im importing my data from mysql to cql this error i get
./dse sqoop cql-import --table npa_nxx --connect jdbc:mysql://localhost/npa_nxx_demo --username root --password 123
ERROR 13:20:53,886 Imported Failed: Parameter 'directory' is not a directory.
Please help me to solve it.!!
It's difficult to be exact about the problem but I'd suggest that you aren't including any cassandra parameters to the cql-import command is attempting to import to an hdfs directory that also isn't declared. Try including a cassandra-keyspace and cassandra-table on the command. Like:
./dse sqoop cql-import --table npa_nxx --connect jdbc:mysql://localhost/npa_nxx_demo --username root --password 123 --cassandra-keyspace npa_nxx --cassandra-table npa_nxx_data
This assumes that the cassandra keyspace and table are correctly setup.
Since this import looks like it is from the dse sqoop demo I'd suggest following the README.txt more closely as it has the correct options for this import in it.
I am trying to import data from Mysql to Hbase using sqoop.
I am running following command.
sqoop import --connect jdbc:mysql://localhost/database --table users --columns "loginid,email" --username tester -P -m 8 --hbase-table hbaseTable --hbase-row-key user_id --column-family user_info --hbase-create-table
But i am getting below error :-
13/05/08 10:42:10 WARN hbase.ToStringPutTransformer: Could not insert
row with null value for row-key column: user_id
please help here
Got the solution.
I was not including my rowKey i.e. user_id in the columns list.
After including it , it worked like a charm.
Thanks..
Your columns should be upper status, not seq_id but SEQ_ID.
I think sqoop considering it as a different column.which is null(Of course).