I tried to import data through Sqoop using the following command.
sqoop import -connect jdbc:mysql://localhost/test_sqoop --username root --table test
but I got the connection refuse error.
And I found out I can't connect to mysql and got this error:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
And I also found out if I don't execute start-dfs.sh,mysql.sock exists in /var/lib/mysql/mysql.sock.
mysql
After I executed start-dfs.sh,mysql.sock would be gone and I can't connect to mysql.
start-dfs.sh
Below is /etc/my.cnf configuration.
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
jdbc string should be: jdbc:mysql://localhost:3306/test_sqoop, best practice is to use server name intesad of localhost or 127.0.0.1. you can get the server name from this command hostname -f. so jdbc string should be jdbc:mysql://servername:3306/test_sqoop - replace the server name by out put of hostname -f command.
you need -P or --password or --connection-param-file to pass the password to the sqoop command. sqoop doesn't read from .my.cnf file. - see usage here
Related
I'm trying to do database management via SSH for Bitnami's WordPress Multi-tier Stack/ Specifically I want to export and do an initial import (though I will probably just create a new database).
When I run the following commands, I get the following errors:
Command: mysqldump -u root -p bitnami_wordpress > bitnami_wordpress.sql
Output: mysqldump: Got error: 2002: "Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)" when trying to connect
This also creates an 0B SQL file in my home directory.
Command: mysqladmin -u root -p status (I enter my password)
Output:
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket .
'/opt/bitnami/mariadb/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket:
'/opt/bitnami/mariadb/tmp/mysql.sock' exists!
Command: cd /opt/bitnami/mariadb/ + ls
Output:
bin CREDITS include licenses README-wsrep
COPYING data INSTALL-BINARY plugin sbin
COPYING.thirdparty EXCEPTIONS-CLIENT lib README.md share
Command: sudo find . -name mysql
Output:
./root/.nami/components/com.bitnami.mysql-client/lib/databases/mysql
./root/.nami/components/com.bitnami.mysql-client/lib/handlers/databases/mysql
./root/.nami/components/com.bitnami.libphp/lib/databases/mysql
./root/.nami/components/com.bitnami.libphp/lib/handlers/databases/mysql
./root/.nami/components/com.bitnami.wordpress/lib/databases/mysql
./root/.nami/components/com.bitnami.wordpress/lib/handlers/databases/mysql
./root/.nami/components/com.bitnami.php/lib/databases/mysql
./root/.nami/components/com.bitnami.php/lib/handlers/databases/mysql
./root/.nami/components/com.bitnami.apache/lib/databases/mysql
./root/.nami/components/com.bitnami.apache/lib/handlers/databases/mysql
./root/.nami/components/com.bitnami.mariadb/lib/databases/mysql
./root/.nami/components/com.bitnami.mariadb/lib/handlers/databases/mysql
./opt/bitnami/mysql
./opt/bitnami/mysql/bin/mysql
./opt/bitnami/mariadb/include/mysql
./opt/bitnami/mariadb/include/mysql/server/mysql
./opt/bitnami/mariadb/include/mysql/mysql
./opt/bitnami/mariadb/bin/mysql
./usr/share/bash-completion/completions/mysql
Commands:
find /opt/bitnami/mysql/ -name "*.cnf"
Output: Nothing
find /opt/bitnami/mariadb/ -name "my.cnf"
Output:
/opt/bitnami/mariadb/share/my-medium.cnf
/opt/bitnami/mariadb/share/my-small.cnf
/opt/bitnami/mariadb/share/my-large.cnf
/opt/bitnami/mariadb/share/my-innodb-heavy-4G.cnf
/opt/bitnami/mariadb/share/my-huge.cnf
/opt/bitnami/mariadb/share/wsrep.cnf
Command: nano /opt/bitnami/mariadb/share/my-medium.cnf (what's the difference between my-medium, my-small, and my-large)?
Output:
# The following options will be passed to all MariaDB clients
[client]
#password = your_password
port = 3306
socket = /opt/bitnami/mariadb/tmp/mysql.sock
NOTE: /opt/bitnami/mariadb/tmp/mysql.sock does not exist.
I've poked around a bit and came across MariaDB's Documentation about 2002 errors, but I don't seem to have the same .conf file (nor do I know where to look)
...from here I have no idea where to go, I've only done limited database management via shell.
Concise questions:
How do I export my database without getting the 2002 error?
How do I overwrite / update my database?
Any help would be much appreciated and thanks in advance!
The folks at Bitnami came through. I was connecting to the wrong host.
Find the host:
sudo cat /opt/bitnami/wordpress/wp-config.php | grep 'DB_HOST'
To export:
mysqldump -h provisioner-peer -u root -p bitnami_wordpress > bitnami_wordpress.sql
I'm trying to connect to a remote MySQL database via an Ubuntu server using a command like:
mysql -u oooo -p -h ooo.ooo.oo.oo
Then it just returned:
mysql: unknown variable 'log=/var/log/mysql/logfile.log'
I've checked that the file 'logfile.log' does not exist and I have no idea what is the next step.
Search your machine for a file named my.cnf. That file will probably contain your offending variable. Comment it out.
I am using hadoop-2.6.0 with kerberos security. I have installed hbase with kerberos security and could able to create table and scan it.
I could run sqoop job as well to import data from mysql into hdfs but sqoop job fails when trying to import from mysql into HBase.
Sqoop Command
sqoop import --hbase-create-table --hbase-table newtable --column-family ck --hbase-row-key id --connect jdbc:mysql://localhost/sample --username root --password root --table newtable -m 1
Exception
15/01/21 16:30:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x734c0647, quorum=localhost:2181, baseZNode=/hbase
15/01/21 16:30:24 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknownerror)
15/01/21 16:30:24 INFO zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
15/01/21 16:30:24 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14b0ac124600016, negotiated timeout = 40000
15/01/21 16:30:25 ERROR tool.ImportTool: Error during import: Can't get authentication token
Could you please try the following :
In the connection string add the port number as :
jdbc:mysql://localhost:3306/sample
Remove --table newtable. Create the required table on Hbase first with the column family.
mention --split-by id
Finally mention a specific --fetch-size , as the sqoop client for MySQL have an error internally which attempts to set the default MIN fetch size which will run into an exception.
Could you attempt the import again and let us know ?
I Installed Sqoop in my local machine. Following are the config information.
Bash.bashrc:
export HADOOP_HOME=/home/hduser/hadoop
export HBASE_HOME=/home/hduser/hbase
export HIVE_HOME=/home/hduser/hive
export HCAT_HOME=/home/hduser/hive/hcatalog
export SQOOP_HOME=/home/hduser/sqoop
export PATH=$PATH:$HIVE_HOME/bin
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HBASE_HOME/bin
export PATH=$PATH:$SQOOP_HOME/bin
export PATH=$PATH:$HCAT_HOME/bin
Hadoop:
Version: Hadoop 1.0.3
Hive:
Version: hive 0.11.0
Mysql Connector driver
version: mysql-connector-java-5.1.29
"The driver is copied to the lib folder of sqoop"
Sqoop :
version: sqoop 1.4.4
After making all the installation I create a table in mysql named practice_1, But when I run the load command to load data from mysql to hdfs the command throws an exception:
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
Coud anyone please guide me what can be the possible problem.
You need database driver in 'SQOOP' classpath check this
It has wonderful explanation about the 'SQOOP'
SQOOP has other options like
Ex: --driver com.microsoft.jdbc.sqlserver.SQLServerDriver -libjars=".*jar"
from here
You can use Sqoop with any other JDBC-compliant database. First, download the appropriate JDBC driver for the type of database you want to import, and install the .jar file in the $SQOOP_HOME/lib directory on your client machine. (This will be /usr/lib/sqoop/lib if you installed from an RPM or Debian package.) Each driver .jar file also has a specific driver class which defines the entry-point to the driver. For example, MySQL's Connector/J library has a driver class of com.mysql.jdbc.Driver. Refer to your database vendor-specific documentation to determine the main driver class. This class must be provided as an argument to Sqoop with --driver.
You may be interested in understanding the difference between connector and driver here is the article
Another solution which avoids using a shared library is adding the driver jar to the classpath of sqoop by using HADOOP_CLASSPATH. I haven't got the -libjars option to work. This solution works also on a secure cluster using kerberos.
HADOOP_CLASSPATH=/use.case/lib/postgresql-9.2-1003-jdbc4.jar
sqoop export --connect jdbc:postgresql://db:5432/user \
--driver org.postgresql.Driver \
--connection-manager org.apache.sqoop.manager.GenericJdbcManager \
--username user \
-P \
--export-dir /user/hive/warehouse/db1/table1 \
--table table2
This one works at least with sqoop 1.4.3-cdh4.4.0
You need to add the MySql connector to /usr/lib/sqoop/lib.
MySQL JDBC Driver by default is not present in Sqoop distribution in order to ensure that the default distribution is fully Apache license compliant.
Hope this helps...!!!
copy the 'mysql-connector-java-5.1.41-bin.jar' into sqoop/lib folder and execute sqoop import statements
If you have copied mysql driver to the sqoop lib folder. It will work for sure. Make sure you sqoop command is correct
/home/hduser/sqoop/bin/sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password root -–table practice_1 -m 1
It's a Oozie ShareLib problem. The script below works for my:
At Shell
sudo -u hdfs hadoop fs -chown cloudera:cloudera /user/oozie/share/lib/lib_20170719053712/sqoop
hdfs dfs -put /var/lib/sqoop/mysql-connector-java.jar /user/oozie/share/lib/lib_20170719053712/sqoop
sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20170719053712/sqoop
oozie admin -oozie http://localhost:11000/oozie -sharelibupdate
oozie admin -oozie http://localhost:11000/oozie -shareliblist sqoop
At Hue Sqoop Client
sqoop list-tables --connect jdbc:mysql://localhost/retail_db --username root --password cloudera
More detail at:
https://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
You need to grant priveleges to the tables as below:
grant all privileges on marksheet.* to 'root'#'192.168.168.1'
identified by 'root123';
flush privileges;
Here is sample command that I have successfully executed:
sqoop import --verbose --fields-terminated-by ',' --connect
jdbc:mysql://192.168.168.1/test --username root --password root123
--table student --hive-import --create-hive-table --hive-home /home/training/hive --warehouse-dir /user/hive/warehouse
--fields-terminated-by ',' --hive-table studentmysql
Is there a way to get a MySQL Dump of a database that is not stored locally. I have the connection string to the database located on another server, but it doesn't seem like MySQLDump wants to do anything if the server is remote.
MySQLDump has a -h parameter to connect to a remote host.
First try the mysql client application:
mysql -h your.server.com -uYourUser -pYourPass
If that works, use the same format for MySQLDump
mysqldump -h your.server.com -uYourUser -pYourPass --all-databases
Edit for ajreal:
By default, mysqld (the MySQL server) will run on 3306, and mysql (the client application) will connect using that port. However, if you changed your configuration, update your command accordingly. For example for port 3307, use
mysql -h your.server.com -P 3307 -uYourUser -pYourPass
Check your MySQL configfile to see how you can connect to your MySQL server.
Here is example, how to extract mysql db named 'abc123' direct to zip, w/o super big text dump file on disk.
mysqldump -u root --opt --databases abc123 | gzip > /tmp/abc123.export.sql.gz