I'm new to Hadoop cluster and trying to deploy a multi-node cluster on ubuntu 15.10 with ONE master and TWO slaves. After configuration, there's TWO active nodes(two slaves). However, when I tried hadoop example program below
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 3 100
I got an error of connection refused:
Job job_1459774851310_0001 failed with state FAILED due to: Application application_1459774851310_0001 failed 2 times due to Error launching appattempt_1459774851310_0001_000002. Got exception: java.net.ConnectException: Call From ubuntu/127.0.1.1 to ubuntu:36380 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
To deploy this cluster, I disabled ipv6 in all machines, and edited configuration files as following shows:
In file core-site.xml:
fs.defaultFS = hdfs://master:8020
In file hdfs-site.xml:
dfs.namenode.name.dir = $HADOOP_PREFIX/namenode
dfs.datanode.data.dir = $HADOOP_PREFIX/datanode
In file yarn-site.xml:
yarn.resourcemanager.address = master:8084
yarn.resourcemanager.schedular.address = master:8085
yarn.resourcemanager.resource-tracker.address = master:8086
yarn.resourcemanager.admin.address = master:8087
yarn.resourcemanager.webapp.address = master:8088
yarn.nodemanager.aux-services = mapreduce_shuffle
In file mapred_site.xml
mapreduce.framework.name = yarn
mapreduce.jobhistory.address = master:10020
mapreduce.jobhistory.address = master:19888
Those 4 files are the same on all machines.
Where did I make mistakes? How to fix it?
In file slaves, I wrote only the IP addresses of two slaves.
Related
I want to read a csv file from hadoop with Pyspark with the following code:
dfcsv = spark.read.csv("hdfs://my_hadoop_cluster_ip:9000/user/root/input/test.csv")
dfcsv.printSchema()
My cluster hadoop is on a Docker container on my local machine and link with two other slave container for the workers.
As you see in this picture from my ui hadoop cluster, the path is the right path.
But when I submit my script with this command :
spark-submit --master spark://my_cluster_spark_ip:7077 test.py
My script stuck on the read, and after few minutes I have this following error :
22/02/09 15:42:29 WARN TaskSetManager: Lost task 0.1 in stage 4.0 (TID 4) (my_slave_spark_ip executor 1): org.apache.hadoop.net.ConnectTimeoutException: Call From spark-slave1/my_slave_spark_ip to my_hadoop_cluster_ip:9000 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=my_hadoop_cluster_ip/my_hadoop_cluster_ip:9000]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
...
For information, my csv file is very small, just 3 lines and 64 KB.
Have you any solution to fix this issue?
I have 4 Elastic Beanstalk deployments: 3 are Corretto 8 and the other one is Corretto 11.
On the Corretto 8 deployments, I can set new configuration without issue. On the Corretto 11 instance, however, any attempt to set a new configuration fails and causes a rollback.
The Corretto versions might not be the problem, but it's the only difference I can see. All 4 apps are Spring Boot apps that run as web servers (i.e embedded tomcat with exposed web ports). I am trying to set the exact same configuration name and value, and it only fails on the one instance.
The configuration I'm trying to set is pretty simple:
VALIDATE_RENEWALS = true
Even just trying to set DEBUG = true causes a failure and rollback.
I don't see a lot of information from the console about what's failing. Here is the event log:
2020-03-16 13:55:17 UTC-0600 INFO The environment was reverted to the previous configuration setting.
2020-03-16 13:54:45 UTC-0600 ERROR During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
2020-03-16 13:54:45 UTC-0600 ERROR Failed to deploy configuration.
2020-03-16 13:54:45 UTC-0600 ERROR Unsuccessful command execution on instance id(s) 'i-00553f4ac36afd327'. Aborting the operation.
2020-03-16 13:54:45 UTC-0600 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-03-16 13:54:45 UTC-0600 ERROR [Instance: i-00553f4ac36afd327] Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].
2020-03-16 13:54:20 UTC-0600 INFO Updating environment XXX's configuration settings.
2020-03-16 13:54:15 UTC-0600 INFO Environment update is starting.
I've also downloaded the full set of logs for the instance and don't see anything obvious. The app stdout doesn't have any errors or exceptions, it just starts normally and then gets terminated. None of the other log files have messages around the times above, so I'm really not sure what else I can look at.
Edit
The times don't line up but I do see this in eb-engine.log file:
2020/03/16 17:54:38.508634 [INFO] checking whether command is applicable to this instance...
2020/03/16 17:54:38.508658 [INFO] this command is applicable to the instance, thus instance should execute command
2020/03/16 17:54:38.508665 [INFO] check whether this is an enhanced env...
2020/03/16 17:54:38.508794 [INFO] Executing instruction: StageJavaApplication
2020/03/16 17:54:38.508858 [ERROR] GetArchivedFileType with file /opt/elasticbeanstalk/deployment/app_source_bundle failed with error open /opt/elasticbeanstalk/deployment/app_source_bundle: no such file or directory
2020/03/16 17:54:38.508868 [ERROR] An error occurred during execution of command [config-deploy] - [StageJavaApplication]. Stop running the command. Error: staging java app failed with error GetArchivedFileType with file /opt/elasticbeanstalk/deployment/app_source_bundle failed with error open /opt/elasticbeanstalk/deployment/app_source_bundle: no such file or directory
I am testing Apache Drill with a two server cluster.
Let's say their external IPs are:
1.1.1.1
2.2.2.2
I first setup Zookeeper to run on both, and when I do the status command I get positive response:
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.8/bin/../conf/zoo.cfg
Mode: leader
The way I have my zoo.cfg to get it working was like this:
Server 1:
// other default values omitted
clientPort=2181
server.1=0.0.0.0:2888:3888
server.2=2.2.2.2:2888:3888
Server 2:
// other default values omitted
clientPort=2181
server.1=1.1.1.1:2888:3888
server.2=0.0.0.0:2888:3888
Next I wanted to get Drill running with this cluster, so I modify the drill-override.conf file for the 2 servers as follows:
Server 1:
drill.exec: {
cluster-id: "test",
zk.connect: "1.1.1.1:2181,2.2.2.2:2181"
}
Server 2:
drill.exec: {
cluster-id: "test",
zk.connect: "2.2.2.2:2181,1.1.1.1:2181"
}
I can start a drillbit on both servers, and when I do status I get this response on both servers:
drillbit is running.
But when I then try to open the console via bin/drill-conf I get this stack trace:
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client. (state=,code=0)
java.sql.SQLException: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client.
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:159)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:64)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client.
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:208)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:151)
... 18 more
Caused by: java.io.IOException: Failure to connect to the zookeeper cluster service within the allotted time of 10000 milliseconds.
at org.apache.drill.exec.coord.zk.ZKClusterCoordinator.start(ZKClusterCoordinator.java:123)
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:206)
... 19 more
apache drill 1.7.0
"start your sql engine"
Why would drill fail to connect to the ZK cluster, which is running just fine?
All ports are open between these two boxes.
Pre-Requisites
Prerequisites for starting drill in distributed mode:
(Required) Running Oracle JDK version 7
(Required) Running a ZooKeeper quorum
(Recommended) Running a Hadoop cluster
(Recommended) Using DNS
Configuration
As your server IP address:
Server 1 - 1.1.1.1
Server 2 - 2.2.2.2
Put same configuration in zoo.cfg in both Server 1 and Server 2
clientPort=2181
server.1=1.1.1.1:2888:3888
server.2=2.2.2.2:2888:3888
Similarly same configuration in drill-override.conf for both the servers
drill.exec: {
cluster-id: "test",
zk.connect: "1.1.1.1:2181,2.2.2.2:2181"
}
Starting Drill
Start drillbit on all the cluster nodes using
bin/drillbit.sh start
Using Drill
Web UI:
Open web UI using any node address. For example:
1.1.1.1:8047
Via Shell:
Fire bin/drill-localhost command and drill shell will appear.
Verify Installation
From drill shell or UI fire
SELECT * FROM sys.drillbits;
Drill lists information about the Drillbits that are running
Stopping Drill
Fire command
bin/drillbit.sh stop
I have a problem with the pool connections of pax-jdbc in karaf, I'm trying to inject a Mysql DataSource (DS) through
blueprint.xml into my project, for test it, I have built a karaf command where injects the DS into karaf command class
and execute a query with that connection. That it's OK, but the problem is when I execute the command a lot times, for
each execution a new instance of the DS is created and the pool connection cannot open new connections to MySQL, because
the pool had reached the limit.
I have uploaded my code to github in this link: https://github.com/christmo/karaf-pax-jdbc , you can give a pull request
if you find an error in this project.
For test this project you can do:
1. Download karaf 4.0.4 or apache-karaf-4.1.0-SNAPSHOT
2. Copy the file karaf-pax-jdbc/etc/org.ops4j.datasource-my-ds.cfg to ${karaf}/etc, this file have the mysql
configuration change with your mysql configuration data.
4. Start mysql database engine
3. Start karaf -> cd ${karaf}/bin/; ./karaf
4. Add the repo of this project with this karaf command: feature:repo-add mvn:pax/features/1.0-SNAPSHOT/xml/features
5. Install the feature created for this project: feature:install mysql-test
6. Execute the command for test this problem: mysql-connection, this command only execute "Select 1" in mysql
If you execute 9 times this command "mysql-connection", it will freeze the prompt of karaf and if you interrupt the
execution you can get this exception:
java.sql.SQLException: Cannot get a connection, general error at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:146)
at com.twim.OrmCommand.execute(OrmCommand.java:53) at
org.apache.karaf.shell.impl.action.command.ActionCommand.execute(ActionCommand.java:83)
at
org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:67)
at
org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:87)
at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
at
org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108) at
org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182) at
org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119) at
org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
at
org.apache.karaf.shell.impl.console.ConsoleSessionImpl.run(ConsoleSessionImpl.java:270)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.lang.InterruptedException at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
... 12 more
The problem in your code is in the line System.out.println("--DS--: " + ds.getConnection());.
There you create a connection but never close it. So with every call you drain the pool.
After manually upgrading PHP and MySQL on my Cent OS box then logging in to my Plesk administration panel, I'm receiving the following error when attempting to carry out numerous tasks. PHP and MySQL are functioning correctly in their own right.
New files of configuration for Apache web server were not built due to errors in configuration templates: Template processing failed: file = /usr/local/psa/admin/conf/templates/default/domainVhost.php, error = [unixODBC][Driver Manager]Can't open lib '/usr/lib64/libmyodbc3.so' : /usr/lib64/libmyodbc3.so: cannot open shared object file: No such file or directory. The detailed error message was e-mailed to you, so please check the e-mail, fix the errors, and click here to retry generating broken configuration or click here to retry generating of all configurations.
I have located the ODBC configuration details in the following file:
/etc/odbcinst.ini
The contents of which are as follows:
# Driver from the MyODBC package
# Setup from the unixODBC package
[MySQL]
Description = MySQL driver for Plesk
Driver = /usr/lib64/libmyodbc3.so
Setup =
FileUsage = 1
Driver64 = /usr/lib64/libmyodbc3.so
Setup64 =
UsageCount = 1
I expect that the Driver & Driver64 parameters must be updated to reflect the changes that occurred when MySQL was upgraded however I am unable to locate the path to the appropriate driver and am out of my depth now. Any advice greatly welcomed.
UPDATE: a '# locate libmyodbc3' returns no results.
Turns out the ODBC and mysql-connector-odbc packages were missing.
They were installed using YUM as follows:
# yum install unixODBC
# yum mysql-connector-odbc packages