MiniDFSCluster gives ioexception - exception

Question

I am trying to test in hadoop. have the code as:
System.setProperty("test.build.data","/folder");
config = new Configuration();
cluster = new MiniDFSCluster(config,1,true,null);
but in new MiniDFSCluster(config,1,true,null), it throws exception:
java.io.IOException: Cannot run program "du": CreateProcess error=2, The system cannot find the file specified.
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DU.<init>(DU.java:53)
at org.apache.hadoop.fs.DU.<init>(DU.java:63)
at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:333)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:689)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:302)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
at ebay.Crawler.TestAll.testinit(TestAll.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified.
at java.lang.ProcessImpl.<init>(ProcessImpl.java:92)
at java.lang.ProcessImpl.start(ProcessImpl.java:41)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:463)
... 33 more
Could someone please give me some hint how to solve this?
Thank you very much.

Answer 1

Looks like the du command is not there on the system or is not in the PATH. If using Hadoop on Windows then Cygwin has to be installed. Anyway, which du will give the location of du binary.

Answer 2

I suspect you're using a Cloudera distribution of Hadoop. Version 1.0.0 of 'vanilla' Hadoop does work on Windows - at least create and writing to a file does.
If you need to run unit tests in a local Windows environment, try using Maven profile properties to set a version of 1.0.0 in you local Maven config, and in the POM specify the 'remote' config. The global setting will override the POM-specific one.
settings.xml
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<profiles>
<profile>
<id>windows</id>
<properties>
<hadoop.version>1.0.0</hadoop.version>
</properties>
</profile>
</profiles>
<activeProfiles>
<activeProfile>windows</activeProfile>
</activeProfiles>
</settings>
pom.xml
<properties>
<hadoop.version>0.20.2-cdh3u2</hadoop.version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoop.version}</version>
</dependency>