Following the question in this link, there is another question about the creating the directory on Hadoop HDFS.
I am new to Hadoop/Flume and I have picked up a project which use Flume to save csv data into HDFS. The setting for the Flume sink will be as follows:
contract-snapshot.sinks.hdfs-sink-contract-snapshot.hdfs.path = /dev/wimp/contract-snapshot/year=%Y/month=%n/day=%e/snapshottime=%k%M
With this Flume setting, the corresponding csv file will be saved into the HDFS, under the folder:
"/wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/contract-snapshot.1591779548475.csv"
I am trying to setup the whole system locally, I have hadoop installed locally on my windows pc, how can I create a directory of "/wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/" on the local hdfs?
In the cmd terminal, the code:
hadoop fs -mkdir /wimp/contract-snapshot
can create a folder /wimp/contract-snapshot. However the following code does not work in the cmd terminal
hadoop fs -mkdir /wimp/contract-snapshot/year=2020
How to create hdfs directory by year, month, day?
hadoop fs -mkdir "/wimp/contract-snapshot/year=2020"
Adding quotation solves the problem.
Related
I am working on hadoop and facing one issue. i have uploaded file in hadoop files view which is .sql file. and im validating its here in hadoop at mysqluploads/sqoop-data-setup.sql
I have tested it, by command
hadoop fs -ls /mysqluploads/sqoop-data-setup.sql hadoop fs -cat /mysqluploads/sqoop-data-setup.sql
now issue is, i want to import this data into mysql database; I am typing after logon in mysql source /mysqluploads/sqoop-data-setup.sql
It is showing me the following error
ERROR: Failed to open file 'root\mysqluploads\sqoop-data-setup.sql', error: 2
I am new to all this as I am only in my second semester and I just need help understanding a command I need to do. I am trying to load a local csv file to hdfs on cloudera using the terminal. I have to use that data and work with Pig for an assignment. I have tried everything and it still gives me 'no such file or directory'. I have turned off safe mode, checked the directories and even made sure the file could be read. Here are the commands I have tried to load the data:
hadoop fs -copyFromLocal 2008.csv
hdfs dfs -copyFromLocal 2008.csv
hdfs dfs -copyFromLocal 2008.csv /user/root
hdfs dfs -copyFromLocal 2008.csv /home/cloudera/Desktop
Nothing at all has worked and keeps giving me
'2008.csv' no such file or directory
. What could I do to fix this? Thank you very much.
I have to use that data and work with Pig for an assignment
You can run Pig without HDFS.
pig -x local
I have tried everything and it still gives me 'no such file or directory'
Well, that error is not from HDFS, it seems to be from your local shell.
ls shows you the files available to use in the current directory for -copyFromLocal or -put to work without an absolute path.
For complete assurance for what you are copying, as well as to where, use full paths in both arguments. The second path is always HDFS if using those two flags.
Try this
hadoop fs -mkdir -p /user/cloudera # just in case
hadoop fs -copyFromLocal ./2008.csv /user/cloudera/
Or even
hadoop fs -copyFromLocal /home/cloudera/Desktop/2008.csv /user/cloudera/
What I think you are having issues with, is that /user/root is not correct unless you are running commands as the root user, and neither is /home/cloudera/Desktop because HDFS has no concept of a Desktop.
The default behavior without the second path is
hadoop fs -copyFromLocal <file> /user/$(whoami)/
(Without the trailing slash, or a pre-existing directory, it'll copy <file> literally as a file, which can be unexpected in certain situations, for example, when trying to copy a file into a user directory, but the directory doesn't exist yet)
I believe you already check and made yourself sure that 2008.csv exists. That's why I think the permissions on this file not allowing you to copy it.
try: sudo -u hdfs cat 2008.csv
If you get permission denied error, this is your issue. Arrange permissions of the file or create a new one if so. If again you get "no file" error, try to use whole path for the file like:
hdfs dfs -copyFromLocal /user/home/csvFiles/2008.csv /user/home/cloudera/Desktop
I have graph.db folder from neo4j. It contains lot of neostore*.* files. How do i export a csv file from this ?
Note: I have this graph.db sent from my friend.
Download and install Neo4j if you haven't already
Move the graph.db directory that you have now into the data/ directory of the fresh Neo4j installation, replacing the existing graph.db directory in the fresh Neo4j instance. (Note: If you are using the desktop Neo4j application you can simply choose the location of your existing graph.db directory when starting Neo4j).
Start Neo4j server
To generate CSVs you have a few options:
Export from Neo4j Browser With Neo4j running, open your web browser and navigate to http://localhost:7474. Execute a Cypher query. Click on the "Download" icon and choose "Export CSV" to download a CSV representation of the data returned. See screenshot below.
neo4j-shell-tools Use neo4j-shell-tools to export results of a Cypher query. Use -o file.csv to specify output should be written to CSV file.
See this blog post for more info.
I am new with Hadoop, I have a file to import into hadoop via command line (I access the machine through SSH)
How can I import the file in hadoop?
How can I check afterward (command)?
2 steps to import csv file
move csv file to hadoop sanbox (/home/username) using winscp or cyberduck.
use -put command to move file from local location to hdfs.
hdfs dfs -put /home/username/file.csv /user/data/file.csv
There are three flags that we can use for load data from local machine into HDFS,
-copyFromLocal
We use this flag to copy data from the local file system to the Hadoop directory.
hdfs dfs –copyFromLocal /home/username/file.csv /user/data/file.csv
If the folder is not created as HDFS or root user we can create the folder:
hdfs dfs -mkdir /user/data
-put
As #Sam mentioned in the above answer we also use -put flag to copy data from the local file system to the Hadoop directory.
hdfs dfs -put /home/username/file.csv /user/data/file.csv
-moveFromLocal
we also use -moveFromLocal flag to copy data from the local file system to the Hadoop directory. But this will remove the file from the local directory
hdfs dfs -moveFromLocal /home/username/file.csv /user/data/file.csv
I'm trying to import a csv file from the command line after connecting to my RDS database. Unfortunately I'm fairly new to the mysql command and file navigation commands. I'm looking to navigate to my directory where the csv I'm importing is located. This is the directory path ~/Desktop/images.csv. I know I use mysqlimport, but can't figure out the command to change directory