I am having a problem in executing a script in Apache Pig. I have 3 files namely movies.csv, ratings.csv, tags.csv. First I want to load "movies.csv", then load "ratings.csv" and join both tables. But I am encountering an error in loading of the files. Code given by me is as follows,
register 'piggybank-0.15.0.jar'
DEFINE CSVLoader org.apache.pig.piggybank.storage.CSVLoader();
part1 = LOAD '/home/cloudera/ml-20m/movies' as (movieId: chararray, title: chararray, genre: chararray);
cat part1;
When I give "cat" command, I am getting an error,as
Pig Stack Trace
ERROR 2997: Encountered IOException. Directory part1 does not exist.
java.io.IOException: Directory part1 does not exist.
at org.apache.pig.tools.grunt.GruntParser.processCat(GruntParser.java:677)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:547)
at org.apache.pig.Main.main(Main.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
But I have the file at specified location. I don't know why pig is not able to recognize the input file. I have tried by placing the input file in hdfs and loading the file. But the error is same though. Can anyone please help me. Thanks in advance.
part1 is not a file but a relation.When you use the LOAD command in Pig,you are instructing to load the contents of the file into a relation.You cannot use cat on a relation since the most common use of cat is to read the contents of files.
To display the content of part1 use
DUMP part1;
else if you insist on using cat, then specify the full path to the file
cat /home/cloudera/ml-20m/movies;
Related
I tried converting my .csv file to .dat format and tried to load the file into Octave. It throws an error:
unable to find file filename
I also tried to load the file in .csv format using the syntax
x = csvread(filename)
and it throws the error:
'filename' undefined near line 1 column 13.
I also tried loading the file by opening it on the editor and I tried loading it and now it shows me
warning: load: 'filepath' found by searching load path
error: load: unable to determine file format of 'Salary_Data.dat'.
How can I load my data?
>> load Salary_Data.dat
error: load: unable to find file Salary_Data.dat
>> Salary_Data
error: 'Salary_Data' undefined near line 1 column 1
>> Salary_Data
error: 'Salary_Data' undefined near line 1 column 1
>> Salary_Data
error: 'Salary_Data' undefined near line 1 column 1
>> x = csvread(Salary_Data)
error: 'Salary_Data' undefined near line 1 column 13
>> x = csvread(Salary_Data.csv)
error: 'Salary_Data' undefined near line 1 column 13
>> load Salary_Data.dat
warning: load: 'C:/Users/vaith/Desktop\Salary_Data.dat' found by searching load path
error: load: unable to determine file format of 'Salary_Data.dat'
>> load Salary_Data.csv
warning: load: 'C:/Users/vaith/Desktop\Salary_Data.csv' found by searching load path
error: load: unable to determine file format of 'Salary_Data.csv'
Salary_Data.csv
YearsExperience,Salary
1.1,39343.00
1.3,46205.00
1.5,37731.00
2.0,43525.00
2.2,39891.00
2.9,56642.00
3.0,60150.00
3.2,54445.00
3.2,64445.00
3.7,57189.00
3.9,63218.00
4.0,55794.00
4.0,56957.00
4.1,57081.00
4.5,61111.00
4.9,67938.00
5.1,66029.00
5.3,83088.00
5.9,81363.00
6.0,93940.00
6.8,91738.00
7.1,98273.00
7.9,101302.00
8.2,113812.00
8.7,109431.00
9.0,105582.00
9.5,116969.00
9.6,112635.00
10.3,122391.00
10.5,121872.00
Ok, you've stumbled through a whole pile of issues here.
It would help if you didn't give us error messages without the commands that produced them.
The first message means you were telling Octave to open something called filename and it couldn't find anything called filename. Did you define the variable filename? Your second command and the error message suggests you didn't.
Do you know what Octave's working directory is? Is it the same as where the file is located? From the response to your load commands, I'd guess not. The file is located at C:/Users/vaith/Desktop. Octave's working directory is probably somewhere else.
(Try the pwd command and see what it tells you. Use the file browser or the cd command to navigate to the same location as the file. help pwd and help cd commands would also provide useful information.)
The load command, used as a command (load file.txt) can take an input that is or isn't defined as a string. A function format (load('file.txt') or csvread('file.txt')) must be a string input, hence the quotes around file.txt. So all of your csvread input commands thought you were giving it variable names, not filenames.
Last, the fact that load couldn't read your data isn't overly surprising. Octave is trying to guess what kind of file it is and how to load it. I assume you tried help load to see what the different command options are? You can give it different options to help Octave figure it out. If it actually is a csv file though, and is all numbers not text, then csvread might still be your best option if you use it correctly. help csvread would be good information for you.
It looks from your data like you have a header line that is probably confusing the load command. For data that simply formatted, the csvread command can bring in the data. It will replace your header text with zeros.
So, first, navigate to the location of the file:
>> cd C:/Users/vaith/Desktop
then open the file:
>> mydata = csvread('Salary_Data.csv')
mydata =
0.00000 0.00000
1.10000 39343.00000
1.30000 46205.00000
1.50000 37731.00000
2.00000 43525.00000
...
If you plan to reuse the filename, you can assign it to a variable, then open the file:
>> myfile = 'Salary_Data.csv'
myfile = Salary_Data.csv
>> mydata = csvread(myfile)
mydata =
0.00000 0.00000
1.10000 39343.00000
1.30000 46205.00000
1.50000 37731.00000
2.00000 43525.00000
...
Notice how the filename is stored and used as a string with quotation marks, but the variable name is not. Also, csvread converted non-numeric header data to 'zeros'. The help for csvread and dlmread show you how to change it to something other than zero, or to skip a certain number of rows. If you want to preserve the text, you'll have to use some other input function.
I'm getting this error on Arduino(1.8.9).
usage: esptool write_flash [-h] [--erase-all]
[--flash_freq {keep,40m,26m,20m,80m}]
[--flash_mode {keep,qio,qout,dio,dout}]
[--flash_size FLASH_SIZE]
[--spi-connection SPI_CONNECTION] [--no-progress]
[--verify] [--compress | --no-compress]
<address> <filename> [<address> <filename> ...]
esptool write_flash: error: argument <address>
<filename>: [Errno 2] No such file or directory:'/home/USER/.arduino15/packages/esp32/hardware/esp32/1.0.2/tools/partitions/boot_app0.bin'
esptool write_flash: error: argument <address> <filename>: [Errno 2] No such file or directory: '/home/USER/.arduino15/packages/esp32/hardware/esp32/1.0.2/tools/partitions/boot_app0.bin'
Although boot_app0.bin file is present:
image link
To fix this problem you can try to edit the platform.txt file which you can find in your esp package directory.
So, you need to replace inner section with code written below:
## Combine gc-sections, archives, and objects
recipe.c.combine.pattern={recipe.hooks.linking.prelink.1.pattern} & "{compiler.path}{compiler.c.elf.cmd}" {build.exception_flags} -Wl,-Map "-Wl,{build.path}/{build.project_name}.map" {compiler.c.elf.flags} {compiler.c.elf.extra_flags} -o "{build.path}/{build.project_name}.elf" -Wl,--start-group {object_files} "{archive_file_path}" {compiler.c.elf.libs} -Wl,--end-group "-L{build.path}" & {recipe.objcopy.hex.1.pattern}
The key thing here is "{recipe.hooks.linking.prelink.1.pattern} &" at start and "& {recipe.objcopy.hex.1.pattern}" at the end. The text between is the part of the platform.txt file, that you dont need to change.
The above is true for OS Windows. In OS Linux set {recipe.hooks.linking.prelink.1.pattern} ;" and "; {recipe.objcopy.hex.1.pattern}".
Its solve my problem for ESP8266, I hope its will be helpful for you too.
Reference link https://www.eclipse.org/forums/index.php/t/1095090/
if you are using esptool. Make sure you actually download the files from github using the download button instead of using the save as option when u right click on the file name.
I am trying to run ConnectedThresholdImageFilter example in ITK mentioned "https://itk.org/Doxygen45/html/Segmentation_2ConnectedThresholdImageFilter_8cxx-example.html" here.
But it is showing the following error.
itk::ImageFileWriterException (0x24cb740)
Location: "void itk::ImageFileWriter::Write() [with
TInputImage = itk::Image]" File:
/usr/local/include/ITK-4.13/itkImageFileWriter.hxx Line: 151
Description: Could not create IO object for writing file output
Tried to create one of the following:
BMPImageIO
BioRadImageIO
Bruker2dseqImageIO
GDCMImageIO
GE4ImageIO
GE5ImageIO
GiplImageIO
HDF5ImageIO
JPEGImageIO
LSMImageIO
MINCImageIO
MRCImageIO
MetaImageIO
NiftiImageIO
NrrdImageIO
PNGImageIO
StimulateImageIO
TIFFImageIO
VTKImageIO You probably failed to set a file suffix, or
set the suffix to an unsupported type
I didn't do any changes in code. And I am trying to use dicom image as an input.
It is either
You set the output file name with an unsupported extension.
There is something wrong about how you compiled/linked ITK or how you are linking your example to ITK.
I'm trying to import an edge list using the csv importer, but I get an error message due to what I think is a failure to escape a character. Is there a way around this, or do I have to run some script on the csv files to handle this?
Exception in thread "main" java.lang.NumberFormatException: For input string: "'Abelmoschus esculentus' bunchy to
phytoplasma"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at org.neo4j.batchimport.Importer.id(Importer.java:213)
at org.neo4j.batchimport.Importer.id(Importer.java:181)
at org.neo4j.batchimport.Importer.importRelationships(Importer.java:147)
at org.neo4j.batchimport.Importer.doImport(Importer.java:232)
at org.neo4j.batchimport.Importer.main(Importer.java:83)
use the config option: batch_import.csv.quotes=true
I'm running Pig
example$ pig --version
Apache Pig version 0.8.1-cdh3u1 (rexported)
compiled Jul 18 2011, 08:29:40
on very simple dataset
example$ hadoop fs -cat /user/pavel/trivial.log
1 one
2 two
3 three
I'm trying to save the bag format as json by using the following script:
REGISTER ./pig.jar;
A = LOAD 'trivial.log' USING PigStorage('\t') AS (mynum: int, mynumstr: chararray);
B = GROUP A BY mynum;
DUMP B;
STORE B into 'trivial_json.out' USING JsonStorage();
and I get an error:
Backend error message
---------------------
java.lang.NullPointerException
at org.apache.pig.ResourceSchema.<init>(ResourceSchema.java:239)
at org.apache.pig.builtin.JsonStorage.prepareToWrite(JsonStorage.java:129)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.<init>(PigOutputFormat.java:124)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:85)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:559)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error: java.lang.NullPointerException
org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: java.lang.NullPointerException
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:154)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:382)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1209)
at org.apache.pig.PigServer.execute(PigServer.java:1201)
at org.apache.pig.PigServer.access$100(PigServer.java:129)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1528)
at org.apache.pig.PigServer.executeBatchEx(PigServer.java:373)
at org.apache.pig.PigServer.executeBatch(PigServer.java:340)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:115)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:172)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
================================================================================
I'm not strong enough in Java to debug in minutes, can somebody suggest what might be going on?
Thanks much!
-Pavel
Got some questions regarding this by email, thought this might be useful for others:
It turned out that the JsonStorage class wasn't included in my Pig installation. In fact it's not even out in a stable branch yet, you won't find it in 0.9.2. But if you get the latest trunk
http://svn.apache.org/repos/asf/pig/trunk/
then trunk/test/org/apache/pig/test/TestJsonLoaderStorage.java shows how it works. If you are not averse to unreleased versions then you could try it. If you go this way you should also take a look at Avro (binary format with json metadata). It's not officially out either.
I was trying to run a streaming python job and was trying to avoid manually specifying the schema. I ended up passing Pig bags and parsing them by hand.
Hope this helps.
-Pavel