Hello everyone i want to write a hadoop program using a sequencefile input and output file. I tryed to set up job.setOutputFormatClass(SequenceFileOutputFormat.class) but it cause EOFException:
I used HVPI https://github.com/xmpy/hvpi with xuggler to break a video into frames. I want save the frames in a sequence file and save it in HDFS to use in another map reduce task.
Another way is write a hadoop custom output file but i don't know how do it.
I followed this tutorial http://hadooptutorial.info/hadoop-sequence-files-example/ but i got EOFException on
public void readFields(DataInput in) throws IOException {
int len = WritableUtils.readVInt(in);
byte[] temp = new byte[len];
in.readFully(temp, 0, len);
ByteArrayInputStream byteStream =
new ByteArrayInputStream(temp);
bufferedImage = ImageIO.read(byteStream);
}
16/09/17 17:05:55 INFO mapred.LocalJobRunner: reduce task executor complete.
16/09/17 17:05:55 WARN mapred.LocalJobRunner: job_local72686463_0001
java.lang.Exception: java.io.EOFException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at edu.bupt.videodatacenter.input.ImageWritable.readFields(ImageWritable.java:67)
...
More specifically in line in.readFully(temp, 0, len). This is from ImageWritable class. It's from HVPI. I don't made any change. I just used this.
The error was the custom type implementation. The write method on custom type was wrong. I just add out.write(byteOutputStream.toByteArray()) and now it's work fine.
Related
I am building a device for my research team. To briefly describe it, this device uses a motor and load sensor connected to an Arduino to apply a rotational force to a corn stalk and record the resistance of the stalk. We are in the process of building Bluetooth into the device. We are using this BT module.
We have a BLE GATT Service with 2 characteristics for storing DATA and 1 for holding the command which is an integer that will be read by the device and acted on. Reading the command characteristic is where we encounter our problem.
void get_input(){
uint16_t bufSize = 15;
char inputBuffer[bufSize];
bleParse = Adafruit_ATParser(); // Throws error: bleParse was not declared in this scope
bleParse.atcommandStrReply("AT+GATTCHAR=3",&inputBuffer,bufSize,1000);
Serial.print("input:");
Serial.println(inputBuffer);
}
The functions I am trying to use are found in the library for the module in Adarfruit_ATParser.cpp
/******************************************************************************/
/*!
#brief Constructor
*/
/******************************************************************************/
Adafruit_ATParser::Adafruit_ATParser(void)
{
_mode = BLUEFRUIT_MODE_COMMAND;
_verbose = false;
}
******************************************************************************/
/*!
#brief Send an AT command and get multiline string response into
user-provided buffer.
#param[in] cmd Command
#param[in] buf Provided buffer
#param[in] bufsize buffer size
#param[in] timeout timeout in milliseconds
*/
/******************************************************************************/
uint16_t Adafruit_ATParser::atcommandStrReply(const char cmd[], char* buf, uint16_t bufsize, uint16_t timeout)
{
uint16_t result_bytes;
uint8_t current_mode = _mode;
// switch mode if necessary to execute command
if ( current_mode == BLUEFRUIT_MODE_DATA ) setMode(BLUEFRUIT_MODE_COMMAND);
// Execute command with parameter and get response
println(cmd);
result_bytes = this->readline(buf, bufsize, timeout, true);
// switch back if necessary
if ( current_mode == BLUEFRUIT_MODE_DATA ) setMode(BLUEFRUIT_MODE_DATA);
return result_bytes;
}
None of the examples in the library use this. They all create their own parsers. For example, the neopixel_picker example sketch has a file called packetParser.cpp which I believe retrieves data from the BT module for that specific sketch, but it never includes or uses Adafruit_ATParser.. There are no examples of this constructor anywhere and I cannot figure out how to use it. I have tried these ways:
bleParse = Adafruit_ATParser();
Adafruit_ATParser bleParse();
Adafruit_ATParser();
ble.Adafruit_ATParser bleParse();
note: ble is an object that signifies a Serial connection between arduino and BT created with:
SoftwareSerial bluefruitSS = SoftwareSerial(BLUEFRUIT_SWUART_TXD_PIN, BLUEFRUIT_SWUART_RXD_PIN);
Adafruit_BluefruitLE_UART ble(bluefruitSS, BLUEFRUIT_UART_MODE_PIN,BLUEFRUIT_UART_CTS_PIN, BLUEFRUIT_UART_RTS_PIN);
Can anyone give me a clue on how to use the Adafruit_ATParser() constructor? Also, if the constructor has no reference to the ble object, how does it pass AT commands to the BT module?
I know this is a big ask, I appreciate any input you can give me.
Like this
Adafruit_ATParser bleParse;
You were closest with this one Adafruit_ATParser bleParse();. This is a common beginner mistake because it looks right. Unfortunately it declares a function bleParse which takes no arguments and returns a Adafruit_ATParser object.
I can't answer the second question.
EDIT
I've taken the time to have a look at the code. This is what I found
class Adafruit_BluefruitLE_UART : public Adafruit_BLE
{
and
class Adafruit_BLE : public Adafruit_ATParser
{
what this means is that the Adafruit_BluefruitLE_UART class is derived from the Adafruit_BLE class which in turn is derived from the Adafruit_ATParser class. Derivation means that any public methods in Adafruit_BLE can also be used on a Adafruit_BluefruitLE_UART object. You already have an Adafruit_BluefruitLE_UART object (you called it ble) so you can just use the method you want to use on that object.
SoftwareSerial bluefruitSS = SoftwareSerial(BLUEFRUIT_SWUART_TXD_PIN, BLUEFRUIT_SWUART_RXD_PIN);
Adafruit_BluefruitLE_UART ble(bluefruitSS, BLUEFRUIT_UART_MODE_PIN,BLUEFRUIT_UART_CTS_PIN, BLUEFRUIT_UART_RTS_PIN);
ble.atcommandStrReply( ... );
Any idea how I can add my output stream to the build config?
ConfigurationBuilder<BuiltConfiguration> builder =
ConfigurationBuilderFactory.newConfigurationBuilder();
AppenderComponentBuilder osAppender = builder.newAppender("os", "OutputStream");
osAppender.addAttribute("target", myStream);
builder.add(osAppender);
BuiltConfiguration config = builder.build();
Configurator.initialize(config);
This is the Error message I get:
2022-01-27 15:04:41,203 main ERROR OutputStream contains an invalid element or attribute "target"
2022-01-27 15:04:41,227 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.OutputStreamAppender for element OutputStream: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.appender.OutputStreamAppender.getManager(OutputStreamAppender.java:159)
Thanks
The ConfigurationBuilder API does not allow you to set attributes which can not be serialized to a String. Therefore you'll need to use OutputSreamAppender's builder directly:
final LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
final Configuration config = ctx.getConfiguration();
final Appender appender = OutputStreamAppender.newBuilder()//
.setTarget(myStream)
.setConfiguration(ctx.getConfiguration())
.build();
config.addLoggerAppender(ctx.getRootLogger(), appender);
See this question for another example of ConfigurationBuilder API vs direct instantiation of Log4j components.
Check also Log4j's architecture, which explains how all these components work together.
I have a function which is running in the separate thread. The code which calls this function not waits for result of it.
def sendEmail(email: String): Future[Unit] = {
...
}
def registration: Future[User] = {
...
// I do not want to wait for result of this function, just fire email sending
// in seprate thread and continue
sendEmail(email)
...
// Do another job
}
The problem is that if something went wrong in sendEmail function, I want to see this exception in log file.
Now log file and console output are empty if some exception is thrown there.
Is there a way to log exceptions from that separate thread?
P.S.: I do not want to log exception manually in sendEmail, but force play framework to log it.
In general, you wrap exceptions in the exceptionally block.
In java, it's like :
foobar.thenComposeAsync(arg -> {
sendEmail();
}).exceptionally(throwable -> {
// Do logging
});
I want to find length of given string with out using pub.string.length (built in function) in web methods .
Ex:If we give name "abcdef" i want result like 6 .
You could write your own java service, but that sounds redundant to me.
Here's an (as yet untested) code sample:
public static final void checkStringSize(IData pipeline)
throws ServiceException {
// pipeline
IDataCursor pipelineCursor = pipeline.getCursor();
String inputString = IDataUtil.getString( pipelineCursor, "inputString" );
pipelineCursor.destroy();
long length = -1;
length = inputString.length();
// pipeline
IDataCursor pipelineCursor_1 = pipeline.getCursor();
IDataUtil.put( pipelineCursor_1, "length", ""+length );
pipelineCursor_1.destroy();
}
You could manually count the length of the string by using pub.string:substring until it gets an error.
I wouldn't recommend this - it's inelegant, possibly quite slow and it still uses a function from pub.string, which you appear to be avoiding.
Anyway, here's a way to do it, click the link to see the image.
webMethods code sample - https://i.stack.imgur.com/FE9oU.jpg
Just make sure the input string cannot be null - otherwise the substring won't throw an error and it will have an infinite loop.
There is a web-service deployed on tomcat 6 and exposed via apache-cxf 2.3.3. A generated sources stubs using wsdl2java to be able to call this service.
Things seemed fine until I sent big request(~1Mb). This request wasn't processed and failing with exception:
Interceptor for {http://localhost/}ResourceAllocationServiceSoapService has thrown
exception, unwinding now org.apache.cxf.binding.soap.SoapFault:
Error reading XMLStreamReader.
...
com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
Is some kind of max request length here, I'm totally stuck with it.
Vladimir's suggestion worked. This code below will help others with understanding where to put the 1000000.
public void handleMessage(SoapMessage message) throws Fault {
// Get message content for dirty editing...
InputStream inputStream = message.getContent(InputStream.class);
if (inputStream != null)
{
String processedSoapEnv = "";
// Cache InputStream so it can be read independently
CachedOutputStream cachedInputStream = new CachedOutputStream(1000000);
try {
IOUtils.copy(inputStream,cachedInputStream);
inputStream.close();
cachedInputStream.close();
InputStream tmpInputStream = cachedInputStream.getInputStream();
try{
String inputBuffer = "";
int data;
while((data = tmpInputStream.read()) != -1){
byte x = (byte)data;
inputBuffer += (char)x;
}
/**
* At this point you can choose to reformat the SOAP
* envelope or simply view it just make sure you put
* an InputStream back when you done (see below)
* otherwise CXF will complain.
*/
processedSoapEnv = fixSoapEnvelope(inputBuffer);
}
catch(IOException e){
}
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// Re-set the SOAP InputStream with the new envelope
message.setContent(InputStream.class,new ByteArrayInputStream( processedSoapEnv.getBytes()));
/**
* If you just want to read the InputStream and not
* modify it then you just need to put it back where
* it was using the CXF cached inputstream
*
* message.setContent(InputStream.class,cachedInputStream.getInputStream());
*/
}
}
I figured out what was wrong. Actually it was bug inside interceptor's code:
CachedOutputStream requestStream = new CachedOutputStream()
When I replaced this with
CachedOutputStream requestStream = new CachedOutputStream(1000000);
things start working fine.
So the request was just trunkated during copying of streams.
I run into same issue of geting "com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog" when using CachedOutputStream class.
Looking at sources of CachedOutputStream class the threshold is used to switch between storing stream's data from "in memory" to "a file".
Assuming stream operates on data that exceeds threshold it gets stored in a file thus following code is going to break
IOUtils.copy(inputStream,cachedInputStream);
inputStream.close();
cachedInputStream.close(); //closes the stream, the file on disk gets deleted
InputStream tmpInputStream = cachedInputStream.getInputStream(); //returned tmpInputStream is brand *empty* one
// ... reading tmpInputStream here will produce WstxEOFException
Increasing 'threshold' does help as all stream data is stored into memory and in such scenario calling cachedInputStream.close() does not really close the underlying stream implementation so one can still read from it later on.
Here is 'fixed' version of above code (at least it worked without exception for me)
IOUtils.copy(inputStream,cachedInputStream);
inputStream.close();
InputStream tmpInputStream = cachedInputStream.getInputStream();
cachedInputStream.close();
// reading from tmpInputStream here works fine
Temporary file gets deleted when close() is called on tmpInputStream and there are no more other references to it, see source code of CachedOutputStream.maybeDeleteTempFile()