I created a hadoop multinode cluster and my target is to know processor information of all nodes in the cluster by running jar file in master.I have the code to know processor info in a node.How can i run this code in all nodes and retrieve the processor information using java.
thanks in advance for help..
Driver code:
Here i have used only mapper class
public class ProcInfoCluster extends Configured implements Tool {
#Override
public int run(String[] arg0) throws Exception {
// TODO Auto-generated method stub
if(arg0.length<2)
{
System.out.println("please provide the output file paths");
return -1;
}
JobConf conf=new JobConf(ProcInfoCluster.class);
conf.setJobName("ProcesorInfo");
FileInputFormat.setInputPaths(conf, new Path(arg0[0]));
FileOutputFormat.setOutputPath(conf,new Path(arg0[1]));
conf.setMapperClass(ProcInfoMapper.class);
//conf.setReducerClass(WordReducer.class);
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(Text.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);
JobClient.runJob(conf);
return 0;
}
public static void main(String args[])
throws Exception
{
int exitCode=ToolRunner.run(new ProcInfoCluster(), args);
System.exit(exitCode);
}
}
Mapper code:
Here in this mapper code i have written code to know processor information and tried to send it as key value pairs
public class ProcInfoMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text,Text>
{
#Override
//map function to retrieve processor information of all nodes
public void map(LongWritable key, Text value,
OutputCollector output, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
String no=System.getenv("PROCESSOR_ARCHITEW6432");
//to know processor info
System.out.println(no);
String s=new String("ProcInfo");
output.collect(new Text(s), new Text(no));
}
}
and i had a doubt that whether we can write a map reduce application only with mapper class without the reducer class
Related
I'm trying to write a unit test for an MDB. The goal of my test is to make sure that the logic in the MDB can identify the correct type of object in the ObjectMessage and process it. However, I can't figure out how to make an ObjectMessage so I can test it. I keep getting null pointer exceptions.
Here is my unit test:
/**
* Test of the logic in the MDB
*/
#RunWith(JMockit.class)
#ExtendWith(TimingExtension.class)
class MDBTest
{
protected MyMDB mdb;
#BeforeEach
public void setup() throws NamingException, CreateHeaderException, DatatypeConfigurationException, PropertiesDataException
{
mdb = new MyMDB();
}
/**
* Test the processing of the messages by the MDB
*/
#Test
void testReceivingMessage() throws JMSException, IOException
{
MyFirstObject testMsg = getTestMessage();
ObjectMessage msg = null;
Session session = null;
new MockUp<ObjectMessage>()
{
#Mock
public void $init()
{
}
#Mock
public Serializable getObject()
{
return testMsg;
}
};
new MockUp<Session>()
{
#Mock
public void $init()
{
}
#Mock
public ObjectMessage createObjectMessage(Serializable object)
{
return msg;
}
};
// !!!! Null pointer here on Session !!!!
ObjectMessage msgToSend = session.createObjectMessage(testMsg);
mdb.onMessage(msgToSend);
assertEquals(1, mdb.getNumMyFirstObjectMsgs());
}
/**
* Create a Test Message
*
* #return the test message
* #throws IOException
*/
protected MyFirstObject getTestMessage) throws IOException
{
MyFirstObject myObj = new MyFirstObject();
myObj.id = 0123;
myObj.description = "TestMessage";
return myObj;
}
}
I feel like I should be able to initialize Session somehow, but I need to do it without using an additional library like Mockrunner.
Any suggestions?
I would try to address this in a different style. Provide a mock client, that will just mock the right API.
We should mock only a set of functions required for message retrieval and processing but that means we might have to provide a custom implementation for some of the APIs available in the EJB/JMS library. The mock client will have a function to push messages on a given topic/queue/channel, message can be simple String.
A simple implementation might look like this, in this other methods have been omitted for simplicity.
// JMSClientImpl is an implementation of Connection interface.
public class MyJmsTestClient extends JMSClientImpl{
Map<String, String> channelToMessage = new ConcurrentHashMap<>();
public Map<String, String> getMessageMap(){
return channelToMessage;
}
public void enqueMessage(String channel, String message){
channelToMessage.put(channe, message);
}
#Override
public Session createSession(){
return new MyTestSession(this);
}
}
// A class that implements some of the methods from session interface
public MyTestSession extends SessionImpl{
private MyJmsTestClient jmsClient;
MyTestSession(MyJmsTestClient jmsClient){
this.jmsClient = jmsClient;
}
// override methods that fetches messages from remote JMS
// Here you can just return messages from MyJmsTestClient
// override other necessary methods like ack/nack etc
MessageConsumer createConsumer(Destination destination) throws JMSException{
// returns a test consume
}
}
A class that implements methods from MessageConsumer interface
class TestMessageConsumer extends MessageConsumerImpl {
private MyJmsTestClient jmsClient;
private Destination destination;
TestMessageConsumer(MyJmsTestClient jmsClient, Destination destination){
this.jmsClient = jmsClient;
this.destination = destination;
}
Message receive() throws JMSException{
//return message from client
}
}
There's no straight forward, you can see if there're any library that can provide you embedded JMS client feature.
I am new to flink i am trying write junit test cases to test KeyedBroadCastProcessFunction. Below is my code ,i am currently calling the getDataStreamOutput method in TestUtils class and passing inputdata and patternrules to method once the input data is evaluated against list of pattern rules and if input data satisfy the condition i will get the signal and calling sink function and returning output data as string in getDataStreamOutput method
#Test
public void testCompareInputAndOutputDataForInputSignal() throws Exception {
Assertions.assertEquals(sampleInputSignal,
TestUtils.getDataStreamOutput(
inputSignal,
patternRules));
}
public static String getDataStreamOutput(JSONObject input, Map<String, String> patternRules) throws Exception {
env.setParallelism(1);
DataStream<JSONObject> inputSignal = env.fromElements(input);
DataStream<Map<String, String>> rawPatternStream =
env.fromElements(patternRules);
//Generate a key,value pair of set of patterns where key is pattern name and value is pattern condition
DataStream<Tuple2<String, Map<String, String>>> patternRuleStream =
rawPatternStream.flatMap(new FlatMapFunction<Map<String, String>,
Tuple2<String, Map<String, String>>>() {
#Override
public void flatMap(Map<String, String> patternRules,
Collector<Tuple2<String, Map<String, String>>> out) throws Exception {
for (Map.Entry<String, String> stringEntry : patternRules.entrySet()) {
JSONObject jsonObject = new JSONObject(stringEntry.getValue());
Map<String, String> map = new HashMap<>();
for (String key : jsonObject.keySet()) {
String value = jsonObject.get(key).toString();
map.put(key, value);
}
out.collect(new Tuple2<>(stringEntry.getKey(), map));
}
}
});
BroadcastStream<Tuple2<String, Map<String, String>>> patternRuleBroadcast =
patternStream.broadcast(patternRuleDescriptor);
DataStream<Tuple2<String, JSONObject>> validSignal = inputSignal.map(new MapFunction<JSONObject,
Tuple2<String, JSONObject>>() {
#Override
public Tuple2<String, JSONObject> map(JSONObject inputSignal) throws Exception {
String source =
inputSignal.getSource();
return new Tuple2<>(source, inputSignal);
}
}).keyBy(0).connect(patternRuleBroadcast).process(new MyKeyedBroadCastProcessFunction());
validSignal.map(new MapFunction<Tuple2<String, JSONObject>,
JSONObject>() {
#Override
public JSONObject map(Tuple2<String, JSONObject> inputSignal) throws Exception {
return inputSignal.f1;
}
}).addSink(new getDataStreamOutput());
env.execute("TestFlink");
}
return (getDataStreamOutput.dataStreamOutput);
}
#SuppressWarnings("serial")
public static final class getDataStreamOutput implements SinkFunction<JSONObject> {
public static String dataStreamOutput;
public void invoke(JSONObject inputSignal) throws Exception {
dataStreamOutput = inputSignal.toString();
}
}
I need to test different inputs with same broadcast rules but each time i am calling this function its again and again doing process from beginning take input signal broadcast data, is there a way i can broadcast once and keeping on sending the input to the method i explored i can use CoFlatMapFunction something like below to combine datastream and keep on sending the input rules while method is running but for this one of the datastream has to keep on getting data from kafka topic again it will overburden on method to load kafka utils and server
DataStream<JSONObject> inputSignalFromKafka = env.addSource(inputSignalKafka);
DataStream<org.json.JSONObject> inputSignalFromMethod = env.fromElements(inputSignal));
DataStream<JSONObject> inputSignal = inputSignalFromMethod.connect(inputSignalFromKafka)
.flatMap(new SignalCoFlatMapper());
public static class SignalCoFlatMapper
implements CoFlatMapFunction<JSONObject, JSONObject, JSONObject> {
#Override
public void flatMap1(JSONObject inputValue, Collector<JSONObject> out) throws Exception {
out.collect(inputValue);
}
#Override
public void flatMap2(JSONObject kafkaValue, Collector<JSONObject> out) throws Exception {
out.collect(kafkaValue);
}
}
I found a link in stackoverflow How to unit test BroadcastProcessFunction in flink when processElement depends on broadcasted data but this is confused me a lot
Any way i can only broadcast only once in Before method in test cases and keeping sending different kind of data to my broadcast function
You can use KeyedTwoInputStreamOperatorTestHarness in order to achieve this for example let's assume you have the following KeyedBroadcastProcessFunction where you define some business logic for both DataStream channels
public class SimpleKeyedBroadcastProcessFunction extends KeyedBroadcastProcessFunction<String, String, String, String> {
#Override
public void processElement(String inputEntry,
ReadOnlyContext readOnlyContext, Collector<String> collector) throws Exception {
//business logic for how you want to process your data stream records
}
#Override
public void processBroadcastElement(String broadcastInput, Context
context, Collector<String> collector) throws Exception {
//process input from your broadcast channel
}
Let's now assume your process function is stateful and is making modifications to the Flink internal state, you would have to create a TestHarness inside your test class to ensure you are able to keep track of the state during testing.
I would then create some unit tests using the following approach:
public class SimpleKeyedBroadcastProcessFunctionTest {
private SimpleKeyedBroadcastProcessFunction processFunction;
private KeyedTwoInputStreamOperatorTestHarness<String, String, String, String> testHarness;
#Before
public void setup() throws Exception {
processFunction = new SimpleKeyedBroadcastProcessFunction();
testHarness = new KeyedTwoInputStreamOperatorTestHarness<>(
new CoBroadcastWithKeyedOperator<>(processFunction, ImmutableList.of(BROADCAST_MAP_STATE_DESCRIPTOR)),
(KeySelector<String, String>) string -> string ,
(KeySelector<String, String>) string -> string,
TypeInformation.of(String.class));
testHarness.setup();
testHarness.open();
}
#After
public void cleanup() throws Exception {
testHarness.close();
}
#Test
public void testProcessRegularInput() throws Exception {
//processElement1 send elements into your regular stream, second param will be the event time of the record
testHarness.processElement1(new StreamRecord<>("Hello", 0));
//Access records collected during processElement
List<StreamRecord<? extends String>> records = testHarness.extractOutputStreamRecords();
assertEquals("Hello", records.get(0).getValue())
}
#Test
public void testProcessBroadcastInput() throws Exception {
//processElement2 send elements into your broadcast stream, second param will be the event time of the record
testHarness.processElement2(new StreamRecord<>("Hello from Broadcast", 0));
//Access records collected during processElement
List<StreamRecord<? extends String>> records = testHarness.extractOutputStreamRecords();
assertEquals("Hello from Broadcast", records.get(0).getValue())
}
}
I have CSV files without headers. Since I'm using 'useMaps' I want to specify the headers dynamically. If I set headers statically and then use in route it works fine as below Approach 1 -
#Component
public class BulkActionRoutes extends RouteBuilder {
#Override
public void configure() throws Exception {
CsvDataFormat csv = new CsvDataFormat(",");
csv.setUseMaps(true);
ArrayList<String> list = new ArrayList<String>();
list.add("DeviceName");
list.add("Brand");
list.add("status");
list.add("type");
list.add("features_c");
list.add("battery_c");
list.add("colors");
csv.setHeader(list);
from("direct:bulkImport")
.convertBodyTo(String.class)
.unmarshal(csv)
.split(body()).streaming()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
GenericObjectModel model = null;
HashMap<String, String> csvRecord = (HashMap<String, String>)exchange.getIn().getBody();
}
});
}
}
However, if the list is passed via Camel headers as below then it does not work Approach 2 -
#Component
public class BulkActionRoutes extends RouteBuilder {
#Override
public void configure() throws Exception {
CsvDataFormat csv = new CsvDataFormat(",");
csv.setUseMaps(true);
from("direct:bulkImport")
.convertBodyTo(String.class)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
ArrayList<String> fileHeaders = (ArrayList<String>)headers.get(Constants.FILE_HEADER_LIST);
if (fileHeaders != null && fileHeaders.size() > 0) {
csv.setHeader(fileHeaders);
}
}
})
.unmarshal(csv)
.split(body()).streaming()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
GenericObjectModel model = null;
HashMap<String, String> csvRecord = (HashMap<String, String>)exchange.getIn().getBody();
}
});
}
}
What could be missing in the Approach 2?
The big difference between approach 1 and 2 is the scope.
In approach 1 you fully configure the CSV data format. This is all done when the Camel Context is created, since the data format is shared within the Camel Context. When messages are processed, it is the same config for all messages.
In approach 2 you just configure the basics globally. The header configuration is within the route and therefore can change for every single message. Every message would overwrite the header configuration of the context-global data format instance.
Without being sure about this, I guess that it is not possible to change a context-global DataFormat inside the routes.
What would you expect (just for example) when messages are processed in parallel? They would overwrite the header config against each other.
As an alternative, you could use a POJO where you can do your dynamic marshal / unmarshal from Java code.
Working on a Jersey web application with a team, as the project got bigger and bigger, we decided to switch from Tomcat to Grizzly to allow deploying parts of the project on different port numbers. What I've found out now, that the custom exception handling we have fails to work now, instead I always get the grizzly html page.
Example exception:
public class DataNotFoundException extends RuntimeException{
private static final long serialVersionUID = -1622261264080480479L;
public DataNotFoundException(String message) {
super(message);
System.out.println("exception constructor called"); //this prints
}
}
Mapper:
#Provider
public class DataNotFoundExceptionMapper implements ExceptionMapper<DataNotFoundException>{
public DataNotFoundExceptionMapper() {
System.out.println("mapper constructor called"); //doesnt print
}
#Override
public Response toResponse(DataNotFoundException ex) {
System.out.println("toResponse called"); //doesnt print
ErrorMessage errorMessage = new ErrorMessage(ex.getMessage(), 404, "No documentation yet.");
return Response.status(Status.NOT_FOUND)
.entity(errorMessage)
.build();
//ErrorMessage is a simple POJO with 2 string and 1 int field
}
}
I'm not sure where is the problem source, if needed I can provide more information/code. What's the problem, what can I try?
EDIT:
Main.class:
public class Main {
/**
* Main method.
* #param args
* #throws Exception
*/
public static void main(String[] args) throws Exception {
...
List<ServerInfo> serverList = new ArrayList<ServerInfo>();
serverList.add(new ServerInfo(
"api",8450,
new ResourceConfig().registerClasses(
the.package.was.here.ApiResource.class)
));
for(ServerInfo server : serverList) {
server.start();
}
System.out.println("Press enter to exit...");
System.in.read();
for(ServerInfo server : serverList) {
server.stop();
}
}
}
EDIT2:
based on this question I've tried using this ServerProperties.RESPONSE_SET_STATUS_OVER_SEND_ERROR, "true"property, which only helped a little. I still get the html grizzly page when the exception happens, but now I see my exception (+stack trace) in the body of the page.
You're only registering one resource class for the entire application
new ResourceConfig().registerClasses(
eu.arrowhead.core.api.ApiResource.class
)
The mapper needs to be registered also
new ResourceConfig().registerClasses(
eu.arrowhead.core.api.ApiResource.class,
YourMapper.class)
)
You can also use package scanning, which will pick up all classes and automatically register them, if they are annotated with #Path or #Provider
new ResourceConfig().packages("the.packages.to.scan")
I am struggling on a simple task. I want to create a cxfrs consumer that simply consumes json.
The json should be converted to a simple map (key->value): I created a simple test:
#Test
public final void test() throws Exception {
MockEndpoint mockOut = context.getEndpoint(MOCK_OUT, MockEndpoint.class);
mockOut.expectedMessageCount(1);
context.addRoutes(createRouteBuilder());
context.start();
context.createProducerTemplate().sendBody(DIRECT_A, "{ \"ussdCode\":\"101#\",\"msisdn\":\"491234567\"}");
mockOut.assertIsSatisfied();
}
private RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from(DIRECT_A).to("cxfrs://http://localhost:8085/ussd");
from("cxfrs://http://localhost:8085/ussd")
.unmarshal().json(JsonLibrary.Jackson)
.process(to).to(MOCK_OUT);
}
};
}
The problem is on context.start() i get ServiceConstructionException: No resource classes found. I also tried to create the consumer this way (setting binding style):
private Endpoint fromCxfRsEndpoint() {
CxfRsEndpoint cxfRsEndpoint = context.getEndpoint("cxfrs://http://localhost:8085/ussd", CxfRsEndpoint.class);
cxfRsEndpoint.setBindingStyle(BindingStyle.SimpleConsumer);
return cxfRsEndpoint;
}
This didn't helped neither. So how to create a simple rest/json consumer and unmarshal to a simple map?