I'm loving vertx.io in my spare time. Just now I am switching from plain vertx to rxjava2-fied (vertx-rx-java2, 3.5.1) api version and my VertxUnitRunner tests doesn't complete anymore:
#Test
public void computeSomethingByNullPlan(TestContext ctx) {
query = null;
Async async = ctx.async();
vertx.eventBus().send(HANDLER_ADDRESS, query,
new DeliveryOptions().addHeader("action", ACTION), msg -> {
if (msg.failed())
log.error(msg.cause());
ctx.assertTrue(msg.succeeded());
ctx.assertTrue(new
JsonArray(msg.result().body().toString()).isEmpty());
async.complete();
});
}
test runs fine but once async.complete(); instruction is hit test does not return but hangs. Maybe it is because I'm mixing io.vertx.reactivex.core and io.vertx.core together (e.g.: io.vertx.reactivex.core.Vertx and io.vertx.core.TestContex) or maybe I'm not using the correct VertxUnitRunner.
What am I doing wrong? I search for a while but no success, Is there any example/doc about vertx.io rxjava2 and testing?
i'm able to reproduce the issue of the test "hanging" when this bit:
JsonArray(msg.result().body().toString())
...fails to deserialize the response body as an array of JSON. to verify this, comment out that entire assertion and see if the test completes afterwards. if so, that should narrow down the scope of that particular issue.
it's worth noting that since you're using the Rx-ified version of the API, you can represent asynchronous sources using the standard Rx types (instead of Handler). your test rewritten along those lines (minus the JsonArray stuff) might look something like this:
import io.vertx.ext.unit.Async;
import io.vertx.ext.unit.TestContext;
import io.vertx.ext.unit.junit.VertxUnitRunner;
import io.vertx.reactivex.core.Vertx;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
#RunWith(VertxUnitRunner.class)
public class Blah {
private Vertx vertx;
#Before
public void setUp(TestContext context) {
vertx = Vertx.vertx();
vertx.eventBus().consumer("HANDLER_ADDRESS").handler(message -> {
message.reply("PONG!");
});
}
#Test
public void computeSomethingByNullPlan(TestContext context) {
final String query = "PING?";
final Async async = context.async();
vertx.eventBus().rxSend("HANDLER_ADDRESS", query)
.subscribe(
message -> {
context.assertEquals("PONG!", message.body().toString());
async.complete();
},
context::fail
);
}
}
i hope that helps clear things up!
Related
I wrote a Scala program, created a Kafka topic, show data in console. Now, I'm trying to modify the existing code so it can sink to a MySQ table.
db name: books
table name: authors
Can you help me to modify the code below so data from Kafka topic_json can be sent to a MySQL table?
df.selectExpr("CAST(id AS STRING) AS key", "to_json(struct(*)) AS value")
.writeStream
.format("jdbc")
.outputMode("overwrite")
.option("url", "jdbc:mysql://127.0.0.1:3306/books")
.option("books", "authors")
.start()
.awaitTermination()
ReadStream code for reference:
val df = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "json_topic")
.option("startingOffsets", "earliest") // From starting
.load()
Appreciate all your help.
How about try to use “for each batch”,
and the essential logic to write is about writing a normal DF to MySQL.
See below for the plumbing part.
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch
If you struggle to get the logic of writing a normal data frame to MySQL, I can later paste my working example code which writes the stream to postgresql using for each batch here.
Code to write to postgresql, rushed POC , on Java. Pay attention to method writeToPostgresql. hopefully you can get the idea and get the scala version working. let me know if you need help on scala version.
import org.apache.spark.api.java.function.VoidFunction2;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.streaming.OutputMode;
import java.util.HashMap;
import java.util.Map;
//todo: write a logging class and extends from it.
public class PostgresqlService {
// Transform and write batchDF.
static <T> void postgresSink(Dataset<T> dataset, Long batchId, String tableName) {
dataset.persist();
dataset.write().format("jdbc").option("dbtable", tableName).options(postgresqlOptions()).mode(SaveMode.Append).save();
dataset.write().format("console").mode(SaveMode.Overwrite).save(); //note: on prod do not do this type of stuff.
dataset.unpersist();
}
//TODO(optional): pass in as an option for things like checkpoint.
//Method to write the dataset into postgresql
public static <T> void writeToPostgresql(Dataset<T> dataset, OutputMode outputMode, String tableName) {
try {
dataset
.writeStream()
.option("checkpointLocation", "/tmp/spark-checkpoint/"+tableName) //path/to/HDFS/dir
.outputMode(outputMode)
.foreachBatch(
new VoidFunction2<Dataset<T>, Long>() {
public void call(Dataset<T> dataset, Long batchId) {
postgresSink(dataset, batchId, tableName);
}
}
)
// .trigger(Trigger.Once())
.start()
.awaitTermination();
} catch (Exception e) {
System.out.println(e.toString());
System.exit(1);
}
}
/**
* Spark-PostgreSQL connection properties.
*
* #return Map of String -> String
*/
static Map<String, String> postgresqlOptions() {
//TODO (optional): current is POC level. if i have time: read from config
Map<String, String> map = new HashMap<String, String>() {
{
put("user", "sparkstreaming"); // Database username
put("password", "password"); // Password
put("driver", "org.postgresql.Driver");
put("url", "jdbc:postgresql://localhost:5432/sparkstreaming");
}
};
return map;
}
}
`
when i called above method, i used `OutputMode.Update()`, like
` writeToPostgresql(transformedAggregatedPayload, OutputMode.Update(), "my-table-name");`
In my Junit Jupiter API 5.5 test, I am calling my method which internally makes a HTTP call to a remote service.
Now the remote service can be down or behave incorrectly. I want to skip my test in case the remote service is not behaving expectedly.
#Test
void testMe() {
// do something
Result res1 = myObject.retrieveResults(params)
// assert something
Result res2 = myObject.retrieveResults(param2)
//asert on results
}
Result retrieveResults(Parameters param) {
// do something
// call to remote service
// if they do not give result throw CustomException()
// return Result
}
So basically in my test i would want to check if myObject.retrieveResult is throwing CustomException then skip that test, otherwise evaluate normally.
We have 2 different ways to accomplish this tasks in JUnit 5.
For demo purposes, I have created a basic class which sends a request to the url
that is passed as an argument to its call(String url) method and
returns true or false depending on the request result.
The body of the method is irrelevant here.
Using Assumptions.assumeTrue()/assumeFalse() methods
Assumptions class provides us with two overloaded methods - assumeTrue
and assumeFalse. The idea is that, if the assumption is wrong, the test will be skipped.
So, the test will be something like this.
#Test
void call1() {
Assumptions.assumeTrue(new EndpointChecker(), "Endpoint is not available");
Assertions.assertTrue(HttpCaller.call("https://www.google.com"));
}
Here is the code for EndpointChecker class.
static class EndpointChecker implements BooleanSupplier {
#Override
public boolean getAsBoolean() {
// check the endpoint here and return either true or false
return false;
}
}
When the test is run, the availability of the endpoint will be checked first, if it is up, then the test will run.
Using JUnit 5 extension mechanisms.
So, let's start with creating the annotation. It is pretty straightforward.
#Retention(RetentionPolicy.RUNTIME)
#ExtendWith(EndpointAvailabilityCondition.class)
public #interface SkipWhenEndpointUnavailable {
String uri();
}
And EndpointAvailabilityCondition class. Even though, it looks big, overall logic is very simple.
import static org.junit.platform.commons.util.AnnotationUtils.findAnnotation;
public class EndpointAvailabilityCondition implements ExecutionCondition {
#Override
public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
final var optional = findAnnotation(context.getElement(), SkipWhenEndpointUnavailable.class);
if (optional.isPresent()) {
final SkipWhenEndpointUnavailable annotation = optional.get();
final String uri = annotation.uri();
// check connection here start
boolean result = false; // dummy value
// check connection here end
if (result) {
return ConditionEvaluationResult.enabled("Connection is up");
} else {
return ConditionEvaluationResult.disabled("Connection is down");
}
}
return ConditionEvaluationResult.enabled("No assumptions, moving on...");
}
}
Hence, we can do the following in our tests.
#Test
#SkipWhenEndpointUnavailable(uri = "https://www.google.com")
void call2() {
Assertions.assertTrue(HttpCaller.call("https://www.google.com"));
}
We can go ahead and add #Test annotation over #SkipWhenEndpointUnavailable and remove it from our test code. Like, so:
#Retention(RetentionPolicy.RUNTIME)
#ExtendWith(EndpointAvailabilityCondition.class)
#Test
public #interface SkipWhenEndpointUnavailable {
String uri();
}
class HttpCallerTest {
#SkipWhenEndpointUnavailable(uri = "https://www.google.com")
void call2() {
Assertions.assertTrue(HttpCaller.call("https://www.google.com"));
}
}
I hope it helps.
Whats the simplest code i can use to start an asynchronous thread to fetch json data and then parse it on the UI thread in android using retrofit and rxjava?
I have searched this for two days and every solution so far gives me an error.
I basically want to replace all the old async tasks in my app with the newer retrofit and rxjava way of doing things.
import okhttp3.ResponseBody;
import retrofit2.Response;
import rx.Subscriber;
PoloniexService.getInstance().getAllCoins()
.subscribe(new Subscriber<Response<ResponseBody>>() {
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable e) {
}
#Override
public void onNext(Response<ResponseBody> responseBodyResponse) {
}
});
Something like this.. Except this gives me the error:
Cannot resolve method 'subscribe(anonymous rx.Subscriber<retrofit2.Response<okhttp3.ResponseBody>>)'
I have the following camel route:
from(inputDirectory)
.unmarshal(jaxb)
.process(jaxb2CSVDataProcessor)
.split(body()) //because there is a list of CSVRecords
.marshal(bindyCsvDataFormat)
.to(outputDirectory); //appending to existing file using "?autoCreate=true&fileExist=Append"
for my CSV model class I am using annotations:
#CsvRecord(separator = ",", generateHeaderColumns = true)
...
and for properties
#DataField(pos = 0)
...
My problem is that the headers are appended every time a new csv record is appended.
Is there a non-dirty way to control this? Am I missing anything here?
I made a work around which is working quite nicely, creating the header by querying the columnames of the #DataField annotation. This is happening once the first time the file is written. I wrote down the whole solution here:
How to generate a Flat file with header and footer using Camel Bindy
I ended up adding a processor that checks if the csv file exists just before the "to" clause. In there I do a manipulation of the byte array and remove the headers.
Hope this helps anyone else. I needed to do something similar where after my first split message I wanted to supress the header output. Here is a complete class (the 'FieldUtils' is part of the apache commons lib)
package com.routes;
import java.io.OutputStream;
import org.apache.camel.Exchange;
import org.apache.camel.dataformat.bindy.BindyAbstractFactory;
import org.apache.camel.dataformat.bindy.BindyCsvFactory;
import org.apache.camel.dataformat.bindy.BindyFactory;
import org.apache.camel.dataformat.bindy.FormatFactory;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
import org.apache.commons.lang3.reflect.FieldUtils;
public class StreamingBindyCsvDataFormat extends BindyCsvDataFormat {
public StreamingBindyCsvDataFormat(Class<?> type) {
super(type);
}
#Override
public void marshal(Exchange exchange, Object body, OutputStream outputStream) throws Exception {
final StreamingBindyModelFactory factory = (StreamingBindyModelFactory) super.getFactory();
final int splitIndex = exchange.getProperty(Exchange.SPLIT_INDEX, -1, int.class);
final boolean splitComplete = exchange.getProperty(Exchange.SPLIT_COMPLETE, false, boolean.class);
super.marshal(exchange, body, outputStream);
if (splitIndex == 0) {
factory.setGenerateHeaderColumnNames(false); // turn off header generate after first exchange
} else if(splitComplete) {
factory.setGenerateHeaderColumnNames(true); // turn on header generate when split complete
}
}
#Override
protected BindyAbstractFactory createModelFactory(FormatFactory formatFactory) throws Exception {
BindyCsvFactory bindyCsvFactory = new StreamingBindyModelFactory(getClassType());
bindyCsvFactory.setFormatFactory(formatFactory);
return bindyCsvFactory;
}
public class StreamingBindyModelFactory extends BindyCsvFactory implements BindyFactory {
public StreamingBindyModelFactory(Class<?> type) throws Exception {
super(type);
}
public void setGenerateHeaderColumnNames(boolean generateHeaderColumnNames) throws IllegalAccessException {
FieldUtils.writeField(this, "generateHeaderColumnNames", generateHeaderColumnNames, true);
}
}
}
I am writing integration tests to test existing Routes. The recommended way of getting the response looks something like this (via Camel In Action section 6.4.1):
public class TestGetClaim extends CamelTestSupport {
#Produce(uri = "seda:getClaimListStart")
protected ProducerTemplate producer;
#Test
public void testNormalClient() {
NotifyBuilder notify = new NotifyBuilder(context).whenDone(1).create();
producer.sendBody(new ClientRequestBean("TESTCLIENT", "Y", "A"));
boolean matches = notify.matches(5, TimeUnit.SECONDS);
assertTrue(matches);
BrowsableEndpoint be = context.getEndpoint("seda:getClaimListResponse", BrowsableEndpoint.class);
List<Exchange> list = be.getExchanges();
assertEquals(1, list.size());
System.out.println("***RESPONSE is type "+list.get(0).getIn().getBody().getClass().getName());
}
}
The test runs but I get nothing back. The assertTrue(matches) fails after the 5 second timeout.
If I rewrite the test to look like this I get a response:
#Test
public void testNormalClient() {
producer.sendBody(new ClientRequestBean("TESTCLIENT", "Y", "A"));
Object resp = context.createConsumerTemplate().receiveBody("seda:getClaimListResponse");
System.out.println("***RESPONSE is type "+resp.getClass().getName());
}
The documentation is a little light around this so can anyone tell me what I am doing wrong with the first approach? Is there anything wrong with following the second approach instead?
Thanks.
UPDATE
I have broken this down and it looks like the problem is with the mix of seda as the start endpoint in combination with the use of a recipientList in the Route. I've also changed the construction of the NotifyBuilder (I had the wrong endpoint specified).
If I change the start endpoint to
direct instead of seda then the test will work; or
If I comment out the recipientList
then the test will work.
Here's a stripped down version of my Route that reproduces this issue:
public class TestRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception {
// from("direct:start") //works
from("seda:start") //doesn't work
.recipientList(simple("exec:GetClaimList.bat?useStderrOnEmptyStdout=true&args=${body.client}"))
.to("seda:finish");
}
}
Note that if I change the source code of the NotifyTest from the "Camel In Action" source to have a route builder like this then it also fails.
Try to use "seda:getClaimListResponse" in the getEndpoint to be sure the endpoint uri is 100% correct
FWIW: It appears that notifyBuilder in conjunction with seda queues are not quite working: a test class to illustrate:
public class NotifyBuilderTest extends CamelTestSupport {
// Try these out!
// String inputURI = "seda:foo"; // Fails
// String inputURI = "direct:foo"; // Passes
#Test
public void testNotifyBuilder() {
NotifyBuilder b = new NotifyBuilder(context).from(inputURI)
.whenExactlyCompleted(1).create();
assertFalse( b.matches() );
template.sendBody(inputURI, "Test");
assertTrue( b.matches() );
b.reset();
assertFalse( b.matches() );
template.sendBody(inputURI, "Test2");
assertTrue( b.matches() );
}
#Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from(inputURI).to("mock:foo");
}
};
}
}