I want to load a JSON string to Elasticsearch version 7.3.
Following is the code i am using for this.
private RestHighLevelClient restHighLevelClient;
String jsonString="//here the complete JSON string";
JSONObject jsonObject = new JSONObject(cojsonStringntent1.toString());
HashMap<String, Object> hashMap = new Gson().fromJson(jsonObject.toString(), HashMap.class);
IndexRequest indexRequest = new IndexRequest("index", "type").source(hashMap);
restHighLevelClient.index(indexRequest, RequestOptions.DEFAULT);
Exception :
Exception in thread "main" java.lang.NullPointerException
at line restHighLevelClient.index(indexRequest, RequestOptions.DEFAULT);
If I post the same jsonString via POSTMEN than it is being loaded to ELASTICSEARCH perfectly.
If you are not using spring(as it's not mentioned), you can use below simple code to create a resthighlevelclient.
In below code, I am reading the elasticsearch configuration from a config file, feel free to change it to the way you read the properties or config, or if you just want to quickly test it hardcode the values of host and port
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(
RestClient.builder(new HttpHost(configuration.getElasticsearchConfig().getHost(),
configuration.getElasticsearchConfig().getPort(),
"http")));
Based on your sample code, your restHighLevelClient hasn't been initialized indeed at all. Please find snippet of code below how you could solve this:
#Bean
public RestHighLevelClient elasticRestClient () {
String[] httpHosts = httpHostsProperty.split(";");
HttpHost[] httpHostsAsArray = new HttpHost[httpHosts.length];
int index = 0;
for (String httpHostAsString : httpHosts) {
HttpHost httpHost = new HttpHost(httpHostAsString.split(":")[0], new Integer(httpHostAsString.split(":")[1]), "http");
httpHostsAsArray[index++] = httpHost;
}
RestClientBuilder restClientBuilder = RestClient.builder(httpHostsAsArray)
.setRequestConfigCallback(builder -> builder
.setConnectTimeout(connectTimeOutInMs)
.setSocketTimeout(socketTimeOutInMs)
);
return new RestHighLevelClient(restClientBuilder);
}
and your impl class uses the autowired RestHighLevelClient bean:
#Autowired
private RestHighLevelClient restClient;
Related
I am trying to receive and access JSON data from a Kafka Topic in Flink. What works is, producing data, send it to a Kafka Topic und receive it in Flink as String. But I want to access the data in an object-oriented way (e.g. extract a specific atrribute from every message)?
Therefore I have a Kafka Producer which sends data (e.g. every 1s) to a Kafka Topic:
ObjectMapper test = new ObjectMapper();
ObjectNode jNode= test.createObjectNode();
jNode.put("LoPos", longPos)
.put("LaPos", latPos)
.put("Timestamp", timestamp.toString());
ProducerRecord<String, ObjectNode> rec = new ProducerRecord<String, ObjectNode>(topicName, jNode);
producer.send(rec);
so the JSON data looks like this:
{"LoPos":10.5,"LaPos":2.5,"Timestamp":"2022-10-31 12:45:19.353"}
What works is, receiving the data and print it as string:
DataStream<String> input =
env.fromSource(
KafkaSource.<String>builder()
.setBootstrapServers("localhost:9092")
.setBounded(OffsetsInitializer.latest())
.setValueOnlyDeserializer(new SimpleStringSchema())
.setTopics(topicName)
.build(),
WatermarkStrategy.noWatermarks(),
"kafka-source");
Print the data as string:
DataStream<String> parsed = input.map(new MapFunction<String, String>() {
private static final long serialVersionUID = -6867736771747690202L;
#Override
public String map(String value) {
System.out.println(value);
return "test";
How can I receive the data in Flink and access it in an object-oriented way (e.g. extract LoPos from every message)? Which approach would you recommend? I tried it with JSONValueDeserializationSchema, but without success...
Thanks!
Update1:
I updated to Flink 1.16 to use JsonDeserializationSchema.
Then I created a Flink Pojo Event like this:
public class Event {
public double LoPos;
public double LaPos;
public Timestamp timestamp;
public Event() {}
public Event(final double LoPos, final double LaPos, final Timestamp timestamp) {
this.LaPos=LaPos;
this.LoPos=LoPos;
this.timestamp=timestamp;
}
#Override
public String toString() {
return String.valueOf(LaPos);
}
}
To read the JSON data, I implemented the following:
KafkaSource<Event> source = KafkaSource.<Event>builder()
.setBootstrapServers("localhost:9092")
.setBounded(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new JsonDeserializationSchema<>(Event.class))
.setTopics("testTopic2")
.build();
DataStream<Event> test=env.fromSource(source, WatermarkStrategy.noWatermarks(), "test");
System.out.println(source.toString());
System.out.println(test.toString());
//test.sinkTo(new PrintSink<>());
test.print();
env.execute();
So I would expect, when using source.toString() the value of LaPos is getting returned. But all I get is:
org.apache.flink.connector.kafka.source.KafkaSource#510f3d34
What am I doing wrong?
This topic is covered in one of the recipes in the Immerok Apache Flink Cookbook.
In the examples below, I'm assuming Event is a Flink POJO.
With Flink 1.15 or earlier, you should use a custom deserializer:
KafkaSource<Event> source =
KafkaSource.<Event>builder()
.setBootstrapServers("localhost:9092")
.setTopics(TOPIC)
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new EventDeserializationSchema())
.build();
The deserializer can be something like this:
public class EventDeserializationSchema extends AbstractDeserializationSchema<Event> {
private static final long serialVersionUID = 1L;
private transient ObjectMapper objectMapper;
/**
* For performance reasons it's better to create on ObjectMapper in this open method rather than
* creating a new ObjectMapper for every record.
*/
#Override
public void open(InitializationContext context) {
// JavaTimeModule is needed for Java 8 data time (Instant) support
objectMapper = JsonMapper.builder().build().registerModule(new JavaTimeModule());
}
/**
* If our deserialize method needed access to the information in the Kafka headers of a
* KafkaConsumerRecord, we would have implemented a KafkaRecordDeserializationSchema instead of
* extending AbstractDeserializationSchema.
*/
#Override
public Event deserialize(byte[] message) throws IOException {
return objectMapper.readValue(message, Event.class);
}
}
We've made this easier in Flink 1.16, where we've added a proper JsonDeserializationSchema you can use:
KafkaSource<Event> source =
KafkaSource.<Event>builder()
.setBootstrapServers("localhost:9092")
.setTopics(TOPIC)
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new JsonDeserializationSchema<>(Event.class))
.build();
Disclaimer: I work for Immerok.
We know that we can parse json file into IConfigurationRoot as
public class Startup
{
public IConfigurationRoot Configuration { get; }
public Startup(IHostingEnvironment env)
{
this.Configuration = new ConfigurationBuilder()
.SetBasePath(path)
.AddJsonFile("somefile.json")
.Build();
}
}
But I want to use ConfigurationBuilder to parse a json string so can access just like how it works with json file so that I can do:
string jsonString = XXX(); // external calls to get json
var config = new ConfigurationBuilder().AddJsonString(jsonString).Build();
string name = config["Student:Name"];
So does imaginal AddJsonString exists or is any third party library I need to use to achieve this?
P.S
I can't use JsonSerializer because the json payload has too many property therefore I can't create a POJO model class to be deserialized with, if there are only 3 or 4 property, then I can certainly do that, but 50 properties (that has nested properties) is a different story
You can use new ConfigurationBuilder().AddJsonStream(new MemoryStream(Encoding.ASCII.GetBytes(jsonString))).Build(); to load json from MemoryStream.
My test Code:
public Startup(IConfiguration configuration, IWebHostEnvironment env)
{
Configuration = configuration;
string jsonString = "{\"source\": \"test.com\", \"Time\":\"Feb 2019\" }"; // external calls to get json
var config = new ConfigurationBuilder().AddJsonStream(new MemoryStream(Encoding.ASCII.GetBytes(jsonString))).Build();
string name = config["source"];
}
Test Result(.Net Core 3.1):
My aim is to read a CSV file, convert it to JSON and send the generated JSON one by one to ActiveMQ queue. My Code below:
final BindyCsvDataFormat bindy=new BindyCsvDataFormat(camelproject.EquityFeeds.class);
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
CamelContext _ctx = new DefaultCamelContext();
_ctx.addComponent("jms", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
_ctx.addRoutes(new RouteBuilder() {
public void configure() throws Exception {
from("file:src/main/resources?fileName=data-sample.csv")
.unmarshal(bindy)
.marshal()
.json(JsonLibrary.Jackson).log("${body}")
.to("file:src/main/resources/?fileName=emp.json");
}
});
EquityFeeds is my POJO class in the above code.
Issues:
No Output is produced. "emp.json" file does not get generated at the given location.
Also how do I split the generated JSON into individual JSON's and send it to ActiveMQ queue like what I did for XML as below:
.split(body().tokenizeXML("equityFeeds", null)).streaming().to("jms:queue:xml.upstream.queue");
EquityFeeds (POJO):
#CsvRecord(separator = ",",skipFirstLine = true)
public class EquityFeeds {
#DataField(pos = 1)
private String externalTransactionId;
#DataField(pos = 2)
private String clientId;
#DataField(pos = 3)
private String securityId;
#DataField(pos = 4)
private String transactionType;
#DataField(pos = 5)
private Date transactionDate;
#DataField(pos = 6)
private float marketValue;
#DataField(pos = 7)
private String priorityFlag;
// getters and setters...
}
Please kindly help. Please tell me where I am going wrong. Need help desperately. Stuck in this issue and not able to move forward. Any help would be highly appreciated. I have really tried hard, searched Google and tried various options but nothing is working.
Please Note: I commented the .marshal() and .json() to check if the .unmarshal() is working but the unmarshal is also not working as "emp.json" is not getting created.
If nothing happens at all when starting the route then it is most likely due to the relative path you passed to the file component. Probably the execution directory of your Java process is not where you think it is and the file is not found. To simplify things I suggest you start with an absolute path. Once everything else is working figure out the correct relative path (your base should be the value of the user.dir system property).
Re your question about splitting the contents: This is answered in the documentation.
This works for me (Camel 3.1):
public class CsvRouteBuilder extends EndpointRouteBuilder {
#Override
public void configure() {
DataFormat bindy = new BindyCsvDataFormat(BindyModel.class);
from(file("/tmp?fileName=simpsons.csv"))
.unmarshal(bindy)
.split(body())
.log("Unmarshalled model: ${body}")
.marshal().json()
.log("Marshalled to JSON: ${body}")
// Unique file name for the JSON output
.setHeader(Exchange.FILE_NAME, () -> UUID.randomUUID().toString() + ".json")
.to(file("/tmp"));
}
}
// Use lombok to generate all the boilerplate stuff
#ToString
#Getter
#Setter
#NoArgsConstructor
// Bindy record definition
#CsvRecord(separator = ";", skipFirstLine = true, crlf = "UNIX")
public static class BindyModel {
#DataField(pos = 1)
private String firstName;
#DataField(pos = 2)
private String middleName;
#DataField(pos = 3)
private String lastName;
}
Given this input in /tmp/simpsons.csv
firstname;middlename;lastname
Homer;Jay;Simpson
Marge;Jacqueline;Simpson
the log output looks like this
Unmarshalled model: RestRouteBuilder.BindyModel(firstName=Homer, middleName=Jay, lastName=Simpson)
Marshalled to JSON: {"firstName":"Homer","middleName":"Jay","lastName":"Simpson"}
Unmarshalled model: RestRouteBuilder.BindyModel(firstName=Marge, middleName=Jacqueline, lastName=Simpson)
Marshalled to JSON: {"firstName":"Marge","middleName":"Jacqueline","lastName":"Simpson"}
and two json files are written in /tmp.
I want to use a JdbcTemplate and the Neo4j JDBC driver to query my neo4j database and return a JSON string.
Is there an existing method to do this?
I've googled and I can't find one.
It otherwise looks like a matter of creating a home cooked RowMapper as per here.
The query :
MATCH (s:Site) - [r] - (ss:SiteState) return s,ss;
it return a json but for my use i use an object
public class SiteRowMapper implements RowMapper<Site> {
#Override
public Site mapRow(ResultSet rs, int rowNum) throws SQLException {
Site site = new Site();
SiteState siteState = new SiteState();
Gson json = new Gson();
site = json.fromJson(rs.getString("s"), Site.class);
siteState = json.fromJson(rs.getString("ss"), SiteState.class);
site.setName(siteState.getName());
return site;
}
}
I am trying to insert json data in mySQL database using camel and hibernate.
Everything is working.
for (Module module : modules) {
from("timer://foo?delay=10000")
.loop(7)//not working
.to(module.getUrl() + "/api/json")
.convertBodyTo(String.class)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
int index = (Integer)exchange.getProperty("CamelLoopIndex"); // not working
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(exchange.getIn().getBody().toString());
String[] lijst = {"lastBuild", "lastCompletedBuild", "lastFailedBuild", "lastStableBuild", "lastSuccessfulBuild", "lastUnstableBuild", "lastUnsuccessfulBuild"};
JSONObject obj = new JSONObject();
JsonNode node = root.get(lijst[index]);
JsonNode build = node.get("number");
obj.put("description", lijst[index]);
obj.put("buildNumber", build);
exchange.getIn().setBody(obj.toString());
}
})
.unmarshal(moduleDetail)
.to("hibernate:be.kdg.teamf.model.ModuleDetail")
.end();
}
When I debug, my CamelLoopIndex remains 0 so it is not incremented every time it goes through the loop.
All help is welcome!
In your case the only first instruction is processed in scope of the loop: .to(module.getUrl() + "/api/json"). You can add more instructions into a loop using Spring DSL, but I don't know how to declare a loop scope using Java DSL explicitly. I hope experts will explain more about a loop scope in Java DSL.
As a workaround I suggest to move all iteration instructions to a separate direct: route.
I can't reproduce your problem. This works:
from("restlet:http://localhost:9010}/loop?restletMethod=get")
.loop(7)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
int index = (int) exchange.getProperty("CamelLoopIndex");
exchange.getIn().setBody("index=" + index);
}
})
.convertBodyTo(String.class)
.end();
Output:
index=6