I'm using Flink to process some JSON-format data coming from some Data Source.
For now, my process is quite simple: extract each element from the JSON-format data and print them into log file.
Here is my piece of code:
// create proper deserializer to deserializer the JSON-format data into ObjectNode
PravegaDeserializationSchema<ObjectNode> adapter = new PravegaDeserializationSchema<>(ObjectNode.class, new JavaSerializer<>());
// create connector to receive data from Pravega
FlinkPravegaReader<ObjectNode> source = FlinkPravegaReader.<ObjectNode>builder()
.withPravegaConfig(pravegaConfig)
.forStream(stream)
.withDeserializationSchema(adapter)
.build();
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<ObjectNode> dataStream = env.addSource(source).name("Pravega Stream");
dataStream.???.print();
Saying that the data coming from Pravega is like this: {"name":"titi", "age":18}
As I said, for now I simply need to extract name and age and print them.
So how could I do this?
As my understanding, I need to make some customized codes at ???. I might need to create a custom POJO class which contains ObjectNode. But I don't know how. I've read the official doc of Flink and also tried to google about how to create a custom POJO for Flink but I can't still figure out clearly.
Could you please show me an example?
Why don't You simply use something more meaningful instead of JavaSerializer? Perhaps something from here.
You could then create a POJO with the fields you want to use and simply deserialize JSON data to Your POJO instead of ObjectNode
Also, if there is some specific reason that You need to have ObjectNode on deserialization then You can simply do something like :
//I assume You have created the class named MyPojo
dataStream.map(new MapFunction<ObjectNode, MyPojo>() {
ObjectMapper mapper = new ObjectMapper();
#Override
public MyPojo map(final ObjectNode value) throws Exception {
mapper.readValue(value.asText(), MyPojo.class)
}
})
I'm fairly new to Kotlin and I'm having trouble manipulating a basic JSON string to access its contents. The JSON string looks like this:
"{\"id\":24,\"name\":\"nope\",\"username\":\"unavailable1991\",\"profile_image_90\":\"/uploads/user/profile_image/24/23102ca5-1412-489d-afdf-235c112c7d8e.jpg\",\"followed_tag_names\":[],\"followed_tags\":\"[]\",\"followed_user_ids\":[],\"followed_organization_ids\":[],\"followed_podcast_ids\":[],\"reading_list_ids\":[],\"blocked_user_ids\":[],\"saw_onboarding\":true,\"checked_code_of_conduct\":true,\"checked_terms_and_conditions\":true,\"number_of_comments\":0,\"display_sponsors\":true,\"trusted\":false,\"moderator_for_tags\":[],\"experience_level\":null,\"preferred_languages_array\":[\"en\"],\"config_body_class\":\"default default-article-body pro-status-false trusted-status-false default-navbar-config\",\"onboarding_variant_version\":\"8\",\"pro\":false}"
I've tried using the Gson and Klaxon packages without any luck. My most recent attempt using Klaxon looked like this:
val json: JsonObject? = Klaxon().parse<JsonObject>(jsonString)
But I get the following error: java.lang.String cannot be cast to com.beust.klaxon.JsonObject
I also tried trimming the double quotes (") at the start and end of the string, and also removing all the backslashes like this:
val jsonString = rawStr.substring(1,rawStr.length-1).replace("\\", "")
But when running the same Klaxon parse I now get the following error: com.beust.klaxon.KlaxonException: Unable to instantiate JsonObject with parameters []
Any suggestions (with or without Klaxon) to parse this string into an object would be greatly appreciated! It doesn't matter if the result is a JsonObject, Map or a custom class, as long as I can access the parsed JSON data :)
Gson is perfect library for this kinda task, here how to do it with gson.
Kotlin implementation,
var map: Map<String, Any> = HashMap()
map = Gson().fromJson(jsonString, map.javaClass)
Or if you want to try with Java,
Gson gson = new Gson();
Map<String,Object> map = new HashMap<String,Object>();
map = (Map<String,Object>) gson.fromJson(jsonString, map.getClass());
And also I just tried with your json-string and it is perfectly working,
Kotlin now provides a multiplatform / multi-format reflectionless serialization.
plugins {
kotlin("jvm") version "1.7.10" // or kotlin("multiplatform") or any other kotlin plugin
kotlin("plugin.serialization") version "1.7.10"
}
repositories {
mavenCentral()
}
dependencies {
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.4.0")
}
So now you can simply use their standard JSON serialization library:
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
fun main() {
val jsonString = "{\"id\":24,\"name\":\"nope\",\"username\":\"unavailable1991\",\"profile_image_90\":\"/uploads/user/profile_image/24/23102ca5-1412-489d-afdf-235c112c7d8e.jpg\",\"followed_tag_names\":[],\"followed_tags\":\"[]\",\"followed_user_ids\":[],\"followed_organization_ids\":[],\"followed_podcast_ids\":[],\"reading_list_ids\":[],\"blocked_user_ids\":[],\"saw_onboarding\":true,\"checked_code_of_conduct\":true,\"checked_terms_and_conditions\":true,\"number_of_comments\":0,\"display_sponsors\":true,\"trusted\":false,\"moderator_for_tags\":[],\"experience_level\":null,\"preferred_languages_array\":[\"en\"],\"config_body_class\":\"default default-article-body pro-status-false trusted-status-false default-navbar-config\",\"onboarding_variant_version\":\"8\",\"pro\":false}"
Json.parseToJsonElement(jsonString) // To a JsonElement
.jsonObject // To a JsonObject
.toMutableMap() // To a MutableMap
}
See: Kotlin Serialization Guide for further details.
To do it in Klaxon, you can do:
Klaxon().parse<Map<String,Any>>(jsonString)!!
I am passing a json object to the client side from java object with a time and value as attributes with gson
this.template.convertAndSend("/topic/123", gson.toJson(object, type));
and on the client side i have the following code where the json object data is stored in the body of the payload but I am unable to access the properties with obj.time or obj.value, it tells me undefined after it is parsed, I tried showing the entire 'obj' itself and the format seems fine however:
var subscription_callback1 = function(payload) {
var obj = JSON.parse(payload.body);
alert(obj);
};
output with alert(obj)
{"time":"3:00:34","value":"7989797"}
Nevermind solved. Since I am transfering STOMP protocol messages with the Spring 4 framework. I opted to use the Jackson2 message converter instead of directly using gson and it seems to work
#Configuration
#EnableWebSocketMessageBroker
public class MessageBrokerConfigurer extends AbstractWebSocketMessageBrokerConfigurer {
#Override
public boolean configureMessageConverters(List<MessageConverter> messageConverters) {
messageConverters.add(new MappingJackson2MessageConverter());
return true;
}
then i directly put my java object into the send function instead of using gson to convert it as above
this.template.convertAndSend("/topic/123", event)
I have seen the responses from many other posts but would like to understand if there is a better way to do the same thing.
Requirement:-
I am using restTemplate to talk to web service which returns JSON output which is dynamic. As a consumer I don't want to access all fields but is interested in few of them. I am using Spring framework with Jackson parser and found the way of accessing it
String response = restTemplate.getForObject(targetUrl, String.class);
System.out.println(response);
ObjectMapper mapper = new ObjectMapper();
JsonNode rootNode = mapper.readValue(response, JsonNode.class);
JsonNode uri = rootNode.get("uri");
System.out.println(uri.asText());
Do you know any better way to do it? Mapping to java Object is something that I dont want to do as the json output is not in my control
If your RestTemplate is configured with the default HttpMessageConverters, which is provided by Jackson2ObjectMapperBuilder, you may directly get a JsonNode from restTemplate.getForObject.
For example,
ArrayNode resources = restTemplate.getForObject("/resources", ArrayNode.class);
Or,
ObjectNode resource = restTemplate.getForObject("/resources/123", ObjectNode.class);
Note that ArrayNode and ObjectNode are sub-classes of JsonNode.
I'm currently writing a Jersey REST interface. I'm have this code which I'm trying to create the following JSON response and a list of long is been generated a array of string (Using Jaxb and Jackson)
The code looks like :
#XmlElement(name = "visitorProfiles", required = false)
private List<Long> visitorProfiles;
The JSON reponse looks like
{
"visitorProfiles":["45"]
}
And I correct JSON response should be
{
"visitorProfiles":[45]
}
This is what I'm using for the JSON configuration
context = new JSONJAXBContext(JSONConfiguration.natural().rootUnwrapping(true).build(), JerseyResources.getJaxbClasses());
Tried to reproduce the case with no success, I can't see any reason for this to happen unless the natural context wasn't apply correctly for that class