Extract some value without key from a json object - json

I am using java to extract a list of node ids from an elastic search running tasks response.
The response looks like this
{
"nodes": {
"ZKUuxQZpRCCcJ0njBM1P0A": {
"name": "ZKUuxQZ",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"roles": [
"master",
"data",
"ingest"
],
"tasks": {
"ZKUuxQZpRCCcJ0njBM1P0A:118": {
"node": "ZKUuxQZpRCCcJ0njBM1P0A",
"id": 118,
"type": "transport",
"action": "indices:data/write/delete/byquery",
"start_time_in_millis": 1527808643421,
"running_time_in_nanos": 154234724059,
"cancellable": true,
"headers": {}
}
}
}
}
}
In this example, I want to exact ZKUuxQZpRCCcJ0njBM1P0A:118. Can someone give me an example how to extract this information?
Option 1, java json parser. I wanted to write a response class and parse the json string, but ZKUuxQZpRCCcJ0njBM1P0A:118 is not in the key:value format. I am not sure how to extract it.
Option 2, using regex. The node id part may also have - or _. Can someone provide a neat java regex solution? So far my regex is [a-zA-Z0-9\-\_]+\:[0-9]+, I am not sure if this is safe enough.
Thanks.

One way you can do this is by using org.json library and create JSONObject and using keys() you can get all the keys and use this to achieve your task
Add the following dependency in your pom.xml
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20180130</version>
</dependency>
Then you can do something like below:
String jsonString = "{\n" +
" \"nodes\": {\n" +
" \"ZKUuxQZpRCCcJ0njBM1P0A\": {\n" +
" \"name\": \"ZKUuxQZ\",\n" +
" \"transport_address\": \"127.0.0.1:9300\",\n" +
" \"host\": \"127.0.0.1\",\n" +
" \"ip\": \"127.0.0.1:9300\",\n" +
" \"roles\": [\n" +
" \"master\",\n" +
" \"data\",\n" +
" \"ingest\"\n" +
" ],\n" +
" \"tasks\": {\n" +
" \"ZKUuxQZpRCCcJ0njBM1P0A:118\": {\n" +
" \"node\": \"ZKUuxQZpRCCcJ0njBM1P0A\",\n" +
" \"id\": 118,\n" +
" \"type\": \"transport\",\n" +
" \"action\": \"indices:data/write/delete/byquery\",\n" +
" \"start_time_in_millis\": 1527808643421,\n" +
" \"running_time_in_nanos\": 154234724059,\n" +
" \"cancellable\": true,\n" +
" \"headers\": {}\n" +
" }\n" +
" }\n" +
" }\n" +
" }\n" +
"}";
JSONObject jsonObject = new JSONObject(jsonString);
Set<String> topLevelKeys = jsonObject.keySet();
for (String key : topLevelKeys) {
Object value = jsonObject.get(key);
if (value instanceof JSONObject) {
JSONObject valueObject = jsonObject.getJSONObject(key);
System.out.println(valueObject.toString());
}
}
Extend this as per your requirement.

So regex is too hacky and I figured it out using gson. I wish ElasticSearch can offer us some standard library to extract all kinds of responses. Here is my solution using gson.
import com.google.gson.*;
import org.json.JSONObject;
public class test {
public static void main(String[] args) throws Exception {
String jsonString = "json_format elasticsearch reponse for listing running tasks"
JsonParser parser = new JsonParser();
JsonObject jsonObject = parser.parse(content).getAsJsonObject();
jsonObject.getAsJsonObject("nodes").entrySet().forEach(
s -> {
s.getValue().getAsJsonObject().getAsJsonObject("tasks").entrySet().forEach( s2 -> {
System.out.println(s2.getKey());
});
});
}
It prints all the running task ids like the following. It's technically nodeId:taskId, but ES has a very vague description in their task API docs(It says you can use TASK_ID to check a status of a task, well that TASK_ID is nodeId:taskId).
Mw-3i39gTHGxu5c8z9viQQ:503209021
DZ29LMsWR0aW9guWZTYe2Q:482931604
6CAbDZSWR8SfwZgnRT0qNg:494351185

Use the following regex:
[a-zA-Z0-9-_]*[0-9]+[a-zA-Z]+:[0-9]*
You can find the test result here:
https://regexr.com/3qdug

Related

I want to convert csv to json with key value pair

I have a JSON file as below and want to convert to json with Name key , value pair ,
Eg : "name":"emailAddress",
"value" :"Trest22#gmail.com"
If have multiple JSON tags then it should break into separate tag as shown in image.
Please help
[
{
"emailAddress": "Trest22#gmail.com",
"loginName": "Testinguser222",
"firstName": "Test222",
"surName": "Test22",
"primaryPhone": "",
"companyId": 123445,
"extracompanies": "[12311,33333]",
"middleName": "Test",
"mobilePhone": 6666666666,
"fax": 87687687686
}
]
Want to convert as below
{
{
"name":"emailAddress",
"value" :"Trest22#gmail.com"
}
{
"name":"loginName",
"value":"Testinguser222"
}
{
"name":"firstName",
"value":"Test222"
}
{
"name":"surName",
"value":"Test22"
}
{ "name":"extracompanies",
"value": "[12311,33333]"
}
I am not sure if any library can do that, however, in Java, you can achieve it as shown below,
ObjectMapper mapper = new ObjectMapper();
String test = "{\n" +
" \"emailAddress\": \"Trest22#gmail.com\",\n" +
" \"loginName\": \"Testinguser222\",\n" +
" \"firstName\": \"Test222\",\n" +
" \"surName\": \"Test22\",\n" +
" \"primaryPhone\": \"\",\n" +
" \"companyId\": 123445,\n" +
" \"extracompanies\": \"[12311,33333]\",\n" +
" \"middleName\": \"Test\",\n" +
" \"mobilePhone\": 6666666666,\n" +
" \"fax\": 87687687686\n" +
" \n" +
" }";
Map<String, Object> maps = mapper.readValue(test, Map.class);
List<Map<String,Object>> converted = maps.keySet().stream().map(key -> {
Map<String,Object> internalMap = new HashMap<>();
internalMap.put("name", key);
internalMap.put("value", maps.get(key));
return internalMap;
}).collect(Collectors.toList());
String json = mapper.writeValueAsString(converted);
Output
[{"name":"emailAddress","value":"Trest22#gmail.com"},{"name":"loginName","value":"Testinguser222"},{"name":"firstName","value":"Test222"},{"name":"surName","value":"Test22"},{"name":"primaryPhone","value":""},{"name":"companyId","value":123445},{"name":"extracompanies","value":"[12311,33333]"},{"name":"middleName","value":"Test"},{"name":"mobilePhone","value":6666666666},{"name":"fax","value":87687687686}]

how to rea a json data and sava as parquet in apache beam

This is a problem that has bothered me for two days。
I can read a json data from local ,but when I write it as parquet get a error.
my code as following :
public class parquet_save_convert {
private static final Schema SCHEMA = new Schema.Parser().parse(
"{ \n" +
" \"namespace\": \"com.navteq.avro\", \n" +
" \"name\": \"FacebookUser\", \n" +
" \"type\": \"record\",\n" +
" \"fields\": [\n" +
" {\"name\": \"event_level\", \"type\": \"string\"},\n" +
" {\"name\": \"spm_page\", \"type\": \"string\"},\n" +
" {\"name\": \"spm_module\", \"type\": \"string\"} ]\n" +
"}");
public static void main(String[] args) {
Gson gson=new GsonBuilder().create();
String outputPath = "./output/parquet";
PipelineOptions options = PipelineOptionsFactory.create();
Pipeline pipeline = Pipeline.create(options);
pipeline.apply(TextIO.read().from("./input/event_type.json"))
.apply(ParDo.of(new DoFn<String,GenericRecord>(){
#ProcessElement
public void processElement(ProcessContext c){
HashMap<String,String> map= gson.fromJson(c.element().toString(),HashMap.class);
GenericRecord osRecord = new GenericData.Record(SCHEMA);
map.forEach((k,v)->{
osRecord.put(k,v);
});
c.output(osRecord);
}
}))
.setCoder(AvroCoder.of(GenericRecord.class,SCHEMA))
.apply(FileIO.<GenericRecord>write()
.via(ParquetIO.sink(SCHEMA)).to(outputPath)
.withSuffix(".parquet"));
pipeline.run().waitUntilFinish();}
my input data like:
{"event_level":"item","spm_page":"Activity","spm_module":"click"}
{"event_level":"page","spm_page":"Activity","spm_module":"action"}
{"event_level":"page","spm_page":"Activity","spm_module":"click"}
{"event_level":"item","spm_page":"Activity","spm_module":"action"}
and the exception is :
Exception in thread "main" java.lang.IllegalArgumentException: unable to serialize DoFnWithExecutionInformation{doFn=parquet_save_convert$1#5d10455d, mainOutputTag=Tag<output>, sideInputMapping={}, schemaInformation=DoFnSchemaInformation{elementConverters=[]}}
I also google this problem,but not get a answer ,it makes me feel very frustrated.
Example to read and write parquet file using ParquetIO through Apache Beam
Thanks in advance.
I'm guessing your Gson object is not serializable. The solution is probably to initialize this as a local variable in your DoFn's SetUp method.

BSON to JSON conversion using MongoDB Driver Java API

I am using MongoDB Driver Java API to convert BSON to JSON.
I have test code like this.
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output = " + doc.toJson());
The output is:
input = { "timestamp" : 1486064586641 }
output = { "timestamp" : { "$numberLong" : "1486064586641" } }
Is there an easy way to make the output look like the input?
BSON Documnet's toJson method supports only output to MongoDB Extended JSON (STRICT or SHELL format). If you want to have regular JSON, you can use com.mongodb.util.JSON class:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output (SHELL) = " + doc.toJson(new JsonWriterSettings(JsonMode.SHELL)));
System.out.println("output (STRICT) = " + doc.toJson(new JsonWriterSettings(JsonMode.STRICT)));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc));
This will generate following output:
input = { "timestamp" : 1486064586641 }
output (SHELL) = { "timestamp" : NumberLong("1486064586641") }
output (STRICT) = { "timestamp" : { "$numberLong" : "1486064586641" } }
output (JSON) = { "timestamp" : 1486064586641}
Natalja's answer is excellent, but if you are using the Mongo Java driver 3.8.2 upwards you will notice some deprecation warnings. If you want the output to look like the input you can use RELAXED JsonWriterSettings mode.
Below you can see an example with the possible modes and how the JSON will looks like. There are also some deprecation warnings and alternatives to the deprecated code:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
JsonWriterSettings shellMode = JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build();
System.out.println("output (SHELL) = " + doc.toJson(shellMode));
JsonWriterSettings strictMode = JsonWriterSettings.builder().outputMode(JsonMode.STRICT).build();
System.out.println("output (STRICT) = " + doc.toJson(strictMode)); // deprecated - use extended like below
JsonWriterSettings extendedMode = JsonWriterSettings.builder().outputMode(JsonMode.EXTENDED).build();
System.out.println("output (EXTENDED) = " + doc.toJson(extendedMode));
JsonWriterSettings relaxed = JsonWriterSettings.builder().outputMode(JsonMode.RELAXED).build();
System.out.println("output (RELAXED) = " + doc.toJson(relaxed));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc)); // deprecated - use relaxed like above
Also note that the JsonWriterSettings constructor is deprecated and you can use as an alternative the builder method like e.g:
JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build()

Unable to parse the id from moviedb api

Hi I'm trying to parse out the id's from an array. I have a logging method which shows a toast:
StringBuilder data= new StringBuilder();
JSONArray arrayMovies= response.getJSONArray(KEY_MOVIES);
for (int i = 0; i < arrayMovies.length(); i++) {
JSONObject currentMovie = arrayMovies.getJSONObject(i);
String id = currentMovie.getString(KEY_ID);
data.append(id+"\n");
}
L.t(getActivity(), data.toString());
.....
But It's not parsing it out. Just getting a list of the movies.
Here is the json file:
{
"page": 1,
"results": [
{
"adult": false,
"backdrop_path": "/tbhdm8UJAb4ViCTsulYFL3lxMCd.jpg",
"genre_ids": [
53,
28,
12
],
"id": 76341,
"original_language": "en",
"original_title": "Mad Max: Fury Road",
"overview": "An apocalyptic story set in the furthest reaches of our planet, in a stark desert landscape where humanity is broken, and most everyone is crazed fighting for the necessities of life. Within this world exist two rebels on the run who just might be"
}
]
}
Not sure, but maybe because id value has no quote, and therefore, can not be parsed as a String with getString() method
I think your code is OK, just need check the response and keys again, I have tested with hard-code the response as the String and 2 keys as the following:
String jsonString = "{\n" +
" \"page\": 1,\n" +
" \"results\": [\n" +
" {\n" +
" \"adult\": false,\n" +
" \"backdrop_path\": \"/tbhdm8UJAb4ViCTsulYFL3lxMCd.jpg\",\n" +
" \"genre_ids\": [\n" +
" 53,\n" +
" 28,\n" +
" 12\n" +
" ],\n" +
" \"id\": 76341,\n" +
" \"original_language\": \"en\",\n" +
" \"original_title\": \"Mad Max: Fury Road\",\n" +
" \"overview\": \"An apocalyptic story set in the furthest reaches of our planet, in a stark desert landscape where humanity is broken, and most everyone is crazed fighting for the necessities of life. Within this world exist two rebels on the run who just might be\"\n" +
" }\n" +
" ]\n" +
"}";
StringBuilder data= new StringBuilder();
try {
JSONObject jsonObject = new JSONObject(jsonString);
JSONArray arrayMovies= jsonObject.getJSONArray("results");
for (int i = 0; i < arrayMovies.length(); i++) {
JSONObject currentMovie = arrayMovies.getJSONObject(i);
String id = currentMovie.getString("id");
data.append(id+"\n");
}
Log.i("BNK", data.toString());
} catch (JSONException e) {
e.printStackTrace();
}
And here is the screenshot (please pay attention to logcat):

Deserialize json into complex Map object using jackson

Consider the following json example
{
"key1" : {
"k11":["vala","valb","valc"],
"k12":["vald"],
"k13":["vale","valf"]
},
"key2" : {
"key21":["valg","valh","vali"],
"key22":["valj"],
"key23":["valk","vall"]
}
}
This translates into a Map<String,Map<String,List<String>>>.
Could anyone please let me know how i can convert this in this into this complex Map object. I do a a method called constructMapType, but not sure if it handles complex Map type.
Seems to work fine with .constructMapType(Map.class, String.class, Map.class)
public static void main(String[] args) throws Exception {
final String json
= "{\n"
+ " \"key1\" : {\n"
+ " \"k11\":[\"vala\",\"valb\",\"valc\"],\n"
+ " \"k12\":[\"vald\"],\n"
+ " \"k13\":[\"vale\",\"valf\"]\n"
+ " },\n"
+ " \"key2\" : {\n"
+ " \"key21\":[\"valg\",\"valh\",\"vali\"],\n"
+ " \"key22\":[\"valj\"],\n"
+ " \"key23\":[\"valk\",\"vall\"]\n"
+ " }\n"
+ "}";
ObjectMapper mapper = new ObjectMapper();
Map<String, Map<String, List<String>>> map
= mapper.readValue(json,TypeFactory.defaultInstance()
.constructMapType(Map.class, String.class, Map.class));
for (String outerKey: map.keySet()) {
System.out.println(outerKey + ": " + map.get(outerKey));
for (String innerKey: map.get(outerKey).keySet()) {
System.out.print(innerKey + ": [");
for (String listValue: map.get(outerKey).get(innerKey)) {
System.out.print(listValue + ",");
}
System.out.println("]");
}
}
}
You could go all the way down listing all the generics down to the List<String>, but as seen above it isn't necessary. But just to show what I mean
TypeFactory factory = TypeFactory.defaultInstance();
Map<String, Map<String, List<String>>> map
= mapper.readValue(json, factory.constructMapType(
Map.class,
factory.constructType(String.class),
factory.constructMapType(
Map.class,
factory.constructType(String.class),
factory.constructCollectionType(
List.class,
String.class))));