jackson arraynode - json

I have a input json object that contains a array as shown below:
[{"key11":"value11","key12":"value12"},
{"key21":"value21","key22":"value22"},
...
{"keyn1":"valuen1","keyn2":"valuen2"}]
I would like to use jackson parser ,firstly, to identify the number of rows and secondly, print only the values on the screen. I want it to be as lightweight as possible. Can anyone point me to a direction?

Assuming that a row is one of the entries in a ArrayNode the following simple approach may be useful. It uses the JsonNode abstraction instead of a series of nested Map objects which I personally prefer since JsonNode provides a series of utility methods that are helpful when dealing with this kind of data (data where the structure is possibly unknown or very dynamic so that it can't be easily transformed to a POJO).
The testcase below illustrates how to find the number of rows and how to print the values. To get hold of the values the method JsonNode.elements() is used and the number of rows is simply a call to the size()-method.
public class ArrayNodeTest {
#Test
public void verifySizeAndPrintRows() throws IOException {
final String jsonStr =
"[{\"key11\":\"value11\",\"key12\":\"value12\"},\n" +
"{\"key21\":\"value21\",\"key22\":\"value22\"},\n" +
"{\"keyn1\":\"valuen1\",\"keyn2\":\"valuen2\"}]";
final ObjectMapper mapper = new ObjectMapper();
final JsonNode jsonNode = mapper.readTree(jsonStr);
// Verify size
Assert.assertEquals(3, jsonNode.size());
// And print rows
for (final JsonNode row : jsonNode) {
final Iterable<JsonNode> iterable = () -> row.elements();
iterable.forEach(elem -> System.out.println(elem.asText()));
}
}
}

to identify the number of rows
What's a "row"? Is a key/value pair a row? Is an element of the JSON array (no matter how many key/value pairs it contains) a row?
print only the values on the screen
Jackson has nothing to do with printing anything on any screens. Jackson can be used to populate a Java data structure from the input JSON, and the populated Java data structure can then be used however you want.
Given the JSON structure in the original question, a simple solution would be to bind to a list (or array) of maps, and then just iterate through the list of maps, accessing all of the values. Following is such an example.
import java.io.File;
import java.util.Map;
import org.codehaus.jackson.map.ObjectMapper;
public class JacksonFoo
{
public static void main(String[] args) throws Exception
{
ObjectMapper mapper = new ObjectMapper();
Map<String, String>[] maps = mapper.readValue(new File("input.json"), Map[].class);
for (Map<String, String> map : maps)
{
for (Map.Entry<String, String> entry : map.entrySet())
{
System.out.println(entry.getValue());
}
}
}
}
Output:
value11
value12
value21
value22
valuen1
valuen2

Related

Convert Dataframe Dataset<Row> to JSON Format of String Data Type for particular Columns and convert the JSON String back to Dataframe

I have a dataframe. I need to call a Rest API for each record.
Lets say the Dataframe looks like:
|----|-------------|-----|---------|
|UUID|PID |DEVID|FIRSTNAME|
|----|-------------|-----|---------|
|1111|1234567891011|ABC11|JOHN |
|2222|9876543256827|ABC22|HARRY |
|----|-------------|-----|---------|
The JSON request string for first row should look like(Note: the json is created on 2 columns, not all), as the Rest API to be called requires the input in this format:
{"applicationInfo": {"appId": "ec78fef4-92b9-3b1b-a68d-c45376b6977a"}, "requestData": [{"secureData": "JOHN", "secureDataType": "FIRSTNAME", "index": 1 }, {"secureData": "1234567891011", "secureDataType": "PID", "index": 2 } ] }
The value of index key has to be generted on the fly, using an incremental counter for each row.
Then, i need to call the Rest API sending the above JSON as a string param.
The response from the API after encryption will look like:
{"responseData":[{"resultCode":"00","secureData":"63ygdydshbhgvdyw3et7edgu","secureDataType":"FIRSTNAME","index":1},{"resultCode":"00","secureData":"HKJJBJHVHG66456456FXXFFCGF","secureDataType":"PID","index":2}],"responseCode":"00","responseMessage":"SUCCESS","resultCounts":{"totalCount":2,"successCount":2,"failedCount":0}}
Then I need to read the above response and create a dataframe which should look like:
|----|--------------------------|-----|------------------------|
|UUID|PID |DEVID|FIRSTNAME |
|----|--------------------------|-----|------------------------|
|1111|HKJJBJHVHG66456456FXXFFCGF|ABC11|63ygdydshbhgvdyw3et7edgu|
|----|--------------------------|-----|------------------------|
If i convert the initial input dataframe toJSON().collectAsList(), then it looks like:
[{"UUID":"1111","PID":"1234567891011","DEVID":"ABC11","FIRSTNAME":"JOHN"}, {"UUID":"2222","PID":"9876543256827","DEVID":"ABC22","FIRSTNAME":"HARRY"}]
But this doesnt work as the Rest API requires its input in a certain format, mentioned above.
Please help.
For the above, I assume that the data set has been partitioned across the number of Spark workers and it is a generic data set of Row (data frame), then the below mechanism can be employed.
Define a class with the required attributes as a data container
Take the data set content as a List (takeAsList method if data set, refer)
Create and populate the objects of your data container (and store in such a way to identify them later, you shall have to repopulate them with decrypted data)
Serialize the list into a JSON array with Jackson (refer)
Step 4 & 5 can be combined with Jackson custom serializer refer example
Make the REST call and repopulate the data container objects (after deserializing the response with Jackson)
Create a data frame (an example)
Process the data frame (dataset of rows)
NOTE: The JSON structure you have provided seems not to be correct, JSON array is [{},{},{}]
In your case, given the format of the request JSON, direct conversion of rows will not work, as mentioned in point 1, make a set of model classes, you could consider the below model classes.
package org.test.json;
import java.util.List;
public class RequestModel {
protected ApplicationInfo applicationInfo;
protected List<RequestData> requestData;
public ApplicationInfo getApplicationInfo() {return applicationInfo;}
public void setApplicationInfo(ApplicationInfo applicationInfo) {this.applicationInfo = applicationInfo;}
public List<RequestData> getRequestData() {return requestData;}
public void setRequestData(List<RequestData> requestData) {this.requestData = requestData;}
}//class closing
package org.test.json;
public class ApplicationInfo {
protected String appId;
public String getAppId() {return appId;}
public void setAppId(String appId) {this.appId = appId;}
}//class closing
package org.test.json;
public class RequestData {
protected String secureData;
protected String secureDataType;
protected int index;
public String getSecureData() {return secureData;}
public void setSecureData(String secureData) {this.secureData = secureData;}
public String getSecureDataType() {return secureDataType;}
public void setSecureDataType(String secureDataType) {this.secureDataType = secureDataType;}
public int getIndex() {return index;}
public void setIndex(int index) {this.index = index;}
}//class closing
Process the list as obtained from the data frame and populate the model classes and then convert with Jackson to get the request JSON.
The below should do what you are looking for, don't directly run this, the data set is null
//Do not run this, will generate NullPointer, for example only
Dataset<Row> ds=null;
List<Row> rows=ds.collectAsList();
RequestModel request=new RequestModel();
//Set application id
ApplicationInfo appInfo=new ApplicationInfo();
appInfo.setAppId("some id");
request.setApplicationInfo(appInfo);
List<RequestData> reqData=new ArrayList<>();
for(int i=0;i<rows.size();i++) {
//Incrementally generated for each row
int index=i;
Row r=rows.get(i);
int rowLength=r.size();
for(int j=0;j<rowLength;j++) {
RequestData dataElement=new RequestData();
dataElement.setIndex(index);
switch(j) {
case 1:{dataElement.setSecureData(r.getString(j));dataElement.setSecureDataType("PID");break;}
case 3:{dataElement.setSecureDataType(r.getString(j));dataElement.setSecureDataType("FIRSTNAME");break;}
default:{break;}
}//switch closing
reqData.add(dataElement);
}//for closing
}//for closing

Is there a way to diff two Avro GenericRecords in Java to obtain only the fields and values that changed?

I would like to write Java code to diff two GenericRecords saved in memory, both with the same schema. I've been unable to find any libraries or any Apache Avro Javadoc on how to do this.
My desire is to be as efficient as possible, and converting the GenericRecords to JSON, then diffing them, then converting the JSON back to GenericRecord would be undesirable unless that is the only option that exists.
protected GenericRecord generateDeltaFieldsOnly(GenericRecord storedRecord, GenericRecord newRecord)
{
ObjectMapper objectMapper = new ObjectMapper();
JsonNode storedRecordJson = objectMapper.readTree(storedRecord.toString());
JsonNode newRecordJson = objectMapper.readTree(newRecord.toString());
}
Sorry just seeing this now...
Just keep both as genericRecords and from the schema just iterate through the fields changing the ones you want. I sort of do this but use genericRecords from two different schemas and mark up the schema description with keywords that inform the basically dumb transformer to do the mapping.
#Override
public GenericRecord transform(GenericRecord source, GenericRecord reference) {
GenericRecordBuilder builder = new GenericRecordBuilder(targetSchema);
buildGenericRecord(builder, source);
buildGenericRecord(builder, reference);
return builder.build();
}
private void buildGenericRecord(GenericRecordBuilder builder, GenericRecord genericRecord) {
String fullName = genericRecord.getSchema().getFullName();
String targetField;
for (Entry<String, String> entry : fromFieldToField.entrySet()) {
if (entry.getKey().startsWith(fullName)) {
targetField = entry.getKey().substring(entry.getKey().lastIndexOf('.') + 1);
builder.set(entry.getValue(), genericRecord.get(targetField));
}
}
}
For yours it is much easier if you are using the same schema.
for (Field field : storedRecord.getSchema().getFields()) {
//do your mapping field to field
}

Jersey/Genson: Unmarschalling single object array

Similar to Jersey: Json array with 1 element is serialized as object BUT on the client side. E.g. I recieve a JSON object where a field is an array regulary, but in case there is only one element, it is a single object.
{"fileInfo":[{"fileName":"weather.arff","id":"10"},"fileName":"supermarket.arff","id":"11"}]}
versus
{"fileInfo":{"fileName":"weather.arff","id":"10"}}
I'm parsing/unmarshalling the JSON using Jersey/Genson. Of course, if the JSON doesnt match the target class I recieve an error (such as expected [ but read '{' )
I've read a lot about this bug and how to avoid when creating JSON objects on the SERVER side, but I found nothing about how to handle this issus when dealing on the CLIENT side.
As always, I prefere the most codeless possibility if there are several solutions...
BTW: Moxy works but it does not marshal native Object-type objects which is another requirement...
Update
Starting with Genson 1.3 release you can achieve it by enabling permissiveParsing:
Genson genson = new GensonBuilder().usePermissiveParsing(true).create();
Answer
Uh, do you know what library produces this on server side? I am curious to see who is responsible for all those badly structured jsons out there...
It is not yet supported in Genson. Originally because IMO people should not produce such dynamic json. Anyway, I opened an issue - this can be easily done, you can expect it to be present in the release coming next week.
Otherwise here is a way to achieve it without breaking the existing mechanisms.
You need to register a Factory that will use Gensons collections factory to create an instance of its standard collection converter. Then you will wrap this converter in another one that will handle the object to array logic. Here is the code (not codeless..., but if you wait a bit you won't have to code :)).
import com.owlike.genson.convert.DefaultConverters.CollectionConverterFactory;
import com.owlike.genson.convert.DefaultConverters.CollectionConverterFactory;
class SingleObjectAsCollectionFactory implements Factory<Converter<Collection>> {
// get the default factory
Factory<Converter<Collection<?>>> defaultFactory = CollectionConverterFactory.instance;
#Override
public Converter<Collection> create(Type type, Genson genson) {
// obtain an instance of the correct default converter for this type
final CollectionConverter defaultConverter = (CollectionConverter) defaultFactory.create(type, genson);
// wrap it in your own converter
return new Converter<Collection>() {
#Override
public void serialize(Collection object, ObjectWriter writer, Context ctx) throws Exception {
defaultConverter.serialize(object, writer, ctx);
}
#Override
public Collection deserialize(ObjectReader reader, Context ctx) throws Exception {
if (reader.getValueType() == ValueType.OBJECT) {
Object object = defaultConverter.getElementConverter().deserialize(reader, ctx);
Collection result = defaultConverter.create();
result.add(object);
return result;
} else return defaultConverter.deserialize( reader, ctx );
}
};
}
}
And then register it
Genson genson = new GensonBuilder()
.withConverterFactory(new SingleObjectAsCollectionFactory())
.create();

Can the GXT JsonReader (using AutoBeans) parse complex JSON structures in one request?

I need to work with data returned from a service which has a more complex JSON structure than the examples provided in the GXT documentation and thus far I cannot find any instructions or example which demonstrates how this might be accomplished.
The JSON contains multiple key/value pairs, but some of the key/value pairs are collections. I can have all of the data returned to me in one call from the service in the proper structure, but there does not appear to be a way to parse the data into separate entities. In my particular case I am attempting to configure a loader which will process one of the collections but I also need other key/value pairs from the same message (it is not ok to have the loader make one call and then have another call made for the same data and retrieve the other key/value pairs). Is there any way to accomplish this using GXT3?
Example: let's assume I can make a request from a server which returns JSON containing the name of an author along with a collection of the books the author has written. I want to display the author's name above a grid which lists the books. I want only one request made to the server and then have my view display the author in one component and the book list in a grid. Assume I need a loader instead of just a store as the grid may have to make additional calls (e.g. if it is a paging grid, livegrid, etc.).
Example JSON: (one JSON message returned with and author element along with a collection of book elements - I've indented the JSON to illustrate the structure)
{ "returnData" :
{"author" : "AuthorName"},
{"books" :
{"id" : "1", "name" : "Book1"},{"id" : "2", "name" : "Book2"}
}
}
Using the example for JsonReader (see the javadoc for an example) I can receive the request and parse the links into a collection using AutoBeans. This works fine when I need to have those retrieved and parsed in a loader. However, if I do that then the other properties are ignored. I currently don't see any way to parse the other values in the same request so they can be used elsewhere. My example code for the collection processing is below:
// this is the root JSON object, the AuthorRecord
public interface AuthorRecord {
#PropertyName(value="author")
String getAuthor();
#PropertyName(value="author")
void setAuthor(String author);
#PropertyName(value="books")
List<Book> getBooks();#
#PropertyName(value="books")
void setBooks (List<Book> books);
}
// models the book objects returned
public interface Book {
#PropertyName(value="id")
String getId();
#PropertyName(value="id")
void setId(String id);
#PropertyName(value="name")
String getName();
#PropertyName(value="name")
void setName(String name);
}
public interface ReturnData {
AuthorRootObject getAuthorRoot();
}
public interface LibraryAutoBeanFactory extends AutoBeanFactory {
AutoBean<ReturnData> authorRecord();
AutoBean<ListLoadConfig> loadConfig();
}
public class ReturnDataJsonReader extends JsonReader<ListLoadResult<Book>,
ReturnData> {
public ReturnDataJsonReader(AutoBeanFactory factory,
Class<ReturnData> rootBeanType) {
super(factory, rootBeanType);
}
#Override
protected ListLoadResultBean<Book> createReturnData(Object loadConfig,
ReturnData incomingData) {
return new ListLoadResultBean<Book>(incomingData.getBooks());
}
}
The problem I was having was that I need to have a view that includes a grid (paging grid, etc.) which lists out the books, while having the Author's name sit above the grid. I wanted to get all of this information (or at least the first page of results) with only one request to the server since the JSON message contains all the information I need to accomplish this. The problem is that the loader makes the request and receives the response, and it expects that the reader it will use is going to process a collection. In my case, I need the loader to process the collection of books but also populate another data field. The solution I found was to create an arbitrary collection to pass to the loader and then implement my own load handler to process the return object as needed.
1.The JSON being returned is really just one object of type ReturnData. The extended JsonReader could process this using AutoBeans, but if the reader is to be used for the loader, it needs to return a collection. Therefore, override the createReturnData() method to return a collection of one object.
public class ReturnDataJsonReader extends JsonReader<ListLoadResult<AuthorRecord>,
ReturnData> {
public ReturnDataJsonReader(AutoBeanFactory factory, Class<ReturnData> rootBeanType)
{
super(factory, rootBeanType);
}
#Override
protected ListLoadResultBean<AuthorRecord> createReturnData(Object loadConfig,
ReturnData incomingData) {
List<AuthorRecord> authorDataCollection = new ArrayList<AuthorRecord>();
authorDataCollection.add(incomingData);
return authorDataCollection;
}
}
2.The LoadHandler used in the examples takes a ListStore as an input and populates it with the results from the loader. Since the return object is not what we want populating the loader, and since we need to populate another property on the view, create your own LoadHandler to take the objects needed as input and populate them:
View Class Example:
public class ExampleViewClass {
// truncating most of the code in here for brevity
// note some of the objects referenced here reference objects in the question
private String authorName;
private ListStore<Book> bookList;
// IMPORTANT - create your own LoadHandler
private class LibraryLoadResultistStoreBinding<C, M, D extends
ListLoadResult<AuthorRecord>> implements LoadHandler<ListLoadConfig,
ListLoadResult<AuthorRecord>> {
private final ListStore<Book> bookStore;
private final String authorName;
public LibraryLoadResultistStoreBinding(ListStore<Book> books, String author) {
this.bookStore = books;
this.authorName = author;
}
#Override
public void onLoad(LoadEvent<ListLoadConfig, ListLoadResult<AuthorRecord> event)
{
// the response object
AuthorRecord response = event.getLoadResult().getData().get(0);
bookStore.replaceAll(response.getBooks());
author = response.getAuthor();
}
}
// example uses an HttpProxy but that's not required
public void populateView() {
LibraryAutoBeanFactory factory = GWT.create(LibraryAutoBeanFactory.class);
ReturnDataJsonReader reader = new ReturnDataJsonReader(factory, ReturnData.class);
String path = "http://path.to.resource/getinfo";
RequestBuilder builder = new RequestBuilder(RequestBuilder.GET, path);
HttpProxy<ListLoadConfig> proxy = new HttpProxy<ListLoadConfig>(builder);
final ListLoader<ListLoadConfig, ListLoadResult<AuthorRecord>> loader = new
ListLoader<ListLoadConfig, ListLoadResult<AuthorRecord>> (proxy, reader);
loader.useLoadConfig(ReturnDataAutoBeanFactory.instance.loadConfig().as();
loader.addLoadHandler(new LibraryLoadResultistStoreBinding<ListLoadConfig,
AuthorRecord, ListLoadResult<AuthorRecord>>(bookList, authorName);
// pass in the objects to be populated
loader.load(); // fire the loader
}

How to serialize in jackson json null string to empty string

I need jackson json (1.8) to serialize a java NULL string to an empty string. How do you do it?
Any help or suggestion is greatly appreciated.
Thanks
See the docs on Custom Serializers; there's an example of exactly this, works for me.
In case the docs move let me paste the relevant answer:
Converting null values to something else
(like empty Strings)
If you want to output some other JSON value instead of null (mainly
because some other processing tools prefer other constant values --
often empty String), things are bit trickier as nominal type may be
anything; and while you could register serializer for Object.class, it
would not be used unless there wasn't more specific serializer to use.
But there is specific concept of "null serializer" that you can use as
follows:
// Configuration of ObjectMapper:
{
// First: need a custom serializer provider
StdSerializerProvider sp = new StdSerializerProvider();
sp.setNullValueSerializer(new NullSerializer());
// And then configure mapper to use it
ObjectMapper m = new ObjectMapper();
m.setSerializerProvider(sp);
}
// serialization as done using regular ObjectMapper.writeValue()
// and NullSerializer can be something as simple as:
public class NullSerializer extends JsonSerializer<Object>
{
public void serialize(Object value, JsonGenerator jgen,
SerializerProvider provider)
throws IOException, JsonProcessingException
{
// any JSON value you want...
jgen.writeString("");
}
}