Use Jackson To Stream Parse an Array of Json Objects - json

Question

I have a file that contains a json array of objects:
[
{
"test1": "abc"
},
{
"test2": [1, 2, 3]
}
]
I wish to use use Jackson's JsonParser to take an inputstream from this file, and at every call to .next(), I want it to return an object from the array until it runs out of objects or fails.
Is this possible?
Use case:
I have a large file with a json array filled with a large number of objects with varying schemas. I want to get one object at a time to avoid loading everything into memory.
EDIT:
I completely forgot to mention. My input is a string that is added to over time. It slowly accumulates json over time. I was hoping to be able to parse it object by object removing the parsed object from the string.
But I suppose that doesn't matter! I can do this manually so long as the jsonParser will return the index into the string.

Answer 1

Yes, you can achieve this sort of part-streaming-part-tree-model processing style using an ObjectMapper:
ObjectMapper mapper = new ObjectMapper();
JsonParser parser = mapper.getFactory().createParser(new File(...));
if(parser.nextToken() != JsonToken.START_ARRAY) {
throw new IllegalStateException("Expected an array");
}
while(parser.nextToken() == JsonToken.START_OBJECT) {
// read everything from this START_OBJECT to the matching END_OBJECT
// and return it as a tree model ObjectNode
ObjectNode node = mapper.readTree(parser);
// do whatever you need to do with this object
}
parser.close();

Answer 2

What you are looking for is called Jackson Streaming API. Here is a code snippet using Jackson Streaming API that could help you to achieve what you need.
JsonFactory factory = new JsonFactory();
JsonParser parser = factory.createJsonParser(new File(yourPathToFile));
JsonToken token = parser.nextToken();
if (token == null) {
// return or throw exception
}
// the first token is supposed to be the start of array '['
if (!JsonToken.START_ARRAY.equals(token)) {
// return or throw exception
}
// iterate through the content of the array
while (true) {
token = parser.nextToken();
if (!JsonToken.START_OBJECT.equals(token)) {
break;
}
if (token == null) {
break;
}
// parse your objects by means of parser.getXxxValue() and/or other parser's methods
}

Answer 3

This example reads custom objects directly from a stream:
source is a java.io.File
ObjectMapper mapper = new ObjectMapper();
JsonParser parser = mapper.getFactory().createParser( source );
if ( parser.nextToken() != JsonToken.START_ARRAY ) {
throw new Exception( "no array" );
}
while ( parser.nextToken() == JsonToken.START_OBJECT ) {
CustomObj custom = mapper.readValue( parser, CustomObj.class );
System.out.println( "" + custom );
}

Answer 4

This is a late answer that builds on Ian Roberts' answer. You can also use a JsonPointer to find the start position if it is nested into a document. This avoids custom coding the slightly cumbersome streaming token approach to get to the start point. In this case, the basePath is "/", but it can be any path that JsonPointer understands.
Path sourceFile = Paths.get("/path/to/my/file.json");
// Point the basePath to a starting point in the file
JsonPointer basePath = JsonPointer.compile("/");
ObjectMapper mapper = new ObjectMapper();
try (InputStream inputSource = Files.newInputStream(sourceFile);
JsonParser baseParser = mapper.getFactory().createParser(inputSource);
JsonParser filteredParser = new FilteringParserDelegate(baseParser,
new JsonPointerBasedFilter(basePath), false, false);) {
// Call nextToken once to initialize the filteredParser
JsonToken basePathToken = filteredParser.nextToken();
if (basePathToken != JsonToken.START_ARRAY) {
throw new IllegalStateException("Base path did not point to an array: found "
+ basePathToken);
}
while (filteredParser.nextToken() == JsonToken.START_OBJECT) {
// Parse each object inside of the array into a separate tree model
// to keep a fixed memory footprint when parsing files
// larger than the available memory
JsonNode nextNode = mapper.readTree(filteredParser);
// Consume/process the node for example:
JsonPointer fieldRelativePath = JsonPointer.compile("/test1");
JsonNode valueNode = nextNode.at(fieldRelativePath);
if (!valueNode.isValueNode()) {
throw new IllegalStateException("Did not find value at "
+ fieldRelativePath.toString()
+ " after setting base to " + basePath.toString());
}
System.out.println(valueNode.asText());
}
}