Circular references not handled in Avro - json

Question

There is a tool called Avro-Tools which ships with Avro and can be used to convert between JSON, Avro-Schema (.avsc) and binary formats.
But it does not work with circular references.
We have two files:
circular.avsc (generated by Avro)
circular.json (generated by Jackson because it has circular reference and Avro doesn't like the same).
circular.avsc
{
"type":"record",
"name":"Parent",
"namespace":"bigdata.example.avro",
"fields":[
{
"name":"name",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"child",
"type":[
"null",
{
"type":"record",
"name":"Child",
"fields":[
{
"name":"name",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"parent",
"type":[
"null",
"Parent"
],
"default":null
}
]
}
],
"default":null
}
]
}
circular.json
{
"#class":"bigdata.example.avro.Parent",
"#circle_ref_id":1,
"name":"parent",
"child":{
"#class":"bigdata.example.avro.DerivedChild",
"#circle_ref_id":2,
"name":"hello",
"parent":1
}
}
Command to run avro-tools on the above
java -jar avro-tools-1.7.6.jar fromjson --schema-file circular.avsc circular.json
Output
2014-06-09 14:29:17.759 java[55860:1607] Unable to load realm mapping info from SCDynamicStore
Objavro.codenullavro.schema?
{"type":"record","name":"Parent","namespace":"bigdata.example.avro","fields":[{"name":"name","type":["null","string"],"default":null},{"name":"child","type":["null",{"type":"record","name":"Child","fields":[{"name":"name","type":["null","string"],"default":null},{"name":"parent","type":["null","Parent"],"default":null}]}],"default":null}]}?'???K?jH!??Ė?Exception in thread "main" org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
Some other JSON values tried with the same schema but that did not work
JSON 1
{
"name":"parent",
"child":{
"name":"hello",
"parent":null
}
}
JSON 2
{
"name":"parent",
"child":{
"name":"hello",
}
}
JSON 3
{
"#class":"bigdata.example.avro.Parent",
"#circle_ref_id":1,
"name":"parent",
"child":{
"#class":"bigdata.example.avro.DerivedChild",
"#circle_ref_id":2,
"name":"hello",
"parent":null
}
}
Removing some of the "optional" elements:
circular.avsc
{
"type":"record",
"name":"Parent",
"namespace":"bigdata.example.avro",
"fields":[
{
"name":"name",
"type":
"string",
"default":null
},
{
"name":"child",
"type":
{
"type":"record",
"name":"Child",
"fields":[
{
"name":"name",
"type":
"string",
"default":null
},
{
"name":"parent",
"type":
"Parent",
"default":null
}
]
},
"default":null
}
]
}
circular.json
{
"#class":"bigdata.example.avro.Parent",
"#circle_ref_id":1,
"name":"parent",
"child":{
"#class":"bigdata.example.avro.DerivedChild",
"#circle_ref_id":2,
"name":"hello",
"parent":1
}
}
output
2014-06-09 15:30:53.716 java[56261:1607] Unable to load realm mapping info from SCDynamicStore
Objavro.codenullavro.schema?{"type":"record","name":"Parent","namespace":"bigdata.example.avro","fields":[{"name":"name","type":"string","default":null},{"name":"child","type":{"type":"record","name":"Child","fields":[{"name":"name","type":"string","default":null},{"name":"parent","type":"Parent","default":null}]},"default":null}]}?x?N??O"?M?`AbException in thread "main" java.lang.StackOverflowError
at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:212)
at org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:323)
at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:216)
at org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:323)
at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:216)
at org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:323)
Does anyone know how I can make circular reference work with Avro?

Answer 1

I met this same problem recently and resolved in a work-around way, hopefully it could help.
Based on the Avro specification:
JSON Encoding
Except for unions, the JSON encoding is the same as is used to encode field default values.
The value of a union is encoded in JSON as follows:
if its type is null, then it is encoded as a JSON null;
otherwise it is encoded as a JSON object with one name/value pair whose name is the type's name and whose value is the recursively encoded value. For Avro's named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.
For example, the union schema ["null","string","Foo"], where Foo is a record name, would encode:
null as null;
the string "a" as {"string": "a"};
and a Foo instance as {"Foo": {...}}, where {...} indicates the JSON encoding of a Foo instance.
If the source file could not be changed to follow the requirement, maybe we have to change the code. So I customized the original org.apache.avro.io.JsonDecoder class from avro-1.7.7 package and created my own class MyJsonDecoder.
Here is the key placed I changed besides create new constructors and class name:
#Override
public int readIndex() throws IOException {
advance(Symbol.UNION);
Symbol.Alternative a = (Symbol.Alternative) parser.popSymbol();
String label;
if (in.getCurrentToken() == JsonToken.VALUE_NULL) {
label = "null";
//***********************************************
// Original code: according to Avor document "JSON Encoding":
// it is encoded as a Json object with one name/value pair whose name is
// the type's name and whose value is the recursively encoded value.
// Can't change source data, so remove this rule.
// } else if (in.getCurrentToken() == JsonToken.START_OBJECT &&
// in.nextToken() == JsonToken.FIELD_NAME) {
// label = in.getText();
// in.nextToken();
// parser.pushSymbol(Symbol.UNION_END);
//***********************************************
// Customized code:
// Add to check if type is in the union then parse it.
// Check if type match types in union or not.
} else {
label = findTypeInUnion(in.getCurrentToken(), a);
// Field missing but not allow to be null
// or field type is not in union.
if (label == null) {
throw error("start-union, type may not be in UNION,");
}
}
//***********************************************
// Original code: directly error out if union
// } else {
// throw error("start-union");
// }
//***********************************************
int n = a.findLabel(label);
if (n < 0)
throw new AvroTypeException("Unknown union branch " + label);
parser.pushSymbol(a.getSymbol(n));
return n;
}
/**
* Method to check if current JSON token type is declared in union.
* Do NOT support "record", "enum", "fix":
* Because there types require user defined name in Avro schema,
* if user defined names could not be found in Json file, can't decode.
*
* #param jsonToken JsonToken
* #param symbolAlternative Symbol.Alternative
* #return String Parsing label, decode in which way.
*/
private String findTypeInUnion(final JsonToken jsonToken,
final Symbol.Alternative symbolAlternative) {
// Create a map for looking up: JsonToken and Avro type
final HashMap<JsonToken, String> json2Avro = new HashMap<>();
for (int i = 0; i < symbolAlternative.size(); i++) {
// Get the type declared in union: symbolAlternative.getLabel(i).
// Map the JsonToken with Avro type.
switch (symbolAlternative.getLabel(i)) {
case "null":
json2Avro.put(JsonToken.VALUE_NULL, "null");
break;
case "boolean":
json2Avro.put(JsonToken.VALUE_TRUE, "boolean");
json2Avro.put(JsonToken.VALUE_FALSE, "boolean");
break;
case "int":
json2Avro.put(JsonToken.VALUE_NUMBER_INT, "int");
break;
case "long":
json2Avro.put(JsonToken.VALUE_NUMBER_INT, "long");
break;
case "float":
json2Avro.put(JsonToken.VALUE_NUMBER_FLOAT, "float");
break;
case "double":
json2Avro.put(JsonToken.VALUE_NUMBER_FLOAT, "double");
break;
case "bytes":
json2Avro.put(JsonToken.VALUE_STRING, "bytes");
break;
case "string":
json2Avro.put(JsonToken.VALUE_STRING, "string");
break;
case "array":
json2Avro.put(JsonToken.START_ARRAY, "array");
break;
case "map":
json2Avro.put(JsonToken.START_OBJECT, "map");
break;
default: break;
}
}
// Looking up the map to find out related Avro type to JsonToken
return json2Avro.get(jsonToken);
}
The generate idea is to check the type from source file could be found in union or not.
Here still has some issues:
This solution doesn't support "record", "enum", or "fixed" Avro type because these types require user defined name. E.g. if you want union "type": ["null", {"name": "abc", "type": "record", "fields" : ...}], this code will not work. For Primitive type, this should work. But please test it before your use it for your project.
Personally I think records should not be null because I consider records are what I need to make sure exists, if something is missing, that means I have bigger problem. If it could be omit, I prefer to use "map" as type instead of using "record" when you define the schema.
Hopefully this could help.