U-SQL JsonTuple - How to access specific fields in JSON array - json

Question

I'm extracting AVRO data which has a JSON field that I need to get values from. The JSON has an array, and I don't know what order the different elements of the array may appear in. How can I target specific node/values?
For example, Filters[0] could be Category one time, but could be AddressType another time.
I'm extracting AVRO data - i.e.
#rs =
EXTRACT date DateTime,
Body byte[]
FROM #input_file
USING new Microsoft.Analytics.Samples.Formats.ApacheAvro.AvroExtractor(#"
...
The Body is JSON that can look like this (but Category is not always Filter[0]. This is a small example; there are 7 different types of "Field"s):
{
""TimeStamp"": ""2019-02-19T15:00:29.1067771-05:00"",
""Filters"": [{
""Operator"": ""eq"",
""Field"": ""Category"",
""Value"": ""Sale""
}, {
""Operator"": ""eq"",
""Field"": ""AddressType"",
""Value"": ""Home""
}
]
}
My U-SQL looks like this, which of course does not always work.
#keyvalues =
SELECT JsonFunctions.JsonTuple(Encoding.UTF8.GetString(Body),
"TimeStamp",
"$.Filters[?(#.Field == 'Category')].Value",
"$.Filters[?(#.Field == 'AddressType')].Value"
) AS message
FROM #rs;
#results =
SELECT
message["TimeStamp"] AS TimeStamp,
message["Filters[0].Value"] AS Category,
message["Filters[1].Value"] AS AddressType
FROM #keyvalues;

Answer 1

Although this does not actually answer my question, as a workaround, I modified the Microsoft 'sample' JsonFunctions.JsonTuple method to be able to specify my own key name for extracted values:
/// Added - Prefix a path expression with a specified key. Use key~$e in the expression.
/// eg:
/// JsonTuple(json, "myId~id", "myName~name") -> field names MAP{ {myId, 1 }, {myName, Ed } }
The modified code:
private static IEnumerable<KeyValuePair<string, T>> ApplyPath<T>(JToken root, string path)
{
var keySeparatorPos = path.IndexOf("~");
string key = null;
var searchPath = path;
if (keySeparatorPos > 0) // =0?if just a leading "=", i.e. no key provided, then don't parse out a key.
{
key = path.Substring(0, keySeparatorPos).Trim();
searchPath = path.Substring(keySeparatorPos + 1);
}
// Children
var children = SelectChildren<T>(root, searchPath);
foreach (var token in children)
{
// Token => T
var value = (T)JsonFunctions.ConvertToken(token, typeof(T));
// Tuple(path, value)
yield return new KeyValuePair<string, T>(key ?? token.Path, value);
}
}
For example, I can access vales and name them
#keyvalues =
SELECT JsonFunctions.JsonTuple(Encoding.UTF8.GetString(Body),
"TimeStamp",
"EventName",
"Plan~ $.UrlParams.plan",
"Category~ $.Filters[?(#.Field == 'Category')].Value",
"AddressType~ $.Filters[?(#.Field == 'AddressType')].Value"
) AS message
FROM #rs;
#results =
SELECT
message["TimeStamp"] AS TimeStamp,
message["EventName"] AS EventName,
message["Plan"] AS Plan,
message["Category"] AS Category,
message["AddressType"] AS AddressType
FROM #keyvalues;
(I've not tested to see what would happen if the same Field appears multiple times in the array; That won't happen in my case)