I am using FFProbe to get media file information in JSON format.
I am looking for a complete schema definition for the JSON output option in FFProbe.
See: https://ffmpeg.org/ffprobe.html#json
Without the schema I find that different files produce different output, and I have to add more serialization logic by hand as I discover more properties and more tags in the JSON.
Something equivalent to MkvToolNix's full JSON schema definition, but for FFProbe:
See: https://gitlab.com/mbunkus/mkvtoolnix/-/blob/master/doc/json-schema/mkvmerge-identification-output-schema-v12.json
Any ideas if such a schema exists for FFProbe?
There isn't one but there is a XML schema which you could try to convert. It's at https://github.com/FFmpeg/FFmpeg/blob/master/doc/ffprobe.xsd
Related
I'm able to successfully generate JSON file from Maximo however I would like to modify the JSON before it gets generated. Like below is the sample JSON that gets generated in Maximo,
{"lastreadingdate":"2020-01-30T16:48:33+01:00",
"linearassetmeterid":0,
"sinceinstall":0.0,
"lastreading":"1,150",
"plustinitrdng":0.0,
"sincelastinspect":0.0,
"_rowstamp":"568349195",
"assetnum":"RS100003",
"active":true,
"assetmeterid":85,
"lifetodate":0.0,
"measureunitid":"KWH",
"metername":"1010",
"remarks":"TESTING JSON"}
I need the JSON to be generated as below ,
{"spi:action": "OSLC draft",
"spi:tri1readingdate":"2020-01-30T16:48:33+01:00",
"spi:tryassetmeterid":0,
"spi:install":0.0,
"spi:lastreadingTx":"1,150",
"spi:intrdngtrX":0.0,
and so on...}
Basically I need to change the target attribute names and prefix "spi" Below is the error occuring in JSON Mapping .
You're not specifying how you generate the JSON file but I'll quickly explain how you can achieve this:
As Dex pointed out, there is a JSON Mapping app in the integration module that you can use to map your outbound object structure's fields to your target structure naming.
You define your JSON structure on the JSON Mapping tab by providing a JSON sample.
You then define your mapping with Maximo on the Properties tab, like this:
Reading this IBM doc before jumping right into it should help you a lot:
https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/02db2a84-fc66-4667-b760-54e495526ec1/page/e10f6e96-435d-433c-8259-5690eb756779/attachment/169224c7-10a5-4cee-af72-697a476f8b2e/media/JSON
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, #Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
I've a large JSON string, I want to convert this string into Erlang record.
I found jiffy library but it doesn't completely convert to record.
For example:
jiffy:decode(<<"{\"foo\":\"bar\"}">>).
gives
{[{<<"foo">>,<<"bar">>}]}
but I want the following output:
{ok,{obj,[{"foo",<<"bar">>}]},[]}
Is there any library that can be used for the desired output?
Or is there any library that can be used in combination of jiffy for further modifying the output of it.
Consider the fact the JSON string is large, and I want the output is minimum time.
Take a look at ejson, from the documentation:
JSON library for Erlang on top of jsx. It gives a declarative interface for jsx by which we need to specify conversion rules and ejson will convert tuples according to the rules.
I made this library to make easy not just the encoding but rather the decoding of JSONs to Erlang records...
In order for ejson to take effect the source files need to be compiled with parse_transform ejson_trans. All record which has -json attribute can be converted to JSON later.
I want to delete/ignore the elements in the following json record:
{"_scroll_id":"==","timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":6908915,"max_score":null,"hits":[{"_index":"abc_v1","_type":"composite_request_response_v1","_id":"123","_score":1.0,"_source":{"response":{"testResults":{"docsisResults":{"devices":[{"upstreamSection":{"upstreams":[]},"fluxSection":{"fluxInfo":[{}]}}],"events":[]},"mocaResults":{"statuses":[]}}}},"sort":[null,1.0]}]}},
I have the records in the above format. I wish to delete the highlighted part of the record. Can someone guide me of ways I can accomplish that. Are there anyways I can achieve that using hive/pig/linux/python?
There is the JSON SerDe in Hive, see this: https://cwiki.apache.org/confluence/display/Hive/Json+SerDe
So you can define only columns that you need in table definition, put your file in the table location and then select only defined columns. Alternatively you can pre-process/transform your files before loading them using Java+ Jackson (library to serialize or map Java objects to JSON and vice versa), this will give you maximum flexibility thought this is not so simple as using JSON SerDe.
Using MarkLogic 8, I'm using a custom XML to JSON conversion for json:transform-to-json, and I've got it working just about right except the conversion is outputting numbers as strings.
Is there a way to specify that the value of a particular element should be a number value, not a string?
I don't see anything in the doc for json:config, but just in case there's something I've missed, or if you have a neat post-processing trick, I'd love to hear about how to solve this problem.
You can do that by defining an XML Schema for the non-string type elements. Just make sure it is available in the context (by loading it into xdmp:schemas-database()), and that it is recognized (your XML needs to have a namespace that matches the XML Schema, and you might wanna use import schema)..
HTH!