I m trying to use 'ffprobe' with Java or groovy - json

As per my understanding "ffprobe" will provide file related data in JSON format. So, I have installed the ffprobe in my Ubuntu machine but I don't know how to access the ffprobe JSON response using Java/Grails.
Expected response format:
{
"format": {
"filename": "/Users/karthick/Documents/videos/TestVideos/sample.ts",
"nb_streams": 2,
"nb_programs": 1,
"format_name": "mpegts",
"format_long_name": "MPEG-TS (MPEG-2 Transport Stream)",
"start_time": "1.430800",
"duration": "170.097489",
"size": "80425836",
"bit_rate": "3782576",
"probe_score": 100
}
}
This is my groovy code
def process = "ffprobe -v quiet -print_format json -show_format -show_streams HelloWorld.mpeg ".execute()
println "Found ${process.text}"
render process as JSON
I m able to get the process object and i m not able to get the json response
Should i want to convert the process object to json object?
OUTPUT:
Found java.lang.UNIXProcess#75566697
org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error converting Bean with class java.lang.UNIXProcess

Grails has nothing to do with this. Groovy can execute arbitrary shell commands in a very simplistic way:
"mkdir foo".execute()
Or for more advanced features, you might look into using ProcessBuilder. At the end of the day, you need to execute ffprobe and then capture the output stream of JSON to use in your app.

Groovy provides a simple way to execute command line processes. Simply
write the command line as a string and call the execute() method.
The execute() method returns a java.lang.Process instance.
println "ffprobe <options>".execute().text
[Source]

Related

I can't use Get Value From Json with JsonPAth parameter in Robot Framework - always have a KeyError

When I try to use "Get Value From Json" on a Json with specific JsonPath , I always have KeyError
even web simple example doesn't work for me...
When I try that code :
Library JSONLibrary
*** Test Cases ***
Example
${tmp} = Convert String To JSON {"foo": {"bar": [1,2,3]}}
Log to console tmp: ${tmp}
${values_test}= Get Value From Json ${tmp} $.foo.bar
Log to console values_test: ${values_test}
I always have this kind of Errors and log :
tmp: {'foo': {'bar': [1, 2, 3]}}
...
Resolving variable '${tmp['$.foo.bar']}' failed: KeyError: '$.foo.bar'
Can somebody Help me please ?
it is really basic example by the way and community always says that it works like that in comments..

Unable to parse nested json in logstash

My application generates the below mentioned mulogs, which is in a nested json logs. While am trying to parse it in kibana, the json parse fails. Below is my log sample,
2022-08-04T12:43:03.977Z {"tags":
{"caller":"sphe",
"job-id":"1",
"mulog/duration":"3180930",
"mulog/namespace":"tool.utilities.db",
"mulog/outcome":":ok",
"user-name":"Pol",
"type":":execute!",
"app-name":"kan",
"mulog/parent-trace":"_YznrMCc",
"user-id":"52-7d4128fb7cb7",
"sql":"SELECT data FROM kan.material_estimate_history WHERE job_id = '167aa1cc' ",
"result":"[]",
"within-tx":"false",
"mulog/root-trace":"S0yn8jclLsmNVyKpH",
"mulog/timestamp":"1659616983977",
"uri":"/api/kan/material-estimates/job/f14b167aa1cc",
"mulog/trace-id":"kI4grnAMe4bGmFc_aX",
"request-method":":get",
"mulog/event-name":":kan=.source.db.material-estimates/find-history-by-job-id"},
"localEndpoint":{"serviceName":"kan"},
"name":"kan.source.db.material-estimates/find-history-by-job-id",
"traceId":"112721d07ecc9be",
"duration":3180,"id":"c90a259a2",
"kind":"SERVER","timestamp":1659616983977000,
"parentId":"dd7368"}

Robot Framework bypass HTTPError: 400 Client Error: Bad Request

I have a request which tend to upload a file, if a file with the same name already exists it throws a message that the file already exists. This can be considered as expected result and even though the error I would the test to pass as it is.
This is the code I am using:
Create Session mysession ${test_env}
&{headers} Create Dictionary Content-Type=application/json; charset=utf-8 Authorization=${token}
${json}= Catenate { "FileName": "File.txt", "Content": "PD94bWwg..", "UserId": "email.com" }
${value} Set Variable 2
${value} Convert To Integer ${value}
${json}= Evaluate json.loads('''${json}''') json
#Set To Dictionary ${json["FileName"]}
${json}= Evaluate json.dumps(${json}) json
${resp} POST url=${test_env}/api/nt data=${json} headers=${headers}
${log}= Log To Console ${resp.status_code} 400
Log To Console ${resp.content}
Status Should Be expected_status=any
The test stops at the POST request and does not want to read the expected_status=any and consider the test as pass.
I would appreciate any hints on how to make it pass.
Below code will verify the 400 error and will continue further execution
Run Keyword And Expect Error HTTPError: 400* POST url=${test_env}/api/nt data=${json} headers=${headers}

Parse JSON with missing fields using cjson Lua module in Openresty

I am trying to parse a json payload sent via a POST request to a NGINX/Openresty location. To do so, I combined Openresty's content_by_lua_block with its cjson module like this:
# other locations above
location /test {
content_by_lua_block {
ngx.req.read_body()
local data_string = ngx.req.get_body_data()
local cjson = require "cjson.safe"
local json = cjson.decode(data_string)
local endpoint_name = json['endpoint']['name']
local payload = json['payload']
local source_address = json['source_address']
local submit_date = json['submit_date']
ngx.say('Parsed')
}
}
Parsing sample data containing all required fields works as expected. A correct JSON object could look like this:
{
"payload": "the payload here",
"submit_date": "2018-08-17 16:31:51",
},
"endpoint": {
"name": "name of the endpoint here"
},
"source_address": "source address here",
}
However, a user might POST a differently formatted JSON object to the location. Assume a simple JSON document like
{
"username": "JohnDoe",
"password": "password123"
}
not containing the desired fields/keys.
According to the cjson module docs, using cjson (without its safe mode) will raise an error if invalid data is encountered. To prevent any errors being raised, I decided to use its safe mode by importing cjson.safe. This should return nil for invalid data and provide the error message instead of raising the error:
The cjson module will throw an error during JSON conversion if any invalid data is encountered. [...]
The cjson.safe module behaves identically to the cjson module, except when errors are encountered during JSON conversion. On error, the cjson_safe.encode and cjson_safe.decode functions will return nil followed by the error message.
However, I do not encounter any different error handling behavior in my case and the following traceback is shown in Openresty's error.log file:
2021/04/30 20:33:16 [error] 6176#6176: *176 lua entry thread aborted: runtime error: content_by_lua(samplesite:50):16: attempt to index field 'endpoint' (a nil value)
Which in turn results in an Internal Server Error:
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
I think a workaround might be writing a dedicated function for parsing the JSON data and calling it with pcall() to catch any errors. However, this would make the safe mode kind of useless. What am I missing here?
Your “simple JSON document” is a valid JSON document. The error you are facing is not related to cjson, it's a standard Lua error:
resty -e 'local t = {foo = 1}; print(t["foo"]); print(t["foo"]["bar"])'
1
ERROR: (command line -e):1: attempt to index field 'foo' (a number value)
stack traceback:
...
“Safeness” of cjson.safe is about parsing of malformed documents:
cjson module raises an error:
resty -e 'print(require("cjson").decode("[1, 2, 3"))'
ERROR: (command line -e):1: Expected comma or array end but found T_END at character 9
stack traceback:
...
cjson.safe returns nil and an error message:
resty -e 'print(require("cjson.safe").decode("[1, 2, 3"))'
nilExpected comma or array end but found T_END at character 9

Unable to Read JSON file using Elephant Bird

Trying to load the json file which is having null values in it by using elephant-bird JsonLoader.
sample.json
{"created_at": "Mon Aug 22 10:48:23 +0000 2016","id": 767674772662607873,"id_str": "767674772662607873","text": "KPIT Image Result for https:\/\/t.co\/Nas2ZnF1zZ... https:\/\/t.co\/9TnelwtIvm","source": "\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated": false,"in_reply_to_status_id": 123,"in_reply_to_status_id_str": null,"in_reply_to_user_id": null,"in_reply_to_user_id_str": null,"in_reply_to_screen_name": null,"geo": null,"coordinates": null,"place": null,"contributors": null,"is_quote_status": false,"retweet_count": 0,"favorite_count": 0,"entities": {"hashtags": [],"urls": [{"url": "https:\/\/t.co\/Nas2ZnF1zZ","expanded_url": "http:\/\/miltonious.com\/","display_url": "miltonious.com","indices": [24, 47]}],"user_mentions": [],"symbols": []},"favorited": false,"retweeted": false,"possibly_sensitive": false,"filter_level": "low","lang": "en","timestamp_ms": "1471862903167"}
script:
REGISTER piggybank.jar
REGISTER json-simple-1.1.1.jar
REGISTER elephant-bird-pig-4.3.jar
REGISTER elephant-bird-core-4.1.jar
REGISTER elephant-bird-hadoop-compat-4.3.jar
json = LOAD 'sample.json' USING JsonLoader('created_at:chararray, id:chararray, id_str:chararray, text:chararray, source:chararray, in_reply_to_status_id:chararray, in_reply_to_status_id_str:chararray, in_reply_to_user_id:chararray, in_reply_to_user_id_str:chararray, in_reply_to_screen_name:chararray, geo:chararray, coordinates:chararray, place:chararray, contributors:chararray, is_quote_status:bytearray, retweet_count:long, favorite_count:chararray, entities:map[], favorited:bytearray, retweeted:bytearray, possibly_sensitive:bytearray, lang:chararray');
describe json;
dump json;
When I dump json,I am getting the following output and the worning
(Mon Aug 22 10:48:23 +0000 2016,767674772662607873,767674772662607873,google Image Result for Twitter Web Client,false,1234,12345,3214,43215,,,,,,,,,,,,,,)
WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigHadoopLogger - org.apache.pig.builtin.JsonLoader(UDF_WARNING_1): Bad record, returning null for {complete json}
By warning i guess it is getting NULL values.
So how can we load a Json which is having null values in it.
And I have tried in another way i.e
json = LOAD 'sample.json' USING com.twitter.elephantbird.pig.load.JsonLoader('created_at:chararray, id:chararray, id_str:chararray, text:chararray, source:chararray, in_reply_to_status_id:chararray, in_reply_to_status_id_str:chararray, in_reply_to_user_id:chararray, in_reply_to_user_id_str:chararray, in_reply_to_screen_name:chararray, geo:chararray, coordinates:chararray, place:chararray, contributors:chararray, is_quote_status:bytearray, retweet_count:long, favorite_count:chararray, entities:map[], favorited:bytearray, retweeted:bytearray, possibly_sensitive:bytearray, lang:chararray');
describe json;
Output
Schema for json unknown.
Please suggest me.
Thanks.
You can try something like this,
MY_JSON = LOAD 'sample.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad');
dump MY_JSON;