Getting line number of json file at which the json validation failed - json

I am using json-schema-validator for validating my json.
I want to show the line number in the json data file where the validation failure occurs. I want to show the failure messages in the user friendly manner.
I get the pointer to the json node where the validation failure might have occurred as follows:
JsonNode jsondatanode = JsonLoader.fromFile(new File("jsondata.json"));
JsonNode jsonschemanode = JsonLoader.fromFile(new File("jsonschema.json"));
final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
final JsonSchema datastoreschema = factory.getJsonSchema(jsonschemanode);
ProcessingReport report;
report = datastoreschema.validate(jsondatanode);
However the pointer is inconvenient to locate the json object/attribute when the json file contains many nodes of type specified by the pointer.
I got following validation failure message:
--- BEGIN MESSAGES ---
error: instance value (12) not found in enum (possible values:["true","false","y","n","yes","no",0,1])
level: "error"
schema: {"loadingURI":"#","pointer":"/properties/configuration/items/properties/skipHeader"}
instance: {"pointer":"/configuration/0/skipHeader"}
domain: "validation"
keyword: "enum"
value: 12
enum: ["true","false","y","n","yes","no",0,1]
--- END MESSAGES ---
I want to show the custom message for validation failures with the line number in json data file which caused schema validation failure. I know I can access the individual details of validation report as shown in below code.
I want to show the custom message as follows:
List<ProcessingMessage> messages = Lists.newArrayList((AbstractProcessingReport)report);
JsonNode reportJson = messages.get(0).asJson();
if(reportJson.get("keyword").toString().equals("enum"))
{
System.out.println("Value "+report.Json.get("value").toString() +"is invalid in " + filepath + " at line " + linenumber);
}
else if{
//...
}
//...
What I dont understand is how can I get that linenumber variable in above code.
Edit
Now I realize that
instance: {"pointer":"/configuration/0/skipHeader"}
shows which occurrence of skipHeader is into problem and in this case its 0th instance of skipHeader inside configuration. However I still think its better to get the line number which ran into problem.

(library author here)
While it can be done (I have somewhere an implementation of JsonParser which does just that) the problem is that the line/column information will most of the time be irrelevant.
In order to save bandwidth, most of the time, JSON sent over the wire will always be on a single line, therefore the problem will remain that you would get, say, "line 1, column 202" without getting any the smarter.
I'll probably do this anyway for the next major version but for 2.2.x it is too late...

Related

Couchbase: using QueryAsync<string>(query) fails with "Error reading string. Unexpected token"

When using Couchbase generic QueryAsync, I was hoping to get a normal, un-parsed, string back instead of a JSON parsed string.
However, when making this call, it indeed seems that Couchbase SDK for .NET uses Newtonsoft anyways, and tries to intepret it as JSON (it is a valid JSON string by the way!):
string query = $"SELECT * FROM myCollection WHERE id = {r.Next(1, 100000)}";
await scope.QueryAsync<string>(query); // <-- fails here
Newtonsoft.Json.JsonReaderException: 'Error reading string. Unexpected
token: StartObject. Path '', line 6, position 1.'
If I instead do this, it works well and JSON is correctly parsed (all documents from myCollection are valid JSON strings):
string query = $"SELECT * FROM myCollection WHERE id = {r.Next(1, 100000)}";
await scope.QueryAsync<JObject>(query); // <-- No problem
-- EDIT --
I also tried they Keu Value operation instead:
var result = await collection.GetAsync(key);
var content = result.ContentAs<string>(); // <-- fails here with same error
Also using this approach, that I just copied from here, fails.
Newtonsoft.Json.JsonReaderException: 'Unexpected character encountered
while parsing value: {. Path '', line 1, position 1.'
The JSON is correct though, and I have verified in different parsers.
Isn't it possible to retrieve raw, unparsed, data from Couchbase?
I just found this on Couchbase forums:
When using SDK 3.x when trying to get a JSON document from Couchbase
and using var result = data.ContentAs(); It always return this
error message: “Unexpected character encountered while parsing value:
{. Path ‘’, line 1, position 1.”
The solution presented there is very convoluted and complex, and strikes me as counter-intuitive:
JMorris writes:
Hi #Brian_Davis -
The Transcoder API was redesigned in SDK3; its now much more
specialized and this type of conversion doesn’t work with the default
JsonTranscoder. Fortunately, we do have another transcoder which
mimics the sdk2 behavior called the LegacyTranscoder which should be
able to handle this for you.
And the convoluted solution seems to be to pass a Transcoder into the options of each call:
var result = await collection.GetAsync(key, options: new GetOptions().Transcoder(new LegacyTranscoder()));
var content = result.ContentAs<string>();

Why is initial JSON object parseable, but object within it not?

I'm storing a config file in version control (GitLab) which contains information to be read by my ruby app. This info is stored as an object containing objects containing objects.
(Update adding more detail and examples for clarity as requested...)
From within my app I can successfully GET the file (which returns the following JSON Object (some bits trimmed with ... for readability):
{"file_name"=>"approval_config.json", "file_path"=>"approval_config.json", "size"=>1331, "encoding"=>"base64", "content_sha256"=>"1c21cbb...fa453fe", "ref"=>"master", "blob_id"=>"de...915", "commit_id"=>"07e...4ff", "last_commit_id"=>"07e...942f", "content"=>"ogICAg...AgICB"}
I can JSON parse the above object and access the contents property on that object. The value of the contents property is a base64Encoded string containing the actual contents of my file in GitLab. I can successfully decode this and see the JSON string stored in GitLab:
"{"G000":{"1":{"max":"4000","name":"Matthew Lewis","id":"ord-matthewl","email":"matthew.lewis#companyx.com"},"2":{"max":"4000","name":"Brendan Jones","id":"ord-brendanj","email":"brendan.jones#companyx.com"},"3":{"max":"20000","name":"Henry Orson","id":"ord-henryo","email":"henry.orson#companyx.com"},"4":{"max":"10000000","name":"Chris Adams","id":"ord-chrisa","email":"chris.adams#companyx.com"}},"G15":{"1":{"max":"4000","name":"Mike Butak","id":"ord-mikebu","email":"mike.butak#companyx.com"},"2":{"max":"4000","name":"Joseph Lister","id":"ord-josephl","email":"joseph.lister#companyx.com"},"3":{"max":"20000","name":"Mike Geisler","id":"ord-mikeg","email":"mike.geisler#companyx.com"},"4":{"max":"10000000","name":"Samuel Ahn","id":"ord-samuela","email":"samuel.ahn#companyx.com"}}}"
THIS string (above), I cannot JSON parse. I get an "unexpected token at '{ (JSON::ParserError)" error.
While writing this update it occurs to me that this "un-parsable" string is simply what I put in the file to begin with. Perhaps the method I used to stringify the file's contents in the first place is the issue. I simply pasted a valid javascript object in my browser's console, JSON.stringify'd it, copied the result from the console, and pasted it in my file in GitLab. Perhaps I need to use Ruby's JSON.stringify method to stringify it?
Based on feedback from #ToddA.Jacobs, I tried the following in my ruby script:
require 'rest-client'
require 'json'
require 'base64'
data = RestClient.get 'https://gitlab.companyx.net/api/v4/projects/3895/repository/files/approval_config.json?ref=master', {'PRIVATE-TOKEN':'*********'}
# get the encoded data stored on the 'content' key:
content = JSON.parse(data)['content']
# decode it:
config = Base64.decode64(content)
# print some logs
$evm.log(:info, config)
$evm.log(:info, "config is a Hash? :" + config.is_a?(Hash).to_s) #prints false
$evm.log(:info, "config is a string? :" + config.is_a?(String).to_s) #prints true
hash = JSON.parse(config)
example = hash.dig "G000" "4" "id"
$evm.log(:info, "print exmaple on next line")
$evm.log(:info, example)
That last line prints:
The following error occurred during method evaluation: NoMethodError: undefined method 'gsub' for nil:NilClass (drbunix:///tmp/automation_engine20200903-3826-1nbuvl) /usr/local/ lib/ruby/gems/2.5.0/gems/manageiq-password-0.3.0/lib/manageiq/password.rb:89:in 'sanitize_string'
Remove Outer Quotes
Your input format is invalid: you're nesting unescaped double quotes, and somehow expecting that to work. Just leave off the outer quotes. For example:
require 'json'
json = <<~'EOF'
{"G000":{"1":{"max":"4000","name":"Matthew Lewis","id":"ord-matthewl","email":"matthew.lewis#companyx.com"},"2":{"max":"4000","name":"Brendan Jones","id":"ord-brendanj","email":"brendan.jones#companyx.com"},"3":{"max":"20000","name":"Henry Orson","id":"ord-henryo","email":"henry.orson#companyx.com"},"4":{"max":"10000000","name":"Chris Adams","id":"ord-chrisa","email":"chris.adams#companyx.com"}},"G15":{"1":{"max":"4000","name":"Mike Butak","id":"ord-mikebu","email":"mike.butak#companyx.com"},"2":{"max":"4000","name":"Joseph Lister","id":"ord-josephl","email":"joseph.lister#companyx.com"},"3":{"max":"20000","name":"Mike Geisler","id":"ord-mikeg","email":"mike.geisler#companyx.com"},"4":{"max":"10000000","name":"Samuel Ahn","id":"ord-samuela","email":"samuel.ahn#companyx.com"}}}
EOF
hash = JSON.parse(json)
hash.dig "G000", "4", "id"
#=> "ord-chrisa"
hash.dig "G15", "4", "id"
#=> "ord-samuela"
This question was answered by users on another post I opened: Why can Ruby not parse local JSON file?
Ultimately the issue was not Ruby failing to parse my JSON. Rather it was the logging function being unable to log the hash.

How to split the data of NodeObject in Apache Flink

I'm using Flink to process the data coming from some data source (such as Kafka, Pravega etc).
In my case, the data source is Pravega, which provided me a flink connector.
My data source is sending me some JSON data as below:
{"key": "value"}
{"key": "value2"}
{"key": "value3"}
...
...
Here is my piece of code:
PravegaDeserializationSchema<ObjectNode> adapter = new PravegaDeserializationSchema<>(ObjectNode.class, new JavaSerializer<>());
FlinkPravegaReader<ObjectNode> source = FlinkPravegaReader.<ObjectNode>builder()
.withPravegaConfig(pravegaConfig)
.forStream(stream)
.withDeserializationSchema(adapter)
.build();
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<ObjectNode> dataStream = env.addSource(source).name("Pravega Stream");
dataStream.map(new MapFunction<ObjectNode, String>() {
#Override
public String map(ObjectNode node) throws Exception {
return node.toString();
}
})
.keyBy("word") // ERROR
.timeWindow(Time.seconds(10))
.sum("count");
As you see, I used the FlinkPravegaReader and a proper deserializer to get the JSON stream coming from Pravega.
Then I try to transform the JSON data into a String, KeyBy them and count them.
However, I get an error:
The program finished with the following exception:
Field expression must be equal to '*' or '_' for non-composite types.
org.apache.flink.api.common.operators.Keys$ExpressionKeys.<init>(Keys.java:342)
org.apache.flink.streaming.api.datastream.DataStream.keyBy(DataStream.java:340)
myflink.StreamingJob.main(StreamingJob.java:114)
It seems that KeyBy threw this exception.
Well, I'm not a Flink expert so I don't know why. I've read the source code of the official example WordCount. In that example, there is a custtom splitter, which is used to split the String data into words.
So I'm thinking if I need to use some kind of splitter in this case too? If so, what kind of splitter should I use? Can you show me an example? If not, why did I get such an error and how to solve it?
I guess you have read the document about how to specify keys
Specify keys
The example codes use keyby("word") because word is a field of POJO type WC.
// some ordinary POJO (Plain old Java Object)
public class WC {
public String word;
public int count;
}
DataStream<WC> words = // [...]
DataStream<WC> wordCounts = words.keyBy("word").window(/*window specification*/);
In your case, you put a map operator before keyBy, and the output of this map operator is a string. So there is obviously no word field in your case. If you actually want to group this string stream, you need to write it like this .keyBy(String::toString)
Or you can even implement a customized keySelector to generate your own key.
Customized Key Selector

Azure Stream Analytics Error :Could not deserialize the input event as Json

I am trying to create a Stream Analytics job. The message is being sent in the following format as JSON:
var message = "Name;\n" + Guid.NewGuid().ToString() + ";" ;
When I am running my job I am getting the following error:
Could not deserialize the input event as Json. Some possible reasons:
1) Malformed events
2) Input source configured with incorrect serialization format
Based on your code sample, it appears your input is taking the form of:
Name;
AA7509E7-D482-459B-9689-456A0F952B44;
then the error message you're seeing is correct, this is not valid JSON, so ASA won't be able to deserialize it. Your JSON string should look something like this:
{
"Name": "AA7509E7-D482-459B-9689-456A0F952B44"
}

Token not allowed in path expression - Reading Configuration file in playframework

I am writing junit test cases in play. I want to read certain configurations from a configuration file. So I am loading that file programatically
private Configuration additionalConfigurations;
Config additionalConfig = ConfigFactory.parseFile(new File("conf/application.conf"));
Config resolConfig = additionalConfig.resolve(ConfigResolveOptions.noSystem());
additionalConfigurations = new Configuration(scaleBasedConf);
running(fakeApplication(additionalConfigurations.asMap()), new Runnable() {
public void run() {
// test Code
}
While running my test case using "play test" I am getting error "Token not allowed in path expression: '[' (you can double-quote this token if you really want it here)
" . My configuration where I am getting this error is
Mykey.a.b.c"[]".xyz = "value"
I have double quoted square brackets. But still getting the error.
After hours of research I finally found out the reason why this is throwing exception. It is because when I do
Config additionalConfig = ConfigFactory.parseFile(new File("conf/application.conf"))
additionalConfig.resolve(ConfigResolveOptions.noSystem());
Then it parses the configuration file taking double quotes in consideration and thus dont give any exception. However it does 1 more thing, it removes those double quotes while parsing. Then the map which we get after parsing , we are passing it to
fakeApplication(additionalConfigurations.asMap()
have key like -> Mykey.a.b.c[].xyz
Here, what play does it again parses the map . Now when double quotes are removed, it throws exception . So the solution for it is-
Mykey."\""a.b.c"[]"\"".xyz = "value"
Doing this, in first parse it creates string as - > Mykey."a.b.c[]".xyz and so in second parse it goes well and dont throw any exception.