Why is initial JSON object parseable, but object within it not? - json

I'm storing a config file in version control (GitLab) which contains information to be read by my ruby app. This info is stored as an object containing objects containing objects.
(Update adding more detail and examples for clarity as requested...)
From within my app I can successfully GET the file (which returns the following JSON Object (some bits trimmed with ... for readability):
{"file_name"=>"approval_config.json", "file_path"=>"approval_config.json", "size"=>1331, "encoding"=>"base64", "content_sha256"=>"1c21cbb...fa453fe", "ref"=>"master", "blob_id"=>"de...915", "commit_id"=>"07e...4ff", "last_commit_id"=>"07e...942f", "content"=>"ogICAg...AgICB"}
I can JSON parse the above object and access the contents property on that object. The value of the contents property is a base64Encoded string containing the actual contents of my file in GitLab. I can successfully decode this and see the JSON string stored in GitLab:
"{"G000":{"1":{"max":"4000","name":"Matthew Lewis","id":"ord-matthewl","email":"matthew.lewis#companyx.com"},"2":{"max":"4000","name":"Brendan Jones","id":"ord-brendanj","email":"brendan.jones#companyx.com"},"3":{"max":"20000","name":"Henry Orson","id":"ord-henryo","email":"henry.orson#companyx.com"},"4":{"max":"10000000","name":"Chris Adams","id":"ord-chrisa","email":"chris.adams#companyx.com"}},"G15":{"1":{"max":"4000","name":"Mike Butak","id":"ord-mikebu","email":"mike.butak#companyx.com"},"2":{"max":"4000","name":"Joseph Lister","id":"ord-josephl","email":"joseph.lister#companyx.com"},"3":{"max":"20000","name":"Mike Geisler","id":"ord-mikeg","email":"mike.geisler#companyx.com"},"4":{"max":"10000000","name":"Samuel Ahn","id":"ord-samuela","email":"samuel.ahn#companyx.com"}}}"
THIS string (above), I cannot JSON parse. I get an "unexpected token at '{ (JSON::ParserError)" error.
While writing this update it occurs to me that this "un-parsable" string is simply what I put in the file to begin with. Perhaps the method I used to stringify the file's contents in the first place is the issue. I simply pasted a valid javascript object in my browser's console, JSON.stringify'd it, copied the result from the console, and pasted it in my file in GitLab. Perhaps I need to use Ruby's JSON.stringify method to stringify it?
Based on feedback from #ToddA.Jacobs, I tried the following in my ruby script:
require 'rest-client'
require 'json'
require 'base64'
data = RestClient.get 'https://gitlab.companyx.net/api/v4/projects/3895/repository/files/approval_config.json?ref=master', {'PRIVATE-TOKEN':'*********'}
# get the encoded data stored on the 'content' key:
content = JSON.parse(data)['content']
# decode it:
config = Base64.decode64(content)
# print some logs
$evm.log(:info, config)
$evm.log(:info, "config is a Hash? :" + config.is_a?(Hash).to_s) #prints false
$evm.log(:info, "config is a string? :" + config.is_a?(String).to_s) #prints true
hash = JSON.parse(config)
example = hash.dig "G000" "4" "id"
$evm.log(:info, "print exmaple on next line")
$evm.log(:info, example)
That last line prints:
The following error occurred during method evaluation: NoMethodError: undefined method 'gsub' for nil:NilClass (drbunix:///tmp/automation_engine20200903-3826-1nbuvl) /usr/local/ lib/ruby/gems/2.5.0/gems/manageiq-password-0.3.0/lib/manageiq/password.rb:89:in 'sanitize_string'

Remove Outer Quotes
Your input format is invalid: you're nesting unescaped double quotes, and somehow expecting that to work. Just leave off the outer quotes. For example:
require 'json'
json = <<~'EOF'
{"G000":{"1":{"max":"4000","name":"Matthew Lewis","id":"ord-matthewl","email":"matthew.lewis#companyx.com"},"2":{"max":"4000","name":"Brendan Jones","id":"ord-brendanj","email":"brendan.jones#companyx.com"},"3":{"max":"20000","name":"Henry Orson","id":"ord-henryo","email":"henry.orson#companyx.com"},"4":{"max":"10000000","name":"Chris Adams","id":"ord-chrisa","email":"chris.adams#companyx.com"}},"G15":{"1":{"max":"4000","name":"Mike Butak","id":"ord-mikebu","email":"mike.butak#companyx.com"},"2":{"max":"4000","name":"Joseph Lister","id":"ord-josephl","email":"joseph.lister#companyx.com"},"3":{"max":"20000","name":"Mike Geisler","id":"ord-mikeg","email":"mike.geisler#companyx.com"},"4":{"max":"10000000","name":"Samuel Ahn","id":"ord-samuela","email":"samuel.ahn#companyx.com"}}}
EOF
hash = JSON.parse(json)
hash.dig "G000", "4", "id"
#=> "ord-chrisa"
hash.dig "G15", "4", "id"
#=> "ord-samuela"

This question was answered by users on another post I opened: Why can Ruby not parse local JSON file?
Ultimately the issue was not Ruby failing to parse my JSON. Rather it was the logging function being unable to log the hash.

Related

Post input from http request needs to be passed as a variable to AZ module

I have a powershell based http trigger azure function that uses the Az module to call get-azvm. It get's it's data from a logic-app POST. The output looks correct and it's type shows as string but the cmdlet does not like the variable. The function looks like this:
Write-Host "PowerShell HTTP trigger function processed a request."
$sub = $Request.Body.subject | Out-String
write-host "sub:" $sub
$split = $sub -split "[/]"
write-host "split:" $split
$avd = $split[8] | Out-string
Write-Host "avd" $avd
$rgName_avd = 'rg-azgroup'
Get-AzVM -Name $avd -ResourceGroupName $rgName_avd
The post input looks like:
/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourcegroups/rg-azgroup/providers/Microsoft.Compute/virtualMachines/myvmname-0
The error in the logs starts as:
[Error] ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.Exception :Type : Newtonsoft.Json.JsonReaderExceptionTargetSite :Name : ParseValueDeclaringType : Newtonsoft.Json.JsonTextReaderMemberType
It feels like it's an issue with the input type but using ConvertFrom-Json does not seem to work. Any ideas what I'm doing wrong? If I hard code the VM name or define a variable locally and use it the command executes.
The AZ module versions being used in requirements.psd1:
#{
'Az.Accounts' = '2.7.6'
'Az.Compute' = '4.26.0'
}
Here are the few workarounds that you can try:
Solution 1:
You might not be passing JSON to DeserializeObject.
It looks like that type of tmpfile from File.WriteAllText(tmpfile,... is string that contain file path. JsonConvert.DeserializeObject takes JSON value rather than the file path, it fails trying to convert something like #"c:\temp\fooo" - which is not JSON.
Solution 2:
Check that is the file containing JSON string has BOM.
Once u remove BOM the problem may get resolved.
Solution 3:
For a Web API action that was binding to a string instead to an object or a JObject when JSON was correct, but the binder tried to obtain a string from the JSON structure and failed.
So, instead of:
[HttpPost("[action]")]
public object Search([FromBody] string data)
Try this:
[HttpPost("[action]")]
public object Search([FromBody] JObject data)
References:
Unexpected character encountered while parsing value
Unexpected character encountered while parsing value ASP.NET Core and Newtonsoft

Roku ParseJSON gives Unknow Identifier error when loading json via AJAX

I'm trying to write a simple Roku application.
When I load the JSON file via roURLTransfer ParseJSON function gives me BRIGHTSCRIPT: ERROR: ParseJSON: Unknown identifier.
If I load the JSON file via ReadAsciiFile("pkg:/feed/feed.json") it works.
The JSON file is the same and I'm pretty sure that my JSON is correct.
url = "http://192.168.1.36/misc/roku/ifilm/feed.json"
result = ""
timeout = 10000
ut = CreateObject("roURLTransfer")
ut.SetPort(CreateObject("roMessagePort"))
ut.SetURL(url)
if ut.AsyncGetToString()
event = wait(timeout, ut.GetPort())
if type(event) = "roUrlEvent"
result = event.GetString()
elseif event = invalid
ut.AsyncCancel()
else
print "roUrlTransfer::AsyncGetToString(): unknown event"
end if
end if
' `print result` shows the correct lintable JSON
' print result
' Next line gives me: BRIGHTSCRIPT: ERROR: ParseJSON: Unknown identifier
json = ParseJSON(result)
But putting the JSON file inside the app works:
feed = ReadAsciiFile("pkg:/feed/feed.json")
sleep(2000)
json = ParseJson(feed)
I need to load the data from the Internet and using the embedded version doesn't help me. Does anyone know what should I do to make it work?
The "Unknown identifier" error is usually because there's a character in the json string that ParseJson() does not support. The reason why ReadAsciiFile() works is likely because the function "cleans up" the json string by applying UTF-8 encoding.
A common character that's present at the beginning of some JSON responses that causes this issue is the unicode character Byte Order Mark (BOM)
If you google "byte order mark json" you'll see lots of cases where this affects other platforms as well.
You can just do a simple find and replace to get rid of that character before attempting to parse the string.
bomChar = Chr(65279)
if result.left(len(bomChar)) = bomChar ' Check if the string has the BOM char prefix
result = result.replace(bomChar, "")
end if
If that doesn't work, then your response may have some other conflicting character, in that case I would advise using ifUrlTransfer::AsyncGetToFile() instead of AsyncGetToString() and then use ReadAsciiFile() which should guarantee a properly formatted json string every time (as long as your json is valid).

In Spark JobServer. How to pass a json formated string on the input.string?

Im trying to execute the following curl command to run a job:
curl -k --basic --user 'user:psw' -d 'input.string= {"user":13}' 'https://localhost:8090/jobs?appName=test&classPath=test.ImportCSVFiles&context=import&sync=true'
But I get the following error:
"com.typesafe.config.ConfigException$WrongType: String: 1: input.string has type OBJECT rather than STRING"
My idea is to pass more than one parameter like an sql query. A json format to easy handling on my submitted jar.
I'm on the right way or there is another way?
Copying it from comment.
Issue was json was trying to read it not as string but as object directly because of string starting with braces "{".
Correct input 'input.string= \"{\"user\":13}\" '
input.string is not a reserved keyword -- in fact, you can name the parameters arbitrarily. Let say, you POST two parameterss foo.string and foo.number. You then read the parameters in your SJS job, like this:
// run is the starting point of a SJS job (also see validate function)
override def runJob(sc: SparkContext, config: Config): Any = {
val cmd = config.getString("input.cmd")
val fooString = config.getString("foo.bar")
val fooNum = config.getInt("foo.number")
Just in case you plan to execute SJS jobs from Scala/Java:
Apache Commons Lang (org.apache.commons.lang3) comes with a very helpful class to escape JSON: StringEscapeUtils
I use it to escape inputs from my Scala application that I need to pass to SparkJobServer jobs, like this:
input.schema=\"" + StringEscapeUtils.escapeJson(referenceSchema)+"\""
referenceSchema is a JSON document (in my case a JSON array)
input.schema is then one of many comma-separated parameters in the body of HTTP post from Scala...

Ruby parse string to json

So I have some json that looks like this, which I got after taking it out of some other json by doing response.body.to_json:
{\n \"access_token\": \"<some_access_token>\",\n \"token_type\": \"Bearer\",\n \"expires_in\": 3600,\n \"id_token\": \<some_token>\"\n}\n"
I want to pull out the access_token, so I do
to_return = {token: responseJson[:access_token]}
but this gives me a
TypeError: no implicit conversion of Symbol into Integer
Why? How do I get my access token out? Why are there random backslashes everywhere?
to_json doesn't parse JSON - it does the complete opposite: it turns a ruby object into a string containing the JSON representation of that object is.
It's not clear from your question what response.body is. It could be a string, or depending on your http library it might have already been parsed for you.
If the latter then
response.body["access_token"]
Will be your token, if the former then try
JSON.parse(response.body)["access_token"]
Use with double quotes when calling access_token. Like below:
to_return = {token: responseJson["access_token"]}
Or backslashes are escaped delimiters and make sure you first parse JSON.

Getting line number of json file at which the json validation failed

I am using json-schema-validator for validating my json.
I want to show the line number in the json data file where the validation failure occurs. I want to show the failure messages in the user friendly manner.
I get the pointer to the json node where the validation failure might have occurred as follows:
JsonNode jsondatanode = JsonLoader.fromFile(new File("jsondata.json"));
JsonNode jsonschemanode = JsonLoader.fromFile(new File("jsonschema.json"));
final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
final JsonSchema datastoreschema = factory.getJsonSchema(jsonschemanode);
ProcessingReport report;
report = datastoreschema.validate(jsondatanode);
However the pointer is inconvenient to locate the json object/attribute when the json file contains many nodes of type specified by the pointer.
I got following validation failure message:
--- BEGIN MESSAGES ---
error: instance value (12) not found in enum (possible values:["true","false","y","n","yes","no",0,1])
level: "error"
schema: {"loadingURI":"#","pointer":"/properties/configuration/items/properties/skipHeader"}
instance: {"pointer":"/configuration/0/skipHeader"}
domain: "validation"
keyword: "enum"
value: 12
enum: ["true","false","y","n","yes","no",0,1]
--- END MESSAGES ---
I want to show the custom message for validation failures with the line number in json data file which caused schema validation failure. I know I can access the individual details of validation report as shown in below code.
I want to show the custom message as follows:
List<ProcessingMessage> messages = Lists.newArrayList((AbstractProcessingReport)report);
JsonNode reportJson = messages.get(0).asJson();
if(reportJson.get("keyword").toString().equals("enum"))
{
System.out.println("Value "+report.Json.get("value").toString() +"is invalid in " + filepath + " at line " + linenumber);
}
else if{
//...
}
//...
What I dont understand is how can I get that linenumber variable in above code.
Edit
Now I realize that
instance: {"pointer":"/configuration/0/skipHeader"}
shows which occurrence of skipHeader is into problem and in this case its 0th instance of skipHeader inside configuration. However I still think its better to get the line number which ran into problem.
(library author here)
While it can be done (I have somewhere an implementation of JsonParser which does just that) the problem is that the line/column information will most of the time be irrelevant.
In order to save bandwidth, most of the time, JSON sent over the wire will always be on a single line, therefore the problem will remain that you would get, say, "line 1, column 202" without getting any the smarter.
I'll probably do this anyway for the next major version but for 2.2.x it is too late...