A colleague and I were trying to write the minimal logic that we needed for compacting, validating, parsing, and storing json coming from a client.
Upon doing so, we realized that compacting and validating were two steps that were both being accomplished by json.Compact anyways, since the code indicates json.Compact calls the json Scanner. The scanner then validates json and errors on invalid json.
The docs do not make this explicit, but we think this is the case.
Here is a link: https://forum.golangbridge.org/t/json-compact-appears-to-also-validate-json-but-is-not-documented/23088
Let us know thoughts.
Yes.
json.Compact uses json.scanner while scanning the json. If the scanner encounters invalid JSON sets scanner.err, which is returned by json.Compact if there is an error.
This is the same way json.Valid checks for valid json, by simply scanning the JSON and checking for scanner.err.
Here's the relevant code sections:
https://go.googlesource.com/go/+/go1.16.2/src/encoding/json/indent.go#17
https://go.googlesource.com/go/+/go1.16.2/src/encoding/json/scanner.go#30
Related
Is it advisable or not in a RESTful web service to use JSON literal values (string / number) as input parameter in the payload or in the response body?
If I have an endpoint PUT /mytodolist is it OK for it to accept a JSON string literal value "Take out the rubbish" in the request payload (with Content-Type=application/json) or should it accept a JSON object instead ({"value":"Take out the rubbish"})?
Similarly, is it fine for GET /mytodolist/1 to return "Take out the rubbish" in the response body or should it return a proper JSON object {"value":"Take out the rubbish"}
Spring MVC to makes implementing and testing such endpoints easy, however clients have flagged this as non standard or hard to implement. In my point of view JSON literals are JSON, but not JSON objects, so I'd say it is fine. I have found no recommendations using Google.
EDIT 1: Clafirication
The question is entirely about the 'standard', if it allows this or not.
I understand the problem with the extensibility, but one can never design a fully extensible interface IMHO. If changes need to be done, we can try extending what we have in a backwards compatible way, but there will come a time when it becomes messy and an other approach is required - which is commonly handled by versioning the API in one way or another. I find it a fair point even though, because using literals as request/response body immediately becomes inextensible, while coming up with a reasonable one-attribute JSON object does not.
It is also understood that some frameworks have problems with handling JSON literals, this is the origin of this question. The tool I used happened to support this, so I thought this was all right, but the front-end library did not.
Still, what I am intending to find out right now, is if using JSON literals is according to the de-facto standard (even if it is a cornercase) or not.
I would recommend to use JSON object always. One reason is that for Content-Type application/json people expect something staring with "{" and not all frameworks will handle json literals properly. Second reason is that probably you will add some additional attributes to you list item (due date, category, priority, etc). And then you'll break backward compatibility, by adding new field.
It may be acceptable in the context of your example, but keep in mind that unambiguous interfaces are easier to use and that will encourage adoption.
For example, your interface could interpret "Take out he rubbish" as the same as {task:"take out the rubbish"}, but once you add additional properties (eg "when" or "who") the meaning of a solitary string in the request becomes ambiguous. It's inevitable that you'll add support for new properties as your interface matures.
I am attempting to consume an API that I do not have control over which is somewhat poorly documentented and somewhat inconsistent. This means that sometimes, the API returns a different type than what is documented or what you would normally see. For this example, we'll look at a case when an array was returned in a place where I would normally see a string. That makes a crappy API, but my real problem is: How can I more easily track those things down? Right now, the errors look something like this:
No usable value for identifier
Do not know how to convert JArray(List(JString(3c8723eceb1a), JString(cba8849e7a2f))) into class java.lang.String
After deciphering the problem (why JValue::toString doesn't emit a JSON string is utterly perplexing to me), I can figure out the API returned an array when I made my case class only able to deal with Strings. Great. My issue is that finding this discrepancy between my object model and the contents of the JSON seems significantly more difficult than it should be.
Currently, this is my workflow for hunting down decoding errors:
Hope bad data has some sort of identifying marker. If this is not true, then it is way more guesswork and you will have to repeat the following steps for each entry that looks like the bad bits.
Go through the troubles of converting the JArray(List(JString(...), ...)) from the error message into valid JSON, hoping that I encode JSON the same way at the API endpoint I got the data from does. If this is not true, then I use a JSON formatter (jq) to format all data consistently.
Locate the place in the source data where the decoding error originates from.
Backtrack through arrays and objects to discover how I need to change my object model to more accurately represent what data is coming back to me from the API.
Some background: I'm coming from C++, where I rolled my own JSON deserialization framework for this purpose. The equivalent error when using the library I built is:
Error decoding value at result.taskInstances[914].subtasks[5].identifier: expected std::string but found array value (["3c8723eceb1a","cba8849e7a2f"]) at 1:4084564
This is my process when using my hand-rolled library:
Look at the expected type (std::string) compared with the data that was actually found (["3c8723eceb1a","cba8849e7a2f"]) and alter my data model for the path for the data in the source (result.taskInstances[914].subtasks[5].identifier)
As you can see, I get to jump immediately to the problem that I actually have.
My question is: Is there a way to more quickly debug inconsistencies between my data model and the results I'm getting back from the API?
I'm using json4s-native_2.10 version 3.2.8.
A simplified example:
{ "property": ["3c8723eceb1a", "cba8849e7a2f"] }
Does not mesh with Scala class:
case class Thing(property: String)
The best solution would be to use Try http://www.scala-lang.org/api/current/#scala.util.Try in Scala, but unfortunately json4s API cannot.
So, I think you should use Scala Option type http://www.scala-lang.org/api/current/#scala.Option .
In Scala, and more generally in functional languages, Options are used to represent an object that can be there or not (like à nil value).
For handle parsing failures, you can use parse(str).toOption, which is a function that return an Option[JValue], and you can doing a pattern matching on the resulting value.
For handling extraction of data extraction into case classes, you can use extractOpt function, to do pattern matching on the value.
You can read this answer : https://stackoverflow.com/a/15944506/2330361
I am loading data from a mongodb collection to a mysql table through Kettle transformation.
First I extract them using MongodbInput and then I use json input step.
But since json input step has very low performance, I wanted to replace it with a
javacript script.
I am a beginner in Javascript and even though i tried somethings, the kettle javascript script is not recognizing any keywords.
can anyone give me sample code to convert Json data to different columns using javascript?
To solve your problem you need to see three aspects:
Reading from MongoDB
Reading from JSON
Reading from (probably) String
Reading from MongoDB Except if you changed the interface, MongoDB returns not JSON but BSON files (~binary JSON). You need to see the MongoDB documentation about reading and writing BSON: probably something like BSON.to() and BSON.from() but I don't know it by heart.
Reading from JSON Once you have your BSON in JSON format, you can read it using JSON.stringify() which returns a String.
Reading from (probably) String If you want to use the capabilities of JSON (why else would you use JSON?), you also want to use JSON.parse() which returns a JSON object.
My experience is that to send a JSON object from one step to the other, using a String is not a bad idea, i.e. at the end of a JavaScript step, you write your JSON object to a String and at the beginning of the next JavaScript step (can be further down the stream) you parse it back to JSON to work with it.
I hope this answers your question.
PS: writing JavaScript steps requires you to learn JavaScript. You don't have to be a master, but the basics are required. There is no way around it.
you could use the json input step to get the values of this json and put in common rows
G'day gurus,
I'm calling the REST APIs of an enterprise application that shall remain nameless, and they return JSON such as the following:
throw 'allowIllegalResourceCall is false.';
{
"data": ... loads of valid JSON stuff here ...
}
Is this actually valid JSON? If (as I suspect) it isn't, is there any compelling reason for these kinds of shenanigans?
The response I received from the application vendor is that this is done for security purposes, but I'm struggling to understand how this improves security much, if at all.
Thanks in advance!
Peter
According to
http://jsonlint.com/
It is not.
Something like the below is.
{
"data": "test"
}
Are they expecting you to pull the JSon load out of the message above?
Its not a JSON format at all. From your question it seems you are working with enterprise systems like JIVE :). I am also facing same issue with JIVE api. This is the problem with their V3 API. Not standard , but following thing worked for me. (I am not sure if you are talking about JIVE or not)
//invalid jason response... https://developers.jivesoftware.com/community/thread/2153
jiveResponse = jiveResponse.Replace
("throw 'allowIllegalResourceCall is false.';",String.Empty);
There is a valid reason for this: it protects against CSRF attacks. If you include a JSON url as the target of a <script> tag, then the same-origin policy doesn't apply. This means that a malicious site can include the URL of a JSON API, and any authenticated users will successfully request that data.
By appropriately overriding Object.prototype and/or Array.prototype, the malicious site can get any data parsed as an object literal or array literal (and all valid JSON is also valid javascript). The throw statement protects against this by making it impossible to parse javascript included on a page via <script> tags.
Definitely NOT valid JSON. Maybe there's an error in the implementation that is mixing some kind of debug output with the correct output?
And, by no means this is for security reasons. Seems to me this is a plain bug.
throw 'allowIllegalResourceCall is false.'; is certainly not valid JSON.
What MIME type is reported?
It seems they have added that line to prevent JSON Hijacking. Something like that line is required to prevent JSON Hijacking only if you return a JSON array. But they may have added that line above all of their JSON responses for easier implementation.
Before using it, you have to strip out the first line, and then parse the remaining as JSON.
We are in the middle of a ongoing discussion about how to handle REST exceptions.
Response Content type : JSON
Two solutions we have:
Throw all the unchecked exceptions as a JSON response.
Send Request Invalid Response code.
Arguments:
When its a error, why return JSON? Just send a invalid response code.
Counter Argument:
Response code are too technical to handle for normal developers.
Whats your say??
For a JSON API I recently developed I do both. I always respond with valid JSON (well, assuming I respond at all). If I detect an invalid request, I use status 400. If I detect a server error (which I don't believe is caused by an invalid request), I use a 5xx status. The JSON object contains a special key that is only set for errors, with a string value.
I think this is a good solution that respects REST principles, and can be used in multiple ways. The same solution is used by some other JSON APIs, such as Yahoo Search. Try http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&output=json .
Use error codes like for HTTP. So 50* for any exception cause by some internal problem. And 40* for bad arguments. Avoid using your own defined codes as far as its possible. The idea is to have a "uniform" interface.
In general.
204 for success without sending any content
200 for success with a json representation of the resource
And if its not a successful operation return appropriate response code. You can choose to optionally return a json. To simplify things you can have a common format (json) for all error responses.
http://en.wikipedia.org/wiki/REST is a must read before you freeze on your api specs.