I need to sign a JSON but I noticed that unmarshaling/marshaling can change JSON's order which might make the signature invalid.
Is there anyway to produce the same hash from a JSON string despite its order?
I've had a look at JOSE but couldn't find the function that actually hashes JSON.
JOSE JWS will absolutely do what you want at the cost of having to manage keys for signatures and verification.
But let's assume that you don't really need the whole key management stuff and general crypto functionality in JOSE and you're not SUPER concerned about performance (so a little string mangling in this process is OK).
You could dumbly unmarshal your JSON and re-marshal it, then just hash that:
package main
import (
"crypto/sha256"
"encoding/hex"
"fmt"
json "encoding/json"
)
// NB These docs are strictly-speaking the same.
const DOCA = "{ \"foo\": 1.23e1, \"bar\": { \"baz\": true, \"abc\": 12 } }"
const DOCB = "{ \"bar\": { \"abc\": 12, \"baz\": true }, \"foo\": 12.3 }"
func hash(doc string) string {
// Dumb af, but it's a cheap way to specific the most generic thing
// you can :-/
var v interface{}
json.Unmarshal([]byte(doc), &v) // NB: You should handle errors :-/
cdoc, _ := json.Marshal(v)
sum := sha256.Sum256(cdoc)
return hex.EncodeToString(sum[0:])
}
func main() {
fmt.Println(DOCA)
fmt.Printf("Hash: %s\n", hash(DOCA))
fmt.Println(DOCB)
fmt.Printf("Hash: %s\n", hash(DOCB))
}
The output of this program (at least in the golang docker container) is:
{ "foo": 1.23e1, "bar": { "baz": true, "abc": 12 } }
Hash: d50756fbb830f8335187a3f427603944c566772365d8d8e6f6760cd2868c8a73
{ "bar": { "abc": 12, "baz": true }, "foo": 12.3 }
Hash: d50756fbb830f8335187a3f427603944c566772365d8d8e6f6760cd2868c8a73
The nice thing about this approach is that, for the cost of some performance, you get insulated from whatever dumb junk you did while marshalling your JSON in the first place (so, unlike other suggestions, you don't have to think about what you might be doing with custom Marshallers and whatnot). This is especially a big deal when you forget that this was an issue at all in version 3.8 of your code a year from now, implement something that messes with the marshal order, and start breaking things.
And, of course, you could always add the hash to the resulting struct and marshal again with the extra item in the map. Obviously you want to optimize a bit for performance if you're worried about it at all and properly handle errors, but this is a good prototype anyway :-)
Oh, and if you're super-worried about edge cases biting you, you could also use canonical JSON to marshal, since it's specifically designed for this type of use (though, honestly, I couldn't come up with an example in my testing where c-json worked but go's default json didn't).
Related
below my json :
"temperature": {
"level": 8,
"format": function (value) {
return value + ' (C°)';
},
"minimum": null,
...
...
}
My goal is to write key format, I have try this but without much conviction...
package require rl_json
namespace import rl_json::json
set rj "{}"
set s1 {function (value) {
return value + ' (C°)';
}
}
json set rj temperature [json object [list level {number 8} format [json template {{"~L:s1"}}] minimum {null}]]
Error parsing JSON value: Expecting : after object key at offset 8
What you are trying to produce is not (partial) JSON, as defined, but rather general Javascript. As such, you cannot reasonably expect JSON-specific tooling (such as rl_json) to work with it. It is usually better to keep your JSON and your Javascript separated, with the data in JSON (including any dynamic generation) and your Javascript as a static artefact (or generated from something like Typescript). Mixing code and dynamic generation requires great care to get right; it is usually better to try to write things so you don't need to be that careful!
If you must do this defintely-not-advised thing, use something like subst or format to inject the variable parts (which you can generate with rl_json) in the overall string. No code for that from me: my advice is don't do that.
Im new in golang and i have a question.
I have 5 structs where i use Json , but the JSON file can have more structs than the ones i have predetermined BUT... the structures of the JSON satisfies the structures in my programing ( lets say i have 5 structs "struct1" , 6 structs "struct 2" , 1 struct "struct 3", and so on...)
My question is , i want to make a function were i take the JSON FILE , read the structs of it and have as an output the number of structs of the JSON file.
I think i could use the map[string]interface{} but i dont understand it.
I hope i have explain myself
Thank u very much!
Without example JSON or structs, the exact question you are asking is a bit hard to decipher, specifically the "output the number of structs" bit in the question, as it could be interpreted several different ways. I will do my best to answer what I think are the most probable questions you are asking.
Interfaces
First off, some basic go knowledge that might be useful, but outside JSON marshaling itself. The interface{} type appears special, but is not a hardwired keyword as it first might appear. What the interface keyword does is describe the requirements that an object must have to fulfill that interface. Because interface{} has no requirements, and because everything is automatically interfaced in go, everything satisfies the interface{} type.
Because of this implementation of interfaces, map[string]interface{} is really map[string]. This allows for the JSON un/marshal to not care about what is on the value side of the map. This exactly lines up with the format of JSON, where you have a string key on one side, and a value that could be any of the JSON datatypes on the other.
How many different objects are in the base JSON object?
let us take an example json
{
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
"text": {
"data": "Click Here",
"size": 36,
"style": "bold",
"name": "text1",
"hOffset": 250,
"vOffset": 100,
"alignment": "center",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
}
}
The answer to the question in this circumstance would be four. debug, window, image, and text.
The process for determining the number would then be:
Load the json into a byte array.
Marshal into an interface{}
Determine type (array vs object etc.) using type switch. see this A Tour of Go page
If you know the type, you can skip this step
Convert to desired type
Get length, or perform any other operation as desired.
package main
import (
"encoding/json"
"fmt"
)
func main() {
myJSON := `<see above>`
var outStruct *interface{}
json.Unmarshal([]byte(myJSON), &outStruct)
outMap := (*outStruct).(map[string]interface{})
fmt.Printf("Num Structs: %d", len(outMap))
}
Go Playground
How many json objects that I do not have structs for are present?
This answer has a very similar answer to the first one, and is really about manipulation of the output map and the struct
Taking almost the entire code from the first one to the second, let us assume that you have the following structs set up
type Image struct {
Name string
//etc
}
type Text struct {
Name string
//etc
}
type Window struct {
Name string
//etc
}
type Base struct {
Image Image
Window Window
Text Text
}
In this case, in addition to the previous steps, you would have to
5. Marshal the json into a base object
6. Go through the map[string]interface{}, and for each key
7. Determine if the key is one of the objects in your base struct
total := 0
for k, _ := range outMap {
if k != "image" && k != "text" && k != "window" && k != "other" {
total++
}
}
fmt.Printf("Total unknown structs: %d\n", total)
How many of my structs are empty?
This last question is also rather simple, and could be done by checking the map for a value given the input key, but for completion's sake, the example code marshals the JSON into a struct, and uses that.
Marshal JSON into base
For each of Window, Item, Text, Other in base, determine if empty.
total = 0
if (Image{}) == outBase.Image {
total++
}
if (Window{}) == outBase.Window {
total++
}
if (Text{}) == outBase.Text {
total++
}
if (Other{}) == outBase.Other {
total++
}
fmt.Printf("Total empty structs: %d\n", total)
Go Playground
See this go blog post for more information on golang JSON.
Hopefully this is enough context for the question...
Using Handlebars with Rust, I'm trying to implement a handler to handle this input:
{{toJSON JSON_OBJ_OR_NONE}}
where JSON_OBJ_OR_NONE is either a valid fragment of JSON like
{
"abc": 123,
"def": ["g", "h", "i"],
"jkl": {
"m": 1,
"n": "op"
}
}
or nothing (an empty string).
What it should return is either a pretty-printed version of the supplied JSON, or "{}" if JSON_OBJ_OR_NONE is empty.
The supplied JSON fragment is completely arbitrary; it could contain any valid JSON, or an empty string. Output should be pretty-printed.
I've tried to implement this in a bunch of different ways, and I'm currently at
handlebars_helper!(toJSON: |json_obj_or_none: str|
if json_obj_or_none.is_empty() {
"{}"
} else {
let s = serde_json::to_string_pretty(&json_obj_or_none).is_ok().to_string();
&s[..]
});
This looks close, but I'm seeing
error[E0597]: `s` does not live long enough
--> src/main.rs:145:10
|
145 | &s[..]
| -^----
| ||
| |borrowed value does not live long enough
| borrow later used here
146 | });
| - `s` dropped here while still borrowed
when I compile it
While this seems to be close to a working solution, I suspect there's a much more elegant way of coding it.
Code like:
{
let s = something;
&s
}
is almost always a mistake in Rust. References always borrow from something, and references can't exist after that thing is destroyed. Local variables are destroyed at the end of their scope.
So what this means is:
Make s
Borrow from s
Destroy s and its content
Return a reference to that thing, which has just been destroyed
So you need to pass ownership:
{
let s = something;
s // not borrowed!
}
or ensure that s isn't destroyed to soon by moving let to a scope level up:
let s;
{
s = something;
&s
// `s` isn't destroyed here, because `let s` was outside `{}`
}
// `s` and `&s` are destroyed now
If you mix it with string literals in if/else, Rust will complain that String and &str are different. Use String::from("") for the literal, or see Cow type that holds both. Rust needs to know which strings to free (String) and which it must never free (&str), that's why there are two types.
Just to wrap this up, I eventually got it working as follows:
handlebars_helper!(toJSON: |json_obj_or_none: object|
if json_obj_or_none.is_empty() {
"{}".into()
} else {
let s = serde_json::to_string_pretty(&json_obj_or_none).unwrap_or_else(|_| "{}".to_string());
s
});
I've been playing with Kotlinx.serialisation. I've been trying to find a quick way to use Kotlinx.serialisation to create a plain simple JSON (mostly to send it away), with minimum code clutter.
For a simple string such as:
{"Album": "Foxtrot", "Year": 1972}
I've been doing is something like:
val str:String = Json.stringify(mapOf(
"Album" to JsonPrimitive("Foxtrot"),
"Year" to JsonPrimitive(1972)))
Which is far from being nice. My elements are mostly primitive, so I wish I had something like:
val str:String = Json.stringify(mapOf(
"Album" to "Sergeant Pepper",
"Year" to 1967))
Furthermore, I'd be glad to have a solution with a nested JSON. Something like:
Json.stringify(JsonObject("Movies", JsonArray(
JsonObject("Name" to "Johnny English 3", "Rate" to 8),
JsonObject("Name" to "Grease", "Rate" to 1))))
That would produce:
{
"Movies": [
{
"Name":"Johnny English 3",
"Rate":8
},
{
"Name":"Grease",
"Rate":1
}
]
}
(not necessarily prettified, even better not)
Is there anything like that?
Note: It's important to use a serialiser, and not a direct string such as
"""{"Name":$name, "Val": $year}"""
because it's unsafe to concat strings. Any illegal char might disintegrate the JSON! I don't want to deal with escaping illegal chars :-(
Thanks
Does this set of extension methods give you what you want?
#ImplicitReflectionSerializer
fun Map<*, *>.toJson() = Json.stringify(toJsonObject())
#ImplicitReflectionSerializer
fun Map<*, *>.toJsonObject(): JsonObject = JsonObject(map {
it.key.toString() to it.value.toJsonElement()
}.toMap())
#ImplicitReflectionSerializer
fun Any?.toJsonElement(): JsonElement = when (this) {
null -> JsonNull
is Number -> JsonPrimitive(this)
is String -> JsonPrimitive(this)
is Boolean -> JsonPrimitive(this)
is Map<*, *> -> this.toJsonObject()
is Iterable<*> -> JsonArray(this.map { it.toJsonElement() })
is Array<*> -> JsonArray(this.map { it.toJsonElement() })
else -> JsonPrimitive(this.toString()) // Or throw some "unsupported" exception?
}
This allows you to pass in a Map with various types of keys/values in it, and get back a JSON representation of it. In the map, each value can be a primitive (string, number or boolean), null, another map (representing a child node in the JSON), or an array or collection of any of the above.
You can call it as follows:
val json = mapOf(
"Album" to "Sergeant Pepper",
"Year" to 1967,
"TestNullValue" to null,
"Musicians" to mapOf(
"John" to arrayOf("Guitar", "Vocals"),
"Paul" to arrayOf("Bass", "Guitar", "Vocals"),
"George" to arrayOf("Guitar", "Sitar", "Vocals"),
"Ringo" to arrayOf("Drums")
)
).toJson()
This returns the following JSON, not prettified, as you wanted:
{"Album":"Sergeant Pepper","Year":1967,"TestNullValue":null,"Musicians":{"John":["Guitar","Vocals"],"Paul":["Bass","Guitar","Vocals"],"George":["Guitar","Sitar","Vocals"],"Ringo":["Drums"]}}
You probably also want to add handling for some other types, e.g. dates.
But can I just check that you want to manually build up JSON in code this way rather than creating data classes for all your JSON structures and serializing them that way? I think that is generally the more standard way of handling this kind of stuff. Though maybe your use case does not allow that.
It's also worth noting that the code has to use the ImplicitReflectionSerializer annotation, as it's using reflection to figure out which serializer to use for each bit. This is still experimental functionality which might change in future.
I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.
Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.
Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}