My problem is pretty much the opposite of this one: Unable to unmarshal json to protobuf struct field
I have a message with several nested messages of the following form:
message MyMsg {
uint32 id = 1;
message Attribute {
...
}
repeated Attribute attrs = 2;
message OtherAttribute {
...
}
OtherAttribute oAttr = 3;
...
}
Some external dependencies will send this message JSON form, which needs to then be unmarshalled into a go struct. When trying to use jsonpb like so, where resp is a *http.Response:
msg := &MyMsg{}
jsonpb.Unmarshal(resp.Body, msg)
The message is not fully decoded into the struct, i.e. some of the nested structs are missing. When the message is however decoded simply using encoding/json like so:
msg := &MyMsg{}
json.NewDecoder(resp.Body).Decode(msg)
All attributes are successfully decoded into the struct.
As jsonpb is the official package to (un)marshall between protobuf/json, I was wondering whether anyone has any idea as to why this type of behaviour could occur. Do the default behaviours for jsonpb and encoding/json differ in a way that would explain one being able to unmarshall and the other not? If so where would one configure the behaviour of jsonpb accordingly?
The default behaviour of encoding/json is the following:
Unknown fields are allowed, i.e. in case of a field not matching it is simply ignored without an error being raised.
Before it is ignored, the Decoder attempts to match the field without case sensitivity
The behaviour in point 1 can be replicated in jsonpb by using the Unmarshaller struct and setting the property AllowUnknownFields to true
var umrsh = jsonpb.Unmarshaler{}
umrsh.AllowUnknownFields = true
msg := &MyMsg{}
umrsh.Unmarshal(resp.Body, msg)
It does not seem to be possible to replicate the behaviour from point 2 within jsonpb.
Related
I'm using Beego framework to build a web application in Go. I have to validate the incoming JSON in an API request.
I can unmarshal the JSON into a struct which works fine, but I want to validate the data as well. For example, if the type doesn't match with the type in struct json.Unmarshal will reutrn an error on the first occurence. I want to validate and get all the errors at once for JSON.
I've tried https://github.com/thedevsaddam/govalidator but the package needs a reference to request object which is not available in the controller of Beego. There are other validators which can validate a struct but i want the json validation as well.
EDIT:
A reference to the request object in beego can be found from the controller's context object as:
func (this *MainController) Post() {
fmt.Println(this.Ctx.Request)
}
But the problem remains same with json unmarshal. If there's any slight mismatch in the type, json.unmarshal will immediately panic. I want to be able to validate the type as well.
Since Golang v1.9 json.Valid([]byte) has been a valid method available from the "encoding/json" package.
Reference:
https://golang.org/pkg/encoding/json/#Valid
We've used go-playground/validator.v8 for a similar purpose. You can define the data validations with the tags that come out of the box (basic stuff like equality, min/max and even has somthing of an expression lang). On top of that you can add your custom validations. It's all in the docs, hope it helps.
There is a method checkValid (in encoding/json package) which can do that. This method, however, is not exposed. So, you can use Unmarshal (in same package) to check if JSON is valid.
I had to do this today so I will share my solution. I created a function using unmarshal that returns true or false based on whether the text returns valid json:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(s string) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(s), &j); err != nil {
return false
}
return true
}
Then I used that logic with the validator package to make and register a custom validation tag that I can use in any struct to validate that a field contains json. You can use go playground to check the full solution and experiment:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(fl validator.FieldLevel) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(fl.Field().String()), &j); err != nil {
return false
}
return true
}
// register the function with json tag:
validate.RegisterValidation("json", isJSON)
I am using lua in asterisk pbx. I encounter following problem while processing json string.
json "null" value converted to function type in lua. why?
how to handle this scenario? because i am expecting nil because no value means null in json and nil means nothing in lua.
local json = require( "json" )
local inspect = require("inspect")
local myjson_str='{"Sms":{"key":"xxxxxxxxxxxxxxxxxxxxx","to":"{caller}","senderid":null,"type":"Simple","content":"Your request has been accepted in Previous Miss call. We get back to you very soon."}}'
local myjson_table = json.decode(myjson_str)
print(type(myjson_table["Sms"]["senderid"]))
print(myjson_table)
print(inspect(myjson_table))
print(json.encode(myjson_table))
out put for above is
function
table: 0xf5e770
{
Sms = {
content = "Your request has been accepted in Previous Miss call. We get back to you very soon.",
key = "xxxxxxxxxxxxxxxxxxxxx",
senderid = <function 1>,
to = "{caller}",
type = "Simple"
}
}
{"Sms":{"type":"Simple","key":"xxxxxxxxxxxxxxxxxxxxx","senderid":null,"content":"Your request has been accepted in Previous Miss call. We get back to you very soon.","to":"{caller}"}}
It is up to specific library to decide how to represent null value.
Using nil has its own problem because its not possible find either
original JSON has key with null value or there no such key at all.
So some libraries just return some unique value. Some provide
a way to pass this value like json.deconde(str, NULL_VALUE).
So answer is just read the doc/source of library you use.
Most likely it provide something like json.null value to check
either value is null. But function is really strange choice because
they have some undetermined rules of uniqueness.
Or try another library.
First of all, #moteus is right:
It is up to specific library to decide how to represent null value
If you're using the JSON library by Jeffrey Friedl the solution is to use a placeholder instead of null and serializing the table structure to a json string using designated encode options:
-- define a placeholder
NullPlaceholder = "\0"
-- use it in an internal table
tableStructure = {}
tableStructure['someNullValue'] = NullPlaceholder
-- pass the placeholder to the encode methode
encode_options = { null = NullPlaceholder }
jsonString = JSON:encode(tableStructure, nil, encode_options)
which leads to
{"someNullValue": null}
Is there a way to marshall a struct to JSON that skips any fields that can not be serialised?
E.G. if I marshal
type aStruct struct {
request *http.Request
name string
}
the JSON representation of
type aStruct struct {
name string
}
would result?
By 'fields that can not be serialised' I mean any field that would cause json.Marshal(..) to return an 'error serializing json: unsupported type' error. Having a look through the json package now with a mind to create another Marshal function that will skip a field it fails to serialise rather than abort and return the error.
==== UPDATE ====
I have created this https://github.com/myles-mcdonnell/jsonx which satisfies my use case. At first I thought this is a terrible way to extend the base json package (it's a copy from 1.6.2 with new behaviour added) but the consensus among my colleagues is that this is the idiomatic way to do this.
I have JSON (that I cannot control) like this:
{
"foo1":{
"a":{
"up":10,
"down":5
}
},
"foo2":{
"a":{
"up":1,
"down":1
}
},
"bar":{
"up":11,
"down":6
}
}
"foo1" and "foo2" are dynamic.
How can I properly unmarshal this structure in go?
It would be okay if I could just tell go to not try to deserialize "bar" (the inconsistent property).
Go will by default ignore fields unspecified in the struct you unmarshal into.
In this case, your structure would be set up like this:
type NestedProp2 struct {
Up int
Down int
}
type NestedProp struct {
A NestedProp2
}
type Prop struct {
Foo1 NestedProp
Foo2 NestedProp
}
When you call the the json.Unmarshal function, the extra property will not be deserialized:
var prop Prop
err := json.Unmarshal(jsonBlob, &prop)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", prop)
So you get the following output:
{Foo1:{A:{Up:10 Down:5}} Foo2:{A:{Up:1 Down:1}}}
You can see it in action here.
You said:
I have JSON (that I cannot control)
So to what extent you could control? Here I could provide you with some scenario, and hope some of them match your purpose :)
Remember the general rule first:
In Golang, if a JSON key failed to find a matched field in struct, it will not be unmarshalled.
This means, for a key name in a JSON struct, when unmarshalling, it will look for a field in a golang struct at the same level with the same name case-insensitively. If this search failed, this key won't be unmarshalled.
For example, a key named foo1 will look for a field name foo1 in a golang struct at the same indent level. However it also matches with Foo1 or FoO1, since this matching is case-insensitive.
Remember, you could use field tag to specify the field name as well. Please take a look at the official page.
The value of some of the JSON fields are not consistent, and they could be ignored.
This is the case #gnalck solved in his answer. According to the general rule, if those inconsistent field failed to find a match, they will not be unmarshalled. Therefore, just don't put those inconsistent fields in the struct and you will be fine.
The value of some of the JSON fields are not consistent, but they could not be ignored.
In this case, #gnalck failed since those fields could not be ignored. Now a better way is to unmarshal bar into a json.RawMessage, so that you could unmarshal later.
The keys of the JSON object is undetermined, and their value is undetermined as well.
In this case, we could unmarshal the whole JSON object into a map[string]json.RawMessage, and unmarshal each fields later. When unmarshalling to a map, you could iterate through the map to get all the fields, and unmarshal them into a proper struct later.
I apologise in advance for a very long question. Hopefully you'll bear with me.
I'm working with the goweb library, and experimenting with the example web app.
I've been trying to modify the RESTful example code, which defines a Thing as:
type Thing struct {
Id string
Text string
}
A Thing is created by sending an HTTP Post request, with an appropriate JSON body, to http://localhost:9090/things. This is handled in the example code in the Create function, specifically the lines:
dataMap := data.(map[string]interface{})
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text = dataMap["Text"].(string)
This is all well and good, and I can run the example server (which listens on http://localhost:9090/) and the server operates as expected.
For example:
curl -X POST -H "Content-Type: application/json" -d '{"Id":"TestId","Text":"TestText"}' http://localhost:9090/things
returns with no error, and then I GET that Thing with
curl http://localhost:9090/things/TestId
and it returns
{"d":{"Id":"TestId","Text":"TestText"},"s":200}
So far, so good.
Now, I would like to modify the Thing type, and add a custom ThingText type, like so:
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
This in itself isn't an issue, and I can modify the Create function like so:
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text.Title = dataMap["Title"].(string)
thing.Text.Body = dataMap["Body"].(string)
and the run the previous curl POST request with the JSON set to:
{"Id":"TestId","Title":"TestTitle","Title":"TestBody"}
and it returns with no error.
Once again I can GET the Thing URL and it returns:
{"d":{"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}},"s":200}
Again, so far, so good.
Now, my question:
how do I modify the Create function to allow me to POST complex JSON to it?
For example, that last returned JSON string above includes {"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}}. I'd like to be able to POST that exact JSON to the endpoint and have the Thing created.
I've followed the code back, and it seems that the data variable is of type Context.RequestData() from https://github.com/stretchr/goweb/context, and the internal Map seems to be of type Object.Map from https://github.com/stretchr/stew/, described as "a map[string]interface{} with additional helpful functionality." in particular, I noticed "Supports dot syntax to set deep values."
I can't work out how I can set up the thing.Text.Title = dataMap... statement so that the correct JSON field is parsed into it. I can't seem to use anything other than string types in the dataMap, and if I try that JSON it gives an error similar to:
http: panic serving 127.0.0.1:59113: interface conversion: interface is nil, not string
Once again, sorry for the ridiculously long question. I really appreciate you reading, and any help you may have to offer. Thanks!
As the JSON package documentation and JSON and Go introduction describe, JSON data can be parsed either generically through interface{} / string maps or unmarshalling directly to the struct types.
The example code you linked to and you based your changes on seems to use the generic string-map approach, dataMap := data.(map[string]interface{}).
As your desired JSON data is an object in an object, it’s simply a map within a map.
So you should be able to
dataMap := data.(map[string]interface{})
subthingMap := dataMap["Text"].(map[string]interface{})
thing.Text.Title = subthingMap["Title"].(string)
thing.Text.Body = subthingMap["Body"].(string)
I’m not sure why that code uses casts and generic types over type-safe unmarshalling directly from JSON to struct types (abstraction I guess). Using the json packages unmarshalling to struct types would go something like
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
…
decoder := json.NewDecoder(body)
var thingobj Thing
for {
if err := decoder.Decode(&thingobj); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(thingobj)
}
where body is a io.Reader - in simple/most cases from http.Response.Body.