Is there a way to marshall a struct to JSON that skips any fields that can not be serialised?
E.G. if I marshal
type aStruct struct {
request *http.Request
name string
}
the JSON representation of
type aStruct struct {
name string
}
would result?
By 'fields that can not be serialised' I mean any field that would cause json.Marshal(..) to return an 'error serializing json: unsupported type' error. Having a look through the json package now with a mind to create another Marshal function that will skip a field it fails to serialise rather than abort and return the error.
==== UPDATE ====
I have created this https://github.com/myles-mcdonnell/jsonx which satisfies my use case. At first I thought this is a terrible way to extend the base json package (it's a copy from 1.6.2 with new behaviour added) but the consensus among my colleagues is that this is the idiomatic way to do this.
Related
I am currently writing an application that queries a third party API. Previously when I have done this, I have done it the correct way and created a struct and unmarshalled the response string into the struct and accessed all the data that way. However that only works if the data structure is consistent.
I have the issue of trying to query an API where the structure is inconsistent. If the request was successful I get the response
{'status': 'ok', 'due_date': '2023-01-01', 'library': 'AIEHA1'}
but if it's unsuccessful, depending on the error type, I get different structures; some examples:
{'status': 'unauthorized', 'error': 'Field Bearer empty'}
{'status': 'not-found', 'error-details': {'type': 'file-not-found', 'file': '/index'}}
Obviously, I can pass this into a generic map but I was wondering what the proper practice is for something like this? Yes the API is terrible, yes sadly I have to use it.
One way of dealing with this is to have a structure containing all possible fields:
type apiResult struct {
Status string
DueDate string
Error string
ErrorDetails ErrorDetail
}
You unmarshal the API response, then process the API result structure to determine the actual return type.
Another way of doing it is to unmarshal into the expected struct, and then unmarshal again into an error struct based on status:
json.Unmarshal(result,&data)
if data.Status=="error" {
json.Unmarshal(result,&errorStruct)
} else if data.Status=="not-found" {
... etc.
}
When the response is not an error, this has the benefit of unmarshalling only once to the target struct.
Usually APIs return HTTP status other than 2xx code when there is an error. If that is the case, you can look at the response code and unmarshal based on that.
I have a tuple consisting of an String and a Uuid that I serialize using serde_json:
let log_and_id = (String::from("Test string"), test_id);
let log_and_id_serialized = serde_json::to_string(&log_and_id)
.expect("Serialization failed");
//After serialization (debug print): "[\"Test string\",\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\"]"
Then I transfer this serialized value over the network and receive a BytesMut (serialized_tuple) on the other end, which I try to deserialize:
//Bytesmut value (debug print): b"\"[\\\"Test string\\\",\\\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\\\"]\""
let (log, operation_state_id) = serde_json::from_slice::<(String, Uuid)>(&serialized_tuple)?;
But I get the following error:
ERROR actix_http::response] Internal Server Error: SerdeError(Error("invalid type: string \"[\\\"Test string\\\",\\\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\\\"]\", expected a tuple of size 2", line: 1, column: 68))
(De)serializing single objects this way used to work in other parts of this code, so what could cause it to fail when used with tuples?
You don't have a serialized tuple, but a serialized serialized tuple.
I mean the serialization of the tuple, which was a JSON string, was again serialized.
You can check this with this code (playground):
let serialized_tuple = b"\"[\\\"Test string\\\",\\\"32a8e12d-69d2-421d-a52e-1ee76cc03ed5\\\"]\"";
let serialized_tuple: String = serde_json::from_slice(serialized_tuple).unwrap();
let (log, operation_state_id) = serde_json::from_slice::<(String, String)>(serialized_tuple.as_bytes()).unwrap();
which produces the desired tuple.
Of course, rather than deserializing twice, you should remove the unnecessary serialization from your application (it's not in the code you've shown).
My problem is pretty much the opposite of this one: Unable to unmarshal json to protobuf struct field
I have a message with several nested messages of the following form:
message MyMsg {
uint32 id = 1;
message Attribute {
...
}
repeated Attribute attrs = 2;
message OtherAttribute {
...
}
OtherAttribute oAttr = 3;
...
}
Some external dependencies will send this message JSON form, which needs to then be unmarshalled into a go struct. When trying to use jsonpb like so, where resp is a *http.Response:
msg := &MyMsg{}
jsonpb.Unmarshal(resp.Body, msg)
The message is not fully decoded into the struct, i.e. some of the nested structs are missing. When the message is however decoded simply using encoding/json like so:
msg := &MyMsg{}
json.NewDecoder(resp.Body).Decode(msg)
All attributes are successfully decoded into the struct.
As jsonpb is the official package to (un)marshall between protobuf/json, I was wondering whether anyone has any idea as to why this type of behaviour could occur. Do the default behaviours for jsonpb and encoding/json differ in a way that would explain one being able to unmarshall and the other not? If so where would one configure the behaviour of jsonpb accordingly?
The default behaviour of encoding/json is the following:
Unknown fields are allowed, i.e. in case of a field not matching it is simply ignored without an error being raised.
Before it is ignored, the Decoder attempts to match the field without case sensitivity
The behaviour in point 1 can be replicated in jsonpb by using the Unmarshaller struct and setting the property AllowUnknownFields to true
var umrsh = jsonpb.Unmarshaler{}
umrsh.AllowUnknownFields = true
msg := &MyMsg{}
umrsh.Unmarshal(resp.Body, msg)
It does not seem to be possible to replicate the behaviour from point 2 within jsonpb.
Are there any Go libraries which can tidy up Json output before it is sent to users?
We could unmarshall into a struct and do this manually, but we would like to know if there are any libraries which can make it easier to extract keys into the struct, which we could Marshall and send to the user?
The short answer is not really, due to the way Go handles JSON marshalling and un-marshalling. The common pattern for dealing with your use case is just to define a Response struct.
A classic example would be something like the following:
type User struct {
// fields
}
// Response type used when the user is asking about their own fields
type PrivateUserResponse struct {
// fields with struct tags
}
func (u *User) ToPrivateUserResponse() *PrivateUserResponse { ... }
// Response type used when the user is being listed in a public directory
type PublicUserResponse struct {
// fields with struct tags
}
func (u *User) ToPublicUserResponse() *PublicUserResponse { ... }
Because JSON key configuration is handled by struct tags, a library would be ill-suited to handle the unique business logic cases that arise in dealing with this problem. You might be able to find a code generator that solves this in a more generic way, but I'd recommend just writing the structs yourself - Go favors explicit and clear behavior.
I have JSON (that I cannot control) like this:
{
"foo1":{
"a":{
"up":10,
"down":5
}
},
"foo2":{
"a":{
"up":1,
"down":1
}
},
"bar":{
"up":11,
"down":6
}
}
"foo1" and "foo2" are dynamic.
How can I properly unmarshal this structure in go?
It would be okay if I could just tell go to not try to deserialize "bar" (the inconsistent property).
Go will by default ignore fields unspecified in the struct you unmarshal into.
In this case, your structure would be set up like this:
type NestedProp2 struct {
Up int
Down int
}
type NestedProp struct {
A NestedProp2
}
type Prop struct {
Foo1 NestedProp
Foo2 NestedProp
}
When you call the the json.Unmarshal function, the extra property will not be deserialized:
var prop Prop
err := json.Unmarshal(jsonBlob, &prop)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", prop)
So you get the following output:
{Foo1:{A:{Up:10 Down:5}} Foo2:{A:{Up:1 Down:1}}}
You can see it in action here.
You said:
I have JSON (that I cannot control)
So to what extent you could control? Here I could provide you with some scenario, and hope some of them match your purpose :)
Remember the general rule first:
In Golang, if a JSON key failed to find a matched field in struct, it will not be unmarshalled.
This means, for a key name in a JSON struct, when unmarshalling, it will look for a field in a golang struct at the same level with the same name case-insensitively. If this search failed, this key won't be unmarshalled.
For example, a key named foo1 will look for a field name foo1 in a golang struct at the same indent level. However it also matches with Foo1 or FoO1, since this matching is case-insensitive.
Remember, you could use field tag to specify the field name as well. Please take a look at the official page.
The value of some of the JSON fields are not consistent, and they could be ignored.
This is the case #gnalck solved in his answer. According to the general rule, if those inconsistent field failed to find a match, they will not be unmarshalled. Therefore, just don't put those inconsistent fields in the struct and you will be fine.
The value of some of the JSON fields are not consistent, but they could not be ignored.
In this case, #gnalck failed since those fields could not be ignored. Now a better way is to unmarshal bar into a json.RawMessage, so that you could unmarshal later.
The keys of the JSON object is undetermined, and their value is undetermined as well.
In this case, we could unmarshal the whole JSON object into a map[string]json.RawMessage, and unmarshal each fields later. When unmarshalling to a map, you could iterate through the map to get all the fields, and unmarshal them into a proper struct later.