I have a Json string that I want to unmarshal.
This is working:
jsonString := []byte(`{"my_int": 3, "my_string": null}`)
var data map[string]interface{}
err := json.Unmarshal(jsonString, &data)
if err != nil {
fmt.Println(err)
}
//avroJson := make(map[string]interface{})
for k, v := range data {
fmt.Printf("%v, %T\n", k, v)
}
My issue is: the value of my_int which is 3 is returned as float64.
My question is: how to parse a json string with the "minimum type" so that 3 will return int32 and not the maximum type 3 => float64?
Assumption: my Json is huge and only have primitive types and I want a minimum value that is really float64 to continue to show float64.
Clarification:
A "minimum type" means that if 3 can be considered both int32 and float64 the "minimum type" will be int32, which is the exact type you'll get when running this:
reflect.TypeOf(3).string()
Since you are unmarshaling into a map of interface{}, the following section of the golang json.Unmarshal documentation pertains:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
...
float64, for JSON numbers
string, for JSON strings
...
As such, to unmarshal your sample data into your desired types you should define a struct type which contains the desired field/type mappings, for example:
type MyType struct {
MyInt int `json:"my_int"`
MyString *string `json:"my_string"`
}
foo := MyType{}
jsonstr := `{"my_int": 3, "my_string": null}`
err := json.Unmarshal([]byte(jsonstr), &foo)
if err != nil {
panic(err)
}
// foo => main.MyType{MyInt:3, MyString:(*string)(nil)}
Since you cannot describe your data in a struct then your options are to:
Use a json.Decoder to convert the values to your desired types as they are parsed.
Parse the document into a generic interface and post-process the value types.
Option #1 is the most flexible and can likely be implemented to be more performant than the other option since parsing and transformation could be performed in a single pass of the data.
Option #2 might be simpler but will require two passes over the data. Here is an example of what the post-processing step might look like:
func TransformValueTypes(o map[string]interface{}) {
for k, v := range o {
// Convert nil values to *string type.
if v == interface{}(nil) {
o[k] = (*string)(nil)
}
// Convert numbers to int32 if possible
if x, isnumber := v.(float64); isnumber {
if math.Floor(x) == x {
if x >= math.MinInt32 && x <= math.MaxInt32 {
o[k] = int32(x)
}
// Possibly check for other integer sizes here?
}
// Possibly check if float32 is possible here?
}
// Check for maps and slices here...
}
}
So if you call TransformValueTypes(data) then your types will look like:
// my_int -> 3 (int32)
// my_string -> <nil> (*string)
// my_string2 -> "foo" (string)
// my_float -> 1.23 (float64)
Of course, your transform function could also apply type transformation logic based on the key name.
Importantly, note that if your document might have additional structure not mentioned in your question (such as nested objects or arrays) then your transform function will need to account for them by more value type checking, recursive calls, and iteration.
Related
I'm calling an external API that i'm unmarshalling into a struct.
In the response most fields are integer but as it's json there are several edge cases where it could return a string but still be a valid / useful information :
"NaN" , "N/A"
My struct is looks like this :
type Example struct {
Field1 *int64 `json:"field_1,omitempty"`
Field2 *int64 `json:"field_2,omitempty"`
Field3 *int64 `json:"field_3,omitempty"`
}
We have several requirement :
If the api returns NaN or N/A I should display an error to my user in the FE so I'm thinking to replace the values with null while "catching" the error
beforehand that's why I've chosen a pointer value.
If no value is returned , omit the value altogether when re-marshalling.
In order to do so I'm trying to replace the "NaN" value with JSON null doing
b = bytes.Replace(b, []byte("NaN"), []byte("null"), -1) `
but it doesn't work as "null" is not equal to null and that's problem number 1.
2nd problem is that the omitempty also doesn't distinguish between nil, 0 and empty values when remarshalling.
So the remarshalling also fails. I know it's a "common" problem in go that is being fixed but is there a work around for now?
Because if I pass nil for " N/A " and "NaN" and use omitempty it will remove them. If I pass 0 it won't make sense ( business wise as 0 have meaning other than "not initialized" ) and if I remove Omitempty it will have the
whole struct marshalled everytime ( lots of unnecessary data ) and no way to differentiate between nil ( NA / NaN ) and nil ( no value ).
Last option would be to build a custom type and marshall / unmarshaller like this :
type JSONint64 struct {
value *int64
error string
}
but that would require me to check every number in my json response , every time , when in fact NaN and N/A are very rare occurrences and adds " complexity " on the front end.
I'm assuming it's a common problem as JSON is untyped , how is this generally fixed ?
I would Unmarshal to a map[string]interface{} value and then use reflection or type assertion to figure out the data type of the values.
Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Like so:
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Field1 *int64 `json:"field_1,omitempty"`
Field2 *int64 `json:"field_2,omitempty"`
Field3 *int64 `json:"field_3,omitempty"`
}
func typeof(v interface{}) string {
switch v.(type) {
case float64:
return "float64"
case string:
return "string"
default:
return "unknown"
}
}
func main() {
d := []byte(`{"field_1": "n/a", "field_2": 2 }"`)
e := make(map[string]interface{})
err := json.Unmarshal(d, &e)
if err != nil {
fmt.Printf("%v\n", err)
}
fmt.Printf("%+v\n", e)
example := Example{}
for k, v := range e {
switch typeof(v) {
case "float64":
val := int64(v.(float64))
switch k {
case "field_1":
example.Field1 = &val
case "field_2":
example.Field2 = &val
case "field_3":
example.Field3 = &val
default:
fmt.Printf("Unexpected field: %v\n", k)
}
default:
// display error
}
}
fmt.Printf("Example field_2: %d\n", *example.Field2)
}
I am dealing with an api that returns json data such as:
{
"bpi": {
"2018-06-01": 128.2597,
"2018-06-02": 127.3648
},
"disclaimer": "something here.",
"time": {
"updated": "Sep 6, 2013 00:03:00 UTC",
"updatedISO": "2013-09-06T00:03:00+00:00"
}
However the price data that has the accompanying dates can return a dynamic date range (ie could be anything from 1 data pair to 1000).
I'm trying to take only the date and price pairs and put them into a map for later consumption, but I'm not finding a straight forward way of doing it. When I put this into a json-to-go auto struct generator it will create a statically and named struct for the pricing.
This is my best attempt at handling the data dynamically. I am passing an empty interface from the response body of the http get, specifically:
var unstructuredJSON interface{}
json.Unmarshal(body, &unstructuredJSON)
and passing the unstructuredJSON to the function:
func buildPriceMap(unstructuredJSON interface{}, priceMap map[string]float64) {
jsonBody := unstructuredJSON.(map[string]interface{})
for k, v := range jsonBody {
switch vv := v.(type) {
case string:
// Do Nothing
case float64:
priceMap[k] = vv
case interface{}:
buildPriceMap(vv, priceMap)
default:
log.Fatal("Json unknown data handling unmarshal error: ", k, vv)
}
}
Is there a better way to do this?
Assuming that you know the top level keys, e.g. bpi, disclaimer, time etc and that the "dynamic data pairs" that you are talking about are part of the bpi field, and that the key and value types of each of the members of bpi are always string: decimal number you do something like....
type APIResp struct {
BPI map[string]float64 `json:"bpi"`
Disclaimer string
// other fields
}
Now each of your pairs will be typed correctly and contained in the APIResp.BPI map. Unmarshal as you are doing already;
var r APIResp
err := json.Unmarshal(body, &r)
// TODO: check err
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
I have not been able to find a way around this issue currently. If I have a structure i would like to populate with json from a http.Request I have no way to tell for instance what value was actually passed in for some values. For instance if I pass in an empty json object and run json.Decode on a structure that looks like this...
var Test struct {
Number int `json:"number"`
}
I now have a json object that supposedly was passed with a key of number and a value of zero when in fact I would rather have this return nothing at all. Does go provide another method that would actually allow me to see what JSON has been passed in or not.
Sorry for the rambling I have been trying to figure out how to to this for a few days now and it's driving me nuts.
Thanks for any help.
Edit:
I made this to depict exactly what I am talking about http://play.golang.org/p/aPFKSvuxC9
You could use pointers, for example:
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus"},
{"Name": "Quoll", "Order": 100}
]`)
type Animal struct {
Name string
Order *int
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
for _, a := range animals {
if a.Order != nil {
fmt.Printf("got order, %s : %d\n", a.Name, *a.Order)
}
}
}
I don't see how you could do this by giving a struct to the Unmarshal function. With the following structure for instance:
type A struct {
Hello string
Foo int
Baz string
}
var a A
json.Unmarshal(data, &a)
Even by doing another implementation of Unmarshal, there would be only two (simple) possibilities:
If baz is not in the json data, set a.Baz to a default value, compatible with its type: the empty string (or 0 if it's an integer). This is the current implementation.
If baz is not in the json data, return an error. That would be very inconvenient if the absence of baz is a normal behaviour.
Another possibility would be to use pointers, and use the default value nil in the same spirit than the default value I talked about, but there would still be issue if your json file could be filled with null values: you would not be able to distinguish values that were in the json file, but set as null, and values that were not in the json, and unmarshalled with nil as their default value.
However, this solution might suit you: instead of using a struct, why not using a map[string]interface{} ? The Unmarshall function would not have to add a default value to non-present fields, and it would be able to retrieve any type of data from the json file.
var b = []byte(`[{"Name": "Platypus"}, {"Name": "Quoll", "Order": 100}]`)
var m []map[string]interface{}
err := json.Unmarshal(b, &m)
fmt.Println(m)
// [map[Name:Platypus] map[Name:Quoll Order:100]]
I am trying to define a struct that can hold an array of any type like so:
type APIResonse struct {
length int
data []interface{}
}
I want the data property to be capable of holding an array of any type/struct so I can have a single response type, that will eventually be serialized to json. So what I want to be able to write is something like the following:
someStruct := getSomeStructArray()
res := &APIResponse{
length: len(someStruct),
data: someStruct,
}
enc, err := json.Marshal(res)
Is this possible in Go? I keep getting cannot use cs (type SomeType) as type []interface {} in assignment. Or do I have to create a different response type for every variation of data? Or maybe I am going about this wrong entirely / not Go-like. Any help would be much appreciated!
There are a couple of problems with that code.
You need to use interface{}, not []interface{}, also [] is called a slice, an array is a fixed number of elements like [10]string.
And your APIResponse fields aren't exported, so json.Marshal will not print out anything.
func main() {
d := []dummy{{100}, {200}}
res := &APIResponse{
Length: len(d),
Data: d,
}
enc, err := json.Marshal(res)
fmt.Println(string(enc), err)
}
playground