I am dealing with an api that returns json data such as:
{
"bpi": {
"2018-06-01": 128.2597,
"2018-06-02": 127.3648
},
"disclaimer": "something here.",
"time": {
"updated": "Sep 6, 2013 00:03:00 UTC",
"updatedISO": "2013-09-06T00:03:00+00:00"
}
However the price data that has the accompanying dates can return a dynamic date range (ie could be anything from 1 data pair to 1000).
I'm trying to take only the date and price pairs and put them into a map for later consumption, but I'm not finding a straight forward way of doing it. When I put this into a json-to-go auto struct generator it will create a statically and named struct for the pricing.
This is my best attempt at handling the data dynamically. I am passing an empty interface from the response body of the http get, specifically:
var unstructuredJSON interface{}
json.Unmarshal(body, &unstructuredJSON)
and passing the unstructuredJSON to the function:
func buildPriceMap(unstructuredJSON interface{}, priceMap map[string]float64) {
jsonBody := unstructuredJSON.(map[string]interface{})
for k, v := range jsonBody {
switch vv := v.(type) {
case string:
// Do Nothing
case float64:
priceMap[k] = vv
case interface{}:
buildPriceMap(vv, priceMap)
default:
log.Fatal("Json unknown data handling unmarshal error: ", k, vv)
}
}
Is there a better way to do this?
Assuming that you know the top level keys, e.g. bpi, disclaimer, time etc and that the "dynamic data pairs" that you are talking about are part of the bpi field, and that the key and value types of each of the members of bpi are always string: decimal number you do something like....
type APIResp struct {
BPI map[string]float64 `json:"bpi"`
Disclaimer string
// other fields
}
Now each of your pairs will be typed correctly and contained in the APIResp.BPI map. Unmarshal as you are doing already;
var r APIResp
err := json.Unmarshal(body, &r)
// TODO: check err
Related
I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!
If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]
I think https://github.com/mitchellh/mapstructure also fits into your use case.
I'm having some trouble parsing a JSON file from an API to Go, this is the JSON I want to parse:
{"method":"stats.provider.ex",
"result":{
"addr":"17a212wdrvEXWuipCV5gcfxdALfMdhMoqh",
"current":[{
"algo":3, // algorithm number (3 = X11)
"name":"X11", // algorithm name
"suffix":"MH", // speed suffix (kH, MH, GH, TH,...)
"profitability":"0.00045845", // current profitability in BTC/suffix/Day
"data":[{ // speed object can contain following fields:
// a (accepted), rt (rejected target), rs (rejected stale),
// rd (rejected duplicate) and ro (rejected other)
// if fields are not present, speed is 0
"a":"23.09", // accepted speed (in MH/s for X11)
"rs":"0.54", // rejected speed - stale
},
"0.0001234" // balance (unpaid)
]},
... // other algorithms here
],
"past":[{
"algo":3,
"data":[
[4863234, // timestamp; multiply with 300 to get UNIX timestamp
{"a":"28.6"}, // speed object
"0" // balance (unpaid)
],[4863235,{"a":"27.4"},"0.00000345"],
... // next entries with inc. timestamps
]},
... // other algorithms here
],
"payments":[{
"amount":"0.00431400",
"fee":"0.00023000",
"TXID":"txidhere",
"time":1453538732, // UNIX timestamp
"type":0 // payment type (0 for standard NiceHash payment)
},
... // other payments here
]
}
}
You can find more info about the API in this link: https://www.nicehash.com/doc-api
The problem I'm experiencing is in the data attribute:
"data":[{ // speed object can contain following fields:
// a (accepted), rt (rejected target), rs (rejected stale),
// rd (rejected duplicate) and ro (rejected other)
// if fields are not present, speed is 0
"a":"23.09", // accepted speed (in MH/s for X11)
"rs":"0.54", // rejected speed - stale
},
"0.0001234" // balance (unpaid)
]},
Because of the balance (unpaid) line, since it doesn't have a name I don't know how to do the struct in go.
It seems that this "data" object can be described by the following struct types (assuming its shape doesn't vary from your examples):
type Data struct {
Timestamp *int64
Speed *Speed
Balance *float64
}
type Speed struct {
Accepted *float64 `json:"a,string,omitempty"`
RejectedTarget *float64 `json:"rt,string,omitempty"`
RejectedStale *float64 `json:"rs,string,omitempty"`
RejectedDuplicate *float64 `json:"rd,string,omitempty"`
RejectedOther *float64 `json:"ro,string,omitempty"`
}
The "Speed" struct has JSON tags since that object is well-suited for the default JSON un/marshaler.
The "Data" struct, however, should implement a custom json.UnmarshalJSON so that it can handle the odd choice of a JSON array with varying types to serialize its fields. Note that my sample implementation below uses the json.RawMessage type to simplify things a bit by allowing the JSON unmarshaler to ensure proper JSON array syntax and store the bytes of each element separately so we can unmarshal them according to their respective types and shapes:
// Parse valid JSON arrays as "Data" by assuming one of the following shapes:
// 1: [int64, Speed, string(float64)]
// 2: [Speed, string(float64)]
func (d *Data) UnmarshalJSON(bs []byte) error {
// Ensure that the bytes contains a valid JSON array.
msgs := []json.RawMessage{}
err := json.Unmarshal(bs, &msgs)
if err != nil {
return err
}
// Parse the initial message as "Timestamp" int64, if necessary.
idx := 0
if len(msgs) == 3 {
ts, err := strconv.ParseInt(string(msgs[idx]), 10, 64)
if err != nil {
return err
}
d.Timestamp = &ts
idx++
}
// Parse the mandatory "Speed" struct per usual.
d.Speed = &Speed{}
err = json.Unmarshal(msgs[idx], &d.Speed)
idx++
if err != nil {
return err
}
// Parse the mandatory "Balance" item after trimming quotes.
balance, err := strconv.ParseFloat(string(msgs[idx][1:len(msgs[idx])-1]), 64)
if err != nil {
return err
}
d.Balance = &balance
return nil
}
As such, you can parse valid, properly shaped JSON arrays as "Data" objects like so:
jsonstr := `[
[4863234, {"a":"28.6"}, "0" ],
[{"a":"23.09","rs":"0.54"},"0.0001234"]
]`
datas := []Data{}
err := json.Unmarshal([]byte(jsonstr), &datas)
if err != nil {
panic(err)
}
// datas[0] = Data{Timestamp:4863234,Speed{Accepted:28.6},Balance:0}
// datas[1] = Data{Speed{Accepted:23.09,RejectedStale:0.54},Balance:0.0001234}
Of course, you would also need to implement json.MarshalJSON if you want to serialize "Data" objects into JSON.
The data field in your JSON object has an array […] as its value, and
in your example that array has two elements: an object and a string apparently containing a floating-point number.
As you can see, this is an array of geterogenous types,
hence in Go, you have two options:
Create a custom type for the elements of that array, and have an
that type implement the encoding/json.Unmarshaler interface.
Then, in that method, you can go creative about interpreting what
kind of data you're about to unmarshal, and act accordingly.
Basically, you'd peek into the input data using Decoder.Token and then
unmarshal the whole input byte slice into a value of an appropriate type
Have the value for that data field to be unmarshaled into a
slice of type []interface{} and then inspect the individual elements
by a type switch
or a series of "comma ok" type asserts.
In this case, an object will be unmarshaled into a map of type
map[string]interface{}, and that string will be unmarshaled
to a value of type string.
Basically these two approaches can be classified as "detect type as you go"
vs "unmarshal everything into data structures of the most generic types
and deal with the real typing afterwards".
Here's also a third approach.
First, it may well turn out that the types of objects in the array
which is the value of that data field are implicit from their positions
in the array. You may act accordingly by unmarshaling the value of data
into an object of your custom type implementing json.Unmarshaler, which
knows which is the real type of each data element it processes.
Second, from that
{
// speed object can contain following fields:
// a (accepted), rt (rejected target), rs (rejected stale),
// rd (rejected duplicate) and ro (rejected other)
// if fields are not present, speed is 0
"a":"23.09", // accepted speed (in MH/s for X11)
"rs":"0.54", // rejected speed - stale
}
I'd say that this "object" really can have different combinations of fields,
so to me, this looks like a candidate to be unmarshaled into
into map[string]string or map[string]float,
and not into some struct-typed object.
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
I have not been able to find a way around this issue currently. If I have a structure i would like to populate with json from a http.Request I have no way to tell for instance what value was actually passed in for some values. For instance if I pass in an empty json object and run json.Decode on a structure that looks like this...
var Test struct {
Number int `json:"number"`
}
I now have a json object that supposedly was passed with a key of number and a value of zero when in fact I would rather have this return nothing at all. Does go provide another method that would actually allow me to see what JSON has been passed in or not.
Sorry for the rambling I have been trying to figure out how to to this for a few days now and it's driving me nuts.
Thanks for any help.
Edit:
I made this to depict exactly what I am talking about http://play.golang.org/p/aPFKSvuxC9
You could use pointers, for example:
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus"},
{"Name": "Quoll", "Order": 100}
]`)
type Animal struct {
Name string
Order *int
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
for _, a := range animals {
if a.Order != nil {
fmt.Printf("got order, %s : %d\n", a.Name, *a.Order)
}
}
}
I don't see how you could do this by giving a struct to the Unmarshal function. With the following structure for instance:
type A struct {
Hello string
Foo int
Baz string
}
var a A
json.Unmarshal(data, &a)
Even by doing another implementation of Unmarshal, there would be only two (simple) possibilities:
If baz is not in the json data, set a.Baz to a default value, compatible with its type: the empty string (or 0 if it's an integer). This is the current implementation.
If baz is not in the json data, return an error. That would be very inconvenient if the absence of baz is a normal behaviour.
Another possibility would be to use pointers, and use the default value nil in the same spirit than the default value I talked about, but there would still be issue if your json file could be filled with null values: you would not be able to distinguish values that were in the json file, but set as null, and values that were not in the json, and unmarshalled with nil as their default value.
However, this solution might suit you: instead of using a struct, why not using a map[string]interface{} ? The Unmarshall function would not have to add a default value to non-present fields, and it would be able to retrieve any type of data from the json file.
var b = []byte(`[{"Name": "Platypus"}, {"Name": "Quoll", "Order": 100}]`)
var m []map[string]interface{}
err := json.Unmarshal(b, &m)
fmt.Println(m)
// [map[Name:Platypus] map[Name:Quoll Order:100]]
I have a Json string that I want to unmarshal.
This is working:
jsonString := []byte(`{"my_int": 3, "my_string": null}`)
var data map[string]interface{}
err := json.Unmarshal(jsonString, &data)
if err != nil {
fmt.Println(err)
}
//avroJson := make(map[string]interface{})
for k, v := range data {
fmt.Printf("%v, %T\n", k, v)
}
My issue is: the value of my_int which is 3 is returned as float64.
My question is: how to parse a json string with the "minimum type" so that 3 will return int32 and not the maximum type 3 => float64?
Assumption: my Json is huge and only have primitive types and I want a minimum value that is really float64 to continue to show float64.
Clarification:
A "minimum type" means that if 3 can be considered both int32 and float64 the "minimum type" will be int32, which is the exact type you'll get when running this:
reflect.TypeOf(3).string()
Since you are unmarshaling into a map of interface{}, the following section of the golang json.Unmarshal documentation pertains:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
...
float64, for JSON numbers
string, for JSON strings
...
As such, to unmarshal your sample data into your desired types you should define a struct type which contains the desired field/type mappings, for example:
type MyType struct {
MyInt int `json:"my_int"`
MyString *string `json:"my_string"`
}
foo := MyType{}
jsonstr := `{"my_int": 3, "my_string": null}`
err := json.Unmarshal([]byte(jsonstr), &foo)
if err != nil {
panic(err)
}
// foo => main.MyType{MyInt:3, MyString:(*string)(nil)}
Since you cannot describe your data in a struct then your options are to:
Use a json.Decoder to convert the values to your desired types as they are parsed.
Parse the document into a generic interface and post-process the value types.
Option #1 is the most flexible and can likely be implemented to be more performant than the other option since parsing and transformation could be performed in a single pass of the data.
Option #2 might be simpler but will require two passes over the data. Here is an example of what the post-processing step might look like:
func TransformValueTypes(o map[string]interface{}) {
for k, v := range o {
// Convert nil values to *string type.
if v == interface{}(nil) {
o[k] = (*string)(nil)
}
// Convert numbers to int32 if possible
if x, isnumber := v.(float64); isnumber {
if math.Floor(x) == x {
if x >= math.MinInt32 && x <= math.MaxInt32 {
o[k] = int32(x)
}
// Possibly check for other integer sizes here?
}
// Possibly check if float32 is possible here?
}
// Check for maps and slices here...
}
}
So if you call TransformValueTypes(data) then your types will look like:
// my_int -> 3 (int32)
// my_string -> <nil> (*string)
// my_string2 -> "foo" (string)
// my_float -> 1.23 (float64)
Of course, your transform function could also apply type transformation logic based on the key name.
Importantly, note that if your document might have additional structure not mentioned in your question (such as nested objects or arrays) then your transform function will need to account for them by more value type checking, recursive calls, and iteration.