I want to decode a big set of data from a (static-schema) json file. The file contains exclusively numeric data, and keys are all integers. I know how to decode this json into a struct containing fields of map[string]int or map[string]float32 using json.Unmarshal. But I have no interest in string keys, I'd need to convert them to int somehow.
So what I'd like to know is:
Is there a way to achieve this, .ie getting a struct containing fields of map[int]float32 type directly from decoding,
Otherwise how to achieve this after decoding, in a memory efficient manner ?
Thanks
The JSON format only specifies key/value objects with string keys. Because of this, the encoding/json package only supports string keys as well.
The json/encoding documentation states:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If you want to use encoding/json package and move it over to a map[int]float64, you can do the following (works with float32 as well):
package main
import (
"fmt"
"strconv"
)
func main() {
a := map[string]float64{"1":1, "2":4, "3":9, "5":25}
b := make(map[int]float64, len(a))
for k,v := range a {
if i, err := strconv.Atoi(k); err == nil {
b[i] = v
} else {
// A non integer key
}
}
fmt.Printf("%#v\n", b)
}
Playground
The encoding/json package includes an interface Unmarshaler which has a single method: UnmarshalJSON(data []byte) error.
If you're feeling brave you could implement that for the following:
type IntToFloat map[int]float32
func (itf *IntToFloat) UnmarshalJSON(data []byte) error {
if itf == nil {
return errors.New("Unmarshalling JSON for a null IntToFload")
}
// MAGIC Goes here.
return nil
}
EDIT
Note: http://golang.org/src/pkg/encoding/json/decode.go?s=2221:2269#L53 is where the std library version of Unmarshal comes from.
http://golang.org/pkg/encoding/json/#Unmarshaler is where the interface referenced above comes from.
Related
Currently, I try to parse JSON to map[string][]interface{}, but unmarshalling returns an error. According to (https://golang.org/pkg/encoding/json/), to unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
-[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I wonder if golang is able to unmarshal map[string][]interface{}. The following is code snippet. I am new to Golang, thanks for help in advance.
// emailsStr looks like "{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}"
emailsRaw := make(map[string][]*entities.Email)
err := json.Unmarshal([]byte(emailsStr), &emailsRaw)
Error message:
&json.UnmarshalTypeError{Value:"number", Type:(*reflect.rtype)(0x151c7a0), Offset:44, Struct:"", Field:""}
The Go encoding/json package will only unmarshal dynamically to a map[string]interface{}. From there, you will need to use type assertions and casting to pull out the values you want, like so:
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
dynamic := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &dynamic)
firstEmail := dynamic["unknown.0"].([]interface{})[0].(map[string]interface{})["email_address"]
fmt.Println(firstEmail)
}
(https://play.golang.org/p/VEUEIwj3CIC)
Each time, Go's .(<type>) operator is used to assert and cast the dynamic value to a specific type. This particular code will panic if anything happens to be the wrong type at runtime, like if the contents of unknown.0 aren't an array of JSON objects.
The more idiomatic (and robust) way to do this in Go is to annotate a couple structs with json:"" tags and have encoding/json unmarshal into them. This avoids all the nasty brittle .([]interface{}) type casting:
type Email struct {
Email string `json:"email_address"`
}
type EmailsList struct {
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion int `json:"schemaVersion"`
Emails []Email `json:"unknown.0"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
emails := EmailsList{}
json.Unmarshal([]byte(jsonStr), &emails)
fmt.Printf("%+v\n", emails)
}
(https://play.golang.org/p/iS6e0_87P2J)
A better approach will be to use struct for main schema and then use an slice of email struct for fetching the data for email entities get the values from the same according to requirements. Please find the solution below :-
package main
import (
"fmt"
"encoding/json"
)
type Data struct{
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion float64 `json:"schemaVersion"`
EmailEntity []Email `json:"unknown.0"`
}
// Email struct
type Email struct{
EmailAddress string `json:"email_address"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
var dynamic Data
json.Unmarshal([]byte(jsonStr), &dynamic)
fmt.Printf("%#v", dynamic)
}
I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}
For example:
{["NewYork",123]}
For json array is decoded as a go array, and go array is need to explicit define a type,
I don't know How to deal with it.
First that json is invalid, objects has to have keys, so it should be something like {"key":["NewYork",123]} or just ["NewYork",123].
And when you're dealing with multiple random types, you just use interface{}.
const j = `{"NYC": ["NewYork",123]}`
type UntypedJson map[string][]interface{}
func main() {
ut := UntypedJson{}
fmt.Println(json.Unmarshal([]byte(j), &ut))
fmt.Printf("%#v", ut)
}
playground
The json package uses map[string]interface{} and []interface{} values to store arbitrary JSON objects and arrays...
http://blog.golang.org/json-and-go
Each value in an object must have key. So suppose this is your json :
{"key":["NewYork",123]}
Then your code should be like this:
package main
import (
"encoding/json"
"fmt"
)
type Message map[string]interface{}
func main() {
msg := Message{}
s := `{"key":["Newyork",123]}`
err := json.Unmarshal([]byte(s), &msg)
fmt.Println(msg, err)
}
You can run it : http://play.golang.org/p/yihj6BZHBY
I have this struct with a field of type interface{}. In the process of caching it using memcached (https://github.com/bradfitz/gomemcache), the struct is marshalled into a JSON, which is then unmarshalled back into the struct when retrieved from the cache. The resulting interface{} field inevitably points to an object of type map[string]interface{} (as in, the interface{} field can only type asserted as map[string]interface{}), the marshalling and unmarshalling process not having preserved the type information. Is there any way to save this information in the marshalling process, in such a way that it can be unmarshalled properly? Or do I have to use some other codec or something?
type A struct {
value interface{}
}
type B struct {
name string
id string
}
func main() {
a := A{value: B{name: "hi", id: "12345"}}
cache.Set("a", a) // Marshals 'a' into JSON and stores in cache
result = cache.Get("a") // Retrieves 'a' from cache and unmarshals
fmt.Printf("%s", result.value.(B).name) // Generates error saying that
// map[string]interface{} cannot be type asserted as a 'B' struct
fmt.Printf("%s", result.value.(map[string]interface{})["name"].(string)) // Correctly prints "12345"
}
Short version, no you can't do that, you have few options though.
Change A.Value to use B instead of interface{}.
Add a function to A that converts A.Value from a map to B if it isn't already B.
Use encoding/gob and store the bytes in memcache then convert it back with a function like NewA(b []byte) *A.
For using gob, you have to register each struct with it first before encoding / decoding, example:
func init() {
//where you should register your types, just once
gob.Register(A{})
gob.Register(B{})
}
func main() {
var (
buf bytes.Buffer
enc = gob.NewEncoder(&buf)
dec = gob.NewDecoder(&buf)
val = A{B{"name", "id"}}
r A
)
fmt.Println(enc.Encode(&val))
fmt.Println(dec.Decode(&r))
fmt.Printf("%#v", r)
}
JSON isn't able to encode the depth of type information that you can in Go, so you'll always get back the following basic types when unmarshalling:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
From go docs: http://golang.org/pkg/encoding/json/#Unmarshal
If you have that knowledge about the types you need, you can write some methods to construct the right variables on unmarshalling perhaps?
I have a json document and I'm using a client which decodes the document in an interface (instead of struct) as below:
var jsonR interface{}
err = json.Unmarshal(res, &jsonR)
How can I access the interface fields? I've read the go doc and blog but my head still can't get it. Their example seem to show only that you can decode the json in an interface but doesn't explain how its fields can be used.
I've tried to use a range loop but it seems the story ends when I reach a map[string]interface. The fields that I need seem to be in the interface.
for k, v := range jsonR {
if k == "topfield" {
fmt.Printf("k is %v, v is %v", k, v)
}
}
The value inside the interface depends on the json structure you're parsing. If you have a json dictionary, the dynamic type of jsonR will be: map[string]interface{}.
Here's an example.
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
a := []byte(`{"topfield": 123}`)
var v interface{}
if err := json.Unmarshal(a, &v); err != nil {
log.Fatalf("unmarshal failed: %s", err)
}
fmt.Printf("value is %v", v.(map[string]interface{})["topfield"])
}
Parsing json like this can be very difficult. The default type of a parse is map[string]interface{}. The Problem arises when you have another complex data structure within the main json(like another list or object). The best way to go about decoding json is defining a struct to hold data. Not only will the values be of the correct type but you can extract the specific data you actually care about.
Your struct can look like this:
type Top struct {
Topfield int `json:"topfield"`
}
which can be decoded like this:
a := []byte(`{"topfield": 123}`)
var data Top
json.Unmarshal(a, &data) //parse the json into data
now you can use regular struct operations to access your data like this:
value := data.Topfield
json which contains more complex data can also be easyli decoded. Perhaps you have a list in your data somewhere, you can use a struct like the following to extract it:
type Data struct {
States []string `json:"states"`
PopulationData []Country `json:"popdata"`
}
type Country struct {
Id int `json:"id"`
LastCensusPop int `json:"lcensuspopulation"`
Gdp float64 `json:"gdp"`
}
such a structure can not only parse list but also parse objects withing fields.