Converting YAML to JSON in Go - json

I have a config file in YAML format, which I am trying to output as JSON via an http API call. I am unmarshalling using gopkg.in/yaml.v2. Yaml can have non-string keys, which means that the yaml is unmarshalled as map[interface{}]interface{}, which is not supported by Go's JSON marshaller. Therefore I convert to map[string]interface{} before unmarshalling. But I still get: json: unsupported type: map[interface {}]interface" {}. I don't understand. The variable cfy is not map[interface{}]interface{}.
import (
"io/ioutil"
"net/http"
"encoding/json"
"gopkg.in/yaml.v2"
)
func GetConfig(w http.ResponseWriter, r *http.Request) {
cfy := make(map[interface{}]interface{})
f, err := ioutil.ReadFile("config/config.yml")
if err != nil {
// error handling
}
if err := yaml.Unmarshal(f, &cfy); err != nil {
// error handling
}
//convert to a type that json.Marshall can digest
cfj := make(map[string]interface{})
for key, value := range cfy {
switch key := key.(type) {
case string:
cfj[key] = value
}
}
j, err := json.Marshal(cfj)
if err != nil {
// errr handling. We get: "json: unsupported type: map[interface {}]interface" {}
}
w.Header().Set("content-type", "application/json")
w.Write(j)
}

Your solution only converts values at the "top" level. If a value is also a map (nested map), your solution does not convert those.
Also you only "copy" the values with string keys, the rest will be left out of the result map.
Here's a function that recursively converts nested maps:
func convert(m map[interface{}]interface{}) map[string]interface{} {
res := map[string]interface{}{}
for k, v := range m {
switch v2 := v.(type) {
case map[interface{}]interface{}:
res[fmt.Sprint(k)] = convert(v2)
default:
res[fmt.Sprint(k)] = v
}
}
return res
}
Testing it:
m := map[interface{}]interface{}{
1: "one",
"two": 2,
"three": map[interface{}]interface{}{
"3.1": 3.1,
},
}
m2 := convert(m)
data, err := json.Marshal(m2)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output (try it on the Go Playground):
{"1":"one","three":{"3.1":3.1},"two":2}
Some things to note:
To covert interface{} keys, I used fmt.Sprint() which will handle all types. The switch could have a dedicated string case for keys that are already string values to avoid calling fmt.Sprint(). This is solely for performance reasons, the result will be the same.
The above convert() function does not go into slices. So for example if the map contains a value which is a slice ([]interface{}) which may also contain maps, those will not be converted. For a full solution, see the lib below.
There is a lib github.com/icza/dyno which has an optimized, built-in support for this (disclosure: I'm the author). Using dyno, this is how it would look like:
var m map[interface{}]interface{} = ...
m2 := dyno.ConvertMapI2MapS(m)
dyno.ConvertMapI2MapS() also goes into and converts maps in []interface{} slices.
Also see possible duplicate: Convert yaml to json without struct

Related

Golang - Ristretto Cache returning base 64

I am encountering a problem while using ristretto cache. Indeed, i have a little api that should return me a value stored in my ristretto cache as json.
The problem is that when i call my function, the return is the json encoded in base64 and i just can't find the way to decode it.
Here is the code i have:
Part 1: the code for initializing my ristretto cache:
func InitCache() {
var err error
ristrettoCache, err = ristretto.NewCache(&ristretto.Config{
NumCounters: 3000,
MaxCost: 1e6,
BufferItems: 64,
})
if err != nil {
panic(err)
}
}
Part 2: Putting my values in cache:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, b, 1)
}
Part 3: getting the value from cache
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
json.NewEncoder(w).Encode(value)
}
The result i have when i make a call to my api is:
"eyJTeW1ib2wiOiJic2NDUllQVE8iLCJBZGRyIjoiMHgyQmNBMUFlM0U1MjQ0NzMyM0IzRWE0NzA4QTNkMTg1ODRDYWY4NWE3IiwiTHBzIjpbeyJTeW1ib2xUb2tlbiI6IkZFRyIsIlRva2VuQWRkciI6IjB4YWNGQzk1NTg1RDgwQWI2MmY2N0ExNEM1NjZDMWI3YTQ5RmU5MTE2NyIsIkxwQWRkciI6IjB4NDU5ZTJlMjQ4NGNlMDU2MWRmNTJiYzFlNjkxMzkyNDA2M2JhZDM5MCJ9LHsiU3ltYm9sVG9rZW4iOiJmQk5CIiwiVG9rZW5BZGRyIjoiMHg4N2IxQWNjRTZhMTk1OEU1MjIyMzNBNzM3MzEzQzA4NjU1MWE1Yzc2IiwiTHBBZGRyIjoiMHg3OGM2NzkzZGMxMDY1OWZlN2U0YWJhMTQwMmI5M2Y2ODljOGY0YzI3In1dfQ=="
But i want the base64 decoded version...
If I change the value b to be string when i insert it in cache like so:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, string(b), 1)
}
When i get the response, i get the stringified json like this:
"{"Symbol":"bscCRYPTO","Addr":"0x2BcA1Ae3E52447323B..."
And i can't find a way to get out of this string :/
Anyone would know how i could get the real json please?
Thank you in advance and i wish u a good day!
From my comments, I meant, in this line, value is most likely of type []byte (or []uint8 - which is the same thing)
value, found := ristrettoCache.Get(key)
JSON encoding a []byte will implicitly base64 the output - since JSON is text-based.
json.NewEncoder(w).Encode(value) // <- value is of type []byte
Inspecting the base64 you posted (https://play.golang.org/p/NAVS4qRfDM2) the underlying binary-bytes are already encoded in JSON - so no extra json.Encode is needed.
Just output the raw-bytes in your handler - and set the content-type to application/json:
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
// json.NewEncoder(w).Encode(value) // not this
w.Header().Set("Content-Type", "application/json")
if bs, ok := value.([]byte); ok {
_, err := w.Write(bs) //raw bytes (already encoded in JSON)
// check 'err'
} else {
// error unexpected type behind interface{}
}
}

Is there any convenient way to get JSON element without type assertion?

There is some inconvenience while processing JSON response from a web server.
For example, I don't know the data structure (and don't want to model it) of the JSON in advance, and just want to get the value from it!
So, for Python, I can just write
value = response["body"][4]["data"]["uid"] //response is a dictionary
But for Golang, I need to do the assertion for every element!
value := response["body"].([]interface{})[4].(map[string]interface{})["data"].(map[string]interface{})["uid"]
//response is a map[string]interface{}
This is what I write in golang to get the value I need. Do you have any suggestion on it? It there any useful tips for this kind of case?
If you model your JSON object with a struct and you unmarshal into a value of that, then you don't need those ugly indices and type assertions, you can simply refer to struct fields.
Note that you don't have to be afraid of the response being complex, you only need to model the parts you intend to use. E.g. if the response is an object with a hundred fields but you only need 2, then create a struct containing only those 2 fields.
If you don't want to model your JSON object (or can't because it's dynamic), then you may write a general utility function which gets a value based on the path (series of map keys and slice indices), which you can see in this answer: Taking a JSON string, unmarshaling it into a map[string]interface{}, editing, and marshaling it into a []byte seems more complicated then it should be
And last you may use 3rd party libs which already contain this helper functionality, such as https://github.com/icza/dyno (disclosure: I'm the author).
Using github.com/icza/dyno, it would look like this:
value, err := dyno.Get(response, "body", 4, "data", "uid")
As per icza you can create struct for JSON object. But in case you are getting JSON structure you don't know from start. Then you can create a dynamic parsing using reflections for interface which will be a recursive function to parse JSON data.
func main(){
var data interface{}
err := json.Unmarshal([]bytes(file.json), &data)
if err != nil {
panic(err)
}
var itemData map[string]interface{}
itemsMap := data.(map[string]interface{})
jsonParsedObject := interate(itemsMap)
log.Println(jsonParsedObject)
}
func iterate(data interface{}) interface{} {
if reflect.ValueOf(data).Kind() == reflect.Slice {
d := reflect.ValueOf(data)
tmpData := make([]interface{}, d.Len())
returnSlice := make([]interface{}, d.Len())
for i := 0; i < d.Len(); i++ {
tmpData[i] = d.Index(i).Interface()
}
for i, v := range tmpData {
returnSlice[i] = iterate(v)
}
return returnSlice
} else if reflect.ValueOf(data).Kind() == reflect.Map {
d := reflect.ValueOf(data)
tmpData := make(map[string]interface{})
for _, k := range d.MapKeys() {
typeOfValue := reflect.TypeOf(d.MapIndex(k).Interface()).Kind()
if typeOfValue == reflect.Map || typeOfValue == reflect.Slice {
tmpData[k.String()] = iterate(d.MapIndex(k).Interface())
} else {
tmpData[k.String()] = d.MapIndex(k).Interface()
}
}
return tmpData
}
return data
}

Go: Converting JSON string to map[string]interface{}

I'm trying to create a JSON representation within Go using a map[string]interface{} type. I'm dealing with JSON strings and I'm having a hard time figuring out how to avoid the JSON unmarshaler to automatically deal with numbers as float64s. As a result the following error occurs.
Ex.
"{ 'a' : 9223372036854775807}" should be map[string]interface{} = [a 9223372036854775807 but in reality it is map[string]interface{} = [a 9.2233720368547758088E18]
I searched how structs can be used to avoid this by using json.Number but I'd really prefer using the map type designated above.
The go json.Unmarshal(...) function automatically uses float64 for JSON numbers. If you want to unmarshal numbers into a different type then you'll have to use a custom type with a custom unmarshaler. There is no way to force the unmarshaler to deserialize custom values into a generic map.
For example, here's how you could parse values of the "a" property as a big.Int.
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type MyDoc struct {
A BigA `json:"a"`
}
type BigA struct{ *big.Int }
func (a BigA) UnmarshalJSON(bs []byte) error {
_, ok := a.SetString(string(bs), 10)
if !ok {
return fmt.Errorf("invalid integer %s", bs)
}
return nil
}
func main() {
jsonstr := `{"a":9223372036854775807}`
mydoc := MyDoc{A: BigA{new(big.Int)}}
err := json.Unmarshal([]byte(jsonstr), &mydoc)
if err != nil {
panic(err)
}
fmt.Printf("OK: mydoc=%#v\n", mydoc)
// OK: mydoc=main.MyDoc{A:9223372036854775807}
}
func jsonToMap(jsonStr string) map[string]interface{} {
result := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &result)
return result
}
Example - https://goplay.space/#ra7Gv8A5Heh
Related questions - create a JSON data as map[string]interface with the given data

How do I read one value from a JSON file using Go

I an new to programming in Go so apologies if this is something obvious. I have a JSON file named foo.json:
{"type":"fruit","name":"apple","color":"red"}
and I am writing a Go program that has to do something when the "name" value in the JSON file is "apple". It needs no other information from that JSON file as that file is used for a completely different purpose in another area of the code.
I have read documentation on Decode() and Unmarshal() and abut 30 different web pages describing how to read the whole file into structures, etc. but it all seems extremely complicated for what I want to do which is just write the correct code to implement the first 2 lines of this pseudo-code:
file, _ := os.Open("foo.json")
name = file["name"]
if (name == "apple") {
do stuff
}
such that I end up with a Go variable named name that contains the string value apple. What is the right way to do this in Go?
The easiest method to do what you want is to decode into a struct.
Provided the format remains similar to {"type":"fruit","name":"apple","color":"red"}
type Name struct {
Name string `json:"name"`
}
var data []byte
data, _ = ioutil.ReadFile("foo.json")
var str Name
_ = json.Unmarshal(data, &str)
if str.Name == "apple" {
// Do Stuff
}
Your other option is to use third party libraries such as gabs or jason.
Gabs :
jsonParsed, err := gabs.ParseJSON(data)
name, ok := jsonParsed.Path("name").Data().(string)
Jason :
v, _ := jason.NewObjectFromBytes(data)
name, _ := v.GetString("name")
Update :
The structure
type Name struct {
Name string `json:"name"`
}
is the json equivalent of {"name":"foo"}.
So unmarshaling won't work for the following json with different formats.
[{"name":"foo"}]
{"bar":{"name":"foo"}}
PS : As mentioned by W.K.S. In your case an anonymous struct would suffice since you're not using this structure for anything else.
One thing is reading a file and other one is decoding a JSON document. I leave you a full example doing both. To run it you have to have a file called file.json in the same directory of your code or binary executable:
package main
import (
"encoding/json"
"io/ioutil"
"log"
"os"
)
func main() {
f, err := os.Open("file.json") // file.json has the json content
if err != nil {
log.Fatal(err)
}
bb, err := ioutil.ReadAll(f)
if err != nil {
log.Fatal(err)
}
doc := make(map[string]interface{})
if err := json.Unmarshal(bb, &doc); err != nil {
log.Fatal(err)
}
if name, contains := doc["name"]; contains {
log.Printf("Happy end we have a name: %q\n", name)
} else {
log.Println("Json document doesn't have a name field.")
}
log.Printf("full json: %s", string(bb))
}
https://play.golang.org/p/0u04MwwGfn
I have also tried to find a simple solution such as $d = json_decode($json, true) in PHP and came to the conclusion that there is no such simple way in Golang. The following is the simplest solution I could make (the checks are skipped for clarity):
var f interface{}
err = json.Unmarshal(file, &f)
m := f.(map[string]interface{})
if (m["name"] == "apple") {
// Do something
}
where
file is an array of bytes of JSON string,
f interface serves as a generic container for unknown JSON structure,
m is a map returned by the type assertion.
We can assert that f is a map of strings, because Unmarshal() builds a variable of that type for any JSON input. At least, I couldn't make it return something different. It is possible to detect the type of a variable by means of run-time reflection:
fmt.Printf("Type of f = %s\n", reflect.TypeOf(f))
For the f variable above, the code will print Type of f = map[string]interface {}.
Example
And this is the full code with necessary checks:
package main
import (
"fmt"
"os"
"io/ioutil"
"encoding/json"
)
func main() {
// Read entire file into an array of bytes
file, err := ioutil.ReadFile("foo.json")
if (err != nil) {
fmt.Fprintf(os.Stderr, "Failed read file: %s\n", err)
os.Exit(1)
}
var f interface{}
err = json.Unmarshal(file, &f)
if (err != nil) {
fmt.Fprintf(os.Stderr, "Failed to parse JSON: %s\n", err)
os.Exit(1)
}
// Type-cast `f` to a map by means of type assertion.
m := f.(map[string]interface{})
fmt.Printf("Parsed data: %v\n", m)
// Now we can check if the parsed data contains 'name' key
if (m["name"] == "apple") {
fmt.Print("Apple found\n")
}
}
Output
Parsed data: map[type:fruit name:apple color:red]
Apple found
The proper Go way of doing this would be to decode into an instance of an anonymous struct containing only the field you need.
func main() {
myStruct := struct{ Name string }{}
json.Unmarshal([]byte(`{"type":"fruit","name":"apple","color":"red"}`), &myStruct)
fmt.Print(myStruct.Name)
}
Playground Link
Alternatively, You could use Jeffails/gabs JSON Parser:
jsonParsed,_ := gabs.ParseJSON([]byte(`{"type":"fruit","name":"apple","color":"red"}`));
value, ok = jsonParsed.Path("name").Data().(string)

Why are json package's Decode and Marshal methods used here?

In the following example from Web Development with Go by Shiju Varghese, which is for implementing a HTTP server using a new MongoDB session for each HTTP request:
Why is json package's Decode method used in PostCategory function?
Why is json package's Marshal method used in GetCategories function?
At first I thought that Decode in PostCategory and Marshal in GetCategories are opposite to each other, but later I found that there is a Unmarshal method and maybe a Encode one in the json package. So I asked a question earlier.
Here is the program
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var session *mgo.Session
type (
Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
}
DataStore struct {
session *mgo.Session
}
)
//Close mgo.Session
func (d *DataStore) Close() {
d.session.Close()
}
//Returns a collection from the database.
func (d *DataStore) C(name string) *mgo.Collection {
return d.session.DB("taskdb").C(name)
}
//Create a new DataStore object for each HTTP request
func NewDataStore() *DataStore {
ds := &DataStore{
session: session.Copy(),
}
return ds
}
//Insert a record
func PostCategory(w http.ResponseWriter, r *http.Request) {
var category Category
// Decode the incoming Category json
err := json.NewDecoder(r.Body).Decode(&category)
if err != nil {
panic(err)
}
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
//Insert record
err = c.Insert(&category)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusCreated)
}
//Read all records
func GetCategories(w http.ResponseWriter, r *http.Request) {
var categories []Category
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
iter := c.Find(nil).Iter()
result := Category{}
for iter.Next(&result) {
categories = append(categories, result)
}
w.Header().Set("Content-Type", "application/json")
j, err := json.Marshal(categories)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusOK)
w.Write(j)
}
func main() {
var err error
session, err = mgo.Dial("localhost")
if err != nil {
panic(err)
}
r := mux.NewRouter()
r.HandleFunc("/api/categories", GetCategories).Methods("GET")
r.HandleFunc("/api/categories", PostCategory).Methods("POST")
server := &http.Server{
Addr: ":8080",
Handler: r,
}
log.Println("Listening...")
server.ListenAndServe()
}
I think the main reason for using a json.NewDecoder here is to read directly from response's body (r.Body) here, since NewDecoder takes an io.Reader as an input.
You could have used json.Unmarshal but then you'd have to first read response body into a []byte and pass that value to json.Unmarshal. NewDecoder is more convenient here.
TL;DR — Marshal/Unmarshal take and return byte slices, while Encode/Decode do the same thing, but read the bytes from a stream such as a network connection (readers and writers).
The encoding/json package uses the Encoder and Decoder types to act on streams of data, that is, io.Reader's and io.Writer's. This means that you can take data directly from a network socket (or an HTTP body in this case which implements io.Reader) and transform it to JSON as the bytes come in. Doing it this way, we can go ahead and start processing that JSON as soon as any data is available but before we've received the whole document (on a slow network connection with a big document this could save us a lot of time, and for some streaming protocols with "infinitely sized" document streams this is absolutely necessary!)
Marshal and Unmarshal however operate on byte slices, which means that you have to have the entirety of the JSON document in memory before you can use them. In your example, the author uses Marshal because they already have a []byte slice so there's no point in constructing a buffer using the byte slice, then making an encoder that uses that buffer, then calling encode: Instead they can just let Marshal do that for them.
In reality, Marshal/Unmarshal are just convenience methods on top of Encoders and Decoders. If we look at the source for Unmarshal, we see that under the hood it's just constructing an encoder (or the internal representation of an encoder, but trust me, they're the same thing, if you want proof you can look at the Encode method source and see that it's also creating an encodeState) and then returning the output bytes:
func Marshal(v interface{}) ([]byte, error) {
e := &encodeState{}
err := e.marshal(v)
if err != nil {
return nil, err
}
return e.Bytes(), nil
}