I am using http.Get in go to a url which results in the following
{"name":"cassandra","tags":["2.2.6","latest"]} that means it behaves like map[string]string for the name field but in the tags it behaves like map[string][]string so how can I unmarshal this in Go?
I tried using map[string][]string but it did not work
map_image_tags := make(map[string][]string)
res2, err := http.Get(fmt.Sprintf("%s/v2/%s/tags/lists", sconf.RegistryConf.url, Images[i]))
if err != nil {
w.WriteHeader(500)
log.Errorf("could not get tags: %s", err)
return
}
log.Debugf("OK")
js2, err := ioutil.ReadAll(res2.Body)
if err != nil {
w.WriteHeader(500)
log.Errorf("could not read body: %s", err)
return
}
log.Debugf("OK")
err = json.Unmarshal(js2, map_image_tags)
if err != nil {
w.WriteHeader(500)
log.Errorf("could not unmarshal json: %s", err)
return
}
I am getting this log error: could not unmarshal json: invalid character 'p' after top-level value
To read a json value like {"name":"cassandra", "tags":["2.2.6","latest"], you can use a struct defined as:
type mapImageTags struct {
Name string `json:"name"`
Tags []string `json:"tags"` // tags is a slice (array) of strings
}
To unmarshal JSON data,
m := mapImageTags{}
err = json.Unmarshal(js2, &m)
A simple map[string]string wont help in this case.
If the structure of json data is dynamic you can unmarshal tags into map[string]interface{}:
var encodedTags map[string]interface{}
result := json.Unmarshal([]byte(image_tags), &encodedTags)
Then you can use type assertion to unmarshal the content of tags:
var tags []interface{}
result = json.Unmarshal([]byte(encodedTags["tags"].(string)), &tags)
And here is a full working example on Go Playground.
Try map[string]interface{}, note that this method forces any numbers to float and in general not recommended when your json is complex. abhink's answer is the recommended way.
Related
I have a json string. Like this:
"{"http_requests":[{"http_requests":{"code":"400","method":"PUT","value":89}},{"http_requests":{"code":"200","method":"PUT","value":45}}]}"
I want to insert this json to mongodb. But I have error in my code.
The error is "cannot transform type string to a BSON Document: WriteString can only write while positioned on a Element or Value but is positioned on a TopLevel"
func insertJson(json_value string) {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb+srv://abc:123#cluster0.wrzj3zo.mongodb.net/?retryWrites=true&w=majority"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
myDatabase := client.Database("my_db")
myCollection := myDatabase.Collection("my_collection")
myResult, err := myCollection.InsertOne(ctx, json_value)
if err != nil {
log.Fatal(err)
}
fmt.Println(myResult.InsertedID)
}
How do I insert this json string to mongodb?
First thing's first: Add a ping to check if connection is succeeding after defer client.Disconnect(ctx).
if err = client.Ping(ctx, readpref.Primary()); err != nil {
log.Fatalf("ping failed: %v", err)
}
If that doesn't throw an error, you can unmarshal your JSON string as explained in stackoverflow: How to insert a json object array to mongodb in golang. However, in this case, use interface{} instead of slice as follows:
var v interface{}
if err := json.Unmarshal([]byte(json_value), &v); err != nil {
log.Fatal(err)
}
Pass v to InsertOne.
Note: This is one way of solving the problem. However, the recommended way to go about it is to unmarshal the JSON to go struct with json and bson tags, and pass the struct instance(s) to InsertOne.
Some references:
Go by Example: JSON
How to Use Golang Structs With MongoDB
Use Struct Tags
The insertOne() method has the following syntax:
db.collection.insertOne(
<document>,
{
writeConcern: <document> (optional)
}
)
all you have to do is
myCollection.insertOne(json_metrics)
I have the following type in Golang:
type Base64Data []byte
In order to support unmarshalling a base64 encoded string to this type, I did the following:
func (b *Base64Data) UnmarshalJSON(data []byte) error {
if len(data) == 0 {
return nil
}
content, err := base64.StdEncoding.DecodeString(string(data[1 : len(data)-1]))
if err != nil {
return err
}
*b = []byte(xml)
return nil
}
Now I also want to be able to marshal and unmarshal it to mongo database, using mgo Golang library.
The problem is that I already have documents there stored as base64 encoded string, so I have to maintain that.
I tried to do the following:
func (b Base64Data) GetBSON() (interface{}, error) {
return base64.StdEncoding.EncodeToString([]byte(b)), nil
}
func (b *Base64DecodedXml) SetBSON(raw bson.Raw) error {
var s string
var err error
if err = raw.Unmarshal(&s); err != nil {
return err
}
*b, err = base64.StdEncoding.DecodeString(s)
return err
}
So that after unmarshaling, the data is already decoded, so I need to encode it back, and return it as a string so it will be written to db as a string (and vice versa)
For that I implemented bson getter and setter, but it seems only the getter is working properly
JSON unmarshaling from base64 encoded string works, as well marshaling it to database. but unmarshling setter seems to not be called at all.
Can anyone suggest what I'm missing, so that I'll be able to properly hold the data decoded in memory, but encoded string type?
This is a test I tried to run:
b := struct {
Value shared.Base64Data `json:"value" bson:"value"`
}{}
s := `{"value": "PHJvb3Q+aGVsbG88L3Jvb3Q+"}`
require.NoError(t, json.Unmarshal([]byte(s), &b))
t.Logf("%v", string(b.Value))
b4, err := bson.Marshal(b)
require.NoError(t, err)
t.Logf("%v", string(b4))
require.NoError(t, bson.Unmarshal(b4, &b))
t.Logf("%v", string(b.Value))
You can't marshal any value with bson.Marshal(), only maps and struct values.
If you want to test it, pass a map, e.g. bson.M to bson.Marshal():
var x = Base64Data{0x01, 0x02, 0x03}
dd, err := bson.Marshal(bson.M{"data": x})
fmt.Println(string(dd), err)
Your code works as-is, and as you intend it to. Try to insert a wrapper value to verify it:
c := sess.DB("testdb").C("testcoll")
var x = Base64Data{0x01, 0x02, 0x03}
if err := c.Insert(bson.M{
"data": x,
}); err != nil {
panic(err)
}
This will save the data as a string, being the Base64 encoded form.
Of course if you want to load it back into a value of type Base64Data, you will also need to define the SetBSON(raw Raw) error method too (bson.Setter interface).
I have an end-point that receives data from a Google PubSub request. As per this repo, the object is as so:
type pushRequest struct {
Message struct {
Attributes map[string]string
Data []byte
ID string `json:"message_id"`
}
Subscription string
}
The Data field is consistently formatted as so:
type Data struct {
Key string `json:"key"`
Body string `json:"body"`
Meta map[string]interface{} `json:"meta"`
}
I can obviously unmarshal the JSON request with something like this:
f := &pushRequest{}
json.Unmarshal(msg, &f)
That leaves with the the []bytes field. Which I can do something like this to convert to a string, as per the docs
messages = append(messages, string(f.Message.Data))
Which doesn't help, since I need it as a struct.
I can Unmarshal the array again:
var m Data
json.Unmarshal(f.Message.Data, &m)
Have tried changing the field type in the pushRequest struct to Data without success. Blank...
Is there a way I can unpack things in a single pass? Doing it twice seems ridiculous.
If it's obvious, I just can't see it!
decoder := json.NewDecoder(r.Body)
psmsg := &PushRequest{}
decoderErr := decoder.Decode(&psmsg)
if decoderErr != nil {
// Error...
return
}
data := Data{}
unmarshalErr := json.Unmarshal([]byte(string(psmsg.Message.Data)), &data)
if unmarshalErr != nil {
// Error...
return
}
Here is a snippet from my cloud function, which serves as a pub/sub push endpoint. The key is that you first have to decode the body using the PushRequest struct. Next, you can transform the message data into a struct. According to the documentation, the Data field within Message is a base-64 encoded string, therefore you have to decode it first.
type PushRequest struct {
Message pubsub.PubsubMessage `json:"message"`
Subscription string `json:"subscription"`
}
type Example struct {
ID string `json:"id" firestore:"id"`
}
func HTTPEndpoint(w http.ResponseWriter, r *http.Request) {
var pr common.PushRequest
if err := json.NewDecoder(r.Body).Decode(&pr); err != nil {
log.Fatalf("Could not decode body: %v", err)
return
}
data, err := base64.StdEncoding.DecodeString(pr.Message.Data)
if err != nil {
log.Fatalf("Base64: %v", err)
return
}
var example Example
if err := json.Unmarshal(data, &example); err != nil {
log.Fatalf("Json: %v", err)
return
}
// Do something useful with the struct
}
In python you can take a json object and grab a specific item from it without declaring a struct, saving to a struct then obtaining the value like in Go. Is there a package or easier way to store a specific value from json in Go?
python
res = res.json()
return res['results'][0]
Go
type Quotes struct {
AskPrice string `json:"ask_price"`
}
quote := new(Quotes)
errJson := json.Unmarshal(content, "e)
if errJson != nil {
return "nil", fmt.Errorf("cannot read json body: %v", errJson)
}
You can decode into a map[string]interface{} and then get the element by key.
func main() {
b := []byte(`{"ask_price":"1.0"}`)
data := make(map[string]interface{})
err := json.Unmarshal(b, &data)
if err != nil {
panic(err)
}
if price, ok := data["ask_price"].(string); ok {
fmt.Println(price)
} else {
panic("wrong type")
}
}
Structs are often preferred as they are more explicit about the type. You only have to declare the fields in the JSON you care about, and you don't need to type assert the values as you would with a map (encoding/json handles that implicitly).
Try either fastjson or jsonparser. jsonparser is optimized for the case when a single JSON field must be selected, while fastjson is optimized for the case when multiple unrelated JSON fields must be selected.
Below is an example code for fastjson:
var p fastjson.Parser
v, err := p.Parse(content)
if err != nil {
log.Fatal(err)
}
// obtain v["ask_price"] as float64
price := v.GetFloat64("ask_price")
// obtain v["results"][0] as generic JSON value
result0 := v.Get("results", "0")
Given a URL like the following.
http://127.0.0.1:3001/find?field=hostname&field=App&filters=["hostname":"example.com,"type":"vm"]
How do I extract JSON values corresponding to keys for eg: hostname 'example.com' and type 'vm'.
I am trying
filters := r.URL.Query()["filters"]
which gives following output:
[["hostname":"example.com,"type":"vm"]]
Use the encoding/json package to parse JSON. The query string in the example URL does not contain valid JSON.
Here's an example show how to use the JSON parser on a slightly different URL.
s := `http://127.0.0.1:3001/find?field=hostname&field=App&filters={"hostname":"example.com","type":"vm"}`
u, err := url.Parse(s)
if err != nil {
log.Fatal(err)
}
var v map[string]string
err = json.Unmarshal([]byte(u.Query().Get("filters")), &v)
if err != nil {
log.Fatal(err)
}
fmt.Println(v)
playground example