I have a json string. Like this:
"{"http_requests":[{"http_requests":{"code":"400","method":"PUT","value":89}},{"http_requests":{"code":"200","method":"PUT","value":45}}]}"
I want to insert this json to mongodb. But I have error in my code.
The error is "cannot transform type string to a BSON Document: WriteString can only write while positioned on a Element or Value but is positioned on a TopLevel"
func insertJson(json_value string) {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb+srv://abc:123#cluster0.wrzj3zo.mongodb.net/?retryWrites=true&w=majority"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
myDatabase := client.Database("my_db")
myCollection := myDatabase.Collection("my_collection")
myResult, err := myCollection.InsertOne(ctx, json_value)
if err != nil {
log.Fatal(err)
}
fmt.Println(myResult.InsertedID)
}
How do I insert this json string to mongodb?
First thing's first: Add a ping to check if connection is succeeding after defer client.Disconnect(ctx).
if err = client.Ping(ctx, readpref.Primary()); err != nil {
log.Fatalf("ping failed: %v", err)
}
If that doesn't throw an error, you can unmarshal your JSON string as explained in stackoverflow: How to insert a json object array to mongodb in golang. However, in this case, use interface{} instead of slice as follows:
var v interface{}
if err := json.Unmarshal([]byte(json_value), &v); err != nil {
log.Fatal(err)
}
Pass v to InsertOne.
Note: This is one way of solving the problem. However, the recommended way to go about it is to unmarshal the JSON to go struct with json and bson tags, and pass the struct instance(s) to InsertOne.
Some references:
Go by Example: JSON
How to Use Golang Structs With MongoDB
Use Struct Tags
The insertOne() method has the following syntax:
db.collection.insertOne(
<document>,
{
writeConcern: <document> (optional)
}
)
all you have to do is
myCollection.insertOne(json_metrics)
Related
I wrote some code which hits one public API and saves the JSON output in a file. But the data is storing line by line into the file instead of a single JSON format.
For eg.
Current Output:
{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}
Desired Output:
[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]
I know this should be pretty simple and i am missing out something.
My Current Code is:
To read IP from file and hit the API one by one on each IP.
func readIPfromFile(filename string, outFile string, timeout int) {
data := jsonIn{}
//open input file
jsonFile, err := os.Open(filename) //open input file
...
...
jsonData := bufio.NewScanner(jsonFile)
for jsonData.Scan() {
// marshal json data & check for logs
if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
log.Fatal(err)
}
//save to file
url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
GetGeoIP(url, outFile, timeout)
}
}
To make HTTP Request with custom request header and call write to file function.
func GetGeoIP(url string, outFile string, timeout int) {
geoClient := http.Client{
Timeout: time.Second * time.Duration(timeout), // Timeout after 5 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("accept", "application/json")
res, getErr := geoClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
jsonout := jsonOut{}
jsonErr := json.Unmarshal(body, &jsonout)
if jsonErr != nil {
log.Fatal(jsonErr)
}
file, _ := json.Marshal(jsonout)
write2file(outFile, file)
}
To Write data to file:
func write2file(outFile string, file []byte) {
f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err = f.WriteString(string(file)); err != nil {
log.Fatal(err)
}
if _, err = f.WriteString("\n"); err != nil {
log.Fatal(err)
}
I know, i can edit f.WriteString("\n"); to f.WriteString(","); to add comma but still adding [] in the file is challenging for me.
First, please do not invent a new way of json marshaling, just use golang built-in encoding/json or other library on github.
Second, if you want to create a json string that represents an array of object, you need to create the array of objects in golang and marshal it into string (or more precisely, into array of bytes)
I create a simple as below, but please DIY if possible.
https://go.dev/play/p/RR_ok-fUTb_4
I need to parse really long json file (more than million items). I don't want to load it to the memory and read it chunk by chunk. There's a good example with the array of items here. The problem is that I deal with the map. And when I call Decode I get not at beginning of value.
I can't get what should be changed.
const data = `{
"object1": {"name": "cattle","location": "kitchen"},
"object2": {"name": "table","location": "office"}
}`
type ReadObject struct {
Name string `json:"name"`
Location string `json:"location"`
}
func ParseJSON() {
dec := json.NewDecoder(strings.NewReader(data))
tkn, err := dec.Token()
if err != nil {
log.Fatalf("failed to read opening token: %v", err)
}
fmt.Printf("opening token: %v\n", tkn)
objects := make(map[string]*ReadObject)
for dec.More() {
var nextSymbol string
if err := dec.Decode(&nextSymbol); err != nil {
log.Fatalf("failed to parse next symbol: %v", err)
}
nextObject := &ReadObject{}
if err := dec.Decode(&nextObject); err != nil {
log.Fatalf("failed to parse next object")
}
objects[nextSymbol] = nextObject
}
tkn, err = dec.Token()
if err != nil {
log.Fatalf("failed to read closing token: %v", err)
}
fmt.Printf("closing token: %v\n", tkn)
fmt.Printf("OBJECTS: \n%v\n", objects)
}
TL,DR: when you are calling Token() method for a first time, you move offset from the beginning (of a JSON value) and therefore you get the error.
You are working with this struct (link):
type Decoder struct {
// others fields omits for simplicity
tokenState int
}
Pay attention for a tokenState field. This value could be one of (link):
const (
tokenTopValue = iota
tokenArrayStart
tokenArrayValue
tokenArrayComma
tokenObjectStart
tokenObjectKey
tokenObjectColon
tokenObjectValue
tokenObjectComma
)
Let's back to your code. You are calling Token() method. This method obtains first JSON-valid token { and changes tokenState from tokenObjectValue to the tokenObjectStart (link). Now you are "in-an-object" state.
If you try to call Decode() at this point you will get an error (not at beginning of value). This is because allowed states of tokenState for calling Decode() are tokenTopValue, tokenArrayStart, tokenArrayValue, tokenObjectValue, i.e. "full" value, not part of it (link).
To avoid this you can just don't call Token() at all and do something like this:
dec := json.NewDecoder(strings.NewReader(dataMapFromJson))
objects := make(map[string]*ReadObject)
if err := dec.Decode(&objects); err != nil {
log.Fatalf("failed to parse next symbol: %v", err)
}
fmt.Printf("OBJECTS: \n%v\n", objects)
Or, if you want to read chunk-by-chunk, you could keep calling Token() until you reach "full" value. And then call Decode() on this value (I guess this should work).
After consuming the initial { with your first call to dec.Token(), you must :
use dec.Token() to extract the next key
after extracting the key, you can call dec.Decode(&nextObject) to decode an entry
example code :
for dec.More() {
key, err := dec.Token()
if err != nil {
// handle error
}
var val interface{}
err = dec.Decode(&val)
if err != nil {
// handle error
}
fmt.Printf(" %s : %v\n", key, val)
}
https://play.golang.org/p/5r1d8MsNlKb
I am using http.Get in go to a url which results in the following
{"name":"cassandra","tags":["2.2.6","latest"]} that means it behaves like map[string]string for the name field but in the tags it behaves like map[string][]string so how can I unmarshal this in Go?
I tried using map[string][]string but it did not work
map_image_tags := make(map[string][]string)
res2, err := http.Get(fmt.Sprintf("%s/v2/%s/tags/lists", sconf.RegistryConf.url, Images[i]))
if err != nil {
w.WriteHeader(500)
log.Errorf("could not get tags: %s", err)
return
}
log.Debugf("OK")
js2, err := ioutil.ReadAll(res2.Body)
if err != nil {
w.WriteHeader(500)
log.Errorf("could not read body: %s", err)
return
}
log.Debugf("OK")
err = json.Unmarshal(js2, map_image_tags)
if err != nil {
w.WriteHeader(500)
log.Errorf("could not unmarshal json: %s", err)
return
}
I am getting this log error: could not unmarshal json: invalid character 'p' after top-level value
To read a json value like {"name":"cassandra", "tags":["2.2.6","latest"], you can use a struct defined as:
type mapImageTags struct {
Name string `json:"name"`
Tags []string `json:"tags"` // tags is a slice (array) of strings
}
To unmarshal JSON data,
m := mapImageTags{}
err = json.Unmarshal(js2, &m)
A simple map[string]string wont help in this case.
If the structure of json data is dynamic you can unmarshal tags into map[string]interface{}:
var encodedTags map[string]interface{}
result := json.Unmarshal([]byte(image_tags), &encodedTags)
Then you can use type assertion to unmarshal the content of tags:
var tags []interface{}
result = json.Unmarshal([]byte(encodedTags["tags"].(string)), &tags)
And here is a full working example on Go Playground.
Try map[string]interface{}, note that this method forces any numbers to float and in general not recommended when your json is complex. abhink's answer is the recommended way.
In python you can take a json object and grab a specific item from it without declaring a struct, saving to a struct then obtaining the value like in Go. Is there a package or easier way to store a specific value from json in Go?
python
res = res.json()
return res['results'][0]
Go
type Quotes struct {
AskPrice string `json:"ask_price"`
}
quote := new(Quotes)
errJson := json.Unmarshal(content, "e)
if errJson != nil {
return "nil", fmt.Errorf("cannot read json body: %v", errJson)
}
You can decode into a map[string]interface{} and then get the element by key.
func main() {
b := []byte(`{"ask_price":"1.0"}`)
data := make(map[string]interface{})
err := json.Unmarshal(b, &data)
if err != nil {
panic(err)
}
if price, ok := data["ask_price"].(string); ok {
fmt.Println(price)
} else {
panic("wrong type")
}
}
Structs are often preferred as they are more explicit about the type. You only have to declare the fields in the JSON you care about, and you don't need to type assert the values as you would with a map (encoding/json handles that implicitly).
Try either fastjson or jsonparser. jsonparser is optimized for the case when a single JSON field must be selected, while fastjson is optimized for the case when multiple unrelated JSON fields must be selected.
Below is an example code for fastjson:
var p fastjson.Parser
v, err := p.Parse(content)
if err != nil {
log.Fatal(err)
}
// obtain v["ask_price"] as float64
price := v.GetFloat64("ask_price")
// obtain v["results"][0] as generic JSON value
result0 := v.Get("results", "0")
Given a URL like the following.
http://127.0.0.1:3001/find?field=hostname&field=App&filters=["hostname":"example.com,"type":"vm"]
How do I extract JSON values corresponding to keys for eg: hostname 'example.com' and type 'vm'.
I am trying
filters := r.URL.Query()["filters"]
which gives following output:
[["hostname":"example.com,"type":"vm"]]
Use the encoding/json package to parse JSON. The query string in the example URL does not contain valid JSON.
Here's an example show how to use the JSON parser on a slightly different URL.
s := `http://127.0.0.1:3001/find?field=hostname&field=App&filters={"hostname":"example.com","type":"vm"}`
u, err := url.Parse(s)
if err != nil {
log.Fatal(err)
}
var v map[string]string
err = json.Unmarshal([]byte(u.Query().Get("filters")), &v)
if err != nil {
log.Fatal(err)
}
fmt.Println(v)
playground example