I'm new to Go and trying to write a simple web crawler. I'm using duck duck go's api and trying to display search results.
https://duckduckgo.com/api
This is my code -
package main
import (
"fmt"
"net/http"
)
func main() {
getDuckDuckGo("food")
}
func getDuckDuckGo(keyword string) <- chan string{
resp, _ := http.Get("http://api.duckduckgo.com/?q=" + keyword + "&format=json&pretty=1")
c := make(chan string)
fmt.Println(resp)
var respMap map[string]interface{}
fmt.Println(respMap)
fmt.Println(respMap)
return c
}
My resp println gives me this -
&{200 OK 200 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Type:[application/x-javascript] Date:[Sat, 20 Dec 2014 00:41:49 GMT] Cache-Control:[max-age=1] Expires:[Sat, 20 Dec 2014 00:41:50 GMT] Server:[nginx] X-Duckduckgo-Locale:[en_US]] 0xf840053c20 -1 [chunked] false map[] 0xf84007c000}
Rather than any json.
Am I doing the GET request correctly?
At least, you should do below things:
Check error of http.Get()
Get io.Reader by resp.Body for HTTP body data
Use json.Decoder to decode json
Your getDuckDuckGo() should be became like this:
func getDuckDuckGoImproved(k string) (map[string]interface{}, error) {
resp, err := http.Get("http://api.duckduckgo.com/?=" + k + "&format=json")
if err != nil {
return nil, err
}
defer resp.Body.Close()
r := make(map[string]interface{})
d := json.NewDecoder(resp.Body)
if err := d.Decode(&r); err != nil {
return nil, err
}
return r, nil
}
Related
I inherited someone else's code for an API and since I'm not familiar with the requests that it's receiving I'm trying to print them or log them so I can see their structure. From what I've read about Go, jsons are decoded with Structs but since I don't know how the requests are received I cant write a struct.
I've tried the following on a basic API but they just print me out an empty map or nothing at all:
func createBook(w http.ResponseWriter, r *http.Request) {
var result map[string]interface{}
_ = json.NewDecoder(r.Body).Decode(&result)
fmt.Println(result)
func createBook(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var book Book
_ = json.NewDecoder(r.Body).Decode(&book)
buf := new(bytes.Buffer)
buf.ReadFrom(r.Body)
newStr := buf.String()
reader := strings.NewReader(newStr)
writter := os.Stdout
dec := json.NewDecoder(reader)
enc := json.NewEncoder(writter)
for {
var m map[string]interface{}
if err := dec.Decode(&m); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
if err := enc.Encode(&m); err != nil {
log.Println(err)
}
fmt.Println(m)
}
book.ID = strconv.Itoa(rand.Intn(100000000)) // Mock ID - not safe
books = append(books, book)
json.NewEncoder(w).Encode(book)
}
Is there any other way that it would print the received json without me knowing the strut beforehand?
Use json.Unmarshal function
import "bytes"
func createBook(w http.ResponseWriter, r *http.Request) {
var result map[string]interface{}
data :+ StreamToByte(r.Body)
err := json.Unmarshal(data, &result)
if err !=nil{
fmt.Println(err) //better to use log
}else
fmt.Println(result)
}
}
func StreamToByte(stream io.Reader) []byte {
buf := new(bytes.Buffer)
buf.ReadFrom(stream)
return buf.Bytes()
}
Refer :
https://appdividend.com/2020/02/28/golang-how-to-convert-json-to-map-in-go/
https://gist.github.com/dixudx/3989284b142414e10352fde9def5c771
I am using github.com/pressly/chi to build this simple program where I try to decode some JSON from the http.Request.Body:
package main
import (
"encoding/json"
"fmt"
"net/http"
"github.com/pressly/chi"
"github.com/pressly/chi/render"
)
type Test struct {
Name string `json:"name"`
}
func (p *Test) Bind(r *http.Request) error {
err := json.NewDecoder(r.Body).Decode(p)
if err != nil {
return err
}
return nil
}
func main() {
r := chi.NewRouter()
r.Post("/products", func(w http.ResponseWriter, r *http.Request) {
var p Test
// err := render.Bind(r, &p)
err := json.NewDecoder(r.Body).Decode(&p)
if err != nil {
panic(err)
}
fmt.Println(p)
})
http.ListenAndServe(":8080", r)
}
When I don't use render.Bind() (from "github.com/pressly/chi/render"), it works as expected.
However, when I uncomment the line err := render.Bind(r, &p) and I comment the line err := json.NewDecoder(r.Body).Decode(&p), it panics with EOF :
2017/06/20 22:26:39 http: panic serving 127.0.0.1:39696: EOF
and thus the json.Decode() fails.
Am I doing something wrong or is the http.Request.Body is already read somewhere else before render.Bind() is called?
render.Bind's purpose is to perform decode and execute Bind(r) to do post decode operations.
For eg.:
type Test struct {
Name string `json:"name"`
}
func (p *Test) Bind(r *http.Request) error {
// At this point, Decode is already done by `chi`
p.Name = p.Name + " after decode"
return nil
}
If you have to do only JSON decode no other actions needs to be done after decode with respect to decoded values. Just use:
// Use Directly JSON decoder of std pkg
err := json.NewDecoder(r.Body).Decode(&p)
OR
// Use wrapper method from chi DecodeJSON
err := render.DecodeJSON(r.Body, &p)
Question: Currently I'm printing out my response in the func Index
like this fmt.Fprintf(w, string(response)) however, how can I send JSON properly in the request so that it maybe consumed by a view?
package main
import (
"fmt"
"github.com/julienschmidt/httprouter"
"net/http"
"log"
"encoding/json"
)
type Payload struct {
Stuff Data
}
type Data struct {
Fruit Fruits
Veggies Vegetables
}
type Fruits map[string]int
type Vegetables map[string]int
func Index(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
response, err := getJsonResponse();
if err != nil {
panic(err)
}
fmt.Fprintf(w, string(response))
}
func main() {
router := httprouter.New()
router.GET("/", Index)
log.Fatal(http.ListenAndServe(":8080", router))
}
func getJsonResponse()([]byte, error) {
fruits := make(map[string]int)
fruits["Apples"] = 25
fruits["Oranges"] = 10
vegetables := make(map[string]int)
vegetables["Carrats"] = 10
vegetables["Beets"] = 0
d := Data{fruits, vegetables}
p := Payload{d}
return json.MarshalIndent(p, "", " ")
}
You can set your content-type header so clients know to expect json
w.Header().Set("Content-Type", "application/json")
Another way to marshal a struct to json is to build an encoder using the http.ResponseWriter
// get a payload p := Payload{d}
json.NewEncoder(w).Encode(p)
Other users were commenting that the Content-Type is plain/text when encoding.
You have to set the content type with w.Header().Set() first, then write the HTTP response code with w.WriteHeader().
If you call w.WriteHeader() first, then call w.Header().Set() after you will get plain/text.
An example handler might look like this:
func SomeHandler(w http.ResponseWriter, r *http.Request) {
data := SomeStruct{}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(data)
}
You can do something like this in you getJsonResponse function -
jData, err := json.Marshal(Data)
if err != nil {
// handle error
}
w.Header().Set("Content-Type", "application/json")
w.Write(jData)
In gobuffalo.io framework I got it to work like this:
// say we are in some resource Show action
// some code is omitted
user := &models.User{}
if c.Request().Header.Get("Content-type") == "application/json" {
return c.Render(200, r.JSON(user))
} else {
// Make user available inside the html template
c.Set("user", user)
return c.Render(200, r.HTML("users/show.html"))
}
and then when I want to get JSON response for that resource I have to set "Content-type" to "application/json" and it works.
I think Rails has more convenient way to handle multiple response types, I didn't see the same in gobuffalo so far.
You may use this package renderer, I have written to solve this kind of problem, it's a wrapper to serve JSON, JSONP, XML, HTML etc.
This is a complement answer with a proper example:
func (ch captureHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodPost:
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, fmt.Sprintf("error reading request body, %v", err), http.StatusInternalServerError)
return
}
...do your stuff here...
case http.MethodGet:
w.Header().Set("Content-Type", "application/json")
err := json.NewEncoder(w).Encode( ...put your object here...)
if err != nil {
http.Error(w, fmt.Sprintf("error building the response, %v", err), http.StatusInternalServerError)
return
}
default:
http.Error(w, fmt.Sprintf("method %s is not allowed", r.Method), http.StatusMethodNotAllowed)
}
}
I'm a total noob at go and I'm trying to understand what I'm missing here. I'm expecting to use dec.Decode to loop over the json values and end up with a map of the response. What I'm getting is the entire json string as the key to the first element of the map. What am I missing?
Example response:
2015/03/02 10:03:16 map[error:invalid_request error_description:that is not a recognized WePay API call error_code:1001] map[string]interface {}
package main
import (
"encoding/json"
"io"
"log"
"net/http"
"reflect"
)
func main() {
var v map[string]interface{}
resp, err := http.Get("https://wepayapi.com/v2/")
if err != nil {
log.Println("Error: " + err.Error())
}
defer resp.Body.Close()
// resp.Body is an io.ReadCloser... NewDecoder expects an io.Reader
dec := json.NewDecoder(resp.Body)
// Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.
for err := dec.Decode(&v); err != nil && err != io.EOF; {
log.Println("ERROR: " + err.Error())
return
}
log.Println(v, reflect.TypeOf(v))
}
Decoder will decode the whole JSON value at once (in this case the error object), you don;y have to call it in a loop:
if err := dec.Decode(&v); err != nil {
log.Println("ERROR: " + err.Error())
return
}
As a response you get a map equavalent of this JSON:
{"error":"invalid_request","error_description":"that is not a recognized WePay API call","error_code":1001}
Result:
map[string]interface{} {
"error":"invalid_request",
"error_description":"that is not a recognized WePay API call",
"error_code":1001,
}
I tried working with Apiary and made a universal template to send JSON to mock server and have this code:
package main
import (
"encoding/json"
"fmt"
"github.com/jmcvetta/napping"
"log"
"net/http"
)
func main() {
url := "http://restapi3.apiary.io/notes"
fmt.Println("URL:>", url)
s := napping.Session{}
h := &http.Header{}
h.Set("X-Custom-Header", "myvalue")
s.Header = h
var jsonStr = []byte(`
{
"title": "Buy cheese and bread for breakfast."
}`)
var data map[string]json.RawMessage
err := json.Unmarshal(jsonStr, &data)
if err != nil {
fmt.Println(err)
}
resp, err := s.Post(url, &data, nil, nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("response Status:", resp.Status())
fmt.Println("response Headers:", resp.HttpResponse().Header)
fmt.Println("response Body:", resp.RawText())
}
This code doesn't send JSON properly, but I don't know why. The JSON string can be different in every call. I can't use Struct for this.
I'm not familiar with napping, but using Golang's net/http package works fine (playground):
func main() {
url := "http://restapi3.apiary.io/notes"
fmt.Println("URL:>", url)
var jsonStr = []byte(`{"title":"Buy cheese and bread for breakfast."}`)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
req.Header.Set("X-Custom-Header", "myvalue")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("response Status:", resp.Status)
fmt.Println("response Headers:", resp.Header)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println("response Body:", string(body))
}
you can just use post to post your json.
values := map[string]string{"username": username, "password": password}
jsonValue, _ := json.Marshal(values)
resp, err := http.Post(authAuthenticatorUrl, "application/json", bytes.NewBuffer(jsonValue))
If you already have a struct.
import (
"bytes"
"encoding/json"
"io"
"net/http"
"os"
)
// .....
type Student struct {
Name string `json:"name"`
Address string `json:"address"`
}
// .....
body := &Student{
Name: "abc",
Address: "xyz",
}
payloadBuf := new(bytes.Buffer)
json.NewEncoder(payloadBuf).Encode(body)
req, _ := http.NewRequest("POST", url, payloadBuf)
client := &http.Client{}
res, e := client.Do(req)
if e != nil {
return e
}
defer res.Body.Close()
fmt.Println("response Status:", res.Status)
// Print the body to the stdout
io.Copy(os.Stdout, res.Body)
Full gist.
In addition to standard net/http package, you can consider using my GoRequest which wraps around net/http and make your life easier without thinking too much about json or struct. But you can also mix and match both of them in one request! (you can see more details about it in gorequest github page)
So, in the end your code will become like follow:
func main() {
url := "http://restapi3.apiary.io/notes"
fmt.Println("URL:>", url)
request := gorequest.New()
titleList := []string{"title1", "title2", "title3"}
for _, title := range titleList {
resp, body, errs := request.Post(url).
Set("X-Custom-Header", "myvalue").
Send(`{"title":"` + title + `"}`).
End()
if errs != nil {
fmt.Println(errs)
os.Exit(1)
}
fmt.Println("response Status:", resp.Status)
fmt.Println("response Headers:", resp.Header)
fmt.Println("response Body:", body)
}
}
This depends on how you want to achieve. I made this library because I have the same problem with you and I want code that is shorter, easy to use with json, and more maintainable in my codebase and production system.
Example post request for http or https
//Encode the data
postBody, _ := json.Marshal(map[string]string{
"name": "Test",
"email": "Test#Test.com",
})
responseBody := bytes.NewBuffer(postBody)
//Leverage Go's HTTP Post function to make request
resp, err := http.Post("https://postman-echo.com/post", "application/json", responseBody)
//Handle Error
if err != nil {
log.Fatalf("An Error Occured %v", err)
}
defer resp.Body.Close()
//Read the response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
sb := string(body)
log.Printf(sb)
Use io.Pipe for large request bodies as mentioned in another answer. This approach avoids building the entire request body in memory by streaming the data from the JSON encoder to the network.
This answer builds on the other answer by showing how to handle errors. Always handle errors!
Use the pipe's CloseWithError function to propagate encoding errors back to error returned from http.Post.
Handle the error returned from http.Post
Close the response body.
Here's the code:
r, w := io.Pipe()
go func() {
w.CloseWithError(json.NewEncoder(w).Encode(data))
}()
// Ensure that read side of pipe is closed. This
// unblocks goroutine in scenario where http.Post
// errors out before reading the entire request body.
defer r.Close()
resp, err := http.Post(url, r)
if err != nil {
// Adjust error handling here to meet application requrirements.
log.Fatal(err)
}
defer resp.Body.Close()
// Use the response here.
If you have a lot of data to send, you can use a pipe:
package main
import (
"encoding/json"
"io"
"net/http"
)
func main() {
m := map[string]int{"SNG_ID": 75498415}
r, w := io.Pipe()
go func() {
json.NewEncoder(w).Encode(m)
w.Close()
}()
http.Post("https://stackoverflow.com", "application/json", r)
}
https://golang.org/pkg/io#Pipe
if you want to do it like that, you need to use this map for unmarshalling json string.
var data map[string]interface{}
but if you need to change the json each time and to make initialization of your requst body more convenient, you can use this map for creating json body.
var bodyJsonMap map[string]interface{}{
"key1": val1,
"key2": val2,
...
}
Then marshal it to a json-string.