Posting json data to remote server with http.Post - json

I am stumped on what seems like a very simple problem.
I am receiving a json object from a client. It looks like this:
{
"user": "test#example.com"
}
I need to simply pass this on to another part of the api as a POST request. This is what i've got until now:
//Decode incomming json
decoder := json.NewDecoder(r.Body)
var user UserInformation
err := decoder.Decode(&user)
if err != nil {
log.Println(err)
}
jsonUser, _ := json.Marshal(user)
log.Println(string(jsonUser[:])) //Correct output
buffer := bytes.NewBuffer(jsonUser)
log.Println(string(buffer.Bytes()[:])) //Also correct
resp, err := http.Post("http://example.com/api/has_publisher", "application/json", buffer)
if err != nil {
log.Println(err)
}
As I cannot test this program on a live system, I verified the resulting post request with wireshark only to find that the content is missing along with Content-Length being 0. For some reason, http.Post doesn't read from the buffer.
Am i missing something here? I would greatly appreciate if someone could point me in the right direction.
Thanks!

Shouldn`t be the root cause but replace
buffer := bytes.NewBuffer(jsonUser)
with
buffer := bytes.NewReader(jsonUser)
It is more likely that your test setup is the root cause. I assume you are pointing to a non-existing endpoint. This would result in a failure (TCP SYN fails) before the actual HTTP POST is send.
Check if you can use mockable.io as an alternative to mock your backend.

Related

Stream data from API

I am trying to pull data on mails coming into an API from an email testing tool mailhog.
If I use a call to get a list of emails e.g
GET /api/v1/messages
I can load this data into a struct with no issues and print out values I need.
However if I use a different endpoint that is essentially a stream of new emails coming in, I have different behavior. When I run my go application I get no output whatsoever.
Do I need to do like a while loop to constantly listen to the endpoint to get the output?
My end goal is to pull some information from emails as they come in and then pass them into a different function.
Here is me trying to access the streaming endpoint
https://github.com/mailhog/MailHog/blob/master/docs/APIv1.md
res, err := http.Get("http://localhost:8084/api/v1/events")
if err != nil {
panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
var data Email
json.Unmarshal(body, &data)
fmt.Printf("Email: %v\n", data)
If I do a curl request at the mailhog service with the same endpoint, I do get output as mails come in. However I cant seem to figure out why I am getting no output via my go app. The app does stay running just I dont get any output.
I am new to Go so apologies if this is a really simple question
From ioutil.ReadAll documentation:
ReadAll reads from r until an error or EOF and returns the data it read.
When you use to read the body of a regular endpoint, it works because the payload has an EOF: the server uses the header Content-Length to tell how many bytes the body response has, and once the client read that many bytes, it understands that it has read all of the body and can stop.
Your "streaming" endpoint doesn't use Content-Length though, because the body has an unknown size, it's supposed to write events as they come, so you can't use ReadAll in this case. Usually, in this case, you are supposed to read line-by-line, where each line represents an event. bufio.Scanner does exactly that:
res, err := http.Get("http://localhost:8084/api/v1/events")
if err != nil {
panic(err.Error())
}
scanner := bufio.NewScanner(res.Body)
for e.scanner.Scan() {
if err := e.scanner.Err(); err != nil {
panic(err.Error())
}
event := e.scanner.Bytes()
var data Email
json.Unmarshal(event, &data)
fmt.Printf("Email: %v\n", data)
}
curl can process the response as you expect because it checks that the endpoint will stream data, so it reacts accordinly. It may be helpful to add the response curl gets to the question.

Approach to send a large JSON payload to a web service

Consider a small Go application that reads a large JSON file 2GB+, marshals the JSON data into a struct, and POSTs the JSON data to a web service endpoint.
The web service receiving the payload changed its functionality, and now has a limit of 25MB per payload. What would be the best approach to overcome this issue using Go? I've thought of the following, however I'm not sure it is the best approach:
Creating a function to split the large JSON file into multiple smaller ones (up to 20MB), and then iterate over the files sending multiple smaller requests.
Similar function to the one being used to currently send the entire JSON payload:
func sendDataToService(data StructData) {
payload, err := json.Marshal(data)
if err != nil {
log.Println("ERROR:", err)
}
request, err := http.NewRequest("POST", endpoint, bytes.NewBuffer(payload))
if err != nil {
log.Println("ERROR:", err)
}
client := &http.Client{}
response, err := client.Do(request)
log.Println("INFORMATIONAL:", request)
if err != nil {
log.Println("ERROR:", err)
}
defer response.Body.Close()
}
You can break the input into chunks and send each piece individually:
dec := json.NewDecoder(inputStream)
tok, err := dec.Token()
if err != nil {
return err
}
if tok == json.Delim('[') {
for {
var obj json.RawMessage
if err := dec.Decode(&obj); err != nil {
return err
}
// Here, obj contains one element of the array. You can send this
// to the server.
if !dec.More() {
break
}
}
}
As the server-side can process data progressively, I assume that the large JSON object can be split into smaller pieces. From this point, I can propose several options.
Use HTTP requests
Pros: Pretty simple to implement on the client-side.
Cons: Making hundreds of HTTP requests might be slow. You will also need to handle timeouts - this is additional complexity.
Use WebSocket messages
If the receiving side supports WebSockets, a step-by-step flow will look like this:
Split the input data into smaller pieces.
Connect to the WebSocket server.
Start sending messages with the smaller pieces till the end of the file.
Close connection to the server.
This solution might be more performant as you won't need to connect and disconnect from the server each time you send a message, as you'd do with HTTP.
However, both solutions suppose that you need to assemble all pieces on the server-side. For example, you would probably need to send along with the data a correlation ID to let the server know what file you are sending right now and a specific end-of-file message to let the server know when the file ends. In the case of the WebSocket server, you could assume that the entire file is sent during a single connection session if it is relevant.

Golang http request POST works once

I have a master and slave. Master have api call result, which takes JSON. And i have proplem with slave, which send this result on master, the first time my code sends json well, but second time, code stop(program wait.....) on resp, err := client.Do(req), when create query on master.
salve code:
func main (){
for {
// some code, very long code
sendResult(resFiles)
}
}
func sendResult(rf common.ResultFiles) {
jsonValue, err := json.Marshal(rf)
req, err := http.NewRequest(methodPost, ResultAdress,
bytes.NewBuffer(jsonValue))
req.Header.Set("Content-Type", ContentType)
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("response Status:", resp.Status)
}
master api call:
func result(c echo.Context) error {
rf := &ResultFiles{}
err := c.Bind(rf)
if err != nil {
log.Fatal(err)
}
rfChannel <- *rf
return c.JSON(http.StatusOK, nil)
}
My question: Why? May be problem in the standard client golang (http.Client) or timeout? If i set timout - my code crashed by timeout)))anticipated....
Thank you!
You need to add a timeout to your http.Client. By default http.Client has timeout specified as 0 which means no timeout at all. So if server is not responding than your application will just hang waiting for a response. This problem is fully described in this article Don’t use Go’s default HTTP client (in production). Though you create custom client you still need to specify timeout.
Problem link with channel, i send result work slave on master to channel, but channel work without loop, i add loop for read data from channel and all work.

Unable to successfully POST valid JSON data to a remote url from App Engine using Golang

UPDATE: Please see Alexey's comments below for the solution
I am trying what I thought would be a trivial function to take some valid Json data and post it to a remote url
I've tried every example I could find close to this on StackOverflow, and the receiving side always has an empty payload.
I'm ruling out the receiving side, due to being able to do this:
curl -XPOST 'http://supersecreturl/mypost' -d '[{"i sware to ritchie":"this json is 100 percent valid"},{"i can even":"copy and paste it into a curl POST request and receive it flawlessly on the remote side"}]'
Please help, I'm loosing my mind here..
/// Here is approximately my code - had to remove the valid url and the JSON content
func PutArticlesJSON(c appengine.Context, articles []*Articlez) (*http.Response){
url := "http://mysecreturl/mypost"
client := urlfetch.Client(c)
jsonarts, _ := json.Marshal(articles)
c.Debugf(" --- What do we have - %v", string(jsonarts)) /// the appengine log shows exactly valid json at this point, such as:
/*
[{"i sware to ritchie":"this json is 100 percent valid"},{"i can even":"copy and paste it into a curl POST request and receive it flawless on the remote side"}]
*/
// tried this way too....
//req, err := http.NewRequest("POST", url, strings.NewReader(string(jsonarts)))
//
req, err := http.NewRequest("POST", url, bytes.NewBuffer(string(jsonStr))) /// on the receiving side, the payload is completely empty no matter what I try
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
return resp
}
#
#!/usr/bin/env python
#
from flask import Flask
from flask import request
import urllib
import json
app = Flask(__name__)
#app.route('/mypost', methods = ['GET','POST'])
def esput():
datapack = request.form
datastream = request.stream
with open("/tmp/log", "a") as myf:
myf.write(str(datastream))
myf.write(str(datapack))
myf.write("\n")
return "all good"
if __name__ == '__main__':
app.run(threaded=True,host='0.0.0.0',port='333',debug=False)
There are two problems I can see there.
Although you think you're sending a valid Json, you aren't.
NewBuffer should receive []byte, not string
Try it like this:
s := `[{"i sware to ritchie":"this json is 100 percent valid"},{"i can even":"copy and paste it into a curl POST request and receive it flawless on the remote side"}]`
req, err := http.NewRequest("POST", url, bytes.NewBuffer([]byte(fmt.Sprintf(`{"data":%s}`, s))))

Go Lang Help - Accessing Array/Slice of interfaces

I'm trying to decode dynamic/random JSON responses in GO, with nested data
body, _ := ioutil.ReadAll(response.Body)
resp := make(map[string]interface{})
err = json.Unmarshal(body, &resp)
fmt.Printf("BODY: %T<\n", body)
fmt.Printf("BODY: %s<\n", body)
fmt.Printf("RESP: %s<\n", resp)
fmt.Printf("RESP: %T<\n", resp)
fmt.Printf("RESP[results]: %T<\n", resp["results"])
fmt.Printf("RESP[results]: %s<\n", resp["results"])
body is the JSON result from the HTTP server and I unmarshall it and the result looks to be a slice of bytes.
BODY: []uint8
BODY: {"results":[{"code":500.0,"errors":["Configuration file 'c2-web-2.conf' already exists."],"status":"Object could not be created."}]}
So I unmarshall it into resp and that works as expected.
RESP: map[string]interface {}
RESP: map[results:[map[code:%!s(float64=500) errors:[Configuration file 'c2-web-2.conf' already exists.] status:Object could not be created.]]]<
I'm able to access the map with the key results.
RESP[results]: []interface {}
RESP[results]: [map[code:%!s(float64=500) errors:[Configuration file 'conf.d/hosts/c2-web-2.conf' already exists.] status:Object could not be created.]]<
Now what i want to access it the "code", "errors" and "status" which is in resp["results"] This looks like an array or slice and I've tried indexing it but I get the error at compile time
./create_host.go:62: invalid operation: resp["results"][0] (type interface {} does not support indexing)
I've done a lot of googling, tried unmarshalling the data within resp["results"] etc, but after a few days I have not made much progress.
How should I access the map which seems to be a member of an array? The data structure is not guaranteed so I can't create a structure and unmarshall into that.
Thanks
A co-worker provided the code fragement below which made it possible to access the map entries I was looking for.
respBody, _ := ioutil.ReadAll(response.Body)
var rsp interface{}
if err := json.Unmarshal(respBody, &rsp); err != nil {
log.Fatal(err)
}
resultMap := rsp.(map[string]interface{})["results"].([]interface{})[0].(map[string]interface{})
fmt.Printf("test: %s<\n", resultMap["errors"] )
test: [Configuration file 'c2-web-2.conf' already exists.]<
I believe you need to do a type assertion. You have an interface{}, but you need some sort of slice to index into. Try resp["results"].([]interface{})[0]? (Sorry, haven't had a chance to test this myself.)