I'm building microservices app with go and beego. I'm trying to pass JSON response from service A to service B as following:
func (u *ServiceController) GetAll() {
req := httplib.Get("http://localhost/api/1/services")
str, err := req.String()
// str = {"id":1, "name":"some service"}
if err != nil {
fmt.Println(err)
}
u.Data["json"] = str
u.ServeJSON()
}
However, when I send the response I actually double json encoding:
"{\"id\":\"1\",\"name\":\"some service\"}"
Finally, this is the solution I came up with:
func (u *ServiceController) GetAll() {
req := httplib.Get("http://localhost/api/1/services")
str, err := req.String()
if err != nil {
fmt.Println(err)
}
strToByte := []byte(str)
u.Ctx.Output.Header("Content-Type", "application/json")
u.Ctx.Output.Body(strToByte)
}
Try this:
func (u *ServiceController) GetAll() {
req := httplib.Get("http://localhost/api/1/services")
str, err := req.Bytes()
// str = {"id":1, "name":"some service"}
if err != nil {
fmt.Println(err)
}
u.Ctx.Output.Header("Content-Type", "text/plain;charset=UTF-8")
u.Ctx.ResponseWriter.Write(str)
}
If you call req.String(), it will encode the " in the json string. I suggest you use []byte to handle data usually.
Related
I wrote some code which hits one public API and saves the JSON output in a file. But the data is storing line by line into the file instead of a single JSON format.
For eg.
Current Output:
{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}
Desired Output:
[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]
I know this should be pretty simple and i am missing out something.
My Current Code is:
To read IP from file and hit the API one by one on each IP.
func readIPfromFile(filename string, outFile string, timeout int) {
data := jsonIn{}
//open input file
jsonFile, err := os.Open(filename) //open input file
...
...
jsonData := bufio.NewScanner(jsonFile)
for jsonData.Scan() {
// marshal json data & check for logs
if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
log.Fatal(err)
}
//save to file
url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
GetGeoIP(url, outFile, timeout)
}
}
To make HTTP Request with custom request header and call write to file function.
func GetGeoIP(url string, outFile string, timeout int) {
geoClient := http.Client{
Timeout: time.Second * time.Duration(timeout), // Timeout after 5 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("accept", "application/json")
res, getErr := geoClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
jsonout := jsonOut{}
jsonErr := json.Unmarshal(body, &jsonout)
if jsonErr != nil {
log.Fatal(jsonErr)
}
file, _ := json.Marshal(jsonout)
write2file(outFile, file)
}
To Write data to file:
func write2file(outFile string, file []byte) {
f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err = f.WriteString(string(file)); err != nil {
log.Fatal(err)
}
if _, err = f.WriteString("\n"); err != nil {
log.Fatal(err)
}
I know, i can edit f.WriteString("\n"); to f.WriteString(","); to add comma but still adding [] in the file is challenging for me.
First, please do not invent a new way of json marshaling, just use golang built-in encoding/json or other library on github.
Second, if you want to create a json string that represents an array of object, you need to create the array of objects in golang and marshal it into string (or more precisely, into array of bytes)
I create a simple as below, but please DIY if possible.
https://go.dev/play/p/RR_ok-fUTb_4
I have a JSON file in S3 that takes the format of the following struct:
type StockInfo []struct {
Ticker string `json:"ticker"`
BoughtPrice string `json:"boughtPrice"`
NumberOfShares string `json:"numberOfShares"`
}
and I want to read it into a struct value from S3. In python the code would look something like this:
import boto3
import json
s3 = boto3.client('s3', 'us-east-1')
obj = s3.get_object(Bucket=os.environ["BucketName"], Key=os.environ["Key"])
fileContents = obj['Body'].read().decode('utf-8')
json_content = json.loads(fileContents)
However I'm kinda stuck on how to make this happen in Go. I've gotten this far:
package main
import (
"archive/tar"
"bytes"
"fmt"
"log"
"os"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"github.com/joho/godotenv"
)
type StockInfo []struct {
Ticker string `json:"ticker"`
BoughtPrice string `json:"boughtPrice"`
NumberOfShares string `json:"numberOfShares"`
}
func init() {
// loads values from .env into the system
if err := godotenv.Load(); err != nil {
log.Print("No .env file found")
}
return
}
func main() {
// Store the PATH environment variable in a variable
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
if err != nil {
panic(err)
}
s3Client := s3.New(sess)
bucket := "ian-test-bucket-go-python"
key := "StockInfo.json"
requestInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
result, err := s3Client.GetObject(requestInput)
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
which returns to me the body/object buffer, but im not sure how to read that into a string so I can marshal it into my struct. I found this code in a similar question:
requestInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
buf := new(aws.WriteAtBuffer)
numBytes, _ := *s3manager.Downloader.Download(buf, requestInput)
tr := tar.NewReader(bytes.NewReader(buf.Bytes()))
but I get the following errors:
not enough arguments in call to method expression s3manager.Downloader.Download
have (*aws.WriteAtBuffer, *s3.GetObjectInput)
want (s3manager.Downloader, io.WriterAt, *s3.GetObjectInput, ...func(*s3manager.Downloader))
multiple-value s3manager.Downloader.Download() in single-value context
Can anyone point me in the right direction? kinda frustrating how hard it seems to do this compared to python.
I was able to do it with the following code:
requestInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
result, err := s3Client.GetObject(requestInput)
if err != nil {
fmt.Println(err)
}
defer result.Body.Close()
body1, err := ioutil.ReadAll(result.Body)
if err != nil {
fmt.Println(err)
}
bodyString1 := fmt.Sprintf("%s", body1)
var s3data StockInfo
decoder := json.NewDecoder(strings.NewReader(bodyString1))
err = decoder.Decode(&s3data)
if err != nil {
fmt.Println("twas an error")
}
fmt.Println(s3data)
Alternative solution using json.Unmarshal
besed on aws-sdk-go-v2
...
params := &s3.GetObjectInput{
Bucket: aws.String(s3Record.S3.Bucket.Name),
Key: aws.String(s3Record.S3.Object.Key),
}
result, _ := client.GetObject(context.TODO(), params)
if err != nil {
panic(err)
}
defer result.Body.Close()
// capture all bytes from upload
b, err := ioutil.ReadAll(result.Body)
if err != nil {
panic(err)
}
var temp StockInfo
if err = json.Unmarshal(b, &temp); err != nil {
panic(err)
}
ftm.Println("res: ",b)
I inherited someone else's code for an API and since I'm not familiar with the requests that it's receiving I'm trying to print them or log them so I can see their structure. From what I've read about Go, jsons are decoded with Structs but since I don't know how the requests are received I cant write a struct.
I've tried the following on a basic API but they just print me out an empty map or nothing at all:
func createBook(w http.ResponseWriter, r *http.Request) {
var result map[string]interface{}
_ = json.NewDecoder(r.Body).Decode(&result)
fmt.Println(result)
func createBook(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var book Book
_ = json.NewDecoder(r.Body).Decode(&book)
buf := new(bytes.Buffer)
buf.ReadFrom(r.Body)
newStr := buf.String()
reader := strings.NewReader(newStr)
writter := os.Stdout
dec := json.NewDecoder(reader)
enc := json.NewEncoder(writter)
for {
var m map[string]interface{}
if err := dec.Decode(&m); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
if err := enc.Encode(&m); err != nil {
log.Println(err)
}
fmt.Println(m)
}
book.ID = strconv.Itoa(rand.Intn(100000000)) // Mock ID - not safe
books = append(books, book)
json.NewEncoder(w).Encode(book)
}
Is there any other way that it would print the received json without me knowing the strut beforehand?
Use json.Unmarshal function
import "bytes"
func createBook(w http.ResponseWriter, r *http.Request) {
var result map[string]interface{}
data :+ StreamToByte(r.Body)
err := json.Unmarshal(data, &result)
if err !=nil{
fmt.Println(err) //better to use log
}else
fmt.Println(result)
}
}
func StreamToByte(stream io.Reader) []byte {
buf := new(bytes.Buffer)
buf.ReadFrom(stream)
return buf.Bytes()
}
Refer :
https://appdividend.com/2020/02/28/golang-how-to-convert-json-to-map-in-go/
https://gist.github.com/dixudx/3989284b142414e10352fde9def5c771
I'm trying to implement a go program which can handle http requests and send response in nested JSON. When i run my code and call the URL, I'm getting a runtime error,what does it mean? how can i handle this?
panic serving 192.168.0.101:50760: interface conversion: interface {} is int64, not []uint8
goroutine 5 [running]
This is my function code which is called on hitting the url
func logInPass(res http.ResponseWriter, req *http.Request) {
type Resp struct {
Result []map[string]interface{} `json:"Result,omitempty"`
Status string `json:"Status"`
}
type AxleUser struct {
Mobile string `json:"Mobile"`
Password string `json:"Password"`
}
var Response Resp
Response.Status = "failed"
Result := make(map[string]interface{})
db, err := sql.Open("mysql", "root:chikkIbuddI57#tcp(127.0.0.1:3306)/b2b")
if err != nil {
panic(err.Error())
}
defer db.Close()
rnd := render.New()
b, err := ioutil.ReadAll(req.Body)
defer req.Body.Close()
if err != nil {
panic(err.Error())
}
// Unmarshal the request body
var msg AxleUser
err = json.Unmarshal(b, &msg)
if err != nil {
panic(err.Error())
}
// get shop id from emp table using mobile number and password
userrows, usererr := db.Query("SELECT b2b_emp_id,b2b_shop_id,b2b_shop_name,b2b_emp_name,b2b_emp_mobile_number FROM b2b_employee_tbl WHERE b2b_emp_mobile_number=? and b2b_password=?", msg.Mobile, msg.Password)
if usererr != nil {
panic(usererr.Error())
}
usercolumns, usererr := userrows.Columns()
if usererr != nil {
panic(usererr.Error())
}
usercount := len(usercolumns)
values := make([]interface{}, usercount)
scanArgs := make([]interface{}, usercount)
for i := range values {
scanArgs[i] = &values[i]
}
for userrows.Next() {
usererr := userrows.Scan(scanArgs...)
if usererr != nil {
panic(usererr.Error())
}
for i, v := range values {
if v != nil {
Result[usercolumns[i]] = fmt.Sprintf("%s", string(v.([]byte)))
}
}
Response.Result = append(Response.Result, Result)
Response.Status = "success"
}
res.Header().Set("Content-Type", "application/json")
rnd.JSON(res, http.StatusOK, Response)
}
Thanks in Advance!
I've changed this line
values := make([]interface{}, usercount)
To
values := make([]string, usercount)
And this line
Result[usercolumns[i]] = fmt.Sprintf("%s", string(v.([]byte)))
To
Result[usercolumns[i]] = v
I build a client and a server in golang both are using this functions to encrypt/decrypt
func encrypt(text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
b := base64.StdEncoding.EncodeToString(text)
ciphertext := make([]byte, aes.BlockSize+len(b))
iv := ciphertext[:aes.BlockSize]
if _, err := io.ReadFull(rand.Reader, iv); err != nil {
return nil, err
}
cfb := cipher.NewCFBEncrypter(block, iv)
cfb.XORKeyStream(ciphertext[aes.BlockSize:], []byte(b))
return ciphertext, nil
}
func decrypt(text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(text) < aes.BlockSize {
return nil, errors.New("ciphertext too short")
}
iv := text[:aes.BlockSize]
text = text[aes.BlockSize:]
cfb := cipher.NewCFBDecrypter(block, iv)
cfb.XORKeyStream(text, text)
data, err := base64.StdEncoding.DecodeString(string(text))
if err != nil {
return nil, err
}
return data, nil
}
so yeah I make a normal post request
url := "https://"+configuration.Server+configuration.Port+"/get"
// TODO maybe bugs rest here
ciphertext, err := encrypt([]byte(*getUrl))
if err != nil {
fmt.Println("Error: " + err.Error())
}
fmt.Println(string(ciphertext))
values := map[string]interface{}{"url": *getUrl, "urlCrypted": ciphertext}
jsonValue, _ := json.Marshal(values)
jsonStr := bytes.NewBuffer(jsonValue)
req, err := http.NewRequest("POST", url, jsonStr)
and the servercode is as following
requestContent := getRequestContentFromRequest(req)
url := requestContent["url"].(string)
undecryptedUrl := requestContent["urlCrypted"].(string)
decryptedurl, err := decrypt([]byte(undecryptedUrl))
if err != nil {
fmt.Println("Error: " + err.Error())
}
fmt.Println(decryptedurl)
where getRequestContentFromRequest is as following
func getRequestContentFromRequest(req *http.Request)
map[string]interface{} {
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
data := buf.Bytes()
var requestContent map[string]interface{}
err := json.Unmarshal(data, &requestContent)
if err != nil {
fmt.Println(err)
}
return requestContent
}
Now to the problem.
If I encrypt my string in the client and decrypt it direct after that everything is fine.
But, when I send the encrypted string to the server and try to decrypt it with literrally the same function as in the client, the decrypt function throws an error.
Error: illegal base64 data at input byte 0
I think the Problem is the unmarshalling of the JSON.
Thanks for help.
P.S.
Repos are
github.com/BelphegorPrime/goSafeClient and github.com/BelphegorPrime/goSafe
UPDATE
Example JSON
{"url":"facebook2.com","urlCrypted":"/}\ufffd\ufffd\ufffdgP\ufffdN뼞\ufffd\u0016\ufffd)\ufffd\ufffd\ufffdy\u001c\u000f\ufffd\ufffd\ufffdep\ufffd\rY\ufffd\ufffd$\ufffd\ufffd"}
UPDATE2
I made a playground here
The problem is that you encode in base64 twice. The first time in the encrypt function and the second time during the JSON marshalling. byte slices are automatically converted into base64 strings by the encoding/json marshaller.
The solution is to decode the base64 string before calling decrypt.
Example on the Go PlayGround
EDIT
Working solution here