Print/Log full unstructured json from stream in Go - json

I inherited someone else's code for an API and since I'm not familiar with the requests that it's receiving I'm trying to print them or log them so I can see their structure. From what I've read about Go, jsons are decoded with Structs but since I don't know how the requests are received I cant write a struct.
I've tried the following on a basic API but they just print me out an empty map or nothing at all:
func createBook(w http.ResponseWriter, r *http.Request) {
var result map[string]interface{}
_ = json.NewDecoder(r.Body).Decode(&result)
fmt.Println(result)
func createBook(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var book Book
_ = json.NewDecoder(r.Body).Decode(&book)
buf := new(bytes.Buffer)
buf.ReadFrom(r.Body)
newStr := buf.String()
reader := strings.NewReader(newStr)
writter := os.Stdout
dec := json.NewDecoder(reader)
enc := json.NewEncoder(writter)
for {
var m map[string]interface{}
if err := dec.Decode(&m); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
if err := enc.Encode(&m); err != nil {
log.Println(err)
}
fmt.Println(m)
}
book.ID = strconv.Itoa(rand.Intn(100000000)) // Mock ID - not safe
books = append(books, book)
json.NewEncoder(w).Encode(book)
}
Is there any other way that it would print the received json without me knowing the strut beforehand?

Use json.Unmarshal function
import "bytes"
func createBook(w http.ResponseWriter, r *http.Request) {
var result map[string]interface{}
data :+ StreamToByte(r.Body)
err := json.Unmarshal(data, &result)
if err !=nil{
fmt.Println(err) //better to use log
}else
fmt.Println(result)
}
}
func StreamToByte(stream io.Reader) []byte {
buf := new(bytes.Buffer)
buf.ReadFrom(stream)
return buf.Bytes()
}
Refer :
https://appdividend.com/2020/02/28/golang-how-to-convert-json-to-map-in-go/
https://gist.github.com/dixudx/3989284b142414e10352fde9def5c771

Related

How to set up web server to perform POST Request in Go?

I want to set up a web server to perform a POST request. How does the post request get executed with the code below since only HandleFunc and ListenAndServe are defined in main function?
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"strings"
)
func post(w http.ResponseWriter, r *http.Request) {
const myurl string = "http://localhost:8000/"
request := strings.NewReader(`
{
"Name":"Tom",
"Age":"20"
}
`)
response, err := http.Post(myurl, "application/json", request)
content, err := ioutil.ReadAll(response.Body)
if err != nil {
panic(err)
}
fmt.Println(string(content))
defer response.Body.Close()
}
func main() {
http.HandleFunc("/", post)
log.Fatal(http.ListenAndServe(":8000", nil))
}
Here is a basic example of how you could go about it. I am using the same program to run both, the server and the client. This is just for demonstration purposes. You can of course make them separate programs.
// use struct to represent the data
// to recieve and send
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
// run the example
func main() {
// start the server in a goroutine
go server()
// wait 1 second to give the server time to start
time.Sleep(time.Second)
// make a post request
if err := client(); err != nil {
fmt.Println(err)
}
}
// basic web server to receive a request and
// decode the body into a user struct
func server() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
user := &Person{}
err := json.NewDecoder(r.Body).Decode(user)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
fmt.Println("got user:", user)
w.WriteHeader(http.StatusCreated)
})
if err := http.ListenAndServe(":8080", nil); err != http.ErrServerClosed {
panic(err)
}
}
// a simple client that posts a user to the server
func client() error {
user := &Person{
Name: "John",
Age: 30,
}
b := new(bytes.Buffer)
err := json.NewEncoder(b).Encode(user)
if err != nil {
return err
}
resp, err := http.Post("http://localhost:8080/", "application/json", b)
if err != nil {
return err
}
defer resp.Body.Close()
fmt.Println(resp.Status)
return nil
}
Here is the working example: https://go.dev/play/p/34GT04jy_uA

Golang Read JSON from S3 into struct in memory

I have a JSON file in S3 that takes the format of the following struct:
type StockInfo []struct {
Ticker string `json:"ticker"`
BoughtPrice string `json:"boughtPrice"`
NumberOfShares string `json:"numberOfShares"`
}
and I want to read it into a struct value from S3. In python the code would look something like this:
import boto3
import json
s3 = boto3.client('s3', 'us-east-1')
obj = s3.get_object(Bucket=os.environ["BucketName"], Key=os.environ["Key"])
fileContents = obj['Body'].read().decode('utf-8')
json_content = json.loads(fileContents)
However I'm kinda stuck on how to make this happen in Go. I've gotten this far:
package main
import (
"archive/tar"
"bytes"
"fmt"
"log"
"os"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"github.com/joho/godotenv"
)
type StockInfo []struct {
Ticker string `json:"ticker"`
BoughtPrice string `json:"boughtPrice"`
NumberOfShares string `json:"numberOfShares"`
}
func init() {
// loads values from .env into the system
if err := godotenv.Load(); err != nil {
log.Print("No .env file found")
}
return
}
func main() {
// Store the PATH environment variable in a variable
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
if err != nil {
panic(err)
}
s3Client := s3.New(sess)
bucket := "ian-test-bucket-go-python"
key := "StockInfo.json"
requestInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
result, err := s3Client.GetObject(requestInput)
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
which returns to me the body/object buffer, but im not sure how to read that into a string so I can marshal it into my struct. I found this code in a similar question:
requestInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
buf := new(aws.WriteAtBuffer)
numBytes, _ := *s3manager.Downloader.Download(buf, requestInput)
tr := tar.NewReader(bytes.NewReader(buf.Bytes()))
but I get the following errors:
not enough arguments in call to method expression s3manager.Downloader.Download
have (*aws.WriteAtBuffer, *s3.GetObjectInput)
want (s3manager.Downloader, io.WriterAt, *s3.GetObjectInput, ...func(*s3manager.Downloader))
multiple-value s3manager.Downloader.Download() in single-value context
Can anyone point me in the right direction? kinda frustrating how hard it seems to do this compared to python.
I was able to do it with the following code:
requestInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
result, err := s3Client.GetObject(requestInput)
if err != nil {
fmt.Println(err)
}
defer result.Body.Close()
body1, err := ioutil.ReadAll(result.Body)
if err != nil {
fmt.Println(err)
}
bodyString1 := fmt.Sprintf("%s", body1)
var s3data StockInfo
decoder := json.NewDecoder(strings.NewReader(bodyString1))
err = decoder.Decode(&s3data)
if err != nil {
fmt.Println("twas an error")
}
fmt.Println(s3data)
Alternative solution using json.Unmarshal
besed on aws-sdk-go-v2
...
params := &s3.GetObjectInput{
Bucket: aws.String(s3Record.S3.Bucket.Name),
Key: aws.String(s3Record.S3.Object.Key),
}
result, _ := client.GetObject(context.TODO(), params)
if err != nil {
panic(err)
}
defer result.Body.Close()
// capture all bytes from upload
b, err := ioutil.ReadAll(result.Body)
if err != nil {
panic(err)
}
var temp StockInfo
if err = json.Unmarshal(b, &temp); err != nil {
panic(err)
}
ftm.Println("res: ",b)

Golang API Gorilla MySQL query

So... I'm finally doing my side project containing the super nerdy tabletop game Warhammer in which I've created a database MySQL and my next step is to create a API.
I've got three tables at this moment... "tyranids", "greyknghts" and "deathguard". I want to make a dynamic query to select targeted table. I'm able to do this but as the tables grown I need to make this dynamic.
func getTyranids(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-type", "application/json")
var units []Unit
result, err := db.Query("SELECT * FROM tyranids")
if err != nil {
panic(err.Error)
}
defer result.Close()
for result.Next() {
var unit Unit
err := result.Scan(&unit.ID, &unit.Name, &unit.Type, &unit.Movement, &unit.WeaponsSkill, &unit.BallisticSkill, &unit.Strength, &unit.Toughness, &unit.Wounds, &unit.Attacks, &unit.Leadership, &unit.Initiate, &unit.Points)
if err != nil {
panic(err.Error)
}
units = append(units, unit)
}
json.NewEncoder(w).Encode(units)
}
How can I write this so I won't need a function for each table?
I've made it work with mux.Vars for each individual unit.
func getTyranidUnit(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
params := mux.Vars(r)
result, err := db.Query("SELECT * FROM tyranids WHERE name = ?", params["name"])
if err != nil {
panic(err.Error())
}
defer result.Close()
var unit Unit
for result.Next() {
err := result.Scan(&unit.ID, &unit.Name, &unit.Type, &unit.Movement, &unit.WeaponsSkill, &unit.BallisticSkill, &unit.Strength, &unit.Toughness, &unit.Wounds, &unit.Attacks, &unit.Leadership, &unit.Initiate, &unit.Points)
if err != nil {
panic(err.Error())
}
}
json.NewEncoder(w).Encode(unit)
}
func main() {
db, err = sql.Open("mysql", "xx:xx#tcp(xxx)/Warhammer")
if err != nil {
panic(err.Error())
}
defer db.Close()
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/tyranids", getTyranids).Methods("GET")
router.HandleFunc("/tyranids/{name}", getTyranidUnit).Methods("GET")
http.ListenAndServe(":8001", router)
}
Thank you.
You could just make a map to pair the tables to keys:
speciesUnitMap:= map[string]string{
"Norn-Queen": "tyranids",
"Hive Tyrant": "tyranids",
"Rippers ": "tyranids",
"Hive Ship": "tyranids",
‎ "Ork Boyz": "orcs",
"‎Waaagh!": "orcs",
‎ "Warboss": "orcs",
"‎Blood Axes" "orcs",
}
// This is dirty I would construct the string before
// and do verifications that the received param is conform
result, err := db.Query("SELECT * FROM " +
speciesUnitMap[params["name"]] + " WHERE name = ?", params["name"])

Chi empty http.Request.Body in render.Bind

I am using github.com/pressly/chi to build this simple program where I try to decode some JSON from the http.Request.Body:
package main
import (
"encoding/json"
"fmt"
"net/http"
"github.com/pressly/chi"
"github.com/pressly/chi/render"
)
type Test struct {
Name string `json:"name"`
}
func (p *Test) Bind(r *http.Request) error {
err := json.NewDecoder(r.Body).Decode(p)
if err != nil {
return err
}
return nil
}
func main() {
r := chi.NewRouter()
r.Post("/products", func(w http.ResponseWriter, r *http.Request) {
var p Test
// err := render.Bind(r, &p)
err := json.NewDecoder(r.Body).Decode(&p)
if err != nil {
panic(err)
}
fmt.Println(p)
})
http.ListenAndServe(":8080", r)
}
When I don't use render.Bind() (from "github.com/pressly/chi/render"), it works as expected.
However, when I uncomment the line err := render.Bind(r, &p) and I comment the line err := json.NewDecoder(r.Body).Decode(&p), it panics with EOF :
2017/06/20 22:26:39 http: panic serving 127.0.0.1:39696: EOF
and thus the json.Decode() fails.
Am I doing something wrong or is the http.Request.Body is already read somewhere else before render.Bind() is called?
render.Bind's purpose is to perform decode and execute Bind(r) to do post decode operations.
For eg.:
type Test struct {
Name string `json:"name"`
}
func (p *Test) Bind(r *http.Request) error {
// At this point, Decode is already done by `chi`
p.Name = p.Name + " after decode"
return nil
}
If you have to do only JSON decode no other actions needs to be done after decode with respect to decoded values. Just use:
// Use Directly JSON decoder of std pkg
err := json.NewDecoder(r.Body).Decode(&p)
OR
// Use wrapper method from chi DecodeJSON
err := render.DecodeJSON(r.Body, &p)

Golang Encode/Decode base64 with json post doesn't work

I build a client and a server in golang both are using this functions to encrypt/decrypt
func encrypt(text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
b := base64.StdEncoding.EncodeToString(text)
ciphertext := make([]byte, aes.BlockSize+len(b))
iv := ciphertext[:aes.BlockSize]
if _, err := io.ReadFull(rand.Reader, iv); err != nil {
return nil, err
}
cfb := cipher.NewCFBEncrypter(block, iv)
cfb.XORKeyStream(ciphertext[aes.BlockSize:], []byte(b))
return ciphertext, nil
}
func decrypt(text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(text) < aes.BlockSize {
return nil, errors.New("ciphertext too short")
}
iv := text[:aes.BlockSize]
text = text[aes.BlockSize:]
cfb := cipher.NewCFBDecrypter(block, iv)
cfb.XORKeyStream(text, text)
data, err := base64.StdEncoding.DecodeString(string(text))
if err != nil {
return nil, err
}
return data, nil
}
so yeah I make a normal post request
url := "https://"+configuration.Server+configuration.Port+"/get"
// TODO maybe bugs rest here
ciphertext, err := encrypt([]byte(*getUrl))
if err != nil {
fmt.Println("Error: " + err.Error())
}
fmt.Println(string(ciphertext))
values := map[string]interface{}{"url": *getUrl, "urlCrypted": ciphertext}
jsonValue, _ := json.Marshal(values)
jsonStr := bytes.NewBuffer(jsonValue)
req, err := http.NewRequest("POST", url, jsonStr)
and the servercode is as following
requestContent := getRequestContentFromRequest(req)
url := requestContent["url"].(string)
undecryptedUrl := requestContent["urlCrypted"].(string)
decryptedurl, err := decrypt([]byte(undecryptedUrl))
if err != nil {
fmt.Println("Error: " + err.Error())
}
fmt.Println(decryptedurl)
where getRequestContentFromRequest is as following
func getRequestContentFromRequest(req *http.Request)
map[string]interface{} {
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
data := buf.Bytes()
var requestContent map[string]interface{}
err := json.Unmarshal(data, &requestContent)
if err != nil {
fmt.Println(err)
}
return requestContent
}
Now to the problem.
If I encrypt my string in the client and decrypt it direct after that everything is fine.
But, when I send the encrypted string to the server and try to decrypt it with literrally the same function as in the client, the decrypt function throws an error.
Error: illegal base64 data at input byte 0
I think the Problem is the unmarshalling of the JSON.
Thanks for help.
P.S.
Repos are
github.com/BelphegorPrime/goSafeClient and github.com/BelphegorPrime/goSafe
UPDATE
Example JSON
{"url":"facebook2.com","urlCrypted":"/}\ufffd\ufffd\ufffdgP\ufffdN뼞\ufffd\u0016\ufffd)\ufffd\ufffd\ufffdy\u001c\u000f\ufffd\ufffd\ufffdep\ufffd\rY\ufffd\ufffd$\ufffd\ufffd"}
UPDATE2
I made a playground here
The problem is that you encode in base64 twice. The first time in the encrypt function and the second time during the JSON marshalling. byte slices are automatically converted into base64 strings by the encoding/json marshaller.
The solution is to decode the base64 string before calling decrypt.
Example on the Go PlayGround
EDIT
Working solution here