Correctly remove second json.Marshal in Go - mysql

I have, for whatever reason, while trying to build a simple Rest API in Go with MySQL storage, added a second json.Marshal which is double-encoding and producing results with escaped quotes and such. I could strip the quotes, but I think I shouldn't have two json.Marshal things happening in the first place.
The problem is twofold - 1) which is proper to remove (leaning toward the first because "result" should be the larger array) and 2)how to keep the code functioning after removal? I can't just simply remove the first one as I start encountering all sorts of errors. Here are the relevant portions of the code:
type Volume struct {
Id int
Name string
Description string
}
... skipping ahead ....
var result = make([]string,1000)
switch request.Method {
case "GET":
name := request.URL.Query().Get("name")
stmt, err := db.Prepare("select id, name, description from idm_assets.VOLUMES where name = ?")
if err != nil{
fmt.Print( err );
}
rows, err := stmt.Query(name)
if err != nil {
fmt.Print( err )
}
i := 0
for rows.Next() {
var name string
var id int
var description string
err = rows.Scan( &id, &name, &description )
if err != nil {
fmt.Println("Error scanning: " + err.Error())
return
}
volume := &Volume{Id: id,Name:name,Description: description}
Here is the first json.Marshal ...
b, err := json.Marshal(volume)
fmt.Println(b)
if err != nil {
fmt.Println(err)
return
}
result[i] = fmt.Sprintf("%s", string(b))
i++
}
result = result[:i]
...skipping other cases for PUT, DELETE, Etc. To the second json.Marshal ...
default:
}
json, err := json.Marshal(result)
if err != nil {
fmt.Println(err)
return
}
fmt.Fprintf(response,"'%v'\n",string(json) )

Turn result into an array of *Volume
result := []*Volume{}
and then append new Volume records:
result = append(result, &Volume{Id: id,Name:name,Description: description})
and in the end use Marshal(result) to get the JSON result.

Related

Removed fieldname id and reformat JSON Output in golang

I am developing a rest api using golang based on /v1/public/characters of Marvel API. I need to return all the character ids in the format of
[ 1011234, 1012345, 1009213, 1010788, 1087123, 1222345, ... ]
The way I achieve this is via strings.NewReplacer (in last 2 lines below).
...
type CharacterId struct {
Data struct {
Results []struct {
Id int `json:"id"`
} `json:"results"`
} `json:"data"`
}
func getCharacters(w http.ResponseWriter, _ *http.Request) {
ts := strconv.FormatInt(time.Now().Unix(), 10)
hash := getMd5(ts + conf.privateKey + conf.publicKey)
response, err := http.Get("https://gateway.marvel.com/v1/public/characters?ts=" + ts + "&apikey=" + conf.publicKey + "&hash=" + hash + "&limit=100")
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
responseBytes, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
var characterId CharacterId
err = json.Unmarshal(responseBytes, &characterId)
if err != nil {
log.Fatal(err)
return
}
data, err := json.Marshal(characterId.Data.Results)
replacer := strings.NewReplacer("\"id\":", "", "{", "", "}", "", ",", ", ", "[", "[ ")
fmt.Fprint(w, replacer.Replace(string(data)))
}
Without using strings.NewReplacer like the code snippet above, i.e. fmt.Fprint(w, string(data)) directly gives me the output below which is not what I want.
[{"id":1011334},{"id":1017100},{"id":1009144},{"id":1010699},{"id":1009146},{"id":1016823},{"id":1009148},{"id":1009149}, ... ]
I somehow feel my way of using strings.NewReplacer is not a clean/proper way. Can anyone suggest a better way of making the output I want?
Create int array with your IDs and marshal it and write it to your response.
Replace your last tree lines with following code and test.
var data []int
for _, result := range characterId.Data.Results {
data = append(data, result.Id)
}
c , err := json.Marshal(data)
if err != nil {
//handle error here
}
fmt.Fprint(w, string(c))

how to parse multiple query params and pass them to a sql command

i am playing around rest API's in go and when i do a get call with this
http://localhost:8000/photos?albumId=1&id=1
i want to return only those values from db which corresponds to alubmId=1 and id =1 , or any other key in the query for that matter without storing as variables and then passing it to query, and when i dont give any query params i want it to return all the posts in db
func getPhotos(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var photos []Photo
db, err = sql.Open("mysql", "root:$pwd#tcp(127.0.0.1:3306)/typicode")
if err != nil {
panic(err.Error())
}
defer db.Close()
for k, v := range r.URL.Query() {
fmt.Printf("%s: %s\n", k, v)
}
result, err := db.Query("SELECT id, albumId, Title,thumbnailUrl,url from photo")
if err != nil {
panic(err.Error())
}
defer result.Close()
for result.Next() {
var photo Photo
err := result.Scan(&photo.Id, &photo.AlbumId, &photo.Title, &photo.Url, &photo.Thumbnailurl)
if err != nil {
panic(err.Error())
}
photos = append(photos, photo)
}
json.NewEncoder(w).Encode(photos)
}
First you need a set of valid table columns that can be used in the query, this is required to avoid unnecessary errors from columns with typos and sql injections from malicious input.
var photocolumns = map[string]struct{}{
"id": {},
"albumId": {},
// all the other columns you accept as input
}
Now depending on the database you may, or you may not, need to parse the query values and convert them to the correct type for the corresponding column. You can utilize the column map to associate each column with the correct converter type/func.
// a wrapper around strconv.Atoi that has the signature of the map-value type below
func atoi(s string) (interface{}, error) {
return strconv.Atoi(s)
}
var photocolumns = map[string]func(string) (interface{}, error) {
"id": atoi,
"albumId": atoi,
// all the other columns you accept as input
}
Then, all you need is a single loop and in it you do all the work you need. You get the correct column name, convert the value to the correct type, aggregate that converted value into a slice so that it can be passed to the db, and also construct the WHERE clause so that it can be concatenated to the sql query string.
where := ""
params := []interface{}{}
for k, v := range r.URL.Query() {
if convert, ok := photocolumns[k]; ok {
param, err := convert(v[0])
if err != nil {
fmt.Println(err)
return
}
params = append(params, param)
where += k + " = ? AND "
}
}
if len(where) > 0 {
// prefix the string with WHERE and remove the last " AND "
where = " WHERE " + where[:len(where)-len(" AND ")]
}
rows, err := db.Query("SELECT id, albumId, Title,thumbnailUrl,url from photo" + where, params...)
// ...

merge two map[string]interface{} from json

I have two json inputs built this way
"count: 1 result: fields"
I would like to concatenate the fields that I find within result without using a defined structure. I have tried in many ways but most of the time the result is an error about the type Interface {} or the last map overwritten the data
I would like both the "result" and the first and second map fields to be merged within the result in output.
oracle, err := http.Get("http://XXX:8080/XXXX/"+id)
if err != nil {
panic(err)
}
defer oracle.Body.Close()
mysql, err := http.Get("http://XXX:3000/XXX/"+id)
if err != nil {
panic(err)
}
defer mysql.Body.Close()
oracleJSON, err := ioutil.ReadAll(oracle.Body)
if err != nil {
panic(err)
}
mysqlJSON, err := ioutil.ReadAll(mysql.Body)
if err != nil {
panic(err)
}
var oracleOUT map[string]interface{}
var mysqlOUT map[string]interface{}
json.Unmarshal(oracleJSON, &oracleOUT)
json.Unmarshal(mysqlJSON, &mysqlOUT)
a := oracleOUT["result"]
b := mysqlOUT["result"]
c.JSON(http.StatusOK, gin.H{"result": ????})
this is an example of json
{"count":1,"result":{"COD_DIPENDENTE":"00060636","MATRICOLA":"60636","COGNOME":"PIPPO"}}
If i have two json like this the result of the function it should be
`"result":{"COD_DIPENDENTE":"00060636","MATRICOLA":"60636","COGNOME":"PIPPO","COD_DIPENDENTE":"00060636","MATRICOLA":"60636","COGNOME":"PIPPO"}}`
The output you are looking for is not valid JSON. However with a small change you can output something very similar to your example that is valid JSON.
You probably do want to use a defined structure for the portion of the input that has a known structure, so that you can extract the more abstract "result" section more easily.
If you start at the top of the input structure using a map[string]interface{} then you'll have to do a type assertion on the "result" key. For example:
var input map[string]interface{}
err = json.Unmarshal(data, &input)
if err != nil {
return err
}
keys, ok := input["result"].(map[string]interface{})
if !ok {
return errors.New("wasn't the type we expected")
}
However if you used a defined structure for the top level you can do it like the following which feels much cleaner.
type Input struct {
Count int `json:"count"`
Result map[string]interface{} `json:"result"`
}
var input Input
err = json.Unmarshal(data, &input)
if err != nil {
return err
}
// from here you can use input.Result directly without a type assertion
To generate output that has duplicate keys, you could use an array of objects with a single key/value pair in each, then you end up with a valid JSON structure that does not overwrite keys. Here's how to do that (playground link):
package main
import (
"encoding/json"
"fmt"
)
type Input struct {
Count int `json:"count"`
Result map[string]interface{} `json:"result"`
}
type Output struct {
Count int `json:"count"`
Result []map[string]interface{} `json:"result"`
}
var inputdata = [][]byte{
[]byte(`{"count":1,"result":{"COD_DIPENDENTE":"00060636", "MATRICOLA":"60636", "COGNOME":"PIPPO"}}`),
[]byte(`{"count":1,"result":{"COD_DIPENDENTE":"00060636", "MATRICOLA":"60636", "COGNOME":"PIPPO"}}`),
}
func main() {
inputs := make([]Input, len(inputdata))
for i := range inputs {
err := json.Unmarshal(inputdata[i], &inputs[i])
if err != nil {
panic(err)
}
}
var out Output
out.Count = len(inputs)
for _, input := range inputs {
for k, v := range input.Result {
out.Result = append(out.Result, map[string]interface{}{k: v})
}
}
outdata, _ := json.Marshal(out)
fmt.Println(string(outdata))
}
Which produces output that looks like this when formatted:
{
"count": 2,
"result": [
{"MATRICOLA": "60636"},
{"COGNOME": "PIPPO"},
{"COD_DIPENDENTE": "00060636"},
{"COGNOME": "PIPPO"},
{"COD_DIPENDENTE": "00060636"},
{"MATRICOLA": "60636"}
]
}

Golang - retrieve multiple results from MySQL, then display them as JSON

Recently, I'm learning about Go (Golang). I'm trying to make a simple web service using Martini and jwt-go. I didn't find any difficulty in retrieving a single row data and put in JSON as the response. But, when dealing with multiple-rows, it's a whole different story. Basically, I refer to the accepted answer here.
Here is the piece of my code:
m.Get("/users", func(params martini.Params, r render.Render) {
db, err := sql.Open("mysql", "root:#/sirat_v2")
if err != nil {
panic(err.Error())
}
defer db.Close()
rows, err := db.Query("SELECT user_id, nama FROM `users` WHERE password = ?", "asdfasdf")
defer rows.Close()
cols, err := rows.Columns()
if err != nil {
panic(err.Error())
}
partages := make([]*Partage, 0, 10)
var user_id int
var nama string
for rows.Next() {
err = rows.Scan(&user_id, &nama)
if err != nil { /* error handling */
}
partages = append(partages, &Partage{user_id, nama})
}
})
When trying to build, there's an error said that Partage is undefined.
The error showing up because you use struct Partage to create an object, but you haven't declared it.
type Partage struct {
user_id string
nama string
}
But how do I display the result as JSON response? I've tried r.JSON(200, partages) but the results aren't displayed
In martini you can use r.JSON() to print rows as JSON
m.Get("/users", func(params martini.Params, r render.Render) {
// ...
r.JSON(200, map[string]interface{}{
data: rows
})
})

Go SQL driver get interface{} column values

I am trying to use go sql driver to read from database tables and I am converting the values to []map[string]interface{}. The column name is the key of the map and the values are of interface{}. I am adding all the columns into an array. I am using the code sample for "RawBytes" at https://github.com/go-sql-driver/mysql/wiki/Examples as an example to start with.
However, in the example -all the column values are converted to string as follows,
// Fetch rows
for rows.Next() {
// get RawBytes from data
err = rows.Scan(scanArgs...)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
// Now do something with the data.
// Here we just print each column as a string.
var value string
for i, col := range values {
// Here we can check if the value is nil (NULL value)
if col == nil {
value = "NULL"
} else {
value = string(col) //ATTN : converted to string here
}
fmt.Println(columns[i], ": ", value)
}
fmt.Println("-----------------------------------")
}
Is there a way to retain it as interface{} so I can do the necessary type casting while using the columns from []map[string]interface{}
See this https://stackoverflow.com/questions/20271123/go-lang-sql-in-parameters answer which my answer is based on. Using that you can do something like this:
var myMap = make(map[string]interface{})
rows, err := db.Query("SELECT * FROM myTable")
defer rows.Close()
if err != nil {
log.Fatal(err)
}
colNames, err := rows.Columns()
if err != nil {
log.Fatal(err)
}
cols := make([]interface{}, len(colNames))
colPtrs := make([]interface{}, len(colNames))
for i := 0; i < len(colNames); i++ {
colPtrs[i] = &cols[i]
}
for rows.Next() {
err = rows.Scan(colPtrs...)
if err != nil {
log.Fatal(err)
}
for i, col := range cols {
myMap[colNames[i]] = col
}
// Do something with the map
for key, val := range myMap {
fmt.Println("Key:", key, "Value Type:", reflect.TypeOf(val))
}
}
Using the reflect package you can then get the Type for each column as needed as demonstrated with the loop at the end.
This is generic and will work with any table, number of columns etc.
AFter a long struggle i found out the solution. Check belowfunction that converts sql.RawBytes to Int64. This can be easily altered to fit any data type
func GetInt64ColumnValue(payload sql.RawBytes) (int64, error) {
content := reflect.ValueOf(payload).Interface().(sql.RawBytes) // convert to bytes
data := string(content) //convert to string
i, err := strconv.ParseInt(data,10,64) // convert to int or your preferred data type
if err != nil {
log.Printf("got error converting %s to int error %s ",data,err.Error())
return 0, err
}
return i, nil
}