Go SQL driver get interface{} column values - mysql

I am trying to use go sql driver to read from database tables and I am converting the values to []map[string]interface{}. The column name is the key of the map and the values are of interface{}. I am adding all the columns into an array. I am using the code sample for "RawBytes" at https://github.com/go-sql-driver/mysql/wiki/Examples as an example to start with.
However, in the example -all the column values are converted to string as follows,
// Fetch rows
for rows.Next() {
// get RawBytes from data
err = rows.Scan(scanArgs...)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
// Now do something with the data.
// Here we just print each column as a string.
var value string
for i, col := range values {
// Here we can check if the value is nil (NULL value)
if col == nil {
value = "NULL"
} else {
value = string(col) //ATTN : converted to string here
}
fmt.Println(columns[i], ": ", value)
}
fmt.Println("-----------------------------------")
}
Is there a way to retain it as interface{} so I can do the necessary type casting while using the columns from []map[string]interface{}

See this https://stackoverflow.com/questions/20271123/go-lang-sql-in-parameters answer which my answer is based on. Using that you can do something like this:
var myMap = make(map[string]interface{})
rows, err := db.Query("SELECT * FROM myTable")
defer rows.Close()
if err != nil {
log.Fatal(err)
}
colNames, err := rows.Columns()
if err != nil {
log.Fatal(err)
}
cols := make([]interface{}, len(colNames))
colPtrs := make([]interface{}, len(colNames))
for i := 0; i < len(colNames); i++ {
colPtrs[i] = &cols[i]
}
for rows.Next() {
err = rows.Scan(colPtrs...)
if err != nil {
log.Fatal(err)
}
for i, col := range cols {
myMap[colNames[i]] = col
}
// Do something with the map
for key, val := range myMap {
fmt.Println("Key:", key, "Value Type:", reflect.TypeOf(val))
}
}
Using the reflect package you can then get the Type for each column as needed as demonstrated with the loop at the end.
This is generic and will work with any table, number of columns etc.

AFter a long struggle i found out the solution. Check belowfunction that converts sql.RawBytes to Int64. This can be easily altered to fit any data type
func GetInt64ColumnValue(payload sql.RawBytes) (int64, error) {
content := reflect.ValueOf(payload).Interface().(sql.RawBytes) // convert to bytes
data := string(content) //convert to string
i, err := strconv.ParseInt(data,10,64) // convert to int or your preferred data type
if err != nil {
log.Printf("got error converting %s to int error %s ",data,err.Error())
return 0, err
}
return i, nil
}

Related

Golang: how to proper handle dynamic select query

what's the best way to handle the result of a select sql query in go?
Context(thats what i got so far...):
#Update
func (s *SQLServiceServer) select_query_func() {
// db_connection and validation
rows, err1 := db.Query(request.GetQuery())
if err1 != nil {
panic(err1)
}
defer rows.Close()
columns, err2 := rows.Columns()
if err2 != nil {
panic(err2)
}
// Loop through rows, using Scan to assign column data to struct fields.
for rows.Next() {
values := make([]interface{}, len(columns))
for i := range values {
values[i] = new(interface{})
}
if err2 = rows.Scan(values...); err2 != nil {
panic(err2)
}
for i := range values {
fmt.Println(values[i])
}
}
return requestOutput(REQUEST_OK, RESULT_OK, ERROR_NULL)
}
To sumarize, what's the best approach to do it?
The error that i mentioned
Scan requires allocated, i.e. non-nil, pointers as arguments. The following types are allowed:
*string
*[]byte
*int, *int8, *int16, *int32, *int64
*uint, *uint8, *uint16, *uint32, *uint64
*bool
*float32, *float64
*interface{}
*RawBytes
*Rows (cursor value) any type implementing Scanner (see Scanner docs)
So to fix your code you need to populate the values slice with non-nil pointers, and for your purpose those pointers can be of type *interface{}.
for rows.Next() {
values := make([]interface{}, len(columns))
for i := range values {
values[i] = new(interface{})
}
if err := rows.Scan(values...); err != nil {
return err
}
}

how to parse multiple query params and pass them to a sql command

i am playing around rest API's in go and when i do a get call with this
http://localhost:8000/photos?albumId=1&id=1
i want to return only those values from db which corresponds to alubmId=1 and id =1 , or any other key in the query for that matter without storing as variables and then passing it to query, and when i dont give any query params i want it to return all the posts in db
func getPhotos(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var photos []Photo
db, err = sql.Open("mysql", "root:$pwd#tcp(127.0.0.1:3306)/typicode")
if err != nil {
panic(err.Error())
}
defer db.Close()
for k, v := range r.URL.Query() {
fmt.Printf("%s: %s\n", k, v)
}
result, err := db.Query("SELECT id, albumId, Title,thumbnailUrl,url from photo")
if err != nil {
panic(err.Error())
}
defer result.Close()
for result.Next() {
var photo Photo
err := result.Scan(&photo.Id, &photo.AlbumId, &photo.Title, &photo.Url, &photo.Thumbnailurl)
if err != nil {
panic(err.Error())
}
photos = append(photos, photo)
}
json.NewEncoder(w).Encode(photos)
}
First you need a set of valid table columns that can be used in the query, this is required to avoid unnecessary errors from columns with typos and sql injections from malicious input.
var photocolumns = map[string]struct{}{
"id": {},
"albumId": {},
// all the other columns you accept as input
}
Now depending on the database you may, or you may not, need to parse the query values and convert them to the correct type for the corresponding column. You can utilize the column map to associate each column with the correct converter type/func.
// a wrapper around strconv.Atoi that has the signature of the map-value type below
func atoi(s string) (interface{}, error) {
return strconv.Atoi(s)
}
var photocolumns = map[string]func(string) (interface{}, error) {
"id": atoi,
"albumId": atoi,
// all the other columns you accept as input
}
Then, all you need is a single loop and in it you do all the work you need. You get the correct column name, convert the value to the correct type, aggregate that converted value into a slice so that it can be passed to the db, and also construct the WHERE clause so that it can be concatenated to the sql query string.
where := ""
params := []interface{}{}
for k, v := range r.URL.Query() {
if convert, ok := photocolumns[k]; ok {
param, err := convert(v[0])
if err != nil {
fmt.Println(err)
return
}
params = append(params, param)
where += k + " = ? AND "
}
}
if len(where) > 0 {
// prefix the string with WHERE and remove the last " AND "
where = " WHERE " + where[:len(where)-len(" AND ")]
}
rows, err := db.Query("SELECT id, albumId, Title,thumbnailUrl,url from photo" + where, params...)
// ...

merge two map[string]interface{} from json

I have two json inputs built this way
"count: 1 result: fields"
I would like to concatenate the fields that I find within result without using a defined structure. I have tried in many ways but most of the time the result is an error about the type Interface {} or the last map overwritten the data
I would like both the "result" and the first and second map fields to be merged within the result in output.
oracle, err := http.Get("http://XXX:8080/XXXX/"+id)
if err != nil {
panic(err)
}
defer oracle.Body.Close()
mysql, err := http.Get("http://XXX:3000/XXX/"+id)
if err != nil {
panic(err)
}
defer mysql.Body.Close()
oracleJSON, err := ioutil.ReadAll(oracle.Body)
if err != nil {
panic(err)
}
mysqlJSON, err := ioutil.ReadAll(mysql.Body)
if err != nil {
panic(err)
}
var oracleOUT map[string]interface{}
var mysqlOUT map[string]interface{}
json.Unmarshal(oracleJSON, &oracleOUT)
json.Unmarshal(mysqlJSON, &mysqlOUT)
a := oracleOUT["result"]
b := mysqlOUT["result"]
c.JSON(http.StatusOK, gin.H{"result": ????})
this is an example of json
{"count":1,"result":{"COD_DIPENDENTE":"00060636","MATRICOLA":"60636","COGNOME":"PIPPO"}}
If i have two json like this the result of the function it should be
`"result":{"COD_DIPENDENTE":"00060636","MATRICOLA":"60636","COGNOME":"PIPPO","COD_DIPENDENTE":"00060636","MATRICOLA":"60636","COGNOME":"PIPPO"}}`
The output you are looking for is not valid JSON. However with a small change you can output something very similar to your example that is valid JSON.
You probably do want to use a defined structure for the portion of the input that has a known structure, so that you can extract the more abstract "result" section more easily.
If you start at the top of the input structure using a map[string]interface{} then you'll have to do a type assertion on the "result" key. For example:
var input map[string]interface{}
err = json.Unmarshal(data, &input)
if err != nil {
return err
}
keys, ok := input["result"].(map[string]interface{})
if !ok {
return errors.New("wasn't the type we expected")
}
However if you used a defined structure for the top level you can do it like the following which feels much cleaner.
type Input struct {
Count int `json:"count"`
Result map[string]interface{} `json:"result"`
}
var input Input
err = json.Unmarshal(data, &input)
if err != nil {
return err
}
// from here you can use input.Result directly without a type assertion
To generate output that has duplicate keys, you could use an array of objects with a single key/value pair in each, then you end up with a valid JSON structure that does not overwrite keys. Here's how to do that (playground link):
package main
import (
"encoding/json"
"fmt"
)
type Input struct {
Count int `json:"count"`
Result map[string]interface{} `json:"result"`
}
type Output struct {
Count int `json:"count"`
Result []map[string]interface{} `json:"result"`
}
var inputdata = [][]byte{
[]byte(`{"count":1,"result":{"COD_DIPENDENTE":"00060636", "MATRICOLA":"60636", "COGNOME":"PIPPO"}}`),
[]byte(`{"count":1,"result":{"COD_DIPENDENTE":"00060636", "MATRICOLA":"60636", "COGNOME":"PIPPO"}}`),
}
func main() {
inputs := make([]Input, len(inputdata))
for i := range inputs {
err := json.Unmarshal(inputdata[i], &inputs[i])
if err != nil {
panic(err)
}
}
var out Output
out.Count = len(inputs)
for _, input := range inputs {
for k, v := range input.Result {
out.Result = append(out.Result, map[string]interface{}{k: v})
}
}
outdata, _ := json.Marshal(out)
fmt.Println(string(outdata))
}
Which produces output that looks like this when formatted:
{
"count": 2,
"result": [
{"MATRICOLA": "60636"},
{"COGNOME": "PIPPO"},
{"COD_DIPENDENTE": "00060636"},
{"COGNOME": "PIPPO"},
{"COD_DIPENDENTE": "00060636"},
{"MATRICOLA": "60636"}
]
}

How to extract a spanner row into Json or Parquet format in golang?

I'm new to golang and spanner, I want to save a snapshot of our spanner DB to Google cloud storage in every 5 mins. The format that I want to use is Parquet or JSON.
stmt = spanner.NewStatement("SELECT * FROM " + tableName + " WHERE UpdatedAt >= #startDateTime AND UpdatedAt <= #endDateTime")
iter := txn.Query(ctx, stmt)
defer iter.Stop()
for {
row, err := iter.Next()
if err == iterator.Done {
break
}
if err != nil {
log.Println("Failed to read data, err = %s", err)
}
}
I have got all the rows, but I don't have a clue on how to extract all the column values and write it to a Parquet or JSON file or upload it to GCS. Is it possible to extract all the column values without knowing the type of the value or column name? Any help would be appreciated.
The column type is required to retrieve the value. See the "Supported types and their corresponding Cloud Spanner column type(s)" in the Row documentation. You can get the column names from the Row.ColumnNames. It might make sense to use Row.ToStruct with a struct corresponding to the table, and write that to json, for example using the "encoding/json" package's Marshal.
I would like to share my tedious solution and hope it will help someone in the future. In my case, I got a task to save a snapshot of our spanner DB in a short time interval and save those data in parquet format to GCS. So that later we can use the big query to query those data.
Firstly, I got the spanner rows that I wanted with a simple statement like this:
stmt := spanner.NewStatement(fmt.Sprintf("SELECT * FROM %s WHERE UpdatedAt >= #startDateTime AND UpdatedAt <= #endDateTime", tableName))
stmt.Params["startDateTime"] = time.Unix(1520330400, 0)
stmt.Params["endDateTime"] = time.Unix(1520376600, 0)
iter := txn.Query(ctx, stmt)
values := readRows(iter)
func readRows(iter *spanner.RowIterator) []spanner.Row {
var rows []spanner.Row
defer iter.Stop()
for {
row, err := iter.Next()
if err == iterator.Done {
break
}
if err != nil {
log.Println("Failed to read data, err = %s", err)
}
rows = append(rows, *row)
}
return rows
}
Yea, that was easy. But it was encouraging, cause this is my first time coding in golang. However, took me a while to found out that it's not possible to decode the value without knowing the type of each column, but all I need is each column's string value and save it in parquet format.
So I wrote another query to get the spanner type for each column like this:
typeStmt = spanner.NewStatement("SELECT t.column_name, t.spanner_type FROM information_schema.columns AS t WHERE t.table_name = #tableName")
typeStmt.Params["tableName"] = tableName
iterTypes := txn.Query(ctx, typeStmt)
types := readRows(iterTypes)
// use a map to keep all the types
dataTypes := make(map[string]string)
for i := 0; i < len(types); i++ {
var columnName string
var dataType string
types[i].Column(0, &columnName)
types[i].Column(1, &dataType)
dataTypes[columnName] = dataType
}
formattedRows, md := extractDataByType(dataTypes, values)
and I convert the spanner type to go type with a switch:
func decodeValueByType(index int, row spanner.Row, value interface{}) {
err := row.Column(index, value)
if err != nil {
log.Println("Failed to extract value, err = %s", err)
}
}
func prepareParquetWriter(md *[]string, parquetType string, columnNames []string, index int) {
if len(*md) < len(columnNames) {
*md = append(*md, fmt.Sprintf("name=%s, type=%s", columnNames[index], parquetType))
}
}
func extractDataByType(types map[string]string, rows []spanner.Row) ([][]string, []string) {
var formattedRows [][]string
var md []string
for _, row := range rows {
columnNames := row.ColumnNames()
var vals []string
for i := 0; i < row.Size(); i++ {
switch types[columnNames[i]] {
case "STRING(MAX)":
var value spanner.NullString
decodeValueByType(i, row, &value)
prepareParquetWriter(&md, "UTF8", columnNames, i)
vals = append(vals, fmt.Sprintf("%v", value))
case "TIMESTAMP":
var value spanner.NullTime
decodeValueByType(i, row, &value)
prepareParquetWriter(&md, "TIMESTAMP_MILLIS", columnNames, i)
vals = append(vals, fmt.Sprintf("%v", value))
case "INT64":
var value spanner.NullInt64
decodeValueByType(i, row, &value)
prepareParquetWriter(&md, "INT64", columnNames, i)
vals = append(vals, fmt.Sprintf("%v", value))
case "BOOL":
var value spanner.NullBool
decodeValueByType(i, row, &value)
prepareParquetWriter(&md, "BOOLEAN", columnNames, i)
vals = append(vals, fmt.Sprintf("%v", value))
}
}
formattedRows = append(formattedRows, vals)
}
log.Println("parquet format: %s", md)
return formattedRows, md
}
Finally, I got my data in a 2 dimensional array and generated my parquet configuration in an array.
I haven't finished the parquet writer for GCS, but I used xitongsys/parquet-go to wrote the file locally like this:
fw, err := ParquetFile.NewLocalFileWriter(fmt.Sprintf("dataInParquet/%s_%s.parquet", name, time.Now().Format("20060102150405")))
if err != nil {
log.Println("Can't open file", err)
return
}
pw, err := ParquetWriter.NewCSVWriter(md, fw, 4)
if err != nil {
log.Println("Can't create csv writer", err)
return
}
for _, row := range formattedRows {
rec := make([]*string, len(row))
for i := 0; i < len(row); i++ {
rec[i] = &row[i]
}
if err = pw.WriteString(rec); err != nil {
log.Println("WriteString error", err)
}
}
if err = pw.WriteStop(); err != nil {
log.Println("WriteStop error", err)
}
log.Println("Write Finished")
fw.Close()
Please let me know if anyone know a better way of doing this. Thanks. ;-)
Btw, this is just my experiment code, If you want to use any of this code please adjust accordingly. My production implementation needs to support more features like querying multiple databases with goroutine, support both spanner and MySQL, save data in either parquet or JSON format. Would like to hear more ideas if anyone is doing something samilar.

Correctly remove second json.Marshal in Go

I have, for whatever reason, while trying to build a simple Rest API in Go with MySQL storage, added a second json.Marshal which is double-encoding and producing results with escaped quotes and such. I could strip the quotes, but I think I shouldn't have two json.Marshal things happening in the first place.
The problem is twofold - 1) which is proper to remove (leaning toward the first because "result" should be the larger array) and 2)how to keep the code functioning after removal? I can't just simply remove the first one as I start encountering all sorts of errors. Here are the relevant portions of the code:
type Volume struct {
Id int
Name string
Description string
}
... skipping ahead ....
var result = make([]string,1000)
switch request.Method {
case "GET":
name := request.URL.Query().Get("name")
stmt, err := db.Prepare("select id, name, description from idm_assets.VOLUMES where name = ?")
if err != nil{
fmt.Print( err );
}
rows, err := stmt.Query(name)
if err != nil {
fmt.Print( err )
}
i := 0
for rows.Next() {
var name string
var id int
var description string
err = rows.Scan( &id, &name, &description )
if err != nil {
fmt.Println("Error scanning: " + err.Error())
return
}
volume := &Volume{Id: id,Name:name,Description: description}
Here is the first json.Marshal ...
b, err := json.Marshal(volume)
fmt.Println(b)
if err != nil {
fmt.Println(err)
return
}
result[i] = fmt.Sprintf("%s", string(b))
i++
}
result = result[:i]
...skipping other cases for PUT, DELETE, Etc. To the second json.Marshal ...
default:
}
json, err := json.Marshal(result)
if err != nil {
fmt.Println(err)
return
}
fmt.Fprintf(response,"'%v'\n",string(json) )
Turn result into an array of *Volume
result := []*Volume{}
and then append new Volume records:
result = append(result, &Volume{Id: id,Name:name,Description: description})
and in the end use Marshal(result) to get the JSON result.