json NewDecoder Decode not parsing integers - json

I'm trying to make an API endpoint which would save stuff into a database however, the integers which I pass in via POST request don't seem to be parsing correctly
Here's my structs:
type OnlineTestForm struct {
Form OnlineTestSet `json:"form"`
}
type OnlineTestSet struct {
ID int `db:"id" json:"id"`
OnlineTestSubjectId int `db:"online_test_subject_id" json:"online_test_subject_id"`
Name string `db:"name" json:"name"`
ScoreToPass int `db:"score_to_pass" json:"score_to_pass"`
TimeLimit int `db:"time_limit" json:"time_limit"`
Description string `db:"description" json:"description"`
Enabled bool `db:"enabled" json:"enabled"`
Online bool `db:"online" json:"online"`
TestType string `db:"test_type" json:"test_type"`
DocName string `db:"doc_name" json:"doc_name"`
}
And the "problematic" function where I parse the JSON into a struct:
func NewOnlineTest(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
var jsonForm OnlineTestForm
json.NewDecoder(r.Body).Decode(&jsonForm)
err := addNewOnlineTest(jsonForm.Form)
if err != nil {
fmt.Println(err)
w.WriteHeader(500)
fmt.Fprint(w, "{\"error\": {\"message\": \"An error occured while adding new test - "+err.Error()+"\"}}")
return
}
w.WriteHeader(200)
}
I'm using httprouter for as my router and the route to which I post is defined like this
router.POST("/onlinetest/new", NewOnlineTest)
And finally the test POST request which I send has the following payload:
{
"form":{
"name":"test",
"test_type":"document",
"score_to_pass":32,
"time_limit":324,
"enabled":true,
"online":true,
"online_test_subject_id":1
}
}
The problem occurs when I try to use jsonForm.Form' but the integers I've passed in liketime_limitandscore_to_pass` are 0
{0 0 test 0 0 true true document []}

Found it!
Needed to add string at the end of my json struc declaration, not sure why but it worked
type OnlineTestSet struct {
ID int `db:"id" json:"id"`
OnlineTestSubjectId int `db:"online_test_subject_id" json:"online_test_subject_id,string"`
Name string `db:"name" json:"name"`
ScoreToPass int `db:"score_to_pass" json:"score_to_pass,string"`
TimeLimit int `db:"time_limit" json:"time_limit,string"`
Description string `db:"description" json:"description"`
Enabled bool `db:"enabled" json:"enabled"`
Online bool `db:"online" json:"online"`
TestType string `db:"test_type" json:"test_type"`
DocName string `db:"doc_name" json:"doc_name"`
}

Related

error: json: unsupported type: func() time.Time in Golang

i am new in golang, just try some API in Echo Framework and got some error.
My Models :
package models
import (
"net/http"
"quotes/db"
)
type Quote struct {
Id int `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
}
func GetAll() (Response, error) {
var quotes Quote
var res Response
ctx := db.Init()
ctx.Find(&quotes)
res.Status = http.StatusOK
res.Message = "Success"
res.Data = ctx
return res, nil
}
My Schema table
package schema
type Quotes struct {
Id int `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
}
My Response type for Api
package models
type Response struct {
Status int `json:"status"`
Message string `json:"message"`
Data interface{} `json:"data"`
}
i tried to add this in Models and Schema :
CreatedAt time.Time `gorm:"type:timestamp" json:"created_at,string,omitempty"`
UpdatedAt time.Time `gorm:"type:timestamp" json:"updated_at,string,omitempty"`
DeletedAt time.Time `gorm:"type:timestamp" json:"deleted_at,string,omitempty"`
And Still Not Works, any solutions?
I expect the api work with no errors
When using gorm, you need to embed a gorm.Model struct, which includes fields ID, CreatedAt, UpdatedAt, DeletedAt.
Reference
// gorm.Model definition
type Model struct {
ID uint `gorm:"primaryKey"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt gorm.DeletedAt `gorm:"index"`
}
Not familiar with echo but read below to understand how you use gorm.
In your case you can try doing the following:
package schema
type Quote struct {
gorm.Model
Title string `json:"title"`
Description string `json:"description"`
}
Then to get all the quotes:
func GetAll() (Response, error) {
var quotes []schema.Quote // slice
ctx := db.Init()
// Assuming
// ctx, err := gorm.Open(....)
// https://gorm.io/docs/query.html
result := db.Find(&quotes)
if result.Error != nil {
return Response{
Status: http.StatusInternalServerError,
Message: "Query failed",
},result.Error
}
if result.RowsAffected == 0 {
return Response{
Status: http.StatusNotFound,
Message: "No records found",
},nil
}
return Response{
Status: http.StatusOK,
Message: "Success",
Data: quotes,
},nil
}
Keep in mind that the Data field has type interface{}, which means it can hold a value of any type. If the value wasn't a slice you would be using the & operator you take the address of the Quote value. A slice is already a pointer to underlying slice so need to use the & operator.
If you want to access the slice of Quote values from the Data field, you will need to use a type assertion to convert the value from the interface{} type to the []Quote type. Here's an example of how you could do this:
// Assume that response.Data holds a slice of Quote values
quotes, ok := response.Data.([]Quote)
if !ok {
// Handle the case where response.Data is not a slice of Quote
}
Warning: Since you are returning a slice, then any changes to the returned slice will be modifying the initial slice too. If you wanted to avoid this then copy the slice values to a new slice:
quotesCopy = make([]schema.Quote, len(quotes))
copy(quotesCopy, quotes)

Server does't get all the JSON from remote REST API

My Go server does not retrieve all the JSON data from a remote API
I even tried creating a custom http.Client to make the request...but still it won't retrieve ALL the JSON data,and even tried extending the respective timeouts
Here's my code:
var netTransport = &http.Transport{
Dial: (&net.Dialer{
Timeout: 50 * time.Second,
}).Dial,
TLSHandshakeTimeout: 50 * time.Second,
}
var netClient = &http.Client{
Timeout: time.Second * 40,
Transport: netTransport,
}
res, getErr := netClient.Get(url)
if getErr != nil {
log.Fatal(getErr)
}
data := JSONData{}
if err := json.NewDecoder(res.Body).Decode(&data); err != nil {
log.Fatal(err)
}
data.Username = user
fmt.Println(data)
JSONData is defined as follows:
type Owner struct {
Login string
}
// Item is the single repository data structure
type Item struct {
ID int
Name string
FullName string `json:"full_name"`
Owner Owner
Description string
CreatedAt string `json:"created_at"`
}
// JSONData contains the GitHub API response
type JSONData struct {
Count int `json:"total_count"`
Username string
Items []Item
}
The JSONData needs to define JSON tags.
type Owner struct {
Login string `json:"login"`
}
// Item is the single repository data structure
type Item struct {
ID int `json:"id"`
Name string `json:"name"`
FullName string `json:"full_name"`
Owner Owner `json:"owner"`
Description string `json:"description"`
CreatedAt string `json:"created_at"`
}
// JSONData contains the GitHub API response
type JSONData struct {
Count int `json:"total_count"`
Username string `json:"username"`
Items []Item `json:"items"`
}
If no tag is specified, the name is taken as-is:
type Owner struct {
Login string
}
that would be marshaled (and correspondingly unmarshalled) into similar:
{"Login": "some login"}

How to navigate a nested json with unknown structure in Go?

I am trying to parse and get selected data from a deep nested json data in Go Lang. I'm having issues navigating through the structure and accessing the data. The data is too deep and complex to be parsed with a-priori known structures in Go.
Here is the URL of the file:
-https://www.data.gouv.fr/api/1/datasets/?format=csv&page=0&page_size=20
I did some parsing with map interfaces and using a json string:
resultdata := map[string]interface {}
json.Unmarshal([]byte(inputbytestring), &resultdata) //Inputstring is the string containing the JSON data of the above URL
The problem:
How can turn resultdata into a map (of strings), so I can use methods available for maps?
The JSON data is nested and has several levels. How is it possible to access the lower level JSON fields? is it possible to unmarshal the data recursively?
Once you have data as a map[string]interface{}, you can use type assertions to get to the lower levels of data.
There's a good explanation here of how to do this at https://blog.golang.org/json-and-go
Here's an example to get you most of the way:
https://play.golang.org/p/P8cGP1mTDmD
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
jsonData := `{
"string": "string_value",
"number": 123.45,
"js_array": ["a", "b", "c"],
"integer": 678,
"subtype": {
"number_array": [1, 2, 3]
}
}`
m := map[string]interface{}{}
err := json.Unmarshal([]byte(jsonData), &m)
if err != nil {
log.Fatal(err)
}
for key, value := range m {
switch v := value.(type) {
case int:
fmt.Printf("Key: %s, Integer: %d\n", key, v)
case float64:
fmt.Printf("Key: %s, Float: %v\n", key, v)
case string:
fmt.Printf("Key: %s, String: %s\n", key, v)
case map[string]interface{}:
fmt.Printf("Key: %s, Subtype: %+v\n", key, v)
case []interface{}:
//TODO: Read through each item in the interface and work out what type it is.
fmt.Printf("Key: %s, []interface: %v\n", key, v)
default:
fmt.Printf("Key: %s, unhandled type: %+v\n", key, v)
}
}
}
Alternatively https://mholt.github.io/json-to-go/ does a decent job of turning examples of JSON data into Go structs that can be used for marshalling.
Putting the example in, I get something that isn't too bad.
type AutoGenerated struct {
Data []struct {
Acronym interface{} `json:"acronym"`
Badges []interface{} `json:"badges"`
CreatedAt string `json:"created_at"`
Deleted interface{} `json:"deleted"`
Description string `json:"description"`
Extras struct {
} `json:"extras"`
Frequency string `json:"frequency"`
FrequencyDate interface{} `json:"frequency_date"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
LastUpdate string `json:"last_update"`
License string `json:"license"`
Metrics struct {
Discussions int `json:"discussions"`
Followers int `json:"followers"`
Issues int `json:"issues"`
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Reuses int `json:"reuses"`
Views int `json:"views"`
} `json:"metrics"`
Organization struct {
Acronym string `json:"acronym"`
Class string `json:"class"`
ID string `json:"id"`
Logo string `json:"logo"`
LogoThumbnail string `json:"logo_thumbnail"`
Name string `json:"name"`
Page string `json:"page"`
Slug string `json:"slug"`
URI string `json:"uri"`
} `json:"organization"`
Owner interface{} `json:"owner"`
Page string `json:"page"`
Private bool `json:"private"`
Resources []struct {
Checksum struct {
Type string `json:"type"`
Value string `json:"value"`
} `json:"checksum"`
CreatedAt string `json:"created_at"`
Description interface{} `json:"description"`
Extras struct {
} `json:"extras"`
Filesize int `json:"filesize"`
Filetype string `json:"filetype"`
Format string `json:"format"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
Latest string `json:"latest"`
Metrics struct {
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Views int `json:"views"`
} `json:"metrics"`
Mime string `json:"mime"`
PreviewURL string `json:"preview_url"`
Published string `json:"published"`
Title string `json:"title"`
Type string `json:"type"`
URL string `json:"url"`
} `json:"resources"`
Slug string `json:"slug"`
Spatial interface{} `json:"spatial"`
Tags []interface{} `json:"tags"`
TemporalCoverage interface{} `json:"temporal_coverage"`
Title string `json:"title"`
URI string `json:"uri"`
} `json:"data"`
Facets struct {
Format [][]interface{} `json:"format"`
} `json:"facets"`
NextPage string `json:"next_page"`
Page int `json:"page"`
PageSize int `json:"page_size"`
PreviousPage interface{} `json:"previous_page"`
Total int `json:"total"`
}
If you want inline decode of nested data for quick uses follow the below method:
myJsonData := `{
"code": "string_code",
"data": {
"id": 123,
"user": {
"username": "my_username",
"age": 30,
"posts": [ "post1", "post2"]
}
}
}`
Let's say you have the above nested and unknown JSON data that want to be read and parsed, first read that intomap[string]interface{}:
m := map[string]interface{}{}
err := json.Unmarshal([]byte(myJsonData), &m)
if err != nil {
log.Fatal(err)
}
Now if you want to access code
fmt.Println(m["code"])
For id in nested data block of JSON:
fmt.Println(m["data"].(map[string]interface{})["id"].(float64))
For username in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["username"].(string))
For age in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["age"].(float64))
For post1 in the third level nested posts block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["posts"].([]interface{})[0].(string))
Please check the example in playground

json: cannot unmarshal string into Go value of type map[string]interface {}

I have been working on converting some crypto pool software over to work with an incompatible coin. I believe I have it all about done. But this error keeps popping up and I just can't seem to figure out what the problem is. Here is my code:
type GetBalanceReply struct {
Unspent string `json:"unspent"`
}
type SendTransactionReply struct {
Hash string `json:"hash"`
}
type RPCClient struct {
sync.RWMutex
Url string
Name string
Account string
Password string
sick bool
sickRate int
successRate int
client *http.Client
}
type GetBlockReply struct {
Difficulty string `json:"bits"`
Hash string `json:"hash"`
MerkleTreeHash string `json:"merkle_tree_hash"`
Nonce string `json:"nonce"`
PrevHash string `json:"previous_block_hash"`
TimeStamp string `json:"time_stamp"`
Version string `json:"version"`
Mixhash string `json:"mixhash"`
Number string `json:"number"`
TransactionCount string `json:"transaction_count"`
}
type GetBlockReplyPart struct {
Number string `json:"number"`
Difficulty string `json:"bits"`
}
type TxReceipt struct {
TxHash string `json:"hash"`
}
type Tx struct {
Gas string `json:"gas"`
GasPrice string `json:"gasPrice"`
Hash string `json:"hash"`
}
type JSONRpcResp struct {
Id *json.RawMessage `json:"id"`
Result *json.RawMessage `json:"result"`
Balance *json.RawMessage `json:"balance"`
Transaction *json.RawMessage `json:"transaction"`
Error map[string]interface{} `json:"error"`
}
func (r *RPCClient) GetPendingBlock() (*GetBlockReplyPart, error) {
rpcResp, err := r.doPost(r.Url, "fetchheaderext", []interface{}{r.Account, r.Password, "pending"})
if err != nil {
return nil, err
}
if rpcResp.Result != nil {
var reply *GetBlockReplyPart
err = json.Unmarshal(*rpcResp.Result, &reply)
return reply, err
}
return nil, nil
}
func (r *RPCClient) GetBlockByHeight(height int64) (*GetBlockReply, error) {
//params := []interface{}{fmt.Sprintf("0x%x", height), true}
params := []interface{}{"-t", height}
return r.getBlockBy("fetch-header", params)
}
Whenever I run that manually in the wallet it displays:
"result": {
"bits": "7326472509313",
"hash": "060d0f6157d08bb294ad30f97a2c15c821ff46236281f118d65576b9e4a0ba27",
"merkle_tree_hash": "36936f36718e134e1eecaef05e66ebc4079811d8ee5a543f36d370335adc0801",
"nonce": "0",
"previous_block_hash": "10f4da59558fe41bab50a15864d1394462cd90836aecf52524f4cbce02a74a27",
"time_stamp": "1520718416",
"version": "1",
"mixhash": "0",
"number": "1011160",
"transaction_count": "1"
}'
But, the code errors out with "json: cannot unmarshal string into Go value of type map[string]interface {}"
Can anyone help me figure this one out? I've been hours on trying to figure it out on my own.
Try changing the type of the Error field to interface{} if you are not sure about the error structure, or change it to string if its always a string.
This should get rid of the error.
The error comes from unmarshaling JSONRpcResp. The input data has something like {error: "this is the error message", ...} and unmarshaling this into a map[string]interface{} will simply not work.

Cannot unmarshal string into Go value of type int64

I have struct
type tySurvey struct {
Id int64 `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
I do json.Marshal write JSON bytes in HTML page. jQuery modifies name field in object and encodes object using jQueries JSON.stringify and jQuery posts string to Go handler.
id field encoded as string.
Sent: {"id":1} Received: {"id":"1"}
Problem is that json.Unmarshal fails to unmarshal that JSON because id is not integer anymore.
json: cannot unmarshal string into Go value of type int64
What is best way to handle such data? I do not wish to manually convert every field. I wish to write compact, bug free code.
Quotes is not too bad. JavaScript does not work well with int64.
I would like to learn the easy way to unmarshal json with string values in int64 values.
This is handled by adding ,string to your tag as follows:
type tySurvey struct {
Id int64 `json:"id,string,omitempty"`
Name string `json:"name,omitempty"`
}
This can be found about halfway through the documentation for Marshal.
Please note that you cannot decode the empty string by specifying omitempty as it is only used when encoding.
use json.Number
type tySurvey struct {
Id json.Number `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
You could also create a type alias for int or int64 and create a custom json unmarshaler
Sample code:
Reference
// StringInt create a type alias for type int
type StringInt int
// UnmarshalJSON create a custom unmarshal for the StringInt
/// this helps us check the type of our value before unmarshalling it
func (st *StringInt) UnmarshalJSON(b []byte) error {
//convert the bytes into an interface
//this will help us check the type of our value
//if it is a string that can be converted into a int we convert it
///otherwise we return an error
var item interface{}
if err := json.Unmarshal(b, &item); err != nil {
return err
}
switch v := item.(type) {
case int:
*st = StringInt(v)
case float64:
*st = StringInt(int(v))
case string:
///here convert the string into
///an integer
i, err := strconv.Atoi(v)
if err != nil {
///the string might not be of integer type
///so return an error
return err
}
*st = StringInt(i)
}
return nil
}
func main() {
type Item struct {
Name string `json:"name"`
ItemId StringInt `json:"item_id"`
}
jsonData := []byte(`{"name":"item 1","item_id":"30"}`)
var item Item
err := json.Unmarshal(jsonData, &item)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+v\n", item)
}