Server does't get all the JSON from remote REST API - json

My Go server does not retrieve all the JSON data from a remote API
I even tried creating a custom http.Client to make the request...but still it won't retrieve ALL the JSON data,and even tried extending the respective timeouts
Here's my code:
var netTransport = &http.Transport{
Dial: (&net.Dialer{
Timeout: 50 * time.Second,
}).Dial,
TLSHandshakeTimeout: 50 * time.Second,
}
var netClient = &http.Client{
Timeout: time.Second * 40,
Transport: netTransport,
}
res, getErr := netClient.Get(url)
if getErr != nil {
log.Fatal(getErr)
}
data := JSONData{}
if err := json.NewDecoder(res.Body).Decode(&data); err != nil {
log.Fatal(err)
}
data.Username = user
fmt.Println(data)
JSONData is defined as follows:
type Owner struct {
Login string
}
// Item is the single repository data structure
type Item struct {
ID int
Name string
FullName string `json:"full_name"`
Owner Owner
Description string
CreatedAt string `json:"created_at"`
}
// JSONData contains the GitHub API response
type JSONData struct {
Count int `json:"total_count"`
Username string
Items []Item
}

The JSONData needs to define JSON tags.
type Owner struct {
Login string `json:"login"`
}
// Item is the single repository data structure
type Item struct {
ID int `json:"id"`
Name string `json:"name"`
FullName string `json:"full_name"`
Owner Owner `json:"owner"`
Description string `json:"description"`
CreatedAt string `json:"created_at"`
}
// JSONData contains the GitHub API response
type JSONData struct {
Count int `json:"total_count"`
Username string `json:"username"`
Items []Item `json:"items"`
}
If no tag is specified, the name is taken as-is:
type Owner struct {
Login string
}
that would be marshaled (and correspondingly unmarshalled) into similar:
{"Login": "some login"}

Related

error: json: unsupported type: func() time.Time in Golang

i am new in golang, just try some API in Echo Framework and got some error.
My Models :
package models
import (
"net/http"
"quotes/db"
)
type Quote struct {
Id int `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
}
func GetAll() (Response, error) {
var quotes Quote
var res Response
ctx := db.Init()
ctx.Find(&quotes)
res.Status = http.StatusOK
res.Message = "Success"
res.Data = ctx
return res, nil
}
My Schema table
package schema
type Quotes struct {
Id int `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
}
My Response type for Api
package models
type Response struct {
Status int `json:"status"`
Message string `json:"message"`
Data interface{} `json:"data"`
}
i tried to add this in Models and Schema :
CreatedAt time.Time `gorm:"type:timestamp" json:"created_at,string,omitempty"`
UpdatedAt time.Time `gorm:"type:timestamp" json:"updated_at,string,omitempty"`
DeletedAt time.Time `gorm:"type:timestamp" json:"deleted_at,string,omitempty"`
And Still Not Works, any solutions?
I expect the api work with no errors
When using gorm, you need to embed a gorm.Model struct, which includes fields ID, CreatedAt, UpdatedAt, DeletedAt.
Reference
// gorm.Model definition
type Model struct {
ID uint `gorm:"primaryKey"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt gorm.DeletedAt `gorm:"index"`
}
Not familiar with echo but read below to understand how you use gorm.
In your case you can try doing the following:
package schema
type Quote struct {
gorm.Model
Title string `json:"title"`
Description string `json:"description"`
}
Then to get all the quotes:
func GetAll() (Response, error) {
var quotes []schema.Quote // slice
ctx := db.Init()
// Assuming
// ctx, err := gorm.Open(....)
// https://gorm.io/docs/query.html
result := db.Find(&quotes)
if result.Error != nil {
return Response{
Status: http.StatusInternalServerError,
Message: "Query failed",
},result.Error
}
if result.RowsAffected == 0 {
return Response{
Status: http.StatusNotFound,
Message: "No records found",
},nil
}
return Response{
Status: http.StatusOK,
Message: "Success",
Data: quotes,
},nil
}
Keep in mind that the Data field has type interface{}, which means it can hold a value of any type. If the value wasn't a slice you would be using the & operator you take the address of the Quote value. A slice is already a pointer to underlying slice so need to use the & operator.
If you want to access the slice of Quote values from the Data field, you will need to use a type assertion to convert the value from the interface{} type to the []Quote type. Here's an example of how you could do this:
// Assume that response.Data holds a slice of Quote values
quotes, ok := response.Data.([]Quote)
if !ok {
// Handle the case where response.Data is not a slice of Quote
}
Warning: Since you are returning a slice, then any changes to the returned slice will be modifying the initial slice too. If you wanted to avoid this then copy the slice values to a new slice:
quotesCopy = make([]schema.Quote, len(quotes))
copy(quotesCopy, quotes)

GoLang JSON payload preparation with multiple data

I want to create JSON payload in given below format. I want a code or pattern that prepares the given format.
{
transactiontype: 'DDDDD'
emailType: 'QQQQQQ'
template: {
templateUrl: 'xry.kk'
templateName: 'chanda'
}
date: [
{
UserId: 1
Name: chadnan
},
{
UserId: 2
Name: kkkkkk
}
]
}
Hope this helps :
type Template struct {
TemplateURL string `json:"templateUrl" param:"templateUrl"`
TemplateName string `json:"templateName" param:"templateName"`
}
type Date struct {
UserId string `json:"UserId" param:"UserId"`
Name string `json:"Name" param:"Name"`
}
type NameAny struct {
*Template
TransactionType string `json:"transactiontype" param:"transactiontype"`
EmailType string `json:"emailType" param:"emailType"`
Data []Date `json:"date" param:"date"`
}
Data, _ := json.Marshal(NameAny)
Json(c, string(Data))(w, r)
You can use an online tool to convert a json into a valid Go struct: https://mholt.github.io/json-to-go/
Given your JSON, the Go struct is:
type AutoGenerated struct {
Transactiontype string `json:"transactiontype"`
EmailType string `json:"emailType"`
Template struct {
TemplateURL string `json:"templateUrl"`
TemplateName string `json:"templateName"`
} `json:"template"`
Date []struct {
UserID int `json:"UserId"`
Name string `json:"Name"`
} `json:"date"`
}
After the conversion, use the json.Marshal (Go Struct to JSON) and json.Unmarshal (JSON to Go Struct)
Complete example with your data: https://play.golang.org/p/RJuGK4cY1u-
// Transaction is a struct which stores the transaction details
type Transaction struct {
TransactionType string `json:"transaction_type"`
EmailType string `json:"email_type"`
Template Template `json:"template"`
Date []Date `json:"date"`
}
//Template is a struct which stores the template details
type Template struct {
TemplateURL string `json:"template_url"`
TemplateName string `json:"template_name"`
}
// Date is a struct which stores the user details
type Date struct {
UserID int `json:"user_id"`
Name string `json:"name"`
}
Above given structs are the correct data structure for storing your json body you can use json decoder for perfectly storing the data into struct
func exampleHandler(w http.ResponseWriter, r *http.Request) {
var trans Transaction
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&trans)
if err != nil {
log.Println(err)
}
}

How to navigate a nested json with unknown structure in Go?

I am trying to parse and get selected data from a deep nested json data in Go Lang. I'm having issues navigating through the structure and accessing the data. The data is too deep and complex to be parsed with a-priori known structures in Go.
Here is the URL of the file:
-https://www.data.gouv.fr/api/1/datasets/?format=csv&page=0&page_size=20
I did some parsing with map interfaces and using a json string:
resultdata := map[string]interface {}
json.Unmarshal([]byte(inputbytestring), &resultdata) //Inputstring is the string containing the JSON data of the above URL
The problem:
How can turn resultdata into a map (of strings), so I can use methods available for maps?
The JSON data is nested and has several levels. How is it possible to access the lower level JSON fields? is it possible to unmarshal the data recursively?
Once you have data as a map[string]interface{}, you can use type assertions to get to the lower levels of data.
There's a good explanation here of how to do this at https://blog.golang.org/json-and-go
Here's an example to get you most of the way:
https://play.golang.org/p/P8cGP1mTDmD
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
jsonData := `{
"string": "string_value",
"number": 123.45,
"js_array": ["a", "b", "c"],
"integer": 678,
"subtype": {
"number_array": [1, 2, 3]
}
}`
m := map[string]interface{}{}
err := json.Unmarshal([]byte(jsonData), &m)
if err != nil {
log.Fatal(err)
}
for key, value := range m {
switch v := value.(type) {
case int:
fmt.Printf("Key: %s, Integer: %d\n", key, v)
case float64:
fmt.Printf("Key: %s, Float: %v\n", key, v)
case string:
fmt.Printf("Key: %s, String: %s\n", key, v)
case map[string]interface{}:
fmt.Printf("Key: %s, Subtype: %+v\n", key, v)
case []interface{}:
//TODO: Read through each item in the interface and work out what type it is.
fmt.Printf("Key: %s, []interface: %v\n", key, v)
default:
fmt.Printf("Key: %s, unhandled type: %+v\n", key, v)
}
}
}
Alternatively https://mholt.github.io/json-to-go/ does a decent job of turning examples of JSON data into Go structs that can be used for marshalling.
Putting the example in, I get something that isn't too bad.
type AutoGenerated struct {
Data []struct {
Acronym interface{} `json:"acronym"`
Badges []interface{} `json:"badges"`
CreatedAt string `json:"created_at"`
Deleted interface{} `json:"deleted"`
Description string `json:"description"`
Extras struct {
} `json:"extras"`
Frequency string `json:"frequency"`
FrequencyDate interface{} `json:"frequency_date"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
LastUpdate string `json:"last_update"`
License string `json:"license"`
Metrics struct {
Discussions int `json:"discussions"`
Followers int `json:"followers"`
Issues int `json:"issues"`
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Reuses int `json:"reuses"`
Views int `json:"views"`
} `json:"metrics"`
Organization struct {
Acronym string `json:"acronym"`
Class string `json:"class"`
ID string `json:"id"`
Logo string `json:"logo"`
LogoThumbnail string `json:"logo_thumbnail"`
Name string `json:"name"`
Page string `json:"page"`
Slug string `json:"slug"`
URI string `json:"uri"`
} `json:"organization"`
Owner interface{} `json:"owner"`
Page string `json:"page"`
Private bool `json:"private"`
Resources []struct {
Checksum struct {
Type string `json:"type"`
Value string `json:"value"`
} `json:"checksum"`
CreatedAt string `json:"created_at"`
Description interface{} `json:"description"`
Extras struct {
} `json:"extras"`
Filesize int `json:"filesize"`
Filetype string `json:"filetype"`
Format string `json:"format"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
Latest string `json:"latest"`
Metrics struct {
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Views int `json:"views"`
} `json:"metrics"`
Mime string `json:"mime"`
PreviewURL string `json:"preview_url"`
Published string `json:"published"`
Title string `json:"title"`
Type string `json:"type"`
URL string `json:"url"`
} `json:"resources"`
Slug string `json:"slug"`
Spatial interface{} `json:"spatial"`
Tags []interface{} `json:"tags"`
TemporalCoverage interface{} `json:"temporal_coverage"`
Title string `json:"title"`
URI string `json:"uri"`
} `json:"data"`
Facets struct {
Format [][]interface{} `json:"format"`
} `json:"facets"`
NextPage string `json:"next_page"`
Page int `json:"page"`
PageSize int `json:"page_size"`
PreviousPage interface{} `json:"previous_page"`
Total int `json:"total"`
}
If you want inline decode of nested data for quick uses follow the below method:
myJsonData := `{
"code": "string_code",
"data": {
"id": 123,
"user": {
"username": "my_username",
"age": 30,
"posts": [ "post1", "post2"]
}
}
}`
Let's say you have the above nested and unknown JSON data that want to be read and parsed, first read that intomap[string]interface{}:
m := map[string]interface{}{}
err := json.Unmarshal([]byte(myJsonData), &m)
if err != nil {
log.Fatal(err)
}
Now if you want to access code
fmt.Println(m["code"])
For id in nested data block of JSON:
fmt.Println(m["data"].(map[string]interface{})["id"].(float64))
For username in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["username"].(string))
For age in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["age"].(float64))
For post1 in the third level nested posts block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["posts"].([]interface{})[0].(string))
Please check the example in playground

json NewDecoder Decode not parsing integers

I'm trying to make an API endpoint which would save stuff into a database however, the integers which I pass in via POST request don't seem to be parsing correctly
Here's my structs:
type OnlineTestForm struct {
Form OnlineTestSet `json:"form"`
}
type OnlineTestSet struct {
ID int `db:"id" json:"id"`
OnlineTestSubjectId int `db:"online_test_subject_id" json:"online_test_subject_id"`
Name string `db:"name" json:"name"`
ScoreToPass int `db:"score_to_pass" json:"score_to_pass"`
TimeLimit int `db:"time_limit" json:"time_limit"`
Description string `db:"description" json:"description"`
Enabled bool `db:"enabled" json:"enabled"`
Online bool `db:"online" json:"online"`
TestType string `db:"test_type" json:"test_type"`
DocName string `db:"doc_name" json:"doc_name"`
}
And the "problematic" function where I parse the JSON into a struct:
func NewOnlineTest(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
var jsonForm OnlineTestForm
json.NewDecoder(r.Body).Decode(&jsonForm)
err := addNewOnlineTest(jsonForm.Form)
if err != nil {
fmt.Println(err)
w.WriteHeader(500)
fmt.Fprint(w, "{\"error\": {\"message\": \"An error occured while adding new test - "+err.Error()+"\"}}")
return
}
w.WriteHeader(200)
}
I'm using httprouter for as my router and the route to which I post is defined like this
router.POST("/onlinetest/new", NewOnlineTest)
And finally the test POST request which I send has the following payload:
{
"form":{
"name":"test",
"test_type":"document",
"score_to_pass":32,
"time_limit":324,
"enabled":true,
"online":true,
"online_test_subject_id":1
}
}
The problem occurs when I try to use jsonForm.Form' but the integers I've passed in liketime_limitandscore_to_pass` are 0
{0 0 test 0 0 true true document []}
Found it!
Needed to add string at the end of my json struc declaration, not sure why but it worked
type OnlineTestSet struct {
ID int `db:"id" json:"id"`
OnlineTestSubjectId int `db:"online_test_subject_id" json:"online_test_subject_id,string"`
Name string `db:"name" json:"name"`
ScoreToPass int `db:"score_to_pass" json:"score_to_pass,string"`
TimeLimit int `db:"time_limit" json:"time_limit,string"`
Description string `db:"description" json:"description"`
Enabled bool `db:"enabled" json:"enabled"`
Online bool `db:"online" json:"online"`
TestType string `db:"test_type" json:"test_type"`
DocName string `db:"doc_name" json:"doc_name"`
}

Pass `struct` as type name to function argument

I'm writing some sort of RESTfull API based Object relational mapper in go.
I plan to make it MIT licensed, when i finish it.
The idea is to use some 3rd party REST API as data storage, and the golang client will query it for data needed.
The API responses are JSONs with known structure.
this is my code:
type AClient struct {
Id string `json:"id"`
Uid string `json:"uid"`
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
CreatedAt string `json:"createdAt"`
UpdatedAt string `json:"updatedAt"`
City string `json:"city"`
Address string `json:"address"`
Telefone string `json:"telefone"`
Zip string `json:"zip"`
Telefon string `json:"telefon"`
Comment string `json:"comment"`
}
type AEvents struct {
Id string `json:"id"`
Security bool `json:"security"`
Description string `json:"description"`
Good AGood `json:"good"`
Client AClient `json:"client"`
Author AAuthor `json:"author"`
InFuture bool `json:"inFuture"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
type Entry struct {
AEvents //this have to be changed to `AClients` in runtime when needed
}
type ORM struct {
ApiUrl string
ModelName string
ModelInterface Entry
HuntKey string
HuntSid string
Csrf string
}
func (o *ORM) Query(parameters map[string]string) ([]Entry, AMetadata, error) {
responseParsed := struct {
Status string `json:"status"`
Metadata AMetadata `json:"metadata"`
Data []Entry `json:"data"` //todo - use o.ModelInterface
}{}
client := &http.Client{}
var queryString string
for k, v := range parameters {
queryString = queryString + fmt.Sprintf("%v=%v&", url.QueryEscape(k), url.QueryEscape(v))
}
req, err := http.NewRequest("GET", fmt.Sprintf("%v%v?%v", o.ApiUrl, o.ModelName, queryString), nil)
fmt.Println("---------------------------------------------")
fmt.Println(fmt.Sprintf("[GET] %v%v?%v", o.ApiUrl, o.ModelName, queryString))
req.Header.Set("huntKey", o.HuntKey)
if err != nil {
return nil, AMetadata{}, err
}
res, err1 := client.Do(req)
defer res.Body.Close()
if err1 != nil {
return nil, AMetadata{}, err1
}
if res.StatusCode == 200 {
for _, v := range res.Cookies() {
if v.Name == "XSRF-TOKEN" {
o.Csrf = v.Value
}
if v.Name == "hunt.sid" {
o.HuntSid = v.Value
}
}
fmt.Printf("CSRF %v\n", o.Csrf)
fmt.Printf("HuntSid %v\n", o.HuntSid)
fmt.Println("---------------------------------------------")
raw, err2 := ioutil.ReadAll(res.Body)
if err2 != nil {
return nil, AMetadata{}, err2
} else {
err2 = json.Unmarshal(raw, &responseParsed)
return responseParsed.Data, responseParsed.Metadata, nil
}
} else {
return nil, AMetadata{}, errors.New("Unable to fetch data!")
}
}
How can I make this:
When instantiating the ORM object, how can i pass the struct name, that will be used to parse the JSON response. The current code works with the struct of AEvents, but i want it to be easy changeble t AClient and so on.
UPD:
i have reviewed the code of https://github.com/jinzhu/gorm and find out tons of things how can i implement it.
Than, as i have promised, I publish this code as open source -https://github.com/vodolaz095/hrorm
You can use reflection, but be warned that it is non-trivial and relatively slow. I don't know of any other way.
http://golang.org/pkg/reflect/
http://blog.golang.org/laws-of-reflection
The simplest way to do this is to make your parameter of type interface{} and pass in an empty (uninitialized) instance of the struct you wish to unmarshal into. I highly suggest reading the second of the two links I list - it provides a clear introduction to reflection and how to use it for solving problems like this one.