I am very, very new to golang (using this project as a way to learn the language better). I have an api I'd like to pull data from, but I can't seem to create the right type of struct for it.
The full JSON I am trying to encode is located here. https://prices.runescape.wiki/api/v1/osrs/latest
The structure of the JSON is
{
"data": {
"2": {
"high": 182,
"highTime": 1621811749,
"low": 180,
"lowTime": 1621811755
},
"6": {
"high": 186683,
"highTime": 1621811083,
"low": 184528,
"lowTime": 1621811286
},
... REPEATS THOUSANDS OF TIMES ...
}
}
I have tried using the JSON to Go converter for the JSON data, but that returns a struct that is entirely too large. (Go returns an error) I have tried to manually make a struct with a slice as follows
type osrsPrices []struct {
ID struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
} `json:"id"`
}
Whenever I try to run this from the terminal I am presented with the error "json: cannot unmarshal object into Go value of type main.osrsPrices exit status 1"
Here is the entire code I am using
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type osrsPrices []struct {
ID struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
} `json:"id"`
}
func main() {
url := "https://prices.runescape.wiki/api/v1/osrs/latest"
spaceClient := http.Client{
Timeout: time.Second * 2, // Timeout after 2 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("User-Agent", "skillerscape:LearningGoLang|priceChecker")
res, getErr := spaceClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
items := osrsPrices{}
jsonErr := json.Unmarshal(body, &items)
if jsonErr != nil {
log.Fatal(jsonErr)
}
for _, item := range items {
fmt.Println(item)
}
}
I am pretty sure the error has to do with the way I am trying to process the data, but I cannot figure out the issue directly, nor the solution despite a couple hours of googling. I appreciate any help anyone has on this issue.
At the deepest level, you have:
type Price struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
}
This is in an object with variable keys, so you need:
type Data struct {
Data map[string]Price `json:"data"`
}
Then you can unmarshal into an instance of this struct:
var data Data
json.Unmarshal(input,&data)
Related
i'm having a lot of trouble getting my program to work. I want to unmarshal something pretty simple, but it's giving me a lot of issues, unfortunately.
Here is the response that I want to unmarshal:
{"error":[],"result":{"XXBTZUSD":[[1647365820,"39192.0","39192.0","39191.9","39191.9","39191.9","0.18008008",10],[1647365880,"39186.1","39186.1","39172.0","39176.0","39174.4","0.13120077",10]],"last":1647408900}}
I've wrote these structs to help with unmarshalling
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
Time float64
Open string
High string
Low string
Close string
Vwa string
Volume string
Count float64
}
I have a function call that makes the http request and then unmarshals the data. For whatever reason, my code will end before even starting the function call for the http request and subsequent unmarshalling when the Pair type is []OHLC or []*OHLC. If I change the Pair type to interface{}, then it runs. i want to make it work with the OHLC struct instead though. Below is the complete code:
package main
import (
"fmt"
"net/http"
//"strings"
"io/ioutil"
"encoding/json"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main");
getOhlc()
}
func getOhlc() {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD");
if err != nil {
fmt.Errorf("error after request")
return;
}
defer resp.Body.Close();
body, err := ioutil.ReadAll(resp.Body);
if err != nil {
fmt.Errorf("error when reading")
return;
}
var jsonData Resp;
err = json.Unmarshal(body, &jsonData);
if err != nil {
fmt.Errorf("error when unmarshalling")
return
}
if(len(jsonData.Error) > 0) {
fmt.Errorf("error");
return;
}
fmt.Println(jsonData);
}
Any ideas about what might be happening?
"Any ideas about what might be happening?"
The elements in the "XXBTZUSD" JSON array are arrays themselves, i.e. "XXBTZUSD" is an array of arrays. The OHLC type is a struct type. The stdlib will not, by itself, unmarshal a JSON array into a Go struct. Go structs can be used to unmarshal JSON objects. JSON arrays can be unmarshaled into Go slices or arrays.
You would clearly see that that's the issue if you would just print the error from json.Unmarshal:
json: cannot unmarshal array into Go struct field
Trades.result.XXBTZUSD of type main.OHLC
https://go.dev/play/p/D4tjXZVzDI_w
If you want to unmarshal a JSON array into a Go struct you have to have the Go struct type implement a the json.Unmarshaler interface.
func (o *OHLC) UnmarshalJSON(data []byte) error {
// first unmarshal the array into a slice of raw json
raw := []json.RawMessage{}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
// create a function that unmarshals each raw json element into a field
unmarshalFields := func(raw []json.RawMessage, fields ...interface{}) error {
if len(raw) != len(fields) {
return errors.New("bad number of elements in json array")
}
for i := range raw {
if err := json.Unmarshal([]byte(raw[i]), fields[i]); err != nil {
return err
}
}
return nil
}
// call the function
return unmarshalFields(
raw,
&o.Time,
&o.Open,
&o.High,
&o.Low,
&o.Close,
&o.Vwa,
&o.Volume,
&o.Count,
)
}
https://go.dev/play/p/fkFKLkaNaSU
Your code had some issues:
Remove semicolons from end of lines, it's redundant.
fmt.Errorf return error, and not print it, every time check your error and propagate it.
We can convert array of numbers and string to struct in golang.
for achieving your desired output we need to first convert to intermediate container and then convert to our wanted output:
package main
import (
"errors"
"fmt"
"log"
"net/http"
//"strings"
"encoding/json"
"io/ioutil"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type IntermediateResp struct {
Error []string `json:"error"`
Result IntermediateTrades `json:"result"`
}
type IntermediateTrades struct {
Pair [][]interface{} `json:"XXBTZUSD"`
Last int `json:"last"`
}
type Trades struct {
Pair []OHLC `json:"result"`
Last int `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main")
err := getOhlc()
if err != nil {
log.Fatal(err)
}
}
func buildOHLC(l []interface{}) (*OHLC, error) {
if len(l) < 8 {
return nil, errors.New("short list")
}
return &OHLC{
TT: l[0].(float64),
Open: l[1].(string),
High: l[2].(string),
Low: l[3].(string),
Close: l[4].(string),
Vwap: l[5].(string),
Volume: l[6].(string),
Count: l[7].(float64),
}, nil
}
func convert(r IntermediateResp) (*Resp, error) {
result := &Resp{Error: r.Error, Result: Trades{Pair: make([]OHLC, len(r.Result.Pair)), Last: r.Result.Last}}
for i, v := range r.Result.Pair {
ohlc, err := buildOHLC(v)
if err != nil {
return nil, err
}
result.Result.Pair[i] = *ohlc
}
return result, nil
}
func getOhlc() error {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD")
if err != nil {
return fmt.Errorf("error after request, %v", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
if err != nil {
return fmt.Errorf("error when reading %v", err)
}
var jsonData IntermediateResp
err = json.Unmarshal(body, &jsonData)
if err != nil {
return fmt.Errorf("error when unmarshalling %v", err)
}
if len(jsonData.Error) > 0 {
return fmt.Errorf("error")
}
convertedOhlc, err := convert(jsonData)
if err != nil {
return fmt.Errorf("error when convertedOhlc %v", err)
}
fmt.Println(convertedOhlc)
return nil
}
We define IntermediateResp and IntermediateTrades for Unmarshaling json and then convert it to actual Resp.
I think aother way is using custom Unmarshal for Trades struct.
So I'm trying to get the number of pigeons from a JSON having this format. This JSON contains a lot of bird types and each one is defined by his color and last contact:
{
"url": "http://localhost:9001/",
"pigeons": [
{
"color": "white",
"lastContact": "2020-03-23T14:46:20.806Z"
},
{
"color": "grey",
"lastContact": "2020-03-23T14:46:20.807Z"
}
],
"parrots": [
{
"color": "green",
"lastContact": "2020-03-23T14:46:20.806Z"
}
]
}
Already made this piece of code that gets the JSON from the API, but since I don't have any experience in Go, can you guys help me to count the number of pigeons from here? I don't really care about the number of other bird types.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type pigeons struct {
Number int `json:"something"`
}
func main() {
url := "http://localhost:9001"
birdsClient := http.Client{
Timeout: time.Second * 2, // Maximum of 2 secs
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
res, getErr := birdsClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
pigeons1 := pigeons{}
jsonErr := json.Unmarshal(body, &pigeons1)
if jsonErr != nil {
log.Fatal(jsonErr)
}
fmt.Println(pigeons1.Number)
}
In the JSON document that comes back, pigeons is an array and it looks like the length of that array is the number of pigeons. So if you unmarshal that into a struct that accepts and array for pigeons, you can get the length of it:
type pigeons struct {
Pigeons []interface{} `json:"pigeons"`
}
Above, you can unmarshal the pigeons field to an interface array because you don't care about the contents of the field. If you need to process contents, you need a separate struct and use the array of that. Then:
var p pigeons
json.Unmarshal(body, &p)
fmt.Printf("%d",len(p.Pigeons))
I need to unmarshal json object which may have the following formats:
Format1:
{
"contactType": 2,
"value": "0123456789"
}
Format2:
{
"contactType": "MobileNumber",
"value": "0123456789"
}
The structure I'm using for unmarshalling is:-
type Contact struct {
ContactType int `json:"contactType"`
Value string `json:"value"`
}
But this works only for format 1. I don't want to change the datatype of ContactType but I want to accommodate the 2nd format as well. I heard about json.RawMarshal and tried using it.
type Contact struct {
ContactType int
Value string `json:"value"`
Type json.RawMessage `json:"contactType"`
}
type StringContact struct {
Type string `json:"contactType"`
}
type IntContact struct {
Type int `json:"contactType"`
}
This gets the unmarshalling done, but I'm unable to set the ContactType variable which depends on the type of json.RawMessage. How do I model my structure so that this problem gets solved?
You will need to do the unmarshalling yourself. There is a very good article that shows how to use the json.RawMessage right and a number of other solutions to this very problem, Like using interfaces, RawMessage, implemention your own unmarshal and decode functions etc.
You will find the article here: JSON decoding in GO by Attila Oláh
Note: Attila has made a few errors on his code examples.
I taken the liberty to put together (using some of the code from Attila) a working example using RawMessage to delay the unmarshaling so we can do it on our own version of the Decode func.
Link to GOLANG Playground
package main
import (
"fmt"
"encoding/json"
"io"
)
type Record struct {
AuthorRaw json.RawMessage `json:"author"`
Title string `json:"title"`
URL string `json:"url"`
Author Author
}
type Author struct {
ID uint64 `json:"id"`
Email string `json:"email"`
}
func Decode(r io.Reader) (x *Record, err error) {
x = new(Record)
if err = json.NewDecoder(r).Decode(x); err != nil {
return
}
if err = json.Unmarshal(x.AuthorRaw, &x.Author); err == nil {
return
}
var s string
if err = json.Unmarshal(x.AuthorRaw, &s); err == nil {
x.Author.Email = s
return
}
var n uint64
if err = json.Unmarshal(x.AuthorRaw, &n); err == nil {
x.Author.ID = n
}
return
}
func main() {
byt_1 := []byte(`{"author": 2,"title": "some things","url": "https://stackoverflow.com"}`)
byt_2 := []byte(`{"author": "Mad Scientist","title": "some things","url": "https://stackoverflow.com"}`)
var dat Record
if err := json.Unmarshal(byt_1, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
if err := json.Unmarshal(byt_2, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
}
Hope this helps.
Let's say I have the following Go struct on the server
type account struct {
Name string
Balance int
}
I want to call json.Decode on the incoming request to parse it into an account.
var ac account
err := json.NewDecoder(r.Body).Decode(&ac)
If the client sends the following request:
{
"name": "test#example.com",
"balance": "3"
}
Decode() will return the following error:
json: cannot unmarshal string into Go value of type int
Now it's possible to parse that back into "you sent a string for Balance, but you really should have sent an integer", but it's tricky, because you don't know the field name. It also gets a lot trickier if you have a lot of fields in the request - you don't know which one failed to parse.
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
You can use a custom type with custom unmarshaling algorythm for your "Balance" field.
Now there are two possibilities:
Handle both types:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Int int
type account struct {
Name string
Balance Int
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var s string
err = json.Unmarshal(b, &s)
if err == nil {
var n int
n, err = strconv.Atoi(s)
if err != nil {
return
}
*i = Int(n)
return
}
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
Handle only a numeric type, and fail anything else with a sensible error:
package main
import (
"encoding/json"
"fmt"
)
type Int int
type account struct {
Name string
Balance Int
}
type FormatError struct {
Want string
Got string
Offset int64
}
func (fe *FormatError) Error() string {
return fmt.Sprintf("Invalid data format at %d: want: %s, got: %s",
fe.Offset, fe.Want, fe.Got)
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
return
}
if ute, ok := err.(*json.UnmarshalTypeError); ok {
err = &FormatError{
Want: "number",
Got: ute.Value,
Offset: ute.Offset,
}
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %#v\n", err)
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
There is a third possibility: write custom unmarshaler for the whole account type, but it requires more involved code because you'd need to actually iterate over the input JSON data using the methods of the
encoding/json.Decoder type.
After reading your
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
more carefully, I admit having a custom parser for the whole type is the only sensible possibility unless you are OK with a 3rd-party package implementing a parser supporting validation via JSON schema (I think I'd look at this first as juju is a quite established product).
A solution for this could be to use a type assertion by using a map to unmarshal the JSON data into:
type account struct {
Name string
Balance int
}
var str = `{
"name": "test#example.com",
"balance": "3"
}`
func main() {
var testing = map[string]interface{}{}
err := json.Unmarshal([]byte(str), &testing)
if err != nil {
fmt.Println(err)
}
val, ok := testing["balance"]
if !ok {
fmt.Println("missing field balance")
return
}
nv, ok := val.(float64)
if !ok {
fmt.Println("balance should be a number")
return
}
fmt.Printf("%+v\n", nv)
}
See http://play.golang.org/p/iV7Qa1RrQZ
The type assertion here is done using float64 because it is the default number type supported by Go's JSON decoder.
It should be noted that this use of interface{} is probably not worth the trouble.
The UnmarshalTypeError (https://golang.org/pkg/encoding/json/#UnmarshalFieldError) contains an Offset field that could allow retrieving the contents of the JSON data that triggered the error.
You could for example return a message of the sort:
cannot unmarshal string into Go value of type int near `"balance": "3"`
It would seem that here provides an implementation to work around this issue in Go only.
type account struct {
Name string
Balance int `json:",string"`
}
In my estimation the more correct and sustainable approach is for you to create a client library in something like JavaScript and publish it into the NPM registry for others to use (private repository would work the same way). By providing this library you can tailor the API for the consumers in a meaningful way and prevent errors creeping into your main program.
I have a json in the following format
{
"status_code": 200,
"status_message": "OK",
"response": {
"Messages": [
"CODE_NOT_AVAILABLE"
],
"UnknownDevices": {
"": [
"6",
"7",
"8",
"9",
"10"
]
}
}
}
As we see we are missing one index key, after unknownDevices, i am trying to unmarshal this json using golan in the following way
package main
import (
"encoding/json"
"fmt"
)
type pushWooshResponse struct {
Status int `json:"status_code"`
Status_msg string `json:"status_message"`
Response response
}
type response struct {
Message []string `json:"Messages"`
UnknownDevices devices
}
type devices struct {
Udevices []string `json:""`
}
func main() {
itemInfoR := `{"status_code":200,"status_message":"OK","response":{"Messages":["CODE_NOT_AVAILABLE"],"UnknownDevices":{"devices":["6","7","8","9","10"]}}}`
itemInfoBytes := []byte(itemInfoR)
var ItemInfo pushWooshResponse
er := json.Unmarshal(itemInfoBytes, &ItemInfo)
if er != nil {
fmt.Println("Error", er.Error())
} else {
fmt.Println(ItemInfo)
}
}
Output is
{200 OK {[CODE_NOT_AVAILABLE] {[]}}}
Everything is working fine except for the last part, which i am not able to unmarshall this. Can you help me out to umarshal the last part of the json.
There may be other options, but whenever you see strange JSON you can always fall back to implementing your own custom (un)marshalling by implementing json.Unmarshaler and/or json.Marshaler.
Perhaps something like:
type devices struct {
Udevices []string
}
func (d *devices) UnmarshalJSON(b []byte) error {
var x map[string][]string
err := json.Unmarshal(b, &x)
if err == nil {
// perhaps check that only a single
// key exists in the map as well
d.Udevices = x[""]
}
return err
}
func (d devices) MarshalJSON() ([]byte, error) {
x := map[string][]string{"": d.Udevices}
return json.Marshal(x)
}
Playground