How to fetch JSON from an API and count number of pigeons - json

So I'm trying to get the number of pigeons from a JSON having this format. This JSON contains a lot of bird types and each one is defined by his color and last contact:
{
"url": "http://localhost:9001/",
"pigeons": [
{
"color": "white",
"lastContact": "2020-03-23T14:46:20.806Z"
},
{
"color": "grey",
"lastContact": "2020-03-23T14:46:20.807Z"
}
],
"parrots": [
{
"color": "green",
"lastContact": "2020-03-23T14:46:20.806Z"
}
]
}
Already made this piece of code that gets the JSON from the API, but since I don't have any experience in Go, can you guys help me to count the number of pigeons from here? I don't really care about the number of other bird types.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type pigeons struct {
Number int `json:"something"`
}
func main() {
url := "http://localhost:9001"
birdsClient := http.Client{
Timeout: time.Second * 2, // Maximum of 2 secs
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
res, getErr := birdsClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
pigeons1 := pigeons{}
jsonErr := json.Unmarshal(body, &pigeons1)
if jsonErr != nil {
log.Fatal(jsonErr)
}
fmt.Println(pigeons1.Number)
}

In the JSON document that comes back, pigeons is an array and it looks like the length of that array is the number of pigeons. So if you unmarshal that into a struct that accepts and array for pigeons, you can get the length of it:
type pigeons struct {
Pigeons []interface{} `json:"pigeons"`
}
Above, you can unmarshal the pigeons field to an interface array because you don't care about the contents of the field. If you need to process contents, you need a separate struct and use the array of that. Then:
var p pigeons
json.Unmarshal(body, &p)
fmt.Printf("%d",len(p.Pigeons))

Related

Unmarshaling JSON in golang

i'm having a lot of trouble getting my program to work. I want to unmarshal something pretty simple, but it's giving me a lot of issues, unfortunately.
Here is the response that I want to unmarshal:
{"error":[],"result":{"XXBTZUSD":[[1647365820,"39192.0","39192.0","39191.9","39191.9","39191.9","0.18008008",10],[1647365880,"39186.1","39186.1","39172.0","39176.0","39174.4","0.13120077",10]],"last":1647408900}}
I've wrote these structs to help with unmarshalling
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
Time float64
Open string
High string
Low string
Close string
Vwa string
Volume string
Count float64
}
I have a function call that makes the http request and then unmarshals the data. For whatever reason, my code will end before even starting the function call for the http request and subsequent unmarshalling when the Pair type is []OHLC or []*OHLC. If I change the Pair type to interface{}, then it runs. i want to make it work with the OHLC struct instead though. Below is the complete code:
package main
import (
"fmt"
"net/http"
//"strings"
"io/ioutil"
"encoding/json"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main");
getOhlc()
}
func getOhlc() {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD");
if err != nil {
fmt.Errorf("error after request")
return;
}
defer resp.Body.Close();
body, err := ioutil.ReadAll(resp.Body);
if err != nil {
fmt.Errorf("error when reading")
return;
}
var jsonData Resp;
err = json.Unmarshal(body, &jsonData);
if err != nil {
fmt.Errorf("error when unmarshalling")
return
}
if(len(jsonData.Error) > 0) {
fmt.Errorf("error");
return;
}
fmt.Println(jsonData);
}
Any ideas about what might be happening?
"Any ideas about what might be happening?"
The elements in the "XXBTZUSD" JSON array are arrays themselves, i.e. "XXBTZUSD" is an array of arrays. The OHLC type is a struct type. The stdlib will not, by itself, unmarshal a JSON array into a Go struct. Go structs can be used to unmarshal JSON objects. JSON arrays can be unmarshaled into Go slices or arrays.
You would clearly see that that's the issue if you would just print the error from json.Unmarshal:
json: cannot unmarshal array into Go struct field
Trades.result.XXBTZUSD of type main.OHLC
https://go.dev/play/p/D4tjXZVzDI_w
If you want to unmarshal a JSON array into a Go struct you have to have the Go struct type implement a the json.Unmarshaler interface.
func (o *OHLC) UnmarshalJSON(data []byte) error {
// first unmarshal the array into a slice of raw json
raw := []json.RawMessage{}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
// create a function that unmarshals each raw json element into a field
unmarshalFields := func(raw []json.RawMessage, fields ...interface{}) error {
if len(raw) != len(fields) {
return errors.New("bad number of elements in json array")
}
for i := range raw {
if err := json.Unmarshal([]byte(raw[i]), fields[i]); err != nil {
return err
}
}
return nil
}
// call the function
return unmarshalFields(
raw,
&o.Time,
&o.Open,
&o.High,
&o.Low,
&o.Close,
&o.Vwa,
&o.Volume,
&o.Count,
)
}
https://go.dev/play/p/fkFKLkaNaSU
Your code had some issues:
Remove semicolons from end of lines, it's redundant.
fmt.Errorf return error, and not print it, every time check your error and propagate it.
We can convert array of numbers and string to struct in golang.
for achieving your desired output we need to first convert to intermediate container and then convert to our wanted output:
package main
import (
"errors"
"fmt"
"log"
"net/http"
//"strings"
"encoding/json"
"io/ioutil"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type IntermediateResp struct {
Error []string `json:"error"`
Result IntermediateTrades `json:"result"`
}
type IntermediateTrades struct {
Pair [][]interface{} `json:"XXBTZUSD"`
Last int `json:"last"`
}
type Trades struct {
Pair []OHLC `json:"result"`
Last int `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main")
err := getOhlc()
if err != nil {
log.Fatal(err)
}
}
func buildOHLC(l []interface{}) (*OHLC, error) {
if len(l) < 8 {
return nil, errors.New("short list")
}
return &OHLC{
TT: l[0].(float64),
Open: l[1].(string),
High: l[2].(string),
Low: l[3].(string),
Close: l[4].(string),
Vwap: l[5].(string),
Volume: l[6].(string),
Count: l[7].(float64),
}, nil
}
func convert(r IntermediateResp) (*Resp, error) {
result := &Resp{Error: r.Error, Result: Trades{Pair: make([]OHLC, len(r.Result.Pair)), Last: r.Result.Last}}
for i, v := range r.Result.Pair {
ohlc, err := buildOHLC(v)
if err != nil {
return nil, err
}
result.Result.Pair[i] = *ohlc
}
return result, nil
}
func getOhlc() error {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD")
if err != nil {
return fmt.Errorf("error after request, %v", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
if err != nil {
return fmt.Errorf("error when reading %v", err)
}
var jsonData IntermediateResp
err = json.Unmarshal(body, &jsonData)
if err != nil {
return fmt.Errorf("error when unmarshalling %v", err)
}
if len(jsonData.Error) > 0 {
return fmt.Errorf("error")
}
convertedOhlc, err := convert(jsonData)
if err != nil {
return fmt.Errorf("error when convertedOhlc %v", err)
}
fmt.Println(convertedOhlc)
return nil
}
We define IntermediateResp and IntermediateTrades for Unmarshaling json and then convert it to actual Resp.
I think aother way is using custom Unmarshal for Trades struct.

Having issues creating a struct for this JSON

I am very, very new to golang (using this project as a way to learn the language better). I have an api I'd like to pull data from, but I can't seem to create the right type of struct for it.
The full JSON I am trying to encode is located here. https://prices.runescape.wiki/api/v1/osrs/latest
The structure of the JSON is
{
"data": {
"2": {
"high": 182,
"highTime": 1621811749,
"low": 180,
"lowTime": 1621811755
},
"6": {
"high": 186683,
"highTime": 1621811083,
"low": 184528,
"lowTime": 1621811286
},
... REPEATS THOUSANDS OF TIMES ...
}
}
I have tried using the JSON to Go converter for the JSON data, but that returns a struct that is entirely too large. (Go returns an error) I have tried to manually make a struct with a slice as follows
type osrsPrices []struct {
ID struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
} `json:"id"`
}
Whenever I try to run this from the terminal I am presented with the error "json: cannot unmarshal object into Go value of type main.osrsPrices exit status 1"
Here is the entire code I am using
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type osrsPrices []struct {
ID struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
} `json:"id"`
}
func main() {
url := "https://prices.runescape.wiki/api/v1/osrs/latest"
spaceClient := http.Client{
Timeout: time.Second * 2, // Timeout after 2 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("User-Agent", "skillerscape:LearningGoLang|priceChecker")
res, getErr := spaceClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
items := osrsPrices{}
jsonErr := json.Unmarshal(body, &items)
if jsonErr != nil {
log.Fatal(jsonErr)
}
for _, item := range items {
fmt.Println(item)
}
}
I am pretty sure the error has to do with the way I am trying to process the data, but I cannot figure out the issue directly, nor the solution despite a couple hours of googling. I appreciate any help anyone has on this issue.
At the deepest level, you have:
type Price struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
}
This is in an object with variable keys, so you need:
type Data struct {
Data map[string]Price `json:"data"`
}
Then you can unmarshal into an instance of this struct:
var data Data
json.Unmarshal(input,&data)

How to save a JSON file into csv

I am trying to save this astros.json file into csv file.
Everything is working I don't have any error but my code is not going into my for loop.
So the astros.csv file is created successfully but there are no datas inside.
package main
import (
"encoding/json"
"encoding/csv"
"fmt"
"io/ioutil"
"os"
"net/http"
"strconv"
)
type People struct {
Name string
Craft string
}
type General struct {
//People []People
Number int
Message string
}
func main() {
// Reading data from JSON File
response, err := http.Get("http://api.open-notify.org/astros.json")
if err != nil {
fmt.Printf("The Http request failed with error %s\n", err)
}
data,_ := ioutil.ReadAll(response.Body)
//fmt.Println(string(data))
// Unmarshal JSON data
var general []General
json.Unmarshal([]byte(data), &general)
//fmt.Printf("First person: %s, Message: %s", general.People[0].Name, general.Message)
// Create a csv file
csvdatafile, err := os.Create("./astros.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
for _, obj := range general {
fmt.Println("Are you going into the for ?")
var record []string
//record = append(record, obj.People)
record = append(record, strconv.Itoa(obj.Number), obj.Message)
record = append(record, obj.Message)
//record = append(record, []string{People})
w.Write(record)
fmt.Println("Are you coming here")
record = nil
}
w.Flush()
fmt.Println("Appending succed")
}
It is important to always check for errors; if you change your code to:
err = json.Unmarshal([]byte(data), &general)
if err != nil {
fmt.Printf("Error in unmarshall %s\n", err)
return
}
Then you would see the problem; json.Unmarshal is returning cannot unmarshal object into Go value of type []main.General. This means that general is nil so your loop is never entered.
The reason for this is apparent when you compare the JSON:
{
"people": [{
"name": "Christina Koch",
"craft": "ISS"
}, {
"name": "Alexander Skvortsov",
"craft": "ISS"
}, {
"name": "Luca Parmitano",
"craft": "ISS"
}, {
"name": "Andrew Morgan",
"craft": "ISS"
}, {
"name": "Oleg Skripochka",
"craft": "ISS"
}, {
"name": "Jessica Meir",
"craft": "ISS"
}
],
"number": 6,
"message": "success"
}
With the variable you are unmarshalling into; []General where General is:
type General struct {
//People []People
Number int
Message string
}
It looks like you had the right idea at some point with the []People but then commented it out.
If you uncomment []People, unmarshal into a General (rather than []General) then make your loop for _, obj := range general.People { it should work as expected. This assumes that you want to iterate through the people; general.Message and general.Number are also available (but you dont need a loop to access these).
See this example (have removed file operation as the playground does not support that).
you could try something like this:
package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"net/http"
"os"
"strconv"
)
type People struct {
Name string
Craft string
}
type Response struct {
People []People
Number int
Message string
}
func main() {
// Reading data from JSON File
r, err := http.Get("http://api.open-notify.org/astros.json")
if err != nil {
fmt.Printf("The Http request failed with error %s\n", err)
}
var response Response
err = json.NewDecoder(r.Body).Decode(&response)
if err != nil {
fmt.Printf("decoding error%s\n", err)
return
}
// Create a csv file
csvdatafile, err := os.Create("./astros.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
for _, obj := range response.People {
fmt.Println("Are you going into the for ?")
var record []string
//record = append(record, obj.People)
record = append(record, strconv.Itoa(response.Number), response.Message)
record = append(record, obj.Name)
// record = append(record, []string{obj})
w.Write(record)
fmt.Println("Are you coming here")
record = nil
}
w.Flush()
fmt.Println("Appending succed")
}

Parse JSON HTTP response using golang

I am trying to get the value of say "ip" from my following curl output:
{
"type":"example",
"data":{
"name":"abc",
"labels":{
"key":"value"
}
},
"subsets":[
{
"addresses":[
{
"ip":"192.168.103.178"
}
],
"ports":[
{
"port":80
}
]
}
]
}
I have found many examples in the internet to parse json output of curl requests and I have written the following code, but that doesn't seem to return me the value of say "ip"
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type svc struct {
Ip string `json:"ip"`
}
func main() {
url := "http://myurl.com"
testClient := http.Client{
Timeout: time.Second * 2, // Maximum of 2 secs
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
res, getErr := testClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
svc1 := svc{}
jsonErr := json.Unmarshal(body, &svc1)
if jsonErr != nil {
log.Fatal(jsonErr)
}
fmt.Println(svc1.Ip)
}
I would appreciate if anyone could provide me hints on what I need to add to my code to get the value of say "ip".
You can create structs which reflect your json structure and then decode your json.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type Example struct {
Type string `json:"type,omitempty"`
Subsets []Subset `json:"subsets,omitempty"`
}
type Subset struct {
Addresses []Address `json:"addresses,omitempty"`
}
type Address struct {
IP string `json:"IP,omitempty"`
}
func main() {
m := []byte(`{"type":"example","data": {"name": "abc","labels": {"key": "value"}},"subsets": [{"addresses": [{"ip": "192.168.103.178"}],"ports": [{"port": 80}]}]}`)
r := bytes.NewReader(m)
decoder := json.NewDecoder(r)
val := &Example{}
err := decoder.Decode(val)
if err != nil {
log.Fatal(err)
}
// If you want to read a response body
// decoder := json.NewDecoder(res.Body)
// err := decoder.Decode(val)
// Subsets is a slice so you must loop over it
for _, s := range val.Subsets {
// within Subsets, address is also a slice
// then you can access each IP from type Address
for _, a := range s.Addresses {
fmt.Println(a.IP)
}
}
}
The output would be:
192.168.103.178
By decoding this to a struct, you can loop over any slice and not limit yourself to one IP
Example here:
https://play.golang.org/p/sWA9qBWljA
One approach is to unmarshal the JSON to a map, e.g. (assumes jsData contains JSON string)
obj := map[string]interface{}{}
if err := json.Unmarshal([]byte(jsData), &obj); err != nil {
log.Fatal(err)
}
Next, implement a function for searching the value associated with a key from the map recursively, e.g.
func find(obj interface{}, key string) (interface{}, bool) {
//if the argument is not a map, ignore it
mobj, ok := obj.(map[string]interface{})
if !ok {
return nil, false
}
for k, v := range mobj {
//key match, return value
if k == key {
return v, true
}
//if the value is a map, search recursively
if m, ok := v.(map[string]interface{}); ok {
if res, ok := find(m, key); ok {
return res, true
}
}
//if the value is an array, search recursively
//from each element
if va, ok := v.([]interface{}); ok {
for _, a := range va {
if res, ok := find(a, key); ok {
return res,true
}
}
}
}
//element not found
return nil,false
}
Note, that the above function return an interface{}. You need to convert it to appropriate type, e.g. using type switch:
if ip, ok := find(obj, "ip"); ok {
switch v := ip.(type) {
case string:
fmt.Printf("IP is a string -> %s\n", v)
case fmt.Stringer:
fmt.Printf("IP implements stringer interface -> %s\n", v.String())
case int:
default:
fmt.Printf("IP = %v, ok = %v\n", ip, ok)
}
}
A working example can be found at https://play.golang.org/p/O5NUi4J0iR
Typically in these situations you will see people describe all of these sub struct types. If you don't actually need to reuse the definition of any sub structs (like as a type for a function argument), then you don't need to define them. You can just use one definition for the whole response. In addition, in some cases you don't need to define a type at all, you can just do it at the time of declaration:
package main
import "encoding/json"
const s = `
{
"subsets": [
{
"addresses": [
{"ip": "192.168.103.178"}
]
}
]
}
`
func main() {
var svc struct {
Subsets []struct {
Addresses []struct { Ip string }
}
}
json.Unmarshal([]byte(s), &svc)
ip := svc.Subsets[0].Addresses[0].Ip
println(ip == "192.168.103.178")
}
You can write your own decoder or use existing third-party decoders.
For instance, github.com/buger/jsonparser could solve your problem by iterating throw array (two times).
package main
import (
"github.com/buger/jsonparser"
"fmt"
)
var data =[]byte(`{
"type":"example",
"data":{
"name":"abc",
"labels":{
"key":"value"
}
},
"subsets":[
{
"addresses":[
{
"ip":"192.168.103.178"
}
],
"ports":[
{
"port":80
}
]
}
]
}`)
func main() {
jsonparser.ArrayEach(data, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
jsonparser.ArrayEach(value, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
v, _, _, err := jsonparser.Get(value, "ip")
if err != nil {
return
}
fmt.Println("ip: ", string(v[:]))
}, "addresses")
}, "subsets")
}
Output: ip: 192.168.103.178

anonymous fields in JSON

I'm reverse engineering some JSON that seems to be using anonymous field names. For example:
{
"1": 123,
"2": 234,
"3": 345
}
BTW - it's not simply using "1" and "2" and "3" because they represent userids that are at a minimum int32's.
Is there some way such as using tags to properly Unmarshal the JSON?
I've tried:
package main
import (
"encoding/json"
"fmt"
)
type MyStruct struct {
string `json:",string"`
}
func main() {
jsonData := []byte("{\"1\":123,\"2\":234,\"3\":345}")
var decoded MyStruct
err := json.Unmarshal(jsonData, &decoded)
if err != nil {
panic(err)
}
fmt.Printf("decoded=%+v\n", decoded)
}
Just decode the data into a map (map[string]int):
jsonData := []byte("{\"1\":123,\"2\":234,\"3\":345}")
var decoded map[string]int
err := json.Unmarshal(jsonData, &decoded)
if err != nil {
panic(err)
}
You'll then be able to iterate over and access the elements by the user ID key:
for userID, _ := range decoded {
fmt.Printf("User ID: %s\n", userID)
}
https://play.golang.org/p/SJkpahGzJY