Parse structs from API - json

I am trying to access some information stored in a json file via Go. I have two related issues. One is that I'm not sure how to organize my structs and secondly how do I access them via a variable. I'll notate my code to make a little more sense
// To be clear, this is dummy info and I'm linting my actual json
// data. It loads fine, I just don't want to get hung up on this side
{
"A": {
"lob": "A",
"url": [
"example.com",
"test.com"]
}
"B": {
"lob": "B",
"url": [
"example2.com",
"test2.com"]
}
}
So the concern is that the structure of the options is identical. I am building this as part of a REST AP. The hope is that users can use http://testapi.com/getdata/A and it will return the urls and name info under A and likewise for B. As is, it loads both of them as separate components of the same struct:
type SiteList struct {
A struct {
Lob string `json:"lob"`
URL []string `json:"url"`
} `json:"test"`
B struct {
Lob string `json:"lob"`
URL []string `json:"url"`
} `json:"test2"`
}
I can do .A or .B by hand but I'm wondering how to handle it when the requests come in so that my API will only return the data under A or B.

If you're going to consume the API via accessing the API via http://testapi.com/getdata/A or http://testapi.com/getdata/B then A and B can be considered the parameters that drive the behavior of your API.
If you're passing A, you basically want to access the site data associated with A and if you're passing B, the site data for B should be returned.
An easy way to organize this data internally is to use a dedicated Go type site which holds Lob and URL and arrange everything in a map via map[string]site, which is initialized on startup of your server.
You can then dynamically access the parameter given to your API (A or B, but can be easily extended), lookup the site information from the map and, in case it's a valid site, return the corresponding data encoded as JSON.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
type site struct {
Lob string `json:"lob"`
URL []string `json:"url"`
}
var sites = map[string]site{
"A": site{
Lob: "A",
URL: []string{
"example.com",
"test.com",
},
},
"B": site{
Lob: "B",
URL: []string{
"example2.com",
"test2.com",
},
},
}
const endpoint = "/getdata/"
func handler(w http.ResponseWriter, r *http.Request) {
lob := r.URL.Path[len(endpoint):]
s, ok := sites[lob]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
resp, err := json.Marshal(s)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.Write(resp)
}
func main() {
http.HandleFunc(endpoint, handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}

Missing values in the json will be unmarshalled with their zero values. An int will be 0, a string will be "", a map and any pointer will be nil. Structs will be initialized, but their fields will have the zero value.
In your case, when comcast is missing in the json, B will be initialized as a struct, where Lob is "" and URL is an empty slice of string.

If I understand correctly you have a struct SiteList and depending on if they user navigates to /getData/A or /getData/B you want to serve SiteList.A or SiteList.B.
From the json marshal docs here or this SO answer you can leave out some fields of a struct when they are empty. Basically if there's no data in that field then it will not appear in the marshalled json.
The "omitempty" option specifies that the field should be omitted from
the encoding if the field has an empty value, defined as false, 0, a
nil pointer, a nil interface value, and any empty array, slice, map,
or string.
With that in mind, if you have the control in the handlers of the requests to be able to set the values in SiteList according to the path then you could make use of this feature.
If you extract the shared parts of A and B into a struct:
type Inner struct {
Lob string `json:"lob"`
URL []string `json:"url"`
}
// using pointers here for A and B means setting to nil is empty
type SiteList struct {
A *Inner `json:"test,omitempty"`
B *Inner `json:"test2,omitempty"`
}
And then set the one that you do not want to be in the response body to nil. (nil is empty for a pointer, so it will not get marshalled.)

Related

Unmarshalling Entities from multiple JSON arrays without using reflect or duplicating code

I'm making an JSON API wrapper client that needs to fetch paginated results, where the URL to the next page is provided by the previous page. To reduce code duplication for the 100+ entities that share the same response format, I would like to have a single client method that fetches and unmarshalls the different entities from all paginated pages.
My current approach in a simplified (pseudo) version (without errors etc):
type ListResponse struct {
Data struct {
Results []interface{} `json:"results"`
Next string `json:"__next"`
} `json:"d"`
}
func (c *Client) ListRequest(uri string) listResponse ListResponse {
// Do a http request to uri and get the body
body := []byte(`{ "d": { "__next": "URL", "results": []}}`)
json.NewDecoder(body).Decode(&listResponse)
}
func (c *Client) ListRequestAll(uri string, v interface{}) {
a := []interface{}
f := c.ListRequest(uri)
a = append(a, f.Data.Results...)
var next = f.Data.Next
for next != "" {
r := c.ListRequest(next)
a = append(a, r.Data.Results...)
next = r.Data.Next
}
b, _ := json.Marshal(a)
json.Unmarshal(b, v)
}
// Then in a method requesting all results for a single entity
var entities []Entity1
client.ListRequestAll("https://foo.bar/entities1.json", &entities)
// and somewehere else
var entities []Entity2
client.ListRequestAll("https://foo.bar/entities2.json", &entities)
The problem however is that this approach is inefficient and uses too much memory etc, ie first Unmarshalling in a general ListResponse with results as []interface{} (to see the next URL and concat the results into a single slice), then marshalling the []interface{} for unmarshalling it directly aftwards in the destination slice of []Entity1.
I might be able to use the reflect package to dynamically make new slices of these entities, directly unmarshal into them and concat/append them afterwards, however if I understand correctly I better not use reflect unless strictly necessary...
Take a look at the RawMessage type in the encoding/json package. It allows you to defer the decoding of json values until later. For example:
Results []json.RawMessage `json:"results"`
or even...
Results json.RawMessage `json:"results"`
Since json.RawMessage is just a slice of bytes this will be much more efficient then the intermediate []interface{} you are unmarshalling to.
As for the second part on how to assemble these into a single slice given multiple page reads you could punt that question to the caller by making the caller use a slice of slices type.
// Then in a method requesting all results for a single entity
var entityPages [][]Entity1
client.ListRequestAll("https://foo.bar/entities1.json", &entityPages)
This still has the unbounded memory consumption problem your general design has, however, since you have to load all of the pages / items at once. You might want to consider changing to an Open/Read abstraction like working with files. You'd have some Open method that returns another type that, like os.File, provides a method for reading a subset of data at a time, while internally requesting pages and buffering as needed.
Perhaps something like this (untested):
type PagedReader struct {
c *Client
buffer []json.RawMessage
next string
}
func (r *PagedReader) getPage() {
f := r.c.ListRequest(r.next)
r.next = f.Data.Next
r.buffer = append(r.buffer, f.Data.Results...)
}
func (r *PagedReader) ReadItems(output []interface{}) int {
for len(output) > len(buffer) && r.next != "" {
r.getPage()
}
n := 0
for i:=0;i<len(output)&&i< len(r.buffer);i++ {
json.Unmarshal(r.buffer[i], output[i] )
n++
}
r.buffer = r.buffer[n:]
return n
}

How to Change Field names of json object arrays

So I have a project with lots of incoming data about 15 sources in total, of course there are inconsistencies in how each label there data available in their rest api's. I need to Change some of their field names to be consistent with the others, but I am at a loss on how to do this when the data sources are json object arrays. A working example of what I am trying to do is found here playground and below
however I seem to lack the knowledge as to how to make this work when the data is not a single json object , but instead and array of objects that I am unmarshaling.
Another approach is using Maps like in this example but the result is the same, works great as is for single objects, but I can not seem to get it to work with json object arrays. Iteration through arrays is not a possibility as I am collecting about 8,000 records every few minutes.
package main
import (
"encoding/json"
"os"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
func NewCacheItem() (*CacheItem, error) {
i := &CacheItem{}
return i, json.Unmarshal([]byte(`{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}
}`), i)
}
func main() {
item, _ := NewCacheItem()
json.NewEncoder(os.Stdout).Encode(struct {
*CacheItem
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}{
CacheItem: item,
// Set the int by value:
MaxAge: item.MaxAge,
// Set the nested struct by reference, avoid making a copy:
Value: &item.Value,
})
}
It appears your desired output is JSON. You can accomplish the conversion by unmarshaling into a slice of structs, and then iterating through each of those to convert them to the second struct type (your anonymous struct above), append them into a slice and then marshal the slice back to JSON:
package main
import (
"fmt"
"encoding/json"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
type OutGoing struct {
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
Key string `json:"key"`
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}
func main() {
objects := make([]CacheItem, 0)
sample := []byte(`[
{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}},
{
"key": "baz",
"cacheAge": 123,
"cacheValue": {
"nested": true
}}]`)
json.Unmarshal(sample, &objects)
out := make([]OutGoing, 0, len(objects))
for _, o := range objects {
out = append(out, OutGoing{Key:o.Key, MaxAge:o.MaxAge, Value:&o.Value})
}
s, _ := json.Marshal(out)
fmt.Println(string(s))
}
This outputs
[{"key":"foo","max_age":1234,"value":{"nested":true}},{"key":"baz","max_age":123,"value":{"nested":true}}]
You could probably skip this iteration and conversion code if you wrote custom MarshalJSON and UnmarshalJSON methods for your CacheItem type, instead of relying on struct field tags. Then you could pass the same slice to both Unmarshal and Marshal.
To me there's no obvious performance mistake with these approaches -- contrast with building a string in a loop using the + operator -- and when that's the case it's often best to just get the software to work and then test for performance rather than ruling out a solution based on fears of performance issues without actually testing.
If there's a performance problem with the above approaches, and you really want to avoid marshal and unmarshal completely, you could look into byte replacement in the JSON data (e.g. regexp). I'm not recommending this approach, but if your changes are very simple and the inputs are very consistent it could work, and it would give another approach you could performance test, and then you could compare performance test results.

Converting from JSON to XML

I have seen lots of posts on here about converting from XML to JSON, and I've recently wrote a program to do so, but I was also curious how you would go about converting from JSON to XML?
Sample JSON:
"version":"0.1",
"termsofService":"http://www.wunderground.com/weather/api/d/terms.html",
"features": {
"conditions": 1
}
}
, "current_observation": {
"image": {
"url":"http://icons.wxug.com/graphics/wu2/logo_130x80.png",
"title":"Weather Underground",
"link":"http://www.wunderground.com"
},
"display_location": {
"full":"Kearney, MO",
"city":"Kearney",
"state":"MO",
"state_name":"Missouri",
I'm not sure if it'd be any use to you, but i'll post my JSON to XML program.
package main
import (
"fmt"
"net/url"
"encoding/xml"
"net/http"
"log"
"io/ioutil"
"encoding/json"
)
type reportType struct{
Version xml.CharData `xml:"version"`
TermsOfService xml.CharData `xml:"termsofService"
`
Features xml.CharData `xml:"features>feature"`
Full string `xml:"current_observation>display_location>full"`
StateName string `xml:"current_observation>display_location>state_name"`
WindGust string `xml:"current_observation>observation_location>full"`
Problem myErrorType `xml:"error"`
}
type myErrorType struct{
TypeOfError xml.CharData `xml:"type"`
Desciption xml.CharData `xml:"description"`
}
type reportTypeJson struct{
Version string `json:"version"`;
TermsOfService string `json:"termsofService"`;
Features map[string]string `json:"features"`;
CurrentObservation map[string]map[string]string `json:"current_observation"`
}
func main() {
fmt.Println("data is from WeatherUnderground.")
fmt.Println("https://www.wunderground.com/")
var state, city string
str1 := "What is your state?"
str2 := "What is your city?"
fmt.Println(str1)
fmt.Scanf("%s", &state)
fmt.Println(str2)
fmt.Scanf("%s", &city)
baseURL := "http://api.wunderground.com/api/";
apiKey := "Nunna"
var query string
//set up the query
query = baseURL+apiKey +
"/conditions/q/"+
url.QueryEscape(state)+ "/"+
url.QueryEscape(city)+ ".xml"
fmt.Println("The escaped query: "+query)
response, err := http.Get(query)
doErr(err, "After the GET")
var body []byte
body, err = ioutil.ReadAll(response.Body)
doErr(err, "After Readall")
fmt.Println(body);
fmt.Printf("The body: %s\n",body)
//Unmarshalling
var report reportType
xml.Unmarshal(body, &report)
fmt.Printf("The Report: %s\n", report)
fmt.Printf("The description is [%s]\n",report.Problem.Desciption)
//Now marshal the data out in JSON
var data []byte
var output reportTypeJson
output.Version = string(report.Version);
output.TermsOfService = string(report.TermsOfService)
output.Features= map[string]string{"feature":string(report.Features)} // allocate a map, add the 'features' value to it and assign it to output.Features
output.CurrentObservation = map[string]map[string]string {
"display_location": map[string]string {
"full": report.Full,
"state_name": report.StateName,
},"observation_location":map[string]string {"full": report.WindGust},
}
data,err = json.MarshalIndent(output,""," ")
doErr(err, "From marshalIndent")
fmt.Printf("JSON output nicely formatted: \n%s\n",data)
}
func doErr( err error, message string){
if err != nil{
log.Panicf("ERROR: %s %s \n", message, err.Error())
}
}
//OUTPUT:
//JSON output nicely formatted:
//{
// "version": "0.1",
// "termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
// "features": {
// "feature": "conditions"
// },
// "current_observation": {
// "display_location": {
// "full": "Kearney, MO",
// "state_name": "Missouri"
// },
// "observation_location": {
// "full": "HOMESTEAD HILLS, Kearney, Missouri"
// }
// }
//}
This is the same process as going in the other direction. Define structs/objects to model the input (be it json in this case), create method to assign all the values from the struct modeling the input to the one you're using for output, then marshal the output to get a string. So to give a practice example using one of the more conceptually difficult fields from you type, having unmarshalled some json into an instance of reportTypeJson I can assign to reportType used to model xml like so;
report.StateName = jsonReport.CurrentObservation["display_location"]["state_name"]
The biggest difference here is the struct representing your xml is flat (like it has Something>InnerSomething>InnerInnerSomething to represent it's nested values while the struct in golang has no nesting) where as with the json, your structures in golang tend to have the same amount of nesting (like having a map[string]map[string][string] meaning items are nested 3 levels inside of the main struct). You can observe this by the amount of indirection when you access fields, like in the example above, there is one level of indirection to access CurrentObservation but that is a map of maps, so then I index into it with the display_location key which yields a map[string]string, since I'm looking for the statename value, I have to index into that with state_name to access that value.
Note that in an actual program, a number of checks would be required because these operations are unsafe. For example, if the json read did not contain a display_location object then jsonReport.CurrentObservation["display_location"] would return nil and the attempt to access ["state_name" would result in a panic.
Also, another side note, in your program I would recommend adding two functions, one called NewJsonReport(report *reportType) *reportTypeJson, err and one called NewXmlReport(report *reportTypeJson) *reportType, err in which you initialize/allocate the a new instance of the return type and return it as to avoid code duplication and make your programs main more readable. Doing this type of assignment in more than one location in a program is sloppy coding which will add a lot of maintenance cost and is likely to result in bugs down the line (like if something changes on one of the models or the input you have to fix every reference to it throughout the program rather than just updating the functions I mentioned above).

XML into JSON Multiple Nesting

I'm attempting to write code to translate XML to JSON. The XML I'm trying to translate is as follows...
(Just a snippet)
`<version>0.1</version>
<termsofService>http://www.wunderground.com/weather/api/d/terms.html</termsofService>
<features>
<feature>conditions</feature>
</features>
<current_observation>
<image>
<url>http://icons.wxug.com/graphics/wu2/logo_130x80.png</url>
<title>Weather Underground</title>
<link>http://www.wunderground.com</link>
</image>
<display_location>
<full>Kearney, MO</full>
<city>Kearney</city>
<state>MO</state>
<state_name>Missouri</state_name>`
Current Code:
`package main
import (
"fmt"
"net/url"
"encoding/xml"
"net/http"
"log"
"io/ioutil"
"encoding/json"
)
type reportType struct{
Version xml.CharData `xml:"version"`
TermsOfService xml.CharData `xml:"termsofService"
`
Features xml.CharData `xml:"features>feature"`
Full xml.CharData `xml:"current_observation>display_location>full"`
StateName xml.CharData `xml:"current_observation>display_location>state_name"`
WindGust xml.CharData `xml:"current_observation>observation_location>full"`
Problem myErrorType `xml:"error"`
}
type myErrorType struct{
TypeOfError xml.CharData `xml:"type"`
Desciption xml.CharData `xml:"description"`
}
type reportTypeJson struct{
Version string `json:"version"`;
TermsOfService string `json:"termsofService"`;
Features map[string]string `json:"features"`;
Full map[string]string `json:"display_location"`;
WindGust map[string]string `json:"observation_location"`
}
func main() {
fmt.Println("data is from WeatherUnderground.")
fmt.Println("https://www.wunderground.com/")
var state, city string
str1 := "What is your state?"
str2 := "What is your city?"
fmt.Println(str1)
fmt.Scanf("%s", &state)
fmt.Println(str2)
fmt.Scanf("%s", &city)
baseURL := "http://api.wunderground.com/api/";
apiKey := "3hunna"
var query string
//set up the query
query = baseURL+apiKey +
"/conditions/q/"+
url.QueryEscape(state)+ "/"+
url.QueryEscape(city)+ ".xml"
fmt.Println("The escaped query: "+query)
response, err := http.Get(query)
doErr(err, "After the GET")
var body []byte
body, err = ioutil.ReadAll(response.Body)
doErr(err, "After Readall")
fmt.Println(body);
fmt.Printf("The body: %s\n",body)
//Unmarshalling
var report reportType
xml.Unmarshal(body, &report)
fmt.Printf("The Report: %s\n", report)
fmt.Printf("The description is [%s]\n",report.Problem.Desciption)
//Now marshal the data out in JSON
var data []byte
var output reportTypeJson
output.Version = string(report.Version);
output.TermsOfService = string(report.TermsOfService)
output.Features= map[string]string{"feature":string(report.Features)} // allocate a map, add the 'features' value to it and assign it to output.Features
output.Full=map[string]string{"full":string(report.Full),"state_name":string(report.StateName)}
output.WindGust=map[string]string{"full":string(report.WindGust)}
data,err = json.MarshalIndent(output,""," ")
doErr(err, "From marshalIndent")
fmt.Printf("JSON output nicely formatted: \n%s\n",data)
}
func doErr( err error, message string){
if err != nil{
log.Panicf("ERROR: %s %s \n", message, err.Error())
}
}
As you can see, I'm using maps to map one level nesting such as in the features case. But for in two level nesting cases such as xml:"current_observation>display_location>state_name", I can't figure out how to create the very first level, in this case current_observations. Would there be a way to somehow create a map of maps of sorts? Any and all ideas are much appreciated because I am very confused at the moment, Thanks for your time!
And the Output:
JSON output nicely formatted:
{
"version": "0.1",
"termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
"features": {
"feature": "conditions"
},
"display_location": {
"full": "Kearney, MO",
"state_name": "Missouri"
},
"observation_location": {
"full": "Stonecrest, Kearney, Missouri"
}
}
You could use either structs or a map of maps. I'll give some examples of both, starting with the map of maps. The type would be declares as;
CurrentObservation map[string]map[string]string `json:"current_observation"`
In this case you have a map with strings as the keys and the value is another map that has string for both key and value. As a result when you marshal your json you will end up with something like;
"current_observation" {
"image": { // first level where the value is a map[string]string
"title":"Weather Underground" // inner map, string to string
}
}
If say you wanted to just print the title, you would do;
fmt.Println(reportJson.CurrentObservation["image"]["title"])
Since the data there looks fairly static you could also use structs instead. In which case you'd use something like this;
CurrentObservation CurrentObservation `json:"current_observation"`
type CurrentObservation struct {
Image Image `json:"image"`
}
type Image struct {
Url string `json:"url"`
Title string `json:"title"`
Link string `json:"link"`
}
Both options produce the same output though they could behave differently for different inputs. For example, if another version of current_observation were received as input that for example has another nested item in it call it... previous_observation then the map option would automatically unmarhsal this data as well where the struct options would exclude it since there would be no mapping to any object/type in Go.
Personally I prefer the struct route when possible but it varies from case to case. For your application the map is likely better since you're not working with the input (it comes in as xml) and you just want to print it, you don't really have to deal with the details of current_observation, if it has 3 objects inside it, they'll all be output as expected, if it were 5 it would be the same. With the structs you have to explicitly define every single field which isn't really necessary if you're just transforming the input. The advantage of the struct is more for use later on where you have type safety although in this case, I would say they're still fairly equivalent because for example anytime you want to access something in image, like CurrentObservation.Image.Title you'll have to perform a check to ensure Image is not nil, like;
if CurrentObservation.Image != nil {
fmt.Println(CurrentObservation.Image.Title)
}
With the map you basically have the same overhead, only you're checking for the existence of a key rather than checking if one of the inner structs is nil or not.
EDIT: example of initializing the map of maps using composite literal syntax;
reportJson.CurrentObservation := map[string]map[string]string {
"display_location": map[string]string {
"full": report.Full,
"state_name": report.StateName,
},
}

How to access fields of a JSON in GO

Hi everyone I'm trying to see what the proper way of accessing fields of a json object from a http.get request in go.
I first do an http.get call get the JSON and then print it (which works) but is there a way to access just a field?
for example:
response, err:= http.Get("URL")
//Error checking is done between
contents, err:=ioutil.Readall(response.Body)
//Now at this point I have a json that looks like
{"id": "someID",
"name": "someName",
"test": [{"Name":"Test1",
"Result": "Success"},
{"Name":"Test2",
"Result": "Success"},
{...},
]}
Is there a way to only print the "test" of the Json? What is the proper way of accessing that field?
Use encoding/json package to Unmarshal data into struct, like following.
type Result struct {
ID string `json:"id"`
Name string `json:"name"`
Test []interface{} `json:"test"`
}
var result Result
json.Unmarshal(contents, &result)
fmt.Println(result.Test)
You can also parse Test to specific struct.
Same as the previous answer, use encoding/json package to Unmarshal data. But if you don't want to specify the structure, you could use map[string]interface/bson.M{} to receive the data, and get the field, then cast into types your want.
m := make(map[string]interface{})
err := json.Unmarshal(data, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m["id"])
You may want to try gabs container, if you are not sure how depth JSON hierarchy can be. Have a look at below resources
https://github.com/Jeffail/gabs
https://godoc.org/github.com/Jeffail/gabs
If you just want to access one field then you can use the jsonq module https://godoc.org/github.com/jmoiron/jsonq
For your example you could get the test object with code similar to
jq.Object("test")
Where jq is a jsonq query object constructed from your JSON above (see the godoc page for instructions on how to create a query object from a JSON stream or string).
You can also use this library for retrieving specific String, Integer, Float and Bool values at an arbitrary depth inside a JSON object.
Since you are starting with a URL, Decode is a better option than Unmarshal:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode
You may check this https://github.com/valyala/fastjson
s := []byte(`{"foo": [123, "bar"]}`)
fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))
// Output:
// foo.0=123