Making a struct before the unmarshal in golang - json

Im new in golang and i have a question.
I have 5 structs where i use Json , but the JSON file can have more structs than the ones i have predetermined BUT... the structures of the JSON satisfies the structures in my programing ( lets say i have 5 structs "struct1" , 6 structs "struct 2" , 1 struct "struct 3", and so on...)
My question is , i want to make a function were i take the JSON FILE , read the structs of it and have as an output the number of structs of the JSON file.
I think i could use the map[string]interface{} but i dont understand it.
I hope i have explain myself
Thank u very much!

Without example JSON or structs, the exact question you are asking is a bit hard to decipher, specifically the "output the number of structs" bit in the question, as it could be interpreted several different ways. I will do my best to answer what I think are the most probable questions you are asking.
Interfaces
First off, some basic go knowledge that might be useful, but outside JSON marshaling itself. The interface{} type appears special, but is not a hardwired keyword as it first might appear. What the interface keyword does is describe the requirements that an object must have to fulfill that interface. Because interface{} has no requirements, and because everything is automatically interfaced in go, everything satisfies the interface{} type.
Because of this implementation of interfaces, map[string]interface{} is really map[string]. This allows for the JSON un/marshal to not care about what is on the value side of the map. This exactly lines up with the format of JSON, where you have a string key on one side, and a value that could be any of the JSON datatypes on the other.
How many different objects are in the base JSON object?
let us take an example json
{
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
"text": {
"data": "Click Here",
"size": 36,
"style": "bold",
"name": "text1",
"hOffset": 250,
"vOffset": 100,
"alignment": "center",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
}
}
The answer to the question in this circumstance would be four. debug, window, image, and text.
The process for determining the number would then be:
Load the json into a byte array.
Marshal into an interface{}
Determine type (array vs object etc.) using type switch. see this A Tour of Go page
If you know the type, you can skip this step
Convert to desired type
Get length, or perform any other operation as desired.
package main
import (
"encoding/json"
"fmt"
)
func main() {
myJSON := `<see above>`
var outStruct *interface{}
json.Unmarshal([]byte(myJSON), &outStruct)
outMap := (*outStruct).(map[string]interface{})
fmt.Printf("Num Structs: %d", len(outMap))
}
Go Playground
How many json objects that I do not have structs for are present?
This answer has a very similar answer to the first one, and is really about manipulation of the output map and the struct
Taking almost the entire code from the first one to the second, let us assume that you have the following structs set up
type Image struct {
Name string
//etc
}
type Text struct {
Name string
//etc
}
type Window struct {
Name string
//etc
}
type Base struct {
Image Image
Window Window
Text Text
}
In this case, in addition to the previous steps, you would have to
5. Marshal the json into a base object
6. Go through the map[string]interface{}, and for each key
7. Determine if the key is one of the objects in your base struct
total := 0
for k, _ := range outMap {
if k != "image" && k != "text" && k != "window" && k != "other" {
total++
}
}
fmt.Printf("Total unknown structs: %d\n", total)
How many of my structs are empty?
This last question is also rather simple, and could be done by checking the map for a value given the input key, but for completion's sake, the example code marshals the JSON into a struct, and uses that.
Marshal JSON into base
For each of Window, Item, Text, Other in base, determine if empty.
total = 0
if (Image{}) == outBase.Image {
total++
}
if (Window{}) == outBase.Window {
total++
}
if (Text{}) == outBase.Text {
total++
}
if (Other{}) == outBase.Other {
total++
}
fmt.Printf("Total empty structs: %d\n", total)
Go Playground
See this go blog post for more information on golang JSON.

Related

How do I pass through values from an Elasticsearch document in Go?

I'm experimenting with rewriting parts of our system in Go. They're currently written in Python. Some of the data that I want to serve lives in Elasticsearch.
Across our users, we have a few standard fields, but also allow people to create a number of custom fields specific to their environment. E.g., we have a product object that has some common fields like name and price, but we let someone create a field like discount_price or requires_freight to agree with their use case.
In Python, this is easy to accommodate. The JSON is read in, our chosen JSON parser does some reasonable type inference, and we can then return the data after it's processed.
In Go, any data we want to deal with from the Elasticsearch JSON response has to be mapped to a type. Or at least that's my understanding. For example:
import (
"encoding/json"
)
...
type Product struct {
Name string `json:"name"`
Price string `json:"price"`
}
Here's a simplified example of what the data might look like. I've prefixed the names of the nonstandard fields I'd want to pass through with custom:
{
"id": "ABC123",
"name": "Great Product",
"price": 10.99,
"custom_alternate_names": ["Great Product"],
"custom_sellers": [{
"id": "ABC123",
"name": "Great Product LLC",
"emails": ["great#stuff.com"]
}]
}
It would be fine to special case for places where the route actually needs to process or manipulate a custom field. But in most cases, passing through the JSON data unchanged is fine, so the fact we aren't imposing any type mappings for safety isn't adding anything.
Is there a way to set up the struct with an interface (?) that could act as a passthrough for any unmapped fields? Or a way to take the unparsed JSON data and recombine it with the mapped object before the data are returned?
You can do something like this
package main
import (
"encoding/json"
"log"
)
type Product struct {
//embed product to be able to pull the properties without
//requiring a nested object in the json
KnownFields
OtherStuff json.RawMessage
}
//Custom unmarshaller
func (p *Product) UnmarshalJSON(b []byte) error {
var k KnownFields
//unmarshal the known fields
err := json.Unmarshal(b, &k)
if err != nil {
return err
}
p.KnownFields = k
//You can use json.RawMessage or map[string]interface
p.OtherStuff = json.RawMessage(b)
return nil
}
type KnownFields struct {
Name string `json:"name"`
Price json.Number `json:"price"`
}
const JSON = `
{
"id": "ABC123",
"name": "Great Product",
"price": 10.99,
"custom_alternate_names": ["Great Product"],
"custom_sellers": [{
"id": "ABC123",
"name": "Great Product LLC",
"emails": ["great#stuff.com"]
}]
}`
func main() {
var p Product
err := json.Unmarshal([]byte(JSON), &p)
if err != nil {
log.Panic(err)
}
log.Printf("%v", p)
}
If you are going to mutate and marshall the Product you will have to implement a custom marshaller and also you would need to use map[string]interface{} as the OtherStuff to avoid having duplicate entries for the known fields

How to obtain same hash from json

I need to sign a JSON but I noticed that unmarshaling/marshaling can change JSON's order which might make the signature invalid.
Is there anyway to produce the same hash from a JSON string despite its order?
I've had a look at JOSE but couldn't find the function that actually hashes JSON.
JOSE JWS will absolutely do what you want at the cost of having to manage keys for signatures and verification.
But let's assume that you don't really need the whole key management stuff and general crypto functionality in JOSE and you're not SUPER concerned about performance (so a little string mangling in this process is OK).
You could dumbly unmarshal your JSON and re-marshal it, then just hash that:
package main
import (
"crypto/sha256"
"encoding/hex"
"fmt"
json "encoding/json"
)
// NB These docs are strictly-speaking the same.
const DOCA = "{ \"foo\": 1.23e1, \"bar\": { \"baz\": true, \"abc\": 12 } }"
const DOCB = "{ \"bar\": { \"abc\": 12, \"baz\": true }, \"foo\": 12.3 }"
func hash(doc string) string {
// Dumb af, but it's a cheap way to specific the most generic thing
// you can :-/
var v interface{}
json.Unmarshal([]byte(doc), &v) // NB: You should handle errors :-/
cdoc, _ := json.Marshal(v)
sum := sha256.Sum256(cdoc)
return hex.EncodeToString(sum[0:])
}
func main() {
fmt.Println(DOCA)
fmt.Printf("Hash: %s\n", hash(DOCA))
fmt.Println(DOCB)
fmt.Printf("Hash: %s\n", hash(DOCB))
}
The output of this program (at least in the golang docker container) is:
{ "foo": 1.23e1, "bar": { "baz": true, "abc": 12 } }
Hash: d50756fbb830f8335187a3f427603944c566772365d8d8e6f6760cd2868c8a73
{ "bar": { "abc": 12, "baz": true }, "foo": 12.3 }
Hash: d50756fbb830f8335187a3f427603944c566772365d8d8e6f6760cd2868c8a73
The nice thing about this approach is that, for the cost of some performance, you get insulated from whatever dumb junk you did while marshalling your JSON in the first place (so, unlike other suggestions, you don't have to think about what you might be doing with custom Marshallers and whatnot). This is especially a big deal when you forget that this was an issue at all in version 3.8 of your code a year from now, implement something that messes with the marshal order, and start breaking things.
And, of course, you could always add the hash to the resulting struct and marshal again with the extra item in the map. Obviously you want to optimize a bit for performance if you're worried about it at all and properly handle errors, but this is a good prototype anyway :-)
Oh, and if you're super-worried about edge cases biting you, you could also use canonical JSON to marshal, since it's specifically designed for this type of use (though, honestly, I couldn't come up with an example in my testing where c-json worked but go's default json didn't).

How to Change Field names of json object arrays

So I have a project with lots of incoming data about 15 sources in total, of course there are inconsistencies in how each label there data available in their rest api's. I need to Change some of their field names to be consistent with the others, but I am at a loss on how to do this when the data sources are json object arrays. A working example of what I am trying to do is found here playground and below
however I seem to lack the knowledge as to how to make this work when the data is not a single json object , but instead and array of objects that I am unmarshaling.
Another approach is using Maps like in this example but the result is the same, works great as is for single objects, but I can not seem to get it to work with json object arrays. Iteration through arrays is not a possibility as I am collecting about 8,000 records every few minutes.
package main
import (
"encoding/json"
"os"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
func NewCacheItem() (*CacheItem, error) {
i := &CacheItem{}
return i, json.Unmarshal([]byte(`{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}
}`), i)
}
func main() {
item, _ := NewCacheItem()
json.NewEncoder(os.Stdout).Encode(struct {
*CacheItem
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}{
CacheItem: item,
// Set the int by value:
MaxAge: item.MaxAge,
// Set the nested struct by reference, avoid making a copy:
Value: &item.Value,
})
}
It appears your desired output is JSON. You can accomplish the conversion by unmarshaling into a slice of structs, and then iterating through each of those to convert them to the second struct type (your anonymous struct above), append them into a slice and then marshal the slice back to JSON:
package main
import (
"fmt"
"encoding/json"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
type OutGoing struct {
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
Key string `json:"key"`
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}
func main() {
objects := make([]CacheItem, 0)
sample := []byte(`[
{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}},
{
"key": "baz",
"cacheAge": 123,
"cacheValue": {
"nested": true
}}]`)
json.Unmarshal(sample, &objects)
out := make([]OutGoing, 0, len(objects))
for _, o := range objects {
out = append(out, OutGoing{Key:o.Key, MaxAge:o.MaxAge, Value:&o.Value})
}
s, _ := json.Marshal(out)
fmt.Println(string(s))
}
This outputs
[{"key":"foo","max_age":1234,"value":{"nested":true}},{"key":"baz","max_age":123,"value":{"nested":true}}]
You could probably skip this iteration and conversion code if you wrote custom MarshalJSON and UnmarshalJSON methods for your CacheItem type, instead of relying on struct field tags. Then you could pass the same slice to both Unmarshal and Marshal.
To me there's no obvious performance mistake with these approaches -- contrast with building a string in a loop using the + operator -- and when that's the case it's often best to just get the software to work and then test for performance rather than ruling out a solution based on fears of performance issues without actually testing.
If there's a performance problem with the above approaches, and you really want to avoid marshal and unmarshal completely, you could look into byte replacement in the JSON data (e.g. regexp). I'm not recommending this approach, but if your changes are very simple and the inputs are very consistent it could work, and it would give another approach you could performance test, and then you could compare performance test results.

Using struct with API call in Go

I'm new to Go and working hard to follow its style and I'm not sure how to proceed.
I want to push a JSON object to a Geckoboard leaderboard, which I believe requires the following format based on the API doc and the one for leaderboards specifically:
{
"api_key": "222f66ab58130a8ece8ccd7be57f12e2",
"data": {
"item": [
{ "label": "Bob", "value": 4, "previous_value": 6 },
{ "label": "Alice", "value": 3, "previous_value": 4 }
]
}
}
My instinct is to build a struct for the API call itself and another called Contestants, which will be nested under item. In order to use json.Marshall(Contestant1), the naming convention of my variables would not meet fmt's expectations:
// Contestant structure to nest into the API call
type Contestant struct {
label string
value int8
previous_rank int8
}
This feels incorrect. How should I configure my Contestant objects and be able to marshall them into JSON without breaking convention?
To output a proper JSON object from a structure, you have to export the fields of this structure. To do it, just capitalize the first letter of the field.
Then you can add some kind of annotations, to tell your program how to name your JSON fields :
type Contestant struct {
Label string `json:"label"`
Value int8 `json:"value"`
PreviousRank int8 `json:"previous_rank"`
}

Decoding incoming JSON in Golang based on param values

I am trying to decode an incoming JSON in my REST API written in Go. I am using decoder.Decode() function and my problem is that I need to apply a certain rules on which struct should be used in the process of decoding because sometimes the JSON contains:
"type": {
"type" : "string",
"maxLength" : 30
},
and sometimes:
"type": {
"type" : "integer",
"max" : 30,
"min" : 10
},
I somehow need to tell Go that "If the type.type is string, use this struct (type Type_String struct) and if the type.type is integer, use other struct (type Type_Integer struct)". I am not really sure how to do it. One solution which is on my mind is to make an universal struct with the all possible properties, use it on any kind of object and then filter the properties based on the type property but this is just so dirty. I guess I can also write my own decoder but that seems also a bit strange.
I am new to Go and I am pretty much used to the freedom JavaScript offers.
First of all, if fields of "type" depends on "type.type", in my opinion, it's better to move it one level up. Something like:
...
"type" : "integer",
"intOptions": {
"max" : 30,
"min" : 10
},
....
Then you can create a struct with only one field:
type Type struct {
Type string
}
and do something like:
myType := new(Type)
json.Unmarshal([]byte(yourJsonString), myType)
And now, depending on myType's value you can use different structs for decoding your json.
You can always decode to interface{} like mentioned here: How to access interface fields on json decode?
http://play.golang.org/p/3z8-unhsH4
package main
import (
"encoding/json"
"fmt"
)
var one string = `{"type": {"type": "string", "maxLength":30}}`
var two string = `{"type": {"type": "integer", "max":30, "min":10}}`
func f(data map[string]interface{}) {
t := data["type"]
typemap := t.(map[string]interface{})
t2 := typemap["type"].(string)
switch t2 {
case "string":
fmt.Println("maxlength:", typemap["maxLength"].(float64))
case "integer":
fmt.Println("max:", typemap["max"].(float64))
default:
panic("oh no!")
}
}
func main() {
var jsonR map[string]interface{}
err := json.Unmarshal([]byte(one), &jsonR)
if err != nil {
panic(err)
}
f(jsonR)
json.Unmarshal([]byte(two), &jsonR)
f(jsonR)
}
The idea is to unmarshal to map[string]interface{} and then cast and compare before accessing values.
In the above code, the f function does the cast and compare. Given this poor json, I used poor variable name, t and t2 to represent the json values of "type" at different depths. Once t2 has the value, the switch statement does something with the "string" or the "integer" and what it does is print the maxLength or the max value.