I'm working on unmarshaling some nested json data that I have already written a struct for. I've used a tool that will generate a struct based off json data, but am a bit confused how to work with accessing nested json data (and fields can sometimes be emtpy).
Here is an example of struct:
type SomeJson struct {
status string `json:"status"`
message string `json:"message"`
someMoreData []struct {
constant bool `json:"constant,omitempty"`
inputs []struct {
name string `json:"name"`
type string `json:"type"`
} `json:"inputs,omitempty"`
Name string `json:"name,omitempty"`
Outputs []struct {
Name string `json:"name"`
Type string `json:"type"`
} `json:"outputs,omitempty"`
I'm able to unmarshal the json data and access the top level fields (such as status and message), but am having trouble accessing any data under the someMoreData field. I understand this field is a (I assume an unknown) map of structs, but only have experience working with basic single level json blobs.
For reference this is the code I have to unmarshal the json and am able to access the top level fields.
someData := someJson{}
json.Unmarshal(body, &someData)
So what is exactly the best to access some nested fields such as inputs.name or outputs.name?
to iterate over your particular struct you can use:
for _, md := range someData.someMoreData {
println(md.Name)
for _, out := range md.Outputs {
println(out.Name, out.Type)
}
}
to access specific field:
someData.someMoreData[0].Outputs[0].Name
Couple of things to note:
The struct definition is syntactically incorrect. There are couple of closing braces missing.
type is a keyword.
The status and message and other fields with lower case first letter fields are unexported. So, the Json parser will not throw error, but you will get zero values as output. Not sure that's what you observed.
someMoreData is an array of structs not map.
I am using a JSON API for which I have to parse it into a struct. However, the API returns all values, even numbers, as strings and I need them to be in the format of numbers. So currently, I have a struct which has member fields which are all strings and after I have parsed the data, I loop through the entries to convert the values and add them to a new struct which has the specific entries in float or int values.
Is there any way to do the parsing and do type conversion in one go without having to use an intermediary struct representation from which to convert the values into the desired data types?
Example Code
str := []byte(`
{
"name": "Jim Burnham",
"age": "34",
"dob_day": "22",
"dob_month": "3",
"dob_year": "1984"
}
`)
Here is a sample JSON declaration of a response from an API. Notice how the age, day, month and year are returned as strings rather than integers. Now I declare a struct with the desired fields with JSON tags to map the values correctly:
type person struct {
Name string `json:"name"`
Age int `json:"age"`
DobDay int `json:"dob_day"`
DobMonth int `json:"dob_month"`
DobYear int `json:"dob_year"`
}
Then I declare an instance of the person struct and use the json package to unmarshal it into the instance of the struct:
var p person
_ = json.Unmarshal(str, &p)
fmt.Println(p)
But when I print out the person, the following output is generated:
{Jim Burnham 0 0 0 0}
As you can see, the string has been parsed correctly but the other integer fields remain at their default Golang initialized value. However, when I change the struct definition to :
type person struct {
Name string `json:"name"`
Age string `json:"age"`
DobDay string `json:"dob_day"`
DobMonth string `json:"dob_month"`
DobYear string `json:"dob_year"`
}
I get the following output:
{Jim Burnham 34 22 3 1984}
This means that currently, I have to define a raw struct which parses the information in the format of a string but then define another struct with the desired dtypes and reassign and convert the values separately, which produces untidy code as well. However, this is just one case but in my use case, there are likely thousands or even sometimes millions of such values and it seems to be inefficient, even for a compiled language. This is why I am asking for solutions for such a problem.
As explained well by #mkopriva at this link: https://play.golang.org/p/klHYlMQyb_V
I'm trying to parse part of the Mailgun notify webhook.
It's a POST request with a x-www-form-urlencoded body.
Here's part of the body:
sender: some#email.com
attachments: [{"url": "https://storage.eu.mailgun.net/v3/domains/beep.boop/messages/randomstring/attachments/0", "content-type": "application/pdf", "name": "example.pdf", "size": 345}]"]
The attachments value is a json encoded array
I would like to decode this string from JSON to a StoredAttachment nested struct as I am decoding the response as x-www-form-urlencoded but I don't know how to do it. The target structs are as follow:
type NotifiedMessage struct {
Sender string `schema:"sender"`
Subject string `schema:"subject"`
Attachments []StoredAttachment `schema:"attachments"`
MessageUrl string `schema:"message-url"`
}
// StoredAttachment structures contain information on an attachment associated with a stored message.
type StoredAttachment struct {
Size int `json:"size"`
Url string `json:"url"`
Name string `json:"name"`
ContentType string `json:"content-type"`
}
Here is my non-working code so far: https://play.golang.org/p/Ofbw2VAYV28
You can implement the TextUnmarshaler interface, the schema package will use that interface instead of executing the default procedure and this allows for custom unmarshaling.
1. Declare a named type and use that as the type of the Attachments field. []StoredAttachment is unnamed. So, for example:
type AttachmentList []StoredAttachment
Why? Because methods can be declared only on named types.
2. Implement TextUnmarhsaler interface and do the json unmashaling there.
func (ls *AttachmentList) UnmarshalText(text []byte) (err error) {
return json.Unmarshal(text, (*[]StoredAttachment)(ls))
}
That's it.
https://play.golang.org/p/t65mI7JRFfS
I'm trying to extend the AWS S3 Bucket type to include an additional format and marshal it as JSON, but the marshalling won't pick up the additional field
This is what I have
// AWS has this struct already
type Bucket struct {
// Date the bucket was created.
CreationDate *time.Time `type:"timestamp"
timestampFormat:"iso8601"`
// The name of the bucket.
Name *string `type:"string"`
// contains filtered or unexported fields
}
// Extended struct
type AWSS3Bucket struct {
s3.Bucket
location string
}
somefunc()
{
var region string = "us-west-1"
aws_s3_bucket := AWSS3Bucket{Bucket:*bucket, location:region}
jsonString, err := json.Marshal(&aws_s3_bucket)
fmt.Printf("%s\n", jsonString)
}
What I get is just the encoding of Bucket. For e.g., my output from above is always like this without the region included
{"CreationDate":"2016-10-17T22:33:14Z","Name":"test-bucket"}
Any ideas why the region is not being marshalled into the json buffer ?
The location field of AWSS3Bucket is not exported (i.e. it doesn't begin with an upper case letter) so the json package cannot find it using reflection. If you export the field:
type AWSS3Bucket struct {
s3.Bucket
Location string
}
then it will show up in jsonString. If you want it to show up as "location":... in the JSON then tag it as such:
type AWSS3Bucket struct {
s3.Bucket
Location string `json:"location"`
}
I can't decode the json code below ... any ideas why it doesn't work ? See also play.golang
package main
import (
"encoding/json"
)
type LocationReadable struct {
District string
City string
State string
}
type Locale struct {
Location string
CountryCode string
CurrencyId string
CurrencySymbol string
LocationReadable LocationReadable
}
type Media struct {
Image string
Video string
}
type Variations struct {
FixedPrice float64
Media Media
Quantity int
}
type PaymentData struct {
PaymentName string
PaymentService string
}
type Payment struct {
Online PaymentData
Offline PaymentData
}
type Shipping struct {
ShippingService string
ShippingName string
ShippingCost float64
HandlingTimeMax int
DispatchTimeMin int
DispatchTimeMax int
ShippingAdditionalCost int
}
type Item []struct {
_version string
CategoryId string
Title string
Media Media
SellerId string
Locale Locale
ListingType string
Payment Payment
StartTime string
EndTime string
Shipping Shipping
TitleSlug string
Variations Variations
_fpaiStatus string
}
func main() {
itemInfoR := `{"locale":{"location":"51.51121389999999,-0.11982439999997041","countryCode":"GB","currencyId":"GBP","currencySymbol":"£","locationReadable":{"district":"City of Westminster","city":"London","state":"Greater London"}},"_version":"serving","categoryId":["Root","Cameras \u0026 Photo","Digital Cameras"],"title":"many pictures","media":{"image":["//lh5.ggpht.com/O_o_N6CFkClY5AV0-LqntpyFjor7Of4u23ZcK7lYwc2uY1ea7GWi61VDJZCB7UCb79svkjKPHIenqwEUhjHi0jdIQnnl6z_p03yktPUB1FBHezIQ","//lh6.ggpht.com/ih3q2d7CenGLPyupH9FpfsoJQWQpw1i8wWA2Kd26bFnSF2fbnKyGU9WePIhCgEeqw5p6YMVmFi1c9oS0Ag93aF_oZ3ZiwK7fQuSYIrZ9VhgXbrTHkw","//lh6.ggpht.com/7RJRsapsnwWL3_KiLIjMz4QojDzUvsztXtvKTFvIfde_AHccDnOibAvXRN73tTB4SeHzlj8S1LWxbYwwWFGn9elfCKdSb8BUIU5QJY1LO791HutQ","//lh6.ggpht.com/qAtjgyHAB734Ox_4NC_fa-ZRqrCjCmJu0Tp8bo-HMO88duv8l4hhuv2REBkB--yneFzOL7annecVlGty-YsKouondiOFVnAZWzjpdrfsGfbL6wh2","//lh3.ggpht.com/dWUbASepwHF4lHaXIPnpv4BNm2pCml9MlJt7s86s1cpu-PsYNmS0yQmKFKTM38q_oMLW_YJMJ19civ2gVViKAGYcZylRW7jN3w77AJvhzS6JE2g","//lh6.ggpht.com/9aXLmPRVeZnxkwvNb3mWTF8kvfEY_lho_lOVVc9AbNqLb8GQmiS_XXVZ3OKqMv2pxgYSayMYPPRh6ACYyh0H8KtS8mPD6MKUkEajwxkTtp5Q4Lo","//lh3.ggpht.com/FG_QXZPHJ2tTYwI_t5Fg1KqivglVg9RlJn0JRsu9Ox8vJ7IcBirb2IV_I1LL_WVOMxfTuBBSDLMlrw9v0MCAdmnPCR29sCbRGjhm6zEfIH-3q2QSdw","//lh4.ggpht.com/Y23DqORrVkM2m55f-rq5_BBrlkvQg4uX7AsAt-ixhMobjK_SFgFaDfktgLhkNsyKwSr9HcF8iiGY3Nw0xOKXG1sn6wyAWg_qsolmKjVOrM5V5mIR","//lh6.ggpht.com/mQ62Ly-DjMKPMzU1OcSPJ7SLBqym0uBjawlkTHfmb-HOKaD56dnitk1duwPFJVdbi0GUpd63RQvr2VMpHp6S1OQ3di-hq4-JPeRoS5FJzksXSvW_","//lh3.ggpht.com/dqWjWPcNsvlR1tMC_agizX19f9MDiNGWFYTYVn4kjJxzIIkEe0mLzNcvS62zVJxAOaitT-IgaUfZ-Ze23BgzbqYY-l600i_LbVe35Uinz6sXIyoB","//lh6.ggpht.com/xhSdFc9uHgghs_6gf3seUWYM-PG2oLmjTrpF7ptEEMqaIrQIa8VPfC6tXE7f3M13eZvDXYqMW_k0AHO5vwCEPNp-iObixskd_lBaKNfz3MH3SNQ","//lh5.ggpht.com/kYeoKPoZGJCow-G1FhnD8kzVjNjbQA8-Kyj8eAh0HL-fMZX9tTeFPQikTZdSU0kks4-5Ui54cZF2CjGut9vfMJAVDKIq3T-bAQewCxvfl2120tH5zQ","//lh5.ggpht.com/4qUl3d-G9EPBzcYKrimNsWhQw7CmONV0jgfVhxFgB9mEU_QLRCyNJTWs2A3xf6wc7AUF2DXrKEkoX-SNLMZ6s-O4aXXV9WOjOPcWdAYreMRBld0E","//lh5.ggpht.com/z-0C4G6EWYkelAF1LjPfl_UQcsp92H4joIPt8NfsOl0nPJ2VpzZYahWadKqTLfl6kq3C6aDBcwfGQyMWSozYoZIAOAW0yRvZrwxia321PlsKTxbZ","//lh4.ggpht.com/U7I12JrDYmMC_pUXpw8DVBjBilU67BvbM8qT8gJE0bQfkhHo7FOdMttiz3syP5IR-LyO4J1WBlfmZjvMjRr4GIBt4o3Vqp-hKz7q2_OGwGtsN5s","//lh3.ggpht.com/fF2XWEtqG23ybhzClhC_p8gvKJalf1vg7k3H7UkuAaIVubil7EgOvJUCwAZk2KiCtlPYp1E5Ep2xaxZjJRmg5EFSEAjqlMHJS_Wd1Bcje6xre4s","//lh3.ggpht.com/jgOebMihBoIZvHE4EOklJvZ_k-9egjNIlUKfKFcLkvXJs8g2FXjPvdFUbwqGrkHrMtyis8uOvgt-E51Vm11hq4bieh7h0cegca0VI4vFtFaAemU","//lh3.ggpht.com/MOrI-zKNMNrQE_aHj5hzbojP3T0hEMJKK6K8UO3e1NBC-nkcQeIM1QnvtJdT_G-W4e7-qv4BiqwdWcNHBpZXOmmX3tcuYEV8u_ANEoa9_aUIfeyg","//lh6.ggpht.com/SyIS5sGOkTG7k_jFF14wzH9Evrblv6o4pHBI6z6X070-xhAeyut_kRO6xHtDID4KLcWFvItjQy-plPcJ6K1T9tlFOrtaryEPvuAYdMVx8e0TTw","//lh6.ggpht.com/2Pp9kLYFhDT3USwHinU5OxnzcWWOLI0nOWe29gOD5KMzyEcXoHkTN-AutJV9M8F_9eqAP379XB9O1d0BWPanhr-MguzKxfHeUvYTs6yHzDkxyfe0NA","//lh4.ggpht.com/7aofqklSkF3AMDfF19yqsA9J3EfEiKy1NdOelEGKNnW0Cv5tGEpq2PF_jZO1MVoBbrrmVVRv0Tdq7I8KyZbIlyHdbTs1jMl7dEFqVMvsPcyaORyHlQ","//lh4.ggpht.com/anYJHqkMCkuhmIHQTBspLtWcDTyx1ZRe84_q5pAgVEOVmsKkaKhS725N4YFoj2zpJrBP7iTC2vf1GUtrp6H7kkm8c1k6zkW6I_Gf5f9A3re_I8Ex","//lh3.ggpht.com/OtSw0rU-DvfoXgoWrQdkln6Kz7O14TF9qrPNJSGJnZLeDqUEctOn1DT09pdwwVpNQV-cXmVYQL-PX4XPhpZLWH1ciSkVT6WHNmTz1D9pHphBwJUv","//lh3.ggpht.com/cTCZnXPIjI-EO2bvQdLgeoSLOSlMFcv805n347Zyci9XDYUdcVDC_5H7SFVYDr4pC5HtQDYnrOHL6AinLW7hWtfSCLlvVhVUNQ-DlDn0NwZ-1iCO-g","//lh4.ggpht.com/i-mL_JcF9rwjQq6HnuKzuAHU41_UGxQ62IOPZvaDrATXaPFbhe-EbT7ZIpboyNA5PXRCsxNsZ9hu58edRvNs5ScgKN8Lg-00J2LhlwMAbdEsv7b0nw","//lh6.ggpht.com/D_YV2BG1WWwl67xNloP3sxzRkqhcVTgJi58L-A8nLrOcMR_tBqLz4fHEGQ-qiNcG_-32MNy3dlSPWrTBKzBcweJxgMnRVet5yuGfelUlwehDtXX_3w"],"video":[]},"sellerId":"mihai","listingType":"fixedPrice","payment":{"online":[{"paymentName":"PayPal","paymentService":"paypal"}],"offline":[{"paymentName":"Pay on Pick-up","paymentService":"payOnPickup"}]},"startTime":"2014-01-04T10:02:18+00:00","endTime":"2014-04-04T10:02:18+00:00","shipping":[{"shippingService":"economy","shippingName":"Economy","shippingCost":1.0,"handlingTimeMax":4,"dispatchTimeMin":1,"dispatchTimeMax":10,"shippingAdditionalCost":"2"},{"shippingService":"localPickup","shippingName":"Local Pick-Up","shippingCost":0.0,"handlingTimeMax":2,"dispatchTimeMin":0,"dispatchTimeMax":0,"shippingAdditionalCost":"0"}],"titleSlug":"many-pictures","variations":[{"fixedPrice":222999.0,"media":{"image":["//lh6.ggpht.com/ih3q2d7CenGLPyupH9FpfsoJQWQpw1i8wWA2Kd26bFnSF2fbnKyGU9WePIhCgEeqw5p6YMVmFi1c9oS0Ag93aF_oZ3ZiwK7fQuSYIrZ9VhgXbrTHkw","//lh6.ggpht.com/9aXLmPRVeZnxkwvNb3mWTF8kvfEY_lho_lOVVc9AbNqLb8GQmiS_XXVZ3OKqMv2pxgYSayMYPPRh6ACYyh0H8KtS8mPD6MKUkEajwxkTtp5Q4Lo","//lh3.ggpht.com/FG_QXZPHJ2tTYwI_t5Fg1KqivglVg9RlJn0JRsu9Ox8vJ7IcBirb2IV_I1LL_WVOMxfTuBBSDLMlrw9v0MCAdmnPCR29sCbRGjhm6zEfIH-3q2QSdw"],"video":[]},"quantity":1121,"Brand":"Bell \u0026 Howell"},{"fixedPrice":211.0,"media":{"image":["//lh6.ggpht.com/qAtjgyHAB734Ox_4NC_fa-ZRqrCjCmJu0Tp8bo-HMO88duv8l4hhuv2REBkB--yneFzOL7annecVlGty-YsKouondiOFVnAZWzjpdrfsGfbL6wh2","//lh3.ggpht.com/FG_QXZPHJ2tTYwI_t5Fg1KqivglVg9RlJn0JRsu9Ox8vJ7IcBirb2IV_I1LL_WVOMxfTuBBSDLMlrw9v0MCAdmnPCR29sCbRGjhm6zEfIH-3q2QSdw","//lh6.ggpht.com/9aXLmPRVeZnxkwvNb3mWTF8kvfEY_lho_lOVVc9AbNqLb8GQmiS_XXVZ3OKqMv2pxgYSayMYPPRh6ACYyh0H8KtS8mPD6MKUkEajwxkTtp5Q4Lo","//lh3.ggpht.com/MOrI-zKNMNrQE_aHj5hzbojP3T0hEMJKK6K8UO3e1NBC-nkcQeIM1QnvtJdT_G-W4e7-qv4BiqwdWcNHBpZXOmmX3tcuYEV8u_ANEoa9_aUIfeyg"],"video":[]},"quantity":2,"Brand":"Fujifilm"},{"fixedPrice":22.0,"media":{"image":["//lh3.ggpht.com/jgOebMihBoIZvHE4EOklJvZ_k-9egjNIlUKfKFcLkvXJs8g2FXjPvdFUbwqGrkHrMtyis8uOvgt-E51Vm11hq4bieh7h0cegca0VI4vFtFaAemU","//lh3.ggpht.com/MOrI-zKNMNrQE_aHj5hzbojP3T0hEMJKK6K8UO3e1NBC-nkcQeIM1QnvtJdT_G-W4e7-qv4BiqwdWcNHBpZXOmmX3tcuYEV8u_ANEoa9_aUIfeyg","//lh4.ggpht.com/anYJHqkMCkuhmIHQTBspLtWcDTyx1ZRe84_q5pAgVEOVmsKkaKhS725N4YFoj2zpJrBP7iTC2vf1GUtrp6H7kkm8c1k6zkW6I_Gf5f9A3re_I8Ex"],"video":[]},"quantity":12,"Brand":"Gateway"}],"_fpaiStatus":"published"}`
itemInfoBytes := []byte(itemInfoR)
var ItemInfo Item
er := json.Unmarshal(itemInfoBytes, &ItemInfo)
if er != nil {
panic(er)
}
}
Here's a fixed version of it: http://play.golang.org/p/w2ZcOzGHKR
The biggest fix that was needed is when Unmarshalling an array, that property needs to be an array/slice in the struct as well.
For example:
{ "things": ["a", "b", "c"] }
Would Unmarshal into a:
type Item struct {
Things []string
}
And not into:
type Item struct {
Things string
}
The other thing to watch out for when Unmarshaling is that the types line up exactly. It will fail when Unmarshalling a JSON string representation of a number into an int or float field -- "1" needs to Unmarshal into a string, not into an int like we saw with ShippingAdditionalCost int
Determining of root cause is not an issue since Go 1.8; field name now is shown in the error message:
json: cannot unmarshal object into Go struct field Comment.author of type string
You JSON doesn't match your struct fields: E.g. "district" in JSON and "District" as the field.
Also: Your Item is a slice type but your JSON is a dict value. Do not mix this up. Slices decode from arrays.
Just in case you're looking for an answer to this from an AWS perspective, particularly if you're trying to use UnmarshalListOfMaps() on the items coming back from a DynamoDB query(), then watch out for a simple typo I made.
Given that this function UnmarshalListOfMaps() takes in...a list of maps :-) then it needs to unmarshal into a List of whatever struct you're trying to build, and not just the plain struct itself. This was throwing me off because I was trying to start by querying for something that should only return one row.
movie := Movie{}
// Run the DynamoDB query
resp, err := session.Query(input) // type QueryInput
if err != nil {
log.WithError(err).Error("Error running DynamoDB query")
}
// Unmarshal all the attribute values into a Movie struct
err = dynamodbattribute.UnmarshalListOfMaps(resp.Items, &movie); if err != nil {
log.WithError(err).Error("Error marshaling DynamoDB result into Movie")
return link, err
}
The problem is in the first line. It should be movies := []Movie{} and then the reference &movie needs to change to &movies as well. If you leave it just as it is above, then the AWS Go SDK will throw this error:
cannot unmarshal list into Go value of type (etc).
If you ever face this while typing yml file not copy-pasting, it might be a problem with your yml file - mine was wrong indent.
This happened while making pipeline on concourse.