i'm trying to decode 2 JSON items into the same struct, because the second JSON complet the first one, but it doesn't work (do nothing) have you got some ideas ?
func getUserClip(this *LibraryController, id string) (*Clip){
//Test Api
//Send Request to azure search
Data := Clip{}
if req := GetClipById("b373400a-bd7e-452a-af68-36992b0323a5"); req == nil {
return nil
} else {
str, err := req.String()
if err != nil {
beego.Debug("Error Json req.String: ", err)
}
//Uncode Json to string
if err := json.Unmarshal([]byte(str), &Data); err != nil {
beego.Debug("Error json", err)
}
for i := range Data.Value {
if req = GetCliRedis(Data.Value[i].Id); err != nil {
return nil
} else {
str, err := req.String()
beego.Debug("JSON REDIS DEBUG: ", str)
if err != nil {
beego.Debug("Error Json req.String: ", err)
}
if err := json.Unmarshal([]byte(str), &Data); err != nil {
beego.Debug("Error json", err)
}
}
i++
}
}
return &Data
}
and the struct
type Clip struct {
Value []InfoClip `json:value`
}
type InfoClip struct {
Id string `json:id`
CreatedAt time.Time `json:createdAt`
StartTimeCode int `json:startTimeCode`
EndTimeCode int `json:endTimeCode`
Metas metas `json:metas`
Tags []string `json:tags`
Categories []string `json:categories`
UserId string `json:userId`
SourceId string `json:sourceId`
ProviderName string `json:providerName`
ProviderReference string `json:providerReference`
PublicationStatus string `json:publicationStatus`
Name string `json:name`
FacebookPage string `json:facebookPage`
TwitterHandle string `json:twitterHandle`
PermaLinkUrl string `json:permalinkUrl`
Logo string `json:logo`
Link string `json:link`
Views int `json:views`
}
type metas struct {
Title string `json:title`
Tags []string `json:tags`
Categories []string `json:categories`
PermaLink string `json:permalink`
}
The JSON I receive is:
{
"clipId":"9b2ea9bb-e54b-4291-ba16-9211fa3c755f",
"streamUrl":"https://<edited out>/asset-32e43a5d-1500-80c3-cc6b-f1e4fe2b5c44\/6c53fbf5-dbe9-4617-9692-78e8d76a7b6e_H264_500kbps_AAC_und_ch2_128kbps.mp4?sv=2012-02-12&sr=c&si=17ed71e8-5176-4432-8092-ee64928a55f6&sig=KHyToRlqvwQxWZXVvRYOkBOBOF0SuBLVmKiGp4joBpw%3D&st=2015-05-18T13%3A32%3A41Z&se=2057-05-07T13%3A32%3A41Z",
"startTimecode":"6",
"endTimecode":"16",
"createdAt":"2015-05-19 13:31:32",
"metas":"{\"title\":\"Zapping : Obama, Marine Le Pen et Michael Jackson\",\"tags\":[\"actualite\"],\"categories\":[\"actualite\"],\"permalink\":\"http:\/\/videos.lexpress.fr\/actualite\/zapping-obama-marine-le-pen-et-michael-jackson_910357.html\"}",
"sourceId":"6c53fbf5-dbe9-4617-9692-78e8d76a7b6e",
"providerName":"dailymotion",
"providerReference":"x1xmnxq",
"publicationStatus":"1",
"userId":"b373400a-bd7e-452a-af68-36992b0323a5",
"name":"LEXPRESS.fr",
"facebookPage":"https:\/\/www.facebook.com\/LExpress",
"twitterHandle":"https:\/\/twitter.com\/lexpress",
"permalinkBaseURL":"https:\/\/tym.net\/fr\/{CLIP_ID}",
"logo":"lexpress-120px.png",
"link":"http:\/\/videos.lexpress.fr\/"
}
The Redis complet the azure search missing information :
here the struct :
type Clip struct {
Value []SearchClip `json:value`
}
type SearchClip struct {
Id string `json:id`
CreatedAt string`json:createdAt`
Tags []string `json:tags`
Categories []string `json:categories`
UserId string `json:userId`
SourceId string `json:sourceId`
Views int `json:views`
}
this is the basic information and redis complet this
I don't want to merge 2 struct into a third one i think it's not the better process, i will do it if it's the last solution.
For lack of activity I'm just gonna post the embedding option as a solution. It's probably the simplest way to do what you want.
type ClipInfoAndMeta struct {
Metas
InfoClip
}
Note I upper cased the name on metas not sure it's necessary but I believe it will be. The language feature being used here is called 'embedding' and it works a lot like composition except that the fields/methods for embedded types are more or less 'hoisted' to the containing types scope. IE with an instance of ClipInfoAndMeta you can directly access any exported field that is defined on InfoClip.
One oddity of your set up is that you'll have collisions on field names between the two types. Not sure how that would play out. With all this being said, it would be helpful to see the json string you're trying to Unmarshal from. As I've been writing this I realized that metas is just a subset of InfoClip. Which has confused me about what you are actually trying to do? I mean, if the data coming back is all in one object, it would mean InfoClip is sufficient for storing all of it. If that is the case you have no reason for the other object... And if you want to trim down the fields which get passed to the display layer of your app you should just define a method on the InfoClip type like func (i *InfoClip) GetMetas() Metas { return &Metas{ ... } } then you can just deal with the one type everywhere and hand off the Metas to the display layer when it's needed.
After a lot of trial and error, I present to you this fully functional solution:
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Clip struct {
Value []InfoClip `json:value`
}
type customTime struct {
time.Time
}
const ctLayout = "2006-01-02 15:04:05"
func (ct *customTime) UnmarshalJSON(b []byte) (err error) {
if b[0] == '"' && b[len(b)-1] == '"' {
b = b[1 : len(b)-1]
}
ct.Time, err = time.Parse(ctLayout, string(b))
return
}
type InfoClip struct {
Id string `json:"clipId"`
CreatedAt customTime `json:"createdAt"`
StartTimeCode string `json:"startTimeCode"` //if you want ints here, you'll have to decode manually, or fix the json beforehand
EndTimeCode string `json:"endTimeCode"` //same for this one
Metas metas `json:"-"`
MetasString string `json:"metas"`
Tags []string `json:"tags"`
Categories []string `json:"categories"`
UserId string `json:"userId"`
SourceId string `json:"sourceId"`
ProviderName string `json:"providerName"`
ProviderReference string `json:"providerReference"`
PublicationStatus string `json:"publicationStatus"`
Name string `json:"name"`
FacebookPage string `json:"facebookPage"`
TwitterHandle string `json:"twitterHandle"`
PermaLinkUrl string `json:"permalinkBaseURL"`
Logo string `json:"logo"`
Link string `json:"link"`
Views int `json:"views"`
}
type metas struct {
Title string `json:"title"`
Tags []string `json:"tags"`
Categories []string `json:"categories"`
PermaLink string `json:"permalink"`
}
var jsonString = `{
"clipId":"9b2ea9bb-e54b-4291-ba16-9211fa3c755f",
"streamUrl":"https://<edited out>/asset-32e43a5d-1500-80c3-cc6b-f1e4fe2b5c44\/6c53fbf5-dbe9-4617-9692-78e8d76a7b6e_H264_500kbps_AAC_und_ch2_128kbps.mp4?sv=2012-02-12&sr=c&si=17ed71e8-5176-4432-8092-ee64928a55f6&sig=KHyToRlqvwQxWZXVvRYOkBOBOF0SuBLVmKiGp4joBpw%3D&st=2015-05-18T13%3A32%3A41Z&se=2057-05-07T13%3A32%3A41Z",
"startTimecode":"6",
"endTimecode":"16",
"createdAt":"2015-05-19 13:31:32",
"metas":"{\"title\":\"Zapping : Obama, Marine Le Pen et Michael Jackson\",\"tags\":[\"actualite\"],\"categories\":[\"actualite\"],\"permalink\":\"http:\/\/videos.lexpress.fr\/actualite\/zapping-obama-marine-le-pen-et-michael-jackson_910357.html\"}",
"sourceId":"6c53fbf5-dbe9-4617-9692-78e8d76a7b6e",
"providerName":"dailymotion",
"providerReference":"x1xmnxq",
"publicationStatus":"1",
"userId":"b373400a-bd7e-452a-af68-36992b0323a5",
"name":"LEXPRESS.fr",
"facebookPage":"https:\/\/www.facebook.com\/LExpress",
"twitterHandle":"https:\/\/twitter.com\/lexpress",
"permalinkBaseURL":"https:\/\/tym.net\/fr\/{CLIP_ID}",
"logo":"lexpress-120px.png",
"link":"http:\/\/videos.lexpress.fr\/"
}`
func main() {
res := parseJson(jsonString)
fmt.Printf("%+v\n",res)
}
func parseJson(theJson string) InfoClip {
toParseInto := struct {
InfoClip
MetasString string `json:"metas"`
}{
InfoClip: InfoClip{},
MetasString: ""}
err := json.Unmarshal([]byte(jsonString), &toParseInto)
if err != nil {
log.Panic(err)
}
err = json.Unmarshal([]byte(toParseInto.MetasString), &toParseInto.InfoClip.Metas)
if err != nil {
log.Panic(err)
}
return toParseInto.InfoClip
}
What are we doing in the parseJson function?
We create a new struct and assign that to the toParseInto variable. We design the struct in a way that it contains all of the fields from InfoClip via embedding, and We add a field to temporarily hold the JSON string metas.
We then unmarshal into that struct, which, after fixing the issues listed below, works fine.
After that, we unmarshal that inner JSON into the correct field in the embedded InfoClip.
We can now easily return that embedded InfoClip to get what we really wanted.
Now, all the issues I have identified in your original solution:
The time format in your JSON is not the standard time format to be used in JSON. That is defined in some RFC, but anyways: because of that, we have to use our own type customTime to parse that. It handles just like a normal time.Time, because that is embedded within.
All your json tags were wrong. All of them had missing quotes, and some were just not even correct.
startTimeCode and endTimeCode are strings in the JSON, not ints
Left to you to improve:
Error handling: Don't just panic in the parseJson function, but rather return the error somehow
If you want startTimecode and endTimecode to be available as ints, parse them manually. You can employ a "hack" similar to the one I used to parse the inner JSON.
One final note, not related to this answer but rather to your question: If you had provided both your code and the JSON with your original question, you would have had an answer in probably less than an hour. Please, please don't make this harder than it needs to be.
EDIT: I forgot to provide my sources, I used this question to parse your time format.
Related
I'm reading a file.json into memory. It's an array of objects, sample:
[
{"id":123123,"language":"ja-JP","location":"Osaka"}
,{"id":33332,"language":"ja-JP","location":"Tokyo"}
,{"id":31231313,"language":"ja-JP","location":"Kobe"}
]
I want to manipulate certain keys in this JSON file, so that they start with uppercase. Meaning
"language" becomes "Language" each time it's found. What I've done so far is to make a struct representing each object, as such:
type sampleStruct struct {
ID int `json:"id"`
Language string `json:"Language"`
Location string `json:"Location"`
}
Here, I define the capitalization. Meaning, id shouldn't be capitalized, but location and language should.
Rest of the code is as such:
func main() {
if len(os.Args) < 2 {
fmt.Println("Missing filename parameter.")
return
}
translationfile, err := ioutil.ReadFile(os.Args[1])
fileIsValid := isValidJSON(string(translationfile))
if !fileIsValid {
fmt.Println("Invalid JSON format for: ", os.Args[1])
return
}
if err != nil {
fmt.Println("Can't read file: ", os.Args[1])
panic(err)
}
}
func isValidJSON(str string) bool {
var js json.RawMessage
return json.Unmarshal([]byte(str), &js) == nil
}
// I'm unsure how to iterate through the JSON objects and only uppercase the objects matched in my struct here.
func upperCaseSpecificKeys()
// ...
Desired output, assuming the struct represents the whole data object, transform each key as desired:
[
{"id":123123,"Language":"ja-JP","Location":"Osaka"}
,{"id":33332,"Language":"ja-JP","Location":"Tokyo"}
,{"id":31231313,"Language":"ja-JP","Location":"Kobe"}
]
The documentation on json.Unmarshal says (with added emphasis):
To unmarshal JSON into a struct, Unmarshal matches incoming object
keys to the keys used by Marshal (either the struct field name or its
tag), preferring an exact match but also accepting a case-insensitive
match
See example here: https://play.golang.org/p/1vv8PaQUOfg
One way is to implement custom marshal method, although not very flexible:
type upStruct struct {
ID int `json:"id"`
Language string
Location string
}
type myStruct struct {
ID int `json:"id"`
Language string `json:"language"`
Location string `json:"location"`
}
func (m myStruct) MarshalJSON() ([]byte, error) {
return json.Marshal(upStruct(m))
}
....
func main() {
var mySArr []myStruct
// 1. Unmarshal the input
err := json.Unmarshal([]byte(myJson), &mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Input: \n%+v\n", mySArr)
// 2. Then, marshal it using our custom marshal method
val, err := json.Marshal(mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Output: \n%v\n", string(val))
}
Link to working code: https://play.golang.org/p/T4twqPc34k0
Thanks to mkopriva
My struct:
type User struct {
FirstName string `json:"firstname, omitempty" validate:"required"`
LastName string `json:"lastname, omitempty" validate:"required"`
NumberofDays int `json:"numberofdays, string" validate:"min=0,max=100"`
}
Value for NumberofDays is passed as string from the server but I want to check if it is within range and store as int.
Ex: user := &User{"Michael","Msk","3"}
I'm getting 'cannot unmarshal string into Go value of type int'.
I'm not sure how to typecast to int and do the validation
Remove the space after the comma in the struct tags, e.g. json:"numberofdays, string" should be json:"numberofdays,string". With the space, the json package ignores the string part, hence the error message you're getting. Without the space, the error goes away and behavior is as expected.
Demo comparing the two here: https://play.golang.org/p/AUImnw_PIS
Documentation of json package struct tags: https://golang.org/pkg/encoding/json/#Marshal
You can use a custom type to unmarshal a string
to an integer value using whatever set of parsing rules
you want this mapping to use:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type User struct {
FirstName string `json:"firstname,omitempty" validate:"required"`
LastName string `json:"lastname,omitempty" validate:"required"`
NumberofDays StrInt `json:"numberofdays" validate:"min=0,max=100"`
}
type StrInt int
func (si *StrInt) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
n, err := strconv.ParseInt(s, 10, 0)
if err != nil {
return err
}
*si = StrInt(n)
return nil
}
const data = `{
"FirstName": "John",
"LastName": "Doe",
"NumberOfDays": "42"
}`
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
panic(err)
}
fmt.Println(u)
}
Playground.
The idea is as follows:
Have a custom type which is mostly int but allows defining custom methods on it.
Have that type satisfy the encoding/json.Unmarshaler interface by implementing the UnmarshalJSON() method on a pointer to that type.
In the method, first unmarshal the value as its native type—string—and
then parse it as an integer.
This approach might appear to be weird, but if you'll think of it
a bit more, there's no single "true" string representation of an integer:
there are different bases and different conventions at representing
integers even in a particular base.
In your unmarshaling method you implement the policy of the text representation
of your integers.
The json package defines a custom type json.Number for these cases.
I am new to BoltDB and Golang, and trying to get your help.
So, I understand that I can only save byte array ([]byte) for key and value of BoltDB. If I have a struct of user as below, and key will be the username, what would be the best choice to store the data into BoltDB where it expects array of bytes?
Serializing it or JSON? Or better way?
type User struct {
name string
age int
location string
password string
address string
}
Thank you so much, have a good evening
Yes, I would recommend marshaling the User struct to JSON and then use a unique key []byte slice. Don't forget that marshaling to JSON only includes the exported struct fields, so you'll need to change your struct as shown below.
For another example, see the BoltDB GitHub page.
type User struct {
Name string
Age int
Location string
Password string
Address string
}
func (user *User) save(db *bolt.DB) error {
// Store the user model in the user bucket using the username as the key.
err := db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucketIfNotExists(usersBucket)
if err != nil {
return err
}
encoded, err := json.Marshal(user)
if err != nil {
return err
}
return b.Put([]byte(user.Name), encoded)
})
return err
}
A good option is the Storm package, which allows for exactly what you are wanting to do:
package main
import (
"fmt"
"github.com/asdine/storm/v3"
)
type user struct {
ID int `storm:"increment"`
address string
age int
}
func main() {
db, e := storm.Open("storm.db")
if e != nil {
panic(e)
}
defer db.Close()
u := user{address: "123 Main St", age: 18}
db.Save(&u)
fmt.Printf("%+v\n", u) // {ID:1 address:123 Main St age:18}
}
As you can see, you don't have to worry about marshalling, it takes care of it for you. By default it uses JSON, but you can configure it to use GOB or others as well:
https://github.com/asdine/storm
I have a struct that I'd like to Marshal into JSON differently depending on the context.
For example, sometimes I want to marshal like this:
type MyStruct struct {
Nickname string `json:"nickname"`
EmailAddress string `json:"email_address"`
PhoneNumber string `json:"-"`
MailingAddress string `json:"-"`
}
And sometimes I want to marshal like this:
type MyStruct struct {
Nickname string `json:"nickname"`
EmailAddress string `json:"email_address"`
PhoneNumber string `json:"phone_number"`
MailingAddress string `json:"mailing_address"`
}
Is there a simple way to do this without:
Making 2 separate structs.
Writing a custom marshaller.
Temporarily removing the string values for PhoneNumber and MailingAddress (with an omitempty on the tag), marshaling and then adding them back.
If only there was a way to:
Specify 2 sets of tags and tell the marshaler which ones to use.
Dynamically change the tags at runtime.
I know you explicitly mention "without writing a custom marshaler", but in case someone sees this and thinks it should be avoided because of complexity, a custom marshaler to do what you want to do is really simple:
type MyStruct struct {
Nickname string `json:"nickname"`
EmailAddress string `json:"email_address"`
PhoneNumber string `json:"phone_number"`
MailingAddress string `json:"mailing_address"`
all bool
}
func (ms MyStruct) MarshalJSON() ([]byte, error) {
m := map[string]interface{}{} // ideally use make with the right capacity
m["nickname"] = ms.Nickname
m["email_address"] = ms.EmailAddress
if ms.all {
m["phone_number"] = ms.PhoneNumber
m["mailing_address"] = ms.MailingAddress
}
return json.Marshal(m)
}
If the all field should be set by an external package, then a method could be defined on the struct, or the field could be made public (wouldn't affect the JSON since it is encoded via the custom marshaler).
Runnable example on the playground: http://play.golang.org/p/1N_iBzvuW4
You can use reflection, not really the most efficient solution but it is convenient.
func MarshalSubset(obj interface{}, fields ...string) ([]byte, error) {
if len(fields) == 0 {
return json.Marshal(obj)
}
out := make(map[string]interface{}, len(fields))
val := reflect.ValueOf(obj)
if val.Kind() == reflect.Ptr {
val = val.Elem()
}
if val.Kind() != reflect.Struct {
panic("not a struct")
}
typ := val.Type()
for _, f := range fields {
val := val.FieldByName(f).Interface()
rfld, _ := typ.FieldByName(f)
tag := strings.Split(rfld.Tag.Get("json"), ",")
if len(tag) > 0 {
f = tag[0]
}
out[f] = val
}
return json.Marshal(out)
}
playground
I have struct
type tySurvey struct {
Id int64 `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
I do json.Marshal write JSON bytes in HTML page. jQuery modifies name field in object and encodes object using jQueries JSON.stringify and jQuery posts string to Go handler.
id field encoded as string.
Sent: {"id":1} Received: {"id":"1"}
Problem is that json.Unmarshal fails to unmarshal that JSON because id is not integer anymore.
json: cannot unmarshal string into Go value of type int64
What is best way to handle such data? I do not wish to manually convert every field. I wish to write compact, bug free code.
Quotes is not too bad. JavaScript does not work well with int64.
I would like to learn the easy way to unmarshal json with string values in int64 values.
This is handled by adding ,string to your tag as follows:
type tySurvey struct {
Id int64 `json:"id,string,omitempty"`
Name string `json:"name,omitempty"`
}
This can be found about halfway through the documentation for Marshal.
Please note that you cannot decode the empty string by specifying omitempty as it is only used when encoding.
use json.Number
type tySurvey struct {
Id json.Number `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
You could also create a type alias for int or int64 and create a custom json unmarshaler
Sample code:
Reference
// StringInt create a type alias for type int
type StringInt int
// UnmarshalJSON create a custom unmarshal for the StringInt
/// this helps us check the type of our value before unmarshalling it
func (st *StringInt) UnmarshalJSON(b []byte) error {
//convert the bytes into an interface
//this will help us check the type of our value
//if it is a string that can be converted into a int we convert it
///otherwise we return an error
var item interface{}
if err := json.Unmarshal(b, &item); err != nil {
return err
}
switch v := item.(type) {
case int:
*st = StringInt(v)
case float64:
*st = StringInt(int(v))
case string:
///here convert the string into
///an integer
i, err := strconv.Atoi(v)
if err != nil {
///the string might not be of integer type
///so return an error
return err
}
*st = StringInt(i)
}
return nil
}
func main() {
type Item struct {
Name string `json:"name"`
ItemId StringInt `json:"item_id"`
}
jsonData := []byte(`{"name":"item 1","item_id":"30"}`)
var item Item
err := json.Unmarshal(jsonData, &item)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+v\n", item)
}