Issue with using custom JSON Marshal() for Embedded Structs - json

I am attempting to define a custom JSON marshaler to display some time information in specific formats. Ideally, I'd like to have a struct that stores created/modified values and then embed them into structs that need to keep track of that information. In addition, I want to define a custom date format in the JSON marshaler to use from a client-side application.
I currently have two structs
type Timestamp struct {
Created time.Time
Modified time.Time
}
type Company struct {
Id string
Name string
Timestamp
}
I want to embed the Timestamp struct into objects that will need to record when items are updated/created. Nothing insane there.
My issues comes when I define
func (t Timestamp) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
CreatedFormatted string
}{
CreatedFormatted: t.Created.Format("Monday Jan _2 15:04:05 2006"),
})
}
When I go to marshal Company, I only see the json for the Timestamp but nothing for the Company. I would have thought that the contents of the Company struct and the Timestamp struct would have been displayed. Am I doing something wrong here?

When a type embedded in a struct provides a method, that method becomes a part of the embedding struct. Since Company embeds Timestamp, Timestamp's MarshalJSON is available for Company as well. When json is looking to marshal a Company it sees that it has a MarshalJSON method and calls it — and the method it finds only marshals the timestamp field. The default behavior of structs (to marshal each field into its own key in a JSON object) is overridden.
What you can do:
Don't use struct embedding, just make Timestamp a regular field (even if you declare it as Timestamp Timestamp). Then Company won't inherit its methods and JSON will work as expected (but other parts of your code that expect embedding might change). Or:
Give Company its own MarshalJSON method that marshalls all of the fields including the timestamp. You could do this by
a. Copying the fields into a different type that is identical to Company but without the embedding, and marshalling that.
b. Copying the fields into a map and marshalling that.
c. Marshalling each of the fields independently and string-pasting fmt.Sprintf({"key1":%s,"key2":%s,...}, m1, m2, ...) yourself.

By embedding Timestamp in Company you not only shared member variables, but also the methods. This means you provided Company.MarshalJSON method, which is then used by json package for marshalling the whole structure. In order to see all the fields you would need to implement an explicit marshaler for Company structure as well.
If you only want to format the timestamp in a specific way, the other solution would be to provide your own time.Time and providing JSON marshaller there.
For example:
type AccessTime time.Time
func (t AccessTime) MarshalJSON() ([]byte, error) {
return json.Marshal(time.Time(t).Format("Monday Jan _2 15:04:05 2006"))
}
type Timestamp struct {
Created AccessTime
Modified AccessTime
}
https://play.golang.org/p/PhZXPauSyz

Related

How to unmarshall time string into time.Time in golang?

I am reading data from multiple tables using JOIN, CONCAT, GROUP_CONCAT, JSON_OBJECT. The data is read into the below mentioned model using gorm.
type OrgUserDisPublisherData struct {
Disciplines datatypes.JSON `json:"disciplines" example:"[]"`
User datatypes.JSON `json:"user"`
}
This process is successfully completed. But then when I try to unmarshal the OrgUserDisPublisherData.Disciplines into another struct which has time.Time datatypes. I am getting the following error parsing time "\"2022-11-03 07:08:09.000000\"" as "\"2006-01-02T15:04:05Z07:00\"": cannot parse " 07:08:09.000000\"" as "T"
Final model used for unmarshalling
type Discipline struct {
Name string `json:"name"`
Code string `json:"code"`
IsPrimary uint `json:"isPrimary"`
IsAligned uint `json:"isAligned"`
IsTrainingFaculty uint `json:"isTrainingFaculty"`
AlignmentDate time.Time `json:"alignmentDate"`
UnalignmentDate time.Time `json:"UnalignmentDate"`
ExpiryDate time.Time `json:"expiryDate"`
ExternalId string `json:"externalId"`
Status string `json:"status"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
At the same time, while inserting data into the tables the same model was used and it does not throw any error related to time.
How can I handle time while unmarshalling, irrespective of the data that is present against the time property?
The problem here is that the default JSON marshalling behaviour for date/time types in GoLang structs is to use ISO8601 formatted date/time strings.
This is identified by the format string in the error message with the T separator between date and time and time-zone suffix. The values in your Discipline JSON string don't conform to this format, lacking both the T separator and any time-zone. Hence the error.
If you can influence the formatting of the JSON string produced by gorm, (not something I'm familiar with so cannot say whether you can or how to do so), then the simplest solution would be to ensure that your JSON string time fields are formatted as ISO8601/RFC3339 strings.
If you have no control over that, then you have two options:
Implement some pre-processing of the JSON via an intermediate map[string]any and reformat the appropriate fields. If the gorm formatting is at least consistent then this could be as easy as splitting the string on the space, remove the final 3dps from the time, append an appropriate timezone (or just Z if the times are UTC) then re-assemble with a T separator.
Use a custom time type with a json.Marshaller implementation that works correctly with the gorm formatted values (you still need to know what time zone applies to the persisted values and apply that correctly when marshalling).
Both of these are vulnerable to a change in the gorm formatting of date/time variables and mis-use (failing to pre-process in the case of Option #1 and mistakenly using time.Time rather than the custom type in the case of Option #2).
For this reason, modifying the formatted output from gorm would be the preferred approach for me, if it is possible.
The quickest solution was to format the data/value while reading it from DB, as suggested by #Deltics.
While querying the data from the DB, using the DATE_FORMAT() I am formatting the data into the format required by go/json
DATE_FORMAT(actual_data, '%Y-%m-%dT%TZ')

Dealing with NULLs when migrating from MySQL to Mongo in Go

Setup
I started a project using MySQL and as such, my project has some helper types that assist with dealing with nulls, both when unmarshalling incoming data on the API, inputting data into the DB, and then the inverse of that, pulling data out of the Database and responding with said data to the API.
For the purposes of this question, we'll deal with a struct i have that represents a Character.
type Character struct {
MongoID primitive.ObjectID `bson:"_id" json:"-"`
ID uint64 `bson:"id" json:"id"`
Name string `bson:"name" json:"name"`
CorporationID uint `bson:"corporation_id" json:"corporation_id"`
AllianceID null.Uint `bson:"alliance_id" json:"alliance_id,omitempty"`
FactionID null.Uint `bson:"faction_id" json:"faction_id,omitempty"`
SecurityStatus float64 `bson:"security_status" json:"security_status"`
NotModifiedCount uint `bson:"not_modified_count" json:"not_modified_count"`
UpdatePriority uint `bson:"update_priority" json:"update_priority"`
Etag null.String `bson:"etag" json:"etag"`
CachedUntil time.Time `bson:"cached_until" json:"cached_until"`
CreatedAt time.Time `bson:"created_at" json:"created_at"`
UpdatedAt time.Time `bson:"updated_at" json:"updated_at"`
}
I want to specifically concentrate on the AllianceID property of type null.Uint which is represented with the following struct:
// Uint is an nullable uint.
type Uint struct {
Uint uint
Valid bool
}
In an API setup using JSON and MySQL (i.e. My setup, but this is not exclusive), this structure allows me to easily deal with values that are "nullable" without having to deal with Pointers. I've always heard that it is best to avoid Pointers with the exception of pointers to structures (Slices, Slices of Structs, Map of Structs, etc). If you have a primitive type (int, bool, float, etc), try to avoid using a pointer to that primitive type.
This type has functions like MarshalJSON, UnmarshalJSON, Scan, and Value with logic inside those functions that leverage the Vaild property to determine what type of value to return. This works really really well with this setup.
Question
After some research, I've come to realize that Mongo would suit me better than a relational database, but due to the fluidity of a Mongo Document (Schemaless), I'm having a hard time understanding how to handle scenarios where a field maybe missing, or a property that i have in MySQL that would normally be null and I can easily unmarshal ontop this struct and use the helper functions logically, is handled. Also, when I setup a connection to Mongo and pull a couple of rows from MySQL and created Documents in Mongo from these rows, the BSON layer is marshalling the entire type for Alliance ID and sticking it in the DB.
Example:
"alliance_id" : {
"uint" : NumberLong(99007760),
"valid" : true
},
Where as in MySQL, the Value function implementing Valuer interface would be called and return 99007760 and that is the value in the DB.
Another scenario would be if valid was false. In MySQL this would mean a null value and when the Value function is called, it would return nil and the mysql driver would populate the field with NULL
So my question is how do I do this? Do I need to start from scratch and rebuild my models and redo some of the logic in my application that leverages the Valid property and use *Pointers or can I do what I am attempting to do using these helper types.
I do want to say that I have tried implementing the Marshaller, and Unmarshaller interfaces on the bson package and the alliane_id in the document is still set to the json encoded version of this type as I outlined above. I wanted to point this out to rule out any suggestions of implemeting those interfaces. If what I am attempting to achieve is counter intuitive to Mongo, please link some guides that can help me achieve what im attempting to do.
Thank you to all who can assist with this.
The easiest way to deal with optional fields like this is to use a pointer:
type Character struct {
ID *uint64 `bson:"id,omitempty" json:"id"`
Name string `bson:"name" json:"name"`
...
}
Above, the ID field will be written if it s non-nil. When unmarshaling, it will be set to a non-nil value if database record has a value for it. If you omit the omitempty flag, marshaling this struct will write null to the database.
For strings, you may use omitempty to omit the field completely if it is empty. If you want to store empty strings, omit omitempty.

UnmarshalJSON any type from []interface{} possible?

When you unmarshal JSON to []interface{} is there any way to automatically detect the type besides some standard types like bool, int and string?
What I noticed is the following, Let's say I marshal [uuid.UUID, bool] then the JSON I get looks like:
[[234,50,7,116,194,41,64,225,177,151,60,195,60,45,123,106],true]
When I unmarshal it again, I get the types as shown through reflect:
[]interface{}, bool
I don't understand why it picked []interface{}. If it cannot detect it, shouldn't it be at least interface{}?
In any case, my question is, is it possible to unmarshal any type when the target is of type []interface{}? It seems to work for standard types like string, bool, int but for custom types I don't think that's possible, is it? You can define custom JSON marshal/unmarshal methods but that only works if you decode it into a target type so that it can look up which custom marshal/unmarshal methods to use.
You can unmarshal any type into a value of type interface{}. If you use a value of type []interface{}, you can only unmarshal JSON arrays into it, but yes, the elements of the array may be of any type.
Since you're using interface{} or []interface{}, yes, type information is not available, and it's up to the encoding/json package to choose the best it sees fit. For example, for JSON objects it will choose map[string]interface{}. The full list of default types is documented in json.Unmarshal():
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Obviously if your JSON marshaling/unmarshaling logic needs some pre- / postprocessing, the json package will not miraculously find it out. It can know about those only if you unmarshal into values of specific types (which implement json.Unmarshaler). The json package will still be able to unmarshal them to the default types, but custom logic will obviously not run on them.

Is it possible to have a struct with multiple JSON tags?

I post a request to a server and get a reply in JSON format. I'm able to unmarshal it to a struct. Then I need to create a new JSON file with the same data but different JSON tags.
Example:
In the following code, I get {"name":"Sam","age":20} from a server and unmarshal it to the struct Foo:
type Foo struct {
Name string `json:"name"`
Age int `json:"age"`
}
Then I need to change the tag name to employee_name and omit age:
type Bar struct {
Name string `json:"employee_name"`
Age int `json:"-"`
}
After that I send this modified data to another server.
I know I could just create a new Bar and copy all data into it, but there are a lot of fields. I was wondering if there is a way to attach multiple JSON tags like this:
type Foo struct {
Name string `json:"name" json:"employee_name"`
Age int `json:"age" json:"-"`
}
Thanks in advance.
It's not possible. The encoding/json package only handles the json key in struct tags. If the json key is listed multiple times (as in your example), the first occurrence will be used (this is implemented in StructTag.Get()).
Note that this is an implementation restriction of the encoding/json package and not that of Go. One could easily create a similar JSON encoding package supporting either multiple tag keys (e.g. json1, json2) or multiple occurrences of the same key (as in your example).
What is possible though, with 2 identically laid out structs (namin, types and ordering of fields needs to match exactly) is to cast from one to the other. I would be very cautious of doing this though and make sure the 2nd type (bar in your example) is unexported to prevent from being used elsewhere.
https://play.golang.org/p/y8EH1U9_3jN

Unmarshal a JSON array of heterogeneous structs

I want to deserialise an object that includes an array of a some interface Entity:
type Result struct {
Foo int;
Bar []Entity;
};
Entity is an interface that is implemented by a number of struct types. JSON data identifies the struct type with a "type" field in each entity. E.g.
{"type":"t1","field1":1}
{"type":"t2","field2":2,"field3":3}
How would I go about deserialising the Result type in such a way that it correctly populates the array. From what I can see, I have to:
Implement UnmarshalJSON on Result.
Parse Bar as a []*json.RawMessage.
Parse each raw message as map[string]interface{}.
Check "type" field in the raw message.
Create a struct of appropriate type.
Parse the raw message again, this time into the just created struct.
This all sounds very tedious and boring. Is there a better way to do this? Or am I doing it backwards, and there is a more canonical method to handle an array of heterogeneous objects?
I think your process is probably a bit more complicated than it has to be, see http://play.golang.org/p/0gahcMpuQc. A single map[string]interface{} will handle a lot of that for you.
Alternatively, you could make a type like
struct EntityUnion {
Type string
// Fields from t1
// Fields from t2
// ...
}
Unmarshal into that; it will set the Type string and fill in all the fields it can get from the JSON data. Then you just need a small function to copy the fields to the specific type.