I am writing a Swift program to interact with our backend Database. These database tables can change format, and each row is from DynamoDB and is represented by a JSON object. Within the program, it will grab the cache from the server which contains the Database structures and then the current contents of the database. I don't want to hardcode the database structures into the app as that would obviously make this too very hard to maintain. What is the best way to read in these rows into a generic dictionary object with dynamic names that can hold any of the JSON types?
One way you can handle parsing dynamic JSON is to have a key or a set of keys that gives information about which type of model to decode. For example, you can have a key type in your JSON objects:
{
...
"type": "some type"
...
}
and an intermediate model:
struct IntermediateModel: Codable {
let type: String // or enum
}
you first decode all your JSON objects to the intermediate model and then decode them to the actual model type based on the value of type.
I have written some helper protocols that make this easier here.
Related
I need to upload through a GraphQL mutation an object whose schema is not known at design time. Basically it is a raw JSON blob that may or may not come from other, possibly non-GraphQL services. I am using JSON scalar from https://github.com/graphql-java/graphql-java-extended-scalars:
scalar JSON
input MyInput {
jsonField: JSON
}
It works nicely except when the input blob contains keys like "$ref", which is the standard way to describe references in JSON. However, field name $ref does not seem to be accepted as it does not conform to the GraphQL naming conventions.
Obvious solutions appear to be to encode-decode field names in some way, or send the whole JSON as a String, perhaps with a custom scalar type. But is there perhaps a more elegant way to achieve this?
I'm trying to retrieve data from an API call and pass them to another service. The data i received are in a specific JSON structure and i would like to map it to a struct, but without the multiple levels of data. I tried the dot notation to access deeper value, but it doesn't work.
Basically, i'm trying to get a struct with an array of "issues" (key, self, description), but without having the "fields.description" structure.
The JSON:
{
"ticket": {
"number": "2",
"issues": [
{
"key": "TA-2",
"self": "http://localhost:8080/rest/api/2/issue/10100",
"fields": {
"description": "This template is used to create openshift project.\n|Type|Value|Help|\n|project_name|personal_project|Enter the name of your openshift project|\n|memory|8GB|Enter the desired amount of memory|"
}
},
{
"key": "TA-1",
"self": "http://localhost:8080/rest/api/2/issue/10000",
"fields": {
"description": "This template is used to create openshift project.\n|Type|Value|Help|\n|project_name|my_openshift_project|Enter the name of your openshift project|\n|memory|4GB|Enter the desired amount of memory|"
}
}
]
}
}
The Struct:
type Todo struct {
Number string `json:"number"`
Issue []struct {
Key string `json:"key"`
Self string `json:"self"`
Description string `json:"field.description"` //doesn't work. don't know what to put ...
} `json:"issues"`
}
The expected / desired struct:
{
"number": "2",
"issues": [{
"key": "TA-2",
"self": "http://localhost:8080/rest/api/2/issue/10100",
"description": "This template ..."
}, {
"key": "TA-1",
"self": "http://localhost:8080/rest/api/2/issue/10000",
"description": "This template ..."
}]
}
Is it possible? if yes, how to do it? Using nested struct won't change the initial JSON structure.
Thanks
You have at least three options:
Create a separate set of struct types to represent the data in the format provided by the JSON. This might include a TodoList struct with a ticket field, referring to a separate Todo struct. This struct might have a number field and an issues field. Each Issue struct might have a fields field, which in turn has a description field. Basically, you recreate the entire domain as represented in the JSON, then you can simply call json.Unmarshal() to unmarshal the JSON into the TodoList. Once you have a TodoList, simply convert it to the representation that you desire by hand.
Write your own custom UnmarshalJSON() function for the Todo and Issue structs. Click here for an example from the GoLang docs.
Unmarshal the entire JSON document manually in an external function. This might involve unmarshalling the data to a map[string]json.RawMessage, and then mapping many of those json.RawMessage objects to other maps and objects, and so on.
The first option is better for cases where you simply have an alternate representation of the data which you desire, but all of the data is still in the JSON document, and all of the data in the JSON document is still relevant. This seems to match your predicament, and so this is the option I would recommend.
The second option is better for cases where you would simply like to recognize that the JSON encoding for a structure should provide a consistently different representation from the GoLang object encoding. In other words, when the structure is encoded as a GoLang object, it should take one form, whereas when the structure is encoded in JSON, it should take another form. These two forms must be strictly tied to their encodings (JSON vs GoLang object). If you plan on unmarshaling the JSON to a GoLang object, and then marshaling it back into a different JSON representation from what you had to begin with, this is probably not the best option because json.Unmarshal() and json.Marshal() should be inverses of each other. Otherwise, you have violated the principle of least astonishment. This option can also be used for delaying or preventing unmarshaling of nested properties by way of json.RawMessage objects. Thus, if you have some dynamically typed JSON field whose type is determined upon examination of other fields, or if you have large amounts of irrelevant data in the JSON document, you can delay or prevent the unmarshalling of such data until desired.
In practice, the third option is very similar to the second. However, by providing an external unmarshaling mechanism rather than implementing UnmarshalJSON(), you are less likely to break the principle of least astonishment. That is, the perceived contract between json.Unmarshal() and json.Marshal() is necessarily preserved. Thus, you can marshal and unmarshal data however you would like, and you can maintain as many different representations per encoding as you would like. That being said, the first option provides this very same benefit, and it is much easier. In reality, the third option should only ever be chosen if the first and second options will not suffice, such as when you may need to delay or prevent unmarshaling of nested properties (like in the second option) while simultaneously providing multiple different JSON representations and preserving the contract between json.Marshal() and json.Unmarshal() (like in the first option).
You cannot add a dot notation in json struct tags. If you expect multiple fields under fields and if you are sure that each field value is a string, then you can unmarshal your JSON into
type Todo struct {
Number string `json:"number"`
Issue []struct {
Key string `json:"key"`
Self string `json:"self"`
Fields map[string]string `json:"fields"`
} `json:"issues"`
}
If its only description everytime, then you can add
Fields struct {
Description string `json:"description"`
} `json:"fields"`
The majority of my development experience has been from dynamically typed languages like PHP and Javascript. I've been practicing with Golang for about a month now by re-creating some of my old PHP/Javascript REST APIs in Golang. I feel like I'm not doing things the Golang way most of the time. Or more generally, I'm not use to working with strongly typed languages. I feel like I'm making excessive use of map[string]interface{} and slices of them to box up data as it comes in from http requests or when it gets shipped out as json http output. So what I'd like to know is if what I'm about to describe goes against the philosophy of golang development? Or if I'm breaking the principles of developing with strongly typed languages?
Right now, about 90% of the program flow for REST Apis I've rewritten with Golang can be described by these 5 steps.
STEP 1 - Receive Data
I receive http form data from http.Request.ParseForm() as formvals := map[string][]string. Sometimes I will store serialized JSON objects that need to be unmarshaled like jsonUserInfo := json.Unmarshal(formvals["user_information"][0]) /* gives some complex json object */.
STEP 2 - Validate Data
I do validation on formvals to make sure all the data values are what I expect before using it in SQL queries. I treat everyting as a string, then use Regex to determine if the string format and business logic is valid (eg. IsEmail, IsNumeric, IsFloat, IsCASLCompliant, IsEligibleForVoting,IsLibraryCardExpired etc...). I've written my own Regex and custom functions for these types of validations
STEP 3 - Bind Data to SQL Queries
I use golang's database/sql.DB to take my formvals and bind them to my Query and Exec functions like this Query("SELECT * FROM tblUser WHERE user_id = ?, user_birthday > ? ",formvals["user_id"][0], jsonUserInfo["birthday"]). I never care about the data types I'm supplying as arguments to be bound, so they're all probably strings. I trust the validation in the step immediately above has determined they are acceptable for SQL use.
STEP 4 - Bind SQL results to []map[string]interface{}{}
I Scan() the results of my queries into a sqlResult := []map[string]interface{}{} because I don't care if the value types are null, strings, float, ints or whatever. So the schema of an sqlResult might look like:
sqlResult =>
[0] {
"user_id":"1"
"user_name":"Bob Smith"
"age":"45"
"weight":"34.22"
},
[1] {
"user_id":"2"
"user_name":"Jane Do"
"age":nil
"weight":"22.22"
}
I wrote my own eager load function so that I can bind more information like so EagerLoad("tblAddress", "JOIN ON tblAddress.user_id",&sqlResult) which then populates sqlResult with more information of the type []map[string]interface{}{} such that it looks like this:
sqlResult =>
[0] {
"user_id":"1"
"user_name":"Bob Smith"
"age":"45"
"weight":"34.22"
"addresses"=>
[0] {
"type":"home"
"address1":"56 Front Street West"
"postal":"L3L3L3"
"lat":"34.3422242"
"lng":"34.5523422"
}
[1] {
"type":"work"
"address1":"5 Kennedy Avenue"
"postal":"L3L3L3"
"lat":"34.3422242"
"lng":"34.5523422"
}
},
[1] {
"user_id":"2"
"user_name":"Jane Do"
"age":nil
"weight":"22.22"
"addresses"=>
[0] {
"type":"home"
"address1":"56 Front Street West"
"postal":"L3L3L3"
"lat":"34.3422242"
"lng":"34.5523422"
}
}
STEP 5 - JSON Marshal and send HTTP Response
then I do a http.ResponseWriter.Write(json.Marshal(sqlResult)) and output data for my REST API
Recently, I've been revisiting articles with code samples that use structs in places I would have used map[string]interface{}. For example, I wanted to refactor Step 2 with a more standard approach that other golang developers would use. So I found this https://godoc.org/gopkg.in/go-playground/validator.v9, except all it's examples are with structs . I also noticed that most blogs that talk about database/sql scan their SQL results into typed variables or structs with typed properties, as opposed to my Step 4 which just puts everything into map[string]interface{}
Hence, i started writing this question. I feel the map[string]interface{} is so useful because majority of the time,I don't really care what the data is and it gives me to the freedom in Step 4 to construct any data schema on the fly before I dump it as JSON http response. I do all this with as little code verbosity as possible. But this means my code is not as ready to leverage Go's validation tools, and it doesn't seem to comply with the golang community's way of doing things.
So my question is, what do other golang developers do with regards to Step 2 and Step 4? Especially in Step 4...do Golang developers really encourage specifying the schema of the data through structs and strongly typed properties? Do they also specify structs with strongly typed properties along with every eager loading call they make? Doesn't that seem like so much more code verbosity?
It really depends on the requirements just like you have said you don't require to process the json it comes from the request or from the sql results. Then you can easily unmarshal into interface{}. And marshal the json coming from sql results.
For Step 2
Golang has library which works on validation of structs used to unmarshal json with tags for the fields inside.
https://github.com/go-playground/validator
type Test struct {
Field `validate:"max=10,min=1"`
}
// max will be checked then min
you can also go to godoc for validation library. It is very good implementation of validation for json values using struct tags.
For STEP 4
Most of the times, We use structs if we know the format and data of our JSON. Because it provides us more control over the data types and other functionality. For example if you wants to empty a JSON feild if you don't require it in your JSON. You should use struct with _ json tag.
Now you have said that you don't care if the result coming from sql is empty or not. But if you do it again comes to using struct. You can scan the result into struct with sql.NullTypes. With that also you can provide json tag for omitempty if you wants to omit the json object when marshaling the data when sending a response.
Struct values encode as JSON objects. Each exported struct field
becomes a member of the object, using the field name as the object
key, unless the field is omitted for one of the reasons given below.
The encoding of each struct field can be customized by the format
string stored under the "json" key in the struct field's tag. The
format string gives the name of the field, possibly followed by a
comma-separated list of options. The name may be empty in order to
specify options without overriding the default field name.
The "omitempty" option specifies that the field should be omitted from
the encoding if the field has an empty value, defined as false, 0, a
nil pointer, a nil interface value, and any empty array, slice, map,
or string.
As a special case, if the field tag is "-", the field is always
omitted. Note that a field with name "-" can still be generated using
the tag "-,".
Example of json tags
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "-".
Field int `json:"-,"`
As you can analyze from above information given in Golang spec for json marshal. Struct provide so much control over json. That's why Golang developer most probably use structs.
Now on using map[string]interface{} you should use it when you don't the structure of your json coming from the server or the types of fields. Most Golang developers stick to structs wherever they can.
I imagine there must me a solution for this, I haven't found one however.
I receive JSON responses of this structure:
{
description: SomeString,
type: ACTIVITY,
timestamp: 1224043200000,
creationTime: 1224043200000,
userIdentification: 1111-2222-3333,
value: 2000
}
"Activity" is the name of my CoreData entity. How do I map this JSON to my data model? All examples I found so far assume a JSON similar to this format:
{
activity: {
description: SomeString,
timestamp: 1224043200000,
creationTime: 1224043200000,
userIdentification: 1111-2222-3333,
value: 2000
}
}
I'm assuming that you have multiple possible different types, each with an associated entity...
Define a mapping for each entity. Don't connect these directly to the response descriptors. Instead, create an RKDynamicMapping instance with a block (setObjectMappingForRepresentationBlock:, or a matcher) which checks the type of the data coming in and returns the appropriate mapping.
If your responses contain multiple different types in a single response then you may want to look at combining the above with KVC validation to reject any objects created with the wrong type. You would also need some different response descriptors or something to apply each of the different entity mappings...
You can look in the ObjectMapping section of the RestKit repo on github. To map your json response you can make use of the nil keyPath as explained here https://github.com/RestKit/RestKit/wiki/Object-mapping#mapping-values-without-key-paths.
As Wain said, you should use Dynamic Mapping to decide which class will be mapped with the json object that you are receiving depending on the value that certain attribute have. How to use Dynamic Mapping is explained here: https://github.com/RestKit/RestKit/wiki/Object-mapping#dynamic-object-mapping. I know it's late, but this could help somebody in the future.
I want to deserialise an object that includes an array of a some interface Entity:
type Result struct {
Foo int;
Bar []Entity;
};
Entity is an interface that is implemented by a number of struct types. JSON data identifies the struct type with a "type" field in each entity. E.g.
{"type":"t1","field1":1}
{"type":"t2","field2":2,"field3":3}
How would I go about deserialising the Result type in such a way that it correctly populates the array. From what I can see, I have to:
Implement UnmarshalJSON on Result.
Parse Bar as a []*json.RawMessage.
Parse each raw message as map[string]interface{}.
Check "type" field in the raw message.
Create a struct of appropriate type.
Parse the raw message again, this time into the just created struct.
This all sounds very tedious and boring. Is there a better way to do this? Or am I doing it backwards, and there is a more canonical method to handle an array of heterogeneous objects?
I think your process is probably a bit more complicated than it has to be, see http://play.golang.org/p/0gahcMpuQc. A single map[string]interface{} will handle a lot of that for you.
Alternatively, you could make a type like
struct EntityUnion {
Type string
// Fields from t1
// Fields from t2
// ...
}
Unmarshal into that; it will set the Type string and fill in all the fields it can get from the JSON data. Then you just need a small function to copy the fields to the specific type.