Hello i'am trying to unmarshal object that contain ECC public key
but i go umarshaling error saying that can't unmarshal object properly.
i'am tring to do the followeing :
var privateKey *ecdsa.PrivateKey
var publicKey ecdsa.PublicKey
privateKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
log.Fatalf("Failed to generate ECDSA key: %s\n", err)
}
publicKey = privateKey.PublicKey
marshalledKey, err := json.Marshal(publicKey)
if err != nil {
panic(err)
}
var unmarshalledKey ecdsa.PublicKey
err2 := json.Unmarshal(marshalledKey, &unmarshalledKey)
if err2 != nil {
panic(err2)
}
and tthe error returned from (err2) is :
panic: json: cannot unmarshal object into Go struct field PublicKey.Curve of type elliptic.Curve
and there is no way in either eliptoc or 509x function to umarshal and the curve value is alwayes null
i found the answer finally and i will post it in case somone need it.
all we need is just to creat another struct, lets name it "retrieve" and declare the value it wil read from the unmarshal.
now the ECC public key contain 3 values "curve" and will append it as json:"Curve", "Y" ans will append it as json:"Y" and we will do the same for "X"
and it will work like this :
type retrieve struct {
CurveParams *elliptic.CurveParams `json:"Curve"`
MyX *big.Int `json:"X"`
MyY *big.Int `json:"Y"`
}
//UnmarshalECCPublicKey extract ECC public key from marshaled objects
func UnmarshalECCPublicKey(object []byte) (pub ecdsa.PublicKey) {
var public ecdsa.PublicKey
rt := new(retrieve)
errmarsh := json.Unmarshal(object, &rt)
if errmarsh != nil {
fmt.Println("err at UnmarshalECCPublicKey()")
panic(errmarsh)
}
public.Curve = rt.Key.CurveParams
public.X = rt.Key.MyX
public.Y = rt.Key.MyY
mapstructure.Decode(public, &pub)
fmt.Println("Unmarshalled ECC public key : ", pub)
return
}
we will pass the the marshalled public key to UnmarshalECCPublicKey() and it will return the correct unmarshaled obj
so the coorect use will be :
var privateKey *ecdsa.PrivateKey
var publicKey ecdsa.PublicKey
privateKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
log.Fatalf("Failed to generate ECDSA key: %s\n", err)
}
publicKey = privateKey.PublicKey
marshalledKey, err := json.Marshal(publicKey)
if err != nil {
panic(err)
}
var unmarshalledKey ecdsa.PublicKey
unmarshalledKey = UnmarshalECCPublicKey(marshalledKey)
and that should do it, u will get the ECC public unmarshalled successfully.
Related
I have a json response like this
{
"foo" : "bar",
"object" : {
"type" : "action",
"data" : "somedata"
}
}
Here the object could be one of multiple types. I define the types and have them implement a common interface.
type IObject interface {
GetType() string
}
type Action struct {
Type string `json:"type"`
Data string `json:"data"`
}
func (a Action) GetType() string {
return "action"
}
type Activity struct {
Type string `json:"type"`
Duration int `json:"duration"`
}
func (a Activity) GetType() string {
return "activity"
}
And a response struct
type Response struct {
Foo string `json:"foo"`
Object IObject `json:"object"`
}
As the type information of a struct that implements IObject is contained within the struct, there is no way to learn in without unmarshaling. I also cannot change the structure of the json response. Currently I am dealing with this problem using a custom unmarshaller:
func UnmarshalObject(m map[string]interface{}, object *IObject) error {
if m["type"] == "action" {
b, err := json.Marshal(m)
if err != nil {
return err
}
action := Action{}
if err = json.Unmarshal(b, &action); err != nil {
return err
}
*object = action
return nil
}
if m["type"] == "activity" {
b, err := json.Marshal(m)
if err != nil {
return err
}
activity := Activity{}
if err = json.Unmarshal(b, &activity); err != nil {
return err
}
*object = activity
return nil
}
return errors.New("unknown actor type")
}
func (r *Response) UnmarshalJSON(data []byte) error {
raw := struct {
Foo string `json:"foo"`
Object interface{} `json:"object"`
}{}
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
r.Foo = raw.Foo
if err = UnmarshalObject(raw.Object.(map[string]interface{}), &r.Object); err != nil
{
return err
}
return nil
}
So what I do is basically
Unmarshall the object into an interface{}
Typecast to map[string]interface{}
Read the "type" value to determine the type
Create a new instance of the determined type
Marshal back to json
Unmarshal again to the new instance of the determined type
Assign the instance to the field
This feels off and I am not comfortable with it. Especially the marshaling/unmarshaling back and forth. Is there a more elegant way to solve this problem?
You can use json.RawMessage.
func (r *Response) UnmarshalJSON(data []byte) error {
var raw struct {
Foo string `json:"foo"`
Object json.RawMessage `json:"object"`
}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
r.Foo = raw.Foo
var obj struct {
Type string `json:"type"`
}
if err := json.Unmarshal(raw.Object, &obj); err != nil {
return err
}
switch obj.Type {
case "action":
r.Object = new(Action)
case "activity":
r.Object = new(Activity)
}
return json.Unmarshal(raw.Object, r.Object)
}
https://go.dev/play/p/6dqiybS4zNp
I need to parse this json
{
"version": "1.1.29-snapshot",
"linux-amd64": {
"url": "https://origin/path",
"size": 7794688,
"sha256": "14b3c3ad05e3a98d30ee7e774646aec7ffa8825a1f6f4d9c01e08bf2d8a08646"
},
"windows-amd64": {
"url": "https://origin/path",
"size": 8102400,
"sha256": "01b8b927388f774bdda4b5394e381beb592d8ef0ceed69324d1d42f6605ab56d"
}
}
Keys like linux-amd64 are dynamic and theirs amount is arbitrary. I tried something like that to describe it and unmarshal. Obviously it doesn't work. Items is always empty.
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
It's similar to this use case, but has no parent key items. I suppose I can use 3rd party library or map[string]interface{} approach, but I'm interested in knowing how to achieve this with explicitly declared types.
The rest of the parsing code is:
func parseUpdateJson(jsonStr []byte) (UpdateInfo, error) {
var allInfo = UpdateInfo{Items: make(map[string]FileInfo)}
var err = json.Unmarshal(jsonStr, &allInfo)
return allInfo, err
}
Look at the link I attached and you will realize that is not that simple as you think. Also I pointed that I interested in typed approach. Ok, how to declare this map[string]FileInfo to get parsed?
You can create a json.Unmarshaller to decode the json into a map, then apply those values to your struct: https://play.golang.org/p/j1JXMpc4Q9u
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
func (i *UpdateInfo) UnmarshalJSON(d []byte) error {
tmp := map[string]json.RawMessage{}
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
err = json.Unmarshal(tmp["version"], &i.Version)
if err != nil {
return err
}
delete(tmp, "version")
i.Items = map[string]FileInfo{}
for k, v := range tmp {
var item FileInfo
err := json.Unmarshal(v, &item)
if err != nil {
return err
}
i.Items[k] = item
}
return nil
}
This answer is adapted from this recipe in my YouTube video on advanced JSON handling in Go.
func (u *UpdateInfo) UnmarshalJSON(d []byte) error {
var x struct {
UpdateInfo
UnmarshalJSON struct{}
}
if err := json.Unmarshal(d, &x); err != nil {
return err
}
var y map[string]json.RawMessage{}
if err := json.Unsmarshal(d, &y); err != nil {
return err
}
delete(y, "version"_ // We don't need this in the map
*u = x.UpdateInfo
u.Items = make(map[string]FileInfo, len(y))
for k, v := range y {
var info FileInfo
if err := json.Unmarshal(v, &info); err != nil {
return err
}
u.Items[k] = info
}
return nil
}
It:
Unmarshals the JSON into the struct directly, to get the struct fields.
It re-unmarshals into a map of map[string]json.RawMessage to get the arbitrary keys. This is necessary since the value of version is not of type FileInfo, and trying to unmarshal directly into map[string]FileInfo will thus error.
It deletes the keys we know we already got in the struct fields.
It then iterates through the map of string to json.RawMessage, and finally unmarshals each value into the FileInfo type, and stores it in the final object.
If you really don't want to unmarshal multiple times, your next best option is to iterate over the JSON tokens in your input by using the json.Decoder type. I've done this in a couple of performance-sensitive bits of code, but it makes your code INCREDIBLY hard to read, and in almost all cases is not worth the effort.
I have a database table called http_requests. I have modelled following struct to represent rows in this table.
type Map map[string]interface{}
type HTTPRequest struct {
ID int64 `json:"id" gorm:"id"`
RequestURL string `json:"request_url,omitempty" gorm:"request_url"`
RequestParams *RequestParams `json:"request_params,omitempty" gorm:"request_params"`
}
// RequestParams is another struct that holds params from body and URL query
type RequestParams struct {
FromBody Map `json:"body,omitempty"`
FromQuery Map `json:"query,omitempty"`
}
Code to save HTTPRequest:
request := &HTTPRequest{
RequestURL: "dummy/url",
RequestParams: &RequestParams{FromBody: Map{"param1": "value1"}},
}
if err := gorm.DB.Create(request).Error; err != nil {
return err
}
When I try to save this HTTPRequest it results in error:
sql: Scan error on column index 9, name "request_params": unsupported Scan, storing driver.Value type []uint8 into type *RequestParams
I would like to have request_params column to store JSON like this:
{"body":{"param1":"value1"}, "query": {"param2" : "value2"} }
or
{"body":{"param1":"value1"}}
or
{"query": {"param2" : "value2"} }
And this should get parsed into RequestParams struct when reading from database.
As suggested by #mkopriva, I implemented Scan() and Value() methods for my RequestParams type. See code below.
import (
"database/sql/driver"
"encoding/json"
"strings"
)
// Value converts RequestParams to a map
func (reqParams RequestParams) Value() (driver.Value, error) {
reqMap, err := reqParams.ToMap()
if err != nil {
return nil, err
}
return reqMap.ForceJSON(), nil
}
// Scan converts value to RequestParams
func (reqParams *RequestParams) Scan(value interface{}) error {
// set empty struct by default
*reqParams = RequestParams{}
if value == nil {
return nil
}
if s, ok := value.([]byte); ok {
d := json.NewDecoder(strings.NewReader(string(s)))
d.UseNumber()
rp := &RequestParams{}
if err := d.Decode(rp); err == nil {
*reqParams = *rp
}
}
return nil
}
Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing
Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing