Editing Json in Go without Unmarshalling into Structs First - json

I am having a bit of an issue. So I am writing a tool in go that works with some json files.
The way the tool works is the devops member who is using it is supposed to upload their json file into the specified folder within the project and then that json file is used from there to deploy an api in api-gateway (the json is actually a swagger with extensions but that isn't particularly important to my question)
The issue I am having is I need to update ONE line in the json. Each file passed in will be different, but it is guaranteed to have a url in the same spot every time, just due to the nature of the project. I need to update this url in an automated fashion.
Due to the fact the json files are different, setting up hard coded structs and unmarshalling in order to edit is out of the question. The objective is for the devops members to not even have to go into the code, but rather just to deploy their files, which is the reason I was hoping for this to be automated.
So far my research has yielded nothing. It appears that Go only supports editing json if it is first unmarshaled into structs (see Modifying JSON file using Golang). Is there a way to edit without the structs if i know for a fact what I am looking for will always be available within the json, despite each file being different?
This is only my first month using go, so there may be a simple solution. I have seen some mention of scanners from the megajson library, but I cannot seem to get that to work either
{
"paths": {
"/account": {
"post": {
"something": "body",
"api": {
"uri": "http://thisiswhereineedtoedit.com"
}
}
}
}
}

Unmarshal to interface{}. Walk down nested objects to find the object with the value to set. Set the value. Marshal back to JSON.
var root interface{}
if err := json.Unmarshal(d, &root); err != nil {
log.Fatal(err)
}
// Walk down path to target object.
v := root
var path = []string{"paths", "/account", "post", "api"}
for i, k := range path {
m, ok := v.(map[string]interface{})
if !ok {
log.Fatalf("map not found at %s", strings.Join(path[:i+1], ", "))
}
v, ok = m[k]
if !ok {
log.Fatalf("value not found at %s", strings.Join(path[:i+1], ", "))
}
}
// Set value in the target object.
m, ok := v.(map[string]interface{})
if !ok {
log.Fatalf("map not found at %s", strings.Join(path, ", "))
}
m["uri"] = "the new URI"
// Marshal back to JSON. Variable d is []byte with the JSON
d, err := json.Marshal(root)
if err != nil {
log.Fatal(err)
}
Replace calls to log.Fatal with whatever error handling is appropriate for your application.
playground example

One way you can solve this is by reading the file and changing it.
Lets say you have the file as you mentioned:
example.json
{
"paths": {
"/account": {
"post": {
"something": "body",
"api": {
"uri": "http://thisiswhereineedtoedit.com"
}
}
}
}
}
And we want to change the line with "uri" in it.
You should be more specific then I was in this example, make a placeholder or something - to avoid changing the wrong line.
You can use a small program that would look something like this:
package main
import (
"io/ioutil"
"log"
"strings"
)
func main() {
file := "./example.json"
url := "\"uri\": \"supreme-uri\""
// Read the file
input, err := ioutil.ReadFile(file)
if err != nil {
log.Fatalln(err)
}
// Split it into lines
lines := strings.Split(string(input), "\n")
// Find the line that contains our "placeholder" / "uri"
for i, line := range lines {
if strings.Contains(line, "\"uri\":") {
// Replace the line
lines[i] = "\"uri\": " + url
}
}
// Join lines and write to file
output := strings.Join(lines, "\n")
err = ioutil.WriteFile(file, []byte(output), 0644)
if err != nil {
log.Fatalln(err)
}
}
And after running the program our example.json file now looks like this:
{
"paths": {
"/account": {
"post": {
"something": "body",
"api": {
"uri": "supreme-uri"
}
}
}
}
}
Hope you find this solution useful, Good luck! :]

You can try filepath pkg from cross-plane runtime. You specify the JSON path and get or set the result you want like the example in the above link.

Related

How to correct forman JSON by GOlang?

I'm working with the Notion API and I can't use JSON correctly to send
My JSON:
{"parent":{"database_id":"123456"},"properties":{"Description":{"title":[{"text":{"content":"text1"}}]},"Value":{"rich_text":[{"text":{"content":"text2"}}]},"Status":{"rich_text":[{"text":{"content":"text4"
}}]},"Golang":{"rich_text":[{"text":{"content":"huinya"}}]},"Checkbox":{"Done?":true}}}
All correct Everything is correct except for the last paragraph.
"Checkbox":{"Done?":true}}}
Its must look like this :
{
"Done?": {
"checkbox": true
}
}
My data structure in Go big, and u can see it here go.dev
changind by place data structure
Add new type struct I put inside a map with string as key and bool as a value and in the tag add the json key that you want to have after marshalling
I hope this code can help you
type Done struct {
Done map[string]bool `json:"Done?"`
}
Js := Done{
Done: map[string]bool{
"checkbox": true,
},
}
b, err := json.Marshal(&Js)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))

check json array length without unmarshalling

Ive go a request body that is an json array of objects something like,
{
"data": [
{
"id": "1234",
"someNestedObject": {
"someBool": true,
"randomNumber": 488
},
"timestamp": "2021-12-13T02:43:44.155Z"
},
{
"id": "4321",
"someNestedObject": {
"someBool": false,
"randomNumber": 484
},
"timestamp": "2018-11-13T02:43:44.155Z"
}
]
}
I want to get a count of the objects in the array and split them into seperate json outputs to pass onto the next service. Im doing this atm by unmarshalling the original json request body and and then looping over the the elements marshalling each one again and attaching it to whatever outgoing message is being sent. Something like,
requestBodyBytes := []bytes(JSON_INPUT_STRING)
type body struct {
Foo []json.RawMessage `json:"foo"`
}
var inputs body
_ = json.Unmarshal(requestBodyBytes, &inputs)
for input := range inputs {
re, _ := json.Marshal(m)
... do something with re
}
What Im seeing though is the byte array of the before and after is different, even though the string representation is the same. I am wondering if there is a way to do this without altering the encoding or whatever is happening here to change the bytes to safeguard against any unwanted mutations? The actual json objects in the array will all have different shapes so I cant use a structured json definition with field validations to help.
Also, the above code is just an example of whats happening so if there are spelling or syntax errors please ignore them as the actual code works as described.
If you use json.RawMessage, the JSON source text will not be parsed but stored in it as-is (it's a []byte).
So if you want to distribute the same JSON array element, you do not need to do anything with it, you may "hand it over" as-is. You do not have to pass it to json.Marshal(), it's already JSON marshalled text.
So simply do:
for _, input := range inputs.Foo {
// input is of type json.RawMessage, and it's already JSON text
}
If you pass a json.RawMessage to json.Marshal(), it might get reencoded, e.g. compacted (which may result in a different byte sequence, but it will hold the same data as JSON).
Compacting might even be a good idea, as the original indentation might look weird taken out of the original context (object and array), also it'll be shorter. To simply compact a JSON text, you may use json.Compact() like this:
for _, input := range inputs.Foo {
buf := &bytes.Buffer{}
if err := json.Compact(buf, input); err != nil {
panic(err)
}
fmt.Println(buf) // The compacted array element value
}
If you don't want to compact it but to indent the array elements on their own, use json.Indent() like this:
for _, input := range inputs.Foo {
buf := &bytes.Buffer{}
if err := json.Indent(buf, input, "", " "); err != nil {
panic(err)
}
fmt.Println(buf)
}
Using your example input, this is how the first array element would look like (original, compacted and indented):
Orignal:
{
"id": "1234",
"someNestedObject": {
"someBool": true,
"randomNumber": 488
},
"timestamp": "2021-12-13T02:43:44.155Z"
}
Compacted:
{"id":"1234","someNestedObject":{"someBool":true,"randomNumber":488},"timestamp":"2021-12-13T02:43:44.155Z"}
Indented:
{
"id": "1234",
"someNestedObject": {
"someBool": true,
"randomNumber": 488
},
"timestamp": "2021-12-13T02:43:44.155Z"
}
Try the examples on the Go Playground.
Also note that if you do decide to compact or indent the individual array elements in the loop, you may create a simple bytes.Buffer before the loop, and reuse it in each iteration, calling its Buffer.Reset() method to clear the previous array's data.
It could look like this:
buf := &bytes.Buffer{}
for _, input := range inputs.Foo {
buf.Reset()
if err := json.Compact(buf, input); err != nil {
panic(err)
}
fmt.Println("Compacted:\n", buf)
}

Converting Protobuf3 with enum to JSON in Go

How can I convert grpc/protobuf3 message to JSON where the enum is represented as string?
For example, the protobuf message:
enum Level {
WARNING = 0;
FATAL = 1;
SEVERE = 2;
...
}
message Http {
string message = 1;
Level level = 2;
}
Is converted by:
j, _ := json.MarshalIndent(protoMessage, "", "\t")
To:
{
"message": "Hello world!",
"level": 2,
}
I wish to get:
{
"message": "Hello world!",
"level": "SEVERE",
}
Thanks
I found out that I should use the protobuf/jsonpb package and not the standard json package.
so:
j, _ := json.MarshalIndent(protoMessage, "", "\t")
Should be:
m := jsonpb.Marshaler{}
result, _ := m.MarshalToString(protoMessage)
Update
As noted bellow, jsonpb is depricated and the new solution is to use protojson
I found some of these modules (jsonpb) to be deprecated. What worked for me was the google encoding version:
import "google.golang.org/protobuf/encoding/protojson"
jsonString := protojson.Format(protoMessage)
Level is not a string though, it is an emum. There are really only two choices I see.
Write a custom marshaller that does this for you
Generate code that does this for you.
For #2, gogoprotobuf has an extension (still marked as experimental) that let's you do exactly this:
https://godoc.org/github.com/gogo/protobuf/plugin/enumstringer and
https://github.com/gogo/protobuf/blob/master/extensions.md
For my use case, I wanted to write it to a file. Using the most recent packages as of this date, this was as close to a regular encoding/json marshal as I could get.
I used the google.golang.org/protobuf/encoding/protojson package and the .ProtoReflect().Interface() methods of my protocol buffer data structure.
package main
import (
"io/ioutil"
"log"
"google.golang.org/protobuf/encoding/protojson"
"myproject/proto"
)
func main() {
myProtoStruct := proto.MyType{}
data, err := protojson.Marshal(myProtoStruct.ProtoReflect().Interface())
if err != nil {
log.Fatalf("Failed to JSON marhsal protobuf.\nError: %s", err.Error())
}
err = ioutil.WriteFile("my.proto.dat", data, 0600)
if err != nil {
log.Fatalf("Failed to write protobuf data to file.\nError: %s", err.Error())
}
log.Println("Written to file.")
}
When it comes to serialize the json object, this would be helpful.
var msg bytes.Buffer
m := jsonpb.Marshaler{}
err := m.Marshal(&msg, event)
msg.Bytes() converts msg to byte stream.
this is a very old question, but I will post my solution, maybe someone on the future will also face the same issue.
firstly, jsonpb is deprecated.
secondly, when you use protojson, you can set your options, which will affect the outcome JSON. In your case, the option UseEnumNumbers says how you want to marshall enums, strings or ints. Like so:
arr, err := protojson.MarshalOptions{
UseEnumNumbers: true, // uses enums as int, not as strings
}.Marshal(a)

Golang - Parsing JSON string arrays from Twitch TV RESTful service

I've been working on parsing a JSON object that I retrieve through an HTTP GET request using Go's built in HTTP library. I initially tried using the default JSON library in Go in order to do this, but I was having a difficult time (I am a novice in Go still). I eventually resorted to using a different library and had little trouble after that, as shown below:
package main
import (
"github.com/antonholmquist/jason"
"fmt"
"net/http"
)
func main() {
resp, err := http.Get("http://tmi.twitch.tv/group/user/deernadia/chatters")
if nil != err {
panic(err)
}
defer resp.Body.Close()
body, err := jason.NewObjectFromReader(resp.Body)
chatters, err := body.GetObject("chatters")
if nil != err {
panic(err)
}
moderators, err := chatters.GetStringArray("moderators")
if nil != err {
panic(err)
}
for _, moderator := range moderators {
fmt.Println(moderator)
}
}
Where github.com/antonholmquist/jason corresponds to the custom JSON library I used.
This code produces something similar to the following output when run in a Linux shell (the RESTful service will update about every 30 seconds or so, which means the values in the JSON object will potentially change):
antwan250
bbrock89
boxception22
cmnights
deernadia
fartfalcon
fijibot
foggythought
fulc_
h_ov
iceydefeat
kingbobtheking
lospollogne
nightbot
nosleeptv
octaviuskhan
pateyy
phosphyg
poisonyvie
shevek18
trox94
trox_bot
uggasmesh
urbanelf
walmartslayer
wift3
And the raw JSON looks similar to this (with some of the users removed for brevity):
{
"_links": {},
"chatter_count": 469,
"chatters": {
"moderators": [
"antwan250",
"bbrock89",
"boxception22",
"cmnights",
"deernadia",
"fartfalcon",
"fijibot",
"foggythought",
"fulc_",
"h_ov",
"iceydefeat",
"kingbobtheking",
"lospollogne",
"nightbot",
"nosleeptv",
"octaviuskhan",
"pateyy",
"phosphyg",
"poisonyvie",
"shevek18",
"trox94",
"trox_bot",
"uggasmesh",
"urbanelf",
"walmartslayer",
"wift3"
],
"staff": [
"tnose"
],
"admins": [],
"global_mods": [],
"viewers": [
"03xuxu30",
"0dominic0",
"3389942",
"812mfk",
"910dan",
"aaradabooti",
"admiralackbar99",
"adrian97lol",
"aequitaso_o",
"aethiris",
"afropigeon",
"ahhhmong",
"aizaix",
"aka_magosh",
"akitoalexander",
"alex5761",
"allenhei",
"allou_fun_park",
"amilton_tkm",
"... more users that I removed...",
"zachn17",
"zero_x1",
"zigslip",
"ziirbryad",
"zonato83",
"zorr03body",
"zourtv"
]
}
}
As I said before, I'm using a custom library hosted on Github in order to accomplish what I needed, but for the sake of learning, I'm curious... how would I accomplish this same thing using Go's built in JSON library?
To be clear, what I'd like to do is be able to harvest the users from each JSON array embedded within the JSON object returned from the HTTP GET request. I'd also like to be able to get the list of viewers, admins, global moderators, etc., in the same way, but I figured that if I can see the moderator example using the default Go library, then reproducing that code for the other user types will be trivial.
Thank you in advance!
If you want to unmarshal moderators only, use the following:
var v struct {
Chatters struct {
Moderators []string
}
}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
for _, mod := range v2.Chatters.Moderators {
fmt.Println(mod)
}
If you want to get all types of chatters, use the following:
var v struct {
Chatters map[string][]string
}
if err := json.Unmarshal(data, &v); err != nil {
handle error
}
for kind, users := range v1.Chatters {
for _, user := range users {
fmt.Println(kind, user)
}
}
run the code on the playground

Iterate over json array in Go to extract values

I have the following in my json array (conf.json file).
{
"Repos": [
"a",
"b",
"c"
]
}
I am attempting to read this json and then iterate over it but get stuck. I am very new to go (and to programming) so I am having a hard time understanding what is happening here.
import (
"encoding/json"
"fmt"
"os"
)
type Configuration struct {
Repos []string
}
func read_config() {
file, _ := os.Open("conf.json")
decoder := json.NewDecoder(file)
configuration := Configuration{}
err := decoder.Decode(&configuration)
if err != nil {
fmt.Println("error:", err)
}
fmt.Println(configuration.Repos)
}
So far this is as far as I have been able to get. This will print out the values okay, [a, b, c].
What I would like to do is be able to iterate over the array and split out each value individually but have not had any luck at doing this. Am I taking the wrong approach to this? Is there a better way to do this?
You mean something like this:
for _, repo := range configuration.Repos {
fmt.Println(repo)
}
Note that the code in your example should not work with the JSON that you have given. There is no mapping between value and Repos. You either have posted incorrect JSON or omitted a tag on the Configuration struct to map it correctly.
Everything worked fine, it's just your printing that is not doing what you expect. Since Repos is an array you'll have to iterate it to print each value individually. Try something like this;
for _, repo := range configuration.Repos {
fmt.Println(repo)
}