Effectively remove invalid characters from JSON file? - json

Question

I am reading in a file via the command line.
As the file is a JSON export from Oracle, it has a certain structure. This default structure is not valid JSON for some reason. Example:
// This isn't valid JSON
,"items":
[
{"id":123,"language":"ja-JP","location":"Osaka"}
,{"id":33,"language":"ja-JP","location":"Tokyo"}
,{"id":22,"language":"ja-JP","location":"Kentok"}
]}
I wish for it to only be an array of object, thus having the expected output:
// This is valid json
[
{"id":123,"language":"ja-JP","location":"Osaka"}
,{"id":33,"language":"ja-JP","location":"Tokyo"}
,{"id":22,"language":"ja-JP","location":"Kentok"}
]
Therefore, I need to remove line 1(entirely) as well as the last } from the last line of the file.
The file is being parsed via commandline from the input:
file, err := ioutil.ReadFile(os.Args[1])
I am trying to remove the invalid strings/words this way, but it does not reformat anything:
// in func main()
removeInvalidJSON(file, os.Args[1])
// later on ..
func removeInvalidJSON(file []byte, path string) {
info, _ := os.Stat(path)
mode := info.Mode()
array := strings.Split(string(file), "\n")
fmt.Println(array)
//If we have the clunky items array which is invalid JSON, remove the first line
if strings.Contains(array[0], "items") {
fmt.Println("Removing items")
array = append(array[:1], array[1+1:]...)
}
// Finds the last index of the array
lastIndex := array[len(array)-1]
// If we have the "}" in the last line, remove it as this is invalid JSON
if strings.Contains(lastIndex, "}") {
fmt.Println("Removing }")
strings.Trim(lastIndex, "}")
}
// Nothing changed?
fmt.Println(array)
ioutil.WriteFile(path, []byte(strings.Join(array, "\n")), mode)
}
The above function does write to the file I can see - but it does not alter the array as far as I can tell, and does not write it into the file.
How do I effectively remote the first line of the file, as well as the last false curly brace } from the file?
I unmarshall the JSON in another function: Is there a method of doing it more "cleanly" using the "encoding/json" library?

Answer 1

There are several significant issues with this code causing it to behave not as intended. I've noted these with comments below:
func removeInvalidJSON(file []byte, path string) {
info, _ := os.Stat(path)
mode := info.Mode()
array := strings.Split(string(file), "\n")
fmt.Println(array)
//If we have the clunky items array which is invalid JSON, remove the first line
if strings.Contains(array[0], "items") {
fmt.Println("Removing items")
// If you just want to remove the first item, this should be array = array[1:].
// As written, this appends the rest of the array to the first item, i.e. nothing.
array = append(array[:1], array[1+1:]...)
}
// Finds the last ~index~ *line* of the array
lastIndex := array[len(array)-1]
// If we have the "}" in the last line, remove it as this is invalid JSON
if strings.Contains(lastIndex, "}") {
fmt.Println("Removing }")
// Strings are immutable. `strings.Trim` does nothing if you discard the return value
strings.Trim(lastIndex, "}")
// After the trim, if you want this to have any effect, you need to put it back in `array`.
}
// Nothing changed?
fmt.Println(array)
ioutil.WriteFile(path, []byte(strings.Join(array, "\n")), mode)
}
I think what you want is something more like:
func removeInvalidJSON(file []byte, path string) {
info, _ := os.Stat(path)
mode := info.Mode()
array := strings.Split(string(file), "\n")
fmt.Println(array)
//If we have the clunky items array which is invalid JSON, remove the first line
if strings.Contains(array[0], "items") {
fmt.Println("Removing items")
array = array[1:]
}
// Finds the last line of the array
lastLine := array[len(array)-1]
array[len(array)-1] = strings.Trim(lastLine, "}")
fmt.Println(array)
ioutil.WriteFile(path, []byte(strings.Join(array, "\n")), mode)
}