Create list of dicts within a dict in TCL - json

Question

I am forced to use TCL for something and I need to create a json string like this:
{ "mainKey": "mainValue", "subKey": [{"key1":"value1"},{"key2":"value2"}]}
So I am trying to do this:
set subDict1 [dict create key1 value1]
set subDict2 [dict create key2 value2]
set subDictList [list $subDict1 $subDict2]
set finalDict [dict create mainKey mainValue subKey $subDictList]
When I convert this dict to json, I get:
{"mainKey":"mainValue", "subKey":{"key1 value1":{"key2":"value2"}}}
instead of the required:
{ "mainKey": "mainValue", "subKey": [{"key1":"value1"},{"key2":"value2"}]}
What am I doing wrong?

Answer 1

First you have to understand that TCL is a very typeless language. What exactly are list and dicts in tcl?
In Tcl a list is a string that is properly formatted where each member of the list is separated by spaces (space, tab or newline) and if the data contained by an item contains spaces they can be escaped either by:
using backslash escaping:
"this is a list\ of\ four\ items"
using "" grouping:
{this is a "list of four items"}
using {} grouping:
{this is a {list of four items}}
Note that internally, once a string has been parsed as a list, Tcl uses a different internal data structure to store the list for speed. But semantically it is still a string. Just like HTML is a specially formatted string or JSON is a specially formatted string Tcl takes the attitude that lists are nothing but specially formatted strings.
So, what are dicts? In Tcl dicts are lists with even number of elements. That's it. Nothing special. A dict is therefore also semantically a string (though as mentioned above, once tcl sees you using that string as a dict it will compile it to a different data structure for optimizing speed).
Note again the core philosophy in tcl: almost all data structures (with the exception of arrays) are merely strings that happens to be formatted in a way that has special meaning.
This is the reason you can't auto-convert tcl data structures to JSON - if you ask Tcl to guess what the data structure is you end up with whatever the programmer who wrote the guessing function want it to be. In your case it looks like it defaults to always detecting lists with even number of elements as dicts.
So how can you generate JSON correctly?
There are several ways to do this. You can of course use custom dedicated for loops or functions to convert your data structure (which again, is just a specially formatted string) to JSON.
Several years ago I've written this JSON compiler:
# data is plain old tcl values
# spec is defined as follows:
# {string} - data is simply a string, "quote" it if it's not a number
# {list} - data is a tcl list of strings, convert to JSON arrays
# {list list} - data is a tcl list of lists
# {list dict} - data is a tcl list of dicts
# {dict} - data is a tcl dict of strings
# {dict xx list} - data is a tcl dict where the value of key xx is a tcl list
# {dict * list} - data is a tcl dict of lists
# etc..
proc compile_json {spec data} {
while [llength $spec] {
set type [lindex $spec 0]
set spec [lrange $spec 1 end]
switch -- $type {
dict {
lappend spec * string
set json {}
foreach {key val} $data {
foreach {keymatch valtype} $spec {
if {[string match $keymatch $key]} {
lappend json [subst {"$key":[
compile_json $valtype $val]}]
break
}
}
}
return "{[join $json ,]}"
}
list {
if {![llength $spec]} {
set spec string
} else {
set spec [lindex $spec 0]
}
set json {}
foreach {val} $data {
lappend json [compile_json $spec $val]
}
return "\[[join $json ,]\]"
}
string {
if {[string is double -strict $data]} {
return $data
} else {
return "\"$data\""
}
}
default {error "Invalid type"}
}
}
}
(See http://wiki.tcl.tk/JSON for the original implementation and discussion of JSON parsing)
Because tcl can never correctly guess what your "string" is I've opted to supply a format string to the function in order to correctly interpret tcl data structures. For example, using the function above to compile your dict you'd call it like this:
compile_json {dict subKey list} finalDict
I've begged the tcllib maintainers to steal my code because I still believe it's the correct way to handle JSON in tcl but so far it's still not in tcllib.
BTW: I license the code above as public domain and you or anyone may claim full authorship of it if you wish.

Answer 2

It's not completely wrong to say that Tcl is a typeless language, because the types of the data objects in a Tcl program aren't expressed fully in the code, and not always even in the Tcl_Obj structures that represent data objects internally. Still, types are certainly not absent from a Tcl program, it's just that the type system is a lot less intrusive in Tcl than in most other programming languages.
The complete type definition in a Tcl program emerges from a dynamic combination of code and data objects as the program executes. The interpreter trusts you to tell it how you want your data objects to behave.
As an example, consider the following string:
set s {title: Mr. name: Peter surname: Lewerin}
Is this a string, an array, or a dictionary? All of the above, actually. (At least it's not an integer, a double or a boolean, other possible Tcl types.)
Using this string, I can answer a number of questions:
Tell me about your name
puts $s
# => title: Mr. name: Peter surname: Lewerin
What do polite people call you?
puts [dict values $s]
# => Mr. Peter Lewerin
What was your last name again?
puts [lindex $s end]
# => Lewerin
Here, I used the same string as a string, as a dictionary, and as an array. The same string representation was used for all three types of object, and it was the operations I used on it that determined the type of the object in that precise moment.
Similarly, the literal 1 can mean the integer 1, the single-character string 1, or boolean truth. There is no way to specify which kind of 1 you mean, but there is no need either, since the interpreter won't complain about the ambiguity.
Because Tcl doesn't store complete type information, it's quite hard to serialize arbitrary collections of data objects. That doesn't mean Tcl can't play well with serialization, though: you just need to add annotations to your data.
This string:
di [dm [st mainKey] [st mainValue]] [dm [st subKey] [ar [di [dm [st key1] [st value1]]] [di [dm [st key2] [st value2]]]]]
can be fed into the Tcl interpreter, and given the proper definitions of di, dm, st, and ar (which I intend to signify "dictionary", "dictionary member", "string", and "array", respectively), I can have the string construct a dictionary structure equivalent to the one in the question, or the string representation of such an object, just a bare list of keys and values, or XML, or JSON, etc. By using namespaces and/or slave interpreters, I can even dynamically switch between various forms. I won't provide examples for all forms, just JSON:
proc di args {return "{[join $args {, }]}"}
proc st val {return "\"$val\""}
proc ar args {return "\[[join $args {, }]]"}
proc dm {k v} {return "$k: $v"}
The output becomes:
{"mainKey": "mainValue", "subKey": [{"key1": "value1"}, {"key2": "value2"}]}
This example used the command nesting of the Tcl interpreter to define the structure of the data. Tcl doesn't need even that: a list of token classes and tokens such as a scanner would emit will suffice:
< : ' mainKey ' mainValue : ' subKey ( < : ' key1 ' value1 > < : ' key2 ' value2 > ) >
Using these simple commands:
proc jsonparseseq {endtok args} {
set seq [list]
while {[lsearch $args $endtok] > 0} {
lassign [jsonparseexpr {*}$args] args expr
lappend seq $expr
}
list [lassign $args -] $seq
}
proc jsonparseexpr args {
set args [lassign $args token]
switch -- $token {
' {
set args [lassign $args str]
set json \"$str\"
}
: {
lassign [jsonparseexpr {*}$args] args key
lassign [jsonparseexpr {*}$args] args val
set json "$key: $val"
}
< {
lassign [jsonparseseq > {*}$args] args dict
set json "{[join $dict {, }]}"
}
( {
lassign [jsonparseseq ) {*}$args] args arr
set json "\[[join $arr {, }]]"
}
}
list $args $json
}
proc jsonparse args {
lindex [jsonparseexpr {*}$args] end
}
I can parse that stream of token classes (<, (, ', :, ), >) and tokens into the same JSON string as above:
jsonparse < : ' mainKey ' mainValue : ' subKey ( < : ' key1 ' value1 > < : ' key2 ' value2 > ) >
# -> {"mainKey": "mainValue", "subKey": [{"key1": "value1"}, {"key2": "value2"}]}
Tcl offers quite a lot of flexibility; few languages will be as responsive to the programmer's whim as Tcl.
For completeness I will also demonstrate using the Tcllib huddle package mentioned by slebetman to create a the kind of structure mentioned in the question, and serialize that into JSON:
package require huddle
# -> 0.1.5
set subDict1 [huddle create key1 value1]
# -> HUDDLE {D {key1 {s value1}}}
set subDict2 [huddle create key2 value2]
# -> HUDDLE {D {key2 {s value2}}}
set subDictList [huddle list $subDict1 $subDict2]
# -> HUDDLE {L {{D {key1 {s value1}}} {D {key2 {s value2}}}}}
set finalDict [huddle create mainKey mainValue subKey $subDictList]
# -> HUDDLE {D {mainKey {s mainValue} subKey {L {{D {key1 {s value1}}} {D {key2 {s value2}}}}}}}
huddle jsondump $finalDict {} {}
# -> {"mainKey":"mainValue","subKey":[{"key1":"value1"},{"key2":"value2"}]}
Another approach is to create regular Tcl structures and convert ("compile") them to huddle data according to a type specification:
set subDict1 [dict create key1 value1]
set subDict2 [dict create key2 value2]
set subDictList [list $subDict1 $subDict2]
set finalDict [dict create mainKey mainValue subKey $subDictList]
huddle compile {dict mainKey string subKey {list {dict * string}}} $finalDict
The result of the last command is the same as of the last huddle create command in the previous example.
Documentation: dict, join, lappend, lassign, lindex, list, lsearch, proc, puts, return, set, switch, while