Exception to the "brace your expr expressions" rule. What's going on? - tcl

Question

I have a number, say 10, in one variable and a string like +1 or -2 in another. I need to evaluate 10+1 or 10-2 in the above cases.
So, I have
set foo 10
set garp -1
If I do
expr $foo $garp
all is well (I get 9).
Ah! But in general, you should put braces around the expr expression.
expr {$foo $garp}
fails with missing operator at _#_ in expression $foo _#_$garp.
Similarly,
expr [concat $foo $garp]
works nicely but
expr {[concat $foo $garp]}
returns 10 -1.
I don't want to leave the expression unbraced without really understanding what's going on as I'm afraid that otherwise I, or someone else, is going to put braces around the expression and the code will stop working.
What's the "correct" way to do this?

Answer 1

In general, expr involves two rounds of substitution.
The first round of substitution is performed by the command parser on those arguments of the expr command that are not enclosed in braces. The resulting strings are concatenated (by adding separator spaces between them) into a single expression string, which is then parsed (and later evaluated) by the expression processor.
During parsing, the expression is decomposed into operators and operands. Operands must be delimited with operators. Assuming focus on mathematical expressions (i.e. discarding string operations), an operand may be one of the following:
a numeric value
a Tcl variable, using standard $ notation. The variable's value will be used as the operand.
a Tcl command enclosed in brackets. The command will be executed and its result will be used as the operand.
a parenthesized sub-expression, which is parsed using the same rules.
a mathematical function whose arguments are sub-expressions, parsed using the same rules.
Items 2 and 3 correspond to the second round of substitution, which is performed by the expression processor during evaluation. Each substitution performed at this step is expected to yield a numeric value that is directly usable in further evaluation, without needing to re-parse and re-evaluate it.
Having all this said, let's look at your examples:
expr $foo $garp
The command processor expands this during the 1st round of substitution to expr 10 -1, the expression string after concatenation of the arguments is {10 -1}, and the expression processor parses it into a valid expression 10 subtract 1.
expr [concat $foo $garp]
During the 1st round of substitution the command processor expands this to expr {10 -1}, effectively producing the same expression string as in the previous case.
expr {$foo $garp}
The command processor leaves this intact, and the expression processor sees two consecutive operands (corresponding to clause 2 above), without any operator between them.
expr {[concat $foo $garp]}
Again, the 1st round of substitution is not performed. Parsing this expression extracts a single operand [concat $foo $garp] corresponding to clause 3. Expression processor evaluates the command and substitutes its result (i.e. the string "10 -1") for the result of the full expression.
So the correctly braced version of your expression must read:
expr {$foo + $garp}
which will be parsed as $foo add $garp.

Answer 2

In this case,
expr {$foo + $garp}
The rule "always brace your expressions" stems from the fact that it is a good idea to bypass the argument evaluation step and leave the evaluation of the expression string completely to expr (because it is more secure1 and results in more effective bytecode2).
For this to work, the string passed to expr needs to be legal according to the rules laid out in the expr docs (an unbraced expression doesn't have to be legal as long as the argument evaluation step makes it legal). From this follows that anytime you need the argument evaluation to help you create a legal expression string is an exception to the "always brace" rule (and possibly a hint that you need to rethink the structure of your code3).
The string {$foo $garp} is illegal because variable substitutions can only be operands in an expression, meaning that we have two operands without an operator. The string "$foo $garp" is transformed by the argument evaluation into a legal expression as the minus operator is reinterpreted into a subtraction operator.
If you have a bunch of values that you are getting in pairs, a and b, and you want to add those, expr $a $b might work if you are sure that they always have a sign. That's brittle, though. It's better to use one of
expr {$a + $b}
tcl::mathop::+ $a $b
expr [join [list $a $b] +]
(The first one is the solution we've discussed above. The second one avoids double substitution by using the + operator outside of expr: the variables are evaluated by the argument evaluator but not by the command. The third variant has all the problems of double substitution and is mentioned mostly for completeness. It's still better than just expr $a $b, though.)
Documentation:
+ (operator),
expr,
join,
list,
Mathematical operators as Tcl commands
1) The argument evaluator, given hostile arguments, could for instance replace $foo in the expression with [exec rm -rf *] or whatever you crazy Linux kids call it, and then the command substitution will be performed inside expr. This is less likely to happen if you disallow double substitution by bracing the expression.
2) The byte compiler can analyze a braced expression and replace the call to expr with more efficient inlined calculations. For an unbraced string, the compiler has no other option than to set up a call to expr whatever the expression is.
3) Seemingly paradoxically, it is not a problem to construct an expression by some trusted method and pass it unbraced via a variable (set myexpr [...] ; expr $myexpr), because this way you are still in full control of the content of the expression, and you are certainly not depending on the argument evaluator to patch it up for you. You won't get the bytecode optimization, though.