Finding the popular (frequent) number in Tcl - tcl

Question

It is pretty much to look for the "mode", which is the value that appears most frequently in a data set.
Here is my test code in TCL:
proc mode {list} {
foreach val $list {dict incr h $val}
set h [lsort -stride 2 -real -index 1 -decreasing $h]
return [lindex $h 0]
}
set a [list 0 0 0 0.4 0.4 0.4 0.4 0.4 0.1 0.2 0.4 0.35 0.29 0.19 0.15 0.45 0.39 0.39 0.39 0.39 0.39 0.39 0.39]
set m [mode $a]
puts $m
Is it a good/efficient way for a large dataset?
How to remove those "0" elements before the mode calculation?

Answer 1

Is it a good/ efficient way for a large dataset?
Define "large" and measure!
As pointed out to you by others, your combo of dict incr/ lsort -integer is a solid choice iff length was the only factor:
Length | Method 1 | Method 2 | Ratio (m1 / m2)
-------+----------+----------+----------------
46 | 4.4 | 7.9 | 0.55
92 | 7.2 | 14.6 | 0.49
184 | 12.7 | 28.2 | 0.45
368 | 23.8 | 55.1 | 0.43
736 | 46.2 | 108.5 | 0.43
1472 | 90.6 | 217.3 | 0.42
2944 | 180.2 | 428.3 | 0.42
5888 | 359.8 | 857.4 | 0.42
11776 | 715.9 | 1704.9 | 0.42
23552 | 1437.0 | 3408.9 | 0.42
47104 | 2878.2 | 6855.4 | 0.42
94208 | 5741.7 | 13664.4 | 0.42
Method 1:
proc mode {list} {
set h [dict create]
foreach val $list {dict incr h $val}
set h [lsort -stride 2 -integer -index 1 -decreasing $h]
return [lindex $h 0]
}
Method 2:
proc mode2 {list} {
set maxCount 0
set mode ""
foreach val $list {
dict incr h $val
set count [dict get $h $val]
if {$count > $maxCount} {
set maxCount $count
set mode $val
}
}
return $mode
}
You mentioned "real" numbers. In many cases, the distribution of levels/ bins (unique values) in a collection is even more important. Let's take the worst case that each measurement point is unique, so the length equals the number of levels/ bins:
Length | Method 1 | Method 2 | Ratio (m1 / m2)
-------+----------+----------+----------------
23 | 4.3 | 4.8 | 0.90
46 | 7.7 | 8.9 | 0.86
92 | 15.6 | 17.1 | 0.91
184 | 31.0 | 34.3 | 0.90
368 | 63.7 | 67.9 | 0.94
736 | 133.2 | 137.8 | 0.97
1472 | 300.8 | 300.8 | 1.00
2944 | 651.5 | 628.0 | 1.04
5888 | 1560.8 | 1310.8 | 1.19
11776 | 2886.6 | 2702.7 | 1.07
23552 | 6408.2 | 5654.6 | 1.13
47104 | 23331.4 | 19110.5 | 1.22
94208 | 69697.9 | 55569.2 | 1.25
... then lsort will start becoming overly heavy on your bill. Also, if you want to detect more than one mode (bimodal etc.), then the picture changes. In either case, Method 2 above might become a valid candidate for large and heterogeneous data sets (w/ and w/o multi modes).
This is the driver code for the above measurement tables:
namespace import tcl::unsupported::timerate
timerate -calibrate {}
proc r {} {expr {10+rand()*40}}
puts " Length | Method 1 | Method 2 | Ratio (m1 / m2)"
puts " -------+----------+----------+----------------"
set l 23
while {$l <= 100000} {
set a [list]
for {set i 0} {$i<$l} {incr i} { lappend a [r]}
set m1 [lindex [timerate {mode $a} 1000] 0]
set m2 [lindex [timerate {mode2 $a} 1000] 0]
set ratio [expr {double($m1) / double($m2)}]
puts [format " %6d | %8.1f | %8.1f | %9.2f" $l $m1 $m2 $ratio]
incr l $l
}
puts " Length | Method 1 | Method 2 | Ratio (m1 / m2)"
puts " -------+----------+----------+----------------"
set a [list 0 0 0 0.4 0.4 0.4 0.4 0.4 0.1 0.2 0.4 0.35 0.29 0.19 0.15 0.45 0.39 0.39 0.39 0.39 0.39 0.39 0.39]
while {[llength $a]*2 <= 100000} {
lappend a {*}$a
set m1 [lindex [timerate {mode $a} 1000] 0]
set m2 [lindex [timerate {mode2 $a} 1000] 0]
set ratio [expr {double($m1) / double($m2)}]
puts [format " %6d | %8.1f | %8.1f | %9.2f" [llength $a] $m1 $m2 $ratio]
}