I want to display a large number of columns in html (> 10k Columns and max 200-300 rows) to display a schedule.
The content will be some blocks that are moveable with some text in it and a background color.
Is it possible to display such a large amount of columns with a table/tr/td? Is there a max number of columns or rows that can be displayed with table/tr/td?
Or would you use a canvas? Any other suggestions to solve this?
What is the best way (with great performance when scrolling, ...) to display such a large table?
With this many columns and rows (representing about a minimum of 3,000,000 entries) you will probably avoid filling DOM with those at all cost as any change would trigger reflow which would in this case take a significant amount of time. The number above is also for entries, if each entry takes 100 bytes to represent you are already up at about 280 mb which the browser has to manage.
Normally you would buffer the drawing so only the visible part is drawn/shown on the screen. This means of course you have to implement a custom manager for this. Whether you draw to canvas or use DOM elements sparingly will be up to you.
A challenge with this many columns is to get a scroll bar that is detailed enough to represent a single column, which it likely won't (except from the arrow buttons on the scroll bar), or you would need a very wide screen area to render to.
In addition to this you will also have to consider buffering the data itself by only loading a chunk of it into a buffer. How big that buffer needs to be depends on many factor including connection speed, server, location, data type, browser, scope etc. I would not recommend loading this amount into memory though as it can affect the client system severely resource-wise.
The example given below is just a rough implementation of the rendering principles (buffering) to show what I mean (run it in "Full page" to access the scrollbar intended for the demo):
var ctx = c.getContext("2d"),
rowWidth = 70, // max width of each column
count = Math.ceil(c.width / rowWidth), // number of rows that fit
entries = 10000, // demo
rows = 19,
columns = [];
// Create some dummy columns and row (10,000 x 19)
for(var x = 0; x < entries; x++) {
var row = [];
for(var y = 0; y < rows; y++) row.push(Math.round(Math.random() * 10000));
columns.push(row);
}
// calc scroller size in pixels
document.querySelector("dummy").style.width = (entries * rowWidth) + "px";
// handle updates
document.getElementById("scroll").onscroll = function() {render(+this.scrollLeft)};
// Render rows based on scroller position (rough calcs just for demo)
function render(scrollX) {
var first = Math.floor(scrollX / rowWidth); // first row to display
ctx.clearRect(0,0,c.width,c.height);
// render based on scroller offset and how many rows fit into canvas
for(var x = first, offset = 0; x < first + count; x++, offset += rowWidth) {
if (columns[x]) {
ctx.fillText("Header " + x, offset, 15);
for(var y = 0; y < rows; y++) ctx.fillText(columns[x][y], offset, y * 20 + 35);
}
}
}
render(0); // initial render
#c {background:#ddd}
#scroll {
max-width:700px;
overflow-x:scroll;
height:16px;
}
dummy {display:inline-block}
<div>
<canvas id=c width=700 height=400></canvas>
<div id=scroll><dummy></dummy></div>
</div>
(my code is written in Java but the question is agnostic; I'm just looking for an algorithm idea)
So here's the problem: I made a method that simply finds the median of a data set (given in the form of an array). Here's the implementation:
public static double getMedian(int[] numset) {
ArrayList<Integer> anumset = new ArrayList<Integer>();
for(int num : numset) {
anumset.add(num);
}
anumset.sort(null);
if(anumset.size() % 2 == 0) {
return anumset.get(anumset.size() / 2);
} else {
return (anumset.get(anumset.size() / 2)
+ anumset.get((anumset.size() / 2) + 1)) / 2;
}
}
A teacher in the school that I go to then challenged me to write a method to find the median again, but without using any data structures. This includes anything that can hold more than one value, so that includes Strings, any forms of arrays, etc. I spent a long while trying to even conceive of an idea, and I was stumped. Any ideas?
The usual algorithm for the task is Hoare's Select algorithm. This is pretty much like a quicksort, except that in quicksort you recursively sort both halves after partitioning, but for select you only do a recursive call in the partition that contains the item of interest.
For example, let's consider an input like this in which we're going to find the fourth element:
[ 7, 1, 17, 21, 3, 12, 0, 5 ]
We'll arbitrarily use the first element (7) as our pivot. We initially split it like (with the pivot marked with a *:
[ 1, 3, 0, 5, ] *7, [ 17, 21, 12]
We're looking for the fourth element, and 7 is the fifth element, so we then partition (only) the left side. We'll again use the first element as our pivot, giving (using { and } to mark the part of the input we're now just ignoring).
[ 0 ] 1 [ 3, 5 ] { 7, 17, 21, 12 }
1 has ended up as the second element, so we need to partition the items to its right (3 and 5):
{0, 1} 3 [5] {7, 17, 21, 12}
Using 3 as the pivot element, we end up with nothing to the left, and 5 to the right. 3 is the third element, so we need to look to its right. That's only one element, so that (5) is our median.
By ignoring the unused side, this reduces the complexity from O(n log n) for sorting to only O(N) [though I'm abusing the notation a bit--in this case we're dealing with expected behavior, not worst case, as big-O normally does].
There's also a median of medians algorithm if you want to assure good behavior (at the expense of being somewhat slower on average).
This gives guaranteed O(N) complexity.
Sort the array in place. Take the element in the middle of the array as you're already doing. No additional storage needed.
That'll take n log n time or so in Java. Best possible time is linear (you've got to inspect every element at least once to ensure you get the right answer). For pedagogical purposes, the additional complexity reduction isn't worthwhile.
If you can't modify the array in place, you have to trade significant additional time complexity to avoid avoid using additional storage proportional to half the input's size. (If you're willing to accept approximations, that's not the case.)
Some not very efficient ideas:
For each value in the array, make a pass through the array counting the number of values lower than the current value. If that count is "half" the length of the array, you have the median. O(n^2) (Requires some thought to figure out how to handle duplicates of the median value.)
You can improve the performance somewhat by keeping track of the min and max values so far. For example, if you've already determined that 50 is too high to be the median, then you can skip the counting pass through the array for every value that's greater than or equal to 50. Similarly, if you've already determined that 25 is too low, you can skip the counting pass for every value that's less than or equal to 25.
In C++:
int Median(const std::vector<int> &values) {
assert(!values.empty());
const std::size_t half = values.size() / 2;
int min = *std::min_element(values.begin(), values.end());
int max = *std::max_element(values.begin(), values.end());
for (auto candidate : values) {
if (min <= candidate && candidate <= max) {
const std::size_t count =
std::count_if(values.begin(), values.end(), [&](int x)
{ return x < candidate; });
if (count == half) return candidate;
else if (count > half) max = candidate;
else min = candidate;
}
}
return min + (max - min) / 2;
}
Terrible performance, but it uses no data structures and does not modify the input array.
I got a numberArray.
It contains intergers - randomised, within a specific range.
I want to get a specific sum, but not for everything inside the numberArray,
more of trying to sum up different amount of numbers (total of 5 only) inside the numberArray and see if it'll get the specific total required. and if not, it'll randomise another number to take over one of the numbers inside the numberArray.
What's the easiest way to do this ?
doing lots of
if (numberArray[1] + numberArray[2] == specificNumber)
{
}
if (numberArray[1] + numberArray[3] == specificNumber)
{
}
etc. etc. etc.
have too many lines of codes, and it seems like there are easier codes. right now i only have 5 different numbers in the array, so it's still bearable, but if the amount of numbers are higher.... ....
Reading your question like this: For your array of random integers, find a (or all) set(s) of integers that have a given sum.
This is an NP-Complete problem - i.e. there's no known algorithm that solves it efficiently.
The fastest known way is rather complex, so we'll go with a naive solution - should be good enough if you're not doing this on every frame or the input set is huge.
This should also work with 0 or negative values in the input set.
// The sum we're looking for:
var requiredSum:int = 8;
// Our input set:
var numberArray:Array = [1, 2, 3, 4, 5, 2, 3];
// Results will be stored here:
var resultSets:Array = [];
// Go through all possible subset sizes.
// This allows subset sizes all the way up to the size of
// the input set (numberArray.length).
// You can modify it to a fixed value (say, 5), of course:
for (var subsetSize:int = 1; subsetSize <= numberArray.length; subsetSize++)
{
// We'll use the same array for all our attempts of this size:
var subset:Array = new Array(subsetSize);
findSum(numberArray, subset, 0, 0);
}
// Output results:
for (var i:int = 0; i < resultSets.length; i++)
{
trace(resultSets[i].join("+"));
}
// numberArray : Our input set
// subset : The set we're currently filling
// setIndex : The position we're at in numberArray
// subsetIndex : The position we're at in the set we're filling
function findSum(numberArray:Array, subset:Array, setIndex:int,
subsetIndex:int):void
{
// Try every value from the input set starting from our current position,
// and insert the value at the current subset index:
for (var index:int = setIndex ; index < numberArray.length; index++)
{
subset[subsetIndex] = numberArray[index];
// Have we filled the subset?
if (subsetIndex == subset.length - 1)
{
var sum:int = 0;
for (var i:int = 0; i < subset.length; i++)
{
sum += subset[i];
}
if (sum == requiredSum)
{
// Clone the array before adding it to our results,
// since we'll be modifying it if we find more:
resultSets.push(subset.concat());
}
}
else
{
// Recursion takes care of combining our subset so far
// with every possible value for the remaining subset indices:
findSum(numberArray, subset, index + 1, subsetIndex + 1);
}
}
}
Output for the values used in the above code:
3+5
5+3
1+2+5
1+3+4
1+4+3
1+5+2
2+3+3
2+4+2
3+2+3
1+2+3+2
1+2+2+3
If we only need to know IF a sum exists, there's no need for the result set - we just return true/false, and break out of the recursive algorithm completely when a sum has been found:
var requiredSum:int = 8;
var numberArray:Array = [1, 2, 3, 4, 5, 2, 3];
// Go through all possible subset sizes:
for (var subsetSize:int = 1; subsetSize <= numberArray.length; subsetSize++)
{
// We'll use the same array for all our attempts of this size:
var subset:Array = new Array(subsetSize);
if (findSum(numberArray, subset, 0, 0))
{
trace("Found our sum!");
// If we found our sum, no need to look for more sets:
break;
}
}
// numberArray : Our input set
// subset : The set we're currently filling
// setIndex : The position we're at in numberArray
// subsetIndex : The position we're at in the set we're filling
// RETURNS : True if the required sum was found, otherwise false.
function findSum(numberArray:Array, subset:Array, setIndex:int,
subsetIndex:int):Boolean
{
// Try every value from the input set starting from our current position,
// and insert the value at the current subset index:
for (var index:int = setIndex ; index < numberArray.length; index++)
{
subset[subsetIndex] = numberArray[index];
// Have we filled the subset?
if (subsetIndex == subset.length - 1)
{
var sum:int = 0;
for (var i:int = 0; i < subset.length; i++)
{
sum += subset[i];
}
// Return true if we found our sum, false if not:
return sum == requiredSum;
}
else
{
if (findSum(numberArray, subset, index + 1, subsetIndex + 1))
{
// If the "inner" findSum found a sum, we're done, so return
// - otherwise stay in the loop and keep looking:
return true;
}
}
}
// We found no subset with our required sum this time around:
return false;
}
ETA: How this works... As mentioned, it's the naive solution - in other words, we're simply checking every single permutation of numberArray, summing each permutation, and checking if it's the sum we want.
The most complicated part is making all the permutations. The way this code does it is through recursion - i.e., the findSum() function filling a slot then calling itself to fill the next one, until all slots are filled and it can check the sum. We'll use the numberArray [1, 5, 4, 2] as an example here:
Go through all subset sizes in a loop - i.e., start by making all [a], then all [a,b], [a,b,c], [a,b,c,d]... etc.
For each subset size:
Fill slot 1 of the subset...
... with each value of numberArray - [1, ?, ?], [5, ?, ?], [4, ?, ?]...
If all slots in subset have been filled, check if the sum matches and skip step 4.
(Recursively) call findSum to:
Fill slot 2 of the subset...
... with each remaining value of numberArray - [1, 5, ?], [1, 4, ?], [1, 2, ?]
If all slots in subset have been filled, check if the sum matches and skip step 4.
(Recursively) call findSum to:
Fill slot 3 of the subset
... with each remaining value of numberArray - [1, 5, 4], [1, 5, 2]
If all slots in subset have been filled, check if the sum matches and skip step 4.
(Recursively) call findSum (this goes on "forever", or until all slots are filled and we "skip step 4")
Go to 2.4.4.1. to try next value for slot 3.
Go to 2.4.1 to try next value for slot 2.
Go to 2.1 to try next value for slot 1.
This way, we go through every permutation of size 1, 2, 3, 4...
There's more optimization that could be done here, since the code never checks that it actually has enough values left in the input set to fill the remaining slots - i.e. it does some loops and calls to findSum() that are unneeded. This is only a matter of efficiency, however - the result is still correct.
I would do something like the following:
shuffle array
take random amount of numbers from the array
sum them up
if the sum is not the total sum you want, repeat
hm, not sure what you want to do at the end when a "conclusion" or "no conclusion" is reached, but you could generate a Power set from your set of numbers then for each subset add up all the numbers in it to see if you get your desired sum.
(This would be a 'brute force' approach and could be slow if you have many numbers.)
Possibly useful for how to create a Power set:
Calculating all of the subsets of a set of numbers
Edit: Since it appears nobody is reading the original question this links to, let me bring in a synopsis of it here.
The original problem, as asked by someone else, was that, given a large number of values, where the sum would exceed what a data type of Double would hold, how can one calculate the average of those values.
There was several answers that said to calculate in sets, like taking 50 and 50 numbers, and calculating the average inside those sets, and then finally take the average of all those sets and combine those to get the final average value.
My position was that unless you can guarantee that all those values can be split into a number of equally sized sets, you cannot use this approach. Someone dared me to ask the question here, in order to provide the answer, so here it is.
Basically, given an arbitrary number of values, where:
I know the number of values beforehand (but again, how would your answer change if you didn't?`)
I cannot gather up all the numbers, nor can I sum them (the sum will be too big for a normal data type in your programming language)
how can I calculate the average?
The rest of the question here outlines how, and the problems with, the approach to split into equally sized sets, but I'd really just like to know how you can do it.
Note that I know perfectly well enough math to know that in math theory terms, calculating the sum of A[1..N]/N will give me the average, let's assume that there are reasons that it isn't just as simple, and I need to split up the workload, and that the number of values isn't necessarily going to be divisable by 3, 7, 50, 1000 or whatever.
In other words, the solution I'm after will have to be general.
From this question:
What is a good solution for calculating an average where the sum of all values exceeds a double’s limits?
my position was that splitting the workload up into sets is no good, unless you can ensure that the size of those sets are equal.
Edit: The original question was about the upper limit that a particular data type could hold, and since he was summing up a lot of numbers (count that was given as example was 10^9), the data type could not hold the sum. Since this was a problem in the original solution, I'm assuming (and this is a prerequisite for my question, sorry for missing that) that the numbers are too big to give any meaningful answers.
So, dividing by the total number of values directly is out. The original reason for why a normal SUM/COUNT solution was out was that SUM would overflow, but let's assume, for this question that SET-SET/SET-SIZE will underflow, or whatever.
The important part is that I cannot simply sum, I cannot simply divide by the number of total values. If I cannot do that, will my approach work, or not, and what can I do to fix it?
Let me outline the problem.
Let's assume you're going to calculate the average of the numbers 1 through 6, but you cannot (for whatever reason) do so by summing the numbers, counting the numbers, and then dividing the sum by the count. In other words, you cannot simply do (1+2+3+4+5+6)/6.
In other words, SUM(1..6)/COUNT(1..6) is out. We're not considering NULL's (as in database NULL's) here.
Several of the answers to that question alluded to being able to split the numbers being averaged into sets, say 3 or 50 or 1000 numbers, then calculating some number for that, and then finally combining those values to get the final average.
My position is that this is not possible in the general case, since this will make some numbers, the ones appearing in the final set, more or less valuable than all the ones in the previous sets, unless you can split all the numbers into equally sized sets.
For instance, to calculate the average of 1-6, you can split it up into sets of 3 numbers like this:
/ 1 2 3 \ / 4 5 6 \
| - + - + - | + | - + - + - |
\ 3 3 3 / \ 3 3 3 / <-- 3 because 3 numbers in the set
---------- -----------
2 2 <-- 2 because 2 equally sized groups
Which gives you this:
2 5
- + - = 3.5
2 2
(note: (1+2+3+4+5+6)/6 = 3.5, so this is correct here)
However, my point is that once the number of values cannot be split into a number of equally sized sets, this method falls apart. For instance, what about the sequence 1-7, which contains a prime number of values.
Can a similar approach, that won't sum all the values, and count all the values, in one go, work?
So, is there such an approach? How do I calculate the average of an arbitrary number of values in which the following holds true:
I cannot do a normal sum/count approach, for whatever reason
I know the number of values beforehand (what if I don't, will that change the answer?)
Well, suppose you added three numbers and divided by three, and then added two numbers and divided by two. Can you get the average from these?
x = (a + b + c) / 3
y = (d + e) / 2
z = (f + g) / 2
And you want
r = (a + b + c + d + e + f + g) / 7
That is equal to
r = (3 * (a + b + c) / 3 + 2 * (d + e) / 2 + 2 * (f + g) / 2) / 7
r = (3 * x + 2 * y + 2 * z) / 7
Both lines above overflow, of course, but since division is distributive, we do
r = (3.0 / 7.0) * x + (2.0 / 7.0) * y + (2.0 / 7.0) * z
Which guarantees that you won't overflow, as I'm multiplying x, y and z by fractions less than one.
This is the fundamental point here. Neither I'm dividing all numbers beforehand by the total count, nor am I ever exceeding the overflow.
So... if you you keep adding to an accumulator, keep track of how many numbers you have added, and always test if the next number will cause an overflow, you can then get partial averages, and compute the final average.
And no, if you don't know the values beforehand, it doesn't change anything (provided that you can count them as you sum them).
Here is a Scala function that does it. It's not idiomatic Scala, so that it can be more easily understood:
def avg(input: List[Double]): Double = {
var partialAverages: List[(Double, Int)] = Nil
var inputLength = 0
var currentSum = 0.0
var currentCount = 0
var numbers = input
while (numbers.nonEmpty) {
val number = numbers.head
val rest = numbers.tail
if (number > 0 && currentSum > 0 && Double.MaxValue - currentSum < number) {
partialAverages = (currentSum / currentCount, currentCount) :: partialAverages
currentSum = 0
currentCount = 0
} else if (number < 0 && currentSum < 0 && Double.MinValue - currentSum > number) {
partialAverages = (currentSum / currentCount, currentCount) :: partialAverages
currentSum = 0
currentCount = 0
}
currentSum += number
currentCount += 1
inputLength += 1
numbers = rest
}
partialAverages = (currentSum / currentCount, currentCount) :: partialAverages
var result = 0.0
while (partialAverages.nonEmpty) {
val ((partialSum, partialCount) :: rest) = partialAverages
result += partialSum * (partialCount.toDouble / inputLength)
partialAverages = rest
}
result
}
EDIT:
Won't multiplying with 2, and 3, get me back into the range of "not supporter by the data type?"
No. If you were diving by 7 at the end, absolutely. But here you are dividing at each step of the sum. Even in your real case the weights (2/7 and 3/7) would be in the range of manageble numbers (e.g. 1/10 ~ 1/10000) which wouldn't make a big difference compared to your weight (i.e. 1).
PS: I wonder why I'm working on this answer instead of writing mine where I can earn my rep :-)
If you know the number of values beforehand (say it's N), you just add 1/N + 2/N + 3/N etc, supposing that you had values 1, 2, 3. You can split this into as many calculations as you like, and just add up your results. It may lead to a slight loss of precision, but this shouldn't be an issue unless you also need a super-accurate result.
If you don't know the number of items ahead of time, you might have to be more creative. But you can, again, do it progressively. Say the list is 1, 2, 3, 4. Start with mean = 1. Then mean = mean*(1/2) + 2*(1/2). Then mean = mean*(2/3) + 3*(1/3). Then mean = mean*(3/4) + 4*(1/4) etc. It's easy to generalize, and you just have to make sure the bracketed quantities are calculated in advance, to prevent overflow.
Of course, if you want extreme accuracy (say, more than 0.001% accuracy), you may need to be a bit more careful than this, but otherwise you should be fine.
Let X be your sample set. Partition it into two sets A and B in any way that you like. Define delta = m_B - m_A where m_S denotes the mean of a set S. Then
m_X = m_A + delta * |B| / |X|
where |S| denotes the cardinality of a set S. Now you can repeatedly apply this to partition and calculate the mean.
Why is this true? Let s = 1 / |A| and t = 1 / |B| and u = 1 / |X| (for convenience of notation) and let aSigma and bSigma denote the sum of the elements in A and B respectively so that:
m_A + delta * |B| / |X|
= s * aSigma + u * |B| * (t * bSigma - s * aSigma)
= s * aSigma + u * (bSigma - |B| * s * aSigma)
= s * aSigma + u * bSigma - u * |B| * s * aSigma
= s * aSigma * (1 - u * |B|) + u * bSigma
= s * aSigma * (u * |X| - u * |B|) + u * bSigma
= s * u * aSigma * (|X| - |B|) + u * bSigma
= s * u * aSigma * |A| + u * bSigma
= u * aSigma + u * bSigma
= u * (aSigma + bSigma)
= u * (xSigma)
= xSigma / |X|
= m_X
The proof is complete.
From here it is obvious how to use this to either recursively compute a mean (say by repeatedly splitting a set in half) or how to use this to parallelize the computation of the mean of a set.
The well-known on-line algorithm for calculating the mean is just a special case of this. This is the algorithm that if m is the mean of {x_1, x_2, ... , x_n} then the mean of {x_1, x_2, ..., x_n, x_(n+1)} is m + ((x_(n+1) - m)) / (n + 1). So with X = {x_1, x_2, ..., x_(n+1)}, A = {x_(n+1)}, and B = {x_1, x_2, ..., x_n} we recover the on-line algorithm.
Thinking outside the box: Use the median instead. It's much easier to calculate - there are tons of algorithms out there (e.g. using queues), you can often construct good arguments as to why it's more meaningful for data sets (less swayed by extreme values; etc) and you will have zero problems with numerical accuracy. It will be fast and efficient. Plus, for large data sets (which it sounds like you have), unless the distributions are truly weird, the values for the mean and median will be similar.
When you split the numbers into sets you're just dividing by the total number or am I missing something?
You have written it as
/ 1 2 3 \ / 4 5 6 \
| - + - + - | + | - + - + - |
\ 3 3 3 / \ 3 3 3 /
---------- -----------
2 2
but that's just
/ 1 2 3 \ / 4 5 6 \
| - + - + - | + | - + - + - |
\ 6 6 6 / \ 6 6 6 /
so for the numbers from 1 to 7 one possible grouping is just
/ 1 2 3 \ / 4 5 6 \ / 7 \
| - + - + - | + | - + - + - | + | - |
\ 7 7 7 / \ 7 7 7 / \ 7 /
Average of x_1 .. x_N
= (Sum(i=1,N,x_i)) / N
= (Sum(i=1,M,x_i) + Sum(i=M+1,N,x_i)) / N
= (Sum(i=1,M,x_i)) / N + (Sum(i=M+1,N,x_i)) / N
This can be repeatedly applied, and is true regardless of whether the summations are of equal size. So:
Keep adding terms until both:
adding another one will overflow (or otherwise lose precision)
dividing by N will not underflow
Divide the sum by N
Add the result to the average-so-far
There's one obvious awkward case, which is that there are some very small terms at the end of the sequence, such that you run out of values before you satisfy the condition "dividing by N will not underflow". In which case just discard those values - if their contribution to the average cannot be represented in your floating type, then it is in particular smaller than the precision of your average. So it doesn't make any difference to the result whether you include those terms or not.
There are also some less obvious awkward cases to do with loss of precision on individual summations. For example, what's the average of the values:
10^100, 1, -10^100
Mathematics says it's 1, but floating-point arithmetic says it depends what order you add up the terms, and in 4 of the 6 possibilities it's 0, because (10^100) + 1 = 10^100. But I think that the non-commutativity of floating-point arithmetic is a different and more general problem than this question. If sorting the input is out of the question, I think there are things you can do where you maintain lots of accumulators of different magnitudes, and add each new value to whichever one of them will give best precision. But I don't really know.
Here's another approach. You're 'receiving' numbers one-by-one from some source, but you can keep track of the mean at each step.
First, I will write out the formula for mean at step n+1:
mean[n+1] = mean[n] - (mean[n] - x[n+1]) / (n+1)
with the initial condition:
mean[0] = x[0]
(the index starts at zero).
The first equation can be simplified to:
mean[n+1] = n * mean[n] / (n+1) + x[n+1]/(n+1)
The idea is that you keep track of the mean, and when you 'receive' the next value in your sequence, you figure out its offset from the current mean, and divide it equally between the n+1 samples seen so far, and adjust your mean accordingly. If your numbers don't have a lot of variance, your running mean will need to be adjusted very slightly with the new numbers as n becomes large.
Obviously, this method works even if you don't know the total number of values when you start. It has an additional advantage that you know the value of the current mean at all times. One disadvantage that I can think of is the it probably gives more 'weight' to the numbers seen in the beginning (not in a strict mathematical sense, but because of floating point representations).
Finally, all such calculations are bound to run into floating-point 'errors' if one is not careful enough. See my answer to another question for some of the problems with floating point calculations and how to test for potential problems.
As a test, I generated N=100000 normally distributed random numbers with mean zero and variance 1. Then I calculated their mean by three methods.
sum(numbers) / N, call it m1,
my method above, call it m2,
sort the numbers, and then use my method above, call it m3.
Here's what I found: m1 − m2 ∼ −4.6×10−17, m1 − m3 ∼ −3×10−15, m2 − m3 ∼ −3×10−15. So, if your numbers are sorted, the error might not be small enough for you. (Note however that even the worst error is 10−15 parts in 1 for 100000 numbers, so it might be good enough anyway.)
Some of the mathematical solutions here are very good. Here's a simple technical solution.
Use a larger data type. This breaks down into two possibilities:
Use a high-precision floating point library. One who encounters a need to average a billion numbers probably has the resources to purchase, or the brain power to write, a 128-bit (or longer) floating point library.
I understand the drawbacks here. It would certainly be slower than using intrinsic types. You still might over/underflow if the number of values grows too high. Yada yada.
If your values are integers or can be easily scaled to integers, keep your sum in a list of integers. When you overflow, simply add another integer. This is essentially a simplified implementation of the first option. A simple (untested) example in C# follows
class BigMeanSet{
List<uint> list = new List<uint>();
public double GetAverage(IEnumerable<uint> values){
list.Clear();
list.Add(0);
uint count = 0;
foreach(uint value in values){
Add(0, value);
count++;
}
return DivideBy(count);
}
void Add(int listIndex, uint value){
if((list[listIndex] += value) < value){ // then overflow has ocurred
if(list.Count == listIndex + 1)
list.Add(0);
Add(listIndex + 1, 1);
}
}
double DivideBy(uint count){
const double shift = 4.0 * 1024 * 1024 * 1024;
double rtn = 0;
long remainder = 0;
for(int i = list.Count - 1; i >= 0; i--){
rtn *= shift;
remainder <<= 32;
rtn += Math.DivRem(remainder + list[i], count, out remainder);
}
rtn += remainder / (double)count;
return rtn;
}
}
Like I said, this is untested—I don't have a billion values I really want to average—so I've probably made a mistake or two, especially in the DivideBy function, but it should demonstrate the general idea.
This should provide as much accuracy as a double can represent and should work for any number of 32-bit elements, up to 232 - 1. If more elements are needed, then the count variable will need be expanded and the DivideBy function will increase in complexity, but I'll leave that as an exercise for the reader.
In terms of efficiency, it should be as fast or faster than any other technique here, as it only requires iterating through the list once, only performs one division operation (well, one set of them), and does most of its work with integers. I didn't optimize it, though, and I'm pretty certain it could be made slightly faster still if necessary. Ditching the recursive function call and list indexing would be a good start. Again, an exercise for the reader. The code is intended to be easy to understand.
If anybody more motivated than I am at the moment feels like verifying the correctness of the code, and fixing whatever problems there might be, please be my guest.
I've now tested this code, and made a couple of small corrections (a missing pair of parentheses in the List<uint> constructor call, and an incorrect divisor in the final division of the DivideBy function).
I tested it by first running it through 1000 sets of random length (ranging between 1 and 1000) filled with random integers (ranging between 0 and 232 - 1). These were sets for which I could easily and quickly verify accuracy by also running a canonical mean on them.
I then tested with 100* large series, with random length between 105 and 109. The lower and upper bounds of these series were also chosen at random, constrained so that the series would fit within the range of a 32-bit integer. For any series, the results are easily verifiable as (lowerbound + upperbound) / 2.
*Okay, that's a little white lie. I aborted the large-series test after about 20 or 30 successful runs. A series of length 109 takes just under a minute and a half to run on my machine, so half an hour or so of testing this routine was enough for my tastes.
For those interested, my test code is below:
static IEnumerable<uint> GetSeries(uint lowerbound, uint upperbound){
for(uint i = lowerbound; i <= upperbound; i++)
yield return i;
}
static void Test(){
Console.BufferHeight = 1200;
Random rnd = new Random();
for(int i = 0; i < 1000; i++){
uint[] numbers = new uint[rnd.Next(1, 1000)];
for(int j = 0; j < numbers.Length; j++)
numbers[j] = (uint)rnd.Next();
double sum = 0;
foreach(uint n in numbers)
sum += n;
double avg = sum / numbers.Length;
double ans = new BigMeanSet().GetAverage(numbers);
Console.WriteLine("{0}: {1} - {2} = {3}", numbers.Length, avg, ans, avg - ans);
if(avg != ans)
Debugger.Break();
}
for(int i = 0; i < 100; i++){
uint length = (uint)rnd.Next(100000, 1000000001);
uint lowerbound = (uint)rnd.Next(int.MaxValue - (int)length);
uint upperbound = lowerbound + length;
double avg = ((double)lowerbound + upperbound) / 2;
double ans = new BigMeanSet().GetAverage(GetSeries(lowerbound, upperbound));
Console.WriteLine("{0}: {1} - {2} = {3}", length, avg, ans, avg - ans);
if(avg != ans)
Debugger.Break();
}
}