I'm trying to understand a little more about how Octave calculates quartiles and interquartile range. Consider the following:
A=[1 4 7 10 14];
quantile(A, [0.25 0.75])
ans = 3.2500 11.0000
This result seems consistent with Method 3 on the Wikipedia page about quartiles. Given that the interquartile range is Q3-Q1, I'd expect the result to be 7.75.
However, running iqr(A) gives a result of 6. Clearly this is calculated from 10 minus 4 from the original data, which is consistent with Method 2 from the same Wikipedia page.
What is the reason for using two different methods for calculating Q1 and Q3?
Related
I am transforming a signal using the ZeroFFT library. The results I get from it, are not what I would intuitively expect.
As a test, I feed the FFT algorithm with a buffer that contains two full cycles of cosine:
Which is sampled over 512 samples.
Which are fed as int16_t values. What I expected to get back, is 256 amplitudes, with the values [ 0, 4095, 0, 0, ..., 0 ].
Instead, this is the result:
2 2052 4086 2053 0 2 2 2 1 2 2 2 4 4 3 4...
And it gets weirder! If I feed it the same signal, but shifted (so sinf() over 0 .. 4*pi instead of cosf() function) I get a completely different result: 4 10 2 16 2 4 4 4 2 2 2 3 2 4 3 4
This throws up the questions:
1. Doesn't a sine signal and cosine signal with same period, contain exactly the same frequencies?
2. If I feed it a buffer with exactly 2 cycles of cosine, wouldn't the Fourier transform result in all zeros, except for 1 frequency?
I generate my test signal as:
static void setup_reference(void)
{
for (int i=0; i<CAPTURESZ; ++i)
{
const float phase = 2 * 3.14159f * i / (256.0f);
reference_wave[i] = (int16_t) (cosf(phase) * 0x7fff);
}
}
And call the ZeroFFT function as:
ZeroFFT(reference_Wave, CAPTURESZ);
Note: the ZeroFFT docs state that a Hanning window is applied.
Windowing causes some spectral leakage. Including the window function, the wave now looks like this:
If I feed it a buffer with exactly 2 cycles of cosine, wouldn't the Fourier transform result in all zeros, except for 1 frequency?
Yes, if you do it without windowing. Actually two frequencies: both the positive frequency that you expect, and the equivalent negative frequency, though not all FFT functions will include the negative frequencies in their output (for Real input, the result is Hermitian-symmetric, there is no extra information in the negative frequencies). For practical reasons, since neither the input signal nor the FFT calculation are exact, you may not get exactly zero everywhere else either, but it should be close - that's mainly a concern for floating point output.
By the way by this I don't mean that windowing is bad, but in this special case (perfectly periodic input) it didn't work out in your favour.
As for the sine wave, the magnitudes of the result should be similar (within reason - exactness shouldn't be expected), but the comments on the FFT function you used mention
The complex portion is discarded, and the real values are returned.
While phase shifts would not change the magnitudes much, they change the phases of the results, and therefore also their Real component.
I am trying to present time series of multiple sensors on a single SSRS (v14) line chart
I need to plot N series, with each independently plotting the series data in the space provided by the chart (independent vertical axis)
More about the data
There can be anywhere from ~1-10 series
The challenge is that they are different orders of magnitude.
One might be degrees F (~0-212)
One might be Carbon ppm (~1-16)
One might be Ftlbs Thrust (~10k-100k)
the point is , they have no relation and can be very different
The exact value is not important. I can hide the vertical axis
More about what I am trying to do
The idea is to show the multiple time series, plotted together against time for the 4 hours before and after
'an event'. Its not the necessarily the exact value that is important. the subject matter expert would be looking for something odd (temperature falls, thrust spikes, etc).
Things I have tried
If there were just 2 series, i could easily use the 2nd axis available in the SSRS chart. Thats exactly the idea I am chasing. But in this case, I want N series to plot using its own axis.
I have tried stacking N transparent graphs on top of each other. This would be a really ugly solution, but SSRS even wont let you do it. It unstacks them for you.
I have experimented with the Allow Scale Breaks property on the Vert Axis. This would solve the problem but we don't like the 'double jagged line'
Turning on Logarithmic scale is a possibility. It does do a better job of displaying all the data. but its not really what we want. Its going to change the shape of data that ranges over a couple orders of magnitude.
I tried the sparkline component and am having the same problem.
This approach is essentially the same a Greg's answer above. I've had to do this same process in the past comparing trends of data even though the units were dissimilar.
I took a very simple approach of adding an additional column to the query that showed each value as a percentage of the maximum value in each series.
As an example (just 2 series here for clarity) I started with data like this in myTable
Series Month myValue
A Jan 4
A Feb 8
A Mar 16
B Jan 200
B Feb 300
B Mar 400
My Dataset query would be something like.
SELECT *, myValue / MAX(myValue) OVER(PARTITION BY Series) as myPlotValue FROM myTable
This gives us a final dataset which looks liek this.
Series Month myValue myPlotValue
A Jan 4 0.25
A Feb 8 0.5
A Mar 16 1
B Jan 200 0.5
B Feb 300 0.75
B Mar 400 1
As you can see all plot values are now between 0 and 1.
I created that charts using the myPlotValue field and had the option of using the original values from the myValue field as datapoint labels.
After talking to some math people, this is a standard problem and it is solved by a process called normalization of the data.
Essentially you are changing all the series to fit in a given range (usually 0-1)
You can scale and add an offset if that makes sense for your problem domain somehow.
https://www.statisticshowto.datasciencecentral.com/normalized/
I need to use a for-loop in a function in order to find spring constants of all possible combinations of springs in series and parallel. I have 5 springs with data therefore I found the spring constant (K) of each in a new matrix by using polyfit to find the slope (using F=Kx).
I have created a function that does so, however it returns data not in a matrix, but as individual outputs. So instead of KP (Parallel)= [1 2 3 4 5] it says KP=1, KP=2, KP=3, etc. Because of this, only the final output is stored in my workspace. Here is the code I have for the function. Keep in mind that the reason I need to use the +2 in the for loop for b is because my original matrix K with all spring constants is ten columns, with every odd number being a 0. Ex: K=[1 0 2 0 3 0 4 0 5] --- This is because my original dataset to find K (slope) was ten columns wide.
function[KP,KS]=function_name1(K)
L=length(K);
c=1;
for a=1:2:L
for b=a+2:2:L
KP=K(a)+K(b)
KS=1/((1/K(a))+(1/K(b)))
end
end
c=c+1;
and then a program calling that function
[KP,KS]=function_name1(K);
What I tried: - Suppressing and unsuppressing lines of code (unsuccessful)
Any help would be greatly appreciated.
hmmm...
your code seems workable, but you aren't dealing with things in the most practical manner
I'd start be redimensioning K so that it makes sense, that is that it's 5 spaces wide instead of your current 10 - you'll see why in a minute.
Then I'd adjust KP and KS to the size that you want (I'm going to do a 5X5 as that will give all the permutations - right now it looks like you are doing some triangular thing, I wouldn't worry too much about space unless you were to do this for say 50,000 spring constants or so)
So my code would look like this
function[KP,KS]=function_name1(K)
L=length(K);
KP = zeros(L);
KS = zeros(l);
c=1;
for a=1:L
for b=1:L
KP(a,b)=K(a)+K(b)
KS(a,b)=1/((1/K(a))+(1/K(b)))
end
end
c=c+1;
then when you want the parallel combination of springs 1 and 4 KP(1,4) or KP(4,1) will do the trick
I have 3 matrices that I would like to be plotted on a boxplot (two of them are 22 rows by 83 columns, and the other is 7 rows by 83 columns) within Octave.
I've tried:
boxplot([red(:,1),blue(:,1),purple(:,1)])
error: horizontal dimensions mismatch
error: evaluating argument list element number 1
But, I keep getting the above error. I assume it's because I have one matrix with 7 rows instead of 22? If so, is there any possible way of getting them both plotted on the same boxplot?
When you pass [a,b,c] you are trying to build a matrix by concatenating horizontally the other three. Since they do not have the same number of rows that will never work.
If you want to do the boxplot use cells (as indicated in help boxplot) that is
boxplot ({red(:,1),blue(:,1),purple(:,1)})
(I wish my mathematical vocabulary was more developed)
I have a website. On that website is a video. As a user watches the video, a bit of javascript stores how far they have gotten so far in the video. When they stop watching the video, that number of seconds is stored. There's no pattern to when the js will do this, unfortunately.
So if one person is watching the video, we might see this set:
3
6
8
10
12
16
And another person might get bored immediately:
1
3
This data is all stored in the same place, anonymously. So the sorted table with all this info would look like this:
1
3
3
6
8
10
12
16
Finally, the amount of times the video is started at all is stored. In this case it would be 2.
So. How do I get the average 'high-time' (the farthest reached point in the video) for all of the times the video was played?
I know that if we had a value for every second:
1
2
3
4
5
6
7
...
14
15
16
1
2
3
Then we could count up the values and divide by the number of plays:
(19) / 2 = 9.5
Or if the data was otherwise uniform, say in increments of 5, then we could count that up and multiply it by 5 (in the example, we would have some loss of precision, but that's ok):
5
10
15
5
(4) * 5 / 2 = 10
So it seems like I have a general function which would work:
count * 1/d = avg
where d is the density of the numbers (in the example above with 5 second increments, 1/5).
Is there a way to derive the density, d, from a set of effectively random numbers?
Why not just keep the last time that has been provided, and average across those? If you either throw away, or only pay attention to, the last number, it seems like you could just average over these.
You might also want to check out the term standard deviation as the raw average of this might not be the most useful measurement. If you have the standard deviation as well, it could help you realize that you have an average of 7, but it is composed of mostly 1's and 15's.
If you HAVE to have all the data, like you suggested, I will try and think about this a little bit more. I'm not totally certain how you can associate a value with all the previous values that came with it. Do you ALWAYS know the sequence by which numbers are output? If so, I think I know of a way you could derive the 'last' one, which might be slightly computationally expensive.
If you only have a sequence of integers, I think you may be able to increase each value (exponentially?) to 'compensate' for the fact that a later value 'contains' earlier values. I'm still working through this idea, but maybe it will give someone else a seed. What if you average over the sum of these, and then take the base2 logarithm of this average? Does that provide any kind of useful metric? That should 'weight' the later values to the point where they compensate for the sum of earlier values. I think.
In python-esk:
sum = 0
numberOf = 0
for node in nodes:
sum = sum + node.value ^ 2
numberOf = numberOf + 1
weightedAverage = log(sum/numberOf, 2)
print weightedAverage
print "Thanks Brian"
I think that #brian-stiner is on the right track in one of his comments.
Start with something like:
1
3
3
6
8
10
12
16
Turn that into numbers and counts.
1, 1
3, 2
6, 1
8, 1
10, 1
12, 1
16, 1
And then reading from the end down, find all of the points that happened more often than any remaining ones.
3, 2
16, 1
Take differences in counts.
3, 1
16, 1
And you have an estimate of stopping places.
This will not be an unbiased estimate. But if the JavaScript is independently inconsistent and the number of people is large, the biases should be fairly small.
It won't be right, but it will be close enough for government work.
Assuming increments are always around 5, some missing, some a bit longer or shorter. Then it won't be easy (possible?) to do this exactly. My suggestion: compute something like a 'moving count'. Similar to moving average.
So, for second 7: count how many numbers are 5,6,7,8 or 9 and divide by 5. That will give you a pretty good guess of how many people watched the 7th second. Do the same for second 10. The difference would be close to the number of the people who left between second 7 and 10.
To get the total time watched for each user, you'll have parse the list smallest to largest. If you have 4 views, you'll go through your list until you find that you no longer have 4 identical numbers, the last number where you had 4 identical numbers is the maximum of the first view. Then you'll look for when the 3 identical numbers stop, and so on. For example:
4 views data:
1111222233334445566778
4 views side by side:
1 1 1 1
2 2 2 2
3 3 3 3 <- first view max is 3 seconds
4 4 4 <- second view max is 4 seconds
5 5
6 6
7 7 <- third view max is 7 seconds
8 <- fourth view max is 8 seconds
EDIT- Oh, I just noticed that they are not uniform. In that case, the moving average would probably be your best bet.
The number of values roughly corresponds to the number of time periods in which your javascript sends the values (minus 1/2 if the video stop is accompanied with a obligatory time posting, since its moment is random within the interval).
If all clients have similar intervals and you know them, you may just use:
SELECT (COUNT(*) - 0.5) * 5.0 / (SELECT counter FROM countertable)
FROM ticktable
5.0 is the interval between the posts here.
Note that it does not even look at the values: you could as well just store "ticks".
For the max time, you could use MAX() on your field. Perhaps something like...
SELECT MAX(play_time) AS maxTime FROM video
Which would give you the longest time someone has played the video for.
If you want other things, like AVG() then you'll need more complex queries, for collecting on a per-user basis etc etc.
MySQL also contains a Standard Deviation function called STDDEV() and STD() which could help you too.