Related
How can I use scipy.interpolate.interp1d when my x array is an N-D array, instead of a 1-D array, without using a loop?
The function f from interp1d then needs to be used with numpy.percentile with one of the arrays as an input.
I think there should be a way to do it with a list comprehension or lambda function, but I am still learning these tools.
(Note that this is different than my recent question here because I mixed up the x and y arrays in the posted question, and this problem was not reproducible.)
Problem statement/example:
# a is y in interp1d docs
a = np.array([97,4809,4762,282,3879,17454,103,2376,40581,])
# b is x in interp1d docs
b = np.array([
[0.14,0.11,0.29,0.11,0.09,0.68,0.09,0.18,0.5,],
[0.32,0.25,0.67,0.25,0.21,1.56,1.60, 0.41,1.15,],]
)
Just trying this, below, fails with ValueError: x and y arrays must be equal in length along interpolation axis. The expected return is array(97, 2376). Using median here, but will need to consider 10th, 90th, etc. percentiles.
f = interpolate.interp1d(b, a, axis=0)
f(np.percentile(b, 50, axis=0))
However this, below, works and prints array(97.)
f = interpolate.interp1d(b[0,:], a, axis=0)
f(np.percentile(b[0,:], 50, axis=0))
A loop works, but I am wondering if there is a solution using list comprehensions, lambda functions, or some other technique.
l = []
for _i in range(b.shape[0]):
_f = interpolate.interp1d(b[_i,:], a, axis=0)
l.append(_f(np.percentile(b[_i,:], 50, axis=0)))
print(out)
# returns
# [array(97.), array(2376.)]
Efforts:
I understand I can loop through the b array with a list comprehension.
[b[i,:] for i in range(b.shape[0])]
# returns
# [array([0.14, 0.11, 0.29, 0.11, 0.09, 0.68, 0.09, 0.18, 0.5 ]),
# array([0.32, 0.25, 0.67, 0.25, 0.21, 1.56, 1.6 , 0.41, 1.15])]
And I also understand that I can use a list comprehension to create the scipy function f for each dimension in b:
[interpolate.interp1d(b[i, :], a, axis=0) for i in range(b.shape[0])]
# returns
# [<scipy.interpolate.interpolate.interp1d at 0x1b72e404360>,
# <scipy.interpolate.interpolate.interp1d at 0x1b72e404900>]
But I don't know how to combine these two list comprehensions to apply the np.percentile function.
Using Python 3.8.3, NumPy 1.18.5, SciPy 1.3.2
If you have large data arrays, you want to stay away from for loops, map, np.vectorize and comprehensions. They will all be slow. Instead, it's always better to use vectorized numpy or scipy operations whenever possible.
In this particular case, you can implement the vectorization pretty trivially yourself. interp1d defaults to a linear interpolation, which is very simple to code by hand. For a general interpolator, the first step would be to sort x and y, which is why scipy can't support multiple x for a given y. If the x rows all have different sort order, what do you do with the y?
Luckily, there are a couple of things you can do to make this much faster than having to build a full interpolator or argsort y multiple times. For example, start by argsorting x:
idx = b.argsort(axis=1)
idx is now an array such that b[np.arange(2)[:, None], idx] gives the sorted version of b along axis 1, and also, a[idx] is the corresponding y-values. Since you are taking the median (50th precentile), and the rows have an odd number of elements, the value of x is just the middle of each row, and y is given by
a[idx[:, len(a) // 2]]
If you had an even number of elements, you would have to average the elements surrounding the middle:
i = len(a) // 2 - 1
a[idx[:, i:i + 2]].mean(axis=1)
You can reduce algorithmic complexity by using np.argpartition instead of a full-blown np.argsort to get the middle element(s).
interp1d and other interpolators from scipy.interpolate only support 1D x arrays. So you'll need to loop over the dimensions of x manually.
I would like to smooth an Impulse Response audio file. The FFT of the file shows that it is very spikey. I would like to smooth out the audio file, not just its plot, so that I have a smoother IR file.
I have found a function that shows the FFT plot smoothed out. How could this smoothing be applied to the actual FFT data and not just to the plot of it?
[y,Fs] = audioread('test\test IR.wav');
function x_oct = smoothSpectrum(X,f,Noct)
%SMOOTHSPECTRUM Apply 1/N-octave smoothing to a frequency spectrum
%% Input checking
assert(isvector(X), 'smoothSpectrum:invalidX', 'X must be a vector.');
assert(isvector(f), 'smoothSpectrum:invalidF', 'F must be a vector.');
assert(isscalar(Noct), 'smoothSpectrum:invalidNoct', 'NOCT must be a scalar.');
assert(isreal(X), 'smoothSpectrum:invalidX', 'X must be real.');
assert(all(f>=0), 'smoothSpectrum:invalidF', 'F must contain positive values.');
assert(Noct>=0, 'smoothSpectrum:invalidNoct', 'NOCT must be greater than or equal to 0.');
assert(isequal(size(X),size(f)), 'smoothSpectrum:invalidInput', 'X and F must be the same size.');
%% Smoothing
% calculates a Gaussian function for each frequency, deriving a
% bandwidth for that frequency
x_oct = X; % initial spectrum
if Noct > 0 % don't bother if no smoothing
for i = find(f>0,1,'first'):length(f)
g = gauss_f(f,f(i),Noct);
x_oct(i) = sum(g.*X); % calculate smoothed spectral coefficient
end
% remove undershoot when X is positive
if all(X>=0)
x_oct(x_oct<0) = 0;
end
end
endfunction
function g = gauss_f(f_x,F,Noct)
% GAUSS_F calculate frequency-domain Gaussian with unity gain
%
% G = GAUSS_F(F_X,F,NOCT) calculates a frequency-domain Gaussian function
% for frequencies F_X, with centre frequency F and bandwidth F/NOCT.
sigma = (F/Noct)/pi; % standard deviation
g = exp(-(((f_x-F).^2)./(2.*(sigma^2)))); % Gaussian
g = g./sum(g); % normalise magnitude
endfunction
% take fft
Y = fft(y);
% keep only meaningful frequencies
NFFT = length(y);
if mod(NFFT,2)==0
Nout = (NFFT/2)+1;
else
Nout = (NFFT+1)/2;
end
Y = Y(1:Nout);
f = ((0:Nout-1)'./NFFT).*Fs;
% put into dB
Y = 20*log10(abs(Y)./NFFT);
% smooth
Noct = 12;
Z = smoothSpectrum(Y,f,Noct);
% plot
semilogx(f,Y,'LineWidth',0.7,f,Z,'LineWidth',2.2);
xlim([20,20000])
grid on
PS. I have Octave GNU, so I don't have the functions that are available with Matlab Toolboxes.
Here is the test IR audio file.
I think I found it. Since the FFT of the audio file (which is real numbers) is symmetric, with the same real part on both sides but opposite imaginary part, I thought of doing this:
take the FFT, keep the half of it, and apply the smoothing function without converting the magnitudes to dB
then make a copy of that smoothed FFT, and invert just the imaginary part
combine the two parts so that I have the same symmetric FFT as I had in the beginning, but now it is smoothed
apply inverse FFT to this and take the real part and write it to file.
Here is the code:
[y,Fs] = audioread('test IR.wav');
function x_oct = smoothSpectrum(X,f,Noct)
x_oct = X; % initial spectrum
if Noct > 0 % don't bother if no smoothing
for i = find(f>0,1,'first'):length(f)
g = gauss_f(f,f(i),Noct);
x_oct(i) = sum(g.*X); % calculate smoothed spectral coefficient
end
% remove undershoot when X is positive
if all(X>=0)
x_oct(x_oct<0) = 0;
end
end
endfunction
function g = gauss_f(f_x,F,Noct)
sigma = (F/Noct)/pi; % standard deviation
g = exp(-(((f_x-F).^2)./(2.*(sigma^2)))); % Gaussian
g = g./sum(g); % normalise magnitude
endfunction
% take fft
Y = fft(y);
% keep only meaningful frequencies
NFFT = length(y);
if mod(NFFT,2)==0
Nout = (NFFT/2)+1;
else
Nout = (NFFT+1)/2;
end
Y = Y(1:Nout);
f = ((0:Nout-1)'./NFFT).*Fs;
% smooth
Noct = 12;
Z = smoothSpectrum(Y,f,Noct);
% plot
semilogx(f,Y,'LineWidth',0.7,f,Z,'LineWidth',2.2);
xlim([20,20000])
grid on
#Apply the smoothing to the actual data
Zreal = real(Z); # real part
Zimag_neg = Zreal - Z; # opposite of imaginary part
Zneg = Zreal + Zimag_neg; # will be used for the symmetric Z
# Z + its symmetry with same real part but opposite imaginary part
reconstructed = [Z ; Zneg(end-1:-1:2)];
# Take the real part of the inverse FFT
reconstructed = real(ifft(reconstructed));
#Write to file
audiowrite ('smoothIR.wav', reconstructed, Fs, 'BitsPerSample', 24);
Seems to work! :) It would be nice if someone more knowledgeable could confirm that the thinking and code are good :)
I can't make matrices with variables in it for some reason. I get following message.
>>> A= [a b ;(-1-a) (1-b); (1+a) b]
error: horizontal dimensions mismatch (2x3 vs 1x1)
Why is it? Please show me correct way if I'm wrong.
In Matlab you first need to assign a variable before you can use it,
a = 1;
b = a+1;
This will thus give an error,
clear;
b = a+1; % ERROR! Undefined function or variable 'a
Matlab does never accept unassigned variables. This is because, on the lowest level, you do not have a. You will have machine code which is assgined the value of a. This is handled by the JIT compiler in Matlab, so you do not need to worry about this though.
If you want to use something as the variable which you have in maths you can specifically express this to matlab. The object is called a sym and the syntax that define the sym x to a variable xis,
syms x;
That said, you can define a vector or a matrix as,
syms a b x y; % Assign the syms
A = [x y]; % Vector
B = A= [a b ;(-1-a) (1-b); (1+a) b]; % Matrix.
The size of a matrix can be found with size(M) or for dim n size(M,n). You can calcuate the matrix product M3=M1*M2 if and only if M1 have the size m * n and M2 have the size n * p. The size of M3 will then be m * p. This will also mean that the operation A^N = A * A * ... is only allowed when m=n so to say, the matrix is square. This can be verified in matlab by the comparison,
syms a b
A = [a,1;56,b]
if size(A,1) == size(A,2)
disp(['A is a square matrix of size ', num2str(size(A,1)]);
else
disp('A is not square');
end
These are the basic rules for assigning variables in Matlab as well as for matrix multiplication. Further, a google search on the error error: 'x' undefined does only give me octave hits. Are you using octave? In that case I cannot guarantee that you can use sym objects or that the syntaxes are correct.
I am having a problem graphing a 3d function - when I enter data, I get a linear graph and the values don't add up if I perform the calculations by hand. I believe the problem is related to using matrices.
INITIAL_VALUE=999999;
INTEREST_RATE=0.1;
MONTHLY_INTEREST_RATE=INTEREST_RATE/12;
# ranges
down_payment=0.2*INITIAL_VALUE:0.1*INITIAL_VALUE:INITIAL_VALUE;
term=180:22.5:360;
[down_paymentn, termn] = meshgrid(down_payment, term);
# functions
principal=INITIAL_VALUE - down_payment;
figure(1);
plot(principal);
grid;
title("Principal (down payment)");
xlabel("down payment $");
ylabel("principal $ (amount borrowed)");
monthly_payment = (MONTHLY_INTEREST_RATE*(INITIAL_VALUE - down_paymentn))/(1 - (1 + MONTHLY_INTEREST_RATE)^-termn);
figure(2);
mesh(down_paymentn, termn, monthly_payment);
title("monthly payment (principal(down payment)) / term months");
xlabel("principal");
ylabel("term (months)");
zlabel("monthly payment");
The 2nd figure like I said doesn't plot like I expect. How can I change my formula for it to render properly?
I tried your script, and got the following error:
error: octave_base_value::array_value(): wrong type argument `complex matrix'
...
Your monthly_payment is a complex matrix (and it shouldn't be).
I guess the problem is the power operator ^. You should be using .^ for element-by-element operations.
From the documentation:
x ^ y
x ** y
Power operator. If x and y are both scalars, this operator returns x raised to the power y. If x is a scalar and y is a square matrix, the result is computed using an eigenvalue expansion. If x is a square matrix. the result is computed by repeated multiplication if y is an integer, and by an eigenvalue expansion if y is not an integer. An error results if both x and y are matrices.
The implementation of this operator needs to be improved.
x .^ y
x .** y
Element by element power operator. If both operands are matrices, the number of rows and columns must both agree.
Question
Let's say I have a string or array which represents a number in base N, N>1, where N is a power of 2. Assume the number being represented is larger than the system can handle as an actual number (an int or a double etc).
How can I convert that to a decimal string?
I'm open to a solution for any base N which satisfies the above criteria (binary, hex, ...). That is if you have a solution which works for at least one base N, I'm interested :)
Example:
Input: "10101010110101"
-
Output: "10933"
It depends on the particular language. Some have native support for arbitrary-length integers, and others can use libraries such as GMP. After that it's just a matter of doing the lookup in a table for the digit value, then multiplying as appropriate.
This is from a Python-based computer science course I took last semester that's designed to handle up to base-16.
import string
def baseNTodecimal():
# get the number as a string
number = raw_input("Please type a number: ")
# convert it to all uppercase to match hexDigits (below)
number = string.upper(number)
# get the base as an integer
base = input("Please give me the base: ")
# the number of values that we have to change to base10
digits = len(number)
base10 = 0
# first position of any baseN number is 1's
position = 1
# set up a string so that the position of
# each character matches the decimal
# value of that character
hexDigits = "0123456789ABCDEF"
# for each 'digit' in the string
for i in range(1, digits+1):
# find where it occurs in the string hexDigits
digit = string.find(hexDigits, number[-i])
# multiply the value by the base position
# and add it to the base10 total
base10 = base10 + (position * digit)
print number[-i], "is in the " + str(position) + "'s position"
# increase the position by the base (e.g., 8's position * 2 = 16's position)
position = position * base
print "And in base10 it is", base10
Basically, it takes input as a string and then goes through and adds up each "digit" multiplied by the base-10 position. Each digit is actually checked for its index-position in the string hexDigits which is used as the numerical value.
Assuming the number that it returns is actually larger than the programming language supports, you could build up an array of Ints that represent the entire number:
[214748364, 8]
would represent 2147483648 (a number that a Java int couldn't handle).
That's some php code I've just written:
function to_base10($input, $base)
{
$result = 0;
$length = strlen($input);
for ($x=$length-1; $x>=0; $x--)
$result += (int)$input[$x] * pow($base, ($length-1)-$x);
return $result;
}
It's dead simple: just a loop through every char of the input string
This works with any base <10 but it can be easily extended to support higher bases (A->11, B->12, etc)
edit: oh didn't see the python code :)
yeah, that's cooler
I would choose a language which more or less supports natively math representation like 'lisp'. I know it seems less and less people use it, but it still has its value.
I don't know if this is large enough for your usage, but the largest integer number I could represent in my common lisp environment (CLISP) was 2^(2^20)
>> (expt 2 (expt 2 20)
In lisp you can easily represent hex, dec, oct and bin as follows
>> \#b1010
10
>> \#o12
10
>> 10
10
>> \#x0A
10
You can write rationals in other bases from 2 to 36 with #nR
>> #36rABCDEFGHIJKLMNOPQRSTUVWXYZ
8337503854730415241050377135811259267835
For more information on numbers in lisp see: Practical Common Lisp Book