currently I'm using nonlin_curvefit function from GNU Octave's 'optim' package to fit data with . But this time I did also need the uncertainty of the returned parameters to determine the quality of the fit. After reading through the documentation I tied using the function curvefit_stat.
However whenever I alwas get errors using this functiona and I can't make any sense of the error message. I'm using Octave 4.2.2 from Ubuntu 18.04's default repository.
Standalone minimal example and error messages below. Startparameters init_cvg usually produces good result, while using init_dvg usually results in poor fits:
1;
x = linspace(-2*pi, 2*pi, 600);
ydata = 3.4*sin(1.6*x) + rand(size(x))*1.3;
f = #(p, x) p(1)*sin(p(2).*x);
function y = testfun(p ,x)
y = p(1).*sin(p(2).*x);
endfunction
init_cvg = [1; 1.1];
init_dvg = [1; 1.0];
[pc, mod_valc, cvgc, outpc] = nonlin_curvefit(f, init_cvg, x, ydata);
[pd, mod_vald, cvgd, outpd] = nonlin_curvefit(f, init_dvg, x, ydata);
hold off
plot(x, yd, "b.");
hold on;
plot(x, mod_valc, "r-");
plot(x, mod_vald, "color", [0.9 0.4 0.3]);
settings = optimset("ret_covp", true);
covpc = curvefit_stat(f, pc, x, ydata, settings);
covpd = curvefit_stat(f, pd, x, ydata, settings);
puts("sqrt(diag(covpc))")
sqrt(diag(covpc))
puts("sqrt(diag(covpd))")
sqrt(diag(covpd))
The first error message occurs when I use f as a model function, the second occurs when I use testfun instead:
>> curvefit_stat_TEST
error: label of objective function must be specified
error: called from
__residmin_stat__ at line 566 column 7
curvefit_stat at line 56 column 7
curvefit_stat_TEST at line 25 column 7
>> curvefit_stat_TEST
error: 'p' undefined near line 8 column 7
error: called from
testfun at line 8 column 5
curvefit_stat_TEST at line 25 column 7
>>
Could somebody confirm this error ?
I would appreciate any help.
I found the problem.
I needed to add "abjf_type", "wls" as arguments to optimset.
Related
I've got a little MatLab script, which I try to understand. It doesn't do very much. It only reads a text from a file and encode and decode it with the Huffman-functions.
But it throws an error while decoding:
"error: out of memory or dimension too large for Octave's index type
error: called from huffmandeco>dict2tree at line 95 column 19"
I don't know why, because I debugged it and don't see a large index type.
I added the part which calculates p from the input text.
%text is a random input text file in ASCII
%calculate the relative frequency of every Symbol
for i=0:127
nlet=length(find(text==i));
p(i+1)=nlet/length(text);
end
symb = 0:127;
dict = huffmandict(symb,p); % Create dictionary
compdata = huffmanenco(fdata,dict); % Encode the data
dsig = huffmandeco(compdata,dict); % Decode the Huffman code
I can oly use octave instead of MatLab. I don't know, if there is an unexpected error. I use the Octave Version 6.2.0 on Win10. I tried the version for large data, it didn't change anything.
Maybe anyone knows the error in this context?
EDIT:
I debugged the code again. In the function huffmandeco I found the following function:
function tree = dict2tree (dict)
L = length (dict);
lengths = zeros (1, L);
## the depth of the tree is limited by the maximum word length.
for i = 1:L
lengths(i) = length (dict{i});
endfor
m = max (lengths);
tree = zeros (1, 2^(m+1)-1)-1;
for i = 1:L
pointer = 1;
word = dict{i};
for bit = word
pointer = 2 * pointer + bit;
endfor
tree(pointer) = i;
endfor
endfunction
The maximum length m in this case is 82. So the function calculates:
tree = zeros (1, 2^(82+1)-1)-1.
So it's obvious why the error called a too large index type.
But there must be a solution or another error, because the code is tested before.
I haven't weeded through the code enough to know why yet, but huffmandict is not ignoring zero-probability symbols the way it claims to. Nor have I been able to find a bug report on Savannah, but again I haven't searched thoroughly.
A workaround is to limit the symbol list and their probabilities to only the symbols that actually occur. Using containers.Map would be ideal, but in Octave you can do that with a couple of the outputs from unique:
% Create a symbol table of the unique characters in the input string
% and the indices into the table for each character in the string.
[symbols, ~, inds] = unique(textstr);
inds = inds.'; % just make it easier to read
For the string
textstr = 'Random String Input.';
the result is:
>> symbols
symbols = .IRSadgimnoprtu
>> inds
inds =
Columns 1 through 19:
4 6 11 7 12 10 1 5 15 14 9 11 8 1 3 11 13 16 15
Column 20:
2
So the first symbol in the input string is symbols(4), the second is symbols(6), and so on.
From there, you just use symbols and inds to create the dictionary and encode/decode the signal. Here's a quick demo script:
textstr = 'Random String Input.';
fprintf("Starting string: %s\n", textstr);
% Create a symbol table of the unique characters in the input string
% and the indices into the table for each character in the string.
[symbols, ~, inds] = unique(textstr);
inds = inds.'; % just make it easier to read
% Calculate the frequency of each symbol in table
% max(inds) == numel(symbols)
p = histc(inds, 1:max(inds))/numel(inds);
dict = huffmandict(symbols, p);
compdata = huffmanenco(inds, dict);
dsig = huffmandeco(compdata, dict);
fprintf("Decoded string: %s\n", symbols(dsig));
And the output:
Starting string: Random String Input.
Decoded string: Random String Input.
To encode strings other than the original input string, you would have to map the characters to symbol indices (ensuring that all symbols in the string are actually present in the symbol table, obviously):
>> [~, s_idx] = ismember('trogdor', symbols)
s_idx =
15 14 12 8 7 12 14
>> compdata = huffmanenco(s_idx, dict);
>> dsig = huffmandeco(compdata, dict);
>> fprintf("Decoded string: %s\n", symbols(dsig));
Decoded string: trogdor
I wrote my very first octave script which is a code for the incremental search method for root finding but I encountered numerous errors that I found hard to understand.
The following is the script:
clear
syms x;
fct=input('enter your function in standard form: ');
f=str2func(fct); % This built in octave function creates functions from strings
Xmax=input('X maximum= ');
Xinit=input('X initial= ');
dx=input('dx= ');
epsi=input('epsi= ');
N=10; % the amount by which dx is decreased in case a root was found.
while (x<=Xmax)
f1=f(Xinit);
x=x+dx
f2=f(x);
if (abs(f2)>(1/epsi))
disp('The function approches infinity at ', num2str(x));
x=x+epsi;
else
if ((f2*f1)>0)
x=x+dx;
elseif ((f2*f1)==0)
disp('a root at ', num2str );
x=x+epsi;
else
if (dx < epsi)
disp('a root at ', num2str);
x=x+epsi;
else
x=x-dx;
dx=dx/N;
x=x+dx;
end
end
end
end
when running it the following errors showed up:
>> Incremental
enter your function in standard form: 1+(5.25*x)-(sec(sqrt(0.68*x)))
warning: passing floating-point values to sym is dangerous, see "help sym"
warning: called from
double_to_sym_heuristic at line 50 column 7
sym at line 379 column 13
mtimes at line 63 column 5
Incremental at line 3 column 4
warning: passing floating-point values to sym is dangerous, see "help sym"
warning: called from
double_to_sym_heuristic at line 50 column 7
sym at line 379 column 13
mtimes at line 63 column 5
Incremental at line 3 column 4
error: wrong type argument 'class'
error: str2func: FCN_NAME must be a string
error: called from
Incremental at line 4 column 2
Below is the flowchart of the incremental search method:
The problem happens in this line:
fct=input('enter your function in standard form: ');
Here input takes the user input and evaluates it. It tries to convert it into a number. In the next line,
f=str2func(fct)
you assume fct is a string.
To fix the problems, tell input to just return the user's input unchanged as a string (see the docs):
fct=input('enter your function in standard form: ', 's');
I want to run a MATLAB script M-file to reconstruct a point cloud in Octave. Therefore I had to rewrite some parts of the code to make it compatible with Octave. Actually the M-file works fine in Octave (I don't get any errors) and also the plotted point cloud looks good at first glance, but it seems that the variables are only half the size of the original MATLAB variables. In the attached screenshots you can see what I mean.
Octave:
MATLAB:
You can see that the dimension of e.g. M in Octave is 1311114x3 but in MATLAB it is 2622227x3. The actual number of rows in my raw file is 2622227 as well.
Here you can see an extract of the raw file (original data) that I use.
Rotation angle Measured distance
-0,090 26,295
-0,342 26,294
-0,594 26,294
-0,846 26,295
-1,098 26,294
-1,368 26,296
-1,620 26,296
-1,872 26,296
In MATLAB I created my output variable as follows.
data = table;
data.Rotationangle = cell2mat(raw(:, 1));
data.Measureddistance = cell2mat(raw(:, 2));
As there is no table function in Octave I wrote
data = cellfun(#(x)str2num(x), strrep(raw, ',', '.'))
instead.
Octave also has no struct2array function, so I had to replace it as well.
In MATLAB I wrote.
data = table2array(data);
In Octave this was a bit more difficult to do. I had to create a struct2array function, which I did by means of this bug report.
%% Create a struct2array function
function retval = struct2array (input_struct)
%input check
if (~isstruct (input_struct) || (nargin ~= 1))
print_usage;
endif
%convert to cell array and flatten/concatenate output.
retval = [ (struct2cell (input_struct)){:}];
endfunction
clear b;
b.a = data;
data = struct2array(b);
Did I make a mistake somewhere and could someone help me to solve this problem?
edit:
Here's the part of my script where I'm using raw.
delimiter = '\t';
startRow = 5;
formatSpec = '%s%s%[^\n\r]';
fileID = fopen(filename,'r');
dataArray = textscan(fileID, formatSpec, 'Delimiter', delimiter, 'HeaderLines' ,startRow-1, 'ReturnOnError', false, 'EndOfLine', '\r\n');
fclose(fileID);
%% Convert the contents of columns containing numeric text to numbers.
% Replace non-numeric text with NaN.
raw = repmat({''},length(dataArray{1}),length(dataArray)-1);
for col=1:length(dataArray)-1
raw(1:length(dataArray{col}),col) = mat2cell(dataArray{col}, ones(length(dataArray{col}), 1));
end
numericData = NaN(size(dataArray{1},1),size(dataArray,2));
for col=[1,2]
% Converts text in the input cell array to numbers. Replaced non-numeric
% text with NaN.
rawData = dataArray{col};
for row=1:size(rawData, 1)
% Create a regular expression to detect and remove non-numeric prefixes and
% suffixes.
regexstr = '(?<prefix>.*?)(?<numbers>([-]*(\d+[\.]*)+[\,]{0,1}\d*[eEdD]{0,1}[-+]*\d*[i]{0,1})|([-]*(\d+[\.]*)*[\,]{1,1}\d+[eEdD]{0,1}[-+]*\d*[i]{0,1}))(?<suffix>.*)';
try
result = regexp(rawData(row), regexstr, 'names');
numbers = result.numbers;
% Detected commas in non-thousand locations.
invalidThousandsSeparator = false;
if numbers.contains('.')
thousandsRegExp = '^\d+?(\.\d{3})*\,{0,1}\d*$';
if isempty(regexp(numbers, thousandsRegExp, 'once'))
numbers = NaN;
invalidThousandsSeparator = true;
end
end
% Convert numeric text to numbers.
if ~invalidThousandsSeparator
numbers = strrep(numbers, '.', '');
numbers = strrep(numbers, ',', '.');
numbers = textscan(char(numbers), '%f');
numericData(row, col) = numbers{1};
raw{row, col} = numbers{1};
end
catch
raw{row, col} = rawData{row};
end
end
end
You don't see any raw in my workspaces because I clear all temporary variables before I reconstruct my point cloud.
Also my original data in row 1311114 and 1311115 look normal.
edit 2:
As suggested here is a small example table to clarify what I want and what MATLAB does with the table2array function in my case.
data =
-0.0900 26.2950
-0.3420 26.2940
-0.5940 26.2940
-0.8460 26.2950
-1.0980 26.2940
-1.3680 26.2960
-1.6200 26.2960
-1.8720 26.2960
With the struct2array function I used in Octave I get the following array.
data =
-0.090000 26.295000
-0.594000 26.294000
-1.098000 26.294000
-1.620000 26.296000
-2.124000 26.295000
-2.646000 26.293000
-3.150000 26.294000
-3.654000 26.294000
If you compare the Octave array with my original data, you can see that every second row is skipped. This seems to be the reason for 1311114 instead of 2622227 rows.
edit 3:
I tried to solve my problem with the suggestions of #Tasos Papastylianou, which unfortunately was not successful.
First I did the variant with a struct.
data = struct();
data.Rotationangle = [raw(:,1)];
data.Measureddistance = [raw(:,2)];
data = cell2mat( struct2cell (data ).' )
But this leads to the following structure in my script. (Unfortunately the result is not what I would like to have as shown in edit 2. Don't be surprised, I only used a small part of my raw file to accelerate the run of my script, so here are only 769 lines.)
[766,1] = -357,966
[767,1] = -358,506
[768,1] = -359,010
[769,1] = -359,514
[1,2] = 26,295
[2,2] = 26,294
[3,2] = 26,294
[4,2] = 26,296
Furthermore I get the following error.
error: unary operator '-' not implemented for 'cell' operands
error: called from
Cloud_reconstruction at line 137 column 11
Also the approach with the dataframe octave package didn't work. When I run the following code it leads to the error you can see below.
dataframe2array = #(df) cell2mat( struct(df).x_data );
pkg load dataframe;
data = dataframe();
data.Rotationangle = [raw(:, 1)];
data.Measureddistance = [raw(:, 2)];
dataframe2array(data)
error:
warning: Trying to overwrite colum names
warning: called from
df_matassign at line 147 column 13
subsasgn at line 172 column 14
Cloud_reconstruction at line 106 column 20
warning: Trying to overwrite colum names
warning: called from
df_matassign at line 176 column 13
subsasgn at line 172 column 14
Cloud_reconstruction at line 106 column 20
warning: Trying to overwrite colum names
warning: called from
df_matassign at line 147 column 13
subsasgn at line 172 column 14
Cloud_reconstruction at line 107 column 23
warning: Trying to overwrite colum names
warning: called from
df_matassign at line 176 column 13
subsasgn at line 172 column 14
Cloud_reconstruction at line 107 column 23
error: RHS(_,2): but RHS has size 768x1
error: called from
df_matassign at line 179 column 11
subsasgn at line 172 column 14
Cloud_reconstruction at line 107 column 23
Both error messages refer to the following part of my script where I'm doing the reconstruction of the point cloud in cylindrical coordinates.
distLaserCenter = 47; % Distance between the pipe centerline and the blind zone in mm
m = size(data,1); % Find the length of the first dimension of data
zincr = 0.4/360; % z increment in mm per deg
data(:,1) = -data(:,1);
for i = 1:m
data(i,2) = data(i,2) + distLaserCenter;
if i == 1
data(i,3) = 0;
elseif abs(data(i,1)-data(i-1)) < 100
data(i,3) = data(i-1,3) + zincr*(data(i,1)-data(i-1));
else abs(data(i,1)-data(i-1)) > 100;
data(i,3) = data(i-1,3) + zincr*(data(i,1)-(data(i-1)-360));
end
end
To give some background information for a better understanding. The script is used to reconstruct a pipe as a point cloud. The surface of the pipe was scanned from inside with a laser and the laser measured several points (distance from laser to the inner wall of the pipe) at each deg of rotation. I hope this helps to understand what I want to do with my script.
Not sure exactly what you're trying to do, but here's a toy example of how a struct could be used in an equivalent manner to a table:
matlab:
data = table;
data.A = [1;2;3;4;5];
data.B = [10;20;30;40;50];
table2array(data)
octave:
data = struct();
data.A = [1;2;3;4;5];
data.B = [10;20;30;40;50];
cell2mat( struct2cell (data ).' )
Note the transposition operation (.') before passing the result to cell2mat, since in a table, the 'fieldnames' are arranged horizontally in columns, whereas the struct2cell ends up arranging what used to be the 'fieldnames' as rows.
You might also be interested in the dataframe octave package, which performs similar functions to matlab's table (or in fact, R's dataframe object): https://octave.sourceforge.io/dataframe/ (you can install this by typing pkg install -forge dataframe in your console)
Unfortunately, the way to display the data as an array is still not ideal (see: https://stackoverflow.com/a/55417141/4183191), but you can easily convert that into a tiny function, e.g.
dataframe2array = #(df) cell2mat( struct(df).x_data );
Your code can then become:
pkg load dataframe;
data = dataframe();
data.A = [1;2;3;4;5];
data.B = [10;20;30;40;50];
dataframe2array(data)
I have a function which I have defined as follows, you can see that it clearly requires 7 arguments;
def calc_z(w,S,var,a1,a2,yt1,yt2):
mu = w*S
sigma = mt.sqrt(var)
z = np.random.normal(mu,sigma)
u = [a1,a2,z]
yt = [yt1,yt2,1]
thetaset = np.random.rand(len(u))
m = [i for i in range(len(u))]
max_iter = 30
#Calculate E-step
for i in range(max_iter):
print 'Iteration:', i
print 'z:', z
print 'thetaset', thetaset
devLz = eq6(var,w,S,z,yt,u,thetaset,m)
dev2Lz2 = eq9(var,thetaset,u)
#Calculate M-Step
z = z - (devLz / dev2Lz2)
w = lambdaw * z
for i in range(len(thetaset)):
devLTheta = eq7(yt,u,thetaset,lambdatheta)
dev2LTheta2 = eq10(thetaset,u,lambdatheta)
thetaset = thetaset - (devLTheta / dev2LTheta2)
return z
I am using pyspark so I convert this to a udf
calc_z_udf = udf(calc_z,FloatType())
and then run it as follows (where I am clearly passing in 7 arguments - Or am I going mad!?);
data = data.withColumn('z', calc_z_udf(data['w'],data['Org_Depth_Diff_S'],data['var'],data['proximity_rank_a1'],data['cotravel_count_a2'],data['cotravel_yt1'],data['proximity_yt2']))
When I run this however I am getting an error which states:
TypeError: calc_z() takes exactly 7 arguments (6 given)
Could anyone help me with why this might be as it is clear that when I am running the function I am infact passing in 7 arguments and not 6 as the error states?
I am not sure its is the reason no need to send column objects you can just pass strings:
data = data.withColumn('z', calc_z_udf('w', 'Org_Depth_Diff_S','var', 'proximity_rank_a1', 'cotravel_count_a2', 'cotravel_yt1', 'proximity_yt2'))
I have been wanting to figure this out for a long time, but have had no success yet. I am assuming I will use arrayfun, but I couldn't figure it yet. Appreciate help. Here is the problem:
Given a matrix of many rows and N^2 columns, reshape every row to NxN matrix and calculate eigenvalues, and do this in a vectorized way not using for loop. For example
A=
0.6060168 0.8340029 0.0064574 0.7133187
0.6325375 0.0919912 0.5692567 0.7432627
0.8292699 0.5136958 0.4171895 0.2530783
0.7966113 0.1975865 0.6687064 0.3226548
0.0163615 0.2123476 0.9868179 0.1478827
for every **i**
m=reshape(A(i,:),2,2)
[vc vl]=eig(m)
I am inclined to do something like
f = #(x) eig(reshape(x,2,2))
arrayfun(f,A)
but of course I am getting an error like
octave:5> arrayfun(f,A)
error: reshape: can't reshape 1x1 array to 2x2 array
error: evaluating argument list element number 1
error: evaluating argument list element number 1
error: called from:
error: at line -1, column -1
error: cellfun: too many output arguments
error: /usr/share/octave/3.2.4/m/general/arrayfun.m at line 168, column 21
A = [0.6060168 0.8340029 0.0064574 0.7133187;
0.6325375 0.0919912 0.5692567 0.7432627;
0.8292699 0.5136958 0.4171895 0.2530783;
0.7966113 0.1975865 0.6687064 0.3226548;
0.0163615 0.2123476 0.9868179 0.1478827];
N = 2;
[mc, ml] = arrayfun (#(row) eig (reshape (A (row, :), N, N)), 1:rows(A), "UniformOutput", false)
mc =
{
[1,1] =
-0.170783 -0.044626
0.985309 -0.999004
[1,2] =
-0.95343 -0.89053
0.30161 -0.45492
(cropped)
}
ml =
{
[1,1] =
Diagonal Matrix
0.56876 0
0 0.75057
[1,2] =
Diagonal Matrix
0.45246 0
0 0.92334
(cropped)
With the Ndpar package, calculations can be parallelized over multiple cores.
Borrowing from Andy's answer,
pkg load ndpar
A = [0.6060168 0.8340029 0.0064574 0.7133187;
0.6325375 0.0919912 0.5692567 0.7432627;
0.8292699 0.5136958 0.4171895 0.2530783;
0.7966113 0.1975865 0.6687064 0.3226548;
0.0163615 0.2123476 0.9868179 0.1478827];
N = 2;
[eigenvectors, eigenvalues] = ndpar_arrayfun(nproc,
#(row) eig(reshape(row, N, N)),
A, "IdxDimensions", 1, "Uniformoutput", false)
yields the same output.
EDIT - Or with the original pararrayfun from the octave-forge parallel package:
[eigenvectors, eigenvalues] = pararrayfun(nproc,
#(row_idx) eig(reshape(A(row_idx, :), N, N)),
1:rows(A), "UniformOutput", false)