Related
I have data as real and imaginary parts of complex number, and I want to fit them according to a complex function. More in detail, they are data from electrochemical impedance spectroscopy (EIS) experiments, and the function comes from an equivalent circuit.
I am using Octave 7.2.0 in a Windows 10 computer. I need to use the leasqr algorithm, present in the Optim package. The leasqr used the Levenberg-Marquardt nonlinear regression, typical in EIS data fitting.
Regarding the data, xdata are linear frequency, y data are ReZ + j*ImZ.
If I try to fit the complex data with the complex fitting function, I get the following error:
error: weighted residuals are not real
error: called from
__lm_svd__ at line 147 column 20
leasqr at line 662 column 25
Code_for_StackOverflow at line 47 column 73
I tried to fit the real part of the data with the real part of the fitting function, and the imaginary parts of the data, with the imaginary part of the function. The fits are successfully performed, but I have two sets of fitted parameters, while I need only one set.
Here the code I wrote.
clear -a;
clf;
clc;
pkg load optim;
pkg load symbolic;
Linear_freq = [1051.432, 394.2871, 112.6535, 39.42871, 11.59668, 3.458659, 1.065641, 0.3258571, 0.1000221];
ReZ = [84.10412, 102.0962, 178.8031, 283.0663, 366.7088, 431.3653, 514.4105, 650.5853, 895.9588];
MinusImZ = [27.84804, 59.56786, 116.5972, 123.2293, 102.6806, 117.4836, 178.1147, 306.256, 551.2337];
Z = [88.5946744, 118.2030626, 213.4606653, 308.7264008, 380.8131426, 447.0776424, 544.3739605, 719.0646495, 1051.950932];
MinusPhase = [18.32042302, 30.26135402, 33.1083029, 23.52528583, 15.64255593, 15.23515301, 19.09841797, 25.2082044, 31.60167787];
ImZ = -MinusImZ;
Angular_freq = 2*pi*Linear_freq;
xdata = Angular_freq;
ydata = ReZ + j*ImZ;
Fitting_Function = #(xdata, p) (p(1) + ((p(2) + (1./(p(3)*(j*xdata).^0.5))).^-1 + (1./(p(4)*(j*xdata).^p(5))).^-1).^-1);
p = [80, 300, 6.63E-3, 5E-5, 0.8]; # True parameters values, taken with a dedicated software: 76, 283, 1.63E-3, 1.5E-5, 0.876
options.fract_prec = [0.0005, 0.0005, 0.0005, 0.0005, 0.0005].';
niter=400;
tol=1E-12;
dFdp="dfdp";
dp=1E-9*ones(size(p));
wt = abs(sqrt(ydata).^-1);
#[Fitted_Parameters_ReZ pfit_real cvg_real iter_real corp_real covp_real covr_real stdresid_real z_real r2_real] = leasqr(xdata, ReZ, p, Fitting_Function_ReZ, tol, niter, wt, dp, dFdp, options);
#[Fitted_Parameters_ImZ pfit_imag cvg_imag iter_imag corp_imag covp_imag covr_imag stdresid_imag z_imag r2_imag] = leasqr(xdata, ImZ, p, Fitting_Function_ImZ, tol, niter, wt, dp, dFdp, options);
[Fitted_Parameters pfit cvg iter corp covp covr stdresid z r2] = leasqr(xdata, ydata, p, Fitting_Function, tol, niter, wt, dp, dFdp, options);
#########################################################################
# Calculate the fitted functions, with the fitted paramteres array
#########################################################################
Fitted_Function_Real = real(pfit_real(1) + ((pfit_real(2) + (1./(pfit_real(3)*(j*xdata).^0.5))).^-1 + (1./(pfit_real(4)*(j*xdata).^pfit_real(5))).^-1).^-1);
Fitted_Function_Imag = imag(pfit_imag(1) + ((pfit_imag(2) + (1./(pfit_imag(3)*(j*xdata).^0.5))).^-1 + (1./(pfit_imag(4)*(j*xdata).^pfit_imag(5))).^-1).^-1);
Fitted_Function = Fitted_Function_Real + j.*Fitted_Function_Imag;
Fitted_Function_Mod = abs(Fitted_Function);
Fitted_Function_Phase = (-(angle(Fitted_Function))*(180./pi));
################################################################################
# Calculate the residuals, from https://iopscience.iop.org/article/10.1149/1.2044210
# An optimum fit is obtained when the residuals are spread randomly around the log Omega axis.
# When the residuals show a systematic deviation from the horizontal axis, e.g., by forming
# a "trace" around, above, or below the log co axis, the complex nonlinear least squares (CNLS) fit is not adequate.
################################################################################
Residuals_Real = (ReZ-Fitted_Function_Real)./Fitted_Function_Mod;
Residuals_Imag = (ImZ-Fitted_Function_Imag)./Fitted_Function_Mod;
################################################################################
# Calculate the chi-squared - reduced value, with the fitted paramteres array NOVA manual page 452
################################################################################
chi_squared_ReZ = sum(((ReZ-Fitted_Function_Real).^2)./Z.^2)
chi_squared_ImZ = sum(((ImZ-Fitted_Function_Imag).^2)./Z.^2)
Pseudo_chi_squared = sum((((ReZ-Fitted_Function_Real).^2)+((ImZ-Fitted_Function_Imag).^2))./Z.^2)
disp('The values of the parameters after the fit of the real function are '), disp(pfit_real);
disp('The values of the parameters after the fit of the imaginary function are '), disp(pfit_imag);
disp("R^2, the coefficient of multiple determination, intercept form (not suitable for non-real residuals) is "), disp(r2_real), disp(r2_imag);
###################################################
## PLOT Data and the Function
###################################################
#Set plot parameters
set(0, "defaultlinelinewidth", 1);
set(0, "defaulttextfontname", "Verdana");
set(0, "defaulttextfontsize", 20);
set(0, "DefaultAxesFontName", "Verdana");
set(0, 'DefaultAxesFontSize', 12);
figure(1);
## Nyquist plot (Argand diagram)
subplot(1,2,1, "align");
plot((ReZ), (MinusImZ), "o", "markersize", 2, (Fitted_Function_Real), -(Fitted_Function_Imag), "-k");
axis ("square");
grid on;
daspect([1 1 2]);
title ('Nyquist Plot - Argand Diagram');
xlabel ('Z'' / \Omega' , 'interpreter', 'tex');
ylabel ('-Z'''' / \Omega', 'interpreter', 'tex');
## Bode Modulus
subplot (2, 2, 2);
loglog((Linear_freq), (Z), "o", "markersize", 2, (Linear_freq), (Fitted_Function_Mod), "-k");
grid on;
title ('Bode Plot - Modulus');
xlabel ('\nu (Hz)' , 'interpreter', 'tex');
ylabel ('|Z| / \Omega', 'interpreter', 'tex');
## Bode Phase
subplot (2, 2, 4);
semilogx((Linear_freq), (MinusPhase), "o", "markersize", 2, (Linear_freq), (Fitted_Function_Phase), "-k");
set(gca,'YTick',0:10:90);
grid on;
title ('Bode Plot - Phase');
xlabel ('\nu (Hz)' , 'interpreter', 'tex');
ylabel ('-\theta (°)', 'interpreter', 'tex');
figure(2)
## Bode Z'
subplot (2, 1, 1);
semilogx((Linear_freq), (ReZ), "o", "markersize", 2, (Linear_freq), (Fitted_Function_Real), "-k");
grid on;
title ('Bode Plot Z''');
xlabel ('\nu (Hz)' , 'interpreter', 'tex');
ylabel ('Z'' / \Omega', 'interpreter', 'tex');
## Bode -Z''
subplot (2, 1, 2);
#subplot (2, 2, 4);
semilogx((Linear_freq), (MinusImZ), "o", "markersize", 2, (Linear_freq), -(Fitted_Function_Imag), "-k");
grid on;
title ('Bode Plot -Z''''');
xlabel ('\nu (Hz)' , 'interpreter', 'tex');
ylabel ('-Z'''' / \Omega', 'interpreter', 'tex');
figure(3)
## Residuals Real
subplot (2, 1, 1);
semilogx((Angular_freq), (Residuals_Real), "-o", "markersize", 2);
grid on;
title ('Residuals Real');
xlabel ('\omega (Hz)' , 'interpreter', 'tex');
ylabel ('\Delta_{re} / \Omega', 'interpreter', 'tex');
## Residuals Imaginary
subplot (2, 1, 2);
#subplot (2, 2, 4);
semilogx((Angular_freq), (Residuals_Imag), "-o", "markersize", 2);
grid on;
title ('Residuals Imaginary');
xlabel ('\omega (Hz)' , 'interpreter', 'tex');
ylabel ('\Delta_{im} / \Omega', 'interpreter', 'tex');
Octave should be able to handle complex numbers. What do I do wrong?
I was thinking to fit the real part of the data with the real part of the fitting function, and then using the Kramers-Kronig relations to get the imaginary part of the fitted function, but I would like to avoid this method, if possible.
Any help would be greatly appreciated, thanks in advance.
From your data drawing the complex impedances diagram makes appear a rather common shape that can be model with a lot of equivalent circuits :
Reference : https://fr.scribd.com/doc/71923015/The-Phasance-Concept
You chose the model n°2 probably according to some physical considerations. This is not the subject to be discussed here.
Also according to physical consideration and/or by graphical inspection you correctly assumed that one phasance is of Warbourg kind (Phi=-pi/4 ; nu=-1/2).
The problem is to fit an equation with five adjustable parameters. This is a difficult problem of non linear regression of a complex equation. The usual method consists in an iterative process starting from "guessed values" of the five parameters.
The "guessed values" have to be not far from the unknown correct values. One can find some approximates from graphical inspection of the impedances diagram. Often this is a cause of failure of convergence of the iterative process.
A more reliable method consists in using a combination of linear regression wrt most of the parameters and non-linear regression wrt only few parameters.
In the present case it is shown below that the nonlinear regression can be reduced to only one parameter while the other parameters can be handled by a simple linear regression. This is a big simplification.
A software for mixed linear and nonlinear regression (in cases involving several phasors) was developed in years 1980-1990. Infortunately I have no access to it presently.
Nevertheless in the present case of one phasor only we don't need a sledgehammer to crack a nut. The Newton-Raphson method is sufficient. Graphical inspection gives a rough approximate of (nu) between -0.7 and -0.8 The chosen initial value is nu=-0.75 giving the next first run :
Since all calculus are carried out in complex numbers the resulting values are complex instead of real as expected. They are noted ZR1, ZR2, ZP1, ZP2 to distinguish from real R1, R2, P1, P2. This is because the value of (nu) isn't optimal.
The more (nu) converges to the final value the more the imaginary parts vanishes. After a few runs of the Newton-Raphson process the imaginary parts become quite negligible. The final result is shown below.
Publications :
"Contribution à l'interprétation de certaines mesures d'impédances". 2-ième Forum sur les Imédances Electrochimiques, 28-29 octobre 1987.
"Calcul de réseau électriques équivalents à partir de mesures d'impédances". 3-ième Forum sur les Imédances Electrochimiques, 24 novembre 1988.
"Synthèse de circuits électiques équivalents à partir de mesures d'impédances complexes". 5-ième Forum sur les Imédances Electrochimiques, 28 novembre 1991.
geopandas.sjoin_nearest takes parameters max_distance and distance_col. What is the units of the distances / how do I interpret them? Is it degrees?
https://geopandas.org/en/stable/docs/reference/api/geopandas.sjoin_nearest.html#geopandas.sjoin_nearest
While geopandas provides utilities for converting between coordinate systems (e.g. to_crs), most operations in geopandas ignore the projection information. Spatial operations such as distance, area, buffer, etc. are done in whatever units the geometries are in. If your geometries are in meters, these will be in meters. If they're in degrees, they'll be in degrees.
For example, let's take a look at the natural earth dataset. You can see that the geometry column is in lat/lon coordinates by just looking at the values:
In [1]: import geopandas as gpd
In [2]: gdf = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
In [3]: gdf
Out[3]:
pop_est continent name iso_a3 gdp_md_est geometry
0 920938 Oceania Fiji FJI 8374.0 MULTIPOLYGON (((180.00000 -16.06713, 180.00000...
1 53950935 Africa Tanzania TZA 150600.0 POLYGON ((33.90371 -0.95000, 34.07262 -1.05982...
2 603253 Africa W. Sahara ESH 906.5 POLYGON ((-8.66559 27.65643, -8.66512 27.58948...
3 35623680 North America Canada CAN 1674000.0 MULTIPOLYGON (((-122.84000 49.00000, -122.9742...
4 326625791 North America United States of America USA 18560000.0 MULTIPOLYGON (((-122.84000 49.00000, -120.0000...
.. ... ... ... ... ... ...
172 7111024 Europe Serbia SRB 101800.0 POLYGON ((18.82982 45.90887, 18.82984 45.90888...
173 642550 Europe Montenegro MNE 10610.0 POLYGON ((20.07070 42.58863, 19.80161 42.50009...
174 1895250 Europe Kosovo -99 18490.0 POLYGON ((20.59025 41.85541, 20.52295 42.21787...
175 1218208 North America Trinidad and Tobago TTO 43570.0 POLYGON ((-61.68000 10.76000, -61.10500 10.890...
176 13026129 Africa S. Sudan SSD 20880.0 POLYGON ((30.83385 3.50917, 29.95350 4.17370, ...
[177 rows x 6 columns]
Specifically, it's in WGS84 (aka EPSG:4326). The units are degrees:
In [4]: gdf.crs
Out[4]:
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- name: World.
- bounds: (-180.0, -90.0, 180.0, 90.0)
Datum: World Geodetic System 1984 ensemble
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich
If we call the area property, geopandas will issue a warning, but it will still calculate the area for us. The total area of the earth comes out to 21,497 degrees^2, which roughly 1/3 of 180*360:
In [6]: gdf.area.sum()
<ipython-input-6-10238de14784>:1: UserWarning: Geometry is in a geographic CRS. Results from 'area' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.
gdf.area.sum()
Out[6]: 21496.990987992736
If we instead use an equal area projection, we'll get something much closer to the land area of the earth, in m^2:
In [10]: gdf.to_crs('+proj=cea').area.sum() / 1e3 / 1e3 / 1e6
Out[10]: 147.36326937311017
The output of this image from gdalinfo 20141030-wv03.tif is like this:
Driver: GTiff/GeoTIFF
Files: 20141030-wv03.tif
Size is 16484, 15253
Coordinate System is:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (113.959353776485997,23.091020758099145)
Pixel Size = (0.000002966620901,-0.000002966620901)
Metadata:
AREA_OR_POINT=Area
DataType=Generic
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 113.9593538, 23.0910208) (113d57'33.67"E, 23d 5'27.67"N)
Lower Left ( 113.9593538, 23.0457709) (113d57'33.67"E, 23d 2'44.78"N)
Upper Right ( 114.0082556, 23.0910208) (114d 0'29.72"E, 23d 5'27.67"N)
Lower Right ( 114.0082556, 23.0457709) (114d 0'29.72"E, 23d 2'44.78"N)
Center ( 113.9838047, 23.0683958) (113d59' 1.70"E, 23d 4' 6.22"N)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
NoData Value=256
Band 2 Block=128x128 Type=Byte, ColorInterp=Green
NoData Value=256
Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
NoData Value=256
The spatial resolution is (0.000002966620901,-0.000002966620901), how to understand this value?
I also check another image from WorldView 2, ths output is:
Driver: GTiff/GeoTIFF
Files: 20150708.tif
Size is 9984, 10132
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 50N",
BASEGEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]],
CONVERSION["UTM zone 50N",
METHOD["Transverse Mercator",
ID["EPSG",9807]],
PARAMETER["Latitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",117,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Scale factor at natural origin",0.9996,
SCALEUNIT["unity",1],
ID["EPSG",8805]],
PARAMETER["False easting",500000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["(E)",east,
ORDER[1],
LENGTHUNIT["metre",1]],
AXIS["(N)",north,
ORDER[2],
LENGTHUNIT["metre",1]],
USAGE[
SCOPE["unknown"],
AREA["World - N hemisphere - 114°E to 120°E - by country"],
BBOX[0,114,84,120]],
ID["EPSG",32650]]
Data axis to CRS axis mapping: 1,2
Origin = (291153.100000000034925,2705938.760000000242144)
Pixel Size = (0.510000000000000,-0.510000000000000)
Metadata:
AREA_OR_POINT=Area
TIFFTAG_XRESOLUTION=1
TIFFTAG_YRESOLUTION=1
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 291153.100, 2705938.760) (114d56'22.85"E, 24d27'10.88"N)
Lower Left ( 291153.100, 2700771.440) (114d56'25.58"E, 24d24'22.98"N)
Upper Right ( 296244.940, 2705938.760) (114d59'23.60"E, 24d27'13.32"N)
Lower Right ( 296244.940, 2700771.440) (114d59'26.26"E, 24d24'25.41"N)
Center ( 293699.020, 2703355.100) (114d57'54.57"E, 24d25'48.15"N)
Band 1 Block=9984x1 Type=Byte, ColorInterp=Red
Band 2 Block=9984x1 Type=Byte, ColorInterp=Green
Band 3 Block=9984x1 Type=Byte, ColorInterp=Blue
The spatial resolution is (0.510000000000000,-0.510000000000000). How do I understand their difference between them? Thanks.
Your images are in two different coordinate systems.
Your second file 20150708.tif is in an UTM projection (UTM 50N to be exact) which has map units in meters - that's why the pixel resolutions is in meters (0.51m).
Your first file 20141030-wv03.tif is in a geographic coordinate system, the widely used World Geodetic System 1984 (or WGS84) which has map units in degrees, giving you the pixel resolution also in (decimal) degrees. On the equator 0.00001 degrees is around 1.11 meters so both images have likely the same resolution.
For more info on WGS84 vs UTM, this post on GIS stackexchange might be interesting.
I'm working on a Named Entity Recognition on resume dataset and we have entities like dates, phone, email etc.,,
And I'm working how to preprocess those entities. I'm currently adding a space after each puncuation like this,
DAVID B-Name
John I-Name
, O
IT O
Washington B-Address
, I-Address
DC I-Address
( B-Phone
107 I-Phone
) I-Phone
155
- I-Phone
4838 I-Phone
david B-Email
. I-Email
John I-Email
# I-Email
gmail I-Email
. I-Email
com I-Email
But I'm starting to question the process on how to handle such text during inference. I'm assuming even at inference we have to preprocess text using same process that is adding a space after each puncuation isn't it?
But it won't be so readable right?
For example at inference I have to provide input text like test # example . com? which is not readable isn't it? It only be able to predict entities in such format.
The problem you're trying to deal with is called tokenization. To deal with the formatting issue that you raise, often frameworks will extract the tokens from the underlying text in a way preserves the original text, such as keeping track of the character starts and ends for each token.
For instance, SpaCy in Python returns an object that stores all of this information:
import spacy
from pprint import pprint
nlp = spacy.load("en_core_web_sm")
doc = nlp("DAVID John, IT\nWashington, DC (107) 155-4838 david.John#gmail.com")
pprint([(token.text, token.idx, token.idx + len(token.text)) for token in doc])
output:
[('DAVID', 0, 5),
('John', 6, 10),
(',', 10, 11),
('IT', 12, 14),
('\n', 14, 15),
('Washington', 15, 25),
(',', 25, 26),
('DC', 27, 29),
('(', 30, 31),
('107', 31, 34),
(')', 34, 35),
('155', 36, 39),
('-', 39, 40),
('4838', 40, 44),
('david.John#gmail.com', 45, 65)]
You could either do the same sort of thing for yourself (e.g. keep a counter as you add spaces) or use an existing tokenizer (such as SpaCy, CoreNLP, tensorflow, etc.)
I am getting this error message while I am trying to insert my data into line table, using coordinates for points between road segments.
ERROR: parse error - invalid geometry
HINT: "SRID=27700;LINESTRING((5" <-- parse error at position 24 within geometry
SQL state: XX000
Part of my code:
NSERT INTO public."RoadSegments"("no", "seg_ID", "description", "location", "length", "the_geom")
VALUES
(1,'Seg_1','Shephards Bush to Royal Crescent','Shephards Bush','540',ST_GeomFromEWKT('SRID=27700;LINESTRING((51.504593 -0.220437),(51.505233 -0.214105))')),
(2,'Seg_2','Royal Crescent to Norland Square','Notting Hill','306',ST_GeomFromEWKT('SRID=27700;LINESTRING((51.505233 -0.214105),(51.506053 -0.209956))')),
(3,'Seg_3','Norland Square to Holland Park','Notting Hill','383',ST_GeomFromEWKT('SRID=27700;LINESTRING((51.506053 -0.209956),(51.507575 -0.204795))')),
(4,'Seg_4','Holland Park to Notting Hill Gate','Notting Hill','477',ST_GeomFromEWKT('SRID=27700;LINESTRING((51.507575 -0.204795),(51
You seem to have too much parentheses in your LINESTRING parameters. Try the following:
INSERT INTO public."RoadSegments"("no", "seg_ID", "description", "location", "length", "the_geom")
VALUES
(1,'Seg_1','Shephards Bush to Royal Crescent', 'Shephards Bush','540',ST_GeomFromEWKT('SRID=27700;LINESTRING(51.504593 -0.220437,51.505233 -0.214105)')),
(2,'Seg_2','Royal Crescent to Norland Square', 'Notting Hill', '306',ST_GeomFromEWKT('SRID=27700;LINESTRING(51.505233 -0.214105,51.506053 -0.209956)')),
(3,'Seg_3','Norland Square to Holland Park', 'Notting Hill', '383',ST_GeomFromEWKT('SRID=27700;LINESTRING(51.506053 -0.209956,51.507575 -0.204795)')),
(4,'Seg_4','Holland Park to Notting Hill Gate','Notting Hill', '477',ST_GeomFromEWKT('SRID=27700;LINESTRING(51.507575 -0.204795,51 ... '))
This should be the correct syntax (no extra parentheses around the coordinates)