Is it possible using Python libraries like pysolar, pvlib or similarly to calculate the
irradiance (Perez model) per day of the year
of a 1 m^2 flat square
situated at a certain latitude longitude
If you do not already have irradiance data I would recommend either using data from PVGIS or NREL's NSRDB PSM3 (Americas).
In either case, I would suggest you first get familiar with the service through the web application. This should give you a good idea of the geographical coverage and the available parameters and settings.
Then, you can use the corresponding pvlib function to programmatically retrieve data. The function to retrieve PVGIS hourly data is: pvlib.iotools.get_pvgis_hourly().
A short example of how to retrieve irradiance data from PVGIS is shown below:
import pvlib
import pandas as pd
data, inputs, meta = pvlib.iotools.get_pvgis_hourly(
latitude=55.7905, # North is positive
longitude=12.5250, # East is positive
start=pd.Timestamp('2020-01-01'), # First available year is 2005
end=pd.Timestamp('2020-12-31'), # Last available year is 2020 (depends on database)
raddatabase='PVGIS-SARAH2',
surface_tilt=5, # surface tilt angle
surface_azimuth=0, # 0 degrees corresponds to south
components=True, # Whether you want the individual components or just the total
url='https://re.jrc.ec.europa.eu/api/v5_2/', # URL for version 5.2
)
data[['poa_direct','poa_sky_diffuse','poa_ground_diffuse']].plot(
figsize=(6,4), subplots=True, sharex=True)
Related
So I developed a code where I read a file with some locations (15 to be more precise), where the first one is the depot and the other 14 locations are the places where a bus needs to pass to collect patients. In order to do that, I use Google Maps API key in order to collect the real distances and write them in a .txt file in the end.
import pandas as pd
import googlemaps
from itertools import tee
import numpy as np
#Read CSV file into data frame named 'df'
#change seperator (sep e.g. ',') type if necessary
df = pd.read_csv("D:/Utilizadores/Documents/FEUP/2018-2019/1º Semestre/PDI/Relatório/locais2.txt", sep='\\t',
engine='python', skiprows=[0], names=["address", "latitude", "longitude"])
lat = np.expand_dims(np.array(df["latitude"]), 1)
lon = np.expand_dims(np.array(df["longitude"]), 1)
coordinates = np.concatenate((lat, lon), axis=1)
coordinates = list(coordinates)
coordinates = [list(coor) for coor in coordinates]
#Perform request to use the Google Maps API web service
#API_key = 'AIzaSyCi8DDz_CCiVwW2JtvT6i-XpJYiEwxFryI'
API_key = 'AIzaSyCpumDcRbbteV64xlGOUQ5_Bah8Ja5gdJ4'
gmaps = googlemaps.Client(key=API_key)
result = gmaps.distance_matrix(coordinates, coordinates, mode='driving')["rows"]
distance_matrix = np.zeros((len(result), len(result)))
for i in range(len(result)):
for j in range(len(result)):
distance_matrix[i, j] = result[i]["elements"][j]["distance"]["value"]
np.savetxt("D:/Utilizadores/Documents/FEUP/2018-2019/1º Semestre/PDI/Relatório/locais_output.txt", distance_matrix, delimiter=",", fmt='%d')
print(distance_matrix)
The distances I want are from one place to every place, so the result I want is a matrix of 15x15 where the diagonal is filled with 0's. But it keeps printing this error:
"googlemaps.exceptions.ApiError: MAX_ELEMENTS_EXCEEDED".
The only way to not have an error is to put a limit on the reading of the file of 10 locations, including the depot:
result = gmaps.distance_matrix(coordinates[0:10], coordinates[0:10], mode='driving')["rows"]
Why is that? Anyone?
from the documentation
MAX_ELEMENTS_EXCEEDED indicates that the product of origins and destinations exceeds the per-query limit.
from "usage and billing":
Each query sent to the Distance Matrix API generates elements, where the number of origins times the number of destinations equals the number of elements.
Other Usage Limits
While you are no longer limited to a maximum number of elements per day (EPD), the following usage limits are still in place for the Distance Matrix API:
Maximum of 25 origins or 25 destinations per request.
Maximum 100 elements per server-side request.
Maximum 100 elements per client-side request.
1000 elements per second (EPS), calculated as the sum of client-side and server-side queries.
15 x 15 = 225 which is greater than the maximum allowed (100)
That you don't want or need some of the responses is not relevant.
I'm new here and I really want some help. I have a dataset including geographical information (longitude, latitude.. ) and I want to ensure the prediction of some aspects using this dataset with Support Vector Regression, but I don't know how to perform this task. I have the following inquires,
Is there a specific precessing I need to go through?
Does SVR consider a geographic dataset as normal data set or are there some specificities in term of tools and treatment?
Any recommended prediction analytics tools (including SVR) considering geographical data?
This given solution is for the situation that you want to extract the independent variable base on the dependent variable from a raster.
but if you have you all dependent and independent data with their corresponding location you simply use svm function in R and you then add a raster or vector (new) data to your predict function for prediction, or you also can use the estimated coefficient of dependent variable in raster calculator in GIS and multiply them to the corresponding independent variable and finally you will get your predicted raster.
Simply you can do the following for spatial data in R.
First of all, the support vector regression can be used for prediction of real value and you can use the library("e1071") in R in order to execute this algorithm.
you can import your dataset as CSV along with lat and long columns.
transform your data.fram to Spatial data.frame
#Read data
dat<-read.csv(choose.files())
#convert the data to SPDF.
dat_sp=SpatialPoints(cbind(dat$x,dat$y))
#add your Geographical referense system
dat_crs=CRS("+proj=utm +zone=39 +datum=WGS84")
#Data Frams for SpatialPoint Data(Creating a SpatialPoints data frame for dat)
dat_spdf=SpatialPointsDataFrame(coords = dat_sp,data = dat, proj4string = dat_crs)
plot(dat_spdf, col='blue', cex=1, pch=16, axes=TRUE)
#Extract value
dat_spdf$ref <- extract(raster , dat_spdf)
then you can extract your data on a raster data or whatever you have(your independent variable).
and finally, you can use the following cold in R.
SVM(dependent ~.,independent)
But you need to really have an intuition about what the SVR is and how to evaluate the result.
you also can show your result as a final raster map.
you can use toolbox package or you may use raster package.
Bit of a challenge here which I've been grappling with for some time. I'll explain my full work flow so you can reproduce if needed.
I'm creating virtual landscapes in Google SketchUp which I ultimately would like to use in Netlogo to examine how turtles interact with them.
My problem is that by the time I get the landscapes into Netlogo the units don't seem to relate to the original 3D model.
Step 1: Create simple hill on a 50m by 50m square in Sketchup using the Toposhaper extension.
Step 2: Export to .dae file and import into Meshlab, ensure the Meshlab model has the same dimensions as the Sketchup model by adjusting the units with the assistance of the measuring tool. Export from meshlab as .xyz file.
Step 3: Import .xyz file into QGis as points by adding a new layer from delimited file. Selecting field_1 and field_2 as X and Y fields.
Step 4: Create raster of points using Raster > Interpolation > Interpolation. Add field_3 as interpolation attribute, set number of columns to 50 by 50 (to correspond to the 50m x 50m 3D model), adjust cell size X and Y to match to ensure Netlogo will read the resulting .asc file.
Step 5: Finally, I setup a model in Netlogo to receive the raster. Firstly, in model settings I set the the min and max pxor and pycor to 0 and 50. Then, using the Gis Extension, I import the raster apply the z-value to a patch variable called elevation:
to load-gis
set elevation gis:load-dataset "cone_50.asc"
gis:set-world-envelope-ds gis:envelope-of elevation
gis:apply-raster elevation target-elev
end
Now, each patch of my 50 by 50 Netlogo world should have an elevation value taken from my 50 by 50 raster. In theory, adding all the elevation values together should (roughly) give me the total volume of the raised area of the hill? The figure I get is higher however and the problem gets worse with larger volumes.
Can anyone help?
I am trying to get the current traffic conditions at a particular location. The GTrafficOverlay object mentioned here only provides an overlay on an existing map.
Does anyone know how I can get this data from Google using their API?
It is only theorical, but there is perhaps a way to extract those data using the distancematrix api.
Method
1)
Make a topological road network, with node and edge, something like this:
Each edge will have four attributes: [EDGE_NUMBER;EDGE_SPEED;EDGE_TIME,EDGE_LENGTH]
You can use the openstreetmap data to create this network.
At the begining each edge will have the same road speed, for example 50km/h.
You need to use only the drivelink and delete the other edges. Take also into account that some roads are oneway.
2)
Randomly chose two nodes that are not closer than 5 or 10km
Use the dijsktra shortest path algorithm to calculate the shortest path between this two nodes (the cost = EDGE_TIME). Use your topological network to do that. The output will look like:
NODE = [NODE_23,NODE_44] PATH = [EDGE_3,EDGE_130,EDGE_49,EDGE_39]
Calculate the time needed to drive between the two nodes with the distance matrix api.
Preallocate a matrix A of size N X number_of_edge filled with zero value
Preallocate a matrix B of size 1 X number_of_edge filled with zero value
In the first row of matrix A fill each column (corresponding to each edge) with the length of the edge if the corresponding edge is in the path.
[col_1,col_2,col_3,...,col_39,...,col_49,...,col_130]
[0, 0, len_3,...,len_39,...,len_49,...,len_130] %row 1
In the first row of matrix B put the time calculated with the distance matrix api.
Then select two news node that were not used in the first path and repeat the operation until that there is no node left. (so you will fill the row 2, the row 3...)
Now you can solve the linear equation system: Ax = B where speed = 1/x
Assign the new calculated speed to each edge.
3)
Iterate the point 2) until your calculated speed start to converge.
Comment
I'm not sure that the calculated speed will converge, it will be interesting to test the method.I will try to do that if I got some time.
The distance matrix api don't provide a traveling time more precise than 1 minute, that's why the distance between the pair of node need to be at least 5 or 10 or more km.
Also this method fails to respect the Google's terms of service.
Google does not make available public API for this data.
Yahoo has a feed (example) with traffic conditions -- construction, accidents, and such. A write-up on how to access it is here.
If you want actual road speeds, you will probably need to work with a commercial provider.
I am reverse engineering a transportation visualization app. I need to find out the latitude for the origin of their data feed. Specifically what XY 0,0 is. The only formulas I have found calculate distance between two points, or location of a bearing/distance.
They use the XY to display a map in a very legacy application. The XY is in FEET.
I have these coordinates:
47.70446615506108, -122.34469839507263: x=1268314, y=260622
47.774182540800616,-122.3412994737105: x=1269649, y=286031
47.60024792289405, -122.32767331735774: x=1271767, y=222532
47.57012494413499, -122.29129609983679: x=1280532, y=211374
I need to find out what the latitude and longitude of x=0, y=0 is and what the formula would be to find this out.
They have two data feeds, one is more current than the other. The feed with the most current data does NOT include latitude, longitude, but only XY. I am trying to extrapolate based on their less current, yet more informative (includes lat, lon) data feed what 0,0 is so I can simply convert their (more current) data feed's XY coordinates to latitude and longitude.
If you look at the first 2 lines of data, and subtract the latitude
47.7044 - 47.7741 = -0.06972 degrees
There are 60 nautical miles per degree of latitude, and 6076 feet per nautical mile.
-.06972 * 60 * 6076 = 25,415 ft
Subtracting the two 'Y' values:
260662 - 286031 = 25,409 ft
So indeed that seems to prove the X and Y values are in feet.
If you take any of the Y values, and convert back to degrees, for example
260622 ft / ( 6076 ft/nm ) / ( 60 nm/degree ) = .71
286031 ft / 6076 / 60 = .78
So subtracting those values from the latitudes of (47.70 and 47.77) gives you very close to exactly 47 degrees, which should be your y=0 point.
For longitude, a degree is 60 nautical miles at the equator and 0 miles at the poles. So the number of miles per degree has to be multiplied by the cosine of the latitude, so approx cos(47 degrees), or .68. So instead of 6076 nm per degree, it's about 4145 nm.
So for the X values,
1268314 ft / ( 4145 ft/nm ) / ( 60 nm/degree ) = 5.10 degrees
1269649 ft / 4145 / 60 = 5.10 degrees
These X numbers increase as the latitude increases (less negative), so I believe you should add 5.1 degrees, which means the X base point is about
-122.3 + 5.1 = 117.2 West longitude for your x=0 point.
This is roughly the position of Spokane WA.
So given X=1280532, Y=211374
Lat = 47 + ( 211374 / 6096 / 60 ) = 47.58
Lon = -117.2 - ( 1280532 / ( 6096 * cos(47.58)) / 60 ) = -122.35
Which is roughly equivalent to the given data 47.57 and -122.29
The variance may be due to different projections - the X,Y system may be a "flattened" projection as opposed to lat/long which apply to a spherical projection? So to be accurate you may yet need more advanced math or that open source library :)
This question may also be helpful, it contains code for calculating great circle distances:
Calculate distance between two latitude-longitude points? (Haversine formula)
There are many different coordinate systems. You need to find out the what the coordinate systems are for both the lat/lon's (e.g. WGS84 etc) and x/y's first (e.g. some sort of projected system probably).
Once you have that information there are several tools you can use to do conversions and manipulations. One example (of a free open source coding library) is proj4.
Ask them what coordinate system they're using! (or if you got the dataset from some database, look at the metadata for the dataset and it should tell you. Otherwise I'd be skeptical of its value)
Most likely this is one of the state plane coordinate systems. They're for localized areas of the earth (kind of like UTM), and are frequently used for surveying.
You can use CORPSCON (or other GIS programs; ExpertGPS will do this if you have the GIS Option Pack but it's not free. I forget whether GPSBabel does conversion) to convert between lat/long and any of the state plane coordinate systems. You'll also need to know which datum the coordinates are in. WGS84 and NAD83 are very close but NAD27 is different.
You've got good advice on coordinate systems already, so I'll just chime in with the library I've used with great success in the past.
Geotrans is approved for use by the US Department of Defence, so you can be sure that it is well tested. You can grab it from here:
http://earth-info.nga.mil/GandG/geotrans/index.html
That might not be the right link as that page talks about the application, not the library. I expect the library is in the Developers package. Licensing terms were very liberal from memory, but make sure you review the terms before using it commercially.
Edit:
An interesting discussion on Geotrans licensing can be found here:
http://www.mail-archive.com/debian-legal#lists.debian.org/msg39263.html
Over here, I said this:
In Java, I would use the OpenMap converter from a point's expression in UTM to one using Latitude and Longitude (assuming a WGS-84 ellipsoid which is most commonly used in GPS).
OpenMap is open source and I would post a link to their download page but they have a short license script in the way. So, to avoid being rude, I won't deep link. Instead, head to their homepage and click Downloads.
That should either solve your problem directly or at least point you towards a useful algorithm.
I've used Brenor Brophey's gPoint PHP class to do this on a couple of occasions. Solid results, GPL code, and easily deployed. Recommended.