Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Is there any open source alternative for Zonal Statistics tool (ArcGIS Spatial Analyst)?
What is the best tool (which I can use in script) dor counting statistics of raster files?
You can do this with GRASS using various methods. Which one is most suitable will depend on your data and the required output. Note that you can use GRASS also from within QGIS using the GRASS toolbox or Sextante toolbox.
Let's assume you have:
a vector map, e.g., vector_zones with the zones defined in the
column myzones in the attribute table.
a raster layer 'values' for which you want to calculate your zonal statistics
r.statistics
To use r.statistics, you first need to convert the vector map to a raster layer, which you can do with v.to.rast. Next, use r.statistics to calculate the zonal statistics.
v.to.rast input=vector_zones output=zones column=myzones
r.statistics base=zones cover=values out=outputmap method=average
This will give you a new layer with the selected zonal statistic, which could be average, mode, median, variance, etc. (see the man page link above).
r.univar
The r.univar function also works on raster layers.
v.to.rast input=vector_zones output=zones column=myzones
r.univar map=values zones=zones output=output.file fs=;
The output is a table with the zonal statistics.
v.rast.stats
This does not require you to convert the vector layer to a raster layer (this is done internally). The function calculates basic univariate statistics per vector category (cat) from the raster map.
v.rast.stats vector=vector_zones layer=1 raster=values column_prefix=val
The results are uploaded to the vector map attribute table.
you can use the raster package in R
library(raster)
v <- raster('raster filename')
z <- raster('zones raster filename')
zv <- zonal(v, z, fun=mean)
Correct me if I'm wrong, RobertH, but I believe zonal() requires that the zones are already 'rasterized' in some sense, whereas many times one will want the statistics of raster cells that fall within polygons. The various overlay methods in R within the sp package (see: ?"overlay-methods" ) are necessary for this, though if I am wrong I would be glad to hear it. I quite prefer the raster package over using SpatialGridsDataFrames, but I think one must rely on sp classes to mix polygons and gridded data. Which is ok, except becomes problematic because it lacks the great memory management of the raster package, which making point-in-polygons style operations really hard to do in R on large rasters.
I am also led to believe, but have not tried, that this can be done within GRASS, and/or through QGIS, with the next release of QGIS (1.7) to have some sort of built-in zonal stats feature.
The Rasterstats package is a nice open source tool that worked well for me:
http://blog.perrygeo.net/2013/09/24/python-raster-stats/
I started using it as a work-around because arcpy's ZonalStatistics method was producing a problematic raster that lead to an odd error when trying to convert the raster to an array (https://gis.stackexchange.com/questions/110274/save-fails-on-raster-object-created-from-numpyarraytoraster). Rasterstats worked well and provided an efficient solution for my problem.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
For my task, I am given a series of frames extracted from a Tom and Jerry video. I need to detect the objects in a frame(in my case, the objects are tom and jerry) and their location. Since my dataset is different from the classes in ImageNet, I am stuck with no training data.
I have searched extensively and there seem to be some tools where I need to manually crop the location of images, is there any way to do this without such manual work ?
Any suggestions would be really helpful, thanks a lot !
is there any way to do this without such manual work ?
Welcome to the current state of machine learning, driven by data-hungry networks and much labor and work on creating datasets side :) Labels are here and will stay for a while, to tell your network (via loss function) what you want to do.
but but.. you are not in that bad situation at all, because you can go for the pre-trained net and just fine-tune into your lovely Jerry and Tom (acquiring training data will be 1-2h itself).
So what is this fine-tuning and how does it work? Let's say you are taking a pre-trained net trained on Imagenet and this net can perform reasonably well on classes defined in Imagenet. It will be your starting point. This network already has learned quite abstract features about all these objects from ImageNet, that's why the network is capable of transfer learning with a reasonably small amount of new class samples. Now when you add Tom and Jerry to the network output and fine-tune it on a small amount of data (20-100 samples) it will perform not that bad (I guess acc will be somewhere in 65-85%). So here is what I suggest:
google some pre-trained net easy to interact. I found this. See chapter 4. Transfer Learning with Your Own Image Dataset.
pick some labeling tool.
label 20-100 Toms, Jerries with bounding box. For a small dataset like this, divide it to ./train (80%) and ./test (20%). Try to catch different poses, different backgrounds, frames distinct from each other. Go for some augmentation.
Remove last network layer and add layer for 2 new outputs, Tom and Jerry.
train it (fine-tune it), check accuracy on your test set.
have fun! Train it again with more data.
"Is it possible to perform Object Detection without proper training
data?"
It's kinda is, but I can't imagine anything simpler than fine-tuning. We can talk here about:
A. non-machine learning approaches: which is computer vision + hand-crafting features + manually defining parametes and using it as a detector, but in your case, it is rather not the way you want to go; however some box sliding and manually color histogram thresholding may work for Tom and Jerry (this thresholding parameter naturally might be subject to train). This is quite often more work to do than proposed fine-tuning. Sometimes it is a way to label thousands of samples that way, then correct labels, then train more powerful detectors. There are numerous tasks that this approach is enough and the benefit can be lightweight and speed.
B. machine learning approaches which deal with no proper training data. Or maybe which deal with a small amount of data as humans do. This is mainly emerging filed, currently under active R&D and few of my favorites are:
fine-tuning pre-trained nets. Hey, we are using this because it is so simple!
one-shot approaches, like triplet-loss+deep-metrics
memory augmented neural networks used in one/few shot context
unsupervised, semi-supervised approaches
bio-plausible nets, including no-backprop approach with only last layer tuned via supervision
I have data that I want to create a GIS type application that will have the typical features of adding and removing layers of different types. What is the best architectural approach?
The data consists of property locations in Eastings and Northings.
I also have ordnance survey data in GML and Shapefiles.
I know this is a very broad question but the subject area also seems very broad to me and I am unsure which direction to go.
I was thinking of using SQL 2008 spatial and Bing Silverlight control to visualise that maps.To do this would I have to convert the eastings and northings to GWS84 geography datatype? But then if I converted the shapefiles to GML and imported all the GML files to sql using GeomFromGML they would be in geometry datatypes. Wouldn’t the two types be incompatible?
Also, should ESRI ArcGIS API for Silverlight
feature in the equation? Is this a good environment to create maps that I can point as SQL sqerver 2008 as the datasource (using a WCF service if necessary)?
Any advice greatly appreciated!
This is something I've done several times, using OS data from SQL Server in both Bing Maps AJAX and Silverlight controls. Some general comments below (in no particular order!):
Don't expect to implement full-blown
GIS functionality using Bing Maps.
Simple querying, retrieval and
display of the data is all fine (+
some simple editing), but after that
you'll be struggling with what can be
achieved in the browser.
All vector shapes supplied to Bing Maps need to be in (geography)
WGS85 coordinates, EPSG:4326.
However, all data will be projected
and displayed using (projected)
Spherical Mercator system, EPSG:3857.
In terms of vector shapes, you can expect to achieve a similar level of performance as you get in the SSMS spatial results tab - that is, (with careful architecture) you can plot up to about 5,000 features on the map at once, zoom / pan around them, click on them to bring up various properties and attributes etc. However, after that you'll find the UI becomes fairly unresponsive (which
I imagine is the reason why the spatial results tabs itself limits you to displaying 5,000 records at once).
If you want to display more features than this, one approach is to rasterize them by projecting them into the EPSG:3857 projection, creating a .PNG/.JPG image file of the features and then cutting that image into tiles according to the Bing Maps quadkey tile numbering system as explained here:
http://msdn.microsoft.com/en-us/library/bb259689.aspx and displaying them as a tilelayer. Tile layers are significantly faster than displaying the equivalent vector shapes, although it means the data is static.
If you do create raster tiles, you can either render them dynamically
or pre-render them to improve performance - i.e. you could set
up a job to render and update the tileset for slowly-changing data
every night/every month etc.
If you're talking about OS Mastermap data, the sheer level of detail
involved means that you need to think more carefully about what
features you want to display, and how you want to display them. Take
greater London, for example, which covers an area about 50km x 40km. To
create raster tiles (each of which are 256px x 256px) at zoom level 19
covering this area you'd need to render and store 1.3 million separate
tiles. If each of those is generated from a database query that takes, say
200ms to run, it's gonna take a looooonggggg time to prepare all your
data. Also, once the files have been generated, you might want to think
about storing them in a DB rather than saving them on a file system.
As for loading the OS data into SQL Server in the first place - there
are several tools that will import either from GML or shapefile into SQL
Server, and handle the projection from EPSG:27700 (Ordnance Survey
National Grid) to WGS84 along the way. Try GDAL/OGR or Safe FME for
starters.
I have a blog at http://alastaira.wordpress.com that has several blog posts that you may find useful describing various aspects of integrating Bing Maps and SQL Server. Particularly, you might want to look at:
http://alastaira.wordpress.com/2011/02/16/loading-ordnance-survey-open-data-into-sql-server-2008/
http://alastaira.wordpress.com/2011/01/23/the-google-maps-bing-maps-spherical-mercator-projection/
http://alastaira.wordpress.com/2011/02/21/using-ogr2ogr-to-convert-reproject-and-load-spatial-data-to-sql-server/
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I have a project in a course at the university that requires various Matlab functions. My version of Matlab is supplied by my workplace, and it doesn't have some of the toolboxes that are required for the project. Is there some repository of such implementations anywhere? I couldn't find anything relevant when I googled.
While my question is general, I'm listing the functions that I need and can't find, since my question is also specific:
knnclassify - an implementation of K-nearest neighbors
svmclassify - an implementation of Support Vector Machines
svmtrain - part of the SVM implementation
mapstd - normalizes a matrix to have a mean of 0 and standard deviation of 1
An alternative I'm considering is working in Python with Numpy and Pylab. Are there toolboxes in Pylab that are equivalent to these Matlab functions?
The first place I would check is the MathWorks File Exchange. There are over 10,000 code submissions from MATLAB users, and you may be able to find alternatives to the various MATLAB toolboxes. Here's something that may be helpful:
Luigi Giaccari has a few highly-rated submissions related to KNN searching here, here, and here.
Another alternative for a simpler function like MAPSTD is to try and implement a stripped-down version of it yourself. Here's some sample code that replicates the basic behavior of MAPSTD:
M = magic(5); %# Create an example matrix
rowMeans = mean(M,2); %# Compute the row means
rowStds = std(M,0,2); %# Compute the row standard deviations
rowStds(rowStds == 0) = 1; %# Set standard deviations of 0 to 1
for i = 1:size(M,1)
M(i,:) = (M(i,:)-rowMeans(i))./rowStds(i); %# Normalize each row of M
end
%# Or you could avoid the for loop with a vectorized solution...
M = bsxfun(#rdivide,bsxfun(#minus,M,rowMeans),rowStds);
This obviously won't cover all of the options in MAPSTD, but captures the basic functionality. I can confirm that the above code gives the same result as mapstd(M).
You might want to consider getting your own copies of Matlab and the toolboxes you need. Mathworks has VERY attractive pricing for University students.
GNU Octave is the free Matlab more-or-less-work-alike. I don't know how well the toolboxes are covered.
Alternatively, if the assignment needs them, the school probably has them on some lab machines somewhere, and you MIGHT be able to login remotely, using an Xterm of some kind. Ask the TA.
You could also look at R which is very strong in many data-driven fields, including Machine Learning.
Also of note is the MLOSS directory of open-source machine learning software.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I've been provided with a table of data which includes columns for latitude and longitude. The data is US only. What I've like to do is to feed this data to Google Maps or a similar tool like Live Maps and have the data points plotted.
Does anyone have a code sample or know of a library that makes this task simple? I can read the values if something else can plot them (or generate the JavaScript to do it). I'm familar with C#, PHP, Python etc so language is not a priority. My preference is something that the is simple and robust.
I've dealt with this problem before many times. While KML is one option, if you have a CSV with the latitude and longitude already in it, my quick and dirty method is to use Google Fusion Tables: http://www.google.com/fusiontables/Home
Even if you have just names of cities and towns (anywhere in the world!) it does an okay job.
If you only care about US locations, and the G20 (rich, well geocoded nations), then batch GEO is probably the best fast method, but limits you to 2500 points on the free version: http://batchgeo.com/
But for my needs, BatchGeo's method falls short. When you drill down in Kenya you see that it guesses at many locations wrong. Those 666 locations in Nairobi are slums with names that appear in no official database. Since Kenya isn't one of the top 20 tech-friendly countries, this is a common source of error. Most interesting locations appear in slums. (See also www.mapKibera.org for the effort involved in fixing blank spots in google maps)
I have a more extensive blog about this with lots of images of what your output will look like: chewychunks.wordpress.com/2011/06/09/how-to-geomap-story-locations-across-east-africa/
(but newbies are not allowed to post images here directly :( )
The best solution for me required downloading 44,000 locations from Geonames.org list for Kenya and Uganda, adding a custom slum-locations geo-lookup-list, and a python multi-step matching algorythm based on DIFFLIB and REGEX.
Answer: use this website http://www.darrinward.com/lat-long/?id=257165 and it will plot them for you.
Unrelated, screenshot is locations of #thefappening leaks. I wish leakers would scrub EXIF before publishing this crap online, this is really not right.
You can plot in Google Earth by creating a special XML document known as a KVM document. See this tutorial for details.
I plotted all of our website visitor's coordinates using this plus a GeoIP service. Really fun stuff.
CSV is widely used in Google Maps mashups. You can find the code including the CSV parser here.
This link has information about using KML and the Google Earth plugin API to display geo-referenced data in csv files.
http://www.cs.umb.edu/~joecohen/#kml
I found this page while looking for some csv files to use while teaching a class about this. If you have some good play data I'd like to see it.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
What's a good library for geospatial functions? I mean things like
distance between two points on the globe
coordinates of a circle of a given radius from a particular point
etc.
Bonus if there's an interface to the various ways different databases
represent geolocations.
I'm a geo-noob (in case this question didn't make it obvious), so pointers to other geolocation/geospatial resources are welcome.
C++ and Python preferred, but all pointers welcome.
I've enjoyed using geopy. It's a simple library that finds great-circle distance in a number of projections. Geopy also provides a single interface to multiple geocoders like Google Maps and Microsoft Earth to give you coordinates for a street address.
You might be interested in the Topic :: Scientific/Engineering :: GIS section in PyPi.
Some options for functions from a useful article on the O'Reilly website. There are other options in the article.
GEOS open source C++ geometry /
topology engine. Might suit you?
PostGIS - a PostgreSQL database
that can also store geometric
(spatial) data types. This provides
GIS-like abilities within an SQL
database environment, so you could do manipulations through SQL.
I'm not sure about interfaces to different databases but the article mentions a number of libraries that convert geospatial data between different formats. You might also be interested in the OGC standards. If all your target databases support WFS you should be able to access them with exactly the same code. EDIT: Increasing numbers of GIS packages support WFS but I don't think pure databases do.
EDIT: you could also check out OSGeo which is a collaboration to support open source geospatial stuff. FDO might do the interfaces to different databases.
If you were using ruby with or without rails, I'd recommend the GeoKit gem: http://geokit.rubyforge.org/
Check GDAL/OGR which is two libraries in one - GDAL for raster data and OGR for vector data. The OGR part provides API to manipulate geometries - it uses GEOS library.
I was looking at something similar, there's the Geolocator Library that I found, it provides some distance calculation functions.