I have an asset of 25k geometries. I also have a function that maps over a list of years to find mean rainfall for a region. I was able to do this using a for loop when I had fewer regions:
def yearlyRainfall(year):
startDate = ee.Date.fromYMD(year, 1, 1)
endDate = startDate.advance(1, 'year')
filtered = chirps.filter(ee.Filter.date(startDate, endDate))
total = filtered.reduce(ee.Reducer.sum())
stats = total.reduceRegion(
reducer = ee.Reducer.mean(),
geometry = transect_region,
scale = 5566,
)
f = ee.Feature(None,{
'year': year,
'precipitation': stats.get('precipitation_sum')
})
return f
years = ee.List.sequence(1981, 2019)
for area in range(len(transect_areas)):
transect_region = transect_areas[area].geometry()
rainfallYears = ee.FeatureCollection(years.map(yearlyRainfall))
rainfalldf = geemap.ee_to_pandas(rainfallYears)
print(rainfalldf)
print(rainfalldf["precipitation"].mean())
Where transect_areas is a dictionary of regions. However, I now have to do this over 25k geometries held in a csv asset, and cannot achieve this through a for loop. I know I have to map it somehow, but I can't figure out a workaround. Any tips would be appreciated.
Tried: looping over dictionary of regions, but computation does not complete
Related
I've already trained several models for a binary classification problem, basing my election on F-Score and AUC. The code used has been the following:
svm = StandardScaler()
svm.fit(feat_train)
feat_train_std = svm.transform(feat_train)
feat_test_std = svm.transform(feat_test)
model_10= BalancedBaggingClassifier(base_estimator=SVC(C=1.0, random_state=1, kernel='linear'),
sampling_strategy='auto',
replacement=False,
random_state=0)
model_10.fit(feat_train_std, target_train)
pred_target_10 = model_10.predict(feat_test)
mostrar_resultados(target_test, pred_target_10)
pred_target_10 = model_10.predict_proba(feat_test)[:, 1]
average_precision_10 = average_precision_score(target_test, pred_target_10)
precision_10, recall_10, thresholds = precision_recall_curve(target_test, pred_target_10)
auc_precision_recall_10 = auc(recall_10, precision_10)
disp_10 = plot_precision_recall_curve(model_10, feat_test, target_test)
disp_10.ax_.set_title('Binary class Precision-Recall curve: '
'AUC={0:0.2f}'.format(auc_precision_recall_10))
Afterwards, I load the model as follows:
modelo_pickle = 'modelo_pickle.pkl'
joblib.dump(model_10,modelo_pickle)
loaded_model = joblib.load(modelo_pickle)
Then, the aim is to load a new dataset, which columns are the same as the model's variables, and make a prediction for each line:
lista_x=x.to_numpy().tolist()
resultados=[]
for i in lista_x:
pred = loaded_model.predict([i])
resultados.append(pred)
print(resultados)
However, every single result is equal to 1, which does not make any sense. Would anyone tell me what am I missing, please?
Thank you in advance.
Regards,
Previously described.
I have a FeatureCollection made up of many (100-200) polygons ('ftr_polygons'). I also have an ImageCollection made up of monthly median Landsat8 bands and indices ('byMonth'). I want to ReduceRegions and save a median (or mean) spatial average from each polygon in the FeatureCollection. End goal is to export to csv a timeseries of monthly mean bands/indices within each polygons over multiple years (2013-2019).
With the code below, I am able to do this for ~1 year, but any more than that, and I get an error: 'FeatureCollection (Error) Computation timed out’. Is there a better way to do this?
// define the function that will grab median (or mean) spatial reductions for each polygon, for each month
var extractdata = function(medianImage,ftr_polygons) {
var date_start = ee.Date(medianImage.get('system:time_start')).format("YYYY-MM"); // get date as string to append to each property
// spatial MEDIAN
ftr_polygons = medianImage.reduceRegions({ // create feature collection with new properties, bands for each month, uniquely named
collection: ftr_polygons,
reducer: ee.Reducer.median(),
scale: 30,
tileScale: 1}); // tile scale
var ftr_polygons_propnames = ftr_polygons.first().propertyNames(); // get property names first
var ftr_polygons_newnames = ftr_polygons_propnames.replace('NDVI_median',
ee.String('NDVI_median_').cat(date_start)); //replace property names with band+date
ftr_polygons_newnames = ftr_polygons_newnames.replace('EVI_median',
ee.String('EVI_median_').cat(date_start)); //replace property names with band+date
ftr_polygons_newnames = ftr_polygons_newnames.replace('NIRv_median',
ee.String('NIRv_median_').cat(date_start)) ; //replace property names with band+date
ftr_polygons = ftr_polygons.map(function(f) {return f.select(ftr_polygons_propnames,ftr_polygons_newnames)});
return ftr_polygons;
};
// apply the function over ImageCollection byMonth, beginning with feature collection ftr_polygons
var ftr_polygons = ee.FeatureCollection(byMonth.iterate(extractdata,ftr_polygons));
// remove geometry on each feature before printing or exporting
var myproperties=function(feature){
feature=ee.Feature(feature).setGeometry(null);
return feature;
};
var ftr_polygon_export = ftr_polygon.map(myproperties)
print(ftr_polygon_export.limit(1), 'For export w monthly properties');
Maybe this answer: https://stackoverflow.com/a/48412324/12393507 alludes to a better way:
The same approach can be used with reduceRegions() as well, mapping over images and then over regions. However, you will have to map over the resulting features to set dates.
I would appreciate more info on this approach.
Thanks.
For computationally intensive operations that will run for a long time you should always export your results instead of visualizing/printing them.
For more info read through this section of the debugging page in the Earth Engine manual.
I have a JSON file of a long route. The file contains the lat and long of of this route.
I'm trying to mark different sections of this route based on a set of criteria (which I've compiled in a dataframe). However, I'm facing to problems:
1) How do I break up this long set of lat and longs into segments? (can't do this manually because I have many route variations)
2) How do I assign a variable color to each segment?
I intend to use leaflet map (for its interactivity), but I'm open to better suggestions.
When working with spatial data, it helps to know spatial classes! I am assuming you know hoe to read your JSON file as a data frame into R.
Here's a reproducible example:
library(mapview)
library(sp)
### create some random data with coordinates from (data("breweries91", package = "mapview"))
set.seed(123)
dat <- data.frame(val = as.integer(rnorm(32, 10, 2)),
lon = coordinates(breweries91)[, 1],
lat = coordinates(breweries91)[, 2])
### state condition for creation of lines
cond <- c(8, 9, 10)
### loop through conditions and create a SpatialLines object for each condition
lns <- lapply(seq(cond), function(i) {
ind <- dat$val == cond[i]
sub_dat <- dat[ind, ]
coords <- cbind(sub_dat$lon, sub_dat$lat)
ln <- coords2Lines(coords, ID = as.character(cond[i]))
proj4string(ln) <- "+init=epsg:4326"
return(ln)
})
### view lines with mapview
mapview(lns[[1]], col = "darkred") +
mapview(lns[[2]], col = "forestgreen") +
mapview(lns[[3]], col = "cornflowerblue")
Essentially, what we are doing here is create a valid sp::SpatialLines object for each condition we specify. The we plot those using mapview given you mentioned interactivity. Plotting of spatial objects can be achieved in many ways (base, lattice, ggplot2, leaflet, ...) so there's many options to choose. Have a look at sp Gallery for a nice tutorial.
Note: This answer is only valid for non-projected geographic coordinates (i.e. latitude/longitude)!
I working with Database first C# MVC, EF6, LINQ and JSon to try and pass data to both Highcharts and Google Maps for some of my reporting.
If I could add an image I would show you the relevant portion of my model, but sadly I need more reputation to do that...
The portion of the Entity Model I'm concentrating on right now is based on a central Docket that contains a BuildingCode as part of a one-to-many relationship to a building with and address and further relationship to the buildings polygons (for mapping). Dockets are also classified by one or more DocketTypes and thus there is a many-to-many relationship between Dockets and DocketTypes, which is not directly exposed to through the EF.
As an example a Docket which represents an investigation, could be related to the theft of a mobile phone in building A located on Campus X, not only was the cellphone stolen but the assailant also assaulted the victim in order to steal the mobile phone. So there are 2 DocketTypes here 1. Theft of mobile phone and 2. assault. Note: this is fictitious and for illustration purposes only .
One of my fundamental reports requires that I count how many docketTypes affect each building and each campus in a given period. When I display this I also need to show what the DocketTypes are.
I have no end of nightmare trying to find a way to get this right, I keep running into circular reference errors and needing to use explicit conversions when trying to model the data with LINQ so that I can pass a single nested object through JSON to the client side where displaying will occur.
In the below code I am told I need an Explicit conversion:
Cannot implicitly convert type 'Campus_Investigator.ViewModels.DocketTypeViewModel' to 'System.Collections.Generic.IEnumerable<Campus_Investigator.ViewModels.DocketTypeViewModel>'. An explicit conversion exists (are you missing a cast?)
var currentDocketQuery = from d in db.Dockets
from dt in d.DocketTypes
from bp in d.BuildingDetail.BuildingPolygons
where d.OccurrenceStartDate >= datetime && d.BuildingDetail.CampusName == Campus
select new CampusBuildingDocketTypeViewModel()
{
BuildingCode = d.BuildingDetail.BuildingCode,
BuildingName = d.BuildingDetail.BuildingName,
//BuildingPolygons = d.BuildingDetail.BuildingPolygons,
DocketTypes = new DocketTypeViewModel()
{
Category = dt.Category,
SubCategory = dt.SubCategory,
ShortDescription = dt.ShortDescription
}
};
I appreciate any ideas on how I can explicitly convert this or is that a better method I can use and avoid the circular reference error?
You included some redundant part in your query (which performs some inner join). The from bp in d.BuildingDetail.BuildingPolygons is joined in but then is not shown in the result. So it totally does not make sense. There may be duplicated elements in the result due to that. The from dt in d.DocketTypes is wrong joined in, although you need it in the result but because the DocketTypes is output per d in db.Dockets, so it's just simply queried like this:
var currentDocketQuery = from d in db.Dockets
where d.OccurrenceStartDate >= datetime && d.BuildingDetail.CampusName == Campus
select new CampusBuildingDocketTypeViewModel()
{
BuildingCode = d.BuildingDetail.BuildingCode,
BuildingName = d.BuildingDetail.BuildingName,
//BuildingPolygons = d.BuildingDetail.BuildingPolygons,
DocketTypes = d.DocketTypes
};
In fact I can see the commented line //BuildingPolygons = d.BuildingDetail.BuildingPolygons, so if you want to include that, it should also work.
If the DocketTypes has different type of d.DocketTypes, then you need a simple projection like this:
var currentDocketQuery = from d in db.Dockets
where d.OccurrenceStartDate >= datetime && d.BuildingDetail.CampusName == Campus
select new CampusBuildingDocketTypeViewModel()
{
BuildingCode = d.BuildingDetail.BuildingCode,
BuildingName = d.BuildingDetail.BuildingName,
//BuildingPolygons = d.BuildingDetail.BuildingPolygons,
DocketTypes = d.DocketTypes.Select(e => new DocketTypeViewModel()
{
Category = e.Category,
SubCategory = e.SubCategory,
ShortDescription = e.ShortDescription
})
};
I managed to solve this one by using the below. The major hassle with this is the circular referencing that exists in the model. When JSON serializes these, everything falls apart so it takes a lot of transforming to make sure that I only extract what I need. In this case grouped campus and building data (below includes the polygons which where only half commented out in the above) and then the include the detail of the DocketTypes that occurred at each building.
var datetime = DateTime.Now.AddDays(-30);
var campusDocket = from d in db.Dockets
where d.OccurrenceStartDate >= datetime && d.BuildingDetail.CampusName == Campus
group d by new { d.BuildingDetail.CampusName, d.BuildingDetail.BuildingCode, d.BuildingDetail.BuildingName } into groupdata
select new CampusBuildingDocketTypeViewModel
{
BuildingCode = groupdata.Key.BuildingCode,
BuildingName = groupdata.Key.BuildingName,
CampusName = groupdata.Key.CampusName,
Count = groupdata.Count(),
BuildingPolygons = from bp in db.BuildingPolygons
where bp.BuildingCode == groupdata.Key.BuildingCode
select new BuildingPolygonViewModel
{
Accuracy = bp.Accuracy,
BuildingCode = bp.BuildingCode,
PolygonOrder = bp.PolygonOrder,
Latitude = bp.Latitude,
Longitude = bp.Longitude
},
DocketTypes = from doc in db.Dockets
from dt in doc.DocketTypes
where doc.OccurrenceStartDate >= datetime && doc.BuildingCode == groupdata.Key.BuildingCode
select new DocketTypeViewModel
{
Category = dt.Category,
SubCategory = dt.SubCategory,
ShortDescription = dt.ShortDescription
}
};
The Answer again is ViewModels. I'm finding ViewModels seem to solve a lot of problems...
Is there a straightforward way to load a .csv file into Simplegeo Storage? I don't have great coding skills and I'm trying to get things set up so I can ask a freelancer to create some maps for my app. If someone has existing code to do this I can probably figure out how to make it work for my situation.
I just skimmed over the api. Here's a basic example in python
Assumed csv format:
layer, id, lat, lon
python
from simplegeo.models import Record, Client
lines = open('file.csv').split('\n')
client = Client('your-oauth-token', 'your-oauth-secret')
for line in lines:
parts = line.split(',')
if len(parts) == 4:
layer = parts[0].strip()
id = parts[1].strip()
lat = float(parts[2].strip())
lon = float(parts[3].strip())
r = Record(layer, id, lat, lon)
client.storage.add_record(r)
After a bit more digging, I found a python example on their site for this exact purpose
https://simplegeo.com/docs/tutorials/general-hackery#how-import-csv-file-simplegeo
import csv
import simplegeo
OAUTH_TOKEN = '[insert_oauth_token_here]'
OAUTH_SECRET = '[insert_oauth_secret_here]'
CSV_FILE = '[insert_csv_file_here]'
LAYER = '[insert_layer_name_here]'
client = simplegeo.Client(OAUTH_TOKEN, OAUTH_SECRET)
def insert(data):
layer = LAYER
id=data.pop("id")
lat=data.pop("latitude")
lon=data.pop("longitude")
# Grab more columns if you wish
record = simplegeo.Record(layer,id,lat,lon,**data)
client.add_record(record)
r = csv.DictReader(open(CSV_FILE, mode='U'))
for l in r:
insert(l)