I have a FeatureCollection made up of many (100-200) polygons ('ftr_polygons'). I also have an ImageCollection made up of monthly median Landsat8 bands and indices ('byMonth'). I want to ReduceRegions and save a median (or mean) spatial average from each polygon in the FeatureCollection. End goal is to export to csv a timeseries of monthly mean bands/indices within each polygons over multiple years (2013-2019).
With the code below, I am able to do this for ~1 year, but any more than that, and I get an error: 'FeatureCollection (Error) Computation timed out’. Is there a better way to do this?
// define the function that will grab median (or mean) spatial reductions for each polygon, for each month
var extractdata = function(medianImage,ftr_polygons) {
var date_start = ee.Date(medianImage.get('system:time_start')).format("YYYY-MM"); // get date as string to append to each property
// spatial MEDIAN
ftr_polygons = medianImage.reduceRegions({ // create feature collection with new properties, bands for each month, uniquely named
collection: ftr_polygons,
reducer: ee.Reducer.median(),
scale: 30,
tileScale: 1}); // tile scale
var ftr_polygons_propnames = ftr_polygons.first().propertyNames(); // get property names first
var ftr_polygons_newnames = ftr_polygons_propnames.replace('NDVI_median',
ee.String('NDVI_median_').cat(date_start)); //replace property names with band+date
ftr_polygons_newnames = ftr_polygons_newnames.replace('EVI_median',
ee.String('EVI_median_').cat(date_start)); //replace property names with band+date
ftr_polygons_newnames = ftr_polygons_newnames.replace('NIRv_median',
ee.String('NIRv_median_').cat(date_start)) ; //replace property names with band+date
ftr_polygons = ftr_polygons.map(function(f) {return f.select(ftr_polygons_propnames,ftr_polygons_newnames)});
return ftr_polygons;
};
// apply the function over ImageCollection byMonth, beginning with feature collection ftr_polygons
var ftr_polygons = ee.FeatureCollection(byMonth.iterate(extractdata,ftr_polygons));
// remove geometry on each feature before printing or exporting
var myproperties=function(feature){
feature=ee.Feature(feature).setGeometry(null);
return feature;
};
var ftr_polygon_export = ftr_polygon.map(myproperties)
print(ftr_polygon_export.limit(1), 'For export w monthly properties');
Maybe this answer: https://stackoverflow.com/a/48412324/12393507 alludes to a better way:
The same approach can be used with reduceRegions() as well, mapping over images and then over regions. However, you will have to map over the resulting features to set dates.
I would appreciate more info on this approach.
Thanks.
For computationally intensive operations that will run for a long time you should always export your results instead of visualizing/printing them.
For more info read through this section of the debugging page in the Earth Engine manual.
Related
I carried out join operation to help in smoothing out my images in a Landsat ImageCollection by getting at least 3 Images in a specified time window, obtain the median and then adding those images to the ImageCollection. I expected to obtain an ImageCollection that I could still carry out some filter functions to create a time series with the median images, but it didn't work.
// 1. selecting the time window
var days = 32;
var milli = ee.Number(days).multiply(1000*24*60*60)
console.log(milli)
var join = ee.Join.saveAll({
matchesKey:"images"
})
// 2. Apply the filter
var dif_filter = ee.Filter.maxDifference({
difference:milli,
leftField:"system:time_start",
rightField:"system:time_start"
})
// 3. Apply the join
var joined_collection = join.apply({
primary: original_collection,
secondary:original_collection,
condition:dif_filter
})
// Adding the median images to our collection
function medianCalculator(image){
var imageGetter = ee.ImageCollection.fromImages(image.get("images"))
var medianImage = imageGetter.reduce(ee.Reducer.median())
return ee.Image(image).addBands(medianImage).select("NDVI_median", "EVI_median", "NDVI", "EVI")
}
// collection with smoothed Images
var collection_2 = joined_collection.map(medianCalculator)
print(collection_2)
I wanted to carry out some filters on "collection_2" but there was an error, "Line 110: collection_2.filter(...).median is not a function". What am I missing.
I suspect that after performing the Join, the ImageCollection got turned into a FeatureCollection so I tried looking for ways of converting a FeatureCollection to an ImageCollection...to no avail.
I am working on a small MeteorJS app that plots points on a map based on popular areas for work.
I have this:
Template.list.jobs = function() {
if(Session.get('currentIndustryOnet')) {
jobs = Jobs.find({onet: Session.get('currentIndustryOnet')}).fetch();
// Session.set('jobCount', jobs.count());
var cnt = _.groupBy(jobs, 'address');
console.log(cnt);
return Pagination.collection(jobs);
} else {
jobs = Jobs.find()
Session.set('jobCount', jobs.count());
return Pagination.collection(jobs.fetch());
}
}
The cnt variable returns a properly grouped array (the key of the array is an address like Allentown, PA). I have a collection of Cities which have ever city in the USA along with their LAT/LONGs to plot on a Google Map. So I will take the top 100 from the grouped array, find the lat/long in the Cities collection and plot those points on a map.
I am not familiar with working with a groupedBy method to sort the list based on the length and then pull out the key to use as my search.
I'm not 100% certain about the data structure... but assuming jobs have an address field, and you want them sorted by frequency of occurrence and capped, you could do something like this:
var addresses = _.chain(jobs)
.countBy('address')
.pairs()
.sortBy(function(j) {return -j[1];})
.map(function(j) {return j[0];})
.first(100)
.value();
Note there may be a more clever way to use underscore to arrive at this result. Once you have the capped, sorted list of addresses, you can probably get the lat/long values via a find like:
Cities.find({address: {$in: addresses}}).fetch();
I'm trying to do some re-factoring on my charts to make them re-usable using this as a guide: http://bost.ocks.org/mike/chart/
I'm having problems drawing the lines in my multi-line graph though - specifically passing the data to the x and y values. If I hard code the element names it works, but if I try to use the xValue and yValue objects this does not work. I'm assuming that this is because I'm trying to call a function within the parameter of an other object, but I'm not sure how to get around this. In the exmaple Mike uses d[0] and d[1], but this won't work with JSON data (or I'm not sure how to make it work).
I've posted this JSFiddle so you can see the code. The problem lines are 125 to 131 which in turn is being called from line 165.
var main_line = d3.svg.line()
.interpolate("cardinal")
// Hard coding the elements works
//.x(function(d) { return main_x(d.date); })
//.y(function(d) { return main_y(d.buildFixTime); });
// Passing xValue and yValue does not work
.x(function(d) { return main_x(xValue); })
.y(function(d) { return main_y(yValue); });
http://jsfiddle.net/goodspeedj/fDyLY/
Thank you in advance.
You need to redefine your accessor method within .x() and .y(). The accessor method defines the way that a datum is pulled out of the data that is bound to the selection that you call the line generator on.
Suppose you have a relatively flat data structure such as the following.
data = [{x : 1, y : 2}, {x:1, y:3}, {x:4, y:5}];
You then bind the data to a selection with the following statement
d3.select("body").datum(data).append("path").attr("d",lineGenerator);
Quite a bit is going on underneath this statement. I'll give you a bit more of a walkthrough after showing you a commonly used example.
The important aspect to understand is that similarly to other calls in d3 such as
var exampleRectangles = d3.select("body")
.data(data).enter()
.append("rect")
.attr("width",2)
.attr("height", 3)
.attr("x",function(datum){return datum.x}) // pay attention to this line
.attr("y",0);
d3 is implicitly iterating over each element in your data. For each datum in your data array, in this case there is a total of three datum, you are going to add a rectangle to the dom.
In the line that I tell you to pay attention to you notice that you're defining an anonymous (unnamed) function. What is that datum parameter coming from? It's implicitly being passed to your anonymous function.
So each rectangle has it's own corresponding datum {x : 1, y : 2}, {x:1, y:3}, {x:4, y:5} respectively. Each rectangle's x coordinate is defined by the respective datum.x attribute. Under the sheets, d3 is implicitly looping over the data array that you've defined. A similar approach to the example d3 code could be written as above.
for (var i = 0; i < data.length; i++)
{
d3.select("body").append("rect")
.attr("width",2)
.attr("height", 3)
.attr("x",data[i].x)
.attr("y",0);
}
This follows from the notion of data driven documents (d3). For each item added (a rectangle in the above example a piece of data is tied to it. In the above example you see that there is something kind of similar to your .x() and .y() accessor functions :
.attr("x",function(datum){return datum.x})
This function is telling d3 how to filter over the total datum that's being passed to the .attr() accessor method.
So, you need to determine which data you need to get a hold of to make your .attr("d", lineGenerator)call make sense. The difference between your.datum(data)call and the typical.data(data)call is that instead of parceling the data that's being passed to.data(data)`, the whole array is given as a single piece of data to the line generator function (similar to main_line(data), wherein it will again implicitly loop over the points to construct your path.
So, what you need to do is determine what a single datum will be defined as for your function to operate on.
I'm not going to define that as I don't seem to know quite which information you are operating on, but I would hazard a guess at something like.
.x(xAccessor)
.y(yAccessor)
function xAccessor(datum)
{
return xScale(datum._id.month);
}
function yAccessor(datum)
{
return yScale(datum.buildFixTime);
}
The way you have it set up, xValue and yValue are functions; you have to actually execute them on something to get a value back.
.x(function(d) { return main_x( xValue(d) ); })
.y(function(d) { return main_y( yValue(d) ); });
If you weren't using a scale, you could use
.x(xValue)
.y(yValue);
but only because if you pass in a function d3 executes it for you with the data as a parameter. And that only works for d3 methods that expect functions as possible input -- the scale functions expect data values as input.
I wrote a long piece work for another user last week that you may find useful, explaining methods that accept functions as parameters.
I guess a step back is in order. My original question is at the bottom of this post for reference.
I am writing a word guessing game and wanted a way to:
1. Given a word length of 2 - 10 characters, randomly generate a valid english word to guess
2.given a 2 - 10 character guess, ensure that it is a valid english word.
I created a vector of 9 objects, one for each word length and dynamically created 172000
property/ value pairs using the words from a word list to name the properties and setting their value to true. The inner loop is:
for (i = 0; i < _WordCount[wordLength] - 2; i)
{
_WordsList[wordLength]["" + _WordsVector[wordLength][i++]] = true;
}
To validate a word , the following lookup returns true if valid:
function Validate(key:String):Boolean
{
return _WordsList[key.length - 2][key]
}
I transferred them from a vector to objects to take advantage of the hash take lookup of the properties. Haven't looked at how much memory this all takes but it's been a useful learning exercise.
I just wasn't sure how best to randomly choose a property from one of the objects. I was thinking of validating whatever method I chose by generating 1000 000 words and analyzing the statistics of the distribution.
So I suppose my question should really first be am I better off using some other approach such as keeping the lists in vectors and doing a search each time ?
Original question
Newbie first question:
I read a thread that said that traversal order in a for.. in is determined by a hash table and appears random.
I'm looking for a good way to randomly select a property in an object. Would the first element in a for .. in traversing the properties, or perhaps the random nth element in the iteration be truly random. I'd like to ensure that there is approximately an equal probability of accessing a given property. The Objects have between approximately 100 and 20000 properties. Other approaches ?
thanks.
Looking at the scenario you described in your edited question, I'd suggest using a Vector.<String> and your map object.
You can store all your keys in the vector and map them in the object, then you can select a random numeric key in the vector and use the result as a key in the map object.
To make it clear, take a look at this simple example:
var keys:Vector.<String> = new Vector.<String>();
var map:Object = { };
function add(key:String, value:*):void
{
keys.push(key);
map[key] = value;
}
function getRandom():*
{
var randomKey = keys[int(Math.random() * keys.length)];
return map[randomKey];
}
And you can use it like this:
add("a", "x");
add("b", "y");
add("c", "z");
var radomValue:* = getRandom();
Using Object instead of String
Instead of storing the strings you can store objects that have the string inside of them,
something like:
public class Word
{
public var value:String;
public var length:int;
public function Word(value:String)
{
this.value = value;
this.length = value.length;
}
}
Use this object as value instead of the string, but you need to change your map object to be a Dictionary:
var map:Dictionary = new Dictionary();
function add(key:Word, value:*):void
{
keys.push(key);
map[key] = value;
}
This way you won't duplicate every word (but will have a little class overhead).
Introduction:
I have an AdvancedDataGrid displaying hierarchical data illustrated by the image below:
The branch nodes "Prosjekt" and "Tiltak" display the sum of the leaf nodes below.
Problem: I want the root node "Tavle" to display the total sum of the branch nodes below. When i attempted to do this by adding the same SummaryRow the sum of the root node was not calculcated correctly(Every node's sum was calculated twice).
dg_Teknikktavles = new AutoSizingAdvancedDataGrid();
dg_Teknikktavles.sortExpertMode="true";
dg_Teknikktavles.headerHeight = 50;
dg_Teknikktavles.variableRowHeight = true;
dg_Teknikktavles.addEventListener(ListEvent.ITEM_CLICK,dg_TeknikktavlesItemClicked);
dg_Teknikktavles.editable="false";
dg_Teknikktavles.percentWidth=100;
dg_Teknikktavles.minColumnWidth =0.8;
dg_Teknikktavles.height = 1000;
var sumFieldArray:Array = new Array(context.brukerList.length);
for(var i:int = 0; i < context.brukerList.length; i++)
{
var sumField:SummaryField2 = new SummaryField2();
sumField.dataField = Ressurstavle.ressursKey + i;
sumField.summaryOperation = "SUM";
sumFieldArray[i] = sumField;
}
var summaryRow:SummaryRow = new SummaryRow();
summaryRow.summaryPlacement = "group";
summaryRow.fields = sumFieldArray;
var summaryRow2:SummaryRow = new SummaryRow();
summaryRow2.summaryPlacement = "group";
summaryRow2.fields = sumFieldArray;
var groupField1:GroupingField = new GroupingField();
groupField1.name = "tavle";
//groupField1.summaries = [summaryRow2];
var groupField2:GroupingField = new GroupingField();
groupField2.name = "kategori";
groupField2.summaries = [summaryRow];
var group:Grouping = new Grouping();
group.fields = [groupField1, groupField2];
var groupCol:GroupingCollection2 = new GroupingCollection2();
groupCol.source = ressursTavle;
groupCol.grouping = group;
groupCol.refresh();
Main Question: How do i get my AdvancedDataGrid's (dg_Teknikktavles) root node "Tavle" to correctly display the sum of the two branch nodes below?
Side Question: How do i add a red color to the numbers of the root node's summary row that exceed 5? E.g the column displaying 8 will exceed 5 in the root node's summary row, and should therefore be marked red
Thanks in advance!
This is a general answer, without code examples, but I had to do the same just couple of days ago, so my memory is still fresh :) Here's what I did:
Created a class A to represent an item renderer data, extended it from Proxy (I had field names defined at run time), and let it contain a collection of values as it's data member. Once accessed through flash_proxy::getPropery(fieldName) it would find a corresponding value in the data member containing the values and return it. Special note: implement IUID, just do it, it'll save you couple of days of frustration.
Extended A in B, added a children property containing ArrayCollection of A (don't try to experiment with other collection types, unless you want to find yourself examining tons of framework code, trust me, it's ugly and is impossible to identify as interesting). Let B override flash_proxy::getPropery - depending of your compiler this may, or may not be possible, if not possible - call some function from A.flash_proxy::getPropery() that you can override in B. Let this function query every instance of A, which is a child of B, asking the same thing, as DataGrid itself would, when building item renderers - this way you would get the total.
When creating a data provider. Create an ArrayCollection of B (again, don't try to experiment with other collections--unless you are ready for lots of frustration). Create Hierarchical data that uses this array collection as a source.
Colors - that's what you use item renderers for, just look up any tutorial on using item renderers, that must be pretty basic.
In case someone else has the same problem:
The initial problem that everything was summed twice, was the result of using the same Array of SummaryField2 (sumFieldArray in the code) for both grouping fields(GropingField2 tavle and kategori)
The Solution to the main question: was to create a new array of summaryfields for the root node(in my intial for loop):
//Summary fields for root node
var sumFieldRoot:SummaryField2 = new SummaryField2();
sumFieldRoot.dataField = Ressurstavle.ressursKey + i;
sumFieldRoot.summaryOperation = "SUM";
sumFieldArrayRoot[i] = sumFieldRoot;
Answer to the side question:
This was pretty much as easy as pointed out by wvxyw. Code for this solution below:
private function summary_styleFunction(data:Object, col:AdvancedDataGridColumn):Object
{
var output:Object;
var field:String = col.dataField;
if ( data.children != null )
{
if(data[field] >5){
output = {color:0xFF0000, fontWeight:"bold"}
}
else {
output = {color:0x006633, fontWeight:"bold"}
}
//output = {color:0x081EA6, fontWeight:"bold", fontSize:14}
}
return output;
}