Join operation in an ImageCollection - gis

I carried out join operation to help in smoothing out my images in a Landsat ImageCollection by getting at least 3 Images in a specified time window, obtain the median and then adding those images to the ImageCollection. I expected to obtain an ImageCollection that I could still carry out some filter functions to create a time series with the median images, but it didn't work.
// 1. selecting the time window
var days = 32;
var milli = ee.Number(days).multiply(1000*24*60*60)
console.log(milli)
var join = ee.Join.saveAll({
matchesKey:"images"
})
// 2. Apply the filter
var dif_filter = ee.Filter.maxDifference({
difference:milli,
leftField:"system:time_start",
rightField:"system:time_start"
})
// 3. Apply the join
var joined_collection = join.apply({
primary: original_collection,
secondary:original_collection,
condition:dif_filter
})
// Adding the median images to our collection
function medianCalculator(image){
var imageGetter = ee.ImageCollection.fromImages(image.get("images"))
var medianImage = imageGetter.reduce(ee.Reducer.median())
return ee.Image(image).addBands(medianImage).select("NDVI_median", "EVI_median", "NDVI", "EVI")
}
// collection with smoothed Images
var collection_2 = joined_collection.map(medianCalculator)
print(collection_2)
I wanted to carry out some filters on "collection_2" but there was an error, "Line 110: collection_2.filter(...).median is not a function". What am I missing.
I suspect that after performing the Join, the ImageCollection got turned into a FeatureCollection so I tried looking for ways of converting a FeatureCollection to an ImageCollection...to no avail.

Related

In Google Earth Engine: Most efficiently reduceRegions over each image in ImageCollection, saving mean as a Feature property?

I have a FeatureCollection made up of many (100-200) polygons ('ftr_polygons'). I also have an ImageCollection made up of monthly median Landsat8 bands and indices ('byMonth'). I want to ReduceRegions and save a median (or mean) spatial average from each polygon in the FeatureCollection. End goal is to export to csv a timeseries of monthly mean bands/indices within each polygons over multiple years (2013-2019).
With the code below, I am able to do this for ~1 year, but any more than that, and I get an error: 'FeatureCollection (Error) Computation timed out’. Is there a better way to do this?
// define the function that will grab median (or mean) spatial reductions for each polygon, for each month
var extractdata = function(medianImage,ftr_polygons) {
var date_start = ee.Date(medianImage.get('system:time_start')).format("YYYY-MM"); // get date as string to append to each property
// spatial MEDIAN
ftr_polygons = medianImage.reduceRegions({ // create feature collection with new properties, bands for each month, uniquely named
collection: ftr_polygons,
reducer: ee.Reducer.median(),
scale: 30,
tileScale: 1}); // tile scale
var ftr_polygons_propnames = ftr_polygons.first().propertyNames(); // get property names first
var ftr_polygons_newnames = ftr_polygons_propnames.replace('NDVI_median',
ee.String('NDVI_median_').cat(date_start)); //replace property names with band+date
ftr_polygons_newnames = ftr_polygons_newnames.replace('EVI_median',
ee.String('EVI_median_').cat(date_start)); //replace property names with band+date
ftr_polygons_newnames = ftr_polygons_newnames.replace('NIRv_median',
ee.String('NIRv_median_').cat(date_start)) ; //replace property names with band+date
ftr_polygons = ftr_polygons.map(function(f) {return f.select(ftr_polygons_propnames,ftr_polygons_newnames)});
return ftr_polygons;
};
// apply the function over ImageCollection byMonth, beginning with feature collection ftr_polygons
var ftr_polygons = ee.FeatureCollection(byMonth.iterate(extractdata,ftr_polygons));
// remove geometry on each feature before printing or exporting
var myproperties=function(feature){
feature=ee.Feature(feature).setGeometry(null);
return feature;
};
var ftr_polygon_export = ftr_polygon.map(myproperties)
print(ftr_polygon_export.limit(1), 'For export w monthly properties');
Maybe this answer: https://stackoverflow.com/a/48412324/12393507 alludes to a better way:
The same approach can be used with reduceRegions() as well, mapping over images and then over regions. However, you will have to map over the resulting features to set dates.
I would appreciate more info on this approach.
Thanks.
For computationally intensive operations that will run for a long time you should always export your results instead of visualizing/printing them.
For more info read through this section of the debugging page in the Earth Engine manual.

Using ItemCollection on a BoxFolder type with Box API only returns 100 results and cannot retrieve the remaining ones

For a while now, I've been using the Box API to connect Acumatica ERP to Box and everything has been going fine until recently. Whenever I try to use a BoxCollection type with the property ItemCollection, I'll only get the first 100 results no matter the limit I set in the GetInformationAsync(). Here is the code snippet:
[PermissionSet(SecurityAction.Assert, Name = "FullTrust")]
public BoxCollection<BoxItem> GetFolderItems(string folderId, int limit = 500, int offset = 0)
{
var response = new BoxCollection<BoxItem>();
var fieldsToGet = new List<string>() { BoxItem.FieldName, BoxItem.FieldDescription, BoxItem.FieldParent, BoxItem.FieldEtag, BoxFolder.FieldItemCollection };
response = Task.Run(() => Client.FoldersManager.GetFolderItemsAsync(folderId, limit, offset)).Result;
return response;
}
I then pass that information on to a BoxFolder type variable, and then try to use the ItemCollection.Entries property, but this only returns 100 results at a time, with no visible way to extract the remaining 61 (in my case, the Count = 161, but Entries = 100 always)
Another code snippet of the used variable, I am basically trying to get the folder ID based on the name of the folder inside Box:
private static void SyncProcess(BoxFolder rootFolder, string folderName)
{
var boxFolder = rootFolder.ItemCollection.Entries.SingleOrDefault(ic => ic.Type == "folder" && ic.Name == folderName);
}
I wasn't able to find anything related to that limit = 100 in the documentation and it only started to give me problems recently.
I had to create a work around by using the following:
var boxCollection = client.GetFolderItems(rootFolder.Id);
var boxFolder = boxCollection.Entries.SingleOrDefault(ic => ic.Type == "folder" && ic.Name == folderName);
I was just wondering if there was a better way to get the complete collection using the property ItemCollection.Entries like I used to, instead of having to fetch them again.
Thanks!
Box pages folder items to keep response times short. The default page size is 100 items. You must iterate through the pages to get all of the items. Here's a code snippet that'll get 100 items at a time until all items in the folder are fetched. You can request up to 1000 items at a time.
var items = new List<BoxItem>();
BoxCollection<BoxItem> result;
do
{
result = await Client.FoldersManager.GetFolderItemsAsync(folderId, 100, items.Count());
items.AddRange(result.Entries);
} while (items.Count() < result.TotalCount);
John's answer can lead to a duplicate values in your items collection if there will be external/shared folders in your list. Those are being hidden when you are calling "GetFolderItemsAsync" with "asUser" header set.
There is a comment about it in the Box API's codeset itself (https://github.com/box/box-windows-sdk-v2/blob/main/Box.V2/Managers/BoxFoldersManager.cs)
Note: If there are hidden items in your previous response, your next offset should be = offset + limit, not the # of records you received back.
The total_count returned may not match the number of entries when using enterprise scope, because external folders are hidden the list of entries.
Taking this into account, it's better to not rely on comparing the number of items retrieved and the TotalCount property.
var items = new List<BoxItem>();
BoxCollection<BoxItem> result;
int limit = 100;
int offset = 0;
do
{
result = await Client.FoldersManager.GetFolderItemsAsync(folderId, limit, offset);
offset += limit;
items.AddRange(result.Entries);
} while (offset < result.TotalCount);

Loading multiple CSV in DC.js, adding a value, and concatenating the results into a single dataTable

I have four CSVs with the same header information, each representing a quarterly result within a year.
Therefore for one result I can load it and display it into a dataTable simple via
d3.csv("data/first-quarter"), function(dataQ1){
dataQ1.forEach(function(d){
d.callTypes = d['Call Types'];
d.callDesc = d['Call Description'];
d.callVol = d['Call Volume'];
d.quarter = 'Q1';
});
var facts = crossfilter(dataQ1);
var timeDimension = facts.dimension(function(d){
return d.quarter;
});
dataTable
... //data table attributes
dc.renderAll();
});
However complications arise when I try to retrieve from multiple sources and append the results.
One approach I took was to place all the file path names into an array and iterate through a forEach, with a flag to show when it was the last iteration to render the table. But this failed with a "Too many recursion" error.
And the next was to nest as such
d3.csv(filesPathNames[0], function(dataQ1){
d3.csv(filesPathNames[1], function(dataQ2){
d3.csv(filesPathNames[2], function(dataQ3){
d3.csv(filesPathNames[3], function(dataQ4){
But both of these methods seem to not work due to the fact that I can't simply add one CSV value to another. So I think where I'm having an issue is that I'm not sure how to concatenate dataQ1, dataQ2, dataQ3, and dataQ4 properly.
Is the only solution to manually append one to another with an added value of Q1, Q2, Q3, and Q4 as the time dimension?
Like Lars said, you can use the queue library. Here is an example of how this might work:
Step 1) Queue up your files:
<script type="text/javascript" src="http://d3js.org/queue.v1.min.js"></script>
var q = queue()
.defer(d3.csv, "data/first-quarter")
.defer(d3.csv, "data/second-quarter");
Step 2) Wait for the files to load:
q.await(function(error, q1data, q2data) {
Step 3) Add the data to crossfilter:
var ndx = crossfilter();
ndx.add(q1data.map(function(d) {
return { callTypes: d['Call Types'],
callDesc: d['Call Description'],
callVol: d['Call Volume'],
quarter: 'Q1'};
}));
ndx.add(q2data.map(function(d) {
return { callTypes: d['Call Types'],
callDesc: d['Call Description'],
callVol: d['Call Volume'],
quarter: 'Q2'};
}));
Step 4) Use your cross filter as you wish:
var timeDimension = ndx.dimension(function(d){
return d.quarter;
});
dataTable
... //data table attributes
dc.renderAll();
Here is an example using this approach with the dc.js library: https://github.com/dc-js/dc.js/blob/master/web/examples/composite.html

How to push instantiated MC into Array dynamically?

Im really stuck. I have 5 MC´s that are being spliced from one array at a certain time. In that same function I want to push another movieclips into another array. The two arrays holds mc's that represent right or wrong answers. So when one question is being given a correct answer that questions visualisation alters.
This function holds a incrementing variable as I do want the mc's to be pushed by the user and one at the time. The thing is I cant seem to refer them properly.
I´ve tried
pQuestSum = this[pQuest + pQuestNumber];
and
pQuestSum = this[pQuest] + pQuestNumber;
and pretty much everything I´ve imagined would work...but the problem is I havent tried
the right thing.
when I trace pQuestSum (which would be the reference) I get an error saying thats its not a number.
this is one of 5 mc's named from 1-5:
var passedquest1:PassedQuest = new PassedQuest();
this is the vars that i try to to build a reference of
var pQuest = "passedquest";
var pQuestNumber = 1;
var pQuestSum;
var questCorrArray:Array = [];
if(event.target.hitTestObject(questArray[ix])){
removeChild(questArray[ix]);
questArray.splice(ix,1);
pQuestNumber ++;
pQuestSum = this[pQuest] + pQuestNumber;
trace("pQuestSum"); // NaN
questCorrArray.push(pQuestSum);
//trace(questArray.length);
pointsIncreased = false;
questPoints = 0;
}
How do I refer an existing movieclip when the reference consists of both a string and a number? Hope I made myself somewhat clear:)
If you had an instance of an object on your timeline called "passedquest1" (as an example), then you could access it this way:
var myObj = this["passedquest" + 1];
Or,
var pQuest = "passedquest";
var pQuestNumber = 1;
var myObj = this[pQuest+ pQuestNumber.toString()];
When you do this: pQuestSum = this[pQuest] + pQuestNumber;, you are trying add the number to an object (this[pQuest]), unless you have number/int var called "passedquest", this will result in NaN.

AdvancedDataGrid total sum of branch nodes

Introduction:
I have an AdvancedDataGrid displaying hierarchical data illustrated by the image below:
The branch nodes "Prosjekt" and "Tiltak" display the sum of the leaf nodes below.
Problem: I want the root node "Tavle" to display the total sum of the branch nodes below. When i attempted to do this by adding the same SummaryRow the sum of the root node was not calculcated correctly(Every node's sum was calculated twice).
dg_Teknikktavles = new AutoSizingAdvancedDataGrid();
dg_Teknikktavles.sortExpertMode="true";
dg_Teknikktavles.headerHeight = 50;
dg_Teknikktavles.variableRowHeight = true;
dg_Teknikktavles.addEventListener(ListEvent.ITEM_CLICK,dg_TeknikktavlesItemClicked);
dg_Teknikktavles.editable="false";
dg_Teknikktavles.percentWidth=100;
dg_Teknikktavles.minColumnWidth =0.8;
dg_Teknikktavles.height = 1000;
var sumFieldArray:Array = new Array(context.brukerList.length);
for(var i:int = 0; i < context.brukerList.length; i++)
{
var sumField:SummaryField2 = new SummaryField2();
sumField.dataField = Ressurstavle.ressursKey + i;
sumField.summaryOperation = "SUM";
sumFieldArray[i] = sumField;
}
var summaryRow:SummaryRow = new SummaryRow();
summaryRow.summaryPlacement = "group";
summaryRow.fields = sumFieldArray;
var summaryRow2:SummaryRow = new SummaryRow();
summaryRow2.summaryPlacement = "group";
summaryRow2.fields = sumFieldArray;
var groupField1:GroupingField = new GroupingField();
groupField1.name = "tavle";
//groupField1.summaries = [summaryRow2];
var groupField2:GroupingField = new GroupingField();
groupField2.name = "kategori";
groupField2.summaries = [summaryRow];
var group:Grouping = new Grouping();
group.fields = [groupField1, groupField2];
var groupCol:GroupingCollection2 = new GroupingCollection2();
groupCol.source = ressursTavle;
groupCol.grouping = group;
groupCol.refresh();
Main Question: How do i get my AdvancedDataGrid's (dg_Teknikktavles) root node "Tavle" to correctly display the sum of the two branch nodes below?
Side Question: How do i add a red color to the numbers of the root node's summary row that exceed 5? E.g the column displaying 8 will exceed 5 in the root node's summary row, and should therefore be marked red
Thanks in advance!
This is a general answer, without code examples, but I had to do the same just couple of days ago, so my memory is still fresh :) Here's what I did:
Created a class A to represent an item renderer data, extended it from Proxy (I had field names defined at run time), and let it contain a collection of values as it's data member. Once accessed through flash_proxy::getPropery(fieldName) it would find a corresponding value in the data member containing the values and return it. Special note: implement IUID, just do it, it'll save you couple of days of frustration.
Extended A in B, added a children property containing ArrayCollection of A (don't try to experiment with other collection types, unless you want to find yourself examining tons of framework code, trust me, it's ugly and is impossible to identify as interesting). Let B override flash_proxy::getPropery - depending of your compiler this may, or may not be possible, if not possible - call some function from A.flash_proxy::getPropery() that you can override in B. Let this function query every instance of A, which is a child of B, asking the same thing, as DataGrid itself would, when building item renderers - this way you would get the total.
When creating a data provider. Create an ArrayCollection of B (again, don't try to experiment with other collections--unless you are ready for lots of frustration). Create Hierarchical data that uses this array collection as a source.
Colors - that's what you use item renderers for, just look up any tutorial on using item renderers, that must be pretty basic.
In case someone else has the same problem:
The initial problem that everything was summed twice, was the result of using the same Array of SummaryField2 (sumFieldArray in the code) for both grouping fields(GropingField2 tavle and kategori)
The Solution to the main question: was to create a new array of summaryfields for the root node(in my intial for loop):
//Summary fields for root node
var sumFieldRoot:SummaryField2 = new SummaryField2();
sumFieldRoot.dataField = Ressurstavle.ressursKey + i;
sumFieldRoot.summaryOperation = "SUM";
sumFieldArrayRoot[i] = sumFieldRoot;
Answer to the side question:
This was pretty much as easy as pointed out by wvxyw. Code for this solution below:
private function summary_styleFunction(data:Object, col:AdvancedDataGridColumn):Object
{
var output:Object;
var field:String = col.dataField;
if ( data.children != null )
{
if(data[field] >5){
output = {color:0xFF0000, fontWeight:"bold"}
}
else {
output = {color:0x006633, fontWeight:"bold"}
}
//output = {color:0x081EA6, fontWeight:"bold", fontSize:14}
}
return output;
}