I am trying to find ways to filter and render 100 million+ data points as a heat map in real time.
Each point in addition to the (x,y) coordinates has a fixed set of attributes (int, date, bit flags) which can be dynamically chosen by the user in order to filter down the data set.
Would it be feasible to accelerate all or parts of this task on GPUs?
It would help if you were more specific, but I'm assuming that you want to apply a user specified filter to the same 2D spatial data. If this is the case, you could consider organizing your data into a spatial datastructure, such as a Quadtree or K-d tree.
Once you have done this, you could run a GPU kernel for each region in your datastructure based on the filter you want to apply. Each thread will figure out which points in its region satisfy the specified filter.
Definitely, this is the kind of problem that fits into the GPGPU spectrum.
You could decide to create your own kernel to filter your data or simply use some functions of vendor's libraries to that end. Probably, you would normalize, interpolate, and so on, which are common utilities in those libraries. These kind of functions are typically embarrassingly parallel, at it shouldn't be difficult to create your own kernel.
I'd rather use a visualization framework that allows you to filter and visualize your data in real time. Vispy is a great option but, of course, there are some others.
I am developing a massive multiplayer online 2D game in Haxe/OpenFL, where world is divided into two-dimensional grid of sectors, therefore each sector acts as one object. On server, these sector objects may or may not exist in memory (depending on whether players are currently in them or not) and I need to access these sector objects via their x and y coordinate. Theoretically there can be several thousands of these objects at any given time.
My question is, what is the best (fastest, most efficient) way of storing and accessing these objects. I tried to use two-dimensional arrays with x and y as indexes, but my implementation was extremely slow.
In server side you may use a hashmap data structure to save the sectors,the key would be x_y.Check the memory after You initialize all the sectors when server startup, if it cost two much, you may initialize some main sectors at first and another sector should be initialized only when it's actually been accessed.
"modern" because that definition may change over time (and specifically I mean desktop browsers)
"handle" because that may vary depending on machine configurations/memory, but specifically I mean a general use case.
This question came to mind over a particular problem I'm trying to solve involving large datasets.
Essentially, whenever a change is made to a particular dataset I get the full dataset back and I have to render this data in the browser.
So for example, over a websocket I get a push event that tells me a dataset has changes, and then I have to render this dataset in HTML by grabbing an existing DOM element, duplicating it, populating the elements with data from this set using classnames or other element identifiers, and then add it back to the DOM.
Keep in mind that any object (JSON) in this dataset may have as many as 1000+ child objects, and there may be as many as 10,000+ parent objects, so as you can see there may be an instance where the returned dataset is upwards towards 1,000,000 => 10,000,000 data points or more.
Now the fun part comes when I have to render this stuff. For each data point there may be 3 or 4 tags used to render and style the data, and there may be event listeners for any of these tags (maybe on the parent container to lighten things up using delegation).
To sum it all up, there can be a lot of incoming information that needs to be rendered and I'm trying to figure out the best way to handle this scenario.
Ideally, you'd just want to render the changes for that single data point that has changes rather than re-rendering the whole set, but this may not be an option due to how the backend was designed.
My main concern here is to understand the limitations of the browser/DOM and looking at this problem through the lense of the frontend. There are some changes that should happen on the backend for sure (data design, caching, pagination), but that isnt the focus here.
This isn't a typical use case for HTML/DOM, as I know there are limitations, but what exactly are they? Are we still capped out at about 3000-4000 elements?
I've got a number of related subquestions for this that I'm actively looking up but I thought it'd be nice to share some thoughts with the rest of the stackoverflow community and try to pool some information together about this issue.
What is "reasonable" amount of DOM elements that a modern browser can handle before it starts becoming slow/non-responsive?
How can I benchmark the number of DOM elements a browser can handle?
What are some strategies for handling large datasets that need to be rendered (besides pagination)?
Are templating frameworks like mustache and handlebars more performant for rendering html from data/json (on the frontend) than using jQuery or Regular Expressions?
Your answer is: 1 OR millions. I'm going to copy/paste an answer from a similar question on SO.
To be honest, if you really need an absolute answer to this question, then you might want to reconsider your design.
No answer given here will be right, as it depends upon many factors that are specific to your application. E.g. heavy vs. little
CSS use, size of the divs, amount of actual graphics rendering
required per div, target browser/platform, number of DOM event
listeners etc..
Just because you can doesn't mean that you should! :-)"
See: how many div's can you have before the dom slows and becomes unstable?
This really is an unanswerable question, with too many factors at too many angles. I will say this however, in a single page load, I used a javascript setinterval at 1ms to continually add new divs to a page with the ID incrementing by 1. My Chrome browser just passed 20,000, and is using 600MB Ram.
This is a question for which only a statistically savvy answer could be accurate and comprehensive.
Why
The appropriate equation is this, where N is the number of nodes, bytesN is the total bytes required to represent them in the DOM, the node index range is n ∈ [0, N), bytesOverhead is the amount of memory used for a node with absolute minimum attribute configuration and no innerHTML, and bytesContent is the amount of memory used to fill such a minimal node.
bytesN = ∑N (bytesContentn + bytesOverheadn)
The value requested in the question is the maximum value of N in the worst case handheld device, operating system, browser, and operating conditions. Solving for N for each permutation is not trivial. The equation above reveals three dependencies, each of which could drastically alter the answer.
Dependencies
The average size of a node is dependent on the average number of bytes used in each to hold the content, such as UTF-8 text, attribute names and values, or cached information.
The average overhead of a DOM object is dependent on the HTTP user agent that manages the DOM representation of each document. W3C's Document Object Model FAQ states, "While all DOM implementations should be interoperable, they may vary considerably in code size, memory demand, and performance of individual operations."
The memory available to use for DOM representations is dependent upon the browser used by default (which can vary depending on what browser handheld device vendors or users prefer), user override of the default browser, the operating system version, the memory capacity of the handheld device, common background tasks, and other memory consumption.
The Rigorous Solution
One could run tests to determine (1) and (2) for each of the common http user agents used on handheld devices. The distribution of user agents for any given site can be obtained by configuring the logging mechanism of the web server to place the HTTP_USER_AGENT if it isn't there by default and then stripping all but that field in the log and counting the instances of each value.
The number of bytes per character would need to be tested for both attributes values and UTF-8 inner text (or whatever the encoding) to get a clear pair of factors for calculating (1).
The memory available would need to be tested too under a variety of common conditions, which would be a major research project by itself.
The particular value of N chosen would have to be ZERO to handle the actual worst case, so one would chose a certain percentage of typical cases of content, node structures, and run time conditions. For instance, one may take a sample of cases using some form of randomized in situ (within normal environmental conditions) study and find N that satisfies 95% of those cases.
Perhaps a set of cases could be tested in the above ways and the results placed in a table. Such would represent a direct answer to your question.
I'm guessing it would take an well educated mobile software engineer with flare for mathematics, especially statistics, five full time weeks to get reasonable results.
A More Practical Estimation
One could guess the worst case scenario. With a few full days of research and a few proof-of-concept apps, this proposal could be refined. Absent of the time to do that, here's a good first guess.
Consider a cell phone that permits 1 Gbyte for DOM because normal operating conditions use 3 Gbytes out of the 4 GBytes for the above mentioned purposes. One might assume the average consumption of memory for a node to be as follows, to get a ballpark figure.
2 bytes per character for 40 characters of inner text per node
2 bytes per character for 4 attribute values of 10 characters each
1 byte per character for 4 attribute names of 4 characters each
160 bytes for the C/C++ node overhead in the less efficient cases
In this case Nworst_case, the worst case max nodes,
= 1,024 X 1,024 X 1,024
/ (2 X 40 + 2 X 4 X 10 + 1 X 4 X 4 + 160)
= 3,195,660 . 190,476.
I would not, however, build a document in a browser with three million DOM nodes if it could be at all avoided. Consider employing the more common practice below.
Common Practice
The best solution is to stay far below what Nworst_case might be and simply reduce the total number of nodes to the degree possible using standard HTTP design techniques.
Reduce the size and complexity of that which is displayed on any given page, which also improves visual and conceptual clarity.
Request minimal amounts of data from the server, deferring content that is not yet visible using windowing techniques or balancing response time with memory consumption in well-planned ways.
Use asynchronous calls to assist with the above minimalism.
For those wondering: Google has it's Dom size recommendation:
Domsize recommandations
"
An optimal DOM tree:
Has less than 1500 nodes total.
Has a maximum depth of 32 nodes.
Has no parent node with more than 60 child nodes.
In general, look for ways to create DOM nodes only when needed, and destroy them when no longer needed.
If your server ships a large DOM tree, try loading your page and manually noting which nodes are displayed. Perhaps you can remove the undisplayed nodes from the loaded document, and only create them after a user gesture, such as a scroll or a button click.
If you create DOM nodes at runtime, Subtree Modification DOM Change Breakpoints can help you pinpoint when nodes get created.
If you can't avoid a large DOM tree, another approach for improving rendering performance is simplifying your CSS selectors. See Reduce The Scope And Complexity Of Style Calculations.
"
There are a number of ways the DOM elements can become too many. Here is a React + d3 component I have been using to render many elements and get a more real-world sense of the DOM's limits:
export const App = React.memo((props) => {
const gridRef = React.useRef(null);
React.useEffect(() => {
if (gridRef.current) {
const table = select(gridRef.current);
table
.selectAll("div")
.data([...new Array(10000)])
.enter()
.append("div")
.text(() => "testing");
}
if (props.onElementRendered) {
props.onElementRendered();
}
}, []);
return <div ref={gridRef} />;
});
On a 2021 Macbook Pro with 16GB of memory running Chrome I'm seeing serious delay (I think on the paint step) starting at around 30,000 elements
Just to add another data point. I loaded the single-page GNU Bison manual, which claims to be 2064K bytes. In the console, I typed document.querySelectorAll('*') and the answer was 22183 nodes, which rather exceeds Google's alleged "optimal sizes".
I detected no delay loading the page (50Mb ethernet connection). Once loaded, I detected no delay whatsoever clicking on internal links, scrolling, etc.
This was on my relatively massively powered desktop machine. Tried the same thing on my Galaxy Note 4 (ancient wifi connection, def not 50Mb). This time (no surprise) I had to wait a few seconds (<5) for it to load. After that, clicking on links and scrolling was again about as instantaneous as my eye could see.
I don't doubt that 30,000 nodes of React could spell trouble, nor that I can have vastly more than that number of framework-free simple HTML nodes without the slightest problem. The notion that I should worry about more than 1500 nodes sounds pretty crazy to me, but I'm sure YMMV.
To traverse a tree data structure in whatever form one might represent, one needs to use either recursion or use iteration with stacks and queues.
How would one do this on the GPU using CUDA? As far as I know, neither recursion nor stack structures likes stacks and queues are supported in CUDA.
In context my problem is that of range searching, where given a point I want to traverse an octree data structure to find all points within a radius of 'r' centered at that point.
The most effecient serial algorithms / data structures do not necessarily make the most effecient parallel implementations.
That said, this is not a new question and a little bit of googling can turn up interesting results.
What are some more interesting graph data structures for working with networks? I am interested in structures which may offer some particular advantage in terms of traversing the network, finding random nodes, size in memory or for insertion/deletion/temporary hiding of nodes for example.
Note: I'm not so much interested in database like designs for addressing external memory problems.
One of my personal favorites is the link/cut tree, a data structure for partitioning a graph into a family of directed trees. This lets you solve network flow problems asymptotically faster than more traditional methods and can be used as a more powerful generalization of the union/find structure you may have heard of before.
I've heard of Skip Graphs ( http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=skip+graphs ), a probabilistic graph structure that is - as far as I know - already in use in some peer-to-peer applications.
These graphs are kind of self-organizing and their goal is to achieve a good connectivity and a small diameter. There is a distributed algorithm that tries to achieve such graphs: http://www14.informatik.tu-muenchen.de/personen/jacob/Publications/podc09.pdf