Load Compressed csv data in d3 for page optimization - csv

Question

I am having a 50MB csv data, is there any possibility i can compress the data to load
d3.js/ dc.js charts, now the page is too slow i would like to optimise it.. any help is much appreciated
Thanks in advance

Answer 1

I think it would be the best to implement a lazy loading solution. The idea is simple: you create a small, say 2MB CSV file and render your visualization using it. At the same time you start loading your full 50MB CSV.
Here is a small snippet:
DS = {} // your app holder for keeping global scope clean
d3.csv('data/small.csv', function(err, smallCSV) {
// Start loading big file immediately
d3.csv('data/big.csv', function(err, bigCSV) {
DS.data = bigCSV // when big data is loaded it replaces old partial data
DS.drawViz() // redraw viz
})
// This portion of code also starts immediately, while big file is still loading
DS.data = smallCSV
DS.drawViz() // function which has all your d3 code and uses DS.data inside
})
The change from small to big could be done in such way that user would have no clue, that something happened in the background. Consider this example where quite big data file is loaded and you can feel the lag at start. This app could load much faster if data would be loaded in two rounds.

Answer 2

That's a lot of data; give us a sample of the first couple rows. What are you doing with it, and how much of it affects what's on screen? Where does the csv come from (i.e., local or web service)?
If it's a matter of downloading the resource, depending on how common and large the values are, you may be able to refactor them into 1-byte keys with definitions pre-loaded (hash maps are O(1) access). Also if you're using a large amount of numerical data, perhaps a different number space (i.e., something that uses less characters than base 10) can shave some bytes off the final size since the CSV values are strings.
It sounds like CSV may not be the way to go, though, especially if your CSV is mostly unique strings or certain numerical data that won't benefit from the above optimizations. If you're loading the CSV from a web service, you could change it so that certain chunks are returned via some passed key (or handle it smarter server-side). So you would load only what you need at any given time, and probably cache it.
Finally, you could schedule multiple async calls to load the the whole thing in small chunks similar to what was suggested by leakyMirror. Since it would probably make most sense to use a lot of chunks, you'd want to do it with code (instead of typing all of those callbacks), and use an async event scheduler. I know there's a popular async library (https://github.com/caolan/async) that has a bunch of ways to do this, or you can write your own callback scheduler.