Query Large Data from Fusion Tables from Google Apps Script - google-apps-script

Question

I've loaded a 66 MB csv file to Fusion Tables. It's about 475k rows long and 12 columns wide.
I'm using Google Apps Script and trying to query the data within there.
One of the columns is the name of the person who that data belongs to, for instance, Joe.
If I want to pull all of Joe's data out so I can display it to him in a nice format, I'm using this query:
var tableId = my_table_id;
var sql1 = "SELECT * FROM " + tableId + " WHERE 'User' = 'Joe'";
var result = FusionTables.Query.sql(sql1,{hdrs : false});
The issue is that Joe has about 52k lines of data. I want to return it so I can load it to a datable and the user can sort through it and view all of the data. I get one of two errors:
If I run the query as above I get:
Response Code: 413. Message: response too large.
If I just try to select it all (SELECT * FROM tableId), I get:
Response size is larger than 10 MB. Please use media download
For media download, I've tried specifying alt : 'media' in the parameters, but I don't think that works within Google Apps script (I can't find documentation on it anywhere).
I have also tried looping through the queries, so select * limit 0,1000, then select * limit 1001,2000, ect. However, fusion tables SQL doesn't seem to support that either.
At this point, I may just leave the CSV in my drive, parse it in on the fly, but that's my last resort. Any advice would be appreciated!

Answer 1

So I think I figured this out. I'm sure it's not the most elegant solution, but here goes:
I run a quick query to check the count() for Joe to see how many records there are and only run loops if needed. I set the max to 40,000 records:
var total_rows_query = "SELECT COUNT() FROM " + tableId + " WHERE 'User' = " + username;
var total_rows = FusionTables.Query.sql(total_rows_query,{hdrs : false}).rows[0][0];
If the total rows are greater than I want, I use the OFFSET and LIMIT parameters to structure the queries:
max_rows = 40000;
if(total_rows > max_rows){
var counter = 0;
//adding in a zero to the ranges since the last query will be the offset of 0, meaning all of them
var ranges = [0]
while(counter + chunk_size < total_rows){
counter = counter + chunk_size;
ranges.push(counter)
}
ranges.push(total_rows)
//Now ranges is an array with zero at the beginning, and counting up by the chunk size I want, ending with the total_rows for the user as the last oen
//This is the array that will be output after concating
var output = []
//looping through the array, setting the offset to the first item, and the limit to the next item minus the first
for(i=0;i<ranges.length-1;i++){
var offset = ranges[i]
var limit = ranges[i+1] - offset
var query = "SELECT * FROM " + tableId + " WHERE 'User' = '" + username + "' OFFSET " + offset + " LIMIT " + limit;
output = output.concat(FusionTables.Query.sql(query,{hdrs : false}).rows)
}
}else{
//if the count is less or equal to the chunk size, just run the one query
var query = "SELECT * FROM " + tableId + " WHERE 'User' = " + username;
var output = FusionTables.Query.sql(query,{hdrs : false}).rows
}
The last thing to note is that if the username is two words, for instance 'John Smith', you may need to add in quotes around your username, so instead of
var total_rows_query = "SELECT COUNT() FROM " + tableId + " WHERE 'User' = " + username;
It would be:
var total_rows_query = "SELECT COUNT() FROM " + tableId + " WHERE 'User' = '" + username + "'";
I spend the last two days trying to figure this out, so I hope it helps someone out there!