Getting a server error on
var contents = folder.getFiles(1501,500);
Ran fine for the first three iterations, 0,500 and 501,500 and 1001,500. There are over 3,000 files in the folder. Clicked it many times over the last several hours. Would anything else cause the error?
Also how are files added to a folder. If I add files can I get the increment by looking at the end of the list or the beginning?
Just played with the 500, reducing it and I got it to run a few more times, 1900 files total and it stopped again. Is there some limit on the max number. Looks like no more than 1999 files. Is this true? Guess I have to split the directory.
Related
Been using getFileById & insertImage in a sheet/program for months with no problem but it suddenly stopped working two days ago. In order to solve the issue I created a fresh sheet and did this simple code:
function myFunction() {
var picId = "1x2jkgSQvVKAIOg6DQ0rss82YhTQl918c";
var picToPlace = DriveApp.getFileById(picId);
SpreadsheetApp.getActiveSpreadsheet().insertImage(picToPlace,1,1);
}
It works just like it should however if I stick the same code into my main sheet I get "Exception: The image could not be inserted. Please verify it is valid and try again."
Why is this happening? I'm not doing anything different, the function is sitting by itself. Been building this sheet/program for months. Have I overloaded the sheet somehow? Again, if I make a fresh, plain sheet, stick this code in it, it works fine, but if I go back to my main sheet, stick in the same function, it just refuses to work. Its in its own function, nothing else is happening w/in it, it's the only thing I'm calling. I'm at a lost trying to figure out why insertImage has completely stopped working in this sheet. There are thousands and thousands lines of code, over 15K rows of data in some tabs, it's referencing hundreds and hundreds of pictures I've uploaded to a server, and processing various aspects of thousands of people I've inputted into this program. Did I overload this sheet somehow or is Google just over me right now (ie, does it not like me trying to use insertImage, for example, 100 times in one call..) Sorry, I've put so much effort into this and the "insertImage" function suddenly not working is a major blow to it when it is NOT the code that is malfunctioning. If it was the code at least I could fix it.
What's odd is I was running the program 2 days ago. Would look up, for example, 100 people and then have it show me all the pics on one page. Ran this a few times and in the middle one of the outputs it suddenly stopped working. I re-ran a query, asked it to show me pictures and it started placing like 20 of them and then just refused to show any more at all. This really seems like Google telling me to f8ck off.
I have a following scenario, 2 revit files, ModelA.rvt and ModelB.rvt. They are cross-referenced together, zipped and uploaded twice under diferrent object key (ModelA.zip, ModelB.zip). ZIP files are identical, very small(4MB) and containing both files. They both are uploaded succesfuly in a loop using:
PUT https://developer.api.autodesk.com/oss/v2/buckets/:bucketKey/objects/:objectName
Files are overwritten with token scope data:write and a post job called with x-ads-force = true in case of model update. Then I call the POST JOB 2x in a loop, once with ModelA.rvt as rootFilename for ModelA.zip and secondly with ModelB.rvt for ModelB.zip. Both post jobs are done sucesfully.
Right after I am getting manifest for both zip files each 10 secs. ModelB.zip is translated 100% in a few secs, but ModelA.zip never finishes (few hours so far), just hangs for no reason. On friday I thought that is just temporary issue, but no it still lasts.
I tried this scenario 3x times, each time with different set of files today and 3 days back. Same result. This one is the easiest one and they are all already present on the cloud. Still have no idea what is going on.
When I list bucket objects, zip files are never present. Another weird thing. Other files with non-zip extension are.
Does anyone have a clue what is causing this, what could be possible workaround? That is serious issue, because it corrupts usability and reliability of the whole API.
The linked revit files need to be in one zipfile with the new v2 API. See this post for more details: http://adndevblog.typepad.com/cloud_and_mobile/2016/07/translate-referenced-files-by-derivative-api.html
I am trying to gather all of the files and folders that are descendants of a given folder.
To do this I use file.list() with q="'FOLDER_ID' in parent" and trashed=false with FOLDER_ID being the ID of the folder I am interested in. As I process the results I keep track of all of the folders that get returned from this request and then repeat the files.list() call using the new folders in the q parameter. I combine multiple folders in one request by using or and continue to repeat this until no new folders are returned.
Example:
Initial Request: q="('FOLDER_ID' in parent) and trashed=false"
All Subsequent Requests: q="('FOLDER_ID_1' in parent or 'FOLDER_ID_2' in parent or 'FOLDER_ID_3' in parent ...) and trashed=false"
(For more information about how to create queries see Drive REST API - Search for Files)
Sometimes this returns all the folders it should and other times some are left out. This doesn't happen if I remove the q parameter as every single file and folder are returned, none are missing.
After some testing/trial and error, I discovered that if I am not receiving all the folders I should be, sending a request with no q seems to "fix" the problem. The next time I run my application and it uses q, all the correct folders do get returned.
Other Information:
It is not a permissions issue, I am using drive.readonly
It is not a pageSize issue as I have tried different values for this and get different results.
It is not a pageToken issue as I make sure to send a request again with the given nextPageToken when it exists.
I am running this on a folder that has a little under 4,000 descendant folders in it and a little under 25,000 descendant files in it.
I feel like this must be a bug related to using multiple folders in the q parameter in a single request, considering that I can perform the exact same process and will get different results seemingly randomly.
I suggest you abandon the approach you've taken. Making so many calls to Drive will take forever and possibly give you quota problems.
It's much, much simpler to simply fetch all the folders in a single query, and then build an in-memory hierarchy of the folder ID's you're interested in. Then run a second set of queries to fetch files with those parents.
Alternatively, if these files are being created by an application, make them all children of a common dummy parent folder that you can query against.
I found a similar issue when looking for all files a given user owns, eg:
'example.user#company.com' in owners and trashed=false
I have about 5000 files and usually I can iterate through all of them via pagination. Some days however (like today) I only get <100 results with the query above. When I rewrite my code to fetch files for a given parent-ID and then recursively iterate through the sub-folders, I will get all files. Afterwards the original query succeeds again as well.
It looks like some kind of caching issue on the google-drive server to me.
I have recently deleted quite a number of files in Google Drive, and they were moved to the trash. I would like to permanently delete them, but the files in the Trash can't be sorted by Deleted Date, but only Modified Date (which is not updated upon delete).
Therefore, I would like to leverage the Google Drive API to enumerate the trash and determine the date it was trashed, to find the set of files I would like to subsequently permanently delete.
From what I can gather, there is a file property which indicates if it has been trashed, but not the date.
I tried searching for this answer, but it's not easy to find, if it is possible. Is anyone familiar with this area of the API?
Is there a better strategy than what I am attempting?
Using Todd's detailed answer, I wrote a Python script to achieve this. It's published on GitHub (cfbao/google-drive-trash-cleaner).
E.g. to view items trashed in the last 30 days, run
python cleaner.py -v -d 30
or
cleaner.exe -v -d 30 on Windows
Edit:
Oops! big mistake!
clean -v -d 30 shows you files trashed more than 30 days ago.
If you want to see files trashed in the last 30 days, run clean -v -d 0 and see all trashed files sorted by their trash date.
Or you can modify my script so -d 30 means "in the last 30 days". Shouldn't be too big a modification.
Directly using https://developers.google.com/drive/v3/reference/changes/list, you can retrieve your most recent file/folder changes. This would seem to list those changes for folder movements as well as trashed files.
Getting the data:
Get a page token from - https://developers.google.com/drive/v3/reference/changes/getStartPageToken
Put that token into - https://developers.google.com/drive/v3/reference/changes/list, with page size of 500, includeRemoved on true, and select all fields with the fields editor link button.
You will see no results, because you are using the most recent page token. You need to manually reduce the page token until the returned changes.time is before the desired date range. (There is no filter for this in the query). Once you have established the correct page token, continue with the steps below.
Before Executing, open the Developer Tools (I was using Chrome) and view the Network section
Look for a transfer name staring with "changes?pageToken=...", and select it, copy the Response to Notepad++
Note the nextPageToken field, and update the requested page token, repeating step 5 until the nextPageToken not longer advances.
I used http://www.jsonquerytool.com/ to paste the data from notepad++ and to query out and find the relevant fileids.
Querying the data for IDs (for each of the results returned):
Paste the JSON into the JSON section
Select JSPath Query Type
Use this query to double-check document name ".changes{.file.trashed === true}.file.name"
Use this query to get ID list to keep ".changes{.file.trashed === true}.file.id"
Copy Results to Notepad++
Permanent Deleting
If you have a small amount of FileIDs (< 100), you may wish to manually run them through https://developers.google.com/drive/v3/reference/files/delete, and optionally check them with https://developers.google.com/drive/v3/reference/files/get on each manual iteration.
Automation
Obviously, if you're dealing with more data it would make sense to write a Script or App to accomplish the above. But at least you can see it's possible.
I was able to setup exporting module using java and phantomjs. I am also able to see the image file getting generated in temp folder but it gets deleted after some time 30 secs to be exact. After going through API definition I found something called as async which caught my eyes. I tried playing around with this option but didnt worked. Image file gets deleted evertime, I want to permanently save this file on file system. Any pointers in this directions would be very helpful.
Thanks
Open AbstractPool.java file, in line 117 you can find:
Collection<File> oldFiles = FileUtils.listFiles(TempDir.outputDir.toFile(),filter, null);
for (File file : oldFiles) {
file.delete();
}
Simply delete that lines, and should be enough. Or above that lines you can set higher fixedRate.
In addition to Pawel's answer you can also change the time limit in the app-convert.properties file:
# Keep files in the temp folder for a certain retentionTime, defined in miliseconds
retentionTime = 30000
That is the default. So, 30s makes sense.