Unstable number of listed files by Google Drive Api after deleting a bunch of files and shared folders - google-drive-api

Question

I'm testing scripts for a client. For this I created a ~10k files which I uploaded to a test folder using the web UI. Then I trashed and then deleted this folder.
Then I added a shared folder from the client and listed all the files using the /v3/files with the proper query parameters to include files from other drives.
I noticed my script was not functioning well due to a lot of 404 responses. It turns out, deleting 10k files is not instantaneous for Google Drive, at least from the API's point of views. The listed files also included the listed files that I've just deleted who were then deleted later.
From what I've seen, Google Drive is able to process about 200 files/s.
I could just wait, but then I found another problem after I deleted the shared folder and replaced it with another shared folder from my client, all of which had ten of thousand of files. As expected, it took some time to see the number of files to go down. But then I saw the number increasing slowly then decreasing.
I suspect this is the adding of a folder that increases the number at the same time the deleting of the other decreases the files count but I am not sure.
Am I the only one who experienced this? Is there something in the API that I've missed that could mitigate this or at least could tell me when Google Drive has ended processing all operations?
Edit: steps to reproduce:
Code that I used to create a bunch of files:
#!/usr/bin/env bash
#create_lots_of_files.sh
mkdir -p lots_of_files
cd lots_of_files
for i in $(seq 10000); do
FOLDER=$(("$i"%10))
FILE="file_$i".txt
mkdir -p "$FOLDER"
echo "$FILE" > "$FOLDER/$FILE"
done
Then upload this folder to your drive. Grab a coffee this will take time.
Code to fetch the file ids using the api:
#!/usr/bin/env bash
# list_ids.sh <output file path> <bearer token>
set -e
# shellcheck disable=SC2128
SCRIPTDIR="$(dirname "$(realpath "$BASH_SOURCE")")"
PAGE_SIZE=1000
if [[ -z $1 ]]; then
echo "First argument must specify a path to store the files ids"
exit 1
fi
IDS_FILE="$1"
if [[ -z $2 ]]; then
echo "Second argument must be the bearer token"
exit 1
fi
ACCESS_TOKEN="$2"
if ! jq -h &> /dev/null; then
echo "error: need to install jq: sudo apt-get install jq"
exit 1
fi
cd "$SCRIPTDIR"
BASE_QUERY_STRING="https://www.googleapis.com/drive/v3/files\
?corpora=allDrives\
&includeItemsFromAllDrives=true\
&supportsAllDrives=true\
&pageSize=$PAGE_SIZE\
"
true > "$IDS_FILE"
while true; do
# If pageToken is empty then it defaults to the first page
QUERY_STRING="$BASE_QUERY_STRING&pageToken=$NEXT_PAGE_TOKEN"
RESPONSE="$(curl \
--silent \
--fail \
-H 'GData-Version: 3.0' \
-H "Authorization: Bearer $ACCESS_TOKEN" \
--request GET \
"$QUERY_STRING" \
)"
jq -r '.files | map(select(.mimeType != "application/vnd.google-apps.folder")) | .[].id' <<<"$RESPONSE" | tee -a "$IDS_FILE"
NEXT_PAGE_TOKEN="$(jq -r '.nextPageToken' <<< "$RESPONSE")"
if [[ -z "$NEXT_PAGE_TOKEN" || "$NEXT_PAGE_TOKEN" = 'null' ]]; then
break
fi
done
Keep track of the number of files with:
while true; do date; ./list_ids.sh ids.txt '<bearer token>' | wc -l; sleep 5; done
Delete lots_of_files on your drive and watch the files count.

Answer 1

I can tell you, you are not the first one. In my experience, this behaviour you are reporting is expected, you will see, changes need to be replicated across all Google Workspace servers, this has a delay usually referred to as 'propagation', as mentioned in this Help Center article https://support.google.com/drive/answer/7166529.
If you share or unshare folders with a lot of files or subfolders, it might take time before all permissions change. If you change a lot of edit or view permissions at once, it might take time before you see the changes.
Although the task you are doing is different to just sharing files, due to the high volume of files and folders you are working with and the fact that you are working with shared folders you will experience a delay. As outlined in this other Help Center article https://support.google.com/a/answer/7514107, you can expect changes to be fully applied within 24 hours.
I have assisted multiple data migrations with Google Workspace admins and this is also expected when working with large amounts of data.