I've been using BigQuery for a while and load my data by fetching a CSV from an http address, uploading this to Google Drive using the Drive API, then attaching this to BigQuery using the BigQuery API.
I always specified auto-detect schema via the API and it has worked perfectly on a cron until March 16, 2017.
On March 16 it stopped working. The CSV still loads to Google Drive fine, but BigQuery won't pick it up.
I started troubleshooting by attempting to load the same CSV manually using the BigQuery UI, and noticed something strange: using auto-detect schema seems to prevent the loading of the CSV, because when I enter the schema manually it loads fine.
I thought maybe some rogue data might be the problem, but auto-detect schema isn't working for me now even with incredibly basic test tables, like...
id name
1 Paul
2 Peter
Has anyone else found auto-detect schema suddenly stopped working.
Maybe some default setting has changed on the API?
I could not get it to work either from GDrive today - 23 mar.
Note: first time ever using BigQuery/Google Cloud Storage.
I had a large CSV of bus stops 134MB.
Tried uploading it to GDrive but couldnt get it to import to big Query.
Just Tried Google Cloud Storage Bucket and it worked ok.
Related
I'm working with GCP Bigquery and Its very common for me export some queries as CSV in my local computer. The thing is that a few weeks ago the exportation name of the tables change from a format like "results-YYYYMMDD-TIME" to "bquxjob_etc etc etc". Besides, since yesterday when I export it download automatically in my download folder, without asking where I want to save it.
On the other hand, when some value is null, came the word "null" as a result. Could it be related?
I have a macbook air, I'm using Google Chrome and yes, I have the option of automatically download off.
Any idea of What is going on or how can I solve it?
Thanks! :)
There could have been changes done internally to affect the things you have mentioned.
I can confirm that the filename of the CSV is the "bquxjob_etc etc etc".csv
Downloaded CSV file is automatically be downloaded to your browsers default download location. This was added to the documentation.
If you want naming convention "results-YYYYMMDD-TIME" when downloading, you can opt to downloading the data to Google Sheets.
The data downloaded to Google Sheets does not have the word "null".
Actual Query result:
Exported Query results to sheets:
As a workaround if you want to retain the "results-yyyy-mmdd-time" naming convention and no "null" values. I suggest to export query results to Google Sheets and download the data as CSV.
You can also send a feedback to the BigQuery team about the changes made.
To send a feedback with regards to the new UI:
Open any BigQuery documentation
Click "Send feedback"
Choose "BigQuery Product feedback"
Fill up the form with your feed back
I am currently working on a dictionary like app in android studio for freshmen in my school's robotics team. I used google spreadsheets as a database and until about a month ago it worked perfectly. one day I open my app and it fails to load the JSON so I try to open it on my web browser and it does not work anymore.
The process to make a JSON url out of a google spreadsheets used to be like this:
https://spreadsheets.google.com/feeds/cells/SPREADSHEET-ID/od6/public/basic?alt=json
In my case the complete url is:
https://spreadsheets.google.com/feeds/cells/1HljdyhGauam-1WYa_C1ho2KOQMUrbGUuR3s5OEU8XtE/od6/public/basic?alt=json
And the url opened a JSON containing all the information in the spreadsheet.
Due to this problem I consider switching to firebase but I am frustrated because it used to work and I do not want to learn an entire new database framework.
Thanks in advance!
I'm Android developer. I'm using Google Drive Java API to access files in hidden Application Data Folder. Since 14 May 2015, LIST method doesn't return newly created files. Also in the Web interface "Hidden data size" is 0.
Files are created correctly (response 200 OK, file created), but I can't list them.
I was using LIST method for 2 years and everything was fine until tomorrow. After few hours files are listed by LIST method, but I want access them immediately after create.
When I save files into visible data folder everything is fine, but problem is with hidden Application Data Folder.
Could you please help me?
Regards
This was a caching issue specifically for App Folder content that affected list queries.
We've rectified the issue so you should no longer see this issue. Thanks for reporting.
Writes were not affected.
I've developed an extension for Google Chrome that HEAVILY relies on Google Drive API (the extension is LBTimer, available in Google Chrome web store), to store data in the appfolder, using XMLHttpRequests
Since May, 13th 2015 I'm finding a problem when using the list method.
If I programmatically create a file in the appfolder, I receive the response: 200 OK and the file created. If then I use the list method to list the files in the appfolder, the file just created is not listed. It happened with several files yesterday. This morning, the files were listed normally, but with any file I create today it happens the same (correctly created but not listed).
Three screens follow: the 1st one is creating a test file in the appfolder using the extension's code. the image shows the server response (200 OK, file created). The second screen shows the list request (list all files whose title contains 'test', it should include the file just created). The third screen shows the response from the server (an empty items list).
There is a way to get them listed: If I create a file, it returns (among other data) the file Id. If I make a simple GET request for that Id, then it is listed from then on.
All other methods are working as expected (as usual), but the list method is giving me this problem since yesterday. Since there was no change in the extension's code, I assume there must have been a change in the API code.
Apparently, it was a caching issue in Google Drive, which has been resolved as of May 15th 2015: Google Drive developers community post
Yes, I noticed the same issue from today. I'm using Google Drive Java API to create files in hidden Application data folder (appfolder). Files are created correctly, but "Hidden data size" i 0. I noticed that my files appears after few hours! I reported this issue to Google several hours ago by Google Developers Console, but the issue still occurs. I think more users should do this to get their attention to this critical issue.
I'm developing an application using the Google drive api (java version). The application saves files on a Google drive mimicking a file system (i.e. has a folder tree). I started by using the files.list() method to retrieve all the existing files on the Google drive but the response got slower as the number of files increased (after a couple of hundred).
The java Google API hardcodes the response timeout to 20 seconds. I changed the code to load one folder at a time recursively instead (using files.list().setQ("'folderId' in parents) ). This method beats the timeout problem but it consistently misses about 2% of the files in my folders (the same files are missing each time). I can see those files through the Google drive web browser interface and even through the Google drive API if I search the file name directly files.list().setQ("title='filename'").
I'm assuming that the "in parents" search uses some inexact indexing which may only be updated periodically. I need a file listing that's more robust and accurate.
Any ideas?
could you utilize the Page mechanism to do multiple times of queries and each query just asks for a small mount of result ?