I am training a bert model using a TPU vm on GCP.
I want to use my bucket as the Datasets library Cache filepath. I have followed instructions from
https://cloud.google.com/tpu/docs/tutorials/bert-2.x and set my bucket link in the HF_DATASETS_CACHE environment variable.
But I am getting this error : Exception in device=TPU:0: [Errno 2] No such file or directory: 'gs:/darijabert_bucket/datasets_Cache
my HF_DATASETS_CACHE is set to gs://darijabert_bucket/datasets_Cache
Thanks
The error seems to suggest that the specified bucket does not exist. Does the bucket gs:/darijabert_bucket/datasets_Cache exist in your project and in the correct region? You can double check to see if it exists by checking it in the GCP console.
Also, make sure that the bucket exists in the same region as your TPU VM, as the tutorial states:
The bucket location must be in the same region as your Compute Engine (VM) and your Cloud TPU node.
Related
I'm trying to create a Cloud Function that access to a website and download CSV file to Cloud Storage.
I managed to access the site using headless-chrominium and chromedriver.
On my local environment I can set up the download directory like below
options.add_experimental_option("prefs", {
"download.default_directory": download_dir,
"plugins.always_open_pdf_externally": True
})
where download_dir is like "/usr/USERID/tmp/"
How in Cloud Function could I assigned the value so that it points to the right Cloud Storage?
As I understand, a GCS bucket cannot be mounted as a local drive in runtime environment used for cloud functions.
Thus, you might need to download the source csv file into the cloud function memory and save it, for example, as a file in the "/tmp" directory.
Then, you can upload it from that location into a GCS bucket. A more detailed explanation how to upload - is provided here: Move file from /tmp folder to Google Cloud Storage bucket
Note: cloud functions have some restrictions - i.e. memory and timeout. Make sure that you allocated (during deployment) enough memory and time to process your csv files.
In addition, make sure that a service account, which is used by your cloud function, has relevant IAM roles for the GCS bucket under discussion.
I had always successfully created new environment by clone environment from a healthy running environment. But suddenly start get failures since today. Is there are limitation for clone environment for one account? Can you please give me some comments on this error?
This issue is caued by insufficient resources. When the environment is created, a log bucket for logging needs to be created in S3. But the S3 bucket already has the maximum number of buckets and not allowed new bucket created.
Keep getting this error:
Error, failed to create instance group-database-instance: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission
I have added a service account with editor permissions to use all gcp resources and added directed terraform to a credentials file generated.
Would this be an error in the code or something else?
Based on the error message you have provided and the task you would like to accomplish, it would seem that you might need to add a scope when creating your instance.
To use the Google Kubernetes Engine API for a GCE virtual machines, you will need to add the Cloud Platform Scope ("https://www.googleapis.com/auth/cloud-platform") to your VM when it is created.
Additionally, if you are using the gcloud command-line, you can follow along with something like:
gcloud compute instances create NAME --scopes=https://www.googleapis.com/auth/cloud-platform
If you are using the Cloud Console UI, when you are creating a VM instance, look for the "Identity and API access" section, and select "Allow full access to all Cloud APIs".
I want some suggestion on Upload Image file in S3 bucket using Lambda function.I am able to create bucket using lambda function but unable to upload file to S3 using Lambda function. It is possible? can we upload local system files(image,text etc) files to S3 bucket using lambda?.
when I am trying upload file using C:\users\images.jpg to S3 using Lambda function its showing me error ..Error: ENOENT, no such file or directory 'C:\Users\Images'.
Please suggest.
Thanks
You have to imagine where your code is running.
If you have a desktop application, you can access to local files such as C:\users\images.jpg becasue the process is has access to the file system.
Your lambda functions are maintained by AWS and they run on Amazon's infrastructure.
Also in general you have to design your functions stateless:
Local file system access, child processes, and similar artifacts may
not extend beyond the lifetime of the request, and any persistent
state should be stored in Amazon S3, Amazon DynamoDB, or another
Internet-available storage service.
Reference: AWS Lambda FAQs
So in your case I'd upload everything to S3 first, or create a background process that does this periodically. That way you can access them via Lambda functions but not directly from your local file system.
I have error when I run google cloud command, this is the error message which I get
$ gcloud compute instances list
NAME ZONE MACHINE_TYPE INTERNAL_IP EXTERNAL_IP STATUS
ERROR: (gcloud.compute.instances.list) Some requests did not succeed:
- Access Not Configured. The API is not enabled for your project, or there is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your configuration.
I have two machine running and I already updated to new version.
I don't know if you are encountering this problem, but if you specify the project name instead of the project ID when you do "gcloud config set project " then you will currently get the "Access Not Configured" error. I've pointed it out on the #gcloud IRC so hopefully it gets fixed. There may be other issues like this so it is best to ensure your parameters are sane.
If your project hasn't been marked for abuse and/or deletion, you have to enable 'Google Compute Engine' API in the Developers Console to solve the problem.
Configure your project using following command
gcloud config set project <project-id>
This is the exact API we need to enable in order to get rid of this error
Compute Engine API