Why can't Cloud Function access metadata server in Vertex AI example? - google-cloud-functions

Question

I followed this Vertex AI tutorial. However, at the last step, as the Cloud Function calls the prediction endpoint, it gets this failure.
This means it could not even access the metadata server. I.e., is not a permissions failure (though I did check that the myproject#appspot.gserviceaccount.com service account does have Project Editor role as specified). It is also an error strictly in Functions and IAM, not in Vertex.AI or other ML systems.
What is going wrong here?
Function execution took 673 ms, finished with status code: 500
Prediction request failed: <class 'google.api_core.exceptions.ServiceUnavailable'>: 503 Getting metadata from plugin failed with error: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/myproject#appspot.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform from the Google Compute Enginemetadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/myproject#appspot.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\\n'", <google.auth.transport.requests._Response object at 0x3e095a9f4c50>)
google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/myproject#appspot.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform from the Google Compute Enginemetadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/myproject#appspot.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\\n'", <google.auth.transport.requests._Response object at 0x3e095a9f4c50>)
File "<string>", line 3, in raise_from
six.raise_from(new_exc, caught_exc)
File "/env/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 117, in refresh
self.refresh(request)
File "/env/local/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
self._request, context.method_name, context.service_url, headers
File "/env/local/lib/python3.7/site-packages/google/auth/transport/grpc.py", line 88, in _get_authorization_headers
callback(self._get_authorization_headers(context), None)
File "/env/local/lib/python3.7/site-packages/google/auth/transport/grpc.py", line 101, in __call__
context, _AuthMetadataPluginCallback(callback_state, callback))
File "/env/local/lib/python3.7/site-packages/grpc/_plugin_wrapping.py", line 78, in __call__
Traceback (most recent call last):
The above exception was the direct cause of the following exception:
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/myproject#appspot.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform from the Google Compute Enginemetadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/myproject#appspot.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\\n'", <google.auth.transport.requests._Response object at 0x3e095a9f4c50>)
response,
File "/env/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 187, in get
token_json = get(request, path, params=params)
File "/env/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 263, in get_service_account_token
request, service_account=self._service_account_email, scopes=scopes
File "/env/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 113, in refresh
Traceback (most recent call last):
AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x3e0961671dd0>" raised exception!