Vertex AI - Endpoint Call with JSON - Invalid JSON payload received - json

Question

I successfully trained and deployed a Tensorflow Recommender model on Vertex AI.
Everything is online and to predict the output. In the notebook I do:
loaded = tf.saved_model.load(path)
scores, titles = loaded(["doctor"])
That returns:
Recommendations: [b'Nelly & Monsieur Arnaud (1995)'
b'Three Lives and Only One Death (1996)' b'Critical Care (1997)']
That is, the payload (input for the neural network) must be ["doctor"]
Then I generate the JSON for payload (the error is here):
!echo {"\""instances"\"" : [{"\""input_1"\"" : {["\""doctor"\""]}}]} > instances0.json
And submit to the endpoint:
!curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-west1-aiplatform.googleapis.com/v1/projects/my_project/locations/us-west1/endpoints/123456789:predict \
-d #instances0.json > results.json
... as seen here: https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/vertex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb#scrollTo=35348dd21acd
However, when I use this payload, I get error 400:
code: 400
message: "Invalid JSON payload received. Expected an object key or }. s" : [{"input_1" : {["doctor"]}}]} ^"
status: "INVALID_ARGUMENT"
This below don't work either:
!echo {"inputs": {"input_1": ["doctor"]}} > instances0.json
Even with validated JSON Lint, it does not return the proper prediction.
In another Stackoverflow question is suggested to remove the " \ " in the payload, but this didn't work either.
Running:
!saved_model_cli show --dir /home/jupyter/model --all
I get:
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_1'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: serving_default_input_1:0
The given SavedModel SignatureDef contains the following output(s):
outputs['output_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 10)
name: StatefulPartitionedCall_1:0
outputs['output_2'] tensor_info:
dtype: DT_STRING
shape: (-1, 10)
name: StatefulPartitionedCall_1:1
Method name is: tensorflow/serving/predict
Concrete Functions:
Function Name: '__call__'
Option #1
Callable with:
Argument #1
input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: True
Option #2
Callable with:
Argument #1
queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: True
Option #3
Callable with:
Argument #1
input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: False
Option #4
Callable with:
Argument #1
queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: False
Function Name: '_default_save_signature'
Option #1
Callable with:
Argument #1
input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
Function Name: 'call_and_return_all_conditional_losses'
Option #1
Callable with:
Argument #1
input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: False
Option #2
Callable with:
Argument #1
queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: True
Option #3
Callable with:
Argument #1
queries: TensorSpec(shape=(None,), dtype=tf.string, name='queries')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: False
Option #4
Callable with:
Argument #1
input_1: TensorSpec(shape=(None,), dtype=tf.string, name='input_1')
Argument #2
DType: NoneType
Value: None
Argument #3
DType: bool
Value: True
The point is: I'm passing an array and I'm not sure if it must be in b64 format.
This Python code works, but returns a different result than expected:
import tensorflow as tf
import base64
from google.protobuf import json_format
from google.protobuf.struct_pb2 import Value
import numpy as np
from google.cloud import aiplatform
import os
vertex_model = tf.saved_model.load("gs://bucket/model")
serving_input = list(
vertex_model.signatures["serving_default"].structured_input_signature[1].keys()
)[0]
print("Serving input :", serving_input)
aip_endpoint_name = (
f"projects/my-project/locations/us-west1/endpoints/12345567"
)
endpoint = aiplatform.Endpoint(aip_endpoint_name)
def encode_input(input):
return base64.b64encode(np.array(input)).decode("utf-8")
instances_list = [{serving_input: {"b64": encode_input(np.array(["doctor"]))}}]
instances = [json_format.ParseDict(s, Value()) for s in instances_list]
results = endpoint.predict(instances=instances)
print(results.predictions[0]["output_2"])
['8 1/2 (1963)', 'Sword in the Stone, The (1963)', 'Much Ado About Nothing (1993)', 'Jumanji (1995)', 'As Good As It Gets (1997)', 'Age of Innocence, The (1993)', 'Double vie de Véronique, La (Double Life of Veronique, The) (1991)', 'Piano, The (1993)', 'Eat Drink Man Woman (1994)', 'Bullets Over Broadway (1994)']
Any ideas on how to fix / encode the payload ?

Answer 1

I solved the issue. The problem was base64 encoding. This is working perfectly:
def encode_64(input):
message = input
message_bytes = message.encode('ascii')
base64_bytes = base64.b64encode(message_bytes)
base64_message = base64_bytes.decode('ascii')
return base64_message
instances_list = [{serving_input: {"b64": encode_64("doctor")}}]
instances = [json_format.ParseDict(s, Value()) for s in instances_list]
results = endpoint.predict(instances=instances)
print(results.predictions[0]["output_2"][:3])
['Nelly & Monsieur Arnaud (1995)', 'Three Lives and Only One Death (1996)', 'Critical Care (1997)']