I have a scikit-learn model successfully trained and loaded onto Vertex AI, but I can't seem to do batch prediction with jsonl. I've tried using these formats with jsonl:
{"dense_input": [1, 2, 3, ...]}
{"dense_input": [4, 5, 6, ...]}
and
{"val_1": 1, "val_2": 2, ...}
{"val_1": 4, "val_2": 5, ...}
but I get this error for both:
('Post request fails. Cannot get predictions. Error: Predictions are not in the response. Got: {"error": "Prediction failed: Exception during sklearn prediction: float() argument must be a string or a number, not 'dict'"}.', 2)
I've tried batch prediction using a CSV file and it works fine, but I'm having difficulty with the jsonl file. Does anyone know what's the problem? Thanks!
Related
I use kafka topic to receive message from mysql database.I need to write python code to push the data in json format from mysql to kafka topic.My requirement is to get the output in json format but not in raw strings.
Below is the python code to dump the mysql table data to kafka topic in json format.
Code:
connection = mysql.connector.connect(host='xyz.us-east-1.rds.amazonaws.com', database='testdb',user='stdnt', password='pssw123')
cursor=connection.cursor()
statement='SELECT * FROM patients_vital_info'
cursor.execute(statement)
data=cursor.fetchall()
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
api_version=(0,11,5),value_serializer=lambda x:
json.dumps(x).encode('utf-8'))
for i in data:
producer.send('test',i)
sleep(1)
Output from kafka topic in raw string format:
[3, 69, 175]
[4, 68, 171]
[5, 72, 177]
[1, 78, 162]
[2, 66, 157]
[3, 72, 156]
The output should be pushed in json format while writing the message to kafka queue.
Expected output:
{"bp":140,"heartBeat":73,"Customerid":1}
cursor.fetchall() returns a row iterator, not a dictionary with key-value pairs of column to value. Your data is also, correctly, a JSON array
You'd need to build the JSON yourself if you want to include column names or use Kafka Connect JDBC source / Debezium rather than Python to do exactly what you're looking for
I tried to run ResNet50 computation graph provided in ZooModel of deeplearning4j but got following error while calling init() method of resNet. I have 10 classes and each image is 64*64.
Please suggest the correct way of initializing.
Code
ResNet50Builder builder = ResNet50.builder();
ResNet50 resNet = builder.numClasses(10).cacheMode(CacheMode.NONE)
.workspaceMode(WorkspaceMode.NONE).inputShape(new int[] {3, 64, 64}).build();
ComputationGraph computationGraph = resNet.init();
Exception Stack Trace:
Exception in thread "main" org.deeplearning4j.exception.DL4JInvalidConfigException: Invalid configuration for layer (idx=77, name=res4a_branch2b, type=ConvolutionLayer) for height dimension: Invalid input configuration for kernel height. Require 0 < kH <= inHeight + 2*padH; got (kH=3, inHeight=2, padH=0)
Input type = InputTypeConvolutional(h=2,w=2,c=256), kernel = [3, 3], strides = [1, 1], padding = [0, 0], layer size (output channels) = 256, convolution mode = Same
at org.deeplearning4j.nn.conf.layers.InputTypeUtil.getOutputTypeCnnLayers(InputTypeUtil.java:327)
at org.deeplearning4j.nn.conf.layers.ConvolutionLayer.getOutputType(ConvolutionLayer.java:191)
at org.deeplearning4j.nn.conf.graph.LayerVertex.getOutputType(LayerVertex.java:131)
at org.deeplearning4j.nn.conf.ComputationGraphConfiguration.getLayerActivationTypes(ComputationGraphConfiguration.java:536)
at org.deeplearning4j.nn.conf.ComputationGraphConfiguration.addPreProcessors(ComputationGraphConfiguration.java:449)
at org.deeplearning4j.nn.conf.ComputationGraphConfiguration$GraphBuilder.build(ComputationGraphConfiguration.java:1201)
at org.deeplearning4j.zoo.model.ResNet50.init(ResNet50.java:91)
at org.deeplearning4j.examples.convolution.ResNet.run(ResNet.java:145)
at org.deeplearning4j.examples.convolution.ResNet.main(ResNet.java:306)
I got this answer from other forum (https://community.konduit.ai/t/error-while-running-resnet50-zoomodel-example/417)
The image is too small, which results in filters being too large after the first few constitutional layers.
I have a MySQL database where I have Point type location data and a Django (Django Rest Framework) backend where I am trying to retrieve that data. If I try to get that location data from phpMyAdmin the returned location is something like this POINT(23.89826 90.267535). In my Django backend however, I get a bytes as the returned location. The returned value is something like this
b'\x00\x00\x00\x00\x01\x01\x00\x00\x00\x12N\x0b^\xf4\xe57#C\xe2\x1eK\x1f\x91V#'
The database uses utf8mb4_unicode_ci collation.
If I try to convert the returned bytes to a string with .decode('utf-8') I get UnicodeDecodeError
>>> s = b'\x00\x00\x00\x00\x01\x01\x00\x00\x00\x12N\x0b^\xf4\xe57#C\xe2\x1eK\x1f\x91V#'
>>> s.decode('utf-8')
Traceback (most recent call last):
File "<console>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 13: invalid continuation byte
I get the same bytes array even if I perform a raw query from Django with the MySQL function St_AsGeoJson(location).
I then tried geojson. When I feed that bytes to geojson.Point() I get a geojson back but instead of 2 floats the coordinates array consists 25 integer values.
>>> s = b'\x00\x00\x00\x00\x01\x01\x00\x00\x00\x12N\x0b^\xf4\xe57#C\xe2\x1eK\x1f\x91V#'
>>> geojson.Point(s)
{"coordinates": [0, 0, 0, 0, 1, 1, 0, 0, 0, 18, 78, 11, 94, 244, 229, 55, 64, 67, 226, 30, 75, 31, 145, 86, 64], "type": "Point"}
How can I retrieve the Point data from the bytes or this geojson?
I had this problem because I was using plain Django and Django models doesn't have a field type that deals with Geo data. I was using a CharField with a max_length=255 and then tried to parse whatever that CharField retrieved from the database. I have solved the problem by using GeoDjango and Django REST Framework GIS. Django REST Framework GIS is not necessary. I used it because I am using Django REST Framework and it outputs the Geo data in a nice format.
Steps were to
Install GDAL(Geospatial Data Abstraction Library)
sudo apt-get install gdal-bin
sudo apt-get install python3-gdal
Add django.contrib.gis and rest_framework_gis to settings.INSTALLED_APPS
Set GDAL_LIBRARY_PATH in settings, in my case it's GDAL_LIBRARY_PATH = os.getenv('GDAL_LIBRARY_PATH')
Update model import from from django.db import models to from django.contrib.gis.db import models
Update the model to use a Geo field. More: https://docs.djangoproject.com/en/2.1/ref/contrib/gis/model-api/
Links
https://docs.djangoproject.com/en/2.1/ref/contrib/gis/
https://github.com/djangonauts/django-rest-framework-gis
https://github.com/domlysz/BlenderGIS/wiki/How-to-install-GDAL
i am basically trying to build a deep model which consists of many convolution2d layers followed by maxpooling 2d as follows :
model.add(Convolution2D(128, 54, 7, input_shape=(1, 54, 180)))
model.add(MaxPooling2D(pool_size=(1, 3)))
model.add(Convolution2D(128, 1, 7))
model.add(MaxPooling2D(pool_size=(1, 3)))
However , i am getting the following error :
File
"/home/user/anaconda2/lib/python2.7/site-packages/keras/engine/training.py",
line 100, in standardize_input_data
str(array.shape)) Exception: Error when checking model input: expected convolution2d_input_1 to have 4 dimensions, but got array
with shape (8000, 180, 54)
Blockquote
But i am following the (samples, channels, rows, cols) norm. Why is this happening ?
It seems like your input data has the wrong shape. You should print out the shape of the data that you are feeding into the network.
It seems like your array are gray input images and they normally only use 2 dimensions because they only have 1 channel. Therefore the np array is ordered without the third dimension. Normally you have to add this by using np.reshape or allocating your array in another way. When I get an error message like yours i would try:
X # training data
X = np.transpose(X, (0, 2, 1))
X = np.reshape(X, (X.shape[0], 1, X.shape[1], X.shape[2]))
I have a php script that outputs a json array that looks like this...
[{"year":"2016","Month":"Apr","the_days":"16, 29, 30"},
{"year":"2016","Month":"May","the_days":"13, 27"},
{"year":"2016","Month":"Jun","the_days":"10, 11, 24"},
{"year":"2016","Month":"Jul","the_days":"08, 22, 23"},
{"year":"2016","Month":"Aug","the_days":"06, 20"},
{"year":"2016","Month":"Sep","the_days":"02, 03, 16, 17, 30"},
{"year":"2016","Month":"Oct","the_days":"01, 14, 15, 29"},
{"year":"2016","Month":"Nov","the_days":"25"},
{"year":"2016","Month":"Dec","the_days":"09, 10, 23, 24"}]
I'm trying to put together some Python that will (eventually) output something like....
Apr: 16, 29, 30
May: 13, 27
//etc
...but I'm not having any luck pulling the array out.
This is code that I'm using in Python3 (that I've pulled together from other Stack questions that I've searched for).
import urllib.request
import json
response = urllib.request.urlopen('http://www.captainobviousobviously.co.uk/private/Apijson.php')
content = response.read()
data = json.load(content.decode('utf-8'))
print(data)
This is the error that I'm getting...
Traceback (most recent call last):
File "/home/pi/Python/availableDates.py", line 6, in <module>
data = json.load(content.decode('utf-8'))
File "/usr/lib/python3.4/json/__init__.py", line 265, in load
return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'
I'm not really sure how to fix it.
Replace
data = json.load(content.decode('utf-8'))
with
data = json.loads(content.decode('utf-8'))
'load' is for files and 'loads' for strings.
Refer What is the difference between json.dumps and json.load?.
As for the code for your problem
for i in data:
print (str(i['Month'])+":"+str(i['the_days']))
Use json.loads instead. load is for loading from a stream, such as a file, whereas loads loads from a string.
data = json.loads(content.decode('utf-8'))
From the Python documentation:
json.load
Deserialize fp (a .read()-supporting file-like object containing a JSON document) to a Python object using this conversion table.
A string isn't a "file-like object", which is why you get your error - the JSON is trying to call .read on the string, but that doesn't exist.
You need to use json.loads(<json str>). If you want you can do the following
content = response.read().decode()
data = json.loads(content)
for d in data:
print(d["Month"], d["the_days"], sep=":")