Lambda Function not able to invoke sagemaker endpoint - json

predict = [0.1,0.2]
payload = {
"instances": [
{
"features": predict
}
]
}
response = linear_regressor.predict(json.dumps(payload))
predictions = json.loads(response)
print(json.dumps(predictions, indent=2))
The above code is able to invoke the endpoint which is the linear-learner endpoint and give the below result.
{
"predictions": [
{
"score": 0.13421717286109924
}
]
}
But when I try to invoke the endpoint using below lambda function,
import json
import io
import boto3
client = boto3.client('runtime.sagemaker')
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
#data = json.loads(json.dumps(event))
#payload = data['data']
#print(payload)
response = client.invoke_endpoint(EndpointName='linear-learner-2019-12-12-16-xx-xx-xxx',
ContentType='application/json',
Body=json.dumps(event))
return response
I get the following error,
An error occurred during JSON serialization of response:
<botocore.response.StreamingBody object at 0x7fc941e7e828> is not JSON serializable
The input for the lambda function is the same,
{
"instances": [
{
"features": [
0.1,
0.2
]
}
]
}
Not sure what is causing the issue here. Any help would be appreciated, Thanks in advance.

Related

Looping through groovy object(List) using each{it} and pass the the elements into a json payload in Jenkins

I have a list containing the name of workspaces in groovy Jenkinsfile. I wrote an each() loop to iterate through the list and use the names in the endpoint below to get the workspace ID from the api response.
def getWorkspaceId() {
def result = []
Listworkspace.each{
def response = httpRequest(
customHeaders: [
[ name: "Authorization", value: "Bearer " + env.BEARER_TOKEN ],
[ name: "Content-Type", value: "application/vnd.api+json" ]
],
url: "https://app.terraform.io/api/v2/organizations/${TF_ORGNAME}/workspaces/$it
)
def data = new JsonSlurper().parseText(response.content)
println ("Workspace Id: " + data.data.id)
result << data.data.id
}
return result
}
After getting the IDs, I want to pass them as part of a json payload.
def buildPayload() {
def workspaceID = new JsonSlurper().parseText(getWorkspaceId())
workspaceID.each{
def payload = """
{
"data": {
"attributes": {
"is-destroy":false,
"message":
},
"type":"runs",
"relationships": {
"workspace": {
"data": [
{"id": "$it", "type": "workspaces"}
]
}
}
}
}
}
"""
return payload
}
Is there a way I can iterate through the list of IDs returned and append each json object for the key "data" after iteration. See the code below
"relationships": {
"workspace": {
"data": [
{"id": "id1", "type": "workspaces"},
{"id": "id2", "type": "workspaces"},
{"id": "id3", "type": "workspaces"}
]
When making the api call, it throws a 400 Bad request error. I tried to print the payload and I found out it attaches the whole list of IDs to the payload.
Any suggestion will be greatly appreciated. Thank you.
def buildPayload() {
def workspaceID = new JsonSlurper().parseText(getWorkspaceId())
workspaceID.each{
def payload = """
{
"data": {
"attributes": {
"is-destroy":false,
"message":
},
"type":"runs",
"relationships": {
"workspace": {
"data": [
[id1, id2, id3]
]
}
}
}
}
}
"""
return payload
}
I'd recommend using the JsonOutput class to make your life easier. In essence, as long as your getWorkspaceId() method is returning a list of ids, you can do something like this:
import groovy.json.JsonOutput
def buildPayload(def ids) {
def payload = [
data: [
attributes: [
"is-destroy": false,
"message" : "",
],
type: "runs",
relationships: [
workspace: [
data: ids.collect {
return [id: it, type: "workspaces"]
}
]
]
]
]
return JsonOutput.toJson(payload)
}
This will take each id in your ids list and build a map where each id is identified by its number and the type: workspaces key pair. This is all then included in the larger payload.

Reformat JSON with Groovy. Remove child node whilst keeping the contents

I am trying to reformat JSON to remove child nodes from some JSON objects whilst keeping whats inside the child node. (unsure how to explain, like removing the title node). In this example i am trying to have a JSONarray of groups without the UserGroup child node (whilst keeping each UserGroups content intact). Example JSON
{
"groups":[
{
"UserGroup":{
"integrationKey":"0000073807",
"uid":"0000073807"
},
"UserGroup":{
"integrationKey":"0000073810",
"uid":"0000073810"
}
}
]
}
What I'd like after processing
{
"groups":[
{
"integrationKey":"0000073807",
"uid":"0000073807"
},
{
"integrationKey":"0000073810",
"uid":"0000073810"
}
]
}
For the JSON with no duplicated keys, which formats the data, removing child keys may look like
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
def inputJson = '''
{
"groups": [{
"UserGroup1": {
"integrationKey": "0000073807",
"uid": "0000073807"
},
"UserGroup2": {
"integrationKey": "0000073810",
"uid": "0000073810"
}
}
]
}
'''
Map parsed = new JsonSlurper().parseText(inputJson) as Map
parsed.each {
parsed[it.key] = it.value*.values().flatten()
}
println JsonOutput.toJson(parsed)
This code will print
{"groups":[{"integrationKey":"0000073807","uid":"0000073807"},{"integrationKey":"0000073810","uid":"0000073810"}]}
Handle duplicated key
But your provided input example contains 2 items with UserGroup name. It's not common for JSON and most parsers just take the last mentioned value for the key. To handle this you can use below dependency
<dependency>
<groupId>net.sf.json-lib</groupId>
<artifactId>json-lib</artifactId>
<version>2.4</version>
<classifier>jdk15</classifier>
</dependency>
And modify the above script to
import net.sf.json.JSONObject
// ...
ipuptJson = JSONObject.fromObject(inputJson).toString()
Map parsed = new JsonSlurper().parseText(inputJson) as Map
parsed.each {
parsed[it.key] = it.value*.values().flatten()
}
println JsonOutput.toJson(parsed)
Starting from terminology, you are not really modifiyng the json itself. In fact you can modify the in-memory representation of it exposed as a mixture of Lists, Maps and primitive object types.
The simplest way to do what you want would be:
import groovy.json.*
def json = new JsonSlurper().parseText '''
{
"groups": [{
"UserGroup1": {
"integrationKey": "0000073807",
"uid": "0000073807"
},
"UserGroup2": {
"integrationKey": "0000073810",
"uid": "0000073810"
}
}
]
}
'''
def modified = [ groups:json.groups*.values().flatten() ]
println JsonOutput.prettyPrint( JsonOutput.toJson( modified ) )
prints:
{
"groups": [
{
"integrationKey": "0000073807",
"uid": "0000073807"
},
{
"integrationKey": "0000073810",
"uid": "0000073810"
}
]
}

TypeError: list indices must be integers or slices, not str JSON Scrapy

I was scraping a JSON response but getting the following error
values = resp['acf']
TypeError: list indices must be integers or slices, not str
I am not sure where did I do wrong.
Your response is highly appreciated.
# -*- coding: utf-8 -*-
import scrapy
import json
class MainSpider(scrapy.Spider):
name = 'main'
start_urls = 'https://chamber.vinylagency.com/wp-json/wp/v2/directory?industry-type=547&per_page=100'
def parse(self, response):
resp = json.loads(response.body)
values = resp['acf']
for value in values:
name = value['OrgName']
yield {
"Name": name,
}
The exception is raised because the response is a list of objects and you are trying to access it as a dict directly.
Here is a sample of the response:
[
{
"id": 33286,
"date": "2020-05-09T02:38:47",
"date_gmt": "2020-05-09T02:38:47",
"guid":
...
},
{
"id": 32954,
"date": "2020-05-09T02:38:22",
"date_gmt": "2020-05-09T02:38:22",
"guid":
...
}
]
You probably want to parse like this:
def parse(self, response):
resp = json.loads(response.body)
for value in values:
name = value['acf']['OrgName']
yield {
"Name": name,
}

Unexpected character (g) at position 0 when trying to parse json - HttpResponseDecorator

Whenever I try to execute below script I keep getting error, saying unexpected character (g).
Basically I want to be able to parse the json response and be able to get the upstream job name from it.
Script:
#Grapes([
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.1'),
#Grab(group='commons-collections', module='commons-collections', version='3.2.1'),
#Grab(group='org.jsoup', module='jsoup', version='1.10.2'),
#Grab(group='org.json', module='json', version='20190722'),
#Grab(group='com.googlecode.json-simple', module='json-simple', version='1.1.1')
])
import static groovyx.net.http.ContentType.*
import groovyx.net.http.HttpResponseException
import groovyx.net.http.RESTClient
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
import java.net.*
import java.util.*
import org.json.simple.*
import org.json.simple.parser.JSONParser;
def getRestClient(){
String jenkinsUrl="http://somedomainname:8080"
def restClient = new RESTClient(jenkinsUrl)
return restClient
}
def getJobsInfo(String jobname,RESTClient restClient){
def requrl= '/job/'+jobname+'/lastBuild/api/json/?pretty=true'
def response = restClient.get( path : requrl)
return response
}
def writeToPropertyFile(){
def jsonResponseObject = getJobsInfo("sampleJobName",getRestClient())
println "\n\n\n\n\n Json String :---- "+ jsonResponseObject.toString()
JSONParser jsonParser = new JSONParser();
JSONObject jsonObject = (JSONObject) jsonParser.parse(jsonResponseObject.toString());
JSONArray upstreamJobInfoArray = jsonObject.getJSONArray("causes");
for (int i = 0; i < upstreamJobInfoArray.length(); i++) {
JSONObject jobCauses = upstreamJobInfoArray.getJSONObject(i);
String upstreamProjectName = jobCauses.getString("upstreamProject");
println upstreamProjectName
}
}
writeToPropertyFile()
Error :
Json String :---- groovyx.net.http.HttpResponseDecorator#6b063470
Caught: Unexpected character (g) at position 0.
Unexpected character (g) at position 0.
at org.json.simple.parser.Yylex.yylex(Yylex.java:610)
at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
at org.json.simple.parser.JSONParser$parse.call(Unknown Source)
at getUpstreamJob.writeToPropertyFile(getUpstreamJob.groovy:39)
at getUpstreamJob.run(getUpstreamJob.groovy:50)
EDIT 1 : START
JSON response that I am trying to parse :
{
"_class": "hudson.model.FreeStyleBuild",
"actions": [
{
"_class": "hudson.model.CauseAction",
"causes": [
{
"_class": "hudson.model.Cause$UpstreamCause",
"shortDescription": "Started by upstream project \"sampleJobName\" build number 712",
"upstreamBuild": 712,
"upstreamProject": "sampleJobName",
"upstreamUrl": "job/sampleJobName/"
},
{
"_class": "hudson.model.Cause$UserIdCause",
"shortDescription": "Started by user Malick, Asif",
"userId": "asifma00",
"userName": "Malick, Asif"
},
{
"_class": "com.sonyericsson.rebuild.RebuildCause",
"shortDescription": "Rebuilds build #300",
"upstreamBuild": 300,
"upstreamProject": "sampleJobName",
"upstreamUrl": "view/ABCProjectView/job/sampleJobName/"
}
]
},
{
"_class": "hudson.model.ParametersAction",
"parameters": [
{
"_class": "hudson.model.StringParameterValue",
"name": "SNAPSHOTNAME",
"value": "ABCDE_12121.2000-2121212121212"
},
{
"_class": "hudson.model.StringParameterValue",
"name": "BUILD_LABEL",
"value": "ABCDE_12121.2000"
}
]
},
{},
{},
{},
{},
{
"_class": "hudson.plugins.parameterizedtrigger.BuildInfoExporterAction"
},
{},
{},
{},
{}
],
"artifacts": [],
"building": false,
"description": null,
"displayName": "#301",
"duration": 1199238,
"estimatedDuration": 1194905,
"executor": null,
"fullDisplayName": "sampleJobName #301",
"id": "301",
"keepLog": false,
"number": 301,
"queueId": 189076,
"result": "SUCCESS",
"timestamp": 1583500786857,
"url": "http://somedomainname:8080/job/sampleJobName/301/",
"builtOn": "Server12345",
"changeSet": {
"_class": "hudson.scm.EmptyChangeLogSet",
"items": [],
"kind": null
},
"culprits": []
}
EDIT 1 : END
I have tried looking at several Stack Overflow issues, but still haven't been able to resolve it.
Please guide.
It's because you're not getting the actual JSON string, you're getting the toString() output on a groovyx.net.http.HttpResponseDecorator class.
You can see it printing it out here (it's easy to miss though... I missed it the first look 🙂):
Json String :---- groovyx.net.http.HttpResponseDecorator#6b063470
I think you need to call jsonResponseObject.data to get the parsed data, and it might even return you a Map parsed form the JSON (so you can get rid of all your JSONObject code). Test it by changing to:
def data = jsonResponseObject.data
println "\n\n\n\n\n Json ${data.getClass().name} :---- $data" jsonResponseObject.toString()
If it is a java.util.Map, you can simplify your second part from:
JSONParser jsonParser = new JSONParser();
JSONObject jsonObject = (JSONObject) jsonParser.parse(jsonResponseObject.toString());
JSONArray upstreamJobInfoArray = jsonObject.getJSONArray("causes");
for (int i = 0; i < upstreamJobInfoArray.length(); i++) {
JSONObject jobCauses = upstreamJobInfoArray.getJSONObject(i);
String upstreamProjectName = jobCauses.getString("upstreamProject");
println upstreamProjectName
}
To (using data from above):
data.actions.causes.upstreamProject.flatten().each { println it }

How to add Json keys to an array list in Django REST framework

I'm using Django REST framework to make an API.
For my serialized models I get a response looks like this:
[GET] http://12.0.0.1/0.5/barrios/?format=json
[
{
"name": "La Julia",
"zone": 1
},
{
"name": "La Floresta",
"zone": 2
}
]
But, what I want looks like this:
{
"barrios": {
"barrio": [
{
"name": "La Julia",
"zone": 1
},
{
"name": "La Floresta",
"zone": 2
}
]
}
}
Any thoughts?
You can override the list() method in your view to get the desired response.
class MyView(..):
def list(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
serializer = self.get_serializer(queryset, many=True)
serializer_data = serializer.data # get the default serialized representation
custom_data = {'barrios': {'barrio': serializer_data}} # custom representation
return Response(custom_data)