Tornado POST request not detecting json input as argument - json

I have written a service which takes a json as input. I am using the website hurl.it to send post requests to check. Below is my code snippet:
class BatchSemanticSimilarityHandler(tornado.web.RequestHandler):
def post(self):
self.set_header('Access-Control-Allow-Origin', '*')
self.set_header('Access-Control-Allow-Credentials', 'true')
self.set_header('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS')
self.set_header('Access-Control-Allow-Headers','Origin, Accept, Content-Type, X-Requested-With, X-CSRF-Token')
data = json.loads(self.request.body)
apikey = data["apikey"]
try:
UA = self.request.headers["User-Agent"]
except:
UA = "NA"
if bool(usercoll.find_one({"apikey":apikey})) == True:
sentence = data["sentence"]
sentence_array = data["sentence_array"]
n = data["num_of_results"]
if sentence is None or sentence_array is [] or apikey is None or n is None:
self.set_status(200)
output = {"error":[{"code":334,"message":"Bad Input data"}]}
misscoll.insert({"apitype":"batchsemanticsimilarity","timestamp":datetime.datetime.now(), "ip":self.request.remote_ip, "useragent":UA, "uri":self.request.uri,"apikey":apikey, "output":output, "input":{"s1":sentence,"s2":sentence_array}})
self.write(output)
return
results = nb.get_similar(sentence, sentence_array, apikey, n)
print "results is",results
output = {"similar_sentences": results, 'credits':'ParallelDots'}
hitscoll.insert({"apitype":"batchsemanticsimilarity","timestamp":datetime.datetime.now(), "ip":self.request.remote_ip, "useragent":UA, "uri":self.request.uri,"apikey":apikey, "output":output, "input":{"s1":sentence,"s2":sentence_array}})
self.write(output)
return
else:
rejectcoll.insert({"apitype":"batchsemanticsimilarity","apikey":apikey,"timestamp":datetime.datetime.now(), "ip":self.request.remote_ip, "useragent":UA, "url":self.request.uri})
self.write({"error":[{"code":333,"message": "Bad Authentication data"}]})
return
The json that I am giving as the body of the request is as below:
{
"sentence": "BJP leads in Bengaluru civic body`s poll, all eyes on JD(S)",
"sentence_array": [
"Narendra Modi is the prime minister",
"Sonia Gandhi runs Congress",
"Sachin is a good batsman"
],
"apikey": "DyMe1gSNhvMV1I1b20a7KARYIwuQX5GAQ",
"num_of_results": 2
}
I have verified on jsonlint that this is a valid JSON.
However while sending the request it gives me below error:
ValueError: No JSON object could be decoded
Can anyone please help me sort this out!!

The JSON object that you are passing in POST request is encoded into the url.
JSON library cannot read the encoded data.So you need to decode the url first.
Decoding of url can be done using urlparse library in python.so you need something like this.
post_data=urlparse.parse_qsl(self.request.body)
According to your need of final format to read there are various methods in urlparse.check this
or
As specified in the docs you can override a method to enable JSON parsing
def prepare(self):
if self.request.headers["Content-Type"].startswith("application/json"):
self.json_args = json.loads(self.request.body)
else:
self.json_args = None
check this

Related

What is the recommended and most performant API call to check if I have read permission on a dataset in Foundry?

On our Stack users by default have discoverer permissions on resources. I was surprised that this also gives users the ability to query the last_transaction_rid of the dataset (using catalog/datasets/<rid>/reverse-transactions2/<branch>), so this method to check if a user has access is not working.
What would be the recommended and most performant API call to check if I, with my current Foundry token, can read the actual content of a dataset? Note: I don't want to query the content, but just understand if I would have the permissions to do so.
Would Data Lineage App (Monocle) work for you?
Open it workspace/data-integration/monocle/ > Find the dataset(s) > top right dropdown ("Resource Type" -> "Permissions")
The API that powers it can be extracted by checking the Dev Tools > Network Tab if you want / need to automate it (I've done this in the past and worked well for my use case)
I have resorted back to calling the compass/resources API and checking if response['operations'] contains compass:view:
def get_dataset_details(api_base: str, dataset_path_or_rid: str, headers: dict) -> dict:
"""
Returns the resource information of a dataset
Args:
dataset_path_or_rid: The full path or rid to the dataset
Returns: (dict) the json response of the api
"""
if 'ri.foundry.main.dataset' in dataset_path_or_rid:
response = requests.get(f"{api_base}/compass/api/resources/{dataset_path_or_rid}",
headers=headers,
params={'decoration': 'path'})
else:
response = requests.get(f"{api_base}/compass/api/resources",
headers=headers,
params={
'path': dataset_path_or_rid,
'decoration': 'path'
})
response.raise_for_status()
if response.status_code != 200:
raise ValueError(f"Dataset {dataset_path_or_rid} not found; "
f"If you are sure your dataset_path is correct, "
f"check if your jwt token "
f"is still valid!")
return response.json()
details = get_dataset_details(...)
if 'compass:view' not in dataset_details['operations']:
raise ValueError("No compass:view access to dataset")

Not able to capture previous API response and append the response in URL of Post Request API

I am trying to POST API request where I am getting API response as
"d": {
"__metadata": {
"uri": "http://ev-qa02.zs.local/IncentiveManager/0002i1/wcf/v5.svc/InDataRequestCreators('9f31c6da-ec56-4360-8589-d21b6320f99b')",
"type": "ZSAssociates.Javelin.ETL.Rest.v5.InDataRequestCreator"
},
"ScenarioId": "9f31c6da-ec56-4360-8589-d21b6320f99b",
"CallbackUrl": "",
"DataExpiresOnUtc": "/Date(4103913600000)/",
"CreateScenarioIfMissing": false,
"AdapterId": "0fcbd8d2-f5cb-4e2a-bda8-bb37037b022d",
"InDataRequestIdOut": "eb36f8a9-5b7d-4835-88f6-4af67830c1e9",
"InDataRequestUrlOut": "/InDataRequests('eb36f8a9-5b7d-4835-88f6-4af67830c1e9')"
}
}
Now I am trying to hit another API request where my URL would be kind of
http://ev-qa02.zs.local/IncentiveManager/0002i1/WCF/V5.svc/InDataRequests('eb36f8a9-5b7d-4835-88f6-4af67830c1e9')/FileCreator
*InDataRequests('eb36f8a9-5b7d-4835-88f6-4af67830c1e9') This random number is generated from above response value "InDataRequestIdOut"
How can I append the URL taking previous API response and adding in my 2nd POST request.
I am not able to capture my response and used it in other API POST request? i would realy appreciate if you can help me here,been stuck in this issue since couple of days,I went through doc and examples too but couldn't resolve this.I have attached screenshot too.PostUrlFailureScreenshot
My main problem is line number 26 and 27 from eclipe screenshot
Scenario: Verify that JIM Idr request ofr Post
Given header Content-Type = 'Application/JSON'
And header Accept = 'Application/JSON'
And header Authorization = 'Basic
UUEwMl9JbmNlbnRpdmVNYW5hZ2VyXzAwMDJpMTpZWkxaRjlGclR1eWhlcVNJbXlkTlBR'
Given path 'InDataRequestCreators'
* def user =
"""
{
"ScenarioId":"9f31c6da-ec56-4360-8589-d21b6320f99b",
"AdapterId":"0fcbd8d2-f5cb-4e2a-bda8-bb37037b022d",
"DataExpiresOnUtc":"2100-01-18T00:00:00",
"CreateScenarioIfMissing":"false"
}
"""
And request user
When method post
Then status 201
* print 'the value of response is:', response
And def app = response
And path 'app.InDataRequestIdOut' + '/FileCreators'
* def body =
"""
{
"InDataRequestId": "1d6326a2-d25f-41d2-9303-8a6e6101efcc",
"ProcedureName": "",
"SourceWorkspacePath": ""
}
"""
And request body
When method post
Then status 201
First, it looks to me you are using the wrong Eclipse plugin for Cucumber, please refer to this issue and make sure: https://github.com/intuit/karate/issues/90
There are so many things you are doing wrong. For example it should be application/json (lowercase). There are many places you are mixing upper case and lower case in your above description, please take care.
And there is no way to understand how the URL is being set up, without this - I can't provide proper help.
You have a fundamental misunderstanding of how to use Karate expressions, for example this is just concatenating two strings:
And path 'app.InDataRequestIdOut' + '/FileCreators'
This may give you some hints, instead of the above:
When url baseUrl
And path "InDataRequests('" + response.InDataRequestIdOut + "')/FileCreator"
And is it FileCreator or FileCreators. You seem to be quite careless :(

Erlang Chicagoboss unable to get the correct JSON response

In my controller file I have a method that reads the incoming HTTP request, reads the user data from the Database, encodes the result in JSON (using jsx) and sends it in response.
sensorusersdetails('GET', []) ->
Headers = [{'Access-Control-Allow-Origin', "*"},
{'Access-Control-Allow-Methods', "GET, OPTIONS"},
{'Content-Type', "application/json"},
{'Access-Control-Allow-Headers', "X-Requested-With"},
{'Access-Control-Max-Age', "180"}],
Building = Req:query_param("bld"),
io:format("User Data request from Node.js server~n~p~n",
[Req:query_params()]),
{{Year,Month,Day},{_,_,_}} = erlang:localtime(),
StrDate = lists:flatten(io_lib:format("~4..0w-~2..0w-~2..0w",
[Year,Month,Day])),
BUserDataList = boss_db:find(sensoruser_data, [{building, 'equals', Building}]),
io:format("Current Users Data stored in the database: ~n~p~n",[BUserDataList]),
MyUserJSONList = sensor_preapre_data(BUserDataList, StrDate),
io:format("The Present Date Sensor Users Data with Binary 1: ~n~p~n",[MyUserJSONList]),
MyUserJSONListLength = length(MyUserJSONList),
if MyUserJSONListLength > 0 ->
MyFinalList = sensor_data_final(MyUserJSONList),
io:format("The Present Date Sensor Users Data without Binary 2: ~n~p~n",[MyFinalList]),
{200, [MyFinalList], Headers};
%%{json, MyFinalList};
true ->
{200, "NO DATA FOUND", Headers}
%%{json, [{error, "NO DATA FOUND"}]}
end.
In the Chicagoboss Server logs I'm getting:
The Present Date Sensor Users Data with Binary 1:
[[<<"{\"username\":\"KPBatman1\",\"building\":\"A\",\"device\":\"Fitbit\",\"date\":\"2017-07-23\",\"calorie\":732,\"distance\":6.4399999999999995,\"elevation\":0,\"floor\":0,\"steps\":8}">>],
[<<"{\"username\":\"KPSuperman1\",\"building\":\"A\",\"device\":\"Jawbone\",\"date\":\"2017-07-23\",\"calorie\":0,\"distance\":0.0,\"elevation\":0,\"floor\":0,\"steps\":0}">>]]
The Present Date Sensor Users Data without Binary 2:
[["{\"username\":\"KPBatman1\",\"building\":\"A\",\"device\":\"Fitbit\",\"date\":\"2017-07-23\",\"calorie\":732,\"distance\":6.4399999999999995,\"elevation\":0,\"floor\":0,\"steps\":8}"],
["{\"username\":\"KPSuperman1\",\"building\":\"A\",\"device\":\"Jawbone\",\"date\":\"2017-07-23\",\"calorie\":0,\"distance\":0.0,\"elevation\":0,\"floor\":0,\"steps\":0}"]]
However, when I send the HTTP request - the JSON response I am getting:
{"username":"KPBatman1","building":"A","device":"Fitbit","date":"2017-07-23","calorie":732,"distance":6.4399999999999995,"elevation":0,"floor":0,"steps":8}
{"username":"KPSuperman1","building":"A","device":"Jawbone","date":"2017-07-23","calorie":0,"distance":0.0,"elevation":0,"floor":0,"steps":0}
What is the correct way to send JSON response?
However, when I send the HTTP request - the JSON response I am
getting:
{"username":"KPBatman1","building":"A", ...}
{"username":"KPSuperman1","building":"A", ...}
And? What did you expect/want to get?
The following code works for me because the output is what I expected to see:
-module(cb_tutorial_greeting_controller, [Req]).
-compile(export_all).
hello('GET', []) ->
Headers = [
{'Access-Control-Allow-Origin', "*"},
{'Access-Control-Allow-Methods', "GET, OPTIONS"},
{'Content-Type', "application/json"},
{'Access-Control-Allow-Headers', "X-Requested-With"},
{'Access-Control-Max-Age', "180"}
],
Data = [
[<<"{\"username\":\"KPBatman1\",\"building\":\"A\"}">>],
[<<"{\"username\":\"KPSuperman1\",\"building\":\"A\"}">>]
],
Json = jsx:encode(Data),
{200, Json, Headers}.
In my browser, I see:
[["{\"username\":\"KPBatman1\",\"building\":\"A\"}"],["{\"username\":\"KPSuperman1\",\"building\":\"A\"}"]]
Note that MyFinalList isn't even valid JSON:
13> Data = [["{\"a\":\"Batman\"}"], ["{\"b\":\"Superman\"}"]].
[["{\"a\":\"Batman\"}"],["{\"b\":\"Superman\"}"]]
14> jsx:is_json(Data).
false
See what I did there?

KeyError reading a JSON file

EDIT: Here's a bit more context to how the JSON is received. I'm using the ApiAI API to generate a request to their platform, and they have a method to retrieve it, like this:
# instantiate ApiAI
ai = apiai.ApiAI(CLIENT_ACCESS_TOKEN)
# declare a request obect, fill in in lower lines
request = ai.text_request()
# send ApiAI the request
request.query = "{}".format(textobject.body)
# get response from ApiAI
response = request.getresponse()
response_decode = response.read().decode("utf-8")
response_data = json.loads(response_decode)
I'm coding a webapp in Django and trying to read through a JSON response POSTed to a webhook. The code to read through the JSON, after it has been decoded, is:
if response_data['result']['action'] != "":
Request.objects.create(
request = response_data['result']['resolvedQuery']
)
When I try to run this code, I get this error:
KeyError: 'result'
on the line
if response_data['result']['action'] != "":
I'm confused because it looks to me like 'result' should be a valid key to this JSON that is being read:
{
'id':'65738806-eb8b-4c9a-929f-28dc09d6a333',
'timestamp':'2017-07-10T04:59:46.345Z',
'lang':'en',
'result':{
'source':'agent',
'resolvedQuery':'Foobar',
'action':'Baz'
},
'alternateResult':{
'source':'domains',
'resolvedQuery':'abcdef',
'actionIncomplete':False,
},
'status':{
'code':200,
'errorType':'success'
}
}
Is there another way I should be reading this JSON in my program?
Try:
import JSON
if 'action' in response_data:
parsed_data = json.loads(response_data)
if parsed_data['result']['action'] != "":
Request.objects.create(request = parsed_data['result']['resolvedQuery'])
Thanks for everyone's thoughts. It turned out there was an another error with how I was trying to implement the ApiAI API, and that was causing this error. It now reads through the JSON fine, and I'm using #sasuke's suggestion.

Play Framework 2.5 ajax json route parameter async MongoDB

POST ing json from javascript to server in Play Framework:
var myJson = {"name": "joe", "age":20};
var obj = JSON.parse(myJson);
$.ajax(jsRoutes.controllers.MyController.create(obj));
Now, I have the javascript router configured fine. If i recieve the obj as a string I can print it out to the console just fine.
routes.conf:
POST /person/add controllers.MyController.createFromAjax(ajax: String)
BUT, I want to write the json to MongoDB using an Async promise which Activator gives the compile time error:
scala.concurrent.Future[play.api.mvc.Result][error] cannot be applied to (String)
I have other routes that take no parameters that receive json using Postman and write it to MongoDB just fine
routes.conf
POST /heartrates/bulk controllers.HRController.createFromJson
If I omit the parameter on the route that receives the json from Ajax instead of using Postman I get a HTTP 400 error in the browser.
POST http://localhost:9000/person/add 400 (Bad Request)
SO, my question is, Ajax needs a parameter but String wont work. Play documentation says json is always received as a String. What am I doing wrong here?
Scala Controller Code taken from Lightbend seed Play.Reactive.MongoDB:
def createBulkFromAjax = Action.async(parse.json) { request =>
val documents = for {
heartRate <- request.body.asOpt[JsArray].toStream
maybeHeartRate <- heartRate.value
validHeartRate <- maybeHeartRate.transform(transformer).asOpt.toList
} yield validHeartRate
for {
heartRate <- hrFuture
multiResult <- heartRate.bulkInsert(documents = documents, ordered = true)
} yield {
Logger.debug(s"Successfully inserted with multiResult: $multiResult")
Created(s"Created ${multiResult.n} heartRate")
}
}
I think you're getting mixed up between the parameters you pass to your Action as part of the jsRoutes call, and parameters that get passed to endpoints (i.e. the query string, query parameters etc).
Play will return a 400 Bad Request if you've declared a non-optional parameter (like you did with ajax: String) and you don't then actually supply it in your request.
While conceptually you are passing obj to your action, it's not as a query parameter - you've declared that your endpoint expects an HTTP POST - so the JSON should be in the HTTP request body. Notice your other endpoints don't take any query parameters.
So step 1 is to fix your routes file (I've renamed your method to match your other existing working one):
POST /person/add controllers.MyController.createFromJson
If you look at the Play documentation for the Javascript reverse router, you'll see that you'll need to set the type (aka HTTP method) if you're doing something other than a GET. So, step 2, here's what your Javascript should look like to achieve a POST:
var myJson = {"name": "joe", "age":20};
var obj = JSON.stringify(myJson);
var r = controllers.MyController.createFromJson;
$.ajax({url: r.url, type: r.type, data: obj });
After those changes you should be good; your controller code looks fine. If you still get 400 Bad Request responses, check that jQuery is setting your Content-Type header correctly - you may need to use the contentType option in the jQuery $.ajax call.
Edit after still getting 400 errors:
I've just noticed that you were using JSON.parse in your Javascript - as per this answer you should be using JSON.stringify to convert an object into something jQuery can send - otherwise it may try to URLEncode the data and/or send the fields as query parameters.
The other thing to look at is whether the JSON you are sending actually agrees with what you're trying to parse it as. I'm not sure if you've provided a simplified version for this question but it looks like you're trying to parse:
{"name": "joe", "age":20}
Using:
request.body.asOpt[JsArray]
Which will always result in a None - you didn't give it an array.
The Answer to ajax javascript routes in Play Framework 2.5 for ReativeMongo:
routes.conf:
GET /javascriptRoutes controllers.HRController.javascriptRoutes
HRController:
def javascriptRoutes = Action { implicit request =>
Ok(
JavaScriptReverseRouter("jsRoutes")(
routes.javascript.HRController.createBulkFromAjax
)
).as("text/javascript")
}
routes.conf:
POST /heartrates/add controllers.HRController.createBulkFromAjax
main.scala.html:
<script type="text/javascript" src="#routes.HRController.javascriptRoutes"></script>
javascript:
var r = jsRoutes.controllers.HRController.createBulkFromAjax();
$.ajax({url: r.url, type: r.type, contentType: "application/json", data: JsonString });
HRController:
def createBulkFromAjax = Action.async(parse.json) { request =>
//Transformation silent in case of failures.
val documents = for {
heartRate <- request.body.asOpt[JsArray].toStream
maybeHeartRate <- heartRate.value
validHeartRate <- maybeHeartRate.transform(transformer).asOpt.toList
} yield validHeartRate
for {
heartRate <- hrFuture
multiResult <- heartRate.bulkInsert(documents = documents, ordered = true)
} yield {
Logger.debug(s"Successfully inserted with multiResult: $multiResult")
Created(s"Created ${multiResult.n} heartRate")
}
}
HRController.createBulkFromAjax was built from a Lightbend activator ui seed example called play.ReactiveMogno