I'm building an application that communicates with a django backend using json-rpc. So far all has been working well. However I've found an anomaly in sending " ". As far as I know the request works fine, however django interprets the response badly. I've reproduced a simplified request and response below:
Request:
{"jsonrpc":"2.0","id":"1","method":"test","params":
{"id":"80","name":"tests","introduction":"hello there"}}
Django receives:
<QueryDict:u'{"jsonrpc":"2.0","id":"1","method":"test","params":
{"id":"80","name":"tests","introduction":"hello ': [u''], u'nbsp': [u''], u'there"}}': [u'']}>
Expected response:
<QueryDict: {u'{"jsonrpc":"2.0","id":"1","method":"test","params":
{"id":"80","name":"tests","introduction":"hello there"}}': [u'']}>
It seems like django interprets the & and the ; as special characters and so creates an unexpected dictionary in its request.POST variable.
What do I need to do to make sure that the json string doesn't get malformed? I have tried encoding it using the php htmlspecialchars() method, but since that doesn't remove the '&' the problem persists.
Any help will be much appreciated.
Django is handling the (POST?) request by decoding the body (your json string) as if it were a query string, and not a json.
Within a query string, & and ; denote the end of a key:value pair. Splitting up your request body on those two characters yields the key:value pairs you see in the Django QueryDict.
You need to get hold of the POST request body and explicitly decode it to a dict yourself using either the standard lib json, or simplejson module.
I have little experience with Django specifically, but I imagine that somewhere in your view handler you would do something akin to:
try:
data = json.loads(requesst.raw_post_data)
## work with the data...
except ValueError:
## do something...
No doubt Django provides a way to move this json handling out of your views, and to somewhere more suitable.
Related
Tornado 4.5.2 using Python3 represents the request body as a byte object instead of a native dictionary. This presents a problem for methods like RequestHandler.get_body_argument() which will not access the field correctly.
My question is how to correctly have tornado parse these bodies into more useable dictionaries so the standard library will work. I've looked throughout tornado's documentation and there's next to nothing on even the existence of this problem.
Am I missing something here or will I need to re-implement those methods myself?
Tornado never automatically parses JSON; it only automatically parses HTML-standard form encoding (the data models of form encoding and JSON are different, so it wouldn't make sense to use the same family of get_argument/get_arguments methods in the less-ambiguous JSON format). If you want to handle JSON requests, it's one line to parse it yourself:
args = tornado.escape.json_decode(self.request.body)
I am receiving a block of JSON that looks like this (edited for brevity):
... "actions": [{"script": \"{\"inputs\": [], <LOTS MORE JSON>}"}]"
Essentially it's a string nested inside another object.
When I send this JSON to my Node.js API using something Advanced Rest Client in Chrome, it succeeds. When I send it using this Python code, however:
headers={'content-type': 'application/json'}
newrsp = self.session.post("http://my-api", data=str(my_json), headers=headers)
I receive
SyntaxError: Unexpected token i
Looking at the body before JSON.parse is called, it appears that my payload has been modified. Instead of looking like this:
... "actions": [{"script": \"{\"inputs\": [], <LOTS MORE JSON>}"}]"''
it looks like this:
... "actions": [{"script": "{"inputs": [], <LOTS MORE JSON>}"}]"
Calling my API from other programs works and doesn't remove the escape characters, so I'm pretty confident that it's not my code, but the Requests API is stripping out the characters somehow before it sends my JSON.
I'm not a Pythonista by an stretch... any Python people out there know what's going on (and hopefully a way I can tell requests not to do this)?
If you use the argument "data" you should convert by yourself the JSON:
jsonItem = json.dumps(dict(item))
req = requests.post('http://...', data=jsonItem)
If you use the argument "json" you delegate the conversion:
req = requests.post('http://...', json=dict(item))
I'm going to answer my own question, in case someone has a similar problem...
It looks like the requests API has some automatic conversion routine that strips off JSON escape characters from elements that don't need it. I ended up calling json.dumps on the piece of JSON that was having issue BEFORE assembling the entire long string (and calling json.dumps again when passing to requests). This effectively 'double-encoded' it, so when requests stripped off escape characters, it left it the way I wanted it.
Here's my JSON response for http://localhost:8000/characters/api/users/1?format=json
)]}',
{"id":1,"username":"admin","mage_by_user":[3],"mage_last_updated":"2015-02-11T16:13:16.229Z"}
Notice the )]}', on the first line.
Here is my code that gets called to create the JSON:
class UserSerializer(serializers.ModelSerializer):
mage_by_user = serializers.PrimaryKeyRelatedField(
many=True, queryset=Mage.objects.all())
mage_last_updated = serializers.ReadOnlyField(
source='mage_by_user.updated_date')
class Meta:
model = User
fields = ('id', 'username', 'mage_by_user', 'mage_last_updated',)
Further testing:
I've noticed the title of the page is TypeError at <insert url here>.
This happens with all of my endpoints
If I try to access a non-existent object (userId=2 for instance), then renders 'normally' for DRF, e.g:
{
detail: "Not found"
}
Any idea why this would happen?
Those characters are inserted by the Djangular middleware AngularJsonVulnerabilityMiddleware, to inject Json Vulnerability Protection
A JSON vulnerability allows third party website to turn your JSON resource URL into JSONP request under some conditions. To counter this your server can prefix all JSON requests with following string ")]}',\n". Angular will automatically strip the prefix before processing it as JSON.
Unfortunately, it means it breaks various JSON viewers.
Sorry to not be more help, but this looks like something entirely unrelated to REST framework. There's absolutely no way a JSON response there would ever be rendered in that way.
Perhaps you have a custom renderer configured, that's outputting a malformed response, perhaps you have some broken middleware inserting those characters, perhaps its an issue in the client or whatever environment you're making the requests, or perhaps it's something else entirely unrelated to any of those.
I'd start by trying to narrow down the issue as much as possible - remove all the complexity from the view and serializer and attempt to replicate the behavior in a test case.
Most likely there's some sort of unexpected integration issue you're missing or some otherwise obvious code typo that's being overlooked.
I'm brand new to Pentaho and I'm trying to do the following workflow:
read a bunch of lines out of a DB
do some transformations
POST them to a REST web service in JSON
I've got the first two figured out using an input step and the Json Output step.
However I have two problems doing the final step:
1) I can't get the JSON formatted how I want. It insists on doing {""=[{...}]} when I just want {...}. This isn't a big deal - I can work around this since I have control over the web service and I could relax the input requirements a bit. (Note: this page http://wiki.pentaho.com/display/EAI/JSON+output gives an example for the output I want by setting no. rows in a block=1 and an empty JSON block name, but it doesn't work as advertised.)
2) This is the critical one. I can't get the data to POST as JSON. It posts as key=value, where the key is the name I specify in the HTTP Post field name (on the 'Fields' tab) and the value is the encoded JSON. I just want to post the JSON as the request body. I've tried googling on this but can't find anyone else doing it, leading me to believe that I'm just approaching this wrong. Any pointers in the right direction?
Edit: I'm comfortable scripting (in Javascript or another language) but when I tried to use XmlHttpRequest in a custom javascript snippet I got an error that XmlHttpRequest is not defined.
Thanks!
This was trivial...just needed to use the REST Client (http://wiki.pentaho.com/display/EAI/Rest+Client) instead of the HTTP Post task. Somehow all my googling didn't discover that, so I'll leave this answer here in case someone else has the same problem as me.
You need to parse the JSON using a Modified JavaScript step. e.g. if the Output Value from the JSON Output is called result and its contents are {"data"=[{...}]}, you should call var plainJSON = JSON.stringify(JSON.parse(result).data[0]) to get the JSON.
In the HTTP Post step, the Request entity field should be plainJSON. Also, don't forget to add a header for Content-Type as application/json (you might have to add that as a constant)
I'm horribly sorry if there is a post for this, I tried to search but didn't find a answer.
Problem:
I'm calling a web service and receiving not so well formed JSON data from a Dynamics Nav service:
JSON:
"[{\"type\":\"2\",\"number\":\"VHT3866\",\"location\":\"Delta\",\"destinationNo\":\"\",\"contactName\":\"Jesus\",\"shipToName\":\"Lord jesus\",\"highPriority\":\"false\",\"hasComment\":\"true\",\"assignedTo\":\"\",\"source\":\"\"},{\"type\":\"2\",\"number\":\"VHT3866\",\"location\":\"Delta\",\"destinationNo\":\"\",\"contactName\":\"Jesus\",\"shipToName\":\"Lord jesus\",\"highPriority\":\"false\",\"hasComment\":\"true\",\"assignedTo\":\"\",\"source\":\"\"}]"
I then take this JSON and use angular.fromJson(json) to get it properly.
It doesn't seem to change into an array of javascript objects, but just plain text.
However if I take the same JSON and just put it manually in like this:
var json = angular.fromJson(stringfromserver);
It turns into a proper javascript object and ng-repeat throws no error.
I found an answer on Quora:
--- Le Batoure,
Angular from json is now strict so assuming that this string is from a trusted source you would have to use "eval()" plus surround the call in parenthesis for it to work
var hatsData = angular.fromJson(eval("(" + hats + ")"))
If you bring your JSON from http request for example you don't need to use the fromJson method.
The JSON is automatically parsed by Angular and you can use it directly.