Difference between JSONParser and JSONRenderer - json

I was going through django rest framework tutorial on serialization in which I got stuck at JSONRenderers and JSONParsers.Below is the code mentioned there:
from snippets.models import Snippet
from snippets.serializers import SnippetSerializer
from rest_framework.renderers import JSONRenderer
from rest_framework.parsers import JSONParser
from django.utils.six import BytesIO
snippet = Snippet(code='foo = "bar"\n')
snippet.save()
snippet = Snippet(code='print "hello, world"\n')
snippet.save()
serializer = SnippetSerializer(snippet)
serializer.data
# {'pk': 2, 'title': u'', 'code': u'print "hello, world"\n', 'linenos': False, 'language': u'python', 'style': u'friendly'}
content = JSONRenderer().render(serializer.data)
content
# '{"pk": 2, "title": "", "code": "print \\"hello, world\\"\\n", "linenos": false, "language": "python", "style": "friendly"}'
stream = BytesIO(content)
data = JSONParser().parse(stream)
I know JSONParser is used for deserializing the data and JSONRenderer is used for serializing it , but I still don't have a sound understanding of the difference between serializing and deserializing of a data. Can someone provide me a clear understanding of this.(Answer with an example is highly appreciated.)
And also how does JSONRenderer and JSONParser work?

I will start with viewsets. View and viewsets are classes in DRF, where most of application logic happens.
Eg. ModelViewSet is class responsible for CRUD operations in response to POST, PUT, PATCH, GET, DELETE HTTP methods.
Lets take look at default create method from https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/mixins.py this method create instance of your model, from data (if they are valid) send via HTTP POST method and persist them to database.
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
self.perform_create(serializer)
headers = self.get_success_headers(serializer.data)
return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
def perform_create(self, serializer):
serializer.save()
This is whats happening here.
self.get_serializer() create new instance of serializer (you set serializer class before), it takes request.data as argument. request.data is (this is importent) dictionary. Dictionary is generic python data structure.
serializer.is_valid() method checks if request.data are valid. If yes you can access serializer.data - also a dictionary.
serializer.save() method creates and persist actual instance of your model (Snippet) to database.
You can directly access instance like this
instance = serializer.save()
Then you return Response object populated with serializer.data back to client.
As you can see, there is no Form data,JSON, XML, HTML etc. in viewset. You work with python data types and serializer is reponsible of "translating" dictionary to instance of your specific model and backwards.
But client send data (in your case) in HTTP request in form of JSON.
JSONParser is responsible of converting JSON to dictionary. Parsing is implemented inside request class https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/request.py , please notice that is not standard django HttpRequest model.
You can set multiple parsers, then request will choose proper one according to HTTP request header: Content-type.
Second thing is, you have to return serializer.data back to client in form of JSON, not dictionary. Thats what JSONRenderer does. It convert dict to JSON and its implemented inside Response class https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/response.py. Also you can set multiple renderers and then the proper one is choose according to accepted media type http header.
Example of full viewset definition might be:
from snippets.models import Snippet
from snippets.serializers import SnippetSerializer
from rest_framework.renderers import JSONRenderer
from rest_framework.parsers import JSONParser
class SnippetViewSet(viewsets.ModelViewSet):
queryset = Snippet.objects.all()
serializer_class = SnippetSerializer
renderer_classes = (JSONRenderer, )
parser_classes = (JSONParser,)

Related

must we render serialized data to json before sending response? DRF

The answer to this question is confusing me.
Multiple Models in Django Rest Framework?
the answer is to a question of sending multipule models in a response. I have the same use case.
the author of the answer has this:
def get(self, request, format=None, **kwargs):
cart = get_cart(request)
cart_serializer = CartSerializer(cart)
another_serializer = AnotherSerializer(another_object)
return Response({
'cart': cart_serializer.data,
'another': another_serializer.data,
'yet_another_field': 'yet another value',
})
but I am keeping with the documentation.
http://www.django-rest-framework.org/api-guide/serializers/#serializing-objects
EXAMPLE FROM DOCS
serializer = CommentSerializer(comment)
serializer.data
# {'email': 'leila#example.com', 'content': 'foo bar', 'created': '2016-01-27T15:17:10.375877'}
from rest_framework.renderers import JSONRenderer
json = JSONRenderer().render(serializer.data)
json
# b'{"email":"leila#example.com","content":"foo bar","created":"2016-01-27T15:17:10.375877"}'
so which one is it? Do I JSON or not JSON. This is what I currently have.
def get(self, request, format=None):
searchcityqueryset = SearchCity.objects.all()
neighborhoodqueryset = SearchNeighborhood.objects.all()
serializedsearchcity = SearchCitySerializer(searchcityqueryset)
serializedsearchneighborhood = SearchNeighborhoodSerializer(neighborhoodqueryset)
jsonsearchcity = JSONRenderer().render(serializedsearchcity.data)
jsonsearchneighborhood = JSONRenderer().render(serializedsearchneighborhood.data)
return Response({
'searchcity': jsonsearchcity,
'searchneighborhood': jsonsearchneighborhood,
})
You don't need to do it.
From the doc:
Unlike regular HttpResponse objects, you do not instantiate Response
objects with rendered content. Instead you pass in unrendered data,
which may consist of any Python primitives.
Also JSONRenderer is default renderer class, which will be used to render.
So you can simple do this:
return Response({
'cart': cart_serializer.data,
'another': another_serializer.data,
'yet_another_field': 'yet another value',
})

GCP Proto Datastore encode JsonProperty in base64

I store a blob of Json in the datastore using JsonProperty.
I don't know the structure of the json data.
I am using endpoints proto datastore in order to retrieve my data.
The probleme is the json property is encoded in base64 and I want a plain json object.
For the example, the json data will be:
{
first: 1,
second: 2
}
My code looks something like:
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Model(EndpointsModel):
data = ndb.JsonProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):
#Model.method(path='mymodel2', http_method='POST',
name='mymodel.insert')
def MyModelInsert(self, my_model):
my_model.data = {"first": 1, "second": 2}
my_model.put()
return my_model
#Model.method(path='mymodel/{entityKey}',
http_method='GET',
name='mymodel.get')
def getMyModel(self, model):
print(model.data)
return model
API = endpoints.api_server([DataEndpoint])
When I call the api for getting a model, I get:
POST /_ah/api/myapi/v1/mymodel2
{
"data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}
where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}
And the print statement give me: {u'second': 2, u'first': 1}
So, in the method, I can explore the json blob data as a python dict.
But, in the api call, the data is encoded in base64.
I expeted the api call to give me:
{
'data': {
'second': 2,
'first': 1
}
}
How can I get this result?
After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:
It will show as plain JSON after the API call.
You will be able to parse it again to a Python dict using eval.
I hope I understood correctly your issue, and this helps you with it.
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
# URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
#Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
def MyModelInsert(self, my_model):
dict={'first':1, 'second':2}
dict_str=str(dict)
my_model.column1="Year"
my_model.column2=2018
my_model.column3=dict_str
my_model.put()
return my_model
# URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
dict=eval(my_model.column3)
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
application = endpoints.api_server([MyApi], restricted=False)
I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.
After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:
Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:
And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):
I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.
Update:
I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:
#Define the nested model (your JSON object)
class Structured(EndpointsModel):
first = ndb.IntegerProperty()
second = ndb.IntegerProperty()
#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
column4 = ndb.StructuredProperty(Structured)
#Modify this endpoint definition to add a new property
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
#Add the new nested property here
dict=eval(my_model.column3)
my_model.column4=dict
print(json.dumps(my_model.column3))
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
With these changes, the response of the call to the endpoint looks like:
Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.
I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.

Django Rest Framework - use serializers to send json with requests library

I hava DRF setup that receives data in json and stores it inside django. Serializer is following
class ReservationSerializer(serializers.ModelSerializer):
room = RoomSerializer()
reserved_days = DaySerializer(many=True)
additional_services = AdditionalServicesSerializer(many=True)
class Meta:
model = Reservation
fields = [
'start',
'end',
'check_in_time',
'check_out_time',
'reserved_days',
'additional_services',
'room',
'has_refund',
'payed',
'guest_name',
'reservation_number',
]
Can I use this serializer to prepare models in json and then send this json with Requests library ?
It's feasible, if you see the need. Although, the JSON encoding is done in the Response object, which is a full HTTPResponse subclass, so you would need to encode your own data:
import json
import requests
my_objects = Reservation.objects.all()
serializer = ReservationSerializer(data=my_objects, many=True)
if serializer.is_valid():
# now you do your encoding:
encoded_data = json.dumps(serializer.data)
response = requests.post(your_url,
headers={'Content-Type': 'application/json'},
data=encoded_data)

How to get Slurpable data from REST client in Groovy?

I have code that looks like this:
def client = new groovyx.net.http.RESTClient('myRestFulURL')
def json = client.get(contentType: JSON)
net.sf.json.JSON jsonData = json.data as net.sf.json.JSON
def slurper = new JsonSlurper().parseText(jsonData)
However, it doesn't work! :( The code above gives an error in parseText because the json elements are not quoted. The overriding issue is that the "data" is coming back as a Map, not as real Json. Not shown, but my first attempt, I just passed the parseText(json.data) which gives an error about not being able to parse a HashMap.
So my question is: how do I get JSON returned from the RESTClient to be parsed by JsonSlurper?
The RESTClient class automatically parses the content and it doesn't seem possible to keep it from doing so.
However, if you use HTTPBuilder you can overload the behavior. You want to get the information back as text, but if you only set the contentType as TEXT, it won't work, since the HTTPBuilder uses the contentType parameter of the HTTPBuilder.get() method to determine both the Accept HTTP Header to send, as well was the parsing to do on the object which is returned. In this case, you need application/json in the Accept header, but you want the parsing for TEXT (that is, no parsing).
The way you get around that is to set the Accept header on the HTTPBuilder object before calling get() on it. That overrides the header that would otherwise be set on it. The below code runs for me.
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.6')
import static groovyx.net.http.ContentType.TEXT
def client = new groovyx.net.http.HTTPBuilder('myRestFulURL')
client.setHeaders(Accept: 'application/json')
def json = client.get(contentType: TEXT)
def slurper = new groovy.json.JsonSlurper().parse(json)
The type of response from RESTClient will depend on the version of :
org.codehaus.groovy.modules.http-builder:http-builder
For example, with version 0.5.2, i was getting a net.sf.json.JSONObject back.
In version 0.7.1, it now returns a HashMap as per the question's observations.
When it's a map, you can simply access the JSON data using the normal map operations :
def jsonMap = restClientResponse.getData()
def user = jsonMap.get("user")
....
Solution posted by jesseplymale workes for me, too.
HttpBuilder has dependencies to some appache libs,
so to avoid to add this dependencies to your project,
you can take this solution without making use of HttpBuilder:
def jsonSlurperRequest(urlString) {
def url = new URL(urlString)
def connection = (HttpURLConnection)url.openConnection()
connection.setRequestMethod("GET")
connection.setRequestProperty("Accept", "application/json")
connection.setRequestProperty("User-Agent", "Mozilla/5.0")
new JsonSlurper().parse(connection.getInputStream())
}

How to output Django queryset as JSON?

I want to serialize my queryset, and I want it in a format as this view outputs:
class JSONListView(ListView):
queryset = Users.objects.all()
def get(self, request, *args, **kwargs):
return HttpResponse(json.dumps({'data': [['bar','foo','bar','foo'],['foo','bar','foo','bar']]}, indent=4), content_type='application/json')
I simply don't know how to output the queryset instead of the manual data in the example.
I've tried
json.dumps({"data": self.get_queryset()})
and
serializers.serialize("json", {'data': self.get_queryset()})
but it won't work. What am I doing wrong? Do I need to make a custom JSON Encoder?
You can use JsonResponse with values. Simple example:
from django.http import JsonResponse
def some_view(request):
data = list(SomeModel.objects.values()) # wrap in list(), because QuerySet is not JSON serializable
return JsonResponse(data, safe=False) # or JsonResponse({'data': data})
Or another approach with Django's built-in serializers:
from django.core import serializers
from django.http import HttpResponse
def some_view(request):
qs = SomeModel.objects.all()
qs_json = serializers.serialize('json', qs)
return HttpResponse(qs_json, content_type='application/json')
In this case result is slightly different (without indent by default):
[
{
"model": "some_app.some_model",
"pk": 1,
"fields": {
"name": "Elon",
"age": 48,
...
}
},
...
]
I have to say, it is good practice to use something like marshmallow to serialize queryset.
...and a few notes for better performance:
use pagination if your queryset is big;
use objects.values() to specify list of required fields to avoid serialization and sending to client unnecessary model's fields (you also can pass fields to serializers.serialize);
It didn't work, because QuerySets are not JSON serializable.
1) In case of json.dumps you have to explicitely convert your QuerySet to JSON serializable objects:
class Model(model.Model):
def as_dict(self):
return {
"id": self.id,
# other stuff
}
And the serialization:
dictionaries = [ obj.as_dict() for obj in self.get_queryset() ]
return HttpResponse(json.dumps({"data": dictionaries}), content_type='application/json')
2) In case of serializers. Serializers accept either JSON serializable object or QuerySet, but a dictionary containing a QuerySet is neither. Try this:
serializers.serialize("json", self.get_queryset())
Read more about it here:
https://docs.djangoproject.com/en/dev/topics/serialization/
For a efficient solution, you can use .values() function to get a list of dict objects and then dump it to json response by using i.e. JsonResponse (remember to set safe=False).
Once you have your desired queryset object, transform it to JSON response like this:
...
data = list(queryset.values())
return JsonResponse(data, safe=False)
You can specify field names in .values() function in order to return only wanted fields (the example above will return all model fields in json objects).
To return the queryset you retrieved with queryset = Users.objects.all(), you first need to serialize them.
Serialization is the process of converting one data structure to another. Using Class-Based Views, you could return JSON like this.
from django.core.serializers import serialize
from django.http import JsonResponse
from django.views.generic import View
class JSONListView(View):
def get(self, request, *args, **kwargs):
qs = User.objects.all()
data = serialize("json", qs)
return JsonResponse(data)
This will output a list of JSON. For more detail on how this works, check out my blog article How to return a JSON Response with Django. It goes into more detail on how you would go about this.
If the goal is to build an API that allow you to access your models in JSON format I recommend you to use the django-restframework that is an enormously popular package within the Django community to achieve this type of tasks.
Django Rest Framework Website
Github
It include useful features such as Pagination, Defining Serializers, Nested models/relations and more. Even if you only want to do minor Javascript tasks and Ajax calls I would still suggest you to build a proper API using the Django Rest Framework instead of manually defining the JSON response.
Another way to turn queryset into JSON, is appending necessary elements to an empty list with loop. It provides to design customizable JSON.
queryset = Users.objects.all()
output = []
for query in queryset:
output.append('id': query.id, 'name': query.name, etc...)
return JSONResponse(output, safe=False)
Try this:
class JSONListView(ListView):
queryset = Users.objects.all()
def get(self, request, *args, **kwargs):
data = {}
data["users"] = get_json_list(queryset)
return JSONResponse(data)
def get_json_list(query_set):
list_objects = []
for obj in query_set:
dict_obj = {}
for field in obj._meta.get_fields():
try:
if field.many_to_many:
dict_obj[field.name] = get_json_list(getattr(obj, field.name).all())
continue
dict_obj[field.name] = getattr(obj, field.name)
except AttributeError:
continue
list_objects.append(dict_obj)
return list_objects
from django.http import JsonResponse
def SomeFunction():
dict1 = {}
obj = list( Mymodel.objects.values() )
dict1['data']=obj
return JsonResponse(dict1)
Try this code for Django