How to JSON-serialize a dictionary that contains a Django PointField? [duplicate] - json

This question already has an answer here:
Rendering spatial data of GeoQuerySet in a custom view on GeoDjango
(1 answer)
Closed 2 years ago.
I am in the process of building a website on geodjango. On the front end I have a map on openlayers and I want to be able to fire ajax queries at some django url and get back geojson.
The problem is that the standard django json serializer doesn't do geojson. So I have a model like:
class Asset (models.Model):
objects = models.GeoManager()
url_name = models.CharField(max_length=200)
name = models.CharField(max_length=200)
point = models.PointField(srid=4326)
def __unicode__(self):
return self.name
And I want to create a view:
def geojson_query(request):
#geographic query
...
assets = Asset.objects.all().filter(point__contained=bb_4326)
json = serializers.serialize("json", assets)
return HttpResponse(json)
But then I get back a response that looks more like this:
"url_name": "Upper_Svaneti",
"name": "Upper Svaneti",
"point": "POINT (43.0113899999999987 42.9163899999999998)"
Whereas I want something that looks like this:
"url_name": "Upper_Svaneti",
"name": "Upper Svaneti",
"geometry": {
"type": "Point",
"coordinates": [43.0113899999999987 42.9163899999999998]
}
So I guess my question is, what is the best way to implement a geojson serializer in django without totally rolling my own? And if it comes to rolling my own, where do I start?

You need to write your own serializer. Just inherit from the DjangoJSONEncoder, here's one I created that supports the Point type:
from django.core.serializers.json import DjangoJSONEncoder
from django.contrib.gis.geos import Point
class GeoJSONEncoder(DjangoJSONEncoder):
def default(self, obj):
if isinstance(obj, Point):
return obj.coords
return super(GeoJSONEncoder, self).default(obj)
You can then use it like so:
from my_app.serializers import GeoJSONEncoder
from django.utils import simplejson
json = simplejson.dumps(data, cls=GeoJSONEncoder)

So... I have done something slightly unpretty. I hardcoded the non-geojson parts of the serializer and used the json function from GEOS to get the geojson part.
So the method in the model looks like:
def get_footprint_json(self):
geojson=self.footprint.json
json='{"type": "Feature","geometry": %s,"properties": {"name":"%s","url_name":"%s"}}'%(geojson,self.name,self.url_name)
return json
And... I have a view that looks like this:
json='{ "srid":4326, "type": "FeatureCollection","features": ['+','.join([asset.get_footprint_json() for asset in assets])+'] }'
return HttpResponse(json)
I'd be curious to see if anyone else has a better way or if django has updated their serializer to include geojson.

Related

How to convert nested classes to json or dict?

I have class of classes as below:
class Base:
def __init__(self):
self.var1 = “variable”
class Child:
def __init__(self):
self.base = Base()
self.var2 = False
class RootChild:
def __init__(self):
self.child = Child()
self.var3 = “variable”
self.base = Base()
I want to save an instance of ‘RootChild’ with all fields of it represented in json. There are a lot of topics about this task, but I could not understand their differences and use cases. What are most pythonic solutions to this problem?
Your mileage will vary - from using a "serializer" strategy: a layer of mechanisms that will know how to map from your classes to Json and back,
which can be a lot of work (to the point of hand-describing each field that should be serialized/unserialized) - https://docs.pylonsproject.org/projects/colander/en/latest/
Or you could build a custom JSON encoder that will just check if it is a known class and then return a dict with all its fields (which can be extracted automatically by introspection) - (check the example to extend the encoder here: https://docs.python.org/3/library/json.html )
Any way there are lots of trade-offs and differing level of work -
if you plan to read the JSON back in Python apps which have the same classes, you could use "jsonpickle" - it will add extra meta-data to your JSON, but will ensure round tripping of all your objects exactly as they are: https://jsonpickle.github.io/
Without using any other lib, and assuming you just need to export your classes, the custom encoder strategy can be somewhat simple:
import json
class SimpleObject(json.JSONEncoder):
def default(self, obj):
if hasattr(obj, "__dict__"):
return {key:value for key, value in obj.__dict__.items() if not key.startswith("_")}
return super().default(obj)
a = RootChild()
And on the interactive environment:
In [52]: a = RootChild()
In [53]: print(json.dumps(a, cls=SimpleObject, indent=4))
{
"child": {
"base": {
"var1": "variable"
},
"var2": false
},
"var3": "variable",
"base": {
"var1": "variable"
}
}

Django View: Return Queryset in JSON Format

i am trying to make the following view with return JsonResponse() at the end work correctly:
def get_data(request):
full_data = Fund.objects.all()
data = {
"test2": full_data.values('investment_strategy').annotate(sum=Sum('commitment')),
}
return JsonResponse(data)
However, I get an error message saying "Object of type QuerySet is not JSON serializable".
When I put the above Queryset in a view with return render() at the end:
def get_more_data(request):
full_data = Fund.objects.all()
data = {"test2": full_data.values('investment_strategy').annotate(sum=Sum('commitment'))}
return render (request, 'test.html', data)
I get the the following result: <QuerySet [{'investment_strategy': 'Buyout', 'sum': 29}, {'investment_strategy': 'Growth', 'sum': 13}, {'investment_strategy': 'Miscellaneous', 'sum': 14}, {'investment_strategy': 'Venture Capital', 'sum': 23}, {'investment_strategy': 'n/a', 'sum': 36}]>
So the queryset works fine, I just have no clue how to return the data in proper Json format (which I would need to use the data charts.js)
I looked through answers for similar questions such as:
TypeError: object is not JSON serializable in DJango 1.8 Python 3.4
Output Django queryset as JSON
etc.
but could not find a meaningful solution for my problem.
Any help would be much appreciated!
So I managed to find a solution, that worked for me - in case anyone else has the same problem. I changed my view to the following:
def get_data(request):
full_data = Fund.objects.all()
full_data_filtered = full_data.values('investment_strategy').annotate(sum=Sum('commitment'))
labels = []
values = []
for d in full_data_filtered:
labels.append(d['investment_strategy'])
values.append(d['sum'])
data = {
"labels": labels,
"values": values,
}
return JsonResponse(data)
So basically I iterate over the Queryset and assign the values I need to lists, which can be passed to JsonResponse. I don't know if this is the most elegant way to do this (sure not), but it works and I can render my data in charts.js
JsonResponse(list(data)) will evaluate the Queryset (actually perform the query to the database) and turn it into a list that can be passed to JsonResponse.
This works because you use values and annotate, so the list is a list of dictionaries containing serializable fields.
In the example you mentioned, it didn't work because the Queryset was just returning a list of model instances, so wrapping in list() isn't enough. If you hadn't added values, you'd have had a list of Fund instances which are not serializable.
The best way I found was to create a custom QuerySet and Manager, it's not a lot of code and it is reusable!
I began by creating the custom QuerySet:
# managers.py but you can do that in the models.py too
from django.db import models
class DictQuerySet(models.QuerySet):
def dict(self):
values = self.values()
result = {}
for value in values:
id = value['id']
result[id] = value # you can customize the content of your dictionary here
return result
Then I created a custom Manager, it is optional but i prefer this way.
# managers.py, also optional
class DictManager(models.Manager):
def get_queryset(self):
return DictQuerySet(self.model, using=self._db)
Change the default manager in your model:
# models.py
from .managers import DictManager
class Fund(models.Model):
# ...
objects = DictManager()
# ...
And now you can call the dict() method from the query
# views.py
def get_data(request):
full_data = Fund.objects.all().dict()
return JsonResponse(full_data)
The response will be the full_data as a dictionary of dictionaries and each key is the primary key of the corresponding object.
If you intend to keep the same format for your JSONs, then you can use the same custom manager for all your models.

GCP Proto Datastore encode JsonProperty in base64

I store a blob of Json in the datastore using JsonProperty.
I don't know the structure of the json data.
I am using endpoints proto datastore in order to retrieve my data.
The probleme is the json property is encoded in base64 and I want a plain json object.
For the example, the json data will be:
{
first: 1,
second: 2
}
My code looks something like:
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Model(EndpointsModel):
data = ndb.JsonProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):
#Model.method(path='mymodel2', http_method='POST',
name='mymodel.insert')
def MyModelInsert(self, my_model):
my_model.data = {"first": 1, "second": 2}
my_model.put()
return my_model
#Model.method(path='mymodel/{entityKey}',
http_method='GET',
name='mymodel.get')
def getMyModel(self, model):
print(model.data)
return model
API = endpoints.api_server([DataEndpoint])
When I call the api for getting a model, I get:
POST /_ah/api/myapi/v1/mymodel2
{
"data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}
where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}
And the print statement give me: {u'second': 2, u'first': 1}
So, in the method, I can explore the json blob data as a python dict.
But, in the api call, the data is encoded in base64.
I expeted the api call to give me:
{
'data': {
'second': 2,
'first': 1
}
}
How can I get this result?
After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:
It will show as plain JSON after the API call.
You will be able to parse it again to a Python dict using eval.
I hope I understood correctly your issue, and this helps you with it.
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
# URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
#Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
def MyModelInsert(self, my_model):
dict={'first':1, 'second':2}
dict_str=str(dict)
my_model.column1="Year"
my_model.column2=2018
my_model.column3=dict_str
my_model.put()
return my_model
# URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
dict=eval(my_model.column3)
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
application = endpoints.api_server([MyApi], restricted=False)
I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.
After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:
Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:
And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):
I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.
Update:
I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:
#Define the nested model (your JSON object)
class Structured(EndpointsModel):
first = ndb.IntegerProperty()
second = ndb.IntegerProperty()
#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
column4 = ndb.StructuredProperty(Structured)
#Modify this endpoint definition to add a new property
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
#Add the new nested property here
dict=eval(my_model.column3)
my_model.column4=dict
print(json.dumps(my_model.column3))
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
With these changes, the response of the call to the endpoint looks like:
Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.
I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.

How to save a dictionary of objects?

I have a Python 3.5 program that creates an inventory of objects. I created a class of Trampolines (color, size, spring, etc.). I constantly will create new instances of the class and I then save a dictionary of them. The dictionary looks like this:
my_dict = {name: instance} and the types are like so {"string": "object"}
My issue is that I want to know how to save this inventory list so that I can start where I left off the last time I closed the program.
I don't want to use pickle because I'm trying to learn secure ways to do this for more important versions in the future.
I thought about using sqlite3, so any tips on how to do this easily would be appreciated.
My preferred solution would state how to do it with the json module. I tried it, but the error I got was:
__main__.Trampoline object at 0x00032432... is not JSON serializable
Edit:
Below is the code I used when I got the error:
out_file = open(input("What do you want to save it as? "), "w")
json.dump(my_dict, out_file, indent=4)
out_file.close()
End of Edit
I've done a good amount of research, and saw that there's also an issue with many of these save options that you can only do one object per 'save file', but that the work around to this is that you use a dictionary of objects, such as the one I made. Any info clarifying this would be great, too!
What you might be able to do is saving the instance's attributes to a CSV-file and then just create it when starting up. This might be a bit too much code and is possible not the best way. One obvious problem is that it doesn't work if you don't have the same amount of attributes as parameters, which should be possible to fix if necessary I believe. I just thought I might try and post and see if it helps :)
import json
class Trampoline:
def __init__(self, color, size, height, spring):
self.color = color
self.size = size
self.height = height
self.spring = spring
def __repr__(self):
return "Attributes: {}, {}, {}, {}".format(self.color, self.size, self.height, self.spring)
my_dict = {
"name1": Trampoline('red', 100, 2.3, True),
"name2": Trampoline('blue', 50, 2.1, False),
"name3": Trampoline('green', 25, 1.8, True),
"name5": Trampoline('white', 10, 2.6, False),
"name6": Trampoline('black', 0, 1.4, True),
"name7": Trampoline('purple', -33, 3.0, True),
"name8": Trampoline('orange', -999, 2.5, False),
}
def save(my_dict):
with open('save_file.txt', 'w') as file:
temp = {}
for name, instance in my_dict.items():
attributes = {}
for attribute_name, attribute_value in instance.__dict__.items():
attributes[attribute_name] = attribute_value
temp[name] = attributes
json.dump(temp, file)
def load():
with open('save_file.txt', 'r') as file:
my_dict = {}
x = json.load(file)
for name, attributes in x.items():
my_dict[name] = Trampoline(**attributes)
return my_dict
# CHECK IF IT WORKS!
save(my_dict)
my_dict = load()
print("\n".join(["{} | {}".format(name, instance) for name, instance in sorted(my_dict.items())]))
Here is an example of a class that handles datetime objects.
class CustomEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
if obj.tzinfo:
obj = obj.astimezone(isodate.tzinfo.UTC).replace(tzinfo=None)
return obj.isoformat()[:23] + 'Z'
return json.JSONEncoder.default(self, obj)
when you encode to json the default function of the cls is called with object you passed. If you want to handle a type that is not part of the standard json.JSONEncoder.default you need to intercept it and return how you want it handled as a valid json type. In this example I turned the datetime into a str and returned that. If its not one of the types I want to special case, I just pass it along to the standard json.JSONEncoder.default handler.
To use this class you need to pass it in the cls param of json.dump or json.dumps:
json.dumps(obj, cls=CustomEncoder)
Decoding is done the same way but with json.JSONDecoder, json.load, and json.loads. However you can not match on type, so you will need to either add an 'hint' in encoding for decoding or know what type it needs to decode.
For a simple class, you can make an easy serializer as below. This will take all of the properties of your Trampoline object and put them into a dictionary and then into JSON.
class Trampoline(object):
...
def serialize(self):
return json.dumps(vars(self))
If your class is a bit more complicated, then write a more complicated serializer :)

How to output Django queryset as JSON?

I want to serialize my queryset, and I want it in a format as this view outputs:
class JSONListView(ListView):
queryset = Users.objects.all()
def get(self, request, *args, **kwargs):
return HttpResponse(json.dumps({'data': [['bar','foo','bar','foo'],['foo','bar','foo','bar']]}, indent=4), content_type='application/json')
I simply don't know how to output the queryset instead of the manual data in the example.
I've tried
json.dumps({"data": self.get_queryset()})
and
serializers.serialize("json", {'data': self.get_queryset()})
but it won't work. What am I doing wrong? Do I need to make a custom JSON Encoder?
You can use JsonResponse with values. Simple example:
from django.http import JsonResponse
def some_view(request):
data = list(SomeModel.objects.values()) # wrap in list(), because QuerySet is not JSON serializable
return JsonResponse(data, safe=False) # or JsonResponse({'data': data})
Or another approach with Django's built-in serializers:
from django.core import serializers
from django.http import HttpResponse
def some_view(request):
qs = SomeModel.objects.all()
qs_json = serializers.serialize('json', qs)
return HttpResponse(qs_json, content_type='application/json')
In this case result is slightly different (without indent by default):
[
{
"model": "some_app.some_model",
"pk": 1,
"fields": {
"name": "Elon",
"age": 48,
...
}
},
...
]
I have to say, it is good practice to use something like marshmallow to serialize queryset.
...and a few notes for better performance:
use pagination if your queryset is big;
use objects.values() to specify list of required fields to avoid serialization and sending to client unnecessary model's fields (you also can pass fields to serializers.serialize);
It didn't work, because QuerySets are not JSON serializable.
1) In case of json.dumps you have to explicitely convert your QuerySet to JSON serializable objects:
class Model(model.Model):
def as_dict(self):
return {
"id": self.id,
# other stuff
}
And the serialization:
dictionaries = [ obj.as_dict() for obj in self.get_queryset() ]
return HttpResponse(json.dumps({"data": dictionaries}), content_type='application/json')
2) In case of serializers. Serializers accept either JSON serializable object or QuerySet, but a dictionary containing a QuerySet is neither. Try this:
serializers.serialize("json", self.get_queryset())
Read more about it here:
https://docs.djangoproject.com/en/dev/topics/serialization/
For a efficient solution, you can use .values() function to get a list of dict objects and then dump it to json response by using i.e. JsonResponse (remember to set safe=False).
Once you have your desired queryset object, transform it to JSON response like this:
...
data = list(queryset.values())
return JsonResponse(data, safe=False)
You can specify field names in .values() function in order to return only wanted fields (the example above will return all model fields in json objects).
To return the queryset you retrieved with queryset = Users.objects.all(), you first need to serialize them.
Serialization is the process of converting one data structure to another. Using Class-Based Views, you could return JSON like this.
from django.core.serializers import serialize
from django.http import JsonResponse
from django.views.generic import View
class JSONListView(View):
def get(self, request, *args, **kwargs):
qs = User.objects.all()
data = serialize("json", qs)
return JsonResponse(data)
This will output a list of JSON. For more detail on how this works, check out my blog article How to return a JSON Response with Django. It goes into more detail on how you would go about this.
If the goal is to build an API that allow you to access your models in JSON format I recommend you to use the django-restframework that is an enormously popular package within the Django community to achieve this type of tasks.
Django Rest Framework Website
Github
It include useful features such as Pagination, Defining Serializers, Nested models/relations and more. Even if you only want to do minor Javascript tasks and Ajax calls I would still suggest you to build a proper API using the Django Rest Framework instead of manually defining the JSON response.
Another way to turn queryset into JSON, is appending necessary elements to an empty list with loop. It provides to design customizable JSON.
queryset = Users.objects.all()
output = []
for query in queryset:
output.append('id': query.id, 'name': query.name, etc...)
return JSONResponse(output, safe=False)
Try this:
class JSONListView(ListView):
queryset = Users.objects.all()
def get(self, request, *args, **kwargs):
data = {}
data["users"] = get_json_list(queryset)
return JSONResponse(data)
def get_json_list(query_set):
list_objects = []
for obj in query_set:
dict_obj = {}
for field in obj._meta.get_fields():
try:
if field.many_to_many:
dict_obj[field.name] = get_json_list(getattr(obj, field.name).all())
continue
dict_obj[field.name] = getattr(obj, field.name)
except AttributeError:
continue
list_objects.append(dict_obj)
return list_objects
from django.http import JsonResponse
def SomeFunction():
dict1 = {}
obj = list( Mymodel.objects.values() )
dict1['data']=obj
return JsonResponse(dict1)
Try this code for Django