How to save a dictionary of objects? - json

I have a Python 3.5 program that creates an inventory of objects. I created a class of Trampolines (color, size, spring, etc.). I constantly will create new instances of the class and I then save a dictionary of them. The dictionary looks like this:
my_dict = {name: instance} and the types are like so {"string": "object"}
My issue is that I want to know how to save this inventory list so that I can start where I left off the last time I closed the program.
I don't want to use pickle because I'm trying to learn secure ways to do this for more important versions in the future.
I thought about using sqlite3, so any tips on how to do this easily would be appreciated.
My preferred solution would state how to do it with the json module. I tried it, but the error I got was:
__main__.Trampoline object at 0x00032432... is not JSON serializable
Edit:
Below is the code I used when I got the error:
out_file = open(input("What do you want to save it as? "), "w")
json.dump(my_dict, out_file, indent=4)
out_file.close()
End of Edit
I've done a good amount of research, and saw that there's also an issue with many of these save options that you can only do one object per 'save file', but that the work around to this is that you use a dictionary of objects, such as the one I made. Any info clarifying this would be great, too!

What you might be able to do is saving the instance's attributes to a CSV-file and then just create it when starting up. This might be a bit too much code and is possible not the best way. One obvious problem is that it doesn't work if you don't have the same amount of attributes as parameters, which should be possible to fix if necessary I believe. I just thought I might try and post and see if it helps :)
import json
class Trampoline:
def __init__(self, color, size, height, spring):
self.color = color
self.size = size
self.height = height
self.spring = spring
def __repr__(self):
return "Attributes: {}, {}, {}, {}".format(self.color, self.size, self.height, self.spring)
my_dict = {
"name1": Trampoline('red', 100, 2.3, True),
"name2": Trampoline('blue', 50, 2.1, False),
"name3": Trampoline('green', 25, 1.8, True),
"name5": Trampoline('white', 10, 2.6, False),
"name6": Trampoline('black', 0, 1.4, True),
"name7": Trampoline('purple', -33, 3.0, True),
"name8": Trampoline('orange', -999, 2.5, False),
}
def save(my_dict):
with open('save_file.txt', 'w') as file:
temp = {}
for name, instance in my_dict.items():
attributes = {}
for attribute_name, attribute_value in instance.__dict__.items():
attributes[attribute_name] = attribute_value
temp[name] = attributes
json.dump(temp, file)
def load():
with open('save_file.txt', 'r') as file:
my_dict = {}
x = json.load(file)
for name, attributes in x.items():
my_dict[name] = Trampoline(**attributes)
return my_dict
# CHECK IF IT WORKS!
save(my_dict)
my_dict = load()
print("\n".join(["{} | {}".format(name, instance) for name, instance in sorted(my_dict.items())]))

Here is an example of a class that handles datetime objects.
class CustomEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
if obj.tzinfo:
obj = obj.astimezone(isodate.tzinfo.UTC).replace(tzinfo=None)
return obj.isoformat()[:23] + 'Z'
return json.JSONEncoder.default(self, obj)
when you encode to json the default function of the cls is called with object you passed. If you want to handle a type that is not part of the standard json.JSONEncoder.default you need to intercept it and return how you want it handled as a valid json type. In this example I turned the datetime into a str and returned that. If its not one of the types I want to special case, I just pass it along to the standard json.JSONEncoder.default handler.
To use this class you need to pass it in the cls param of json.dump or json.dumps:
json.dumps(obj, cls=CustomEncoder)
Decoding is done the same way but with json.JSONDecoder, json.load, and json.loads. However you can not match on type, so you will need to either add an 'hint' in encoding for decoding or know what type it needs to decode.

For a simple class, you can make an easy serializer as below. This will take all of the properties of your Trampoline object and put them into a dictionary and then into JSON.
class Trampoline(object):
...
def serialize(self):
return json.dumps(vars(self))
If your class is a bit more complicated, then write a more complicated serializer :)

Related

How to convert nested classes to json or dict?

I have class of classes as below:
class Base:
def __init__(self):
self.var1 = “variable”
class Child:
def __init__(self):
self.base = Base()
self.var2 = False
class RootChild:
def __init__(self):
self.child = Child()
self.var3 = “variable”
self.base = Base()
I want to save an instance of ‘RootChild’ with all fields of it represented in json. There are a lot of topics about this task, but I could not understand their differences and use cases. What are most pythonic solutions to this problem?
Your mileage will vary - from using a "serializer" strategy: a layer of mechanisms that will know how to map from your classes to Json and back,
which can be a lot of work (to the point of hand-describing each field that should be serialized/unserialized) - https://docs.pylonsproject.org/projects/colander/en/latest/
Or you could build a custom JSON encoder that will just check if it is a known class and then return a dict with all its fields (which can be extracted automatically by introspection) - (check the example to extend the encoder here: https://docs.python.org/3/library/json.html )
Any way there are lots of trade-offs and differing level of work -
if you plan to read the JSON back in Python apps which have the same classes, you could use "jsonpickle" - it will add extra meta-data to your JSON, but will ensure round tripping of all your objects exactly as they are: https://jsonpickle.github.io/
Without using any other lib, and assuming you just need to export your classes, the custom encoder strategy can be somewhat simple:
import json
class SimpleObject(json.JSONEncoder):
def default(self, obj):
if hasattr(obj, "__dict__"):
return {key:value for key, value in obj.__dict__.items() if not key.startswith("_")}
return super().default(obj)
a = RootChild()
And on the interactive environment:
In [52]: a = RootChild()
In [53]: print(json.dumps(a, cls=SimpleObject, indent=4))
{
"child": {
"base": {
"var1": "variable"
},
"var2": false
},
"var3": "variable",
"base": {
"var1": "variable"
}
}

Micropython: bytearray in json-file

i'm using micropython in the newest version. I also us an DS18b20 temperature sensor. An adress of theses sensor e.g. is "b'(b\xe5V\xb5\x01<:'". These is the string representation of an an bytearray. If i use this to save the adress in a json file, i run in some problems:
If i store directly "b'(b\xe5V\xb5\x01<:'" after reading the json-file there are no single backslahes, and i get b'(bxe5Vxb5x01<:' inside python
If i escape the backslashes like "b'(b\xe5V\xb5\x01<:'" i get double backslashes in python: b'(b\xe5V\xb5\x01<:'
How do i get an single backslash?
Thank you
You can't save bytes in JSON with micropython. As far as JSON is concerned that's just some string. Even if you got it to give you what you think you want (ie. single backslashes) it still wouldn't be bytes. So, you are faced with making some form of conversion, no-matter-what.
One idea is that you could convert it to an int, and then convert it back when you open it. Below is a simple example. Of course you don't have to have a class and staticmethods to do this. It just seemed like a good way to wrap it all into one, and not even need an instance of it hanging around. You can dump the entire class in some other file, import it in the necessary file, and just call it's methods as you need them.
import math, ujson, utime
class JSON(object):
#staticmethod
def convert(data:dict, convert_keys=None) -> dict:
if isinstance(convert_keys, (tuple, list)):
for key in convert_keys:
if isinstance(data[key], (bytes, bytearray)):
data[key] = int.from_bytes(data[key], 'big')
elif isinstance(data[key], int):
data[key] = data[key].to_bytes(1 if not data[key]else int(math.log(data[key], 256)) + 1, 'big')
return data
#staticmethod
def save(filename:str, data:dict, convert_keys=None) -> None:
#dump doesn't seem to like working directly with open
with open(filename, 'w') as doc:
ujson.dump(JSON.convert(data, convert_keys), doc)
#staticmethod
def open(filename:str, convert_keys=None) -> dict:
return JSON.convert(ujson.load(open(filename, 'r')), convert_keys)
#example with both styles of bytes for the sake of being thorough
json_data = dict(address=bytearray(b'\xFF\xEE\xDD\xCC'), data=b'\x00\x01\02\x03', date=utime.mktime(utime.localtime()))
keys = ['address', 'data'] #list of keys to convert to int/bytes
JSON.save('test.json', json_data, keys)
json_data = JSON.open('test.json', keys)
print(json_data) #{'date': 1621035727, 'data': b'\x00\x01\x02\x03', 'address': b'\xff\xee\xdd\xcc'}
You may also want to note that with this method you never actually touch any JSON. You put in a dict, you get out a dict. All the JSON is managed "behind the scenes". Regardless of all of this, I would say using struct would be a better option. You said JSON though so, my answer is about JSON.

Django View: Return Queryset in JSON Format

i am trying to make the following view with return JsonResponse() at the end work correctly:
def get_data(request):
full_data = Fund.objects.all()
data = {
"test2": full_data.values('investment_strategy').annotate(sum=Sum('commitment')),
}
return JsonResponse(data)
However, I get an error message saying "Object of type QuerySet is not JSON serializable".
When I put the above Queryset in a view with return render() at the end:
def get_more_data(request):
full_data = Fund.objects.all()
data = {"test2": full_data.values('investment_strategy').annotate(sum=Sum('commitment'))}
return render (request, 'test.html', data)
I get the the following result: <QuerySet [{'investment_strategy': 'Buyout', 'sum': 29}, {'investment_strategy': 'Growth', 'sum': 13}, {'investment_strategy': 'Miscellaneous', 'sum': 14}, {'investment_strategy': 'Venture Capital', 'sum': 23}, {'investment_strategy': 'n/a', 'sum': 36}]>
So the queryset works fine, I just have no clue how to return the data in proper Json format (which I would need to use the data charts.js)
I looked through answers for similar questions such as:
TypeError: object is not JSON serializable in DJango 1.8 Python 3.4
Output Django queryset as JSON
etc.
but could not find a meaningful solution for my problem.
Any help would be much appreciated!
So I managed to find a solution, that worked for me - in case anyone else has the same problem. I changed my view to the following:
def get_data(request):
full_data = Fund.objects.all()
full_data_filtered = full_data.values('investment_strategy').annotate(sum=Sum('commitment'))
labels = []
values = []
for d in full_data_filtered:
labels.append(d['investment_strategy'])
values.append(d['sum'])
data = {
"labels": labels,
"values": values,
}
return JsonResponse(data)
So basically I iterate over the Queryset and assign the values I need to lists, which can be passed to JsonResponse. I don't know if this is the most elegant way to do this (sure not), but it works and I can render my data in charts.js
JsonResponse(list(data)) will evaluate the Queryset (actually perform the query to the database) and turn it into a list that can be passed to JsonResponse.
This works because you use values and annotate, so the list is a list of dictionaries containing serializable fields.
In the example you mentioned, it didn't work because the Queryset was just returning a list of model instances, so wrapping in list() isn't enough. If you hadn't added values, you'd have had a list of Fund instances which are not serializable.
The best way I found was to create a custom QuerySet and Manager, it's not a lot of code and it is reusable!
I began by creating the custom QuerySet:
# managers.py but you can do that in the models.py too
from django.db import models
class DictQuerySet(models.QuerySet):
def dict(self):
values = self.values()
result = {}
for value in values:
id = value['id']
result[id] = value # you can customize the content of your dictionary here
return result
Then I created a custom Manager, it is optional but i prefer this way.
# managers.py, also optional
class DictManager(models.Manager):
def get_queryset(self):
return DictQuerySet(self.model, using=self._db)
Change the default manager in your model:
# models.py
from .managers import DictManager
class Fund(models.Model):
# ...
objects = DictManager()
# ...
And now you can call the dict() method from the query
# views.py
def get_data(request):
full_data = Fund.objects.all().dict()
return JsonResponse(full_data)
The response will be the full_data as a dictionary of dictionaries and each key is the primary key of the corresponding object.
If you intend to keep the same format for your JSONs, then you can use the same custom manager for all your models.

Django serializing all objects in JSONB

I am trying to serialize a model for displaying on an existing front end interface. The model is setup as such:
class Timevalue(models.Model):
time = models.FloatField(blank=True, null=True)
values = JSONField(blank=True, null=True)
The nature of the values is that it has no defined keys, therefore it is using JSON rather than a structured schema. As an end result, I need to RestAPI to output a list of timevalue objects that is flattened so that each element contains the time key as well as all the keys for the values.
So far I have written the following serializer that can return the data in the format of [{'time': 0.01, 'values': {'value1': 1, 'value2': 2, 'value3': 3}}]
class TimevalueSerializer(serializers.Serializer):
time = serializers.FloatField()
values = serializers.JSONField()
However I cannot achieve getting the output in the necessary format: [{'time': 0.01, 'value1': 1, 'value2': 2, 'value3': 3}].
I have tried the following serializer setup:
class TimevaluechildSerializer(serializers.Serializer):
fields = '*'
class TimevalueSerializer(serializers.Serializer):
time = serializers.FloatField()
values = TimevaluechildSerializer('*')
but I cannot work out what to pass to the child serializer in order for it to return all of the key-value pairs.
As this model is used for other views, I prefer to use a Serializer rather than a ModelSerializer.
Hopefully the answer isn't too difficult.
Stu
Maybe using serializer will be hard for this, rather than that, you can send this response manually. For example:
from rest_framework import status
from rest_framework.response import Response
class SomeApiView(ApiView):
resp_list = list()
for i in Timevalues.objects.all():
t = {'time': i.time}
t.update(i.values)
resp_list.append(t)
return Response(resp_list, status=status.HTTP_200_OK)

GCP Proto Datastore encode JsonProperty in base64

I store a blob of Json in the datastore using JsonProperty.
I don't know the structure of the json data.
I am using endpoints proto datastore in order to retrieve my data.
The probleme is the json property is encoded in base64 and I want a plain json object.
For the example, the json data will be:
{
first: 1,
second: 2
}
My code looks something like:
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Model(EndpointsModel):
data = ndb.JsonProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):
#Model.method(path='mymodel2', http_method='POST',
name='mymodel.insert')
def MyModelInsert(self, my_model):
my_model.data = {"first": 1, "second": 2}
my_model.put()
return my_model
#Model.method(path='mymodel/{entityKey}',
http_method='GET',
name='mymodel.get')
def getMyModel(self, model):
print(model.data)
return model
API = endpoints.api_server([DataEndpoint])
When I call the api for getting a model, I get:
POST /_ah/api/myapi/v1/mymodel2
{
"data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}
where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}
And the print statement give me: {u'second': 2, u'first': 1}
So, in the method, I can explore the json blob data as a python dict.
But, in the api call, the data is encoded in base64.
I expeted the api call to give me:
{
'data': {
'second': 2,
'first': 1
}
}
How can I get this result?
After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:
It will show as plain JSON after the API call.
You will be able to parse it again to a Python dict using eval.
I hope I understood correctly your issue, and this helps you with it.
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
# URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
#Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
def MyModelInsert(self, my_model):
dict={'first':1, 'second':2}
dict_str=str(dict)
my_model.column1="Year"
my_model.column2=2018
my_model.column3=dict_str
my_model.put()
return my_model
# URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
dict=eval(my_model.column3)
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
application = endpoints.api_server([MyApi], restricted=False)
I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.
After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:
Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:
And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):
I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.
Update:
I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:
#Define the nested model (your JSON object)
class Structured(EndpointsModel):
first = ndb.IntegerProperty()
second = ndb.IntegerProperty()
#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
column4 = ndb.StructuredProperty(Structured)
#Modify this endpoint definition to add a new property
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
#Add the new nested property here
dict=eval(my_model.column3)
my_model.column4=dict
print(json.dumps(my_model.column3))
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
With these changes, the response of the call to the endpoint looks like:
Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.
I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.