This is a little bit complicated but I try to explain as best as I can.
I have a class called Event with two attributes:
self.timestamp= datetime.now()
self.data = this is a big dictionary
I put all the instances of this class into a list and finally use json.dumps() to print the whole list to a file. json.dumps(self.timeline, indent=4, default=json_handler)
I am using a python environment where I can install/modify libraries and I only have access to python json <= 2.7.
This is my workaround to handle datetime:
# workaround for python json <= 2.7 datetime serializer
def json_handler(obj):
if hasattr(obj, 'isoformat'):
return obj.isoformat()
elif isinstance(obj, event.Event):
return {obj.__class__.__name__ : obj.data}
else:
raise TypeError("Unserializable object {} of type {}".format(obj, type(obj)))
and everything seemed to work fine until I noticed that json does not print any timestamp.
Why is that? what is happening?
When the serializer encounters your event.Event type you're only serializing its data attribute skipping the timestamp completely. You need to return the timestamp as well somehow. Maybe something like:
def json_handler(obj):
if hasattr(obj, 'isoformat'):
return obj.isoformat()
elif isinstance(obj, Event):
attrs = dict(data=obj.data, timestamp=obj.timestamp)
return {obj.__class__.__name__: attrs}
else:
raise TypeError("Unserializable object {} of type {}".format(obj, type(obj)))
Related
i'm using micropython in the newest version. I also us an DS18b20 temperature sensor. An adress of theses sensor e.g. is "b'(b\xe5V\xb5\x01<:'". These is the string representation of an an bytearray. If i use this to save the adress in a json file, i run in some problems:
If i store directly "b'(b\xe5V\xb5\x01<:'" after reading the json-file there are no single backslahes, and i get b'(bxe5Vxb5x01<:' inside python
If i escape the backslashes like "b'(b\xe5V\xb5\x01<:'" i get double backslashes in python: b'(b\xe5V\xb5\x01<:'
How do i get an single backslash?
Thank you
You can't save bytes in JSON with micropython. As far as JSON is concerned that's just some string. Even if you got it to give you what you think you want (ie. single backslashes) it still wouldn't be bytes. So, you are faced with making some form of conversion, no-matter-what.
One idea is that you could convert it to an int, and then convert it back when you open it. Below is a simple example. Of course you don't have to have a class and staticmethods to do this. It just seemed like a good way to wrap it all into one, and not even need an instance of it hanging around. You can dump the entire class in some other file, import it in the necessary file, and just call it's methods as you need them.
import math, ujson, utime
class JSON(object):
#staticmethod
def convert(data:dict, convert_keys=None) -> dict:
if isinstance(convert_keys, (tuple, list)):
for key in convert_keys:
if isinstance(data[key], (bytes, bytearray)):
data[key] = int.from_bytes(data[key], 'big')
elif isinstance(data[key], int):
data[key] = data[key].to_bytes(1 if not data[key]else int(math.log(data[key], 256)) + 1, 'big')
return data
#staticmethod
def save(filename:str, data:dict, convert_keys=None) -> None:
#dump doesn't seem to like working directly with open
with open(filename, 'w') as doc:
ujson.dump(JSON.convert(data, convert_keys), doc)
#staticmethod
def open(filename:str, convert_keys=None) -> dict:
return JSON.convert(ujson.load(open(filename, 'r')), convert_keys)
#example with both styles of bytes for the sake of being thorough
json_data = dict(address=bytearray(b'\xFF\xEE\xDD\xCC'), data=b'\x00\x01\02\x03', date=utime.mktime(utime.localtime()))
keys = ['address', 'data'] #list of keys to convert to int/bytes
JSON.save('test.json', json_data, keys)
json_data = JSON.open('test.json', keys)
print(json_data) #{'date': 1621035727, 'data': b'\x00\x01\x02\x03', 'address': b'\xff\xee\xdd\xcc'}
You may also want to note that with this method you never actually touch any JSON. You put in a dict, you get out a dict. All the JSON is managed "behind the scenes". Regardless of all of this, I would say using struct would be a better option. You said JSON though so, my answer is about JSON.
i am trying to make the following view with return JsonResponse() at the end work correctly:
def get_data(request):
full_data = Fund.objects.all()
data = {
"test2": full_data.values('investment_strategy').annotate(sum=Sum('commitment')),
}
return JsonResponse(data)
However, I get an error message saying "Object of type QuerySet is not JSON serializable".
When I put the above Queryset in a view with return render() at the end:
def get_more_data(request):
full_data = Fund.objects.all()
data = {"test2": full_data.values('investment_strategy').annotate(sum=Sum('commitment'))}
return render (request, 'test.html', data)
I get the the following result: <QuerySet [{'investment_strategy': 'Buyout', 'sum': 29}, {'investment_strategy': 'Growth', 'sum': 13}, {'investment_strategy': 'Miscellaneous', 'sum': 14}, {'investment_strategy': 'Venture Capital', 'sum': 23}, {'investment_strategy': 'n/a', 'sum': 36}]>
So the queryset works fine, I just have no clue how to return the data in proper Json format (which I would need to use the data charts.js)
I looked through answers for similar questions such as:
TypeError: object is not JSON serializable in DJango 1.8 Python 3.4
Output Django queryset as JSON
etc.
but could not find a meaningful solution for my problem.
Any help would be much appreciated!
So I managed to find a solution, that worked for me - in case anyone else has the same problem. I changed my view to the following:
def get_data(request):
full_data = Fund.objects.all()
full_data_filtered = full_data.values('investment_strategy').annotate(sum=Sum('commitment'))
labels = []
values = []
for d in full_data_filtered:
labels.append(d['investment_strategy'])
values.append(d['sum'])
data = {
"labels": labels,
"values": values,
}
return JsonResponse(data)
So basically I iterate over the Queryset and assign the values I need to lists, which can be passed to JsonResponse. I don't know if this is the most elegant way to do this (sure not), but it works and I can render my data in charts.js
JsonResponse(list(data)) will evaluate the Queryset (actually perform the query to the database) and turn it into a list that can be passed to JsonResponse.
This works because you use values and annotate, so the list is a list of dictionaries containing serializable fields.
In the example you mentioned, it didn't work because the Queryset was just returning a list of model instances, so wrapping in list() isn't enough. If you hadn't added values, you'd have had a list of Fund instances which are not serializable.
The best way I found was to create a custom QuerySet and Manager, it's not a lot of code and it is reusable!
I began by creating the custom QuerySet:
# managers.py but you can do that in the models.py too
from django.db import models
class DictQuerySet(models.QuerySet):
def dict(self):
values = self.values()
result = {}
for value in values:
id = value['id']
result[id] = value # you can customize the content of your dictionary here
return result
Then I created a custom Manager, it is optional but i prefer this way.
# managers.py, also optional
class DictManager(models.Manager):
def get_queryset(self):
return DictQuerySet(self.model, using=self._db)
Change the default manager in your model:
# models.py
from .managers import DictManager
class Fund(models.Model):
# ...
objects = DictManager()
# ...
And now you can call the dict() method from the query
# views.py
def get_data(request):
full_data = Fund.objects.all().dict()
return JsonResponse(full_data)
The response will be the full_data as a dictionary of dictionaries and each key is the primary key of the corresponding object.
If you intend to keep the same format for your JSONs, then you can use the same custom manager for all your models.
I need to dump numpy arrays of dtype float128 to JSON. For that purpose I wrote a custom array encoder, that properly handles arrays by calling tolist() on them. This works perfectly fine for all real valued dtypes, except for float128. See the following example:
import json
import numpy as np
class ArrayEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, np.ndarray):
out = {'__ndarray__': True,
'__dtype__': o.dtype.str,
'data': o.tolist()}
return out
return json.JSONEncoder.default(self, o)
arr64 = np.array([1, 2, 3], dtype='float64')
arr128 = np.array([1, 3, 4], dtype='float128')
json.dumps(arr64, cls=ArrayEncoder) # fine
json.dumps(arr128, cls=ArrayEncoder) # TypeError
TypeError: Object of type float128 is not JSON serializable
The purpose of the encoder is to provide data in a format JSON can handle. In this case the data is converted to a list of plain Python floats, which should not make any trouble. A possible solution would be to change the conversion line in the encoder to
class ArrayEncoder(json.JSONEncoder):
def default(self, o):
...
'data': o.astype('float64').tolist()
...
I am, however, interestend in cause of the problem. Why does JSON raise an error even though it uses an encoder that provides the data in a serializable format?
You have a custom encoder, but it only cares about np.array types - it converts then to a json object and their data as a Python list, and then defaults to normal lists.
The .tolist method you call makes your float128 array elements as standalone Float128 objects, which are not JSON encodable by default, and your encoder explicitly just wants to know about "isinstance(..., np.ndarray)".
You have two options: either check for float128 , so that your encoder is called recursivelly for each value, or convert then on output of np.ndarray either to plain float, or to a string:
class ArrayEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, np.ndarray):
out = {'__ndarray__': True,
'__dtype__': o.dtype.str,
'data': o.tolist()} # <- the other option is to map the results of .tolist here, and convert any float128 to str/float oon this step.
return out
elif isinstance(o, np.float128):
return float(o)
return json.JSONEncoder.default(self, o)
In Django currently Postgres JSONField model field is shown as plain JSON text in CharField form field.
I'd like to present data in CharField as YAML text (while keeping it in JSON format internally) and while saving convert it back to JSON like:
yaml.dump(json.loads(value))
Current:
Necessary:
How to accomplish that?
Thanks.
Solution is to subclass JSONField (both model and form) and just replace json with yaml. Made as separate package:
https://github.com/mike-tk/django-yamlfield
from django.contrib.postgres import fields, forms
import yaml
from django.utils.translation import gettext_lazy as _
class InvalidYAMLInput(str):
pass
class YAMLString(str):
pass
class YAMLFormField(forms.JSONField):
default_error_messages = {
'invalid': _("'%(value)s' value must be valid YAML."),
}
def to_python(self, value):
if self.disabled:
return value
if value in self.empty_values:
return None
elif isinstance(value, (list, dict, int, float, YAMLString)):
return value
try:
converted = yaml.load(value)
except yaml.YAMLError:
raise forms.ValidationError(
self.error_messages['invalid'],
code='invalid',
params={'value': value},
)
if isinstance(converted, str):
return YAMLString(converted)
else:
return converted
def bound_data(self, data, initial):
if self.disabled:
return initial
try:
return yaml.load(data)
except yaml.YAMLError:
return InvalidYAMLInput(data)
def prepare_value(self, value):
if isinstance(value, InvalidYAMLInput):
return value
return yaml.dump(value, default_flow_style=False)
class YAMLField(fields.JSONField):
def formfield(self, **kwargs):
defaults = {'form_class': YAMLFormField}
defaults.update(kwargs)
return super().formfield(**defaults)
I have a Python 3.5 program that creates an inventory of objects. I created a class of Trampolines (color, size, spring, etc.). I constantly will create new instances of the class and I then save a dictionary of them. The dictionary looks like this:
my_dict = {name: instance} and the types are like so {"string": "object"}
My issue is that I want to know how to save this inventory list so that I can start where I left off the last time I closed the program.
I don't want to use pickle because I'm trying to learn secure ways to do this for more important versions in the future.
I thought about using sqlite3, so any tips on how to do this easily would be appreciated.
My preferred solution would state how to do it with the json module. I tried it, but the error I got was:
__main__.Trampoline object at 0x00032432... is not JSON serializable
Edit:
Below is the code I used when I got the error:
out_file = open(input("What do you want to save it as? "), "w")
json.dump(my_dict, out_file, indent=4)
out_file.close()
End of Edit
I've done a good amount of research, and saw that there's also an issue with many of these save options that you can only do one object per 'save file', but that the work around to this is that you use a dictionary of objects, such as the one I made. Any info clarifying this would be great, too!
What you might be able to do is saving the instance's attributes to a CSV-file and then just create it when starting up. This might be a bit too much code and is possible not the best way. One obvious problem is that it doesn't work if you don't have the same amount of attributes as parameters, which should be possible to fix if necessary I believe. I just thought I might try and post and see if it helps :)
import json
class Trampoline:
def __init__(self, color, size, height, spring):
self.color = color
self.size = size
self.height = height
self.spring = spring
def __repr__(self):
return "Attributes: {}, {}, {}, {}".format(self.color, self.size, self.height, self.spring)
my_dict = {
"name1": Trampoline('red', 100, 2.3, True),
"name2": Trampoline('blue', 50, 2.1, False),
"name3": Trampoline('green', 25, 1.8, True),
"name5": Trampoline('white', 10, 2.6, False),
"name6": Trampoline('black', 0, 1.4, True),
"name7": Trampoline('purple', -33, 3.0, True),
"name8": Trampoline('orange', -999, 2.5, False),
}
def save(my_dict):
with open('save_file.txt', 'w') as file:
temp = {}
for name, instance in my_dict.items():
attributes = {}
for attribute_name, attribute_value in instance.__dict__.items():
attributes[attribute_name] = attribute_value
temp[name] = attributes
json.dump(temp, file)
def load():
with open('save_file.txt', 'r') as file:
my_dict = {}
x = json.load(file)
for name, attributes in x.items():
my_dict[name] = Trampoline(**attributes)
return my_dict
# CHECK IF IT WORKS!
save(my_dict)
my_dict = load()
print("\n".join(["{} | {}".format(name, instance) for name, instance in sorted(my_dict.items())]))
Here is an example of a class that handles datetime objects.
class CustomEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
if obj.tzinfo:
obj = obj.astimezone(isodate.tzinfo.UTC).replace(tzinfo=None)
return obj.isoformat()[:23] + 'Z'
return json.JSONEncoder.default(self, obj)
when you encode to json the default function of the cls is called with object you passed. If you want to handle a type that is not part of the standard json.JSONEncoder.default you need to intercept it and return how you want it handled as a valid json type. In this example I turned the datetime into a str and returned that. If its not one of the types I want to special case, I just pass it along to the standard json.JSONEncoder.default handler.
To use this class you need to pass it in the cls param of json.dump or json.dumps:
json.dumps(obj, cls=CustomEncoder)
Decoding is done the same way but with json.JSONDecoder, json.load, and json.loads. However you can not match on type, so you will need to either add an 'hint' in encoding for decoding or know what type it needs to decode.
For a simple class, you can make an easy serializer as below. This will take all of the properties of your Trampoline object and put them into a dictionary and then into JSON.
class Trampoline(object):
...
def serialize(self):
return json.dumps(vars(self))
If your class is a bit more complicated, then write a more complicated serializer :)