I have a model like this:
class MyModel(models.Model):
details = models.JSONField()
# other fields
I want to annotate some fields from this model like this:
qs = MyModel.objects.filter(id__in=given_list).annotate(
first_name=F('details__first_name'),
last_name=F('details__last_name')
)
However the F() expression is not considering the json keys, its just returning the details field only.
I am using MySQL, so cannot use KeyTextTransform.
I tried using RawSQL like this:
qs = MyModel.objects.filter(id__in=given_list).annotate(
first_name=RawSQL("(details->%s)", ('first_name',)),
last_name=RawSQL("(details->%s)", ('last_name',))
)
But it was giving this error:
MySQLdb._exceptions.OperationalError: (3143, 'Invalid JSON path expression. The error is around character position 1.')
So what can I do to make everything work as expected?
You can use JSONExtract, it will be easier to write and understand:
from django_mysql.models.functions import JSONExtract
qs = MyModel.objects.filter(id__in=given_list).annotate(
first_name=JSONExtract('details', '$.first_name'),
last_name=JSONExtract('details', '$.last_name')
)
MySQL json extractions have a special syntax using jsonfield->"$.key". Try with this:
qs = MyModel.objects.filter(id__in=given_list).annotate(
first_name=RawSQL("(details->%s)", ('$.first_name',)),
last_name=RawSQL("(details->%s)", ('$.last_name',))
)
You can just add properties to your MyModel and have them return the corresponding information
class MyModel(models.Model):
details = models.JSONField()
#property
def first_name(self):
return self.details['first_name']
#property
def last_name(self):
return self.details['last_name']
Related
Hi stack overflow community! This is my first question here, but I tried to find an answer beforehand. Right now I am working on loading data from json file like so (I actually have a json file named persons.json, not API) with the use of peewee to SQLite DB. As you can see, the json file has multiple nested dicts. My peewee model is as follows:
import json
import sqlite3
from peewee import *
db = SqliteDatabase('persons.sqlite3')
class Person(Model):
gender = CharField()
name = CharField()
location = CharField()
email = CharField()
login = CharField()
dob = CharField()
registered = CharField()
phone = CharField()
cell = CharField()
id_ = CharField()
picture = CharField()
nat = CharField()
count_dob = IntegerField()
And this is how I load all the data from json file to SQLite DB:
db.connect()
db.create_tables([Person])
with open('persons.json', encoding='utf8') as persons:
persons_data = json.load(persons)
for person in persons_data['results']:
p = Person(gender=person['gender'], name=person['name'], location=person['location'], email=person['email'],
login=person['login'], dob=person['dob'], registered=person['registered'], phone=person['phone'],
cell=person['cell'], id_=person['id'], picture=person['picture'], nat=person['nat'])
My question is, do you think the variables in my model are correctly defined (basically every single one of them as a CharField) ? The thing is, later whenever I query the DB and I need to access some of these nested dictionaries, they are actually a string, which I can convert with the use of ast.literal_eval back to dict, but I don't think it looks nice. I thought of a solution - for all the 'dictionary type' variables in my model ('location', 'dob' etc), instead of using CharField() I could probably use JSONField() - not sure how to do that though. Could you please advise on that one?
Relational databases do not support "nesting". It's foundational. Anything that is nested should probably be in a separate table or in its own column as a flat/scalar value.
https://en.wikipedia.org/wiki/Database_normalization
company = self.env['res.company'].search([('id', '=', 1)])
print company.name # prints 'my company' which is correct
print json.dumps(company) # error TypeError: res.company(1,) is not JSON serializable
Question is how do simply export company object in json?
I am looking for a generic way which would work for any model.
Use Model.read(). You can also specify the fields to be read in the read method (see doc). Also, datetime objects are not json serializable. Fortunately, Odoo already provides a utility method.
import json
from odoo.tools import date_utils
raw_data = company.read()
json_data = json.dumps(raw_data, default=date_utils.json_default)
print(json_data)
Please try this code:
company = self.env['res.company'].search([('id', '=', 1)])
print company.name
print json.dumps(company.name)
Search will return object, so we have to manually add fields to dictionary to build the json. I have added few fields, you can add more fields.
company = self.env['res.company'].search([('id', '=', 1)])
params = {}
data = dict()
data['partner_id'] = company.partner_id
data['name'] = company.name
data['email'] = company.email
data['phone'] = company.phone
data['company_registry'] = company.company_registry
params['params'] = data
print json.dumps(params)
The last answer (Jerther) is the right answer.
You can also use in odoo 16
from odoo.tools import json_default
instead of
from odoo.tools import date_utils
I have two models like this:
class McqQuestion(models.Model):
mcq_question_id = models.IntegerField()
test_id = models.ForeignKey('exam.Test')
mcq_right_answer = models.IntegerField()
class UserMcqAnswer(models.Model):
user = models.ForeignKey('exam.UserInfo')
test_id = models.ForeignKey('exam.Test')
mcq_question_id=models.ForeignKey('exam.McqQuestion')
user_answer = models.IntegerField()
I need to match the user_answer and mcq_right_answer. Able to do that by executing the below raw query.
rightAns=UserMcqAnswer.objects.raw('SELECT B.id, COUNT(A.mcq_question_id) AS RightAns\
FROM exam_mcqquestion AS A\
LEFT JOIN exam_usermcqanswer AS B\
ON A.mcq_question_id=B.mcq_question_id_id\
WHERE B.test_id_id=%s AND B.user_id=%s AND\
A.mcq_right_answer=B.user_answer',[test_id,user_id])
1) But the problem is that couldn't able to pass the result as JSONResponse because it says TypeError: Object of type 'RawQuerySet' is not JSON serializable
2) Is there any alternative to this raw query by using the objects and filtered querysets?
Django's serialize function's second argument can be any iterator that yields Django model instances.
So, in principle, you can use that raw SQL query that you worked on, using something like this:
query = """SELECT B.id, COUNT(A.mcq_question_id) AS RightAns\
FROM exam_mcqquestion AS A\
LEFT JOIN exam_usermcqanswer AS B\
ON A.mcq_question_id=B.mcq_question_id_id\
WHERE B.test_id_id=%s AND B.user_id=%s AND\
A.mcq_right_answer=B.user_answer"""%(test_id, user_id)
and then getting the json data you'll return, as:
from django.core import serializers
data = serializers.serialize('json', UserMcqAnswer.objects.raw(query), fields=('some_field_you_want', 'another_field', 'and_some_other_field'))
Good luck finding the best way to solve your issue
Edit: small fix, added an import
Using raw query is not recommended in Django.
When the model query APIs don’t go far enough, you can fall back to writing raw SQL.
In your case model query API can solve your problem. You can use the following view:
views.py
def get_answers(request):
test = Test.objects.get(name="Test 1")
answers = UserMcqAnswer.objects.filter(test_id=test, user=request.user).annotate(
is_correct=Case(
When(user_answer=F('mcq_question_id__mcq_right_answer'),
then=Value(True)),
default=Value(False),
output_field=BooleanField())
).values()
return JsonResponse(list(answers), safe=False)
Also you can consider Django Rest Framework for serialization of QuerySet.
I discovered the new generated columns functionality of MySQL 5.7, and wanted to replace some properties of my models by those kind of columns. Here is a sample of a model:
class Ligne_commande(models.Model):
Quantite = models.IntegerField()
Prix = models.DecimalField(max_digits=8, decimal_places=3)
Discount = models.DecimalField(max_digits=5, decimal_places=3, blank=True, null=True)
#property
def Prix_net(self):
if self.Discount:
return (1 - self.Discount) * self.Prix
return self.Prix
#property
def Prix_total(self):
return self.Quantite * self.Prix_net
I defined generated field classes as subclasses of Django fields (e.g. GeneratedDecimalField as a subclass of DecimalField). This worked in a read-only context and Django migrations handles it correctly, except a detail : generated columns of MySQL does not support forward references and django migrations does not respect the order the fields are defined in a model, so the migration file must be edited to reorder operations.
After that, trying to create or modify an instance returned the mysql error : 'error totally whack'. I suppose Django tries to write generated field and MySQL doesn't like that. After taking a look to django code I realized that, at the lowest level, django uses the _meta.local_concrete_fields list and send it to MySQL. Removing the generated fields from this list fixed the problem.
I encountered another problem: during the modification of an instance, generated fields don't reflect the change that have been made to the fields from which they are computed. If generated fields are used during instance modification, as in my case, this is problematic. To fix that point, I created a "generated field descriptor".
Here is the final code of all of this.
Creation of generated fields in the model, replacing the properties defined above:
Prix_net = mk_generated_field(models.DecimalField, max_digits=8, decimal_places=3,
sql_expr='if(Discount is null, Prix, (1.0 - Discount) * Prix)',
pyfunc=lambda x: x.Prix if not x.Discount else (1 - x.Discount) * x.Prix)
Prix_total = mk_generated_field(models.DecimalField, max_digits=10, decimal_places=2,
sql_expr='Prix_net * Quantite',
pyfunc=lambda x: x.Prix_net * x.Quantite)
Function that creates generated fields. Classes are dynamically created for simplicity:
from django.db.models import fields
def mk_generated_field(field_klass, *args, sql_expr=None, pyfunc=None, **kwargs):
assert issubclass(field_klass, fields.Field)
assert sql_expr
generated_name = 'Generated' + field_klass.__name__
try:
generated_klass = globals()[generated_name]
except KeyError:
globals()[generated_name] = generated_klass = type(generated_name, (field_klass,), {})
def __init__(self, sql_expr, pyfunc=None, *args, **kwargs):
self.sql_expr = sql_expr
self.pyfunc = pyfunc
self.is_generated = True # mark the field
# null must be True otherwise migration will ask for a default value
kwargs.update(null=True, editable=False)
super(generated_klass, self).__init__(*args, **kwargs)
def db_type(self, connection):
assert connection.settings_dict['ENGINE'] == 'django.db.backends.mysql'
result = super(generated_klass, self).db_type(connection)
# double single '%' if any because it will clash with later Django format
sql_expr = re.sub('(?<!%)%(?!%)', '%%', self.sql_expr)
result += ' GENERATED ALWAYS AS (%s)' % sql_expr
return result
def deconstruct(self):
name, path, args, kwargs = super(generated_klass, self).deconstruct()
kwargs.update(sql_expr=self.sql_expr)
return name, path, args, kwargs
generated_klass.__init__ = __init__
generated_klass.db_type = db_type
generated_klass.deconstruct = deconstruct
return generated_klass(sql_expr, pyfunc, *args, **kwargs)
The function to register generated fields in a model. It must be called at django start-up, for example in the ready method of the AppConfig of the application.
from django.utils.datastructures import ImmutableList
def register_generated_fields(model):
local_concrete_fields = list(model._meta.local_concrete_fields[:])
generated_fields = []
for field in model._meta.fields:
if hasattr(field, 'is_generated'):
local_concrete_fields.remove(field)
generated_fields.append(field)
if field.pyfunc:
setattr(model, field.name, GeneratedFieldDescriptor(field.pyfunc))
if generated_fields:
model._meta.local_concrete_fields = ImmutableList(local_concrete_fields)
And the descriptor. Note that it is used only if a pyfunc is defined for the field.
class GeneratedFieldDescriptor(object):
attr_prefix = '_GFD_'
def __init__(self, pyfunc, name=None):
self.pyfunc = pyfunc
self.nickname = self.attr_prefix + (name or str(id(self)))
def __get__(self, instance, owner):
if instance is None:
return self
if hasattr(instance, self.nickname) and not instance.has_changed:
return getattr(instance, self.nickname)
return self.pyfunc(instance)
def __set__(self, instance, value):
setattr(instance, self.nickname, value)
def __delete__(self, instance):
delattr(instance, self.nickname)
Note the instance.has_changed that must tell if the instance is being modified. If found a solution for this here.
I have done extensive tests of my application and it works fine, but I am far from using all django functionalities. My question is: could this settings clash with some use cases of django ?
How do I update an HSTORE field with Flask-Admin?
The regular ModelView doesn't show the HSTORE field in Edit view. It shows nothing. No control at all. In list view, it shows a column with data in JSON notation. That's fine with me.
Using a custom ModelView, I can change the HSTORE field into a TextAreaField. This will show me the HSTORE field in JSON notation when in edit view. But I cannot edit/update it. In list view, it still shows me the object in JSON notation. Looks fine to me.
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
When I attempt to save/edit the JSON, I receive this error:
sqlalchemy.exc.InternalError
InternalError: (InternalError) Unexpected end of string
LINE 1: UPDATE mytable SET attributes='{}' WHERE mytable.id = ...
^
'UPDATE mytable SET attributes=%(attributes)s WHERE mytable.id = %(mytable_id)s' {'attributes': u'{}', 'mytable_id': 14L}
Now -- using code, I can get something to save into the HSTORE field:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
model.attributes = {"a": "1"}
return
This basically overrides the model and put this object into it. I can then see the object in the List view and the Edit view. Still not good enough -- I want to save/edit the object that the user typed in.
I tried to parse and save the content from the form into JSON and back out. This doesn't work:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
x = form.data['attributes']
y = json.loads(x)
model.attributes = y
return
json.loads(x) says this:
ValueError ValueError: Expecting property name: line 1 column 1 (char
1)
and here are some sample inputs that fail:
{u's': u'ff'}
{'s':'ff'}
However, this input works:
{}
Blank also works
This is my SQL Table:
CREATE TABLE mytable (
id BIGSERIAL UNIQUE PRIMARY KEY,
attributes hstore
);
This is my SQA Model:
class MyTable(Base):
__tablename__ = u'mytable'
id = Column(BigInteger, primary_key=True)
attributes = Column(HSTORE)
Here is how I added the view's to the admin object
admin.add_view(ModelView(models.MyTable, db.session))
Add the view using a custom Model View
admin.add_view(MyView(models.MyTable, db.session))
But I don't do those views at the same time -- I get a Blueprint name collision error -- separate issue)
I also attempted to use a form field converter. I couldn't get it to actually hit the code.
class MyModelConverter(AdminModelConverter):
def post_process(self, form_class, info):
raise Exception('here I am') #but it never hits this
return form_class
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
The answer gives you a bit more then asked
Fist of all it "extends" hstore to be able to store actually JSON, not just key-value
So this structure is also OK:
{"key":{"inner_object_key":{"Another_key":"Done!","list":["no","problem"]}}}
So, first of all your ModelView should use custom converter
class ExtendedModelView(ModelView):
model_form_converter=CustomAdminConverter
Converter itself should know how to use hstore dialect:
class CustomAdminConverter(AdminModelConverter):
#converts('sqlalchemy.dialects.postgresql.hstore.HSTORE')
def conv_HSTORE(self, field_args, **extra):
return DictToHstoreField(**field_args)
This one as you can see uses custom WTForms field which converts data in both directions:
class DictToHstoreField(TextAreaField):
def process_data(self, value):
if value is None:
value = {}
else:
for key,obj in value.iteritems():
if (obj.startswith("{") and obj.endswith("}")) or (obj.startswith("[") and obj.endswith("]")):
try:
value[key]=json.loads(obj)
except:
pass #
self.data=json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
for key,obj in self.data.iteritems():
if isinstance(obj,dict) or isinstance(obj,list):
self.data[key]=json.dumps(obj)
if isinstance(obj,int):
self.data[key]=str(obj)
The final step will be to actual use this data in application
I did not make it in common nice way for SQLalchemy, since was used with flask-restful, so I have only adoption for flask-restful in one direction, but I think it's easy to get the idea from here and do the rest.
And if your case is simple key-value storage so nothing additionaly should be done, just use it as is.
But if you want to unwrap JSON somewhere in code, it's simple like this whenever you use it, just wrap in function
if (value.startswith("{") and value.endswith("}")) or (value.startswith("[") and value.endswith("]")):
value=json.loads(value)
Creating dynamical field for actual nice non-JSON way for editing of data also possible by extending FormField and adding some javascript for adding/removing fields, but this is whole different story, in my case I needed actual json storage, with blackjack and lists :)
Was working on postgres JSON datatype. The above solution worked great with a minor modifications.
Tried
'sqlalchemy.dialects.postgresql.json.JSON',
'sqlalchemy.dialects.postgresql.JSON',
'dialects.postgresql.json.JSON',
'dialects.postgresql.JSON'
The above versions did not work.
Finally the following change worked
#converts('JSON')
And changed class DictToHstoreField to the following:
class DictToJSONField(fields.TextAreaField):
def process_data(self, value):
if value is None:
value = {}
self.data = json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
else:
self.data = '{}'
Although, this is might not be the answer to your question, but by default SQLAlchemy's ORM doesn't detect in-place changes to HSTORE field values. But fortunately there's a solution: SQLAlchemy's MutableDict type:
from sqlalchemy.ext.mutable import MutableDict
class MyClass(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
attributes = Column(MutableDict.as_mutable(HSTORE))
Now when you change something in-place:
my_object.attributes.['some_key'] = 'some value'
The hstore field will be updated after session.commit().