Relation in sqlalchemy
I'll ask this question separately, since it's a continuation of this one, but with a different content.
I want to serialize a join and then serialize it with pydantic, but it doesn't work.
If possible, I want to return it as if it were a single object.
{
id: 1,
parent_column: test1,
child_column: test2
}
You can create a Pydantic model with the information you want, so the model will take care of the serialization to JSON format.
from pydantic import BaseModel
class MyResponse(BaseModel):
id: int
parent: str
child: str
You just have to create a response from your model by providing it with the data in the requested format.
data_json = MyResponse(id= my_data.id, parent=my_data.parent, child=my_data.child)
Below is an example given with a fastAPI endpoint that would take an ID to retrieve a specific data and return a JSON response with a specific template.
from pydantic import BaseModel
from fastapi import status
class MyResponse(BaseModel):
id: int
parent: str
child: str
# router.get('/data',
status_code=status.HTTP_200_OK,
response_model=MyResponse)
def get_data(parent_id:int):
try:
data = get_data(parent_id)
except Exception:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,detail="Server Error")
return MyResponse(id=data.id, parent=data.parent, child=data.child)
Afterwards, it is possible to link Pydantic models with sqlalchemy via the orm_mode of the Pydantic class. More information here: https://fastapi.tiangolo.com/tutorial/sql-databases/#use-pydantics-orm_mode
Related
I follow the FastAPI Tutorial and am not quite sure what the exact relationship between the proposed data objects is.
We have the models.py file:
from sqlalchemy import Boolean, Column, ForeignKey, Integer, String
from sqlalchemy.orm import relationship
from .database import Base
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True, index=True)
email = Column(String, unique=True, index=True)
hashed_password = Column(String)
is_active = Column(Boolean, default=True)
items = relationship("Item", back_populates="owner")
class Item(Base):
__tablename__ = "items"
id = Column(Integer, primary_key=True, index=True)
title = Column(String, index=True)
description = Column(String, index=True)
owner_id = Column(Integer, ForeignKey("users.id"))
owner = relationship("User", back_populates="items")
and the schemas.py file:
from typing import List, Union
from pydantic import BaseModel
class ItemBase(BaseModel):
title: str
description: Union[str, None] = None
class ItemCreate(ItemBase):
pass
class Item(ItemBase):
id: int
owner_id: int
class Config:
orm_mode = True
class UserBase(BaseModel):
email: str
class UserCreate(UserBase):
password: str
class User(UserBase):
id: int
is_active: bool
items: List[Item] = []
class Config:
orm_mode = True
Those classes are then used to define db queries like in the crud.py file:
from sqlalchemy.orm import Session
from . import models, schemas
def get_user(db: Session, user_id: int):
return db.query(models.User).filter(models.User.id == user_id).first()
def get_user_by_email(db: Session, email: str):
return db.query(models.User).filter(models.User.email == email).first()
def get_users(db: Session, skip: int = 0, limit: int = 100):
return db.query(models.User).offset(skip).limit(limit).all()
def create_user(db: Session, user: schemas.UserCreate):
fake_hashed_password = user.password + "notreallyhashed"
db_user = models.User(email=user.email, hashed_password=fake_hashed_password)
db.add(db_user)
db.commit()
db.refresh(db_user)
return db_user
def get_items(db: Session, skip: int = 0, limit: int = 100):
return db.query(models.Item).offset(skip).limit(limit).all()
def create_user_item(db: Session, item: schemas.ItemCreate, user_id: int):
db_item = models.Item(**item.dict(), owner_id=user_id)
db.add(db_item)
db.commit()
db.refresh(db_item)
return db_item
and in the FastAPI code main.py:
from typing import List
from fastapi import Depends, FastAPI, HTTPException
from sqlalchemy.orm import Session
from . import crud, models, schemas
from .database import SessionLocal, engine
models.Base.metadata.create_all(bind=engine)
app = FastAPI()
# Dependency
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
#app.post("/users/", response_model=schemas.User)
def create_user(user: schemas.UserCreate, db: Session = Depends(get_db)):
db_user = crud.get_user_by_email(db, email=user.email)
if db_user:
raise HTTPException(status_code=400, detail="Email already registered")
return crud.create_user(db=db, user=user)
#app.get("/users/", response_model=List[schemas.User])
def read_users(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)):
users = crud.get_users(db, skip=skip, limit=limit)
return users
#app.get("/users/{user_id}", response_model=schemas.User)
def read_user(user_id: int, db: Session = Depends(get_db)):
db_user = crud.get_user(db, user_id=user_id)
if db_user is None:
raise HTTPException(status_code=404, detail="User not found")
return db_user
#app.post("/users/{user_id}/items/", response_model=schemas.Item)
def create_item_for_user(
user_id: int, item: schemas.ItemCreate, db: Session = Depends(get_db)
):
return crud.create_user_item(db=db, item=item, user_id=user_id)
#app.get("/items/", response_model=List[schemas.Item])
def read_items(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)):
items = crud.get_items(db, skip=skip, limit=limit)
return items
Now from what I understand:
the models data classes define the sql tables
the schemas data classes define the api that FastAPI uses to interact with the database
they must be convertible into each other so that the set-up works
What I don't understand:
in crud.create_user_item I expected the return type to be schemas.Item, since that return type is used by FastAPI again
according to my understanding the response model of #app.post("/users/{user_id}/items/", response_model=schemas.Item) in the main.py is wrong, or how can I understand the return type inconsistency?
however inferring from the code, the actual return type must be models.Item, how is that handled by FastAPI?
what would be the return type of crud.get_user?
I'll go through your bullet points one by one.
The models data classes define the sql tables
Yes. More precisely, the ORM classes that map to actual database tables are defined in the models module.
the schemas data classes define the api that FastAPI uses to interact with the database
Yes and no. The Pydantic models in the schemas module define the data schemas relevant to the API, yes. But that has nothing to do with the database yet. Some of these schemas define what data is expected to be received by certain API endpoints for the request to be considered valid. Others define what the data returned by certain endpoints will look like.
they must be convertible into each other so that the set-up works
While the database table schemas and the API data schemas are usually very similar, that is not necessarily the case. In the tutorial however, they correspond quite neatly, which allows succinct CRUD code, like this:
db_item = models.Item(**item.dict(), owner_id=user_id)
Here item is a Pydantic model instance, i.e. one of your API data schemas schemas.ItemCreate containing data you decided is necessary for creating a new item. Since its fields (their names and types) correspond to those of the database model models.Item, the latter can be instantiated from the dictionary representation of the former (with the addition of the owner_id).
in crud.create_user_item I expected the return type to be schemas.Item, since that return type is used by FastAPI again
No, this is exactly the magic of FastAPI. The function create_user_item return an instance of models.Item, i.e. the ORM object as constructed from the database (after calling session.refresh on it). And the API route handler function create_item_for_user actually does return that same object (of class models.Item).
However, the #app.post decorator takes that object and uses it to construct an instance of the response_model you defined for that route, which is schemas.Item in this case. This is why you set this in your schemas.Item model:
class Config:
orm_mode = True
This allows an instance of that class to be created via the .from_orm method. That all happens behind the scenes and again depends on the SQLAlchemy model corresponding to the Pydantic model with regards to field names and types. Otherwise validation fails.
according to my understanding the response model [...] is wrong
No, see above. The decorated route function actually deals with the schemas.Item model.
however inferring from the code, the actual return type must be models.Item
Yes, see above. The return type of the undecorated route handler function create_item_for_user is in fact models.Item. But its return type is not the response model.
I assume that to reduce confusion the documentation example does not annotate the return type of those route functions. If it did, it would look like this:
#app.post("/users/{user_id}/items/", response_model=schemas.Item)
def create_item_for_user(
user_id: int, item: schemas.ItemCreate, db: Session = Depends(get_db)
) -> models.Item:
return crud.create_user_item(db=db, item=item, user_id=user_id)
It may help to remember that a decorator is just syntactic sugar for a function that takes a callable (/function) as argument and returns a function. Typically the returned function actually internally calls the function passed to it as argument and does additional things before and/or after that call. I could rewrite the route above like this and it would be exactly the same:
def create_item_for_user(
user_id: int, item: schemas.ItemCreate, db: Session = Depends(get_db)
) -> models.Item:
return crud.create_user_item(db=db, item=item, user_id=user_id)
create_item_for_user = app.post(
"/users/{user_id}/items/", response_model=schemas.Item
)(create_item_for_user)
what would be the return type of crud.get_user?
That would be models.User because that is the database model and is what the first method of that query returns.
db.query(models.User).filter(models.User.id == user_id).first()
This is then again returned by the read_user API route function in the same fashion as I explained above for models.Item.
Hope this helps.
Suppose I have a simple SQLAlchemy class and a simple Flask o FastAPI implementation like this:
from sqlalchemy.ext.declarative import declarative_base
from pydantic import BaseModel
Base = declarative_base()
class A(Base):
__tablename__ = 'as'
my_id = Column(String)
class AModel(BaseModel):
myId:str = None
And a simple endpoint like this:
#app_router.get('/a')
def get_all_a(session:Session = Depends(get_session)):
return session.query(A).all()
How could I ensure that the returned list of this endpoint yields in camelCase like this:
[{'myId': 'id1'},{'myId': 'id2'}, ...]
Note: My application is rather complex, as I also have pagination implemented and some post-processings require a little bit more that just snake_case to camelCase conversion, so the simplest solution would be the best.
I've tried overriding dict() methods and similar stuff with no luck, simply cannot understand how FastAPI processes the results to obtain a JSON.
require a little bit more that just snake_case to camelCase conversion
Well, if you don't use response_model you don't have a lot of choices.
The solution is returning your dict with a conversion from snake_case to camelCase. You have functions that do it recursively.
Why it is the best solution ?
using regex it's super fast, faster than any lib that convert dict to obj like pydantic
If you definitely don't want to do this, well your only solution is using pydantic models, attrs or dataclasses and convert your db query output to one of those models type with camelCase variable name (dirty).
Since you are using fastapi you should use all the power of it.
I would suggest this:
from typing import List
from sqlalchemy.ext.declarative import declarative_base
from pydantic import BaseModel, Field
from pydantic import parse_obj_as
Base = declarative_base()
class A(Base):
__tablename__ = 'as'
my_id = Column(String)
class AModel(BaseModel):
myId: str = Field(alias="my_id", default=None)
#app_router.get('/a', response_model=List[AModel])
def get_all_a(session:Session = Depends(get_session)):
return parse_obj_as(List[AModel], session.query(A).all())
Keep in mind that having classes variables in CamelCase is not a good practice.
The gold answer would be to not return camel but snake and let your client do the work of conversion if needed.
I have a custom serializer that is returning a string representation of JSON. This serializer uses django.contrib.gis.serializers.geojson.Serializer which is much faster than the DRF serializer. The downside of this serializer is that it returns everything already serialized into a string, rather than as a JSON serializiable object.
Is there a way to shortcut the DRF obj>json string process and just pass the string as the json response?
Currently I am doing the following, but the obj>string>dict>string process is not ideal:
from django.contrib.gis.serializers.geojson import Serializer
from json import loads
class GeoJSONFastSerializer(Serializer):
def __init__(self, *args, **kwargs):
self.instances = args[0]
super().__init__()
#property
def data(self):
# The use of json.loads here to deserialize the string,
# only for it to be reserialized by DRF is inefficient.
return loads(self.serialize(self.instances))
Which is implemented (simplified version) in the view:
from rest_framework.mixins import ListModelMixin
from rest_framework.viewsets import GenericViewSet
class GeoJSONPlaceViewSet(ListModelMixin, GenericViewSet):
serializer_class = GeoJSONFastSerializer
queryset = Places.objects.all()
I think that you could define a custom renderer and just pass the data, something like this,
from django.utils.encoding import smart_unicode
from rest_framework import renderers
class AlreadyJSONRenderer(renderers.BaseRenderer):
media_type = 'application/json'
format = 'json'
def render(self, data, media_type=None, renderer_context=None):
return data
and in the view just add
renderer_classes = [AlreadyJSONRenderer]
Django: v2.1.5
DRF: v3.9.1
mariaDB: v10.3
Hi, I am a DRF newbie and I have been struggling with json field.
DRF does not support official json field type working with mariaDB and even though there is a 3rd-party package for mysql(django-mysql) but not compatible with mariaDB.
So I searched and started implementing custom jsonfield and it looks like:
model.py:
class JSONField(models.TextField):
def to_dict(self, value):
""" convert json string to python dictionary """
return json.loads(value)
def to_json(self, value):
""" convert python dictionary to json string """
return json.dumps(value)
def from_db_value(self, value, expression, connection):
""" convert string from db to python dictionary """
if value is None:
return value
return self.to_dict(value)
def to_python(self, value):
""" convert model input value to python dictionary """
if isinstance(value, dict):
return value
if value is None:
return value
return self.to_dict(value)
def get_prep_value(self, value):
""" convert python dictionary to string before writing to db """
return self.to_json(value)
class Project(models.Model):
objects = models.Manager()
project_user = JSONField(null=True)....
serializers.py:
class ProjectSerializer(serializers.ModelSerializer):
class Meta:
model = Project
fields = ('project_code'...)
def create(self, validated_data):
"""
Create and return a new `Project` instance, given the validated data.
"""
return Project.objects.create(**validated_data)
views.py:
class ListCreateProjectView(APIView):
"""
POST admin/_proj_/
: create a project
"""
def post(self, request, format=None):
serializer = ProjectSerializer(data=request.data)
if serializer.is_valid():
serializer.save()
return Response(data=serializer.data)
else:
return Response(data=serializer.errors)
please let me know what I am doing wrong here or a way to use 3rd party package rather than custom jsonfield
Thanks a lot and have a great day guys!
For people who suffer from this kind of problem, I actually solved this problem, but in an informal way. I set JSONfield as a textfield in DB, and handle (str->json),(json->str) in the view. But again, this is an informal way (I think) and you will need another solution than this. If you find one, please share it for me and other people :)
I have a basic model with a case class
case class Record( id: Option[String],
data: Double,
user: String,
)
object RecordJsonFormats {
import play.api.libs.json.Json
implicit val recordFormat = Json.format[Record]
}
Field user is actually an ObjectId of other module also id is also an ObjectId yet then try to change String type to BSONObjectId macros in play.api.libs.json.Json break... so both user and if saved with object id fields get saved as String not ObjectId.
What is the optimal way to operate with ObjectIds in Play framework?
Maybe I should extend play.api.libs.json.Json with BSONObjectId?
Maybe there is a way to link models and IDs are tracked automatically without a need to declare them in model?
You can override the default type of _id. You just need to specify the type you want in the case class.
import java.util.UUID
import play.api.libs.json._
case class Record (_id: UUID = UUID.randomUUID())
object Record {
implicit val entityFormat = Json.format[Record]
}
MongoDB has a default _id field of type ObjectId, which uniquely identifies a document in a given collection. However, this _id typically does not have a semantic meaning in the context of the application domain. Therefore, a good practice is to introduce an additional id field as index of documents. This id can simply a Long number, no more or less.
Then, you can search documents by id easily, and do not care much about ObjectId.
This, https://github.com/luongbalinh/play-mongo/, is a sample project using Play 2.4.x and ReactiveMongo. Hopefully, it helps you.
For those using Official Mongo Scala Driver and Play Framework 2.6+, Here's my solution: https://gist.github.com/ntbrock/556a1add78dc287b0cf7e0ce45c743c1
import org.mongodb.scala.bson.ObjectId
import play.api.libs.json._
import scala.util.Try
object ObjectIdFormatJsonMacro extends Format[ObjectId] {
def writes(objectId: ObjectId): JsValue = JsString(objectId.toString)
def reads(json: JsValue): JsResult[ObjectId] = json match {
case JsString(x) => {
val maybeOID: Try[ObjectId] = Try{new ObjectId(x)}
if(maybeOID.isSuccess) JsSuccess(maybeOID.get) else {
JsError("Expected ObjectId as JsString")
}
}
case _ => JsError("Expected ObjectId as JsString")
}
}
Use it like this in your business objects:
case class BusinessTime(_id: ObjectId = new ObjectId(), payRate: Double)
object BusinessTime {
implicit val objectIdFormat = ObjectIdFormatJsonMacro
implicit val businessTimeFormat = Json.format[BusinessTime]
}