Avoiding two distinct domain models with Spring Boot and Jackson - json

I'm designing a Spring Boot REST API that will be backed by MySQL. It has occurred to me that I want, effectively, two separate models for all my domain objects:
Model 1: Used between the outside world (REST clients) and my Spring REST controllers; and
Model 2: The entities used internally between by Spring Boot app and the MySQL database
For instance I might have a contacts table for holding personal/contact info:
CREATE TABLE contacts (
contact_id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
contact_ref_id VARCHAR(36) NOT NULL,
contact_first_name VARCHAR(100) NOT NULL,
...many more fields
);
and the respective Spring/JPA/Hibernate entity for it might look like:
// Groovy pseudo-code!
#Entity
class Contact {
#Id
#Column(name = "contact_id")
#GeneratedValue(strategy = GenerationType.IDENTITY)
Long id
#Column(name = "contact_ref_id")
UUID refId
#Column(name = "contact_first_name")
String firstName
// ...etc.
}
If I only had a single model paradigm, then when Jackson goes to serialize a Contact instance (perhaps fetched back from the DB) into JSON and send it back to the client, they'd see JSON that looks like:
{
"id" : 45,
"refId" : "067e6162-3b6f-4ae2-a171-2470b63dff00",
"firstName" : "smeeb",
...
}
Nothing like exposing primary keys to the outside world! Instead, I'd like the serialized JSON to omit the id field (as well as others). Another example might be a lookup/reference table like Colors:
# Perhaps has 7 different color records for ROYGBIV
CREATE TABLE colors (
color_id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
color_name VARCHAR(20) NOT NULL,
color_label VARCHAR(20) NOT NULL,
color_hexcode VARCHAR(20) NOT NULL,
# other stuff here
);
If the corresponding Color entity looked like this:
#Entity
class Color {
#Id
#Column(name = "color_id")
#GeneratedValue(strategy = GenerationType.IDENTITY)
Long id
#Column(name = "color_name")
String name
#Column(name = "color_label")
String label
#Column(name = "color_hexcode")
String hexcode
// ...etc.
}
Then with only one model it would serialize into JSON like so:
{
"id" : 958,
"name" : "Red",
"label" : "RED",
"hexcode" : "ff0000"
}
But maybe I just want it to come back as a simple string value:
{
"color" : "RED"
}
So it seems to me that I either need two separate models (and mapper classes that map between them) or I need a way to annotate my entities or configure either Spring, Jackson or maybe even Hibernate to apply certain transformations on my entities at the right time. Do these frameworks offer anything that can help me here, or am I going to have to go with two distinct domain models here?

You can actually accomplish this with just one model and I think it is the easiest way if you are just looking for hiding fields, custom formatting, simple transformation of attributes etc. Having two models require transformation from one model to another and vice-versa which is a pain. Jackson provides a lot of useful annotations which can be used to customize the output. Some of the annotations that can be useful for you are listed below
#JsonIgnore - ignore a field/attribute. You can hide your id field using this annotation.
#JsonInclude - Can be used to specify when a field should be present in output. For eg: Whether a field should be present in output if it is null
#JsonSerialize - You can specify a custom serializer for an attribute. For eg: You have an attribute 'password' and you want to output password as '****'.
#JsonFormat - You can apply a custom format to a field. This is very useful if you have date/time fields
#JsonProperty - If you want to give a different name for your field in your output. For eg: You have a field 'name' in your model and you want to display it as 'userName' in the output.

Related

'__post_init__' or dynamically create new fields of SQLModel after initiation

My objective is to create a new field based on another field after a Request is posted to FastAPI. My attempt was:
import functools
from datetime import date, datetime
from slugify import slugify
from sqlmodel import Field, SQLModel
class ProjectBase(SQLModel):
is_active: bool = Field(default=True, nullable=False)
name: str = Field(..., nullable=False)
publish_date: date = Field(default_factory=datetime.now().date, nullable=False)
# # post_init
repr_name: str = Field(
default_factory=functools.partial(slugify, name, separator='_'),
description="The project represented name, it must contain no whitespace, each word is separated by an underscore and it is slugified using the python-slugify library.",
nullable=False)
I have also tried __post_init__ but I think SQLModel does not have such a mechanism, it belongs to dataclasses within pydantic.
My desired output would be something like if a Request like the below was POST-ed:
request = {
'is_active': true,
'name': 'hello world bye',
'publish_date': '2023-01-01'
}
Then, the following Response is gotten and inserted into the database:
response = {
'is_active': true,
'name': 'hello world bye',
'repr_name': 'hello_world_bye', # output of slugify(`name`, separator='_')
'publish_date': '2023-01-01'
}
Thankfully sqlmodel.SQLModel inherits from pydantic.BaseModel. (And their metaclasses are also related.)
This is yet another job for custom validators. We just need a default sentinel value to check against, if no explicit value is provided by the user. I would just make the field type a union of str and None and make None the default, but then always ensure a str ends up as the value via the validator.
Here is an example:
from datetime import date, datetime
from typing import Any, Optional
from pydantic import validator
from slugify import slugify
from sqlmodel import Field, SQLModel
class ProjectBase(SQLModel):
name: str = Field(..., nullable=False)
publish_date: date = Field(default_factory=datetime.now().date, nullable=False)
repr_name: Optional[str] = Field(default=None, nullable=False)
#validator("repr_name", always=True)
def slugify_name(cls, v: Optional[str], values: dict[str, Any]) -> str:
if v is None:
return slugify(values["name"], separator="_")
return slugify(v, separator="_")
print(ProjectBase(name="foo bar baz").json(indent=2))
print(ProjectBase(name="spam", repr_name="Eggs and Bacon").json(indent=2))
Output:
{
"name": "foo bar baz",
"publish_date": "2023-02-11",
"repr_name": "foo_bar_baz"
}
{
"name": "spam",
"publish_date": "2023-02-11",
"repr_name": "eggs_and_bacon"
}
The always=True is important to trigger validation, when no value was explicitly provided.
The order of the fields is also important because the values dictionary in the validator method will only contain previously validated fields and field validation occurs in the order the fields were defined (see docs link above), so since name comes before repr_name, it will be validated and its value therefore present in the dictionary, when repr_name validation is triggered.
Important: Validation will not work, if the validators belong to a table=True model. To get around this, define a base model with the fields you need validated and inherit from that in your table=True model; then parse data through the parent model before instantiating the table model with it.
This bit seems pretty limiting, so I would not rule out the possibility that this will change in the future, at least to a certain extent. But in the current state of SQLModel that is what we have.

postgres store reference to field in json

It is possible to store json in postgres using the json data type. Check this tutorial for an introduction: http://www.postgresqltutorial.com/postgresql-json/
Consider I am storing the following json in such a field:
{
"address": {
"street1": "123 seasame st"
}
}
I want a to store separately a reference to the street field. For example, I might have another object which is using data from this json structure and wants to store a reference to where it got the data. Maybe something like this:
class Product():
__tablename__ = 'Address'
street_1 = Column(String)
data_source = ?
Now I could make data_source a string and just store namespaces like address.street, but if I did this postgres has no idea what that means. Working with that in queries would mean parsing the string and other inefficient stuff. Does postgres support referring to fields stored inside json data structures?
This question is related to JSON foreign keys in PostgreSQL , but in this case I don't necessarily want a fk relationship. I just want to create a reference, which is not necessarily enforced in the way a fk is.
update:
To be more clear, I want to reference the location of something in the json structure on another attribute and store that reference in a column. In the below code, Address.data_source is a reference to the location of the street data (for example address.street1 in this case)
class Address():
__tablename__ = 'Address'
street_1 = Column(String)
sample_id = Column(Integer, ForeignKey('DataSample.uid'))
data_source = ?
class DataSample():
__tablename__ = 'DataSample'
uid = Column(Integer, primary_key=True)
data = Column(JSONB)
body = {
"address": {
"street1": "123 seasame st"
}
}
datasample = DataSample(data=body)
address = Address(street_1=datasample.data['address']['street_1'],
sample_id=datasample.uid,
data_source=?)
As clarified, the question is seeking a way to flexibly specify a path within a JSON object of a particular record. Keys are being handled in normal columns. Constraints on JSONB fields are not available, and there is no specific support for specifying paths within JSON objects.
I worked with the following in SQL Fiddle using PostgreSQL 9.6:
CREATE TABLE datasample (
id integer PRIMARY KEY,
data jsonb
);
CREATE TABLE address (
id integer PRIMARY KEY,
street_1 text,
sample_id integer REFERENCES datasample (id),
data_source text
);
INSERT INTO datasample(id, data)
VALUES (1, '{"address":{"street_1": "123 seasame st"}}');
INSERT INTO address(id,street_1, sample_id, data_source)
VALUES (1,'123 seasame st',1,'datasample.data->''address''->>''street''');
A typical lookup of the street address (needed to retrieve street_1) would resemble:
SELECT datasample.data->'address'->>'street_1'
FROM datasample
WHERE id=1;
There is no special postgres type for identifying columns. Strings are the closest available and you will need to retrieve the string (or array of strings, or object containing strings, if one of those simplifies parsing) and use it to build the query. In tbe first code block, I stored it as the (escaped) fragment of query - 'datasample.data->''address''->>''street'''. Though longer, it would require only retrieval and unescaping to use in a new custom query. I did not find a way to use the string as a fragment within the same SQL statement, though it might be possible to combine it with other bits of text to form a full statement that could be run through EXECUTE.

How does sqlmetal generate association names?

The association names sqlmetal generates have been the source of much frustration. Sometimes the association is simply the column name with "Id" taken off the end, sometimes it generates an association name based on the foreign key constraint name.
I simply cannot figure out what steps it uses to generate these names, and a recent schema change has drastically altered the association names once again, so I'd like to get a handle on this.
I have two tables which reference each other in a sort of chain. Something like this:
class In
{
int Id;
EntityRef<Out> Yields; // YieldsId => FK_Out_Source
EntitySet<Out> In_Source; // FK_In_Source
}
class Out
{
int Id;
EntityRef<In> Yields; // YieldsId => FK_In_Source
EntitySet<In> Out_Source; // FK_Out_Source
}
These were the classes prior to the schema change, where there was an extra FK field between In and Out tables. After deleting that field, sqlmetal now generates this:
class In
{
int Id;
EntityRef<Out> Yields; // YieldsId => FK_Out_Source
EntitySet<Out> Out; // FK_In_Source
}
class Out
{
int Id;
EntityRef<In> In; // YieldsId => FK_In_Source
EntitySet<In> Out_Source; // FK_Out_Source
}
The previous classes were perfectly symmetrical as they should be, but now the generated classes are completely asymmetrical. Can anyone explain this?
Since there seems to be no rhyme or reason to this, I created a command line tool that wraps sqlmetal and rewrites the association names. It's included in my open source Sasa utilities framework, and is called sasametal.

Store long strings with Squeryl

I'd like to use VARCHAR(255) or TEXT MySQL data type to store a name of scientific article.
Squeryl creates VARCHAR(128) fields to store strings. How do I configure it to use larger fields?
From http://squeryl.org/schema-definition.html
object Library extends Schema {
...
...
on(borrowals)(b => declare(
b.numberOfPhonecallsForNonReturn defaultsTo(0),
b.borrowerAccountId is(indexed),
columns(b.scheduledToReturnOn, b.borrowerAccountId) are(indexed)
))
on(authors)(s => declare(
s.email is(unique,indexed("idxEmailAddresses")), //indexes can be named explicitely
s.firstName is(indexed),
**s.lastName is(indexed, dbType("varchar(255)")),** // the default column type can be overriden
columns(s.firstName, s.lastName) are(indexed)
))
}

Is it possible for a Grails Domain to have no 'id'?

Is it possible to create a table that has no 'id'? For example, this is my domain:
class SnbrActVector {
int nid
String term
double weight
static mapping = {
version false
id generator: 'identity'
}
static constraints = {
}
}
When I run this SQL statement, it fails:
insert into snbr_act_vector values (5, 'term', 0.5)
I checked the table and 'id' is already set to autoincrement. I'm thinking that another option is to remove the 'id' itself. Or is there another workaround for this? Please assume that it is not an option to change the givent SQL statement.
Gorm requires an id field to work. You can fake an assigned id by using a transient variable like below. The getters and setters map the nid field to the id field.
When saving a domain object using this method you have to do:
snbrActVectgor.save(insert:true)
because grails thinks a non-null id is a persistent instance.
class SnbrActVector {
Integer id
// nid is the actual primary key
static transients = ['nid']
void setNid(Integer nid) {
id = nid
}
Integer getNid() {
return nid
}
static mapping = {
version false
id generator:'assigned', column:'nid', type:'integer'
}
}
You probably need to specify that nid is your id column.
static mapping = {
version false
id generator: 'identity', column: 'nid'
}
There is no way to have no "id". what you can do is change the name of "id" field using assigned id generator.
Try using: "id( generator: 'assigned')" instead of "id generator: 'identity'" and see if that removes the autoincrement property from the "id" database column.
Yes in Oracle you can use ROWID for ID column.
class Document {
String id
static mapping = {
table "DOCUMENTS"
version false
id column: 'ROWID'
}
}