This question is related to a Question posted a few years ago.
I'm using the redshift-sqlalchemy package to connect SQLAlchemy to Redshift.
In Redshift I have a simple "companies" table:
CREATE TABLE apis
(
id INTEGER IDENTITY(1,1) NOT NULL,
name VARCHAR(255) NOT NULL,
PRIMARY KEY(id)
);
On the SQLAlchemy side I have mapped it like so:
Base = declarative_base()
class Company(Base):
__tablename__ = 'companies'
id = Column(BigInteger, primary_key=True, redshift_identity=(1, 1))
name = Column(String, nullable=False)
def __init__(self, name: str):
self.name = name
If I try to create a company:
company = Company(name = 'Hoge')
session.add(company)
session.commit()
then I get this error:
(sqlalchemy.exc.ProgrammingError) (redshift_connector.error.ProgrammingError) {'S': 'ERROR', 'C': '42P01', 'M': 'relation "companies_id_seq" does not exist', 'F': '../src/pg/src/backend/catalog/namespace.c', 'L': '267', 'R': 'LocalRangeVarGetRelid'}
[SQL: INSERT INTO companies (id, name) VALUES (%s, %s)]
[parameters: [{'name': 'Hoge'}]]
(Background on this error at: https://sqlalche.me/e/14/f405)
I think the problem is that you are trying to insert data also in the ID column with the IDENTITY option set.
SQL: INSERT INTO companies (id, name) VALUES (%s, %s)
If I execute SQL directly on a Redshift table I get the following error.
ERROR: cannot set an identity column to a value.
Please tell me how to define the auto-populated ID column in sqlalchemy-redshift model?
Related
I am quite new to sqlalchemy, I guess I am missing just a little piece here.
There is this Database (sql):
create table CEO (
id int not null auto_increment,
name char(255) not null,
primary key(id),
unique(name)
);
create table Company (
id int not null auto_increment,
name char (255) not null,
ceo int not null,
primary key(id),
foreign key(ceo) references CEO(id)
);
That code:
from sqlalchemy import create_engine, Table, Column, Integer, String, ForeignKey
from sqlalchemy.orm import registry, relationship, Session
engine = create_engine(
"mysql+pymysql:xxxxxxxx",
echo=True,
future=True
)
mapper_registry = registry()
Base = mapper_registry.generate_base()
#####################
## MAPPING CLASSES ##
#####################
class CEO(Base):
__table__ = Table('CEO', mapper_registry.metadata, autoload_with=engine)
companies = relationship('Company', lazy="joined")
class Company(Base):
__table__ = Table('Company', mapper_registry.metadata, autoload_with=engine)
##########################
## FINALLY THE QUESTION ##
##########################
with Session(engine, future=True) as session:
for row in session.query(CEO).all():
for company in row.companies:
## Just the id of the Ceo is yielded here
print(company.ceo)
So CEO.companies works as expected, but Company.ceo does not, even though the FOREIGN KEY is defined.
What is a proper setup for the Company Mapper class, such that Company.ceo yields the related object?
I could figure out, that the automatic setup did not work, because the column Company.ceo exists in the Database and represents the ID of a given row. To make everything work, I needed to rename Company.ceo to Company.ceo_id and add the relation manually like so:
CompanyTable = Table('Company', Base.metadata, autoload_with=engine)
class Company(Base):
__table__ = CompanyTable
ceo_id = CompanyTable.c.ceo
ceo = relationship('CEO')
I would like to know if it would be possible to rename the column within the Table(…) call, such that I could get rid of the extra CompanyTable thing.
I want to put a constraint on a table to limit the input of values in the table.
I want to make sure that any combination of values can only be in once.
I.e (1,2) and (2,1) can not be in the same table.
For example I have a table with two columns (c1 and c2):
The following has to be followed.
C1 C2
1 2 OK
2 1 NOT OK
3 1 OK
1 2 NOT OK
1 4 OK
1 3 NOT OK
Is there any way to do this in SQLALchemy?
I used UNIQUE(c1, c2) but that only says (1,2) and (1,2) can't be in the same table but as mentioned I also want to include that (2,1) can't be in the table either.
Thanks
Probably the easiest solution is to add a check constraint for c1 < c2 (or c1 <= c2 if they're allowed to be the same) so that (c1, c2) will always be in "ascending order":
import sqlalchemy as sa
connection_uri = (
"mssql+pyodbc://#localhost:49242/myDb?driver=ODBC+Driver+17+for+SQL+Server"
)
engine = sa.create_engine(connection_uri)
Base = declarative_base()
class So64232358(Base):
__tablename__ = "so64232358"
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
c1 = sa.Column(sa.Integer, nullable=False)
c2 = sa.Column(sa.Integer, nullable=False)
comment = sa.Column(sa.String(50))
sa.CheckConstraint(c1 < c2)
sa.UniqueConstraint(c1, c2)
Base.metadata.drop_all(engine, checkfirst=True)
Base.metadata.create_all(engine)
"""SQL rendered:
CREATE TABLE so64232358 (
id INTEGER NOT NULL IDENTITY,
c1 INTEGER NOT NULL,
c2 INTEGER NOT NULL,
comment VARCHAR(50) NULL,
PRIMARY KEY (id),
CHECK (c1 < c2),
UNIQUE (c1, c2)
)
"""
Session = sessionmaker(bind=engine)
session = Session()
obj = So64232358(c1=2, c2=1, comment="no es bueno")
session.add(obj)
try:
session.commit()
except sa.exc.IntegrityError as ie:
print(ie)
"""console output:
(pyodbc.IntegrityError) ('23000', '[23000] [Microsoft]
[ODBC Driver 17 for SQL Server][SQL Server]The INSERT statement conflicted
with the CHECK constraint "CK__so64232358__429B0397". The conflict
occurred in database "myDb", table "dbo.so64232358".
(547) (SQLExecDirectW);
[23000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
The statement has been terminated. (3621)')
"""
session.rollback()
obj = So64232358(c1=1, c2=2, comment="bueno")
session.add(obj)
session.commit() # no error
obj = So64232358(c1=1, c2=2, comment="duplicado")
session.add(obj)
try:
session.commit()
except sa.exc.IntegrityError as ie:
print(ie)
"""console output:
(pyodbc.IntegrityError) ('23000', "[23000] [Microsoft]
[ODBC Driver 17 for SQL Server][SQL Server]Violation of UNIQUE KEY
constraint 'UQ__so642323__E13250592117193A'. Cannot insert duplicate key
in object 'dbo.so64232358'. The duplicate key value is (1, 2).
(2627)(SQLExecDirectW);
[23000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
The statement has been terminated. (3621)")
"""
session.rollback()
This question already has answers here:
Unique constraint that allows empty values in MySQL
(3 answers)
Closed 2 years ago.
I have a model with a custom _id that has to be unique, and soft delete, deleted objects don't have to have a unique _id, so I did it as follows:
class MyModel(models.Model):
_id = models.CharField(max_length=255, db_index=True)
event_code = models.CharField(max_length=1, blank=True, default='I')
deleted = models.BooleanField(default=False)
deleted_id = models.IntegerField(blank=True, null=True)
objects = MyModelManager() # manager that filters out deleted objects
all_objects = MyModelBaseManager() # manager that returns every object, including deleted ones
class Meta:
constraints = [
UniqueConstraint(fields=['_id', 'event_code', 'deleted', 'deleted_id'], name='unique_id')
]
def delete(self, *args, **kwargs):
self.deleted = True
self.deleted_id = self.max_related_deleted_id() + 1
self.save()
def undelete(self, *args, **kwargs):
self.deleted = False
self.deleted_id = None
self.save()
def max_related_deleted_id(self):
# Get max deleted_id of deleted objects with the same _id
max_deleted_id = MyModel.all_objects.filter(Q(_id=self._id) & ~Q(pk=self.pk) & Q(deleted=True)).aggregate(Max('deleted_id'))['deleted_id__max']
return max_deleted_id if max_deleted_id is not None else 0
The whole logic of the deleted_id is working, I tested it out, the problem is, the UniqueConstraint is not working, for example:
$ MyModel.objects.create(_id='A', event_code='A')
$ MyModel.objects.create(_id='A', event_code='A')
$ MyModel.objects.create(_id='A', event_code='A')
$ MyModel.objects.filter(_id='A').values('pk', '_id', 'event_code', 'deleted', 'deleted_id')
[{'_id': 'A',
'deleted': False,
'deleted_id': None,
'event_code': 'A',
'pk': 1},
{'_id': 'A',
'deleted': False,
'deleted_id': None,
'event_code': 'A',
'pk': 2},
{'_id': 'A',
'deleted': False,
'deleted_id': None,
'event_code': 'A',
'pk': 3}]
Here is the migration that created the unique constraint:
$ python manage.py sqlmigrate myapp 0003
BEGIN;
--
-- Create constraint unique_id on model MyModel
--
ALTER TABLE `myapp_mymodel` ADD CONSTRAINT `unique_id` UNIQUE (`_id`, `event_code`, `deleted`, `deleted_id`);
COMMIT;
Any help is appreciated!
Django version = 2.2
Python version = 3.7
Database = MySQL 5.7
Ok, I figured out my problem, I'm gonna post this here in case someone runs into it:
The problem is with MySQL, as stated in this post, mysql allows multiple null values in a unique constraint, so I had to change the default of deleted_id to 0 and now it works.
I am trying to get an SQLAlchemy ORM class to automatically:
either lookup the foreign key id for a field
OR
for entries where the field isn't yet in foreign key table, add the row to the foreign key table - and use the auto generated id in the original table.
To illustrate:
Class Definition
class EquityDB_Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
__table_args__ = {'mysql_engine': 'InnoDB'}
__mapper_args__= {'always_refresh': True}
id = Column(Integer, primary_key=True)
def fk(tablename, nullable=False):
return Column("%s_id" % tablename, Integer,
ForeignKey("%s.id" % tablename),
nullable=nullable)
class Sector(EquityDB_Base, Base):
name = Column(String(40))
class Industry(EquityDB_Base, Base):
name = Column(String(40))
sector_id = fk('sector')
sector = relationship('Sector', backref='industries')
class Equity(EquityDB_Base, Base):
symbol = Column(String(10), primary_key=True)
name = Column(String(40))
industry_id = fk('industry')
industry = relationship('Industry', backref='industries')
Using the Class to Set Industry and Sector
for i in industry_record[]:
industry = Industry(id=i.id,
name=i.name,
sector=Sector(name=i.sector_name))
session.merge(industry)
Result
Unfortunately, when I run this - the database adds individual rows to the sector table for each duplicate use of 'sector_name' - for instance, if 10 industries use 'Technology' as their sector name, I get 10 unique sector_id for each one of the 10 industries.
What I WANT - is for each time a sector name is presented that is already in the database, for it to auto-resolve to the appropriate sector_id
I am clearly just learning SQLAlchemy, but can't seem to figure out how to enable this behavior.
Any help would be appreciated!
See answer to a similar question create_or_get entry in a table.
Applying the same logic, you would have something like this:
def create_or_get_sector(sector_name):
obj = session.query(Sector).filter(Sector.name == sector_name).first()
if not obj:
obj = Sector(name = sector_name)
session.add(obj)
return obj
and use it like below:
for i in industry_record[:]:
industry = Industry(id=i.id,
name=i.name,
sector=create_or_get_sector(sector_name=i.sector_name))
session.merge(industry)
One thing you should be careful about is which session instance is used there in the create_or_get_sector.
In my database I want to synchronize two tables. I use auth_user(Default table provided by Django) table for registration and there was another table user-profile that contain entities username, email, age etc. During the synchronization how to update Foriegn key?
def get_filename(instance,filename):
return "upload_files/%s_%s" % (str(time()).replace('.','_'),filename)
def create_profile(sender, **kwargs):
if kwargs["created"]:
p = profile(username = kwargs["instance"], email=kwargs["instance"])
p.save()
models.signals.post_save.connect(create_profile, sender=User)
class profile(models.Model):
username = models.CharField(max_length = 30)
email = models.EmailField()
age = models.PositiveIntegerField(default='15')
picture = models.FileField(upload_to='get_filename')
auth_user_id = models.ForeignKey(User)
Here in table profile during synchronization all columns are filled except auth_user_id. and there was an error
Exception Value:
(1048, "Column 'auth_user_id_id' cannot be null")
You have to alter your table and change the column auth_user_id_id datatype attribute that allows null.
Something like this:-
ALTER TABLE mytable MODIFY auth_user_id_id int;
Assuming auth_user_id_id as int datatype.(Columns are nullable by default)