I'm trying to add a method to an (quite big) existing project writtent in python with pyramid framework and sqlalchemy ORM. I've wanted to execute an sql query with sqlalchemy but I've never developped with pyramid or sqlalchemy before. So I would like to test it and see if the query returns what I'm expecting but I don't want to add useless code to test my query ( like a new template, a view etc). My SQL query is :
select a.account_type, u.user_id from accounts a inner join account_users au on a.account_id=au.account_id inner join users u on u.user_id=au.user_id where u.user_id = ?;
And my method is :
def find_account_type_from_user_id(self,user_id):
'''
Method that finds the account type (one/several points of sale...)
from the id of the user who is linked to this account
:param user_id:
:return:(string) account_type
'''
q = self.query(Account)\
.join(AccountUser)\
.join(User)\
.filter(User.user_id == user_id)\
.one()
return q
ps: I've already searched on the internet but I only find things like : unit tests etc and I've never did that. (Noob's sorry).
Unit tests are a must to test new services, fixes, refactoring code, etc, you need a good collection of unit tests.
You can start here.
Two ways to see SQLAlchemy query content
Set sqlalchemy logging level to INFO - see instructions https://opensourcehacker.com/2016/05/22/python-standard-logging-pattern/
Use pyramid_debugtoolbar and it shows all queries your view made
Execute query interactively using pshell - no views need to be added
Related
I'm developing an API using NestJS & TypeORM to fetch data from a MySQL DB. Currently I'm trying to get all the instances of an entity (HearingTonalTestPage) and all the related entities (e.g. Frequency). I can get it using createQueryBuilder:
const queryBuilder = await this.hearingTonalTestPageRepo
.createQueryBuilder('hearing_tonal_test_page')
.innerJoinAndSelect('hearing_tonal_test_page.hearingTest', 'hearingTest')
.innerJoinAndSelect('hearingTest.page', 'page')
.innerJoinAndSelect('hearing_tonal_test_page.frequencies', 'frequencies')
.innerJoinAndSelect('frequencies.frequency', 'frequency')
.where(whereConditions)
.orderBy(`page.${orderBy}`, StringToSortType(pageFilterDto.ascending));
The problem here is that this will produce a SQL query (screenshot below) which will output a line per each related entity (Frequency), when I want to output a line per each HearingTonalTestPage (in the screenshot example, 3 rows instead of 12) without losing its relations data. Reading the docs, apparently this can be easily achieved using the relations option with .find(). With QueryBuilder I see some relation methods, but from I've read, under the hood it will produce JOINs, which of course I want to avoid.
So the 1 million $ question here is: is it possible with CreateQueryBuilder to load the relations after querying the main entities (something similar to .find({ relations: { } }) )? If yes, how can I achieve it?
I am not an expert, but I had a similar case and using:
const qb = this.createQueryBuilder("product");
// apply relations
FindOptionsUtils.applyRelationsRecursively(qb, ["createdBy", "updatedBy"], qb.alias, this.metadata, "");
return qb
.orderBy("product.id", "DESC")
.limit(1)
.getOne();
it worked for me, all relations are correctly loaded.
ref: https://github.com/typeorm/typeorm/blob/master/src/find-options/FindOptionsUtils.ts
You say that you want to avoid JOINs, and are seeking an analogue of find({relations: {}}), but, as the documentation says, find({relations: {}}) uses under the hood, expectedly, LEFT JOINs. So when we talk about query with relations, it can't be without JOIN's.
Now about the problem:
The problem here is that this will produce a SQL query (screenshot
below) which will output a line per each related entity (Frequency),
when I want to output a line per each HearingTonalTestPage
Your query looks fine. And the result of the query, also, is ok. I think that you expected to have as a result of the query something similar to json structure(when the relation field contains all the information inside itself instead of creating new rows and spread all its values on several rows). But that is how the SQL works. By the way, getMany() method should return 3 HearingTonalTestPage objects, not 12, so what the SQL query returns should not worry you.
The main question:
is it possible with CreateQueryBuilder to load the relations after
querying the main entities
I did't get what do you mean by saying "after querying the main entities". Can you provide more context?
I have several procedures with same name but in different schemas. When these procedures raise an error, it is possible in parent procedure (which is calling these nested stored procedures) get schema of the procedure which raised an error ? For example i can get name from ERROR_PROCEDURE() but is there some option to get also SCHEMA ? Because otherwise i am not sure which exactly procedure throwed an error if there are many with same name.
I guess this feature is still missing
https://connect.microsoft.com/SQLServer/feedback/details/124627/schema-not-reported-in-the-error-procedure-function
but is there some workaround for this ?
Sadly, there is no 100% Workaround for this limitation in SQL-Server.
Shame on the MSSQL Dev Team for not rectifying this, well over a Decade later.
It should be as simple as adding a New Function like ERROR_ProcedureSchema() or ERROR_PROCID().
Here is a revived Post Requesting this Feature from way back in May of 2005:
https://feedback.azure.com/forums/908035-sql-server/suggestions/32894584-schema-not-reported-in-the-error-procedure-functio
I prefer to Log as much detail as possible about Exceptions I capture in my custom Error Handling Logic.
This is the best I could come up with to find the Schema Name:
DECLARE #Error_ProcSchemaName nVarChar(128)--Leave as Null if found in more than 1 Schema.
--Only Populate the #Error_ProcSchemaName if it Belongs to 1 Schema. - 04/08/2019 - MCR.
SELECT #Error_ProcSchemaName = S.name
FROM sys.objects as O
JOIN sys.schemas as S
ON S.schema_id = O.schema_id
JOIN
(
SELECT O.name[ObjectName], COUNT(*)[Occurrences]
FROM sys.objects as O
GROUP BY O.name
) AS Total
ON Total.ObjectName = O.name
WHERE O.name = ERROR_PROCEDURE()
AND Total.Occurrences = 1
Avoid using anything like OBJECT_SCHEMA_NAME(OBJECT_ID(ERROR_PROCEDURE())) as the string you pass into OBJECT_ID() should already have the Schema in it (which ERROR_PROCEDURE() does not).
Otherwise it will default to your Default Schema, which (in most cases) is dbo.
Run this Query to View all your Object Names that are Reused across Schemas:
--View Object Names that Exist in Multiple Schemas: - 04/08/2019 - MCR.
SELECT S.name[SchemaName], O.name[ObjectName], Total.Occurrences,
O.type[Type], O.type_desc[TypeDesc],
O.object_id[ObjectID], O.principal_id[PrincipalID], O.parent_object_id[ParentID],
O.is_ms_shipped[MS], O.create_date[Created], O.modify_date[Modified]
FROM sys.objects as O
JOIN sys.schemas as S
ON S.schema_id = O.schema_id
JOIN
(
SELECT O.name[ObjectName], COUNT(*)[Occurrences]
FROM sys.objects as O
GROUP BY O.name
) AS Total
ON Total.ObjectName = O.name
WHERE Total.Occurrences > 1
ORDER BY [ObjectName], [SchemaName]
If you only have a few Objects (Sprocs and Triggers) that overlap, then you might be okay not knowing the Schema as it may be obvious where it originated from.
However, if this is not the case, then you may need to either:
Change the Name of the Sproc/Trigger to make it Unique.
This option goes against the very fiber of my being.
If you are using Advanced Error Handling, then manually add the Schema of your Sproc/Trigger with OBJECT_SCHEMA_NAME(##PROCID)
in your Catch-Block when logging the Error.
Note: These options may not be possible due to the use of 3rd Party Sprocs you are not allowed to edit.
When Troubleshooting with multiple Sprocs/Triggers that share the same Name, you might be able to write a Custom Wrapper-Sproc to call your 3rd Party Sproc, then Log any exception thrown in your Wrapper to know exactly which Schema/Sproc caused it.
The Code Smell:
If you have multiple Sprocs/Triggers with the same name spread across various Schemas
then I would call that a "Code Smell".
Meaning, your Architecture is flawed.
You may not be properly encapsulating your logic for reuse.
There will be times when a name overlaps Schemas, but this should be rare and by coincidence only.
Misappropriating Schemas for Handling Multi-Tenant / UserGroup Access:
If you are trying something Multi-Tenant (storing data from different Organizations/UserGroups in the same Database and preventing them from seeing eachother's info) and running almost the same logic in each Schema that shares the Object Name, then that's a Design Problem.
You should have your data in Different Databases if Users will be accessing it directly
or have a TenantID or UserGroupID you always pass in and filter on everywhere when Users will be accessing from a Custom Application.
Some possible solutions I can think of:
Renaming each Stored Procedure so that they have different names in the different schemas.
Adding some debugging output to the Stored Procedures so that when they are being executed, you can see which one was in progress when your error occurred.
Running the SQL Profiler to see what is being called at the time your error occurs.
However, these are coming more from the perspective of trying to troubleshoot an issue you're having right now, rather than building in some error handling for potential future troubleshooting. You could always get these Stored Procedures to write some log files to disk somewhere so you can interrogate those logs when an error is experienced perhaps.
I'm having a problem in the Django admin. I'm using version 1.5.5
I have a Booth model which has a foreign key to my AreaLayout model, which then goes back through another few models with foreign and many2many keys. My model code can be seen on pastebin. The initial indication of the problem in admin is that AreaLayouts are being duplicated in the select dropdown in Booth admin. The MultipleObjectsReturned error (traceback) is being raised when I then try to save a new booth. I was able to trace this back to the SQL query that django is creating to grab the AreaLayout list:
SELECT `venue_arealayout`.`id`, `venue_arealayout`.`name`, `venue_arealayout`.`description`, `venue_arealayout`.`area_id`, `venue_arealayout`.`additional_notes`
FROM `venue_arealayout`
INNER JOIN `venue_area` ON (`venue_arealayout`.`area_id` = `venue_area`.`id`)
INNER JOIN `venue_venue` ON (`venue_area`.`venue_id` = `venue_venue`.`id`)
INNER JOIN `venue_venue_venue_type` ON (`venue_venue`.`id` = `venue_venue_venue_type`.`venue_id`)
INNER JOIN `venue_venuetype` ON (`venue_venue_venue_type`.`venuetype_id` = `venue_venuetype`.`id`)
WHERE (`venue_arealayout`.`id` = 66 )
This query produces a duplicate in MySQL when I run it there. Removing the final 2 JOINs results in a single result being returned (which is the desired result), whereas removing only the last join still results in duplication.
I tried running that query with SELECT * in place of selecting specific fields and the two results in that case are almost equal. The difference is that the venue in question has multiple venuetypes and I'm getting a result for each of those. Is there any way I can tell django not to include those joins for these queries, or is there a way I can get distinct results as far as AreaLayouts go?
I think you're being caught by the bug reported in ticket 11707. One of the comments mentions that it can cause a MultipleObjectsReturned exception.
All I can suggest is that you stop using limit_choices_to. Your models are fairly complex, so I can't immediately see which one is causing the problem.
I am new in Code Igniter and like its active record feature now is there any useful steps or tips or any guidness how do i convert my pervoiusly written simple SQL Queries in CI style like this is my perviouly written simple query
SELECT *
FROM hs_albums
WHERE id NOT IN (SELECT album_id
FROM hs_delete_albums
WHERE user_id = 72
AND del_type = 1)
AND ( created = 72
OR club_id IN (SELECT cbs.id
FROM hs_clubs cbs
INNER JOIN hs_club_permissions cbp
ON cbs.id = cbp.club_id
WHERE cbp.user_id = 72
AND cbp.status = 2)
OR group_id IN (SELECT gps.id
FROM hs_groups gps
INNER JOIN hs_group_permissions grp
ON gps.id = grp.group_id
WHERE grp.user_id = 72
AND grp.status = 2)
OR comp_id IN (SELECT cmp.id
FROM hs_companies cmp
INNER JOIN hs_comp_permissions comp
ON cmp.id = comp.comp_id
WHERE comp.user_id = 72
AND comp.status = 2) )
The short answer is: You don't.
CodeIgniter's Active Record implementation is basically a layer on top of SQL that makes writing queries easier by:
Automatically escaping values
Automatically generating the appropriate query syntax for the database, so that the application can be more easily ported between databases (for instance, if you didn't use Active Record to write a query, and then wanted to move from MySQL to PostgreSQL, then you might well need to rewrite the query to make it work with PostgreSQL)
Providing a syntax for queries in PHP directly, thus avoiding the context switching between PHP and SQL.
However, it can't do everything SQL can do, and while I would always try to use ActiveRecord where possible, there comes a point where you're better off forgetting about using it and just using $this->db->query() to write your query directly. In this case, as mamdouh alramadan has said, CodeIgniter doesn't support subqueries so you can't replicate this query using ActiveRecord anyway.
The thing to remember is that ActiveRecord is a tool, not a requirement. If you're using CodeIgniter and aren't using an ORM instead, you should use it for the reasons mentioned above. However, once it starts getting in the way, you should consider whether it would be better practice to write your query manually instead.
I have a simple model with 3 ForeignKey fields.
class Car(models.Model):
wheel = models.ForeignKey('Wheel', related_name='wheels')
created = models.DateTimeField(auto_now_add=True)
max_speed = models.PositiveSmallIntegerField(null=True)
dealer = models.ForeignKey('Dealer')
category = models.ForeignKey('Category')
For the list view in the django admin i get 4 queries. One of them is a SELECT with 3 INNER JOINS. That one query is way to slow. Replacing the INNER JOINs with STRAIGHT_JOIN would fix the issue. Is there a way to patch the admin generated query just before it is evaluated?
I've implemented a fix for INNER JOIN for Django ORM, it will use STRAIGHT_JOIN in case of ordering with INNER JOINs. I talked to Django core-devs and we decided to do this as a separate backend for now. So you can check it out here: https://pypi.python.org/pypi/django-mysql-fix
However, there is one other workaround. Use a snippet from James's answer, but replace select_related with:
qs = qs.select_related('').prefetch_related('wheel', 'dealer', 'category')
It will cancel INNER JOIN and use 4 separate queries: 1 to fetch cars and 3 others with car_id IN (...).
UPDATE:
I've found one more workaround. Once you specify null=True in your ForeignKey field, Django will use LEFT OUTER JOINs instead of INNER JOIN. LEFT OUTER JOIN works without performance issues in this case, but you may face other issues that I'm not aware of yet.
You may just specify list_select_related = () to prevent django from using inner join:
class CarAdmin(admin.ModelAdmin):
list_select_related = ()
You could overwrite
def changelist_view(self, request, extra_context=None):
method in your admin class inherited from ModelAdmin class
something like this(but this question is rather old):
Django Admin: Getting a QuerySet filtered according to GET string, exactly as seen in the change list?
Ok, I found a way to patch the admin generated Query. It is ugly but it seems to work:
class CarChangeList(ChangeList):
def get_results(self, request):
"""Override to patch ORM generated SQL"""
super(CarChangeList, self).get_results(request)
original_qs = self.result_list
sql = str(original_qs.query)
new_qs = Car.objects.raw(sql.replace('INNER JOIN', 'STRAIGHT_JOIN'))
def patch_len(self):
return original_qs.count()
new_qs.__class__.__len__ = patch_len
self.result_list = new_qs
class CarAdmin(admin.ModelAdmin):
list_display = ('wheel', 'max_speed', 'dealer', 'category', 'created')
def get_changelist(self, request, **kwargs):
"""Return custom Changelist"""
return CarChangeList
admin.site.register(Rank, RankAdmin)
I came across the same issue in the Django admin (version 1.4.9) where fairly simple admin listing pages were very slow when backed by MySQL.
In my case it was caused by the ChangeList.get_query_set() method adding an overly-broad global select_related() to the query set if any fields in list_display were many-to-one relationships. For a proper database (cough PostgreSQL cough) this wouldn't be a problem, but it was for MySQL once more than a few joins were triggered this way.
The cleanest solution I found was to replace the global select_related() directive with a more targeted one that only joined tables that were really necessary. This was easy enough to do by calling select_related() with explicit relationship names.
This approach likely ends up swapping in-database joins for multiple follow-up queries, but if MySQL is choking on the large query many small ones may be faster for you.
Here's what I did, more or less:
from django.contrib.admin.views.main import ChangeList
class CarChangeList(ChangeList):
def get_query_set(self, request):
"""
Replace a global select_related() directive added by Django in
ChangeList.get_query_set() with a more limited one.
"""
qs = super(CarChangeList, self).get_query_set(request)
qs = qs.select_related('wheel') # Don't join on dealer or category
return qs
class CarAdmin(admin.ModelAdmin):
def get_changelist(self, request, **kwargs):
return CarChangeList
I've had slow admin queries on MySQL and found the easiest solution was to add STRAIGHT_JOIN to the query. I figured out a way to add this to a QuerySet rather than being forced to go to .raw(), which won't work with the admin, and have open sourced it as part of django-mysql. You can then just:
def get_queryset(self, request):
qs = super(MyAdmin, self).get_queryset(request)
return qs.straight_join()
MySQL still has this problem even in version 8 and Django still doesn't allow you to add STRAIGHT_JOIN in the query set. I found a hackish solution to add STRAIGHT_JOIN...:
This was tested with Django 2.1 and MySQL 5.7 / 8.0
def fixQuerySet(querySet):
# complete the SQL with params encapsulated in quotes
sql, params = querySet.query.sql_with_params()
newParams = ()
for param in params:
if not str(param).startswith("'"):
if isinstance(param, str):
param = re.sub("'", "\\'", param)
newParams = newParams + ("'{}'".format(param),)
else:
newParams = newParams + (param,)
rawQuery = sql % newParams
# escape the percent used in SQL LIKE statements
rawQuery = re.sub('%', '%%', rawQuery)
# replace SELECT with SELECT STRAIGHT_JOIN
rawQuery = rawQuery.replace('SELECT', 'SELECT STRAIGHT_JOIN')
return querySet.model.objects.raw(rawQuery)
Important: This method returns a raw query set so should be called just before consuming the query set