I'm making a web service with Django that uses MySQL database. Clients interface with our database through URLs, handled by Django. Right now I'm trying to create a behavior that automatically does some checking/logging whenever a certain table is modified, which naturally means MySQL triggers. However I can also do this in Django, in the request handler that does the table modification. I don't think Django has trigger support yet, so I'm not sure which is better, doing through Django code or MySQL trigger.
Anybody with knowledge on the performance of these options care to shed some light? Thanks in advance!
There are a lot of ways to solve the problem you've described:
Application Logic
View-specific logic -- If the behavior is specific to a single view, then put the changes in the view.
Model-specific logic -- If the behavior is specific to a single model, then override the save() method for the model.
Middleware Logic -- If the behavior relates to multiple models OR needs to wrapped around an existing application, you can use Django's pre-save/post-save signals to add additional behaviors without changing the application itself.
Database Stored Procedures -- Normally a possibility, but Django's ORM doesn't use them. Not portable across databases.
Database Triggers -- Not portable from one database to another (or even one version of a database to the next), but allow you to control shared behavior across multiple (possibly non-Django) applications.
Personally, I prefer using either overriding the save() method, or using a Django signal. Using view-specific logic can catch you out on large applications with multiple views of the same model(s).
What you're describing sounds like "change data capture" to me.
I think the trade-offs might go like this:
Django pros: Middle tier code can be shared by multiple apps; portable if database changes
Django cons: Logically not part of the business transaction
MySQL pros: Natural to do it in a database
MySQL cons: Triggers are very database-specific; if you change vendors you have to rewrite
This might be helpful.
Related
Good Morning,
as from the title, i'd like to create a proprietary database to be integrate in a Typo3 website.
I'd like to receive some advise on which is the best solution:
- is it possible to create tables directly from Typo3?
- is it better creating a database, for example with MySQL and then integrate
it?
In the second case, how coud that be done?
are there other options?
I hope this is not an already answered topic, in case, please send me to it ( i could not find so much information.
Thanks in advance.
If I understand your question correctly, you want to add a custom Extension to TYPO3, containing custom tables. From a content side, this is perceived as a "database", right?
TYPO3 has a framework for that called Extbase. You can "kickstart" a TYPO3 extension with the "Extension Builder" https://typo3.org/extensions/repository/view/extension_builder by entering the "Model" (the data structure) via GUI and then you get all tables etc. automatically set up.
After that (aside from general TYPO3 knowledge), there is some coding involved. In theory, it's possible to make a "round trip" back to the extension builder from the code, but I've never done that.
You need to know / learn the specificities of extbase / php, which is is based on some "convention over configuration" rules and has some additional tweaks to plain PHP (functional comments). Here's a great resource: http://www.extbase-book.org/.
With that, you have great flexibility and powerful tooling to build almost anything inside TYPO3.
From a TYPO3 view it is best if you are able to hold your data in the TYPO3 database. You need to create an extension to handle your data. In TYPO3 an extension can define it's own tables and with updates of the extension updates in the datastructure are handled automatically.
Since version 8 there is a new layer (doctrine) and so it is possible to define further databases for individual tables. With some restrictions you are able to even use different database (-systems) for different tables.
Anyway you could program your own database interface to get and store your data independent from any TYPO3 restrictions, but then you need to handle everything on your own.
Using the TYPO3 core API will help you in multiple ways to handle your data without programming everything anew.
Especially if you use extbase (and the EXT:extensionbuilder) you will get a complete BE data handling, FE-Plugins with Fluid templates to present your data, even data management from the FE could be generated for you just by defining the datastructure. Of course versioning, workspace and timed visibility support are also available if you use TYPO3 structures which includes some (mostly invisible) fields aside from uid, hidden, deleted.
As a Front end developer I have less knowledge on Databases. But recently we started to develop an CRM application.
My question is, how feasible to migrate from one database to other. Lets say our application now supports mysql but later customer comes up with IBM's DB2 or sql lite. What are the things that we need to take care while developing to support easy migration ?
How cloud will help to solve my problem?
Just keep your data model separate from actual database calls and you should be good. Use a database abstraction layer in your model to make calls to the database. You'll only have to change the bottom layer for specific databases.
Some best practices:
Avoid DBMS specific features, data types and SQL/DDL constructs; keep to the SQL[92] standard. Test against e. g. SQLite, which keeps rather close to the standard.
Use an Entity Relationship Modeling tool that supports exporting DDL files for all targeted DBMS, or to standard SQL. Or write and maintain your DDL scripts by hand. Vendor specific tools usually don't do this.
Use the existing SQL abstraction layer that comes with your language/toolkit/environment, or implement one with an eye on portability (which reinvents the wheel another time).
Keep the logic in your application; the DB is for data only. Avoid triggers, stored procedures etc.
Generally apply the KISS principle to your data storage.
You may get more help on specific questions about general/abstract issues (not the implementation details, which belong here) over at Programmers.
I'm going to start a new project which is going to be small initially but may grow to big over the years. I'm strongly convinced that I'm going to use ASP.NET MVC with jQuery for UI. I want to go for MySQL as database for some reasons but worried on few things.
I'm totally new to Linq but it seems that it is easier to use once you are familiar with it.
First thing is that accessing data should be easy. So I thought I should use MySQL to Linq but somewhere I read that it is not directly supported but MySQL .NET connector adds support for EntityFramework. I don't know what are the pros and cons of it. DbLinq is what I also heard. I would love if I can implement repository pattern as it allows to apply filter in logic layer rather than in data access layer. Will it be possible if I use Entity Framework?
I'm also concerned about the performance. Someone told me that if we use Entity framework it fetches lot of data and then filter it. Is that right?
So questions basically are -
Is MySQL to Linq possible? If yes where can I get more details on it?
Pros and cons of using EntityFramework or DbLinq with MySQL?
Will it be easy to access data using EntityFramework or DbLinq with MySQL?
Will I be able to implement repository pattern which allows applying filter in logic layer rather than data access layer (when I use EntityFramework with MySQL)
Does it fetches hell lot of data from database and then apply filter on it?
If it sounds too many questions from my side in that case, if you can just let me know what you will do (with a considerable reason) in this situation as an experienced person in this area, that should answer my question.
As I am fan of ALT.NET I would recomend you to use NHibernate for your project instead of EntityFramework, you may google for the advantages over it, I am convinced you'll choose it.
Based on the points you've mentioned, then I would seriously consider going with MS SQL instead of MySQL initially and implementing LINQ-to-SQL instead of Entity Framework, and here's why:
The fact that you are anticipating a lot of traffic initially tells me that you need to think about where you plan to end up, rather than where to start. I have considerably more experience with MS SQL than I do with MySQL, but if you're talking about starting with the community version of MySQL and upgrading later, you're going to be incurring a significant expense anyway with the Enterprise version.
I have heard there is a version of LINQ that supports MySQL, but, unless things have changed recently, it is still in beta. I am completing an 18-month web-based project that used ASP.NET MVC 1.0, LINQ-to-SQL, JavaScript, jQuery, AJAX, and MS SQL. I implemented the repository pattern, view models, interfaces, unit tests and integration tests using WatiN. The combination of technologies worked very well for me, and I plan to go with the same combination for a personal project I'm developing.
When you get MS SQL with a hosting plan, you typically have the ability to create multiple databases from that single instance. It looks like they give you more storage because they give you multiple MySQL databases, but that's only because the architecture only supports the creation of one database per instance.
I won't use the Entity Framework for my ASP.NET MVC projects, because I wasn't crazy about ADO.NET in the first place. I don't want to have to open a connection, create a command object, populate a parameter collection, issue the execute method, and then iterate through a one-way reader object to get my data. Once you see how LINQ-to-SQL simplifies the process, you won't want to go back either. In the project I mentioned earlier, I have over 60 tables in the database with about 200 foreign key relationships. Because I used LINQ-to-SQL with the repository pattern in my data layer, I was able to build the application using not a single stored procedure. LINQ-to-SQL automatically protects against SQL injection attacks and support optimistic and pessimistic concurrency checking.
I don't know what your project is, but you don't want to get into a situation where you're going to have trouble scaling the application later. Code for the end result, not for the starting point, and you'll save yourself a lot of headaches later.
I have a small amount of experience using SVN on my development projects, and I have just as little experience with relational databases. I know the basic concepts like tables, and SQL statements, but I'm far from being an expert.
What I'd like to know is if there are any generic version control type systems like SVN, but that work with a database rather than files. I would like the same kind of features you get with SVN like the ability to create branches, create tags, and merge branches together. Rather than a revision number being associated to a version of a file repository it would be associated with a version of the database.
Are their any generic solutions available that can add this kind of functionality independent of the actual database schema? I'd be interested in solutions that work with MySQL or MS SQL Server.
I should also clarify that I'm trying to version control the data not the schema. I would expect the schema to remain constant. So really it seems like I want a way to create a log of all the INSERT, UPDATE, and DELETE requests sent the the database between each version of the data. That way any version could be recreated by resending all the SQL statements that have been saved up to the desired version.
You can script all your DDL, stored procedures and such to regular text files.
Then you can simply use SVN for database versioning.
I've never found a solution that works as well as Subversion, but here's a few things I've done that have helped:
Make scripts that will create the schema and populate any initial data. Then make an update script for each change after that. It's a fairly manual process, but it works. There's extra things that help like storing the current version number in a table in the db and making sure that the scripts are idempotent.
Store the full development db in Subversion. This doesn't usually work out too well for me if there is a lot of data or it is frequently changed. But in some projects is could work.
I keep and maintain create scripts in my version control system.
There are two things I can think of:
http://www.liquibase.org/ - provides a way of generally managing database changes. Creates files that get committed into source control, and it helps manage changes across different development databases, etc.
http://www.viget.com/extend/backup-your-database-in-git/ - this describes a strategy for backing up a database into source control, but the same strategy can be used just on the schema. In this scheme, the database would be in a separate area from your main code. (This can be used with other source control systems too.)
First a bit about the environment:
We use a program called Clearview to manage service relationships with our customers, including call center and field service work. In order to better support clients and our field technicians we also developed a web site to provide access to the service records in Clearview and reporting. Over time our need to customize the behavior and add new features led to more and more things being tied to this website and it's database.
At this point we're dealing with things like a Company being defined partly in the Clearview database and partly in the website database. For good measure we're also starting to tie the scripting for our phone system into the same website, which will require talking to the phone system's own database as well.
All of this is set up and working... BUT we don't have a good data layer to work with it all. We moved to Linq to SQL and now have two DBMLs that we can use, along with some custom classes I wrote before I'd ever heard of Linq, along with some of the old style ADO datasets. So yeah, basically things are a mess.
What I want is a data layer that provides a single front end for our applications, and on the back end manages everything into the correct database.
I had heard something about Entity Framework allowing classes to be built from multiple sources, but it turns out there can only be one database. So the question is, how could I proceed with this?
I'm currently thinking of getting the Linq To SQL classes all set for each database, then manually writing Linq compatible front ends that tie those together. Seems like a lot of work, and given Linq's limitations (such as not being able to refresh) I'm not sure it's a good idea.
Could I do something with Entity Framework that would turn out better? Should I look into another tool? Am I crazy?
The Entity Framework does give a certain measure of database independence, insofar as you can build an entity model from one database, and then connect it to a different database by using a different entity connect string. However, as you say, it's still just one database, and, moreover, it's limited to databases which support the Entity Framework. Many do, but not all of them. You could use multiple entity models within a single application in order to combine multiple databases using the Entity Framework. There is some information on this on the ADO.NET team blog. However, the Entity Framework support for doing this is, at best, in an early stage.
My approach to this problem is to abstract my use of the Entity Framework behind the Repository pattern. The most immediate benefit of this, for me, is to make unit testing very simple; instead of trying to mock my Entity model, I simply substitute a mock repository which returns IQueryables. But the same pattern is also really good for combining multiple data sources, or data sources for which there is no Entity Framework provider, such as a non-data-services-aware Web service.
So I'm not going to say, "Don't use the Entity Framework." I like it, and use it, myself. In view of recent news from Microsoft, I believe it is a better choice than LINQ to SQL. But it will not, by itself, solve the problem you describe. Use the Repository pattern.
if you want to use tools like Linq2SQl or EF and don't want to have to manage multiple DBMLS (or whaetever its called in EF or other tools), you could create views in your website database, that reference back to the ClearView or Phone system's DB.
This allows you to decouple your web site from their database structure. I believe Linq2Sql and EF can use a view as the source for an Entity. If they can't look at nHibernate.
This will also let you have composite entities that are pulled from the various data sources. There are some limitations updating views in SQL Server; however, you can define your own Instead of trigger(s) on the view which can then do the actual insert update delete statements.
L2S works with views, perfectly, in my project. You only need to make a small trick:
1. Add a secondary DB table to the current DB as a view.
2. In Designer, add a primary key attribute to a id field on the view.
3. Only now, add an association to whatever other table you want in the original DB.
Now, you might see the view available for the navigation.