how much work should we do in the database?
Ok I'm really confused as to exactly how much "work" should be done IN the database, and how much work had to be done instead at the application level?
I mean I'm not talking about obvious stuff like we should convert strings into SHA2 hashes at the application level instead of the database level..
But rather stuff that are more blur, including, but not limited to "should we retrieve the data for 4 column and do a uppercase/concatenation at the application level, or should we do those stuff at the database level and send the calculated result to the application level?
And if you could list any more other examples it would be great.
It really depends on what you need.
I like to do my business logic in the database, other people are religously against that.
You can use triggers and stored procedures/functions in SQL.
Links for MySQL:
http://dev.mysql.com/doc/refman/5.5/en/triggers.html
http://www.mysqltutorial.org/introduction-to-sql-stored-procedures.aspx
http://dev.mysql.com/doc/refman/5.5/en/stored-routines.html
My reasons for doing business logic in triggers and stored proces
Note that I'm not talking about bending the database structure towards the business logic, I'm talking about putting the business logic in triggers and stored procedures.
It centralizes your logic, the database is a central place, everything has to go through it. If you have multiple insert/update/delete points in your app (or you have multiple apps) you'll need to do the checks multiple times, if you do it in the database you only have to do the checks in one place.
It simplifies the application e.g., you can just add a member, the database will figure out if the member is already known and take the appopriate action.
It hides the internals of your database from the application, if you do all your logic in the application you will need intricate knowledge of your database in the application. If you use database code (triggers/procs) to hide that, you don't need to know every database detail in your app.
It makes it easier to restucture your database If you have the logic in your database, you can just change a tablelayout, replace the old table with a blackhole table, put a trigger on that and let the trigger do the updates to the new table, your app does not even need to know the database has changed, this allows legacy apps to keep working unchanged, whilst new apps can use the improved database layout.
Some things are easier in SQL
Some things work faster in SQL
I don't like to use (lots of and/or complicated) SQL code in my application, I like to put SQL code in a stored procedure/function and try to only put simple queries in my application code, that way I can just write code that explains what I mean in my application and let the database layer do the heavy lifting.
Some people disagree strongly with this, but this approach works well for me and has simplified debugging and maintenance of my applications a lot.
Generally, its a good practice to expect only "Data" from the Database. Its upto Application(s), to apply Business/Domain Logic and make sense of the data retrieved. Its highly recommended to do the following things in the Application Layer:
1) Formatting Date
2) Applying Math functions, such as interpolation/extrapolation, etc
3) Dynamic sorting (based on columns)
However, situations sometime warrant few things to be done at the database level.
In my opinion application should use data and database should provide them and that should be clear separation of concerns. So database gives records sorted, ordered and filtered according to requested conditions but it is up to application to apply some business logic to that records and "convert" them into something meaningful to the user.
For example, in my previous company we worked on big application for work time calculations. One of obvious functionalities in this kind of application is tracking vacation days of employees - how many days employee has per year, how many he used, how many left, etc. Basically we could write some triggers and procedures that would update those columns automatically. So when employee had his vacation days approved amount of days he applied for is taken from his "vacation pool" and added to "vacation days used". Pretty easy stuff but we decided to make it explicit on application level and boy, very soon we were happy we did it that way. Application had to be labor law compliant and it quickly turned out that not for all employees vacation days are calculated equally and sometimes vacation day can be not so vacation day at all but that is beside the point. Had we put this "easy" operation in database we had to version our database with every little change to a vacation days related logic and that would lead us straight to hell in customer support field due to a fact that it was possible to update only application without a need to update database (except clear "breakthrough" moments where database structure was changed of course).
In my experience I've found that many applications start with a straight-forward set of tables and then and handful of stored procedures to provide basic functionality. This works very well; it usually yields high performance and is simple to understand, it also mitigates any need for a complex middle-tier.
However, applications grow. It's not unusual to see large data-driven applications with thousands of stored procedures. Throw triggers into the mix and you have an application which, for anybody other than the original developers (if they're still working on it), is very difficult to maintain.
I will put a word in for applications which place most logic in the database - they can work well when you have some good database developers and/or you have a legacy schema which cannot be changed. The reason I say this is that ORMs take much of the pain out of this part of application development when you let them control the schema (if not, you often need to do a lot of fiddling to get it working).
If I was designing a new application then I would usually opt for a schema which is dictated by my application domain (the design of which will be in code). I would normally let an ORM handle the mapping between the objects and the database. I would treat stored procedures as exceptions to the rule when it came to data access (reporting can be much easier in sprocs than trying to coax an ORM into producing a complex output efficiently).
The most important thing to remember though, is that there are no "best practices" when it comes to design. It is up to you the developer to weigh up the pros and cons of each option in the context of your design.
Related
I'm building up an online (paid) service used for business administration purposes. The database is structured like so:
I have a contacts table filled with persons, contact info and the like. Then I have a few other tables holding information about payments, agreements and appointments. Also statistics like how much money was transferred this month, how many hours worth of appointments this month and the like.
I'm using MySQL (but could also go for MSSQL or some other service if necessary) and I had no formal training in any programming language whatsoever (yet).
I'm building a WPF application for acces to this database. Also planning on building an app so users can access their data and plan new appointments and register payments on the go.
I'm going to go for a login system to verify their right to login and use my service.
My question is about how to structure this. I'm not an SQL expert nor have I had any formal training in SQL or any other programming language. What I do know though is that my client-side app is almost out of the alpha stage.
So far I have come up with two ways to structure this.
1. Users get a seperate database.
My original idea was to give each user a seperate database, this makes it easier to provide people with statistics. Also it makes it easier to spread the workload through multiple, seperate servers. People would login to a master/main server, where their login information is stored, fetch their server info and programatically be 'redirected' to their own database. Spreading these databases also make it easier to provide individual back-ups to users.
The down-side of this is the sheer quantity of databases I'd have to manage. I'm planning on ending up with hundreds of thousands of users. Let's just say I want the system to be able to provide to an infinite amount of users.
2. Everything is stored in one database.
It's also possible to store everything in one database. This would make the database structure somewhat more complicated (while it also makes the whole a lot simpler). I'd have to add 'AND consumer_ID='" + MyID + "' to every query. (Which ofcourse is possible) and add a few tables to handle statistics per user.
It would be simpler to provide every user with the same database updates. Maintenance would be easier.
The down-side of this is that it makes it harder to spread the workload to seperate servers, I'd have to build something to make it possible that seperate servers mirror each other. Also I'd have to make sure that the workload is automatically divided between the servers, instead of simply going for: Fill server with X databases, then new server, fill, new etc.
I'm not in the luxury of hiring someone with any SQL training.
The most important thing for me now is that the system can be easily maintained while still being safe and reliable. I'm an amateur developer, going to college next year. I don't want to spend 50% of my time maintaining the database.
I think I got the major part of the details you might need, if you need anymore please ask for them.
I thank you in advance :)
Just go with solution 2. The downside of spreading the workload to many servers is fullfilled by "partitioning", look here for a starting point: http://dev.mysql.com/doc/refman/5.1/en/partitioning-overview.html
Partitioning would allow you for example to put all information of a table containing even IDs for consumers on the one, all other on the second server. Or whatever you want...
But i wouldn't start that complicated: do you need that now? It burdens you (either way) with such a big additional overhead! You can also look into the NoSQL database world for solutions that can be spread to as many servers as you want with low effort. You loose SQL and it's ACID features in the most cases; if you need those NoSQL is not an option.
We are at the beginning of a new project, and we are really wondering if we should use stored procedures in MySQL or not.
We would use the stored procedures only to insert and update business model entities. There are several tables which represent a model entity, and we would abstract it in those stored procedures insert/update.
On the other hand, we can call insert and update from the Model layer but not in MySQL but in PHP.
In your experience, Which is the best option? advantages and disadvantages of both approaches. Which is the fastest one in terms of high performance?
PS: It is is a web project with mostly read and high performance is the most important requisite.
Unlike actual programming language code, they:
not portable (every db has its own version of PL/SQL. Sometimes different versions of the same database are incompatible - I've seen it!)
not easily testable - you need a real (dev) database instance to test them and thus unit testing their code as part of a build is virtually impossible
not easily updatable/releasable - you must drop/create them, ie modify the production db to release them
do not have library support (why write code when someone else has)
are not easily integratable with other technologies (try calling a web service from them)
they use a language about as primitive as Fortran and thus are inelegant and laborious to get useful coding done, so it is difficult to express business logic, even though typically that is what their primary purpose is
do not offer debugging/tracing/message-logging etc (some dbs may support this - I haven't seen it though)
lack a decent IDE to help with syntax and linking to other existing procedures (eg like Eclipse does for java)
people skilled in coding them are rarer and more expensive than app coders
their "high performance" is a myth, because they execute on the database server they usually increase the db server load, so using them will usually reduce your maximum transaction throughput
inability to efficiently share constants (normally solved by creating a table and questing it from within your procedure - very inefficient)
etc.
If you have a very database-specific action (eg an in-transaction action to maintain db integrity), or keep your procedures very atomic and simple, perhaps you might consider them.
Caution is advised when specifying "high performance" up front. It often leads to poor choices at the expense of good design and it will bite you much sooner than you think.
Use stored procedures at your own peril (from someone who's been there and never wants to go back). My recommendation is to avoid them like the plague.
Unlike programming code, they:
render SQL injection attacks almost
impossible (unless you are are
constructing and executing dynamic
SQL from within your procedures)
require far less data to be sent over
the IPC as part of the callout
enable the database to far better
cache plans and result sets (this is
admittedly not so effective with
MySQL due to its internal caching
structures)
are easily testable in isolation
(i.e. not as part of JUnit tests)
are portable in the sense that they
allow you to use db-specific
features, abstracted away behind a
procedure name (in code you are stuck
with generic SQL-type stuff)
are almost never slower than SQL
called from code
but, as Bohemian says, there are plenty of cons as well (this is just by way of offering another perspectve). You'll have to perhaps benchmark before you decide what's best for you.
As for performances, they have the potential to be really performant in a future MySQL version (under SQL Server or Oracle, they are a real treat!). Yet, for all the rest... They totally blow up competition. I'll summarize:
Security: You can give your app the EXECUTE right only, everything is fine. Your SP will insert update select ..., with no possible leak of any sort. It means global control over your model, and an enforced data security.
Security 2: I know it's rare, but sometimes php code leaks out from the server (i.e. becomes visible to public). If it includes your queries, possible attackers know your model. This is pretty odd but I wanted to signal it anyway
Task force: yes, creating efficient SQL SPs requires some specific resources, sometimes more expensive. But if you think you don't need these resources just because you're integrating your queries in your client... you're going to have serious problems. I'd mention the analogy of web development: it's good to separate the view from the rest because your designer can work on their own technology while the programmers can focus on programming the business layer.
Encapsulating business layer: using stored procedures totally isolates the business where it belongs: the damn database.
Quickly testable: one command line under your shell and your code is tested.
Independence from the client technology: if tomorrow you'd like to switch from php to something else, no problem. Ok, just storing these SQL in a separate file would do the trick too, that's right. Also, good point in the comments about if you decide to switch sql engines, you'd have a lot of work to do. You have to have a good reason to do that anyway, because for big projects and big companies, that rarely happens (due to the cost and HR management mostly)
Enforcing agile 3+-tier developments: if your database is not on the same server than your client code, you may have different servers but only one for the database. In that case, you don't have to upgrade any of your php servers when you need to change the SQL related code.
Ok, I think that's the most important thing I had to say on the subject. I developed in both spirits (SP vs client) and I really, really love the SP style one. I just wished Mysql had a real IDE for them because right now it's kind of a pain in the ass limited.
Stored procedures are good to use because they keep your queries organized and allow you to perform a batch at once. Stored procedures are normally quick in execution because they are pre-compiled, unlike queries that are compiled on every run. This has significant impact in situations where database is on a remote server; if queries are in a PHP script, there are multiple communication between the application and the database server - the query is send, executed, and result thrown back. However, if using stored procedures, it only need to send a small CALL statement instead of big, complicated queries.
It might take a while to adapt to programming a stored procedure because they have their own language and syntaxes. But once you are used to it, you'll see that your code is really clean.
In terms of performance, it might not be any significant gain if you use stored procedures or not.
I will let know my opinion, despite my toughts possibly are not directly related to the question.:
As in many issues, reply about using Stored Procedures or an application-layer driven solution relies on questions that will drive the overall effort:
What you want to get.
Are you trying to do either batch operations or on-line operations? are they completely transactional? how recurrent are those operations? how heavy is the awaited workload for the database?
What you have in order to get it.
What kind of database technology you have? What kind of infrastucture? Is your team fully trained in the database technology? Is your team better capable of building a database-aegnostic solution?
Time for get it.
No secrets about that.
Architecture.
Is your solution required to be distributed onto several locations? is your solution required to use remote communications? is your solution working on several database servers, or possibly using a cluster-based architecture?
Mainteinance.
How much is the application required to change? do you have personal specifically trained for maintain the solution?
Change Management.
Do you see your database technology will change at a short, middle, long time? do you see will be required to migrate the solution frequently?
Cost
How much will cost to implement that solution using one or another strategy?
The overall of those points will drive the answer. So you have to care each of this points when making a decision about using or not any strategy. There are cases where using of stored procedures are better than application-layer managed queries, and others when, conducting queries and using an application-layer based solution is best.
Using of stored procedures tends to be more addequate when:
Your database technology isn't provided to change at a short time.
Your database technology can handle parallelized operations, table partitions or anything else strategy for divide the workload onto several processors, memory and resources (clustering, grid).
Your database technology is fully integrated with the stored proceduce definition language, that is, support is inside the database engine.
You have a development team who aren't afraid about using a procedural language (3rd. Generation language) for getting a result.
Operations you wanna achieve are built-in or supported inside the database (Exporting to XML data, managing data integrity and coherence appropiately with triggers, scheduled operations, etc).
Portability isn't an important issue and you do not whatch a technology change at a short time into your organization, even, it is not desirable. Generally, portability is seen like a milestone by the application-driven and layered-oriented developers. From my point of view, portability isn't an issue when your application isn't required to be deployed for several platforms, less when there are no reasons for making a technology change, or the effort for migrating all the organizational data is higher than the benefit for making a change. What you can win by using an application-layer driven approach (portability) you can loose in performance and value obtained from your database (Why to spend thousands of dollars for to get a Ferrari that you'll drive no more than 60 milles/hr?).
Performance is an issue. First: In several cases, you can achieve better results by using a single stored procedure call than multiple requests for data from another application. Moreover, some characteristics you need to perform may be built-in your database and its use less expensive in terms of workload. When you use an application-layer driven solution you have to take in account the cost associated to make database connections, making calls to the database, network traffic, data wrapping (i.e., using either Java or .NET, there is an implicit cost when using JDBC/ADO.NET calls as you have to wrap your data into objects that represents the database data, so instantiation has an associated cost in terms of processing, memory, and network when data comes from and goes to outside).
Using of application-layer driven solutions tends to be more addequate when:
Portability is an important issue.
Application will be deployed onto several locations with only one or few database repositories.
Your application will use heavy business-oriented rules, that need to be agnostic of the underlying database technology.
You have in mind to do change technology providers based on market tendencies and budget.
Your database isn't fully integrated with the stored procedure language that calls to the database.
Your database capabilities are limited and your requirement goes beyond what you can achieve with your database technology.
Your application can support the penalty inherent to external calls, is more transactional-based with business-specific rules and has to abstract the database model onto a business model for the users.
Parallelizing database operations isn't important, moreover, your database has not parallelization capabilities.
You have a development team which is not well-trained onto the database technology and is better productive by using an application-driven based technology.
Hope this may help to anyone asking himself/herself what is better to use.
I would recommend you don't use stored procedures:
Their language in MySQL is very crappy
There is no way to send arrays, lists, or other types of data structure into a stored procedure
A stored procedure cannot ever change its interface; MySQL permits neither named nor optional parameters
It makes deploying new versions of your application more complicated - say you have 10x application servers and 2 databases, which do you update first?
Your developers all need to learn and understand the stored procedure language - which is very crap (as I mentioned before)
Instead, I recommend to create a layer / library and put all your queries in there
You can
Update this library and ship it on your app servers with your app
Have rich data types, such as arrays, structures etc passed around
Unit test this library, instead of the stored procedures.
On performance:
Using stored procedures will decrease the performance of your application developers, which is the main thing you care about.
It is extremely difficult to identify performance problems within a complicated stored procedure (it is much easier for plain queries)
You can submit a query batch in a single chunk over the wire (if CLIENT_MULTI_STATEMENTS flag is enabled), which means you don't get any more latency without stored procedures.
Application-side code generally scales better than database-side code
If your database is complex and not a forum type with responses, but true warehousing SP will definitely benefit. You can out all your business logic in there and not a single developer is going to care about it, they just call your SP's. I have been doing this joining over 15 tables is not fun, and you cannot explain this to a new developer.
Developers also don't have access to a DB, great! Leave that up to database designers and maintainers. If you also decide that the table structure is going to get changed, you can hide this behind your interface. n-Tier, remember??
High performance and relational DB's is not something that goes together, not even with MySQL InnoDB is slow, MyISAM should be thrown out of the window by now. If you need performance with a web-app, you need proper cache, memcache or others.
in your case, because you mentioned 'Web' I would not use stored procedures, if it was data warehouse I would definitely consider it (we use SP's for our warehouse).
Tip:
Since you mentioned Web-project, ever though about nosql sort of solution? Also, you need a fast DB, why not use PostgreSQL? (trying to advocate here...)
I used to use MySql and my understanding of sql was poor at best, I spent a fair amount of time using Sql Server, I have a clear separation of a data layer and an application layer, I currently look after a server with 0.5 terabytes.
I have felt frustrated at times not using an ORM as development is really quick with stored procedures it is much slower. I think much of our work could have been sped up by using an ORM.
When your application reaches critical mass, the ORM performance will suffer, a well written stored procedure, will give you your results faster.
As an example of performance I collect 10 different types of data in an application, then convert that to XML, which I process in the stored procedure, I have one call to the database rather than 10.
Sql is really good at dealing with sets of data, one thing that gets me frustrated is when I see someone getting data from sql in a raw form and using application code to loop over the results and format and group them, this really is bad practice.
My advice is to learn and understand sql enough and your applications will really benefit.
Lots of info here to confuse people, software development is a evolutionary. What we did 20 years ago isn't best practice now. Back in the day with classic client server you wouldnt dream of anything but SPs.
It is absolutely horses for courses, if you are a big organisation with you will use multi tier, and probably SPs but you will care little about them because a dedicated team will be sorting them out.
The opposite which is where I find myself trying to quickly knock up a web app solution, that fleshes out business requirements, it was super fast to leave the developer (remote to me) to knock up the pages and SQL queries and I define the DB structure.
However complexity is growing and without an easy way to provide APIs, I am staring to use SPs to contain the business logic. I think it is working well and sensible, I control this because I can build logic and provide a simple result set for my offshore developer to build a front end around.
Should I find my software a phenomenal success, then more separation of concerns will occur and different implementations of n teir will come about but for now SPs are perfect.
You should know all the tool sets available to you and match them is wise to start with. Unless you are building an enterprise system to start with then fast and simple is best.
I would recommend that you stay away from DB specific Stored Procedures.
I've been through a lot of projects where they suddently want to switch DB platform and the code inside a SP is usually not very portable = extra work and possible errors.
Stored Procedure development also requires the developer to have access directly to the SQL-engine, where as a normal connection can be changed by anyone in the project with code-access only.
Regarding your Model/layer/tier idea: yes, stick with that.
Website calls Business layer (BL)
BL calls Data layer (DL)
DL calls whatever storage (SQL, XML, Webservice, Sockets, Textfiles etc.)
This way you can maintain the logic level between tiers. IF and ONLY IF the DL calls seems to be very slow, you can start to fiddle around with Stored Procedures, but maintain the original none-SP code somewhere, if you suddently need to transfer the DB to a whole new platform. With all the Cloud-hosting in the business, you never know whats going to be the next DB platform...
I keep a close eye on Amazon AWS of the very same reason.
I think there is a lot of misinformation floating around about database stored queries.
I would recommend using MySQL Stored Procedures if you're doing many static queries for data manipulation. Especially if you're moving things from one table to another (i.e. moving from a live table to a historical table for whatever reason). There are drawbacks of course in that you'll have to keep a separate log of changes to them (you could in theory make a table that just holds changes to the stored procedures that the DBA's update). If you have many different applications interfacing with the database, especially if say you have a desktop program written in C# and a web program in PHP, it might be more beneficial to have some of your procedures stored in the database as they are platform independent.
This website has some interesting information on it you may find useful.
https://www.sitepoint.com/stored-procedures-mysql-php/
As always, build in a sandbox first, and test.
Try to update 100,000,000 records on a live system from a framework, and let me know how it goes. For small apps, SPs are not a must, but for large serious systems, they are a real asset.
Our company has a database of 17,000 entries. We have used MS Access for over 10 years for our various mailings. Is there something new and better out there? I'm not a techie, so keep in mind when answering. Our problems with Access are:
-no record of what was deleted,
-will not turn up a name in a search if cap's or punctuation
is not entered exactly,
-is complicated for us to understand the de-duping process.
- We'd like a more nimble program that we can access from more than one dedicated computer.
The only applications I know of that are comparable to Access are FileMaker Pro, and the database component of the Open Office suite. FM Pro is a full-fledged product and gets good marks for ease of use from non-technical users, while Base is much less robust and is not nearly as easy for creating an application.
All of the answers recommending different databases really completely miss the point here -- the original question is about a data store and application builder, not just the data store.
To the specific problems:
PROBLEM 1: no record of what was deleted
This is a design error, not a flaw in Access. There is no database that really keeps a record of what's deleted unless someone programs logging of deleted data.
But backing up a bit, if you are asking this question it suggest that you've got people deleting things that shouldn't be deleted. There are two solutions:
regular backups. That would mean you could restore data from the last backup and likely recover most of the lost data. You need regular backups with any database, so this is not really something that is specific to Access.
design your database so records are never deleted, just marked deleted and then hidden in data entry forms and reports, etc. This is much more complicated, but is very often the preferred solution, as it preserves all the data.
Problem #2: will not turn up a name in a search if cap's or punctuation is not entered exactly
There are two parts to this, one of which is understandable, and the other of which makes no sense.
punctuation -- databases are stupid. They can't tell that Mr. and Mister are the same thing, for instance. The solution to this is that for all data that needs to be entered in a regularized fashion, you use all possible methods to insure that the user can only enter valid choices. The most common control for this is a dropdown list (i.e., "combo box"), which limits the choices the user has to the ones offered in the list. It insures that all the data in the field conforms to a finite set of choices. There are other ways of maintaining data regularity and one of those involves normalization. That process avoids the issue of repeatedly storing, say, a company name in multiple records -- instead you'd store the companies in a different table and just link your records to a single company record (usually done with a combo box, too). There are other controls that can be used to help insure regularity of data entry, but that's the easiest.
capitalization -- this one makes no sense to me, as Access/Jet/ACE is completely case-insensitive. You'll have to explain more if you're looking for a solution to whatever problem you're encountering, as I can't conceive of a situation where you'd actually not find data because of differences in capitalization.
Problem #3: is complicated for us to understand the de-duping process
De-duping is a complicated process, because it's almost impossible for the computer to figure out which record among the candidates is the best one to keep. So, you want to make sure your database is designed so that it is impossible to accidentally introduce duplicate records. Indexing can help with this in certain kinds of situations, but when mailing lists are involved, you're dealing with people data which is almost impossible to model in a way where you have a unique natural key that will allow you to eliminate duplicates (this, too, is a very complicated topic).
So, you basically have to have a data entry process that checks the new record against the existing data and informs the user if there's a duplicate (or near match). I do this all the time in my apps where the users enter people -- I use an unbound form where they type in the information that is the bare minimum to create a new record (usually some combination of lastname, firstname, company and email), and then I present a list of possible matches. I do strict and loose matching and rank by closeness of the match, with the closer matches at the top of the list.
Then the user has to decide if there's a match, and is offered the opportunity to create the duplicate anyway (it's possible to have two people with the same name at the same company, of course), or instead to abandon adding the new record and instead go to one the existing records that was presented as a possible duplicate.
This leaves it up to the user to read what's onscreen and make the decision about what is and isn't a duplicate. But it maximizes the possibility of the user knowing about the dupes and never accidentally creating a duplicate record.
Problem #4: We'd like a more nimble program that we can access from more than one dedicated computer.
This one confuses me. Access is multi-user out of the box (and has been from the very beginning, nearly 20 years ago). There is no limitation whatsoever to a single computer. There are things you have to do to make it work, such as splitting your database into to parts, one part with just the data tables, and the other part with your forms and reports and such (and links to the data tables in the other file). Then you keep the back end data file on one of the computers that acts as a server, and give a copy of the front end (the reports, forms, etc.) to each user. This works very well, actually, and can easily support a couple of dozen users (or more, depending on what they are doing and how well your database is designed).
Basically, after all of this, I would tend to second #mwolfe02's answer, and agree with him that what you need is not a new database, but a database consultant who can design for you an application that will help you manage your mailing lists (and other needs) without you needing to get too deep into the weeds learning Access (or FileMaker or whatever). While it might seem more expensive up front, the end result should be a big productivity boost for all your users, as well as an application that will produce better output (because the data is cleaner and maintained better because of the improved data entry systems).
So, basically, you either need to spend money upfront on somebody with technical expertise who would design something that allows you to do better work (and more efficiently), or you need to invest time in upping your own technical skills. None of the alternatives to Access are going to resolve any of the issues you've raised without significant investment in interface design to further the goals you have (cleaner data, easier to find information, etc.).
At the risk of sounding snide, what you are really looking for is a consultant.
In the hands of a capable programmer, all of your issues with Access are easily handled. The problems you are having are not the result of using the wrong tool, but using that tool less than optimally.
Actually, if you are not a techie then Access is already the best tool for you. You will not find a more non-techie friendly way to build a data application from bottom to top.
That said, I'd say you have three options at this point:
Hire a competent database consultant to improve your application
Find commercial off-the-shelf (COTS) software that does what you need (I'm sure there are plenty of products to handle mailings; you'll need to research)
Learn about database normalization and building proper MS Access applications
If you can find a good program that does what you want then #2 above will maximize your Return on Investment (ROI). One caveat is that you'll need to convert all of your existing data, which may not be easy or even possible. Make sure you investigate that before you buy anything.
While it may be the most expensive option up-front, hiring a competent database consultant is probably your best option if you need a truly custom solution.
SQL Server sounds like a viable alternative to your scenario. If cost is a concern, you can always use SQL Server Express, which is free. Full blown SQL Server provides a lot more functionality that might not be needed right away. Express is a lot simpler as the number of features provided with it are much smaller. With either version though you will have centralized store for your data and the ability to allow all transactions to be recorded in the transaction log. Also, both have the ability to import data from an Access Database.
The newest version of SQL Server is 2008 R2
You probably want to take a look at modern databases. If you're into Microsoft-based products, start with SQL Server Express
EDIT: However, since I understand that you're not a programmer yourself, you'd probably be better off having someone experienced look into your technical problem more deeply, like the other answer suggests.
It sounds like you may want to consider a front-end for your existing Access data store. Microsoft has yet to replace Access per se, but they do have a new tool that is a lot lower on the programming totem pole than some other options. Check out Visual Studio Lightswitch - http://www.microsoft.com/visualstudio/en-us/lightswitch.
It's fairly new (still in beta) but showing potential. With it, just as with any Visual Studio project, you can connect to an MS Access datasource and design a front-end to interact with it. The plus-side here is programming requirements are much lower than with straight-up Visual Studio (read: Wizards).
Given that replacing your Access DB will require some font-end programming, you may look into VistaDB. It should allow your front end to be created in .NET with an XCopy database on the backend without requiring a server. One plus is that it retains SQL Server syntax, so if you do decide to move to SQL Server you'll be one step ahead.
(Since you're not a techie and may not understand my previous statement, you might pass my answer on to the consultant/programmer/database guy who is going to do the work for you.)
http://www.vistadb.net/
I'm starting with Databases. I've been playing around with MySQL and Informix, but never had a real life project.
What is the real responsibility of a Database? Should we add Store procedures and functions to de Database or just let it to be a data repository with no logic?
What is the real responsibility of a Database?
A database at its core is a system to store and retrieve data. A CSV file on disk + suitable tools (e.g. Excel) is a simple example of this. In addition, a database might provide additional capabilities, such as transaction control, data integrity, and security.
Should we add Store procedures and functions to de Database or just let it to be a data repository with no logic?
What do you want from the database? If all you want is a "bit bucket", then by all means, store it in a plain file on disk and call it "the database". If you want a bit more than that, use a product that suits your needs. If you want to be able to query it using a 4GL like SQL, use MySQL. If you want transaction control, security, advanced query features, etc etc, use another DBMS if appropriate. Whatever product you choose, however, take advantage of that product. Otherwise you're wasting your time and money. Sure, you'll never use all of the features (only a subset will be useful to you), but if you use very few of them, you may as well downgrade to a simpler product.
If you're using Oracle, you can store procedures and functions (even better, whole packages) right there in the database alongside the data. The real question is, what do you need to write in those procedures and functions - business logic or presentation logic?
Personally, I usually prefer to keep business logic close to the data, whereas presentation logic is custom-made for each interface.
It is possible to create an API layer over your data so that no matter how your applications access your database, they will get a consistent view of it, and they will all modify it using a consistent mechanism. In other words, instead of writing the business logic multiple times (once for each interface), you write it once and once only, then re-use it everywhere.
There are two reasons I've heard why business logic should not be stored in the database:
1. Maintainability: it's hard to change. I never really understood this one. How hard is it to type CREATE OR REPLACE PACKAGE? I suspect it's just the burden of having to learn "yet another language".
2. Database independence: what works in Oracle won't work elsewhere. This is a biggie, and better minds than I have written about this one. Basically, if you really need it to be "database agnostic", you won't be able to use any of the advanced features of the database you bought, so you may as well just use the simplest/cheapest one you can find; in which case, you don't need it to work on every database anyway!
Generally it's considered good practice to not place business logic in your database. The main reason is maintainability. It is ok to use stored procedures still, but including business logic within those stored procedures makes your application harder to debug and update.
Including business logic in your database will also effectively tie you to using that one DBMS, and not allow the data layer to remain independent from your application. For example, you may encounter performance and scalability problems with one DB once your application is live, but due to business logic scattered throughout the db, migrating to a more scalable database will be time consuming at best.
If business logic is kept in application code (eg java or c#) and the data layer is abstracted using a data abstraction layer, and an ORM if language permits, then interchanging databases is much less problematic.
We should be striving for separation of concerns, and keeping business logic out of the db helps achieve that.
edit: There are also performance concerns which may dictate that stored procedures are a good place to keep business logic. Containing logic within the data tier (ie the sproc) in some cases reduces the many round trips between the data abstraction layer and the database, which can give a performance boost. I've worked on systems like this in the past, for this reason, but I've always found then difficult to maintain. The problem being that you can look through the classes and procedures to see the business logic and think that's it and you will not see how a particular bug or process can be occurring, then you'll find the stored procedure and see the other half of the business operation (a real pain when the sproc is a 1000 lines!)
As with many things, where you place your business logic depends on the particular problem you're trying to solve.
We have a lot of data around us which can be of great use to us. Ordered collection of information helps businesses to take more proper decisions. Databases are ordered storage of information.
Responsibility: In a common scenario, we can state that there is a lot of information around, ordered collection of information is called data, this information relates to an entity, and ordered collection of data is a database, information relating to a group of entities. Collection of these databases is a DBMS. Responsibility of the database is organizing information.
Stored procedures, functions are more like the business processes that you require in order to collect the data you desire to.
First starting point,
Begin:
Select database in {postgreSQL, MySQL, SQL Server(Express edition)} and install it.
Learn about Codd Rules, Normal forms, Good resource
Start learning SQL, write queries.
Understand the basics involved in schema creation.
Learn procedural language implementation in database.
Ask doubts in SO.
Is it considered crazy to store common SQL queries for my web app in a database for use in execution? Or is that common practice? Or is it impossible?
My thinking is, this way, I avoid hard-coding SQL into my application files, and add another level of abstraction.
Is this crazy? Is this what a stored procedure is? Or is that something else?
EDIT: The below answers are useful as a background for 'stored procedures', but didn't answer my core question: Is a 'stored procedure' just when I have a database table that contains queries that can be called? ie, something like this
INDEX | NAME | QUERY
1 | show_names | "SELECT names.first, names.last FROM names;"
2 | show_5_cities | "SELECT cities.city FROM cities LIMIT 0,5;"
etc.
Or is there a more complicated mechanism that encompasses the concept of stored procedures? Is my example an actual example of something people do?
Along with MUG4N's great reasons on why to use stored procedures, here are three more:
Security
You can grant access to your application to execute stored procedures while denying direct table access.
Think defense in depth. If your app is cracked, then they will be limited to executing ONLY the procedures you have defined. This means things like 'drop table' would be explicitly disallowed, unless, of course, you have a procedure to do that.
Conversely, if your app is cracked and you allow the app to have full access to your sql server, then one of two things will happen. Either your data disappears and/or the cracker easily get's a copy.
Unit Testing.
It's much easier to unit test your queries if you can hit them directly without having to go through the application itself.
In Flight Changes:
If you need to modify a query AFTER you have published your site, it's much easier to just make a proc change than redeploy code that may have undergone other changes since the last deployment. For example, let's say you have a page that isn't performing all that well. After evaluation, you determine that just changing the joins on a query will fix this. Modify the proc and go.
Well in my opinion you should definitly use stored procedures. And this is common practice!
Here are just two advantages of using stored procedures:
They will run in all environments, and there is no need to recreate the logic. Since they are on the database server, it makes no difference what application environment is used - the stored procedure remains consistent. If your setup involves different clients, different programming languages - the logic remains in one place. Web developers typically make less use of this feature, since the web server and database server are usually closely linked. However, in complex client-server setups, this is a big advantage. The clients are automatically always in sync with the procedure logic as soon as its been updated.
They can reduce network traffic. Complex, repetitive tasks may require getting results, applying some logic to them, and using this to get more results. If this only has to be done on the database server, there is no need to send result sets and new queries back and forth from application server to database server. Network traffic is a common bottleneck causing performance issues, and stored procedures can help reduce this. More often though, it is the database server itself that is the bottleneck, so this may not be much of an advantage.
The idea certainly has its appeal -- but the problems is, they are nearly impossible to scale.. I have never seen a scalable solution to maintaining stored procs (especially in MySQL) that has not made me shutter.
Since it seems you're heading the PHP/MySQL route, I'll give a few examples of my experience with stored procs in MySQL:
They are generally far less readable and far more difficult to write than PHP.
They make debugging a nightmare
Trying to figure out why changing a value in table_1 triggers a change in table_2 (if you're even lucky enough to recognize that this happens) is much more difficult to determine by looking through dozens of stored procedures than it is to, say, look in the Model that handles changes to table_1.
To my knowledge there is no standardized & automated way to integrate stored procs / triggers / etc into any revision control system
A stored procedure is just one or more SQL statements that are "pre-compiled" and live inside the database. You call them to return one or more rows of data, or to update, insert, or delete data.
If you tell us what web framework and database you are using, we can give you actual examples of how to call a stored procedure, or at least point you to an article or two to get you going.
You could also consider using an ORM framework, such as Hibernate. This will allow you to get away from dealing with SQL code altogether. I am a .Net developer, so I'm not sure what is available to you on the PHP/MySQL platform, but I am sure there is a lot out there to choose from.
You should think about it, when developing a commercial grade tiered application there is always people behind the database making it secure and reliable, other people are behind the application logic and other people behind the web code, so you can get the best of all working together.
Once the application has been designed, everyone start making their implementations, the db people give to the others some kind of API to use hiding the SQL, the developers won't have to think about it and focus on their code, i had worked as db developer and used some COM techniques to overcome the expansion and modification of the application logic or reuse, the database in these kind of products is too important to leave it in the wild so security it's a really serious issue.
But in most cases, web applications are made by web developers and they tend to have no design time, making big changes on the near time so they don't use stored procedures, also they don't even secure execution or try to leave security to the application leaving the database unprotected and prone to attacks.
If you're doing everything and changing your product too often you should avoid them since it will be double work and most of the times will be useless, once you stabilize your logic then you could start migrating your heavier queries to stored procedures.