So, I have an existing db with some tables for a class of users. We're building a more general app to handle multiple things the company does and this class of users, call them hosts, is a general type used by multiple programs in our company. We want to (eventually) migrate into a centralized app as now we have several. However, we don't have the time to do it completely right now. I need to build a login system for these hosts and I'd like to begin to migrate to this new system with that. I can't figure out a reasonable way to move those tables that are in the legacy DB to the new DB, which (of course) resides on a different server, with out wanting to stab my own eyes out after 30 seconds of having to deal with this. The legacy db has many reports the rely on joining on the current hosts tables.
The only things I can come up with don't seem like very good ideas. Those being, writing to both dbs from both apps (pointless data duplication prone to syncing problems), provide an API from the new app and mash the data coming back together with record sets (just seems... wrong).
Anyone have any ideas how to deal with this?
It has it's limitations, but the FEDERATED storage engine might be of assistance.
Related
For a client I'm going to deliver a SaaS solution, SaaS in that matter it's still closed to a limited clients that has to sign a contract with us, so not shared world wide and the client base will be around 5-10 companies.
Our first client, the pilot client so to speak, has it as a requirement that they can perform SQL queries (read mode only) on the data, so they can make analysis in Excel along with what our application serves.
My question is that I would for maintain reasons prefer to serve everything in the same codebase, but I'm wondering how I can make sure, clients can access other clients SQL records?
I'm using Laravel so the solution for different installations would be to make everything in maintainable packages and upgrade all installations from there, but this can grow to a lot of work.
How to have the solution with only one solution I'm still not sure, maybe it is to have a separate database per client? That would require a central database to point them to the right database of course, or maybe only have some of the tables in another database, but it already sounds like a mess to me
In Laravel it is possible to have multiple database connections. As such your thoughts on giving the clients thier own database is going to be the most secure option.
Have your default database be your main application database which will be settings/auth etc.
For each client store their personal data in a separate database per client and only allow them to query this database.
Although I don't know the specifics of your application my true suggestion is to avoid the SQL queries completely and build an API.
Your SaaS clients should not have to be concerned with the internal implementation of your database structure. A well built API gives you freedom to modify the database as needed and the SaaS client the peace of mind that their "interface" is not in a technically permanent state of flux.
Scenario:
Building a commercial app consisting in an RESTful backend with symfony2 and a frontend in AngularJS
This app will never be used by many customers (if I get to sell 100 that would be fantastic. Hopefully much more, but in any case will be massive)
I want to have a multi tenant structure for the database with one schema per customer (they store sensitive information for their customers)
I'm aware of problem when updating schemas but I will have to live with it.
Today I have a MySQL demo database that I will clone each time a new customer purchase the app.
There is no relationship between my customers, so I don't need to communicate with multiple shards for any query
For one customer, they can be using the app from several devices at the time, but there won't be massive write operations in the db
My question
Trying to set some functional tests for the backend API I read about having a dedicated sqlite database for loading testing data, which seems to be good idea.
However I wonder if it's also a good idea to switch from MySQL to SQLite3 database as my main database support for the application, and if it's a common practice to have one dedicated SQLite3 database PER CLIENT. I've never used SQLite and I have no idea if the process of updating a schema and replicate the changes in all the databases is done in the same way as for other RDBMS
Is this a correct scenario for SQLite?
Any suggestion (aka tutorial) in how to achieve this?
[I wonder] if it's a common practice to have one dedicated SQLite3 database PER CLIENT
Only if the database is deployed along with the application, like on a phone. Otherwise I've never heard of such a thing.
I've never used SQLite and I have no idea if the process of updating a schema and replicate the changes in all the databases is done in the same way as for other RDBMS
SQLite is a SQL database and responds to ALTER TABLE and the like. As for updating all the schemas, you'll have to re-run the update for all schemas.
Schema synching is usually handled by an outside utility, usually your ORM will have something. Some are server agnostic, some only support specific servers. There are also dedicated database change management tools such as Sqitch.
However I wonder if it's also a good idea to switch from MySQL to SQLite3 database as my main database support for the application, and
SQLite's main advantage is not requiring you to install and run a server. That makes sense for quick projects or where you have to deploy the database, like a phone app. For server based application there's no problem having a database server. SQLite's very restricted set of SQL features becomes a disadvantage. It will also likely run slower than a server database for anything but the simplest queries.
Trying to set some functional tests for the backend API I read about having a dedicated sqlite database for loading testing data, which seems to be good idea.
Under no circumstances should you test with a different database than the production database. Databases do not all implement SQL the same, MySQL is particularly bad about this, and your tests will not reflect reality. Running a MySQL instance for testing is not much work.
This separate schema thing claims three advantages...
Extensibility (you can add fields whenever you like)
Security (a query cannot accidentally show data for the wrong tenant)
Parallel Scaling (you can potentially split each schema onto a different server)
What they're proposing is equivalent to having a separate, customized copy of the code for every tenant. You wouldn't do that, it's obviously a maintenance nightmare. Code at least has the advantage of version control systems with branching and merging. I know only of one database management tool that supports branching, Sqitch.
Let's imagine you've made a custom change to tenant 5's schema. Now you have a general schema change you'd like to apply to all of them. What if the change to 5 conflicts with this? What if the change to 5 requires special data migration different from everybody else? Now let's imagine you've made custom changes to ten schemas. A hundred. A thousand? Nightmare.
Different schemas will require different queries. The application will have to know which schema each tenant is using, there will have to be some sort of schema version map you'll need to maintain. And every different possible query for every different possible schema will have to be maintained in the application code. Nightmare.
Yes, putting each tenant in a separate schema is more secure, but that only protects against writing bad queries or including a query builder (which is a bad idea anyway). There are better ways mitigate the problem such as the view filter suggested in the docs. There are many other ways an attacker can access tenant data that this doesn't address: gain a database connection, gain access to the filesystem, sniff network traffic. I don't see the small security gain being worth the maintenance nightmare.
As for scaling, the article is ten years out of date. There are far, far better ways to achieve parallel scaling then to coarsely put schemas on different servers. There are entire databases dedicated to this idea. Fortunately, you don't need any of this! Scaling won't be a problem for you until you have tens of thousands to millions of tenants. The idea of front loading your design with a schema maintenance nightmare for a hypothetical big parallel scaling problem is putting the cart so far before the horse, it's already at the pub having a pint.
If you want to use a relational database I would recommend PostgreSQL. It has a very rich SQL implementation, its fast and scales well, and it has something that renders this whole idea of separate schemas moot: a built in JSON type. This can be used to implement the "extensibility" mentioned in the article. Each table can have a meta column using the JSON type that you can throw any extra data into you like. The application does not need special queries, the meta column is always there. PostgreSQL's JSON operators make working with the meta data very easy and efficient.
You could also look into a NoSQL database. There are plenty to choose from and many support custom schemas and parallel scaling. However, it's likely you will have to change your choice of framework to use one that supports NoSQL.
Little question, I'm developing a saas software (erp).
I designed it with 1 database per account for these reasons :
I make a lot of personalisation, and need to add specific table columns for each account.
Easier to manage db backup (and reload data !)
Less risky : sometimes I need to run SQL queries on a table, in case of an error with bad query (update / delete...), only one customer is affected instead of all of them.
Bas point : I'm turning to have hundreds of databases...
I'm hiring a company to manage my servers, and they said that it's better to have only one database, with a few tables, and put all data in the same tables with column as id_account. I'm very very surprised by these words, so I'm wondering... what are your ideas ?
Thanks !
Frederic
The current environment I am working in, we handle millions of records from numerous clients. Our solution is to use Schema to segregate each individual client. A schema allows you to partition your clients into separate virtual databases while inside a single db. Each schema will have an exact copy of the tables from your application.
The upside:
Segregated client data
data from a single client can be easily backed up, exported or deleted
Programming is still the same, but you have to select the schema before db calls
Moving clients to another db or standalone server is a lot easier
adding specific tables per client is easier (see below)
single instance of the database running
tuning the db affects all tenants
The downside:
Unless you manage your shared schema properly, you may duplicate data
Migrations are repeated for every schema
You have to remember to select the schema before db calls
hard pressed to add many negatives... I guess I may be biased.
Adding Specific Tables: Why would you add client specific tables if this is SAAS and not custom software? Better to use a Postgres DB with a Hstore field and store as much searchable data as you like.
Schemas are ideal for multi-tenant databases Link Link
A lot of what I am telling you depends on your software stack, the capabilities of your developers and the backend db you selected (all of which you neglected to mention)
Your hardware guys should not decide your software architecture. If they do, you are likely shooting yourself in the leg before you even get out of the gate. Get a good senior software architect, the grief they will save you, will likely save your business.
I hope this helps...
Bonne Chance
I'm about to create my first cloud based app using PHP and MYSQL. I'm in a limbo and can't seem to figure out whether I should use one database which stores everything, or should I setup a dedicated database for each user that signs up to my service.
The application itself will be recording hundreds and thousands of records on daily basis. So having a one big shared database could get heavy. It could also lead to performance issues when querying the database. On the other hand, having a dedicated database for each user could potentially lead to maintenance problems (E.g backing up and keeping track of each database instance).
Could anyone please advise me on the best approach? Or is there a better way of doing this?
My site has a MySQL database with about 50 tables. I work hard to make it as safe and secure as possible.
Per our development plan, we will be adding a forum in the not too distant future.
I'm unsure about whether it is better to have the forum in its own database, or to insert all its tables into our existing database. I've listed the pros and cons of both approaches below as I understand them, and would appreciate some advice from those more knowledgeable and experienced than I, which is nearly all of you :-)
Merged into Existing Database
Pros
integrating forum data into existing site is easier (example: using forum thread tags to match threads to site pages and automatically display links to relevant discussions)
can merge existing users table into forum so users need not re-register to begin using the forum
all-in-one backups
Cons
I've instantly added a huge amount of new code, some of which has database access, and all of which is a much higher profile target for shenanigans, meaning my original database is now placed at much more risk of attack
updating the forum software will be more hands-on, as it will not be a straight database flop
Separate Databases for Forum and Main Site
Pros
easy install, testing, upgrade, tear down of forum
forum database security holes don't place my main site at risk (and vice-versa)
Cons
integration into existing site requires querying two databases at once. I suspect this would be fairly more difficult to program.
users would have to re-register on the forum
backing up 2 databases rather than one (this is a minor con, but it is a con)
Your thoughts? :-)
Querying from 2 databases:
select db1.a.field1, db2.b.field2 from db1.a
inner join db2.b on (db1.a.id = db2.b.id);
Just make sure your connect string has access two both databases.
And both databases need to be on the same machine.
The approach that has proven itself for me is:
Install the forum as separate system
write a thin layer to share login (if both use open id or something similar, be happy)
As time goes I slowly and carefully merge the two system where it make sense, usually it does not. I love to share data between the two databases using views.
Merged into Existing Database
Pros
Integrating forum data into existing site is easier .. [nope. from a coding perspective there isn't any difference between running a query on one database versus another. Also, your queries themselves can cross databases.]
can merge existing users table into forum so users need not re-register to begin using the forum. [nope. Yes you can do this, but you could do it even if the forum tables aren't in this database. So it's a wash]
all-in-one backups. [I think you're grasping here. Whether one database or two, the backup procedures are the same. The only difference is you have 1 or 2 files]
Cons
I've instantly added a huge amount of new code, some of which has database access, and all of which is a much higher profile target for shenanigans, meaning my original database is now placed at much more risk of attack. [maybe. IF the new code uses dynamic sql, and/or fails to use parameterized queries, then it's screwed regardless. Further, if your data layer allows the user the queries execute under full access to your server, which unfortunately seems to be par for the course on most applications, then it doesn't matter if the tables are in the same database or not. Interestingly the MySql site was cracked in this manner a month ago.]
updating the forum software will be more hands-on, as it will not be a straight database flop. [? I'm not entirely sure what you mean by this. I've never heard the term "flop" used in this context.]
Separate Databases for Forum and Main Site
Pros
easy install, testing, upgrade, tear down of forum [No.. You have the same issues regardless of what database it lives in]
forum database security holes don't place my main site at risk (and vice-versa). [depends on the types of holes and exactly how security was implemented]
Cons
integration into existing site requires querying two databases at once. I suspect this would be fairly more difficult to program. [It's not. It has exactly the same level of complexity. Also your queries can cross databases.]
users would have to re-register on the forum [Nope. You can reuse the same user table in the other table]
backing up 2 databases rather than one (this is a minor con, but it is a con). [I would disagree, but then again we have dozens of databases on our servers and all of our backups are automated. Heck, as soon as we create one the maintenance plans automatically add it to the nightly backup schedule so it's not even a thought.]
Quite frankly, I'd say the only potential issue is in how the new forum stuff accesses the database and exactly what user rights that account needs in order to do its job. If done right then there is no issue; but if it's done way wrong then the only real protection would be to place the forum software on it's own database server... and even then it might cause problems.
But this should be identified by a proper security audit.