First of all, sorry for the question title - I was unable to think about something better.
I have an interesting problem.
There are three web applications:
1. ApplicationA => example.com -> hosted in Germany
2. ApplicationB => example2.net -> hosted in Australia
3. ApplicationC => anotherexample.com -> hosted in United States
All of them are completely free however owner is planning to implement some paid options. The main issue is that applications are hosted on separate servers, in three different locations.
Now, if owner wants to implement any paid options, he needs to created unified invoicing system (as invoices numbering order needs to be correct).
So, we have situation:
1. user buys a premium option on example.com
2. another user buys a premium option on example2.net
3. third and fourth users buy extra options on anotherexample.com
So we have 4 invoices, so they numbering should be as following: 2011/01, 2011/02, 2011/03, 2001/04.
As mentioned above, the main issue is to unify invoicing system as applications use different databases and are hosted on different servers. Of course, invoices should be stored in application-specified database.
Theoretically we have only one issue: invoices numbers. Obviously we need to create a unified system for invoices storage.
There might be few possible issues:
there might be a lot of API requests to invoicing system
every single invoice needs to be stored in the database
while creating every single invoice in every external application, we need
to query invoicing system for the
latest invoice number.
I'm really interested in your knowing your approaches and suggestions. Any activity in this case is highly appreciated.
First, I would have the independent invoicing systems in example.com, example2.net and anotherexample.com all have their own internal primary keys for the invoices generated from within each of these systems. Each system should have its own independent copy of the invoicing logic because you don't want an outage on one server knocking out invoicing on every server.
Whenever you have a distributed system where local copies are creating records for something that will be amalgamated later, it's a good idea to use a GUID as the local primary key, or if you have a philosophical objection to GUID as PKs, create a GUID as a candidate key. This way, you can bring together invoices from all of your systems (and any future ones) without worrying about key collisions and you'll be able to track the combined records back to the source records, should you ever have to do that.
Next, you'll need an integrated invoice system where all of the invoice details are collected periodically. To facilitate this, you need processes on each local invoicing system pumping their own records up to the centralized system. Keep a flag on the invoices in the local systems to identify which invoices have been successfully uploaded - or if you have very high volumes, use a work-list table containing the invoice keys that still need transmitting instead of a flag right on the local invoice table.
The centralized invoice system will also want to have a source code on the combined invoice table so that you can easily tell which website created the invoice originally.
As to invoice numbers, I'm assuming from your question that the customer is a bit fussy about having proper sequencing of your invoice numbers. You can have the centralized system generate these numbers for you using a web service to pick up the next invoice ID. If the centralized service is down for any reason, you can still give the customer an "order reference" i.e. the GUID and just hold back on the invoice number until it can be generated through the central server. This should satisfy your customer's need for tight sequential invoice numbers while preserving the ability to operate multiple sites on multiple servers.
If your customer actually doesn't care about tight sequencing of the invoice numbers, then another alternative is to have the central system generate blocks of reserved invoice numbers and allot them to each website. When the website is getting low on its allotment, it asks the central server for another block. This gives you breathing room in the sequences in case there are communication difficulties.
In my opinion, it would be better to use an encoded invoice number. This way, you won't need to worry about number order getting mixed up.
For example, invoices can be prefixed with the country domain like de297,de298, etc., for invoices from Germany.
And going one step further, I'd incorporate the year as well. Thus it would reset at the beginning of each year and still maintain no conflicts, while at the same time keep the invoice number within a small length.
Related
I have an application in which we want to provide the functionality using which user can add/update/delete the columns of different tables. My approach is to create a different database for each client so that their changes specific to tables will remain in their database.
Since each client will have their own database, I wonder how can I manage authentication and authorization? Do I need to create a different database for that as well? Will it affect the performance of the application?
Edit: The approach that I am planning to use for authentication and authorization is to create an additional field called "Account" on the login page. This account name will guide the program to connect it to correct database. And each database will have it's own users to authenticate.
The answer to your question is of course (and unfortunately) Yes and No. :)
This is known as multi-tenant data architecture.
Having separate databases can definitely be a great design option however so can using one database shared with all of your clients/customers and you will need to consider many factors before choosing.
Each design has pluses and minuses.
Here are your 3 essential choices
1) Each customer shares the same database and database tables.
2) Each customer shares the same database but they get their own schema inside the database so they each get their own set of tables.
3)Each customer gets their own database.
One major benefit (that I really like) to the separate database approach is data security. What I mean by this is that every customer gets their own database and because of this they will edit/update/delete just their database. Because of this, there is no risk in end users overriding other users data either due to programmatic error on your part or due to a security breach in your application.
When all users are in the same database you could accidentally pull and expose another customers data. Or, worse, you could expose a primary key to a record on screen and forget to secure it appropriately and a power user could override this key very easily to a key that belongs to another customer thus exposing another clients data.
However, lets say that all of your customers are actually subsidieries of 1 large company and you need to roll up financials every day/week/month/year etc.
If this is the case, then having a database for every client could be a reporting nightmare and having everyone in a single database sharing tables would just make life so much easier. When it comes time to report on your daily sales for instance, its easier to just sum up a column then go to 10,000 databases and sum them up. :)
So the answer definitely depends on your applicaton and what it will be doing.
I work on a large enterprise system where we have tens of thousands of clients in the same database and in order to support this we took very great care to secure all of our data very carefully.
I also work on a side project in my spare time which supports a database per customer multi-tenant architecture.
So, consider what your application will do, how you will backup your data, do you need to roll up data etc and this will help you decide.
Heres a grea article on MSDN for this:
https://msdn.microsoft.com/en-us/library/aa479086.aspx
Regarding your question about authentication.
Yes, having a separate database for authentication is a great design. When a customer authenticates, you will authenticate them off of your authentication database and they will receive the connectionstring to their database as part of this authentication. Then all data from that point comes from that clients database.
Hope this was helpful.
Good luck!
I'm trying to create a payment system for my website. The website is a market place for 3d printing blueprint. Users buy credits on my website. When a user purchase a 3d printing blueprint uploaded by other user, it creates a new tuple or a row in the 'purchased' table while deducting credit in user credit table. Here's the important part. My gut tells me to use event scheduler to mark rows of purchased as payed every month and wire the sum of money earned by each seller. My worry is the table will grow infinitely as months pass by.
Is this the right implementation?
Or can I somehow create a new table each month that holds transactions for only this month?
Is there a Nosql equivalent to this?
Stripe.com or Braintree.com might be good options for you.
It is not advisable to create or roll your own payments implementation. These established services not only handle the PCI compliance aspect of payments, but they also have direct support for the use case you're asking about.
In an effort to answer your question further - it's probably not going to be an issue from the stand point of performing inserts into this MySQL table or in terms of iterating across it for batch processing. Querying on the other hand will become more onerous as the data set gets very large.
You can use partitioning in MySQL and perform the partitioning based on date but I doubt this is something you should spend your time accomplishing at this point. Wait until your site blows up and is super popular then come back and update your schema and configuration to meet your actual usage demands.
It's worth noting that you'll also want to make sure to take regular backups of something as important of payments information. Typically you'd also see at least one replica for something this critical.
Again I don't think you should try and solve this yourself. Just pay for a service that does this for you and focus on building the best 3d blueprint marketplace.
I need to architect a database and service, I have resource that I need to deliver to the users. And the delivery takes some time or requires user to do some more job.
These are the tables I store information into.
Table - Description
_______________________
R - to store resources
RESERVE - to reserve requested resources
HACK - to track some requests that couldn`t be made with my client application (statistics)
FAIL - to track requests that can`t be resolved, but the user isn't guilty (statistics)
SUCCESS - to track successfully delivery (statistics)
The first step when a user requests resouce
IF (condition1 is true - user have the right to request resource) THEN
IF (i've successfully RESERVE-d resource and commited the transaction) THEN
nothing to do more
ELSE
save request into FAIL
ELSE
save request into HACK
Then the second step
IF (condition2 is true - user done his job and requests the reserved resource) THEN
IF (the resource delivered successfully) THEN
save request into SUCCESS
ELSE
save request into FAIL
depending on application logic move resource from RESERVE to R or not
ELSE
save request into HACK, contact to the user,
if this is really a hacker move resource from RESERVE to R
This is how I think to implement the system. I've stored transactions into the procedures. But the main application logic, where I decide which procedure to call are done in the application/service layer.
Am I on a right way, is such code division between the db and the service layers normal? Your experienced opinions are very important.
Clarifying and answering to RecentCoin's questions.
The difference between the HACK and FAIL tables are that I store more information in the HACK table, like user IP and XFF. I`m not going to penalize each user that appeared in that table. There can be 2 reasons that a user(request) is tracked as a hack. The first is that I have a bug (mainly in the client app) and this will help me to fix them. The second is that someone does manually requests, and tries to bypass the rules. If he tries 'harder' I'll be able to take some precautions.
The separation of the reserve and the success tables has these reasons.
2.1. I use reserve table in some transactions and queries without using the success table, so I can lock them separately.
2.2. The data stored in success will not slow down my queries, wile I'm querying the reserve table.
2.3. The success table is kind of a log for statistics, that I can delete or move to other database for future analyse.
2.4. I delete the rows from the reserve after I move them to the success table. So I can evaluate approximately the max rows count in that table, because I have max limit for reservations for each user.
The points 2.3 and 2.4 could be achieved too by keeping in one table.
So are the reasons 2.1 and 2.2 enough good to keep the data separately?
The resource "delivered successfully" mean that the admin and the service are done everything they could do successfully, if they couldn't then the reservation fails
4 and 6. The restrictions and right are simple, they are like city and country restrictions, The users are 'flat', don't have any roles or hierarchy.
I have some tables to store users and their information. I don't have LDAP or AD.
You're going in the right direction, but there are some other things that need to be more clearly thought out.
You're going to have to define what constitutes a "hack" vs a "fail". Especially with new systems, users get confused and it's pretty easy for them to make honest mistakes. This seems like something you want to penalize them for in some fashion so I'd be extremely careful with this.
You will want to consider having "reserve" and "success" be equivalent. Why store the same record twice? You should have a really compelling reason do that.
You will need to define "delivered successfully" since that could be anything from an entry in a calendar to getting more pens and post notes.
You will want to define your resources as well as which user(s) have rights to them. For example, you may have a conference room that only managers are allowed to book, but you might want to include the managers' administrative assistants in that list since they would be booking the room for the manager(s).
Do you have a database of users? LDAP or Active Directory or will you need to create all of that yourself? If you do have LDAP or AD, can use something like SAML?
6.You are going to want to consider how you want to assign those rights. Will they be group based where group membership confers the rights to reserve, request, or use a given thing? For example, you may only want architects printing to the large format printer.
For clarification, I don't want to store metrics on the database itself - rather, I want to build a database to store metrics from the various controls we measure at my organization for easy reporting. A little background: as manager, I pull metrics from various applications - our two ticketing systems (yeah, I know), our phone system, alerts from our event management software (i.e., Nagios), etc. I report on these on a weekly basis and keep an Excel spreadsheet with the historical data. The spreadsheet is really, really big and inflexible.
I'm not new at writing web apps, but I'm still new to the database design arena. I want to code an app with some awesome rickshaw javascript graphs for some great historical data (and to wow the senior management team with crazy colors and interactivity).
Where do I start with the database? I could create one table for all metrics, but how to index those into the various types (for instance, phone metrics has abandon rate, total inbound calls, total outbound calls, total time on call, average talk time, average hold time, max hold time, etc.). That's one messy, unorganized table.
I could create one table for each type (phone, ticket, event, etc.) but that seems hard to add metrics to the pile later.
I'm hoping someone here has some experience and can give me some pointers on what direction I should head.
PS: It will need to be SQLite or MySQL, just due to the resources I have available at this time.
MySQL design for such a system can be made considering following:
A Table for each type of metrics group for example an entity of ticket system can be a single ticket
If a ticket is connected to single user you may include user name in the previous ticket table otherwise to keep it flexible i would say create a table for each connected element for example ticket is assinged to staff and has multiple telephone calls associated to it so you would need calls table and staff table.
In order to map multiple items create mapping tables for example stafftickets and ticketcalls to associate staff with multiple tickets and tickets with multiple calls
Once you have defined these entities then you can sit on mySQL phpmyadmin and create tables that will work.
For charting side of things use D3.js and just spit out json and use javascript or json2 to bind it to your graphs etc.
I have two separate newsletter databases right now. One is a simple opt-in where a customer can just give their email address to receive our mailings, and the other is a full-blown customer account database (username, password, address, order history, etc.). Both end up getting basically the same newsletters (we do some segmentation and dynamic content where possible). Additionally, I should mention that the opt-in table has some one-off columns where different marketing initiatives have gathered data about the contact (where the contact registered [online/offline] and what promotion it was apart of, their mailing address if we have it, etc.).
I've had these two lists set up for years, but it is getting harder and harder to manage as a customer moves from a simple opt-in to creating an account, they update their email address, etc.. Once the user creates the account, it would make sense that they'll manage their newsletter settings from in there. But sometimes an existing customer enters their email address in the opt-in (or vice versa) and it starts getting complicated to manage the duplicates. We also have to deal with a lot of duplicate data between the two databases (and going across multiple databases, not just tables, is a challenge).
Adding to the complications, we send our newsletter using a third party service. We have some processes that run every-so-often that will do some two-way syncing, so I have to do some juggling each time to determine which database I pull changes from/push changes to, which source was updated last, etc.. If the contact is moving from the opt-in list to the customer list, I have to push changes, deactivate the old relationship; if they added themselves to the opt-in and we already have their email in the other database, I set a flag to ignore them -- it's all a headache.
Now this has to be a pretty common situation. Almost every ecommerce site I go to has at least two distinct lists like this. My question is how are they handling it? What's the best practice?
The way I see it, I have two options:
Use two different tables in the same database (which is pretty close to how I'm doing it now). This allows me to continue to keep things separate and organized, but efficiently check both lists as the same time for conflicting/duplicate information. I still have many of the same problems (moving the relationship or defining a "master record", duplicate data, propagating changes, etc.), but it should simplify a few things at least.
Create a single table (call it NewsletterContacts or something) for all contacts to share. Opt-in users would probably be added directly, but entries from the customer account table would would have to be synced. There are a number of downsides to this too, though. Any change to the customer account table would have to be pushed to NewsletterContacts before it could be pushed to our email software. I'd have some very mixed content in that database -- some records would have an address and source and such, others would just have an email address, and then others would point to a customer account table that would have all types of other information.
TL,DR: How do you manage two separate newsletter lists when you know there will be duplicates?