I have two separate newsletter databases right now. One is a simple opt-in where a customer can just give their email address to receive our mailings, and the other is a full-blown customer account database (username, password, address, order history, etc.). Both end up getting basically the same newsletters (we do some segmentation and dynamic content where possible). Additionally, I should mention that the opt-in table has some one-off columns where different marketing initiatives have gathered data about the contact (where the contact registered [online/offline] and what promotion it was apart of, their mailing address if we have it, etc.).
I've had these two lists set up for years, but it is getting harder and harder to manage as a customer moves from a simple opt-in to creating an account, they update their email address, etc.. Once the user creates the account, it would make sense that they'll manage their newsletter settings from in there. But sometimes an existing customer enters their email address in the opt-in (or vice versa) and it starts getting complicated to manage the duplicates. We also have to deal with a lot of duplicate data between the two databases (and going across multiple databases, not just tables, is a challenge).
Adding to the complications, we send our newsletter using a third party service. We have some processes that run every-so-often that will do some two-way syncing, so I have to do some juggling each time to determine which database I pull changes from/push changes to, which source was updated last, etc.. If the contact is moving from the opt-in list to the customer list, I have to push changes, deactivate the old relationship; if they added themselves to the opt-in and we already have their email in the other database, I set a flag to ignore them -- it's all a headache.
Now this has to be a pretty common situation. Almost every ecommerce site I go to has at least two distinct lists like this. My question is how are they handling it? What's the best practice?
The way I see it, I have two options:
Use two different tables in the same database (which is pretty close to how I'm doing it now). This allows me to continue to keep things separate and organized, but efficiently check both lists as the same time for conflicting/duplicate information. I still have many of the same problems (moving the relationship or defining a "master record", duplicate data, propagating changes, etc.), but it should simplify a few things at least.
Create a single table (call it NewsletterContacts or something) for all contacts to share. Opt-in users would probably be added directly, but entries from the customer account table would would have to be synced. There are a number of downsides to this too, though. Any change to the customer account table would have to be pushed to NewsletterContacts before it could be pushed to our email software. I'd have some very mixed content in that database -- some records would have an address and source and such, others would just have an email address, and then others would point to a customer account table that would have all types of other information.
TL,DR: How do you manage two separate newsletter lists when you know there will be duplicates?
Related
So currently I'm working on a web application for a game and stumbled upon the following "problem"/question: Is it actually safer to have an individual table for the players/users and administrators?
Personally I think it's not and storing them both in one table would also be easier and more efficient, since every administrator is considered a player as well, but some people tell me it's safer to separate them without a clear reason why.
As for now I have them both stored in one table and am using a role based permission system.
By separating them you could control permissions to the tables with more granularity, for example limiting access to the administrators table to only certain database user accounts (so that it cannot be accessed by "game" code/servers at all).
But outside of those kinds of scenarios, I can't really think of one and agree with the second comment to your question.
I am working on a project where I am going to make a community.
The problem I encounter is the following:
There is going to be 8 different types of pages on the site like "Store", "Event" and "Blog".
Some pages will be owned by a company others will be owned by a user.
Some pages should have the possibility of writing messages to users and other pages (pages that is allowed to send messages to each other).
Users are going to be able to:
1. Follow some page types but not others
2. Like some page types but not others
3. Write a public message on some page types profile page but not others
4. Write personal messages to some page types but not others
5. All users who are administrators on a page are going to get an extra inbox where he/she can read and respond to the pages messages
I have been trying out a lot of different approaches to make this work, but I however I do it I get a LOT of tables where I need to make a LOT of joins which makes me worried about the performance later on.
The best solution I have come up with is to make the following tables:
Users
Pages
Stores
Events
Blogs
And set all page types (users, stores, events, blogs) to own a page.
Then all pages are allowed to message each other.
Store_likes
Event_followers
Blog_followers
to control which page types are allowed to be followed or liked
Inboxes
That represents which page inbox belongs to which users
Is this the best way of doing it, or do anyone have a better solution?
Any input is greatly appreciated!
/ Elias
You are asking a very general modeling question here which is difficult to answer. But general my recommendation is to first create a model. This amounts to deciding what the entities are, what are the primary keys and unique keys and properties of those entities and the relationships that exists between the entities. For each property decide whether it is required or optional. For relations between entities decide the cardinality of the relation and the inverse relation. Create a model first without worrying about performance at first. Then implement the model in the database. Your application is very standard. Dont worry about lots of tables and joins. Most likely it will be OK.
Our client would like the user table to be separated from all other tables for "security reasons". Is this a good practice given that our application is built using RoR and MySQL and running on Unicorn and Nginx ?
I can think of two possible ways:
Create two different login accounts, one for the user table and one for the other tables.
OR
Have a separate database for the user data.
I think that both the solutions might create some problems with the migrations and other tasks and I don't know if this is an effective method of protecting user data. I am a junior developer and I am not familiar with some database and security concepts. Any suggestion?
A very common pattern is to have the users table literally just contain details of the user account and NO details of the actual person behind that account. ie, it would have username, email, password, or encrypted password & salt or whatever, but nothing else - not even name. So, all the "glue" that makes the system work stays in the users table in your regular database.
Then, the details of the real person behind the account (name, telephone number, address, card details etc etc) are stored in a different table, or tables, with a foreign key in either table pointing into the other one. You could store them in a different database but i don't know if this makes it more secure. A better way might be to encrypt just the table with the user's personal data, or perhaps encrypt the entire database. see
http://thinkdiff.net/mysql/encrypt-mysql-data-using-aes-techniques/
I get the feeling that your client doesn't know a lot about internet security and just needs to be reassured that some hacker isn't going to put all the customers' credit card details online, as has happened in several high profile cases recently. To satisfy them i would recommend that you research this well and implement at least two different security strategies, each of which on their own would be considered adequate.
Reassuring the client didn't work as this was an acceptance criteria for launch. In the end, I created two separate database with separate login credentials and user permissions. In order to manage multiple database migrations, I initially used multi-database-migrations gem and then customised it into my own gem.
I was new to SO when I posted this question (still am) and I now understand that the topic is too wide to be asked in a single SO question. Thanks for the suggestions anyway and I hope that the answer can help other people.
I am building a small e-mail like messaging app for a project, where a user would send out a message to another with information like meeting times and such, and I'm wondering about how to store all the messages exchanged.
My issues are:
Should I store all messages between all users in one database table (where it would be expensive to get the messages of each user when they log in)? Or should each user have a personal table for his/her messages(would have too many tables)?
I also need to store the events that the user accepts. Again, should these be in one table for all users or a separate table for each (I need to retrieve these quite often)?
I've searched on the site for other similar questions but most seem to focus on real-time messaging or on specific implementation technologies.
Thanks for the help!
A table per user is a bad idea. It means every query will be different and for every new user you will need to modify the database. This is hard to build, hard to maintain, and inefficient for your database too.
So, just store it in one table. A couple of millions of rows won't be a problem if you have proper indexes (and proper hardware).
If you fear for bad performance, you may delete very old messages. Or you can move them to an 'archive' table. If a user wants to view recent messages (of the past year or so), they can get it from the normal table, and older messages can be fetched from the other one. It's usually acceptable that digging into the archives is a bit slower, so it's probably okay if that table grows very large.
That said, you already mentioned e-mail. I'd seriously consider inspecting the possibilities of actual e-mail and the post boxes that come with it. There are many existing implementations, and it's a powerful protocol that has survived since the dawn of the internet, so maybe you shouldn't reinvent the wheel.
E-mail can have headers (custom headers too), and multiple parts, so even if a 'normal' e-mail won't suffice, you can still use e-mail as a transport layer for custom types of messages.
First of all, sorry for the question title - I was unable to think about something better.
I have an interesting problem.
There are three web applications:
1. ApplicationA => example.com -> hosted in Germany
2. ApplicationB => example2.net -> hosted in Australia
3. ApplicationC => anotherexample.com -> hosted in United States
All of them are completely free however owner is planning to implement some paid options. The main issue is that applications are hosted on separate servers, in three different locations.
Now, if owner wants to implement any paid options, he needs to created unified invoicing system (as invoices numbering order needs to be correct).
So, we have situation:
1. user buys a premium option on example.com
2. another user buys a premium option on example2.net
3. third and fourth users buy extra options on anotherexample.com
So we have 4 invoices, so they numbering should be as following: 2011/01, 2011/02, 2011/03, 2001/04.
As mentioned above, the main issue is to unify invoicing system as applications use different databases and are hosted on different servers. Of course, invoices should be stored in application-specified database.
Theoretically we have only one issue: invoices numbers. Obviously we need to create a unified system for invoices storage.
There might be few possible issues:
there might be a lot of API requests to invoicing system
every single invoice needs to be stored in the database
while creating every single invoice in every external application, we need
to query invoicing system for the
latest invoice number.
I'm really interested in your knowing your approaches and suggestions. Any activity in this case is highly appreciated.
First, I would have the independent invoicing systems in example.com, example2.net and anotherexample.com all have their own internal primary keys for the invoices generated from within each of these systems. Each system should have its own independent copy of the invoicing logic because you don't want an outage on one server knocking out invoicing on every server.
Whenever you have a distributed system where local copies are creating records for something that will be amalgamated later, it's a good idea to use a GUID as the local primary key, or if you have a philosophical objection to GUID as PKs, create a GUID as a candidate key. This way, you can bring together invoices from all of your systems (and any future ones) without worrying about key collisions and you'll be able to track the combined records back to the source records, should you ever have to do that.
Next, you'll need an integrated invoice system where all of the invoice details are collected periodically. To facilitate this, you need processes on each local invoicing system pumping their own records up to the centralized system. Keep a flag on the invoices in the local systems to identify which invoices have been successfully uploaded - or if you have very high volumes, use a work-list table containing the invoice keys that still need transmitting instead of a flag right on the local invoice table.
The centralized invoice system will also want to have a source code on the combined invoice table so that you can easily tell which website created the invoice originally.
As to invoice numbers, I'm assuming from your question that the customer is a bit fussy about having proper sequencing of your invoice numbers. You can have the centralized system generate these numbers for you using a web service to pick up the next invoice ID. If the centralized service is down for any reason, you can still give the customer an "order reference" i.e. the GUID and just hold back on the invoice number until it can be generated through the central server. This should satisfy your customer's need for tight sequential invoice numbers while preserving the ability to operate multiple sites on multiple servers.
If your customer actually doesn't care about tight sequencing of the invoice numbers, then another alternative is to have the central system generate blocks of reserved invoice numbers and allot them to each website. When the website is getting low on its allotment, it asks the central server for another block. This gives you breathing room in the sequences in case there are communication difficulties.
In my opinion, it would be better to use an encoded invoice number. This way, you won't need to worry about number order getting mixed up.
For example, invoices can be prefixed with the country domain like de297,de298, etc., for invoices from Germany.
And going one step further, I'd incorporate the year as well. Thus it would reset at the beginning of each year and still maintain no conflicts, while at the same time keep the invoice number within a small length.