So I am going to be building a website using the Django web framework. In this website, I am going to have an advertising component. Whenever an advertisement is clicked, I need to record it. We charge the customer every time a separate user clicks on the advertisement. So my question is, should I record all the click entries into a log file or should I just create a Django model and record the data into a mysql database? I know its easier to create a model but am worried if there is a lot of high traffic to the website. Please give me some advice. I appreciate you taking the time to read and address my concerns.
Awesome. Thank you. I will definitely use a database.
Traditionally, this sort of interactions is stored in a DB. You could do it in a log, but I see at least two disadvantages:
log rotation
the fact that after logging you'll still have to process the data in a meaningful manner.
IMO, you could do it in a separate DB (see the multiple db feature in django). This way, you could have the performance somewhat more balanced.
You should save all clicks to a DB. A database is created to handle the kind of data you are trying to save.
Additionally, a database will allow you to analyze your data a lot more simply then a flat file. If you want to graph traffic from country, or by user agent or by date range, this will be almost trivial in a database, but parsing giganitc log files could be more involving.
Also a database will be easier to extend. Right now you are just tracking clicks but what happens if you want to start pushing advertisements that require some sort of additional user action or conversion. You will be able to extend this beyond clicks extremely easy in a database.
Related
I'm relatively new to Databases and have been looking into a solution that will allow users, under my server, to access their own data and no one else. I want these databases for the user to be scalable, so if more space is needed to store files, they can do so with little intervention.
I was looking into MySQL, as I have only done single database work with it, and was trying to see how this could potentially be done. Would the best course of action be set a database for each user? That way the tables for each database are separate, password protected, and cannot exchange data among the tables in other databases? I know there can be essentially unlimited databases and tables, in addition to table sharding/partitioning, so I think this is a solid choice, but was wondering if anyone who has worked with MySQL more had any input.
Thanks
EDIT: Update for clarification of desires. So what I essentially want is a platform where I am the owner, but I can have users log in to my platform to access their data. This data will probably mostly include files, such as PDF's, but as to their size I cannot tell, but am planning for the worst. They will be able to use a web/application to view their files and download, upload, sort, delete these files. So in addition to creating files, there will be the ability to see historic files and download those as well if desired. What my platform will be providing is the framework for these files with fields being autofilled if I can, as well as the UI for the file management. My concern comes from architecture of having multiple users, with separate data, to be kept separate, scalable, and not completely crash the server with read/writes.
It sounds like you are looking to store the users "files" as BLOBs in the database which doesn't necessarily lend itself to scaling well in the first place. Depending on the type of files generally the best solution would be to provide security in the application layer and use cloud based storage for your files. If you need an additional layer of security (i.e. users can only access the files assigned to them) there are a number of options - one such option, for example, assuming you were using S3 would be to use IAM profiles which could be generated when the user a/c is set up. The same would apply for any third party cloud storage with API.
Having an individual database per user would be an administrative nightmare unless you could generate each database on login (which would mean a separate data store for credentials anyway so it would be somewhat pointless) and also would not work in a BLOB storage scenario.
If you can detail a little more in terms of precisely what you are trying to achieve and why there will for sure be plenty of answers.
I'm looking to implement notifications within my node.js application. I currently use mysql for relational data (users, submissions, comments, etc). I use mongodb for page views only.
To build a notification system, does it make more sense (from a performance standpoint) to use mongodb vs MySQL?
Also, what's the convention for showing new notifications to users? At first, I was thinking that I'd have a notification icon, and they click on it and it does an ajax call to look for all new notifications from the user, but I want to show the user that the icon is actually worth clicking (either with some different color or a bubble with the number of new notifications like Google Plus does).
I could do it when the user logs it, but that would mean the user would only see new notifications when they logged out and back in (because it'd be saved in their session). Should I poll for updates? I'm not sure if that's the recommended method as it seems like overkill to show a single digit (or more depending on the num of notifications).
If you're using node then you can 'push' notifications to a connected user via websockets. The linked document is an example of one well known websocket engine that has good performance and good documentation. That way your application can send notifications to any user, or sets of users, or everyone based on simple queries that you setup.
Data storage is a different question. Generally mysql does have poor perfomance in cases of high scalability, and mongo does generally have a quicker read query response, but it depends on what data structure you wish to use. If your data is in a simple key-value structure with no real need for relational data, then perhaps using a memory store such as Redis would be the most suitable.
This answer has more information on your question too if you want to follow up and investigate more.
For a RoR app I'm helping develop, I need to save all search queries in a database so I can analyze them later.
My plan right now is to create a Result model and table, and just save each search query's text in that table, along with a user's ID, the time, etc.
However, the app has about 15,000 users, so I'm afraid the single table approach won't be super efficient when it comes time to parse that data. (The database is setup via MySQL, if that factors in at all.)
Am I just being paranoid? Is there a Ruby gem that handles this sort of thing, or a better approach I could take?
Any input would be appreciated.
there are couple of approaches you can try:
1. Enable mysql query logging and than analyze these logs
http://dev.mysql.com/doc/refman/5.1/en/query-log.html
2. Use key=>value store (redis comes to mind) to log the search query in a similar way you described
If you decide to go with the 2nd approach i would create an asynch observer on the model you want to track
The answer depends on what you want to do with the data.
If your users don't need access to this, and you're not doing real-time analytics, dump them out of your app and get them into another database to run analytics to your heart's content.
If you want something integrated into your app, try a single mysql table.
Unless your server is tiny or your users are crazy active searchers, it should work just peachy. At a certain point you'll probably want to clear out old records and save them elsewhere though.
I have some items and I want to store statistics like daily clicks and the traffic source of the page and I want to store these statistics for 100`s of items. Based on what stats I will save I want to be able to generate charts with the clicks of each day (like on google analytics) and to show the number of clicks from each traffic source.
I'm not a specialist, but I'm thinking to store statistics in a mysql table for a single day then write them in multiple .xml files. I have a slow, cheap server and I`m searching for the best method, please help me!
These "items" are embedded in other websites. I control these items using php
Since this items are embedded in other websites storing this infos / request is a NO GO.
This means you either need to install and setup mysql on this other websites, which is unlikely.
Or you connect to a remote mysql .. which is quite expensive for each request.
Especially when you say yourself, that you only have a "cheap" server.
Additionally you risk bringing the websites with the embedded items down, when your mysql server fails.
Better to use google analytics to track the visited pages correctly instead of developing one.
It will show you daily, country wise visitors.
I operate a web-based online game with a mySQL backend. Every day many writes are performed against hundreds of related tables holding user data.
Frequently a user's account will become compromised. I would like the ability to restore the user's data to a certain point in time prior to the attack without affecting any other user data.
I'm aware of binary logging in mySQL, but as far as I know this is whole-database recovery up to a certain point in time. I would like a more granular solution, ie able to specify which tables, which rows etc.
What should I be looking into? What are the general best-practices?
If you create and use audit tables (populated through triggers) you can always get back to the data for one particular user in any table.
Be sure to write your general restore script before you need it though. Much easier to put in a userid into a script that you already have available than to sit there looking at the audit tables going, how do I do this again.
MySQL (or any other RDBMS that I'm aware of) is not able to do that by itself. Therefore you should implement that yourself in your application layer.
This is (without external modules) not possible.
As thejh in the comments suggested, revisions would be a good solution. When you only need to work with userdata, create a table that resembles the usertable with additional timestamp or similar and run a cron job once a week/day/.. that copies the userdata that has recently been modified (additional flags/dates in the actual user table) into this table.