I have a MySQL database that has over 50+ tables and constantly increasing but as an example assume the following tables:
dynamic (entity, attribute, value) following a EAV model
tasks (id, name, duration, type, start_date, finish_date, user_id, ...)
users (id, first_name, last_name, username, join_date, ...)
I am working on a dashboard to allow user-created charts which reads from the dynamic table and user can filter through entity, property & values however the user would not be able to access 'static' columns belonging in different tables (e.g. output count of tasks with duration > 2 days).
Two approaches I have brainstormed to tackle this:
Approach 1 -
Create a view table using a cron job to extract all columns and values of every 'static' table in a EAV model where the entity = table name, attribute = column name and value = row value and join with 'dynamic' table, this way the table is in a standardised format where the user can easily pick the attributes to filter/report.
Advantages would be no backend coding required every time a new table is added to the database as the cron will pick up all tables.
Disadvantages are doubling up records and also crons have to run at intervals so the reporting data will not be up-to-date unless cron runs more frequently and consequently consuming more resource.
Approach 2 -
Hard code every table and column as a selection option so the user can select from predefined tables and columns.
Advantages are that there's no double-ups and the information is accessibly easily via SELECT queries.
Disadvantages would be every time a new table is added, it needs to be hard-coded as a predefined option and certainly a lot of if-statements in backend to support different tables and properties. It would need constant updates given the number of tables.
What is the best way to approach this? I believe there must be a better way to do this than the above.
Related
We have a system with two main roles: service provider and customer. The provider side is users like doctors, nurses, and caregivers. The customer side is just the customer. all user types contain some common data and some uncommon data. in the current system, we have a table for each user type, and for common data, we have User table. currect system ERD is:
https://s4.uupload.ir/files/screenshot-20210710165449-1007x662_tpwd.png
in the current system, we have a lot of tables and we think about reducing them. our vision is to bring all user types in a single table called User and instead of a lot of tables, we have more columns. of course in some users, we have empty cells that do not belong to this user type.
I have 4 questions:
is it ok to bring customers and providers to a table like User?
what is the optimal number of columns in a table?
load a row with a lot of columns OR relation between different tables?
provider type should be a separate table or can be an enum?
It is best to put all users in single table. So when you check login there is less place to do mistake. When selecting user you dont need to use SELECT * FROM... You can use SELECT id, username, name FROM...
Dont put too many columns, if there is some data which you dont need when searching or displaying users, you can create helper table "user_meta" with dolumns user_id, meta_key, value where user_id and meta_key are primary key
Answered by first 2 answers
Provider type should be enum if there will not bee needs to expand with additional types.
I want to revisit a project I made to store user data into a database and improve on the way it is stored. I currently went the hard way about it and stored user data in JSON format within a MySQL database field making it difficult to complete CRUD actions. The reason I did this was to keep all the user's data within the user's field. And was reasonably new to this.
I didn't want to store the data mixed with other user's data and as I thought there may be issues with increased users. for example,
If I had 1000 users with 500 rows of data for each, that's 500 000 rows to sort through when reading the data and displaying it on a web page. And is there a risk of mixing the data up or performance issues?
I basically just want a user database that stores the user's id, name, and credentials. Then another database that will store data from a user's activity(run). So at least 5 fields for each event: Time, location, date, duration, etc. And this will be saved for different events(runs) which could end up in the 100's over a period of time.
My question is, Should I design the table as above. Or would it be better to have a table for each user? Or are there other options that I have not explored?
Given the information shared, I believe below mentioned design may be suitable.
Create a Table called User_Details with columns as id (auto increment),user id, name and credentials.
Now create a User_Activity Table with these columns id, user_id, event name, data(json field).
Explanation:
The User Activity table will store the event data for you related to each user through user_id field to user_details table. The data which is a json field will help you to store all the fields for the event. As you are using json field in DB it will allow you to dump any number of fields for the event which may/may not be structured. You can then map this in your middle layer as required.
Also, in case you have finite number of events then you can also create a table called user_event_types and have column id, event name and then in user_activity table you can refer the id instead of event name.
I'm building an stock exchange simulation game. I have a table called 'Market_data' and in the game players simulate being in particular dates and are allowed to use SQL queries to retrieve the historical data and plan their course of action. My difficulty is that I need to limit the rows they can access based on the current date they are playing on so they cant see rows with a date greater than the current date.
Eg: An user is running the game and is currently in the year 2010, if he does a simple select like "SELECT * FROM market_data" I don't want him to see rows with Date > 'x-x-2010'
The only soution that I know of is to parse the user's SQL and add WHERE clauses to remove newer dates but it seems time consuming and prone to errors and I wasn't sure whether there were better alternatives. Any ideas on how to do this right will be thanked.
Solution is SQL Views, Views are used for several different reasons:
*1.*To hide data complexity. Instead of forcing your users to learn the T-SQL JOIN syntax you might wish to provide a view that runs a commonly requested SQL statement.
*2.*To protect the data. If you have a table containing sensitive data in certain columns, you might wish to hide those columns from certain groups of users. For instance, customer names, addresses and their social security numbers might all be stored in the same table; however, for lower level employees like shipping clerks, you can create a view that only displays customer name and address. You can grant permissions to a view without allowing users to query the underlying tables. There are a couple of ways you might want to secure your data:
a.Create a view to allow reading of only certain columns from a table. A common example of this would be the salary column in the employee table. You might not want all personnel to be able to read manager's or each other's salary. This is referred to as partitioning a table vertically and is accomplished by specifying only the appropriate columns in the CREATE VIEW statement.
b.Create a view to allow reading only certain rows from a table. For instance, you might have a view for department managers. This way, each manager can provide raises only to the employees of his or her department. This is referred to as horizontal partitioning and is accomplished by providing a WHERE clause in the SELECT statement that creates a view.
*3.*Enforcing some simple business rules. For example, if you wish to generate a list of customers that need to receive the fall catalog, you can create a view of customers that have previously bought your shirts during the fall.
*4.*Data exports with BCP. If you are using BCP to export your SQL Server data into text files, you can format the data through views since BCP's formatting ability is quite limited.
*5.*Customizing data. If you wish to display some computed values or column names formatted differently than the base table columns, you can do so by creating views.
reference taken from http://sqlserverpedia.com.
1)You can use mysql proxy http://dev.mysql.com/downloads/mysql-proxy/ with custom rules restricting access.
2)You can use stored procedures/functions
3)You can use views
The basic way would be :
-> Prevent that user (or group) from accessing the base table.
-> Define a view on top of that table that shows only the rows these users are supposed to see.
-> Give those users SELECT permission on the view.
-> And you can also use SQL Encryption,Decryption and Hashing concept.
Encryption & Decryption examples can be found here:
http://msdn.microsoft.com/en-us/library/ms179331.aspx
Hashing example can be found here:
http://msdn.microsoft.com/en-us/library/ms174415.aspx
I want to try and keep this as one query and not use PHP, but it's proving to be tough.
I have a table called applications, that stores all the applications and some basic information about them.
Then, I have a table with all the types of applications in it, and that table contains a reference to another table which stores more specific data about the specific type of application in question.
select applications.id as appid, applications.category, type.title as type, type.id as tid, type.valuefld, type.tablename
from applications
left join type on applications.typeid=type.id
left join department on type.deptid=department.id
where not isnull(work_cat)
and work_cat != ''
and applications.deleted=0
and datei between '10-04-14' and '11-04-14'
order by type, work_cat
Now, in the old version, there is another query on every single result. Over hundreds of results... that sucks.
This is the query I'd like to integrate so I can get all the data in one result row. (Old is ASP, I'm re-writing it in PHP)
query = "select sum("&adors.fields("valuefld")&") as cost, description from "&adors.fields("tablename")&" where appid = '"&adors.fields("tablename")&"'"
Prepared statements, I'm aware, are the best solution, but for now they are not an option.
You can't do this with a plain SQL query - you need to have a defined set of tables that your query is based on. The fact that your current implementation queries from whatever table is named by tablename from the first result-set means that to get this all in one query, you will have to restructure your data. You have to know what tables you're querying from rather than having it dynamic.
If the reason for these different tables is the different information stored in each requiring different record (column) structures, you might want to look into Key/Value pair storage in a large table. Once you combine the dynamically named ones into a single location you can integrate your two queries together.
I've learned that views can be used to create custom "table views" (so to say) that aggregate related data from multiple tables.
My question is: what are the advantages of views? Specifically, let's say I have two tables:
event | eid, typeid, name
eventtype | typeid, max_team_members
Now I create a view:
eventdetails | event.eid, event.name, eventtype.max_team_members
| where event.typeid=eventtype.typeid
Now if I want to maximum number of members allowed in a team for some event, I could:
use the view
do a join query (or maybe a stored procedure).
What would be my advantages/disadvantages in each method?
Another query: if data in table events and eventtypes gets updated, is there any overhead involved in updating the data in the view (considering it caches resultant data)?
A view is not stored separately: when you query a view, the view is replaced with the definition of that view. So and changes to the data in the tables will show up immediately via the view.
In addition to the security feature pointed out earlier:
If you're writing a large number of queries that would perform that join, it factors out that SQL code. Like doing some operations in a function used in several places, it can make your code easier to read/write/debug.
It would also allow you to change how the join is performed in the future in one place. Perhaps a 1-to-many relationship could become a many-to-many relationship, introducing an extra table in the join. Or you may decide to denormalize and include all of the eventtype fields in each event record so that you don't have to join each time (trading space for query execution time).
You could further split tables later, changing it to a 3-way join, and other queries using the view wouldn't have to be rewritten.
You could add new columns to the table(s) and change the view to leave out the new columns so that some older queries using "select *" don't break when you change the table definitions.
You can restrict users to the view instead of the underlying table(s), thereby enhancing security.
Advantages of SQL Views
1). You can save a complex query(a query with join multiple tables) as view to reuse it in simple manners.
In other words, you can change your multi-line query into single-line query.
2). You can hide sensitive data by converting your query into view with new name and can give the access only required columns instead of giving the access of actual table.