Maybe the question is a bit complex and complicated (that's why I need to make a brief introduction)
my team and I are developing a Mobile App with Node js. Now we are in the part of the database structure. Our idea is to do it in Azure SQL. But we have a couple of questions regarding the structure of the database.
We offer 5 services (at the moment), of which each user can be assigned several services (may non or all). Based on the services it has, the user will be redirected to a screen where all the services will be and only those assigned with color (to be able to click) and the others in gray (so that they cant click it)
Which is better, create one column per service or all services in a single column array style?
for example
service 1| service 2| service 3|service 4|service 5|
true | fasle | true | true| false| true
or
service
[service 1,service 2,service 3,service 4,service 5]
Because I think that if in the future we have x services, going through the entire array and making a condition to verify what service it has is going to make the latency of the app to slow, instead hitting a certain column maybe makes it faster
I hope the question has been understood, sorry if the maries.
regards
This isn't really an azure SQL question but more of a relational database question.
In general you should avoid both these methods and try to normalize your database.
Your database knows how to query multiple databases without any performance hits, its made for it.
The best option in my experience is to create a many to many table connection
So one table that holds the original data without any mention of a service, Maybe called Entities
Id, Data, Time, Active
Another table that holds the relations to the services, called EntitiesToServices
Id, ServiceId, OtherTableId
And a third table that holds data about the services called Services
Id, Name
In this way you can expand all your services freely and add more tables without anyone interfering with each other.
If all you need is a set of up to 64 true/false values, consider a single column of type SET. Similarly you could some any sized INT (again with a limit of 64 flags) and turn on/off each 'service'. Today you have 5; tomorrow, as you say, there will be more.
The syntax for SET is a bit clumsy. So is using INT for this.
It is very compact; this may or may not be a bonus.
Normalizing (as mentioned in another answer) may be a better solution, especially if you need to store more than just on/off for each service for each user.
Please provide more details on what actions will happen with these flags; then we can get into more detail.
Related
i'm currently searching for a good approach to filter DB results based on permissions which are stored in another services DB.
Let me first show the current state:
There's one Document-Service with 2 tables (permission, document) in its MySQL DB. When documents for a user are requested, a paginated result should be returned. For brevity let's ignore the pagination for now.
Permission table: Document table:
user_id| document_id document_id| more columns
-------|------------ A
1 | A B
2 | A C
2 | B
2 | C
The following request "GET /documents/{userId}" will result in the following query against the DB:
SELECT d.* FROM document d JOIN permission p WHERE p.user_id = '{userId}' AND p.document_id = d.document_id;
That's the current implementation and now i am asked to move the permission table into its own service. I know, one would say that's not a good idea, but this question is just a broken down example and in the real scenario it's a more meaningful change than it looks like. So let's take it as a "must-do".
Now my problem: After i move the table into another DB, i cannot use it in the sql query of Document-Service anymore to filter results.
I also cannot query everything and filter in code, because there will be too much data AND i must use pagination which is currently implemented by LIMIT/OFFSET in the query (ignored in this example for brevity).
I am not allowed to access a DB from any other application except its service.
My question is: Is there any best practise or suggested approach for this kind of situation?
I already had 2 ideas which i would like to list here, even though i'm not really happy with either of them:
Query all document_ids of a user from the new Permission-Service and change the SQL to "SELECT * FROM document WHERE document_id IN {doc_id_array_from_permission_service}". The array could get pretty big and the statement slow; not happy about that.
Replicate the permission table into Document-Service DB on startup and keep the query as it is. But then i need to implement a logic/endpoint to update the table in the Document-Service whenever it changes in the Permission-Service otherwise it get's out of sync. This feels like i'm duplicating so much logic in both services.
For the sake of this answer, I'm going to assume that it is logical for Permissions to exist completely independently of Documents. That is to say - if the ONLY place a Permission is relevant is with respect to a DocumentID, it probably does not make sense to split them up.
That being the case, either of the two options you laid out could work okay; both have their caveats.
Option 1: Request Documents with ID Array
This could work, and in your simplified example you could handle pagination prior to making the request to the Documents service. But, this requires a coordinating service (or an API gateway) that understands the logic of the intended actions here. It's doable, but it's not terribly portable and might be tough to make performant. It also leaves you the challenge of now maintaining a full, current list of DocumentIDs in your Permissions service which feels upside-down. Not to mention the fact that if you have Permissions related to other entities, those all have to get propagated as well. Suddenly your Permissions service is dabbling in lots of areas not specifically related to permissions.
Option 2: Eventual Consistency
This is the approach I would take. Are you using a Messaging Plane in your Microservices architecture? If so, this is where it shines! If not, you should look into it.
So, the way this would work is any time you make a change to Permissions, your Permissions Service generates a permissionUpdatedForDocument event containing the relevant new/changed Permissions info. Your Documents service (and any other service that cares about permissions) subscribes to these events and stores its own local copy of relevant information. This lets you keep your join, pagination, and well-bounded functionality within the Documents service.
There are still some challenges. I'd try to keep your Permissions service away from holding a list of all the DocumentID values. That may or may not be possible. Are your permissions Role or Group-based? Or, are they document-specific? What Permissions does the Documents service understand?
If permissions are indeed tied explicitly to individual documents, and especially if there are different levels of permission (instead of just binary yes/no access), then you'll have to rethink the structure in your Permissions service a bit. Something like this:
Permission table:
user_id| entity_type| entity_id | permission_type
-------|------------|-----------|----------------
1 | document | A | rwcd
2 | document | A | r
2 | document | B | rw
2 | document | C | rw
1 | other | R | rw
Then, you'll need to publish serviceXPermissionUpdate events from any Service that understands permissions for its entities whenever those permissions change. Your Permissions service will subscribe to those and update its own data. When it does, it will generate its own event and your Documents service will see confirmation that its change has been processed and accepted.
This sounds like a lot of complication, but it's easy to implement, performant, and does a nice job of keeping each service pretty well contained. The Messaging plane is where they interact with each other, and only via well-defined contracts (message names, formats, etc.).
Good luck!
We're building a new piece of software for our company, where we want to manage our inventory.
The goal for the tool is to be customizable by the customer.
My part is mostly on the DB side. We have chosen MariaDB as our DB engine, and while we are working with the rather static functionality of a relational DB, we want to realize a rather dynamic solution.
Our chief programmer has explained to me the basics of the concept I shall implement into our DB:
We want a table which basically just consists of other tables.
Lets call it "maintable".
Maintable shall then reference its "attributes", which are the other tables.
For example, maintable references "Workstations".
"Workstations" then contains attributes like CPU, RAM, Drives, PSU etc..
And now comes the part which I didn't completely understand. The actual VALUES to these attributes in "Workstations" shall not be inserted into "Workstations". Instead, they are packed into another (junction?) table.
The reason for this approach is that the customer shall be able to customize the DB to his needs.
When the customer wants to add another attribute, he shall be able to do so. For example, if a new PSU now requires another attribute for an additional serial number, then the customer shall be able to simply create this new attribute in the front-end input form and then persist it to the DB.
If someone could point to good tutorials explaining this type of DB concept, then I would be glad as well! :=)
We are updating table XYZ have following fields:
First Name|Middle Name|Last Name|Address|DOB|Country|County|(etc.)
Initially, we are calling some web service which is sending updated information for a row in XYZ like either update first name or DOB update or both or all or none.
Now there is requirement to create a log table in database which store summary of old records and changes done to XYZ. Every affected row should be reported.
Is it good to create similar fields in new table say ABC:
First Name|Middle Name|Last Name|Address|DOB|Country|County|Update_Date
with additional field called "Update_datetime"
Now each time service called we will select values from previous row i.e from XYZ and update the same to ABC with update date.
What are loopholes in this practice? What other better practices can be followed?
Is there a requirement for a log table or a requirement for a proper history?
Oracle has history functionality Out of the box
I doubt MySQL does - you may have to do a different way.
The pros of Oracle is that it will not fail - it's a core feature. The cons of hand rolled is, well, it's hand rolled. Lots of SPs, triggers or other nastiness that people can deliberately or inadvertently bypass.
I echo the need to know what the requirements are behind this. Is it to be human readable (auditing, debugging etc.) or machine readable (e.g. event sourcing architectural pattern)? How often will you need to go back to look at previous versions? How often do things change?
If it's event sourcing, then there are a few answers around on Stack Overflow about that, e.g. Using an RDBMS as event sourcing storage and best event sourcing db strategy. For more of an introduction, there's e.g. a Martin Fowler video.
There are also SO answers on logging changes in MySQL and Using MySQL triggers to log all table changes to a secondary table and an alternative approach (using 1 table, but adding sort-of version numbers to show each record's validity).
I have a site where some pages (we call them gateway pages) are based loosely on certain departments in the organization. Each department has classes associated with it. Unfortunately some of my pages are not associated with a specific department, but do display information about several classes from a department so I can't just query the database strictly on department alone.
Would it be smarter to create a table called gateway_classes with a fk from the gateway table in each class or form a query to somehow filter out exactly what I need from my existing tables using an array of classes to be pulled during the query?
Here's my tables:
departments_classes | classes_vendors | departments | vendors | classes | products | gateway
Any guidance is greatly appreciated.
More Info: There are roughly 350 classes and 18 departments and 12 gateway pages...
Your indexing table idea sounds like it'd work just fine. The only downside to that is that you've got to maintain it separately, and you want to make sure that the data you hold in that table isn't being duplicated in any of your existing tables.
If you don't want to maintain that data differently than you're currently doing so, you can use CF's arrays (or structs) to hold that correlation data (which you'd have to pull from the db in a separate query) and then loop over it as you construct the query that pulls the classes for a given page.
Either way would work okay, it's more a matter of how you prefer to do it, and what you think would be easiest to build, test, and maintain.
One thing about efficiency - make sure you not only link your tables via Foreign Keys (which helps to maintain data integrity), but also put in (nonclustered) indices, which helps the efficiency of the joins and lookups your queries will be doing.
I've seen dramatic speed improvements in my queries (CFQUERYs operating against MS SQL) with the simple act of putting in indices.
In MS SQL, you do so like this:
CREATE NONCLUSTERED INDEX yourIndexName ON yourTableName(yourFieldName)
I hope this helps!
Your problem sounds similar to a common scenario for determining user rights. A User may belong to some Group that has Rights associated with it or the User may be assigned Rights individually. In your case, the User is the Gateway, the Group is the Department, and the Rights are the Classes. A Gateway can then be linked to any number of Departments and/or Classes.
Using this model, you just need to add the gateway_classes table as you describe along with a gateway_departments table.
You could then use UNION to merge the "gateway classes" query with the "gateway departments" query (or perhaps something more elegant) but I think this schema will do want you need without introducing any redundant information.
I'm writing a CMS for various forms and such, and I find I'm creating a lot of drop-downs. I don't really feel like mucking up my database with tons of random key/string value tables for simple drop-downs with 2-4 options that change very infrequently. What do you do to manage this in a responsible way?
This is language-agnostic, but I'm working in Rails, if anyone has specific advice.
We put everything into a single LookUp table in the database, with a column that mapped to an enum that described which lookup it was for (title, country, etc.).
This enabled us to add the flexibility of an "Other, please specify" option in lookup dropdowns. We made a control that encapsulated this, with a property to turn this behaviour on or off on a case-by-case basis.
If the end user picked "Other, please specify", a textbox would appear for them to enter their own value. This would be added to the lookup table, but flagged as an ad hoc item.
The table contained a flag denoting the status of each lookup value: Active, Inactive, AdHoc. Only Active ones would appear in the dropdown; AdHoc ones were those created via the "Other, please specify" option.
An admin page showed the frequency of usage of the AdHoc values, allowing the administrators of the site to promote common popular values into general usage (i.e. changing their Status flag to Active).
This may well be overkill for your app, but it worked really well for ours: the app was basically almost entirely CRUD operations on very business-specific data. We had dozens of lookups throughout the site that the customer wanted to be able to maintain themselves. This gave them total flexibility with no intervention from us.
You cold have one single dropdown table with an extra column to say what the drop down is for... limit the results with a where clause...
At my current position, we implemented a LookupCode table that contains a CodeGroup,Code, and Meaning column, as well as some others (like active). That way you have a single table that contains all of your lookup values are in a single location and you can do some quick lookups to bind to your dropdown lists.