Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am newbie to web development.
I have an application server where my ASP.NET code resides. My application server communicates to a MySQL instance which is on a different server.
I was wondering, whether it is a good practice to move the computation from the application server to the database server by having a Stored Procedure with Views or should I just move on with all logic kept in application server and query the database only to retrieve data from tables directly without having stored procedures and views.
I am a strong advocate of putting database logic into the database and not splitting it between the application and the server. This means that I prefer to wrap all database calls in stored procedures and views.
The driving reasons are maintenance, security, and functionality, not performance, although performance is often better on the server side.
The number one reason is to isolate the application from changes in the underlying data structure. So, if the data structure changes, the application does not (always) break.
Other reasons the come to mind:
The same logic gets used for the same thing. That is, one piece of code doesn't define "foobar" one way and another "foobar" another way.
Auditing and logging are implemented within stored procedures rather than using triggers.
Database tables are off-limits to all users, unless they go through the defined interface.
A newer version and older version can often co-exist.
Admittedly, for a one-off, quick-and-dirty application these issues may not be important. However, I think it is a good idea to have well defined interfaces (APIs) between different components of a system, and databases and the application layer are a prime example where such APIs are quite useful.
I agree with Gordon on separating out a "layer" of code between the application and the actual database. I dispute how practical Stored Routines are at such.
PHP (etc) is far more expressive than SProcs.
One SProc can execute multiple queries faster because it is closer to the server. This can be an overwhelming performance gain if the client and server are on opposite sides of the country.
Error checking is clumsy in SProcs.
PHP recompiles only when the code changes; SProcs recompile once per connection; Perl always recompiles; etc.
VIEWs are sometimes poorly optimized, so I avoid them.
The secret to a good design for the "layer" is in the compromise between the forces tugging on either side. One example: Can you completely hide a schema change from the app? Even if you split one table into two?
A really bad example was when the UI did pagination by using page numbers. The layer thought in terms of OFFSET and LIMIT, and fed that to the MySQL back-end. Then came an item will 216K pages (Yes, that many!) They found out that OFFSET+LIMIT is not a good way to implement "next page", but fixing it required a changes to all layers of the system.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed last month.
Improve this question
I am a new MySQL learner and like to use it within a NodeJS application. From my previous knowledge with MongoDB all the tables we need will be created by MongoDB automatically. We only needed to connect to the MongoDB and with create or save commands it could find the table and put something inside or create the table if it was not exist.
But from MySQL tutorials I have seen so far they create databases manually by executing SQL codes directly inside a MySQL shell/bash or by using applications like MySQL Workbench.
So I want to make sure which way is correct? Should I create my tables before running my Node app or there is also a way, and it is a good way to create tables dynamically within the code when it's needed regarding the app usage by the user? If the second way is a better way, how can I do that? Is there a code sample for creating databases from inside a Node app?
Moreover the same question for databases. Can we or is it a good way creating them from inside our Node app?
In sql-based applications standard tables are created when you install the application giving it a fix structure. Table structure is only changed during program updates as structure changes can be a very expensive operation. The initial table creation, subsequent changes can be done via sql scripts or by application code - both are acceptable practices. But the main point stands: permanent tables are not created on the fly.
Some complex scripts may require temporary tables that are created and destroyed on the fly, but most of these will be done implicitly (e.g. database engine may materialise the resultset of a subquery as a temporary table), rather than explicitly (application code executing create temporary table statement).
Rationale: RDBMSs have a rigid, inflexible data structure that is optimised towards data storage and retrieval speed. Any solution to introduce flexibility is at odds with relational theory and will come with its drawbacks (see Bill Karwin's excellent answer on this subject, particularly the last two pragraphs).
NoSQL solutions, however, more focus on flexibility, than data storage. Mongodb for example, does not even have tables with rows and columns. It has collections that are made up of documents and every document in a collection may have different properties. This is why you can create these collections and documents on the fly.
Both are possible.
You can create a piece of code that will check the schema and update it if it is not (there are lots of tools and for all languages).
In my experience, in most cases we tend to create the tables, indexes, views, etc. by hand first and then run the application. For larger companies / projects, they often use tools. I personally have seen a lot of flyway in the Java world.
In any case, the most important thing is that your creation script is idempotent. For this, you can use the if not exists statements : https://dev.mysql.com/doc/refman/8.0/en/replication-features-create-if-not-exists.html
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
i have two scenarios here :
migrating mysql database to nosql without code change(no orms are used)
using no foriegn keys and indexes in mysql(because they want to migrate to different database in future)
3.all this done by very less code change
these questions are asked by my team lead. so i dont have a answer to give him properly because i feel it very unlikely to do mysql with no indexes and foreign keys and first of all if they are not meant to use mysql.then why they choose that.
i want to know that people do like this in software industries
ofently or they will choose on their need fits correctly
they are saying that foreign key validitations are done by api level
not by mysql level
i dont understand them becasue i have less experience so i dont have an answer why they are saying like this. please give me some insight to this that if this is a good practice or not ?
I don't think it will be possible without adding code - you need to implement how your data is managed by your nosql dB engine in some way. If the project is coded with a clear separation of business logic and database code, it's a simple matter of using the new database implementation instead of the old one. If that is not the case and your db implementation leaked into your business logic, then it will not be possible to switch without changing code. Depending on the size of the code base it might /will most likely be too expensive.
If you want to see an example of a clean separation of dB logic from business logic, have a look at this repository: https://github.com/fathersson/money-transfer
(this is not my repository, I just stumbled upon it today)
If you want to learn and understand the principles driving that design, start by looking for "clean architecture" and/or "Domain Driven Design" - the first one is easier to understand in my opinion and there are some talks on YouTube by Robert C. Martin that you can have a look at before buying some books.
Edit: The project I'm working on at the moment did change from postgresql running on rds to dynamodb using a different repository without changing any existing business logic. It saves a lot of money that way. So yes, changing the db backend does happen and is driven by requirements.
In addition to that, when I start working on a new feature set/micro service/bounded context I usually start with a simple in memory repository implementation that's using a map. After I'm done with the initial set of use cases, I know more about the db requirements and choose the db engine based on these and the general requirement to limit the number of different technologies in use.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'm working on a project which is somewhat familiar to WhatApp, except that I'm not implementing the chatting function.
The only thing I need to store in the database is user's information, which won't be very large, and I also need an offline database in the mobile app which should be synced with the server database.
Currently I use MySQL for my server database, and I'm thinking of using JSON for the syncing between mobile app and the server, then I found that mongoDB has a natural support for JSON, which caused me wonder should I change to mongoDB.
So here are my questions:
Should I change to mongoDB or should I still use MySQL? The data for each user won't be too large and it does have some requirement for data consistency. But mongoDB's JSON support is somewhat attractive.
I'm not familiar with the syncing procedure, I did some digging and it appears that JSON is a good choice, but what data should I put into the JSON files?
I actually flagged this as too broad and as attracting primarily opinion based answers but I'll give it a go anyhow and hope to stay objective.
First of all you have 2 separate questions here.
What database system should I use.
How do I sync between app and server.
The first one is easily answered because it doesn't really matter. Both are good options for storing data. MySQL is mature and stable and MongoDB although it's newer has very good reviews and I don't know of any known problems which would prevent it from being used. So take the database which you find easy to use.
Now for second I'll first put in a disclaimer that for data synchronization between multiple entities entire books are written and that it is after all this time still the subject of Phds.
I would advice against directly synchronizing between mobile app and database because that requires the database credentials to be contained within the app. Mobile apps can and will be decompiled and credentials extracted which would compromise your entire database. So you'll probably want to create some API which first does device/user authentication and then changes the database.
This already means that using MongoDB for sake of this would probably be a bad idea.
Now JSON itself is just a format of representing data with some structure, just as XML. As such it's not a method of synchronization but transport.
For synchronizing data it's important that you know the source of truth.
If you have 1 device <-> 1 record it's easy because the device will be the source of truth, after all the only mutations that take place are presumably done by the user on the device.
If you have n devices <-> 1 record then it becomes a whole lot more annoying. If you want to allow a device to change the state when offline you'll need to do some tricks to synchronize the data when the device comes back online. But this is probably a question too complex and situation dependent to answer on SO.
If you however force the device to always immediately propagate changes to the database then the database will always contain the most up to date record, or truth. Downside is that part of the app will not be functional when offline.
If offline updates don't change the state but merely add new records then you can push those to the server when it comes online. But keep in mind you won't be able to order these events.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
So I'm working on a task of calculating medians for every 100 records in a giant MySQL table, which appears to be a straightforward problem but ends up with very complex SQL code. One of my friend who saw my work asked me, why don't you load the data into memory and process it with C or Python, wouldn't that be easier? My intuition is that it is a bad idea. But can someone elaborate more about why it is not suggested? Thank you!
I can think of no good reason to tell you that it's a bad idea to use a front-end to process data stored in a MySQL db... to me it's something like "don't use knives to cut your food because you can cut your own finger".
You can, of course write some stored procedures or functions that might give you the results you need, but if you can't make it work with MySQL, then the obvious step is to use another tool.
You must, however, take some precautions:
Don't overload your network connection (trivial if you are working on localhost).
Don't try to store too big resultsets in memory: keep it simple and small (divide and conquer)
Let the database server do the heavy work, and use your front-end to do the fine work (if you need to filter data, let MySQL do that for you, and write the code to do the calculations on the filtered data).
Be sure to take the appropriate precautions when sending queries to MySQL (avoid SQL injection vulnerabilities)
In general, yes you should do your heavy lifting in the database. If your dataset is fairly small, it wouldn't matter whether you do the calculations on the database server or on the database client.
The primary consideration whether to do the calculation in the db server vs on the db client is usually of performance. If you do the heavy calculations on the db client, you may end up having to transfer a lot of data through the db connection. With large datasets, transferring the entire table to the client may become performance issues, and if your database server lives in a different machine than your application server (i.e. not localhost), then the network transfer overhead becomes even worse.
If you have to transfer the entire dataset anyway, then there likely won't be any significant performance difference. The SQL language itself isn't inherently faster than the client languages for doing number crunching, it simply has the advantage of running on the server process and thus can avoid the overhead of data transfer.
There are also applications that uses multiple data sources, for these, generally you'll will often up with no other choice but to do parts your calculations the client side.
Ultimately, you have to measure. It didn't matter whether it's best practice or not, if doing the calculation in the client is fast enough and it simplify the overall code doing that, then do take that route.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I would like to create a database with the goal of populating this database with comprehensive inventory information obtained via a shell script from each client machine. My shell script currently writes this information to a single csv file located on a server through an ssh connection. Of course, if this script were to be run on multiple machines at once it would likely cause issues as each client potentially would try to write to the csv at the same time.
In the beginning, the inventory was all I was after; however after more thought I began to ponder wether or not much much more could be possible after I gathered this information. If I were to have this information contained within a database I might be able to utilize the information to initialize other processes based on the information of a specific machine or group of "like" machines. It is important to note that I am already currently managing a multitude of processes by identifying specific machine information. However pulling that information from a database after matching a unique identifier (in my mind) could greatly improve the efficiency. Also allowing for more of server side approach cutting down on the majority of client side scripting. (Instead of gathering this information from the client machine on the startup of each client I would have it already in a central database allowing a server to utilize the information and kick off specific events)
I am completely foreign to SQL and am not certain if it is 100% necessary. Is it necessary? For now I have decided to download and install both PostgreSQL and MySQL on separate Macs for testing. I am also fairly new to stackoverflow and apologize upfront if this is an inappropriate question or style of question. Any help including a redirection would be appreciated greatly.
I do not expect a step by step answer by any means, rather am just hoping for a generic "proceed..." "this indeed can be done..." or "don't bother there is a much easier solution."
As I come from the PostgreSQL world, I highly recommend using it for it's strong enterprise-level features and high standard compliance.
I always prefer to have a database for each project that I'm doing for the following benefits:
Normalized data is easier to process and build reports on;
Performance of database queries will be much better due to the caching done by the DB engine, indexes on your data, optimized query paths;
You can greatly improve machine data processing by using SQL/MED, which allows querying external data sources from the database directly. You can have a look on the Multicorn project and examples they provide.
Should it be required to deliver any kinds of reports to your management, DB will be your friend, while doing this outside the DB will be overly complicated.
Shortly — go for the database!