Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I'm developing a small experimental project using Node.js and MySQL. The database has about 20 tables only, with less than 100 registers on each table. I work with this project in two different environments and would like to make it easier to keep the databases updated, for both architecture and data. So I desire to replace the MySQL engine with another that better fits the requirements:
SQL;
Compatible with Node js;
Easy to install/access/carry;
Free;
I think SQLite could solve my issue, but I'm not sure how the single database file will behave when managed by git. Another option would be an online database, but I don't know any that is SQL and free.
Do you have any suggestions?
I guess your application must initialize a database the first time you start it up after cloning it. That is, it must create some tables and load rows into them.
You can handle this with SQLite by saving the database file as a binary git object. That should work OK, at least until the next version of SQLite comes out and breaks database file compatibility.
But a better way is to create a SQL file to do the database initialization, and store that in git. It contains the CREATE TABLE and INSERT operations necessary to set up your database. (If you're wise you will write that SQL code so it works on both SQLite and MySQL: you'll be able to switch database servers in the future.)
Then, when your application first runs, it opens the database software and checks whether your tables are present. If they are not, your application loads them from your SQL table.
I guess you also want to share information inserted into the database when the application runs in more than one place. Obviously a shared database server is a good SQL way to do this.
Without a shared server, a good way to do this is to build some kind of "save" operation into your application. It will write out the SQL INSERT for the shared information, which you can then send to another location or commit to git.
Free public shared database servers? Your best bet is probably to run MySQL on a free-tier virtual machine on one of the server-rental ("cloud") services. Both AWS and Azure offer free tier. And Digital Ocean costs US$5 per month for a small virtual machine.
Related
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 5 days ago.
Improve this question
I have a database (MySQL, AWS RDS). This is a production database that has customer information including names, emails, bank account information. Some of it is encrypted, some of it is not.
We want to setup an environment that can be regularly used for automated testing. We want the database for the test environment to be the same as production except we want to replace customer data.
We want to do this in a way where the customer data never leaves the production environment. We don’t mind creating an “intermediate” environment that may initially contain some customer data but then gets removed. From the intermediate environment, we’d transfer the cleaned database to the testing environment.
Appreciate the guidance since I’m way out of my depth here
There is no easy / automated solution to do this that I am aware of. You need to replicate your data to a different system and have that replication service scrub your data for you. A few options come to mind:
You could write a batch processor that dumps the DB down to disk, loads it into a secondary server (staging/scrubbing environment), and then run a series of cleanup scripts. Then you can again dump the data down.
You could write a database trigger that fires on the events you care about and maintains a staging table of sanitized data.
You could write a test data generator that users the patterns of your production data to generate fake data in a testing table. There are many tools that can help with this, both open source and commercial.
Personally, I lean towards the last option because it's the safest and can be used in many places, like on a local dev machine, CI/CD system, shared staging environment, etc. I do believe there is a strong case for sending a copy of a subset of production data to canary systems as part of a rollout strategy, though. Effectively testing your release with live data before wiring it to your live database.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed last month.
Improve this question
I am a new MySQL learner and like to use it within a NodeJS application. From my previous knowledge with MongoDB all the tables we need will be created by MongoDB automatically. We only needed to connect to the MongoDB and with create or save commands it could find the table and put something inside or create the table if it was not exist.
But from MySQL tutorials I have seen so far they create databases manually by executing SQL codes directly inside a MySQL shell/bash or by using applications like MySQL Workbench.
So I want to make sure which way is correct? Should I create my tables before running my Node app or there is also a way, and it is a good way to create tables dynamically within the code when it's needed regarding the app usage by the user? If the second way is a better way, how can I do that? Is there a code sample for creating databases from inside a Node app?
Moreover the same question for databases. Can we or is it a good way creating them from inside our Node app?
In sql-based applications standard tables are created when you install the application giving it a fix structure. Table structure is only changed during program updates as structure changes can be a very expensive operation. The initial table creation, subsequent changes can be done via sql scripts or by application code - both are acceptable practices. But the main point stands: permanent tables are not created on the fly.
Some complex scripts may require temporary tables that are created and destroyed on the fly, but most of these will be done implicitly (e.g. database engine may materialise the resultset of a subquery as a temporary table), rather than explicitly (application code executing create temporary table statement).
Rationale: RDBMSs have a rigid, inflexible data structure that is optimised towards data storage and retrieval speed. Any solution to introduce flexibility is at odds with relational theory and will come with its drawbacks (see Bill Karwin's excellent answer on this subject, particularly the last two pragraphs).
NoSQL solutions, however, more focus on flexibility, than data storage. Mongodb for example, does not even have tables with rows and columns. It has collections that are made up of documents and every document in a collection may have different properties. This is why you can create these collections and documents on the fly.
Both are possible.
You can create a piece of code that will check the schema and update it if it is not (there are lots of tools and for all languages).
In my experience, in most cases we tend to create the tables, indexes, views, etc. by hand first and then run the application. For larger companies / projects, they often use tools. I personally have seen a lot of flyway in the Java world.
In any case, the most important thing is that your creation script is idempotent. For this, you can use the if not exists statements : https://dev.mysql.com/doc/refman/8.0/en/replication-features-create-if-not-exists.html
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am newbie to web development.
I have an application server where my ASP.NET code resides. My application server communicates to a MySQL instance which is on a different server.
I was wondering, whether it is a good practice to move the computation from the application server to the database server by having a Stored Procedure with Views or should I just move on with all logic kept in application server and query the database only to retrieve data from tables directly without having stored procedures and views.
I am a strong advocate of putting database logic into the database and not splitting it between the application and the server. This means that I prefer to wrap all database calls in stored procedures and views.
The driving reasons are maintenance, security, and functionality, not performance, although performance is often better on the server side.
The number one reason is to isolate the application from changes in the underlying data structure. So, if the data structure changes, the application does not (always) break.
Other reasons the come to mind:
The same logic gets used for the same thing. That is, one piece of code doesn't define "foobar" one way and another "foobar" another way.
Auditing and logging are implemented within stored procedures rather than using triggers.
Database tables are off-limits to all users, unless they go through the defined interface.
A newer version and older version can often co-exist.
Admittedly, for a one-off, quick-and-dirty application these issues may not be important. However, I think it is a good idea to have well defined interfaces (APIs) between different components of a system, and databases and the application layer are a prime example where such APIs are quite useful.
I agree with Gordon on separating out a "layer" of code between the application and the actual database. I dispute how practical Stored Routines are at such.
PHP (etc) is far more expressive than SProcs.
One SProc can execute multiple queries faster because it is closer to the server. This can be an overwhelming performance gain if the client and server are on opposite sides of the country.
Error checking is clumsy in SProcs.
PHP recompiles only when the code changes; SProcs recompile once per connection; Perl always recompiles; etc.
VIEWs are sometimes poorly optimized, so I avoid them.
The secret to a good design for the "layer" is in the compromise between the forces tugging on either side. One example: Can you completely hide a schema change from the app? Even if you split one table into two?
A really bad example was when the UI did pagination by using page numbers. The layer thought in terms of OFFSET and LIMIT, and fed that to the MySQL back-end. Then came an item will 216K pages (Yes, that many!) They found out that OFFSET+LIMIT is not a good way to implement "next page", but fixing it required a changes to all layers of the system.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'm working on a project which is somewhat familiar to WhatApp, except that I'm not implementing the chatting function.
The only thing I need to store in the database is user's information, which won't be very large, and I also need an offline database in the mobile app which should be synced with the server database.
Currently I use MySQL for my server database, and I'm thinking of using JSON for the syncing between mobile app and the server, then I found that mongoDB has a natural support for JSON, which caused me wonder should I change to mongoDB.
So here are my questions:
Should I change to mongoDB or should I still use MySQL? The data for each user won't be too large and it does have some requirement for data consistency. But mongoDB's JSON support is somewhat attractive.
I'm not familiar with the syncing procedure, I did some digging and it appears that JSON is a good choice, but what data should I put into the JSON files?
I actually flagged this as too broad and as attracting primarily opinion based answers but I'll give it a go anyhow and hope to stay objective.
First of all you have 2 separate questions here.
What database system should I use.
How do I sync between app and server.
The first one is easily answered because it doesn't really matter. Both are good options for storing data. MySQL is mature and stable and MongoDB although it's newer has very good reviews and I don't know of any known problems which would prevent it from being used. So take the database which you find easy to use.
Now for second I'll first put in a disclaimer that for data synchronization between multiple entities entire books are written and that it is after all this time still the subject of Phds.
I would advice against directly synchronizing between mobile app and database because that requires the database credentials to be contained within the app. Mobile apps can and will be decompiled and credentials extracted which would compromise your entire database. So you'll probably want to create some API which first does device/user authentication and then changes the database.
This already means that using MongoDB for sake of this would probably be a bad idea.
Now JSON itself is just a format of representing data with some structure, just as XML. As such it's not a method of synchronization but transport.
For synchronizing data it's important that you know the source of truth.
If you have 1 device <-> 1 record it's easy because the device will be the source of truth, after all the only mutations that take place are presumably done by the user on the device.
If you have n devices <-> 1 record then it becomes a whole lot more annoying. If you want to allow a device to change the state when offline you'll need to do some tricks to synchronize the data when the device comes back online. But this is probably a question too complex and situation dependent to answer on SO.
If you however force the device to always immediately propagate changes to the database then the database will always contain the most up to date record, or truth. Downside is that part of the app will not be functional when offline.
If offline updates don't change the state but merely add new records then you can push those to the server when it comes online. But keep in mind you won't be able to order these events.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I would like to create a database with the goal of populating this database with comprehensive inventory information obtained via a shell script from each client machine. My shell script currently writes this information to a single csv file located on a server through an ssh connection. Of course, if this script were to be run on multiple machines at once it would likely cause issues as each client potentially would try to write to the csv at the same time.
In the beginning, the inventory was all I was after; however after more thought I began to ponder wether or not much much more could be possible after I gathered this information. If I were to have this information contained within a database I might be able to utilize the information to initialize other processes based on the information of a specific machine or group of "like" machines. It is important to note that I am already currently managing a multitude of processes by identifying specific machine information. However pulling that information from a database after matching a unique identifier (in my mind) could greatly improve the efficiency. Also allowing for more of server side approach cutting down on the majority of client side scripting. (Instead of gathering this information from the client machine on the startup of each client I would have it already in a central database allowing a server to utilize the information and kick off specific events)
I am completely foreign to SQL and am not certain if it is 100% necessary. Is it necessary? For now I have decided to download and install both PostgreSQL and MySQL on separate Macs for testing. I am also fairly new to stackoverflow and apologize upfront if this is an inappropriate question or style of question. Any help including a redirection would be appreciated greatly.
I do not expect a step by step answer by any means, rather am just hoping for a generic "proceed..." "this indeed can be done..." or "don't bother there is a much easier solution."
As I come from the PostgreSQL world, I highly recommend using it for it's strong enterprise-level features and high standard compliance.
I always prefer to have a database for each project that I'm doing for the following benefits:
Normalized data is easier to process and build reports on;
Performance of database queries will be much better due to the caching done by the DB engine, indexes on your data, optimized query paths;
You can greatly improve machine data processing by using SQL/MED, which allows querying external data sources from the database directly. You can have a look on the Multicorn project and examples they provide.
Should it be required to deliver any kinds of reports to your management, DB will be your friend, while doing this outside the DB will be overly complicated.
Shortly — go for the database!