I'm developing one app that store data locally and send CSV when internet is available.
The goal is to populate an on-line MySql db.
The question is: considering an environment with very low quality internet connection, is better to send data as CSV to check and then put in db or directly to db?
I suppose directly populate db is less expensive in terms of connection but is safe for db integrity and consistency if the data connection get lost?
If you can insert the data with a single SQL statement, it will be treated as an atomic operation and either fail completely or succeed completely. If you need multiple insert statements, wrap the whole thing in a transaction so it gets treated as an atomic operation.
Related
I have node project with multiple "storages". More specifically MySQL, Redis and Minio.
When user fill form on frontend, data is send to backend and there are following operations: create record in MySQL (with set transaction), put something into Redis queue and put something into
Minio Storage, exactly in that order.
Imagine that third step - put something into Minio storage, will fail. MySQL is okay, that's treated by rollback, but with Redis? So in general, is there some way, how approach this problem?
I have a system in which data is written constantly. It works on MySQL, I also have a second system that runs on SQL Server and uses some parameters from the first base.
Question: how is it possible (is this even possible) to constantly transfer values from one base (MySQL) to another (SQL Server)? The option to switch to one base is not an option. As I understand it, it will be necessary to write a program for example in Delphi which will transfer values from the other database to another.
You have a number of options.
SQL Server can access another database using ODBC, so you could setup SQL server to obtain the information it needs directly from tables that are held in MySQL.
MySQL supports replication using log files, so you could configure MySQL replication (which does not have to be on all tables) to write relevant transactions to a log file. You would then need to process that log file (which you could do in (almost) real time as the standard MySQL replication does) to identify what needs to be written to the MS SQL Server. Typically this would produce a set of statements to be run against the MS SQL server. You have any number of languages you could use to process the log file and issue the updates.
You could have a scheduled task that reads the required parameters from MySQL and posts it to MS SQL, but this would leave a period of time where the two may not be in sync. Given that you may have an issue with parsing log files and posting the updates you may still want to implement this as a fall back if you are processing log files.
If the SQL Server and the MySQL server are on the same network the external tables method is likely to be simplest and lowest maintenance, but depending on the amount of data involved you may find the overhead of the external connection and queries could affect the overall performace of the queries made against the MS SQL Server.
For a project we are working with an several external partner. For the project we need access to their MySQL database. The problem is, they cant do that. Their databse is hosted in a managed environment where they don't have much configuration possibilities. And they dont want do give us access to all of their data. So the solution they came up with, is the federated storage engine.
We now have one table for each table of their database. The problem is, the amount of data we get is huge and will even increase in the future. That means there are a lot of inserts performed on our database. The optimal solution for us would be to intercept all incoming MySQL traffic, process it and then store it in bulk. We also thought about using someting like redis to store the data.
Additionnaly, we plan to get more data from different partners. They will potentialy provide us the data in different ways. So using redis would allow us, to have all our data in one place.
Copying the data to redis after its stored in the mysql database is not an option. We just cant handle that many inserts and we need the data as fast as possible.
TL;DR
is there a way to pretend to be a MySQL server so we can directly process data received via the federated storage engine?
We also thought about using the blackhole engine in combination with binary logging on our side. So incoming data would only be written to the binary log and wouldn't be stored in the database. But then performance would still be limited by Disk I/O.
I am currently importing a huge CSV file from my iPhone to a rails server. In this case, the server will parse the data and then start inserting rows of data into the database. The CSV file is fairly large and would take a lot time for the operation to end.
Since I am doing this asynchronously, my iPhone is then able to go to other views and do other stuff.
However, when it requests another query in another table.. this will HANG because the first operation is still trying to insert the CSV's information into the database.
Is there a way to resolve this type of issue?
As long as the phone doesn't care when the database insert is complete, you might want to try storing the CSV file in a tmp directory on your server and then have a script write from that file to the database. Or simply store it in memory. That way, once the phone has posted the CSV file, it can move on to other things while the script handles the database inserts asynchronously. And yes, #Barmar is right about using an InnoDB engine rather than MyISAM (which may be default in some configurations).
Or, you might want to consider enabling "low-priority updates" which will delay write calls until all pending read calls have finished. See this article about MySQL table locking. (I'm not sure what exactly you say is hanging: the update, or reads while performing the update…)
Regardless, if you are posting the data asynchronously from your phone (i.e., not from the UI thread), it shouldn't be an issue as long as you don't try to use more than the maximum number of concurrent HTTP connections.
I have a huge SQL 2008 DB with ~300 million rows. some of the tables have columns with an encrypted value.
The DB was gradually build using an application (.net 4).
We are considering to move to some hardware (USB token) like encryption but that mean we will have to change the encrypted values in the DB.
We've written a small app that decrypts with the old key and then encrypt with the token but it takes days for it to run since I have to SELECT the row and UPDATE one at a time by ID. the db is indexed but still...
The Encrypt(string) method is a functionality provided by the USB token and I can access it via .net
I'm looking for a more direct way to use that functionality. maybe access it through SQL or something.
You can use a CLR stored procedure to access the USB dongle on the server. You'd need UNSAFE access of course
This will reduce network overhead because you don't want to do a single 300 million row update. You'd still need it RBAR or batched: I'd suggest RBAR to keep it simple.