MYSQL Dump Formatted Text File - mysql

I have a service which produce formatted text files with information from certain tables in a MYSQL db.
Currently I programmatically query the DB and dump it in a TEXT file.
I wonder if i can do the same from within the MYSQL db.
This means that when a new record is added to a specific table, MYSQL will run a piece of code that will generate that formatted text file.

You didn't tell much about your use cases, but maybe the CSV engine might be what you're looking for?
From the manual:
When you create a CSV table, the server creates a table format file in the database directory. The file begins with the table name and has an .frm extension. The storage engine also creates a data file. Its name begins with the table name and has a .CSV extension. The data file is a plain text file. When you store data into the table, the storage engine saves it into the data file in comma-separated values format.
mysql> CREATE TABLE test (i INT NOT NULL, c CHAR(10) NOT NULL)
-> ENGINE = CSV;
Query OK, 0 rows affected (0.12 sec)
mysql> INSERT INTO test VALUES(1,'record one'),(2,'record two');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT * FROM test;
+------+------------+
| i | c |
+------+------------+
| 1 | record one |
| 2 | record two |
+------+------------+
2 rows in set (0.00 sec)
Starting with MySQL 5.1.9, creating a CSV table also creates a corresponding Metafile that stores the state of the table and the number of rows that exist in the table. The name of this file is the same as the name of the table with the extension CSM.
If you examine the test.CSV file in the database directory created by executing the preceding statements, its contents should look like this:
"1","record one"
"2","record two"
This format can be read, and even written, by spreadsheet applications such as Microsoft Excel or StarOffice Calc.

Related

XML Parsing and query on MySQL select query

I have a table where we have more than 2 Billions rows and my search is always based on Index field .
No new thing that need to try is to get the filed of xml string which is stored in a column which is longtext and varchar(4000)
Both of this column stores xml files one is big xml files and another one is small xmls file.
I need to query on the nodes or filed which is inside the xml files .
for example i need to get the statusCode which is present in fist xml file (DS column 6th column) and providerT which is present in second xml file (SY column last column)
Th search will happen always based on CT_ID which is index column and then on DS SY column .
so i am searching for a query something like
select * from AD_TABLE where CT_ID ='6ththeurtyiru' and DS.statusCode='COMPLETED' AND SY.providerT='noOfNRe';
I know my query is wrong but this can work if i have json but do we have any such thing for xml in MySql?
i have done some search but the query looks very very difficult to understand specially parsing query .
Can some one help me with this please .
ID,VERSION,AN_TYPE,EN_TYPE,CT_ID,DS,OT_TYPE,DATE_TIME,GP_ID,OT_NAME,OT_ID,UR_NAME,UR_ID,PT_ID,NS,SY
6uyuitygjh82mcrzz,0,SD_CASE,MYCHECK,6ththeurtyiru
,<?xml version="1.0" encoding="UTF-8" standalone="yes"?><sCPayload><cId>suerywe7r-21cf-4c7e-8071-suerywe7r</cId><statusCode>COMPLETED</statusCode><sngState>INPROGRESS</sngState><noOfNRe>1</noOfNRe><noOfRRR>0</noOfRRR><noOfER>0</noOfER><noOfARR>1</noOfARR><providerT>WATCHLIST</providerT></sCPayload>,CASE,9/16/2020 9:45,bd7c9519-d726-4672-8599-83d21927bec5,,5f53b903-21cf-4c7e-8071-suerywe7r,System User,USER_SYSTEM,,,<?xml version="1.0" encoding="UTF-8" standalone="yes"?><screenCES><providerT>MYWATCH</providerT><noOfNRe>1</noOfNRe></screenCES>,
In MySQL8.0, you can use ExtractValue
For MariaDB (10.5) the docs are: ExtractValue
mysql> set #xml='<sCPayload><cId>suerywe7r-21cf-4c7e-8071-suerywe7r</cId><statusCode>COMPLETED</statusCode><sngState>INPROGRESS</sngState><noOfNRe>1</noOfNRe><noOfRRR>0</noOfRRR><noOfER>0</noOfER><noOfARR>1</noOfARR><providerT>WATCHLIST</providerT></sCPayload>';
Query OK, 0 rows affected (0.00 sec)
mysql> select ExtractValue(#xml,'//statusCode');
+-----------------------------------+
| ExtractValue(#xml,'//statusCode') |
+-----------------------------------+
| COMPLETED |
+-----------------------------------+
1 row in set (0.00 sec)
mysql>
You can also put the result from extract in a permanent virtual field. so you can build a index or combined index to get a fast result and not a FULL TABLE SCAN.
This field will be automaticly be updated if you change the XML
Add this field to your Table
ALTER TABLE yourTable
ADD COLUMN
`statusCode` varchar(16) AS (ExtractValue(DS,'//statusCode')) PERSISTENT;

Merge CSV data into existing table

I want to merge data from CSV into an already existing table that has data:
The table I already have has 4 columns, 2 of which are NULL. My CSV file has 3 columns: one is the Primary Key to be used to match the table and the other 2 have information to fill my already-existing table.
I want to be able to match the CSV's Primary Key to my table's Primary Key, and insert the information from the CSV into the matched row.
My table goes something like:
+--------+----------------+------+
| CardID | Topic | Number | Date |
+--------+----------------+------+
pk00001 | name | NULL | NULL |
+--------+----------------+------+
Whereas the CSV goes like:
"ID","Number","Date"
"pk00001","100001","1999/01/01"
The table doesn't have any constraints: it's all VARCHAR(256)
However, my table has more rows than the CSV, and the operation shouldn't touch those rows.
Is it possible to create such a query in MySQL?
MySQL doesn't have a function to do exactly what you're asking. There may be database clients that can do this for you, but your best bet would be to do it the old fashioned way.
First, load your csv file into a new table in your database. Most database clients will be able to do this for you, but if you want to do it from your shell, you can look at mysqlimport or LOAD DATA INFILE.
Once your data is in your table (you may want to index the key as well, depending on the size), you can do a simple update with a join to your new table.
UPDATE
original_table
SET
CardName = new.CardName,
CardText = new.CardText,
-- ...And the rest of your columns to update
FROM
original_table old
join imported_table new
on (old.CardID=new.CardID)
Then drop your csv table, and you're good to go.
A simple (but far more dangerous and less performant) alternative would be to open your csv file in Excel and use a formula to generate the UPDATE statements for you. For example:
="UPDATE original_table SET CardName='"&B1&"', CardText='"&C1&"' WHERE CardID='"&A1&"'"
Then just populate each row with this formula, and you'll have an update statement for every row you want to modify.
Please be aware that this Excel method is opening you up to SQL injections (both accidental, as if Card Text said "for John O'Brien", and potentially malicious if any of the csv content comes from an external source).

Calling stored procedures on temporary tables

Is it possible to execute stored procedures on temporary tables in MySQL? I'm trying to create a system for data import that could, theoretically, run concurrently for different users. If one user is importing files into a temporary table while another user is doing the same thing, is it possible for the same shared procedure to be called by both users since the tables referenced in the procedure will match the temporary tables?
The workflow for an individual user would look like this...
Load data into temporary table newdata
Stored procedure is called where munging and updates are done to table newdata
Stored procedure moves data from newdata to the live/permanent tables.
...while another user could, possibly, be doing the same thing.
Yes, you can reference temp tables in a stored procedure:
mysql> create procedure p() select * from t;
Query OK, 0 rows affected (0.03 sec)
mysql> create temporary table t as select 123 union select 456;
Query OK, 2 rows affected (0.02 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> call p;
+-----+
| 123 |
+-----+
| 123 |
| 456 |
+-----+
(Tested on MySQL 5.6.31)
However, most experienced MySQL users try to avoid doing what you're planning, if they use replication. The reason is that when a slave restarts for any reason, it drops any temp tables. So any subsequent UPDATE and INSERT...SELECT referencing the temp table get an error because the temp table no longer exists. An error in the replication stream stops replication.
It might seem like this should be an uncommon occurrence for a slave to restart suddenly, but if your app creates temp tables frequently, there's a high chance for there to be a temp table going at the moment a slave restarts.
The best use of temp tables is to fill them with temp data in one statement, and then use the temp table only by SELECT queries (which are not replicated).

Is there a way how to recover deleted rows in Mysql deleted over a month ago

I am using Mysql and by mistake deleted very important details.Is there a way how to recover deleted records in Mysql?
I know this is not really what you are asking but I find it important that you know this.
The system I suggest is called soft delete. It works as follow:
Make a new column in your table (for ex the table people):
ID
Name
Deleted
1
Bob
0
2
Frank
1
3
Alice
0
With 0 as undeleted and 1 as deleted in last column with the type BOOLEAN. Name and ID are respectively a NVARCHAR and an INT.
If you want delete a record, you don't delete it with a DELETE statement but update it like this:
UPDATE people SET Deleted = 1 WHERE ID = 1
Result:
ID
Name
Deleted
1
Bob
1
2
Frank
1
3
Alice
0
You know ID's 1 and 2 are deleted because the boolean in column Deleted is on 1. Alice with ID 3 is not deleted because the boolean is on 0.
The pro of this system is that you can recover data with only one statement!
UPDATE people SET Deleted = 0 WHERE Deleted = 1
Result:
ID
Name
Deleted
1
Bob
0
2
Frank
0
3
Alice
0
Now all record are recovered because the boolean in column Deleted is on 0.
If you use DELETE statement, it is impossible to recover all records! They are deleted for always and you can never recover it if you are using the DELETE statements. You can only use a backup file for recover it but has also contra's like:
It's an old backup file,
I've forget to make one,
I don't know how you can make it,
I've lost it,
...
By soft delete you change only one column and you have your data back.
Edit:
The contra of the system is that (like you have said) the data is not exactly remove from your database. It's only a column you change from 0 to 1. But if you know that you can make money from data... is this another story.
If you want to delete it exactly you can use a DELETE statement.
Do you have binlog files from the date where you insert these records ?
Then you can get the inserts and put it back
mysql> use employees;
Database changed
mysql> show create table employees;
| employees | CREATE TABLE `employees` (
`emp_no` int(11) NOT NULL,
`birth_date` date NOT NULL,
`first_name` varchar(14) NOT NULL,
`last_name` varchar(16) NOT NULL,
`gender` enum('M','F') NOT NULL,
`hire_date` date NOT NULL,
PRIMARY KEY (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
mysql> select count(*) from employees;
+----------+
| count(*) |
+----------+
| 1000 |
+----------+
1 row in set (0.00 sec)
mysql> delete from employees;
mysql> flush table employees with read lock;
Query OK, 0 rows affected (0.01 sec)
we can directly scan the table .ibd file ,and find the deleted records:
select add .ibd from menu start
input the deleted table's ibd location
right click the ibd file , scan it
click the first page node , and input create table sql
choose the right mysql version , and page format (dynamic in most version)
select the Deleted Data Tab , then you can check deleted records and export them into MYSQLDUMP format SQL file.
You can also use conventional recovery mode in this case.
https://youtu.be/dVMO0YYcJoo
For InnoDB tables, if binary logs are enabled, one could use:
mysqlbinlog path_to_binary_log_file > query_log.sql
If binary logs are not enabled, there is a recovery tool provided by Percona which may help.
Created a script to automate the steps from the Percona tool which recovers the deleted rows (if exist) and provides SQL queries to load the data back into database.
Please note:
The time between the deletion of rows and the database stop is
crucial. If pages are reused you can’t recover the data.
PS: the DBRecover Tool by #ParnassusData is cool and much easier IMO

Easy way to store metadata about MySQL-Database

Earlier today, I asked for an easy way to store a version number for the SQL table layout you are using in SQLite, and got the suggestion to use PRAGMA user_version. As there is no such thing as a Pragma in MySQL, I was wondering on how you would go about this in MySQL (Except for creating a table named "META" with a column "DB-Scheme-Version").
Just to repeat what I said in the linked question: I'm not looking for a way to find out which version of MySQL is installed, but to save a version nuber that tells me what version of my MySQL-Scheme I am using, without checking every table via script.
I also saw this question, but it only allows me to version single tables. Is there something similar or, preferably, easier, for whole Databases (Since it would be no fun to query every single table seperately)? Thanks in advance.
MySQL's SET GLOBAL would probably work, but I prefer a solution that does not reset itself every time the server reboots and does not require SUPER Privilege and / or access to the configuration file to use. To put it short: It should work with a standard MySQL-Database that you get when you rent a small webhosting package, not the ones you get if you rent a full server, as you tend to have more access to those.
There are a couple of choices, depending on the privileges that you have. The higher privileges you have, the more “elegant” the solution.
The most direct route is to create a stored function, which requires the CREATE ROUTINE privilege. e.g.
mysql> CREATE FUNCTION `mydb`.DB_VERSION() RETURNS VARCHAR(15)
RETURN '1.2.7.2861';
Query OK, 0 rows affected (0.03 sec)
mysql> SELECT `mydb`.DB_VERSION();
+--------------+
| DB_VERSION() |
+--------------+
| 1.2.7.2861 |
+--------------+
1 row in set (0.06 sec)
If your privileges limit you to only creating tables, you can create a simple table and put the version in a default value:
mysql> CREATE TABLE `mydb`.`db_version` (
`version` varchar(15) not null default '1.2.7.2861');
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW COLUMNS FROM `mydb`.`db_version`;
+---------+-------------+------+-----+------------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+-------------+------+-----+------------+-------+
| version | varchar(15) | NO | | 1.2.7.2861 | |
+---------+-------------+------+-----+------------+-------+
1 row in set (0.00 sec)