im trying to detect and delete a line break out of a subject (called m.subject) mail information retrieved via CONCAT out of a mysql database.
That said, the linebreak may or may not occur in the subject and therefore must be detected.
My query looks like this:
mysql --default-character-set=utf8 -h $DB_HOST -D $TARGET -u $DB_USER -p$DB_PW -N -r -e "SELECT CONCAT(m.one,';',m.two,';',m.three,';',m.subject,';',m.four';',m.five,';',(SELECT CONCAT(special_one) FROM special_$SQL_TABLE WHERE msg_id = m.six ORDER BY time DESC LIMIT 1)) FROM mails_$SQL_TABLE m WHERE m.rtime BETWEEN $START AND $END AND m.seven = 1 AND m.eight IN (2);"
I tried to delete it afterwards, but getting in performance trouble due to several while operations on all lines already. Is there an easy way to detect and cut it directly via the CONCAT buildup? It is crucial to retrieve only one line after extraction for me.
Updating/changing the database is not an option for me, as I only want to read the current state.
I am storing output of MySQL query in a varible using shell scripting. The output of SQL query is in multiple rows. When I checked the count of the variable (which I think is an array), it is giving 1. My code snippet is as follows:
sessionLogin=`mysql -ugtsdbadmin -pgtsdbadmin -h$MYSQL_HOST -P$MYSQLPORT CMDB -e " select distinct SessionID div 100000 as 'MemberID' from SessionLogin where ClientIPAddr like '10.104%' and LoginTimestamp > 1426291200000000000 order by 1;"`
echo "${#sessionLogin[#]}"
How can I store the MySQL query output in an array in shell scripting?
You can loop over the output from mysql and append to an existing array. For example, in Bash 3.1+, a while loop with process substitution is one way to do it (please replace the mysql parameters with your actual command)
output=()
while read -r output_line; do
output+=("$output_line")
done < <(mysql -u user -ppass -hhost DB -e "query")
echo "There are ${#output[#]} lines returned"
Also take a look at the always excellent BashFaq
I have a table that holds relative paths to real files on HDD. for example:
SELECT * FROM images -->
id | path
1 | /files/1.jpg
2 | /files/2.jpg
Can I create a query to select all records pointing to non-existent files? I need to check it by MySql server exactly, without using an iteration in PHP-client.
I would go with a query like this:
SELECT id, path, ISNULL(LOAD_FILE(path)) as not_exists
FROM images
HAVING not_exists = 1
The function LOAD_FILE tries to load the file as a string, and returns NULL when it fails.
Please notice that a failure in this case might be due to the fact that mysql simply cannot read that specific location, even if the file actually exists.
EDIT:
As #ostrokach pointed out in comments, this isn't standard SQL, even though MySQL allows it, to follow the standard it could be:
SELECT *
FROM images
WHERE LOAD_FILE(PATH) IS NULL
The MySQL LOAD_FILE command has very stringent requirements on the files that it can open. From the MySQL docs:
[LOAD_FILE] Reads the file and returns the file contents as a string. To use this function, the file must be located on the server host, you must specify the full path name to the file, and you must have the FILE privilege. The file must be readable by all and its size less than max_allowed_packet bytes. If the secure_file_priv system variable is set to a non-empty directory name, the file to be loaded must be located in that directory.
So if the file can't be reached by the mysql user or any of the other requirements are not satisfied, LOAD_FILE will return Null.
You can get a list of IDs that correspond to missing files using awk:
mysql db_name --batch -s -e "SELECT id, path FROM images" \
| awk '{if(system("[ -e " $2 " ]") == 1) {print $1}}' \
>> missing_ids.txt
or simply using bash:
mysql db_name --batch -s -e "SELECT id, path FROM images" \
| while read id path ; if [[ -e "$path" ]] ; then echo $id ; done
>> missing_ids.txt
This also has the advantage of being much faster than LOAD_FILE.
MYSQL only handles the Database so there is no way for you to fire an SQL Statement to check on the HDD if the file exists. You need to iterate over the rows and check it with PHP.
It's not possible using stock MySQL. However you can write UDF (user-defined function), probably in C, load it using CREATE FUNCTION statement and use it from MySQL as you would use any built-in function.
This has been asked a few times but I cannot find a resolution to my problem. Basically when using mysqldump, which is the built in tool for the MySQL Workbench administration tool, when I dump a database using extended inserts, I get massive long lines of data. I understand why it does this, as it speeds inserts by inserting the data as one command (especially on InnoDB), but the formatting makes it REALLY difficult to actually look at the data in a dump file, or compare two files with a diff tool if you are storing them in version control etc. In my case I am storing them in version control as we use the dump files to keep track of our integration test database.
Now I know I can turn off extended inserts, so I will get one insert per line, which works, but any time you do a restore with the dump file it will be slower.
My core problem is that in the OLD tool we used to use (MySQL Administrator) when I dump a file, it does basically the same thing but it FORMATS that INSERT statement to put one insert per line, while still doing bulk inserts. So instead of this:
INSERT INTO `coupon_gv_customer` (`customer_id`,`amount`) VALUES (887,'0.0000'),191607,'1.0300');
you get this:
INSERT INTO `coupon_gv_customer` (`customer_id`,`amount`) VALUES
(887,'0.0000'),
(191607,'1.0300');
No matter what options I try, there does not seem to be any way of being able to get a dump like this, which is really the best of both worlds. Yes, it take a little more space, but in situations where you need a human to read the files, it makes it MUCH more useful.
Am I missing something and there is a way to do this with MySQLDump, or have we all gone backwards and this feature in the old (now deprecated) MySQL Administrator tool is no longer available?
Try use the following option:
--skip-extended-insert
It worked for me.
With the default mysqldump format, each record dumped will generate an individual INSERT command in the dump file (i.e., the sql file), each on its own line. This is perfect for source control (e.g., svn, git, etc.) as it makes the diff and delta resolution much finer, and ultimately results in a more efficient source control process. However, for significantly sized tables, executing all those INSERT queries can potentially make restoration from the sql file prohibitively slow.
Using the --extended-insert option fixes the multiple INSERT problem by wrapping all the records into a single INSERT command on a single line in the dumped sql file. However, the source control process becomes very inefficient. The entire table contents is represented on a single line in the sql file, and if a single character changes anywhere in that table, source control will flag the entire line (i.e., the entire table) as the delta between versions. And, for large tables, this negates many of the benefits of using a formal source control system.
So ideally, for efficient database restoration, in the sql file, we want each table to be represented by a single INSERT. For an efficient source control process, in the sql file, we want each record in that INSERT command to reside on its own line.
My solution to this is the following back-up script:
#!/bin/bash
cd my_git_directory/
ARGS="--host=myhostname --user=myusername --password=mypassword --opt --skip-dump-date"
/usr/bin/mysqldump $ARGS --database mydatabase | sed 's$VALUES ($VALUES\n($g' | sed 's$),($),\n($g' > mydatabase.sql
git fetch origin master
git merge origin/master
git add mydatabase.sql
git commit -m "Daily backup."
git push origin master
The result is a sql file INSERT command format that looks like:
INSERT INTO `mytable` VALUES
(r1c1value, r1c2value, r1c3value),
(r2c1value, r2c2value, r2c3value),
(r3c1value, r3c2value, r3c3value);
Some notes:
password on the command line ... I know, not secure, different discussion.
--opt: Among other things, turns on the --extended-insert option (i.e., one INSERT per table).
--skip-dump-date: mysqldump normally puts a date/time stamp in the sql file when created. This can become annoying in source control when the only delta between versions is that date/time stamp. The OS and source control system will date/time stamp the file and version. Its not really needed in the sql file.
The git commands are not central to the fundamental question (formatting the sql file), but shows how I get my sql file back into source control, something similar can be done with svn. When combining this sql file format with your source control of choice, you will find that when your users update their working copies, they only need to move the deltas (i.e., changed records) across the internet, and they can take advantage of diff utilities to easily see what records in the database have changed.
If you're dumping a database that resides on a remote server, if possible, run this script on that server to avoid pushing the entire contents of the database across the network with each dump.
If possible, establish a working source control repository for your sql files on the same server you are running this script from; check them into the repository from there. This will also help prevent having to push the entire database across the network with every dump.
As others have said using sed to replace "),(" is not safe as this can appear as content in the database.
There is a way to do this however:
if your database name is my_database then run the following:
$ mysqldump -u my_db_user -p -h 127.0.0.1 --skip-extended-insert my_database > my_database.sql
$ sed ':a;N;$!ba;s/)\;\nINSERT INTO `[A-Za-z0-9$_]*` VALUES /),\n/g' my_database.sql > my_database2.sql
you can also use "sed -i" to replace in-line.
Here is what this code is doing:
--skip-extended-insert will create one INSERT INTO for every row you have.
Now we use sed to clean up the data. Note that regular search/replace with sed applies for single line so we cannot detect the "\n" character as sed works one line at a time. That is why we put ":a;N;$!ba;" which basically tells sed to search multi-line and buffer the next line.
Hope this helps
What about storing the dump into a CSV file with mysqldump, using the --tab option like this?
mysqldump --tab=/path/to/serverlocaldir --single-transaction <database> table_a
This produces two files:
table_a.sql that contains only the table create statement; and
table_a.txt that contains tab-separated data.
RESTORING
You can restore your table via LOAD DATA:
LOAD DATA INFILE '/path/to/serverlocaldir/table_a.txt'
INTO TABLE table_a FIELDS TERMINATED BY '\t' ...
LOAD DATA is usually 20 times faster than using INSERT statements.
If you have to restore your data into another table (e.g. for review or testing purposes) you can create a "mirror" table:
CREATE TABLE table_for_test LIKE table_a;
Then load the CSV into the new table:
LOAD DATA INFILE '/path/to/serverlocaldir/table_a.txt'
INTO TABLE table_for_test FIELDS TERMINATED BY '\t' ...
COMPARE
A CSV file is simplest for diffs or for looking inside, or for non-SQL technical users who can use common tools like Excel, Access or command line (diff, comm, etc...)
I'm afraid this won't be possible. In the old MySQL Administrator I wrote the code for dumping db objects which was completely independent of the mysqldump tool and hence offered a number of additional options (like this formatting or progress feedback). In MySQL Workbench it was decided to use the mysqldump tool instead which, besides being a step backwards in some regards and producing version problems, has the advantage to stay always up-to-date with the server.
So the short answer is: formatting is currently not possible with mysqldump.
Try this:
mysqldump -c -t --add-drop-table=FALSE --skip-extended-insert -uroot -p<Password> databaseName tableName >c:\path\nameDumpFile.sql
I found this tool very helpful for dealing with extended inserts: http://blog.lavoie.sl/2014/06/split-mysqldump-extended-inserts.html
It parses the mysqldump output and inserts linebreaks after each record, but still using the faster extended inserts. Unlike a sed script, there shouldn't be any risk of breaking lines in the wrong place if the regex happens to match inside a string.
I liked Ace.Di's solution with sed, until I got this error:
sed: Couldn't re-allocate memory
Thus I had to write a small PHP script
mysqldump -u my_db_user -p -h 127.0.0.1 --skip-extended-insert my_database | php mysqlconcatinserts.php > db.sql
The PHP script also generates a new INSERT for each 10.000 rows, again to avoid memory problems.
mysqlconcatinserts.php:
#!/usr/bin/php
<?php
/* assuming a mysqldump using --skip-extended-insert */
$last = '';
$count = 0;
$maxinserts = 10000;
while($l = fgets(STDIN)){
if ( preg_match('/^(INSERT INTO .* VALUES) (.*);/',$l,$s) )
{
if ( $last != $s[1] || $count > $maxinserts )
{
if ( $count > $maxinserts ) // Limit the inserts
echo ";\n";
echo "$s[1] ";
$comma = '';
$last = $s[1];
$count = 0;
}
echo "$comma$s[2]";
$comma = ",\n";
} elseif ( $last != '' ) {
$last = '';
echo ";\n";
}
$count++;
}
add
set autocommit=0;
to first line of your sql script file, then import by:
mysql -u<user> -p<password> --default-character-set=utf8 db_name < <path>\xxx.sql
, it will fast 10x.
Given a file which consists of multiple line MySQL queries, eg
SELECT foo, bar, etc
FROM blah
WHERE something or other
LIMIT etc
Is there any way I can visually select a query in Vim, pipe it through MySQL, and see the query and result in a new buffer?
Clarification: I don't want tabular output, but something that can be further processed in vim or imported into a spreadsheet (like the tab-separated output that you get from mysql --batch)
(Ubuntu Linux).
The Dbext plugin supports this behavior.
Visually select the SQL statement, and run :DBExecRangeSQL to execute it.
The result will be returned into a new split at the bottom of your current viewport.
There are lots and lots of options for controlling the output window. See :help dbext for the glorious details.
Update 1.May.2012
Version 15.0 of the plugin has been released with this functionality built in.
The default -t flag can be overridden
Default setting:
let g:dbext_default_MYSQL_extra = '-t'
Overridden for batch setting
let g:dbext_default_MYSQL_extra = '--batch --raw'
Dbext hard-codes the -t option to MySQL, but if that line is removed from dbext.vim, on line 2278 in DB_MYSQL_execSql (of my current version), you can pass the --batch and --raw options:
:DBSetOption MYSQL_cmd_options='--batch --raw'
To restore tabular output:
:DBSetOption MYSQL_cmd_options='-t'
I tested this successfully on my installation.
Thanks to Michael and Zsolt Botykai for suggesting dbext and other vim plugins - they don't seem to provide raw output though.
I've put the following in my .vimrc, inspired by Matias' answer here. This is my first attempt at vimscript, so caveat emptor...
function Query() range
" use a temp file for result
let s:tmpfile = system('mktemp')
" single line copy of query followed by blank line
echo system('echo '.shellescape(join(getline(a:firstline,a:lastline)," ")).
\ ' > '.s:tmpfile)
echo system('echo >> '.s:tmpfile)
" pipe through mysql into temp file
echo system('echo '.shellescape(join(getline(a:firstline,a:lastline),"\n")).
\ '| mysql --batch --silent --raw &>> '.s:tmpfile)
" and open in new buffer
exec 'ed '.s:tmpfile
endfunction
" select query and <F5>
vmap <F5> :call Query()<cr>
Visually select query, press F5 (or :call Query()), and the result is opened in a new buffer.
It assumes linux (with mktemp) and gets connection details from .my.cnf
Try the dbext plugin.
But there are some others for this task.