I made an experience following an issue I'm having, and discovered that somehow, SQLAlchemy/Flask/Flask-SQLAlchemy is keeping some kind of cache that is causing me troubles.
Here's what I did :
$> ./env/bin/flask shell
$> MyModel.query.filter(MyModel.id == 5).first() # ID 5 doesn't exists yet in the DB
$> # Returns None
$> # Create the entry with ID = 5
$> # Re-query the above request:
$> MyModel.query.filter(MyModel.id == 5).first()
$> # Still returns None !
The last query should return the entry in the database, but somehow, still return none. If I exit the shell, return in it and paste the line, it returns the proper value.
How can I query the database without using the cache?
Related
I need to suppress all error messages that the mysql command prints to stdout. I saw many other similar questions but all answers suggest workarounds to avoid these messages (check database/table/column exists before executing query). But I need the mysql command to return a failure exit code on error and don't print anything to output except the data explicitly requested in a query in a successful run. The -s key doesn't help in hiding error messages.
My task is to execute a MySQL query in a script and get either the requested data (printed with the -s key) or a non-zero exit code. I don't want to check each and every table/column/etc existence before executing a target query. How can I achieve this?
UPD: I tried this but it didn't help:
mysql ... db -s -N -e "SELECT config_id FROM core_config_data LIMIT 1;" 2> /dev/null
To sum it up:
You want the mysql command to:
not print any error
exit with a non-success code when a query fails
Then I've got good news for you! The output of errors will always be on stderr. Therefore you can just redirect the output to null or whatever you like.
root#icarus ~/so # mariadb -Dmaio290sql1 -e 'SELECT * FROM wp_users' -s 2> bla.txt
[actual content]
root#icarus ~/so # echo $?
0
root#icarus ~/so # mariadb -Dmaio290sql1 -e 'SELECT * FROM nope' -s 2> bla.txt
root#icarus ~/so # echo $?
1
The last query is throwing an error and therefore the exit code is not 0.
This was tested on MariaDB though: 10.3.27-MariaDB-0+deb10u1 Debian 10.
have an next case:
test3 = Mixlib::ShellOut.new("echo '1'")
test4 = Mixlib::ShellOut.new("mysql -u root --silent --skip-column-names --password='rootpass' -e 'some sql;'")
test3.run_command
test4.run_command
puts test3.stdout # => 1
puts test4.stdout # => empty string, nothing
expecting test4 to return result as test3
I guess maybe mysql returning result not to stdout, where then and how can I get the result
P.S I know about ruby gem mysql2 which I could use for doing this stuff, but to be honest no time to implement stuff like this just to getting only a simple SQL result, also I have tried run command without --silent and --skip-column-names flags, and this doesn't help too.
So the problem was, my command was returning code exit 1, it seems it was failing, so after fixing the command it works, the problem was in using single comma instead of double comma
I know there are many mysql-proxy questions on SO, however I have read through many of them and none seem to solve my problem. I am simply trying to get mysql-proxy up and running, with the eventual purpose of rewriting some queries that go through the proxy. I am using ubuntu 14.04, I have mysql-proxy version 0.8.1, and mysql version 5.5.37. To start mysql-proxy I run the following line on the command line
sudo mysql-proxy --defaults-file=mysql-proxy.cnf
where the file mysql-proxy.cnf looks like the following:
[mysql-proxy]
log-file= /var/log/mysql/proxy-error.log
log-level= debug
admin-lua-script= /usr/lib/mysql-proxy/lua/admin.lua
proxy-lua-script= /path/to/lua/script/example.lua
admin-username = myusername
admin-password = mypassword
proxy-skip-profiling = true
proxy-address = localhost:4040
proxy-backend-addresses = localhost:3306
plugins = proxy,admin
My example.lua script is very simple, and meant only to verify that the mysql-proxy query is being altered. example.lua is pasted below
-- first_example.lua
function read_query(packet)
if string.byte(packet) == proxy.COM_QUERY then
print("Hello world! Seen the query: " .. string.sub(packet, 2))
end
end
Since I don't run this with the --daemon flag, when I run that line above in the command line it just loops indefinitely, which is expected.
Finally, in separate terminal session, I run the following on the command line and enter my password in order to connect with the proxy
mysql -u myusername -p -h localhost -P 4040
I then select a database to use, and run a simple SELECT query on one of the tables. Based on multiple articles/tutorials I've read on mysql-proxy, my first console session, the one that ran mysql-proxy, should print out some data based on the example.lua file. However this does not happen, in fact nothing happens.
I'm not sure if the following bit of information makes any difference, but in my "my.cnf" mysql configuration file, I have these couple of lines
bind-address = 255.255.255.255
#bind-address = 127.0.0.1
where I have replaced my actual ip address with 255.255.255.255 because I do not want to display my ip address publicly.
Please, I have been trying to figure this out for several days, and no amount of new lua scripts, or changing the host:port parameters in the mysql-proxy.cnf file have solved anything. I
I'm using a MySQL client connection to issue FLUSH TABLES WITH READ LOCK and then create an LVM snapshot. I'd like the higher-level script (Ruby) to capture the return code of the snapshot creation, but cannot determine how to capture that within the MySQL client. To simplify:
mysql> system pwd
/root
I would expect that system to return 0.
mysql> system foo
sh: 1: foo: not found
I would expect that system to return 127 in bash, etc.
Is there some way to retrieve that error code within the MySQL client and return it when the MySQL client exits, write it to a file to read by the controlling script, etc.
To run FTWRL in a context where you're also running system commands, and they depend on each other, indicates you want to flush, then snapshot, then decide what to do based on that before releasing the table lock.
You'll need to do that in the same script to make sure they work together, and you're trying to do that in the SQL script. It's far easier to do it the other way around, by running the SQL script from inside the system script, because you're making decisions based on shell conditions rather than SQL conditions.
Here's how I do that (in Perl):
#!/usr/bin/perl -w
use DBI;
my $d = DBI->connect('dbi:mysql:mysql','root','***');
$d->do('FLUSH TABLES WITH READ LOCK') or die ("Can't flush tables in MySQL");
# Here's the important bit: Capture the status in $s and branch on the result
my $s = system('lvm lvcreate -s -n lv_datadirbackup -L 4G /dev/VG_MYSQL/lv_datadir');
if($s == 0) {
print ("Snapshot created.\n");
$d->do('UNLOCK TABLES') or die("Can't unlock tables");
... # etc
That way you can create the table lock take the snapshot, and use your script to control what happens next based on the snapshot status.
You can do this:
root#localhost> mysql
mysql> system bash
root#localhost> foo
foo: not found
root#localhost> echo $? > logfile
root#localhost> exit
mysql>
logfile now contains your return value.
If you want to automate it, you could eg. create these scripts:
script_bash.sh:
#!/bin/bash
foo
echo $? > logfile
script_mysql.sql:
system script_bash.sh
Then:
root#localhost> chmod +x script_bash.sh
root#localhost> mysql < script_mysql.sql #or mysql -e"script_bash.sh"
root#localhost> cat logfile
127
Whether you can do it does not mean you should do it :) I would definitely recommend firing the system command from the "controlling script".
I'm doing a bash script that interacts with a MySQL datatabase using the mysql command line programme. I want to use table locks in my SQL. Can I do this?
mysql -e "LOCK TABLES mytable"
# do some bash stuff
mysql -u "UNLOCK TABLES"
The reason I ask, is because table locks are only kept for the session, so wouldn't the lock be released as soon as that mysql programme finishes?
[EDIT]
nos had the basic idea -- only run "mysql" once, and the solution nos provided should work, but it left the FIFO on disk.
nos was also correct that I screwed up: a simple "echo X >FIFO" will close the FIFO; I remembered wrongly. And my (removed) comments w.r.t. timing don't apply, sorry.
That said, you don't need a FIFO, you could use an inter-process pipe. And looking through my old MySQL scripts, some worked akin to this, but you cannot let any commands write to stdout (without some "exec" tricks).
#!/bin/bash
(
echo "LOCK TABLES mytable READ ;"
echo "Doing something..." >&2
echo "describe mytable;"
sleep 5
echo "UNLOCK tables;"
) | mysql ${ARGUMENTS}
Another option might be to assign a file descriptor to the FIFO, then have it run in the background. This is very similar to what nos did, but the "exec" option wouldn't require a subshell to run the bash commands; hence would allow you to set "RC" in the "other stuff":
#!/bin/bash
# Use the PID ($$) in the FIFO and remove it on exit:
FIFO="/tmp/mysql-pipe.$$"
mkfifo ${FIFO} || exit $?
RC=0
# Tie FD3 to the FIFO (only for writing), then start MySQL in the u
# background with its input from the FIFO:
exec 3<>${FIFO}
mysql ${ARGUMENTS} <${FIFO} &
MYSQL=$!
trap "rm -f ${FIFO};kill -1 ${MYSQL} 2>&-" 0
# Now lock the table...
echo "LOCK TABLES mytable WRITE;" >&3
# ... do your other stuff here, set RC ...
echo "DESCRIBE mytable;" >&3
sleep 5
RC=3
# ...
echo "UNLOCK TABLES;" >&3
exec 3>&-
# You probably wish to sleep for a bit, or wait on ${MYSQL} before you exit
exit ${RC}
Note that there are a few control issues:
This code has NO ERROR CHECKING for failure to lock (or any SQL commands
within the "other stuff"). And that's definitely non-trivial.
Since in the first example, the "other stuff" is within a subshell, you cannot easily
set the return code of the script from that context.
Here's one way, I'm sure there's an easier way though..
mkfifo /tmp/mysql-pipe
mysql mydb </tmp/mysql-pipe &
(
echo "LOCK TABLES mytable READ ;" 1>&6
echo "Doing something "
echo "UNLOCK tables;" 1>&6
) 6> /tmp/mysql-pipe
A very interesting approach I found out while looking into this issue for my own, is by using MySQL's SYSTEM command. I'm not still sure what exactly are the drawbacks, if any, but it will certainly work for a lot of cases:
Example:
mysql <<END_HEREDOC
LOCK TABLES mytable;
SYSTEM /path/to/script.sh
UNLOCK TABLES;
END_HEREDOC
It's worth noting that this only works on *nix, obviously, as does the SYSTEM command.
Credit goes to Daniel Kadosh: http://dev.mysql.com/doc/refman/5.5/en/lock-tables.html#c10447
Another approach without the mkfifo commands:
cat <(echo "LOCK TABLES mytable;") <(sleep 3600) | mysql &
LOCK_PID=$!
# BASH STUFF
kill $LOCK_PID
I think Amr's answer is the simplest. However I wanted to share this because someone else may also need a slightly different answer.
The sleep 3600 pauses the input for 1 hour. You can find other commands to make it pause here: https://unix.stackexchange.com/questions/42901/how-to-do-nothing-forever-in-an-elegant-way
The lock tables SQL runs immediately, then it will wait for the sleep timer.
Problem and limitation in existing answers
Answers by NVRAM, nos and xer0x
If commands between LOCK TABLES and UNLOCK TABLES are all SQL queries, you should be fine.
In this case, however, why don't we just simply construct a single SQL file and pipe it to the mysql command?
If there are commands other than issuing SQL queries in the critical section, you could be running into trouble.
The echo command that sends the lock statement to the file descriptor doesn't block and wait for mysql to respond.
Subsequent commands are therefore possible to be executed before the lock is actually acquired. Synchronization aren't guaranteed.
Answer by Amr Mostafa
The SYSTEM command is executed on the MySQL server. So the script or command to be executed must be present on the same MySQL server.
You will need terminal access to the machine/VM/container that host the server (or at least a mean to transfer your script to the server host).
SYSTEM command also works on Windows as of MySQL 8.0.19, but running it on a Windows server of course means you will be running a Windows command (e.g. batch file or PowerShell script).
A modified solution
Below is a example solution based on the answers by NVRAM and nos, but waits for lock:
#!/bin/bash
# creates named pipes for attaching to stdin and stdout of mysql
mkfifo /tmp/mysql.stdin.pipe /tmp/mysql.stdout.pipe
# unbuffered option to ensure mysql doesn't buffer the output, so we can read immediately
# batch and skip-column-names options are for ease of parsing the output
mysql --unbuffered --batch --skip-column-names $OTHER_MYSQL_OPTIONS < /tmp/mysql.stdin.pipe > /tmp/mysql.stdout.pipe &
PID_MYSQL=$!
# make sure to stop mysql and remove the pipes before leaving
cleanup_proc_pipe() {
kill $PID_MYSQL
rm -rf /tmp/mysql.stdin.pipe /tmp/mysql.stdout.pipe
}
trap cleanup_proc_pipe EXIT
# open file descriptors for writing and reading
exec 10>/tmp/mysql.stdin.pipe
exec 11</tmp/mysql.stdout.pipe
# update the cleanup procedure to close the file descriptors
cleanup_fd() {
exec 10>&-
exec 11>&-
cleanup_proc_pipe
}
trap cleanup_fd EXIT
# try to obtain lock with 5 seconds of timeout
echo 'SELECT GET_LOCK("my_lock", 5);' >&10
# read stdout of mysql with 6 seconds of timeout
if ! read -t 6 line <&11; then
echo "Timeout reading from mysql"
elif [[ $line == 1 ]]; then
echo "Lock acquired successfully"
echo "Doing some critical stuff..."
echo 'DO RELEASE_LOCK("my_lock");' >&10
else
echo "Timeout waiting for lock"
fi
The above example uses SELECT GET_LOCK() to enter the critical section. It produces output for us to parse the result and decide what to do next.
If you need to execute statements that doesn't produce output (e.g. LOCK TABLES and START TRANSACTION), you may perform a dummy SELECT 1; after such statement and read from the stdout with a reasonable timeout. E.g.:
# ...
echo 'LOCK TABLES my_table WRITE;' >&10
echo 'SELECT 1;' >&10
if ! read -t 10 line <&11; then
echo "Timeout reading from mysql"
elif [[ $line == 1 ]]; then
echo "Table lock acquired"
# ...
else
echo "Unexpected output?!"
fi
You may also want to attach a third named pipe to stderr of mysql to handle different cases of error.