I’m trying to run/load sql file into mysql database using this golang statement but this is not working:
exec.Command("mysql", "-u", "{username}", "-p{db password}", "{db name}", "<", file abs path )
But when i use following command in windows command prompt it’s working perfect.
mysql -u {username} -p{db password} {db name} < {file abs path}
So what is the problem?
As others have answered, you can't use the < redirection operator because exec doesn't use the shell.
But you don't have to redirect input to read an SQL file. You can pass arguments to the MySQL client to use its source command.
exec.Command("mysql", "-u", "{username}", "-p{db password}", "{db name}",
"-e", "source {file abs path}" )
The source command is a builtin of the MySQL client. See https://dev.mysql.com/doc/refman/5.7/en/mysql-commands.html
Go's exec.Command runs the first argument as a program with the rest of the arguments as parameters. The '<' is interpreted as a literal argument.
e.g. exec.Command("cat", "<", "abc") is the following command in bash: cat \< abc.
To do what you want you have got two options.
Run (ba)sh and the command as argument: exec.Command("bash", "-c", "mysql ... < full/path")
Pipe the content of the file in manually. See https://stackoverflow.com/a/36383984/8751302 for details.
The problem with the bash version is that is not portable between different operating systems. It won't work on Windows.
Go's os.exec package does not use the shell and does not support redirection:
Unlike the "system" library call from C and other languages, the os/exec package intentionally does not invoke the system shell and does not expand any glob patterns or handle other expansions, pipelines, or redirections typically done by shells.
You can call the shell explicitly to pass arguments to it:
cmd := exec.Command("/bin/sh", yourBashCommand)
Depending on what you're doing, it may be helpful to write a short bash script and call it from Go.
Related
Consider the following program, running on a Linux machine, which opens a gzipped input file:
#!/usr/bin/env perl
open (my $fileHandle, "-|", "/bin/zcat $ARGV[0]");
my $ff = <$fileHandle>;
close($fileHandle);
That works as expected (it does nothing, but prints no error):
$ bar.pl file.gz
$
Now, if I use the same code but previously connect to a MySQL database, gzip will complain (you can run the code directly, this is an open DB and the credentials will work):
#!/usr/bin/env perl
use DBI;
use strict;
use warnings;
my $dsn = "DBI:mysql:database=hg19;host=genome-mysql.cse.ucsc.edu";
my $db = DBI->connect($dsn, 'genomep', 'password');
my $dbResults = $db->prepare("show tables");
my $ret = $dbResults->execute();
$dbResults->finish();
$db->disconnect();
open (my $fileHandle, "-|", "/bin/zcat $ARGV[0]");
my $ff = <$fileHandle>;
close($ff);
Running the above gives:
$ foo.pl file.gz
gzip: stdout: Broken pipe
This is obviously part of a much more complicated program, but I've managed to trim it down to this silly snippet that reproduces the issue.
What's going on? Why does connecting to a DB affect how gzip behaves? Note that everything seems to work (in the actual program, I do something useful with the gzipped data) but why am I getting that error message?
It turns out this behavior is specific to (slightly) older versions of Perl and/or DBI. On the machines where it failed, I have:
Ubuntu
Perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi
DBI 1.634
DBD 4.033
gzip 1.6
However, on another two machines it did work. These had:
Ubuntu
Perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux-gnu-thread-multi
DBI 1.640
DBD 4.033
gzip 1.6
And
Arch Linux
Perl 5, version 30, subversion 0 (v5.30.0) built for x86_64-linux-thread-multi
DBI 1.642
DBD 4.050
gzip 1.10
At least here, it appears that the MySQL libraries (probably) are masking (ignoring) SIGPIPE, and that's what you're seeing. Comparing strace outputs, I see a line like this in the MySQL run:
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f78bdf16840}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
And it turns out you can duplicate the behavior easily w/o MySQL:
$SIG{PIPE} = 'IGNORE';
open (my $fileHandle, "-|", "/bin/zcat $ARGV[0]");
my $ff = <$fileHandle>;
close($ff);
Or, alternatively, you can reset the signal to the default handler to make the message go away, even after connecting to MySQL by setting it to DEFAULT instead of IGNORE.
This is, by the way, documented behavior of the MySQL library:
To avoid aborting the program when a connection terminates, MySQL blocks SIGPIPE on the first call to mysql_library_init(), mysql_init(), or mysql_connect().
(It may also depend on the gzip version; maybe some versions of gzip set up signal handlers on init.)
Ultimately, what you're seeing is that if gzip gets a SIGPIPE, it just exits. If it gets an error back from write (because SIGPIPE is ignored), it prints an error message.
Most probably the following is happening:
gzip tries to write to the pipe, the program on your side is not reading up to eof, the closes the pipe. Gzip then receives a SIGPIPE, and dies with this error message. Can you confirm that this is taking place?
I have a cypher script file and I would like to run it directly.
All answers I could find on SO to the best of my knowledge use the command neo4j-shell which in my version (Neo4j server 3.5.5) seems to be deprecated and substituted with the command cyphershell.
Using the command sudo ./neo4j-community-3.5.5/bin/cypher-shell --help I got the following instructions.
usage: cypher-shell [-h] [-a ADDRESS] [-u USERNAME] [-p PASSWORD]
[--encryption {true,false}]
[--format {auto,verbose,plain}] [--debug] [--non-interactive] [--sample-rows SAMPLE-ROWS]
[--wrap {true,false}] [-v] [--driver-version] [--fail-fast | --fail-at-end] [cypher]
A command line shell where you can execute Cypher against an
instance of Neo4j. By default the shell is interactive but you can
use it for scripting by passing cypher directly on the command
line or by piping a file with cypher statements (requires Powershell
on Windows).
My file is the following which tries to create a graph from csv files and it comes from the book "Graph Algorithms".
WITH "https://github.com/neo4j-graph-analytics/book/raw/master/data" AS base
WITH base + "transport-nodes.csv" AS uri
LOAD CSV WITH HEADERS FROM uri AS row
MERGE (place:Place {id:row.id})
SET place.latitude = toFloat(row.latitude),
place.longitude = toFloat(row.latitude),
place.population = toInteger(row.population)
WITH "https://github.com/neo4j-graph-analytics/book/raw/master/data/" AS base
WITH base + "transport-relationships.csv" AS uri
LOAD CSV WITH HEADERS FROM uri AS row
MATCH (origin:Place {id: row.src})
MATCH (destination:Place {id: row.dst})
MERGE (origin)-[:EROAD {distance: toInteger(row.cost)}]->(destination)
When I try to pass the file directly with the command:
sudo ./neo4j-community-3.5.5/bin/cypher-shell neo_4.cypher
first it asks for username and password but after typing the correct password (the wrong password results in the error The client is unauthorized due to authentication failure.) I get the error:
Invalid input 'n': expected <init> (line 1, column 1 (offset: 0))
"neo_4.cypher"
^
When I try piping with the command:
sudo cat neo_4.cypher| sudo ./neo4j-community-3.5.5/bin/cypher-shell -u usr -p 'pwd'
no output is generated and no graph either.
How to run a cypher script file with the neo4j command cypher-shell?
Use cypher-shell -f yourscriptname. Check with --help for more description.
I think the key is here:
cypher-shell -- help
... Stuff deleted
positional arguments:
cypher an optional string of cypher to execute and then exit
This means that the paremeter is actual cypher code, not a file name. Thus, this works:
GMc#linux-ihon:~> cypher-shell "match(n) return n;"
username: neo4j
password: ****
+-----------------------------+
| n |
+-----------------------------+
| (:Job {jobName: "Job01"}) |
| (:Job {jobName: "Job02"}) |
But this doesn't (because the text "neo_4.cypher" isn't a valid cypher query)
cypher-shell neo_4.cypher
The help also says:
example of piping a file:
cat some-cypher.txt | cypher-shell
So:
cat neo_4.cypher | cypher-shell
should work. Possibly your problem is all of the sudo's. Specifically the cat ... | sudo cypher-shell. It is possible that sudo is protecting cypher-shell from some arbitrary input (although it doesn't seem to do so on my system).
If you really need to use sudo to run cypher, try using the following:
sudo cypher-shell arguments_as_needed < neo_4.cypher
Oh, also, your script doesn't have a return, so it probably won't display any data, but you should still see the summary reports of records loaded.
Perhaps try something simpler first such as a simple match ... return ... query in your script.
Oh, and don't forget to terminate the cypher query with a semi-colon!
The problem is in the cypher file: each line should end with a semicolon: ;. I still need sudo to run the program.
The file taken from the book seems to contain other errors as well actually.
We recently installed a server dedicated to unit tests, which deploys
updates automatically via Jenkins when commits are done, and sends
mails when a regression is noticed
> This requires our database to always be up-to-date
Since the database-schema-reference is our MWB, we added some scripts
during deploy, which export the .mwb to a .sql (using python) This
worked fine... but still has some issues
Our main concern is that the functions attached to the schema are not exported at all, which makes the DB unusable.
We'd like to hack into the python code to make it export scripts... but didn't find enough informations about it.
Here is the only piece of documentation we found. It's not very clear for us. We didn't find any information about exporting scripts.
All we found is that a db_Script class exists. We don't know where we can find its instances in our execution context, nor if they can be exported easily. Did we miss something ?
For reference, here is the script we currently use for the mwb to sql conversion (mwb2sql.sh).
It calls the MySqlWorkbench from command line (we use a dummy x-server to flush graphical output.)
What we need to complete is the python part passed in our command-line call of workbench.
# generate sql from mwb
# usage: sh mwb2sql.sh {mwb file} {output file}
# prepare: set env MYSQL_WORKBENCH
if [ "$MYSQL_WORKBENCH" = "" ]; then
export MYSQL_WORKBENCH="/usr/bin/mysql-workbench"
fi
export INPUT=$(cd $(dirname $1);pwd)/$(basename $1)
export OUTPUT=$(cd $(dirname $2);pwd)/$(basename $2)
"$MYSQL_WORKBENCH" \
--open $INPUT \
--run-python "
import os
import grt
from grt.modules import DbMySQLFE as fe
c = grt.root.wb.doc.physicalModels[0].catalog
fe.generateSQLCreateStatements(c, c.version, {})
fe.createScriptForCatalogObjects(os.getenv('OUTPUT'), c, {})" \
--quit-when-done
set -e
I know you use the spool command when you are trying to write a report to a file in Oracle SQLplus.
What is the equivalent command in MySQL?
This is my code:
set termout off
spool ${DB_ADMIN_HOME}/data/Datareport.log # ${DB_ADMIN_HOME}/Scripts.Datavalidation/Datareportscript.sql
spool off
exit
How can I write it in MySQL?
In MySQL you need to use the commands tee & notee:
tee data.txt;
//SQL sentences...
notee;
teedata.txt == spooldata.txt
notee == spool off
For the Oracle SQLPlus spool command, there is no equivalent in the mysql command line client.
To get output from the mysql command line client saved to a file, you can have the operating system redirect the output to a file, rather than to the display.
In Unix, you use the > symbol on the command line. (It seems a bit redundant here to give an example of how to redirect output.)
date > /tmp/foo.txt
That > symbol is basically telling the shell to take what is written to the STDOUT handle and redirect that to the named file (overwriting the file if it exists) if you have privileges.
Q: is set pagesize and set linesize used in mysql when you are trying to generate a report?
A: No. Those are specific to Oracle SQLPlus. I don't know of any equivalent functionality in the mysql command line client. The mysql command line client has some powerful features when its run in interactive mode (e.g. pager and tee), but in non-interactive mode, it's an inadequate replacement for SQLPlus.
If I get what you are asking:
mysql dbname < ${DB_ADMIN_HOME}/Scripts.Datavalidation/Datareportscript.sql \
> ${DB_ADMIN_HOME}/data/Datareport.log
Use redirection.
I'm trying to setup an mysql import script in Phing, but the "<" is causing XML errors. Any ideas?
<exec command="mysql -u${mysql.username} -p${mysql.password} -h ${mysql.server} ${mysql.database} < ${sql.file}" />
I'm looking at making dbDeploy doing this, but it'd be great if there was an easy way for this to work (single line command versus the multi-line setup of dbDeploy)
Escape the character by replacing it with <
I'm not familiar with Phing, but it may not be executing the command in a shell, so the redirect operator won't work (it may be treating the operator as an argument to mysql itself).
Instead of < ${sql.file}, try -e 'source ${sql.file}', so your whole line is:
<exec command="mysql -u${mysql.username} -p${mysql.password} -h ${mysql.server} ${mysql.database} -e 'source ${sql.file}'" />
See the MySQL batch mode documentation.