Filtering pg_dump with only INSERT lines and changing table name using awk - mysql

I want to create a bash script that does a pg_dump of only INSERT lines and changes the table title in the INSERT line.
I have the following bash script:
#!bin/bash
#Create temp files to store the PSQL dump
DUMPFILE='poops.dump.sql' || (echo "make sql dump file failed" 1>&2; exit 1)
TMPFILE=`mktemp` || (echo "mktemp failed" 1>&2; exit 1)
#Tables to dump: api_order, poops_order_dates, poops_price
#Dump as INSERT queries statements
pg_dump --username="poops" --host="localhost" \
--table="api_order" --table="poops_order_dates" --table="poops_price" \
--no-password --column-inserts \
--data-only "poops" | \
awk '/^INSERT/ {i=1} {if(i) print}' \
> "$TMPFILE" \
|| (echo "pg_dump failed" 1>&2; exit 1)
(echo "start transaction; truncate table api_order; "; \
echo "truncate table poops_order_dates; "; \
echo "truncate table poops_price; "; \
cat "$TMPFILE"; echo 'commit;' ) \
> "$DUMPFILE" \
|| (echo "parsing dump file failed" 1>&2; exit 1)
rm "$TMPFILE"
It removes all the junk before the first INSERT INTO line, however there is still the following stuff after all the INSERT INTO queries:
--
-- Name: api_order_id_seq; Type: SEQUENCE SET; Schema: public; Owner: poops
--
SELECT pg_catalog.setval('api_order_id_seq', 33146, true);
--
-- Data for Name: poops_order_dates; Type: TABLE DATA; Schema: public; Owner: poops
--
--
-- Name: poops_order_dates_id_seq; Type: SEQUENCE SET; Schema: public; Owner: poops
--
SELECT pg_catalog.setval('poops_order_dates_id_seq', 1, false);
--
-- Data for Name: poops_price; Type: TABLE DATA; Schema: public; Owner: poops
--
--
-- Name: poops_price_id_seq; Type: SEQUENCE SET; Schema: public; Owner: poops
--
SELECT pg_catalog.setval('poops_price_id_seq', 1, false);
--
-- PostgreSQL database dump complete
--
How do I write the following awk line:
awk '/^INSERT/ {i=1} {if(i) print}' \
To make it so that it only outputs the INSERT INTO queries, and changes api_order to api_order_test.
I did try:
awk '/^INSERT/ {gsub("INSERT INTO api_order", "INSERT INTO api_order_test", $0); print $0}' \
But because of print $0, it cuts off part of the query where it starts on a new line. I need it to filter for, output and change only
"INSERT INTO api_order .... );\r"
I believe at the end it is a carriage return (\r) or does pg_dump output each INSERT INTO query with a \n at the end?
Raw SQL Dump:
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
--
-- Data for Name: api_order; Type: TABLE DATA; Schema: public; Owner: poops
--
INSERT INTO api_order (id, order type, …'', 0, NULL);
INSERT INTO api_order (id, order type, …'', 0, NULL);
INSERT INTO api_order (id, order type, …'', 0, NULL);
INSERT INTO api_order (id, order type, …'', 0, NULL);
INSERT INTO api_order (id, order type, …'', 0, NULL);
--
-- Name: api_order_id_seq; Type: SEQUENCE SET; Schema: public; Owner: poops
--
SELECT pg_catalog.setval('api_order_id_seq', 33294, true);
--
-- Data for Name: hoops_price; Type: TABLE DATA; Schema: public; Owner: poops
--
--
-- Name: hoops_price_id_seq; Type: SEQUENCE SET; Schema: public; Owner: poops
--
SELECT pg_catalog.setval('hoops_price_id_seq', 1, false);
--
-- PostgreSQL database dump complete
--

If everything you don't want to print comes after everything you do want to print, you can "turn off" printing in the same way you turn it on. Something like /^INSERT/ {i=1} /^SELECT/ {i=0} i. (Note that the trailing i is equivalent to your { if (i) print }) If you're input is more complicated or there are other considerations, please post a representative sample of the input to awk and your expected output.
– jas

Related

Sqlite3 - How to import NULL values from csv

I have dumped a mysql table as CSV. In this CSV file, the NULL values are written as \N
Now I want to import this data into sqlite database, but I am unable to tell sqlite that \N are null values. It is treating it as a string and that column value is stored as "\N" instead of NULL.
Can anyone guide in how to use .nullvalue dot command from sqlite. I am unable to set \N as nullvalue.
sqlite> .show
nullvalue: ""
sqlite> .nullvalue \N
sqlite> .show
nullvalue: "N"
sqlite> .nullvalue '\N'
sqlite> .show
nullvalue: "\\N"
sqlite> .nullvalue "\N"
sqlite> .show
nullvalue: "N"
sqlite> .nullvalue \\N
sqlite> .show
nullvalue: "\\N"
sqlite> .nullvalue '\'N
Usage: .nullvalue STRING
sqlite> .nullvalue '\\'N
Usage: .nullvalue STRING
sqlite> .nullvalue \\N
sqlite> .show
nullvalue: "\\N"
sqlite>
This is the output after every value of nullvalue
sqlite> .import /tmp/mysqlDump.csv employee
sqlite> select count(*) from employee where updatedon='\N';
94143
sqlite> select count(*) from employee where updatedon is null;
0
How can I tell sqlite to treat \N as NULL value? I cannot use empty string as NULL value as my data contains empty strings.
CSV files contain only text values. It is not possible to import NULL values from a CSV file.
To convert the \N values into NULLs, just use UPDATE afterwards:
UPDATE employee SET updatedon = NULL WHERE updatedon = '\N';
When sqlite is configured to check foreign key references, a method that relies on doing an update after the import is not possible, because the import would fail (the foreign key constraint would fail) and there would be no row to update.
For that, and the cases where doing an update after the import is unacceptable, you must modify the shell.c file in the amalgamation (as shown below), and compile a new sqlite(.exe) binary.
The change to be made is to bind the parameter for the field to NULL when the field is empty (with sqlite3_bind_null) instead of unconditionally binding it as a text field with sqlite3_bind_text as is currently done.
An example of making that change to sqlite version v 3.33.0 2020-08-14 is below (as a patch diff).
The example has the changes behind a new compile-time option, SQLITE_IMPORT_NULL_IF_EMPTY, so to enable it you'd need to define it when compiling like so:
cc -DSQLITE_IMPORT_NULL_IF_EMPTY <other options> shell.c sqlite3.c -o sqlite3
Example full compilation command with recommended options (and a few others set):
cc -Os -DSQLITE_IMPORT_NULL_IF_EMPTY -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_MEMSTATUS=0 -DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS -DSQLITE_MAX_EXPR_DEPTH=0 -DSQLITE_OMIT_DECLTYPE -DSQLITE_OMIT_DEPRECATED -DSQLITE_OMIT_PROGRESS_CALLBACK -DSQLITE_OMIT_SHARED_CACHE -DSQLITE_USE_ALLOCA -DSQLITE_OMIT_AUTOINIT -DSQLITE_DEFAULT_FOREIGN_KEYS=1 -DSQLITE_ENABLE_NULL_TRIM -DSQLITE_ENABLE_RBU -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DHAVE_USLEEP -DHAVE_READLINE shell.c sqlite3.c -lreadline -lncurses -o sqlite3
Patch file contents:
--- sqlite-amalgamation-3330000/shell.c 2020-08-14 13:42:48.000000000 +0000
+++ shell.c 2020-10-07 13:23:39.000000000 +0000
## -17845,7 +17845,12 ##
** the remaining columns.
*/
if( p->mode==MODE_Ascii && (z==0 || z[0]==0) && i==0 ) break;
+#ifdef SQLITE_IMPORT_NULL_IF_EMPTY
+ if (z==0 || z[0]=='\0') sqlite3_bind_null(pStmt, i+1);
+ else sqlite3_bind_text(pStmt, i+1, z, -1, SQLITE_TRANSIENT);
+#else
sqlite3_bind_text(pStmt, i+1, z, -1, SQLITE_TRANSIENT);
+#endif
if( i<nCol-1 && sCtx.cTerm!=sCtx.cColSep ){
utf8_printf(stderr, "%s:%d: expected %d columns but found %d - "
"filling the rest with NULL\n",

remove trailing comma in mysql2sqlite script using awk gsub

I found a script to turn a mysql dump into an sqlite insertable code,
and i am trying to get rid of trailing comma's before parentheses. (so the comma after 'DEFAULT NULL'. I use awk and gsub here and there. As you could guess I am not very familiar with regex.
link to script https://gist.github.com/esperlu/943776
CREATE TABLE "table_name" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"name" varchar(100) NOT NULL,
"created_at" datetime NOT NULL,
"deleted_at" datetime DEFAULT NULL,
);
At this point I use the following line in the script:
#Replace trailing commas
/\,\n\)/ { /\,\n\)/, "\, " }
But this gives me the following error in bash
cmd. line:52: /\,\n\)/ { /\,\n\)/, "\, " }
awk: cmd. line:52: ^ syntax error
awk: cmd. line:52: warning: escape sequence `\,' treated as plain `,'
/\,\n\)/ { /\,\n\)/, "\, " }
awk: cmd. line:52: ^ syntax error
awk: cmd. line:52: warning: escape sequence `\,' treated as plain `,'
sql exported to stub-testdb.sqlite
here is my full script into which I added the lines to remove trailing commas
#!/bin/sh
# Converts a mysqldump file into a Sqlite 3 compatible file. It also extracts the MySQL `KEY xxxxx` from the
# CREATE block and create them in separate commands _after_ all the INSERTs.
# Awk is choosen because it's fast and portable. You can use gawk, original awk or even the lightning fast mawk.
# The mysqldump file is traversed only once.
# Usage: $ ./mysql2sqlite mysqldump-opts db-name | sqlite3 database.sqlite
# Example: $ ./mysql2sqlite --no-data -u root -pMySecretPassWord myDbase | sqlite3 database.sqlite
# Thanks to and #artemyk and #gkuenning for their nice tweaks.
mysqldump --compatible=ansi --skip-extended-insert --compact "$#" | \
awk '
# Replace PRIMARY KEY if there is AUTO_INCREMENT
BEGIN { RS="/CREATE TABLE /" }
/NOT NULL AUTO_INCREMENT/ {
gsub( /\n PRIMARY KEY \(\"[a-z_]+\"\)/, "" )
print
next
}
' | \
awk '
BEGIN {
FS=",$"
print "PRAGMA synchronous = OFF;"
print "PRAGMA journal_mode = MEMORY;"
print "BEGIN TRANSACTION;"
}
# CREATE TRIGGER statements have funny commenting. Remember we are in trigger.
/^\/\*.*CREATE.*TRIGGER/ {
gsub( /^.*TRIGGER/, "CREATE TRIGGER" )
print
inTrigger = 1
next
}
# The end of CREATE TRIGGER has a stray comment terminator
/END \*\/;;/ { gsub( /\*\//, "" ); print; inTrigger = 0; next }
# The r est of triggers just get passed through
inTrigger != 0 { print; next }
# Skip other comments
/^\/\*/ { next }
# Print all `INSERT` lines. The single quotes are protected by another single quote.
/INSERT/ {
gsub( /\\\047/, "\047\047" )
gsub(/\\n/, "\n")
gsub(/\\r/, "\r")
gsub(/\\"/, "\"")
gsub(/\\\\/, "\\")
gsub(/\\\032/, "\032")
print
next
}
# Print the `CREATE` line as is and capture the table name.
/^CREATE/ {
print
if ( match( $0, /\"[^\"]+/ ) ) tableName = substr( $0, RSTART+1, RLENGTH-1 )
}
#Replace AUTO_INCREMENT with AUTOINCREMENT
/int\([0-9]+\) NOT NULL AUTO_INCREMENT/ { gsub( /int\([0-9]+\) NOT NULL AUTO_INCREMENT/, "INTEGER PRIMARY KEY AUTOINCREMENT" ) }
#Replace table and column COMMENT
/ COMMENT '.*'/ { gsub( / COMMENT '.*'/, "" ) }
#Replace trailing commas
/\,\n\)/ { /\,\n\)/, "\, " }
# Replace `FULLTEXT KEY` or any other `XXXXX KEY` except PRIMARY by `KEY`
/^ [^"]+KEY/ && !/^ PRIMARY KEY/ { gsub( /.+KEY/, " KEY" ) }
# Get rid of field lengths in KEY lines
/ KEY/ { gsub(/\([0-9]+\)/, "") }
# Print all fields definition lines except the `KEY` lines.
/^ / && !/^( KEY|\);)/ {
gsub( /AUTO_INCREMENT/, "" )
gsub( /(CHARACTER SET|character set) [^ ]+ /, "" )
gsub( /DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP|default current_timestamp on update current_timestamp/, "" )
gsub( /(COLLATE|collate) [^ ]+ /, "" )
gsub(/(ENUM|enum)[^)]+\)/, "text ")
gsub(/(SET|set)\([^)]+\)/, "text ")
gsub(/UNSIGNED|unsigned/, "")
if (prev) print prev ","
prev = $1
}
# `KEY` lines are extracted from the `CREATE` block and stored in array for later print
# in a separate `CREATE KEY` command. The index name is prefixed by the table name to
# avoid a sqlite error for duplicate index name.
/^( KEY|\);)/ {
if (prev) print prev
prev=""
if ($0 == ");"){
print
} else {
if ( match( $0, /\"[^"]+/ ) ) indexName = substr( $0, RSTART+1, RLENGTH-1 )
if ( match( $0, /\([^()]+/ ) ) indexKey = substr( $0, RSTART+1, RLENGTH-1 )
key[tableName]=key[tableName] "CREATE INDEX \"" tableName "_" indexName "\" ON \"" tableName "\" (" indexKey ");\n"
}
}
# Print all `KEY` creation lines.
END {
for (table in key) printf key[table]
print "END TRANSACTION;"
}
'
exit 0
Using GNU awk for multi-char RS and gensub():
$ gawk -v RS='^$' -v ORS= '{$0=gensub(/,(\s*\))/,"\\1","g")}1' file
CREATE TABLE "table_name" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"name" varchar(100) NOT NULL,
"created_at" datetime NOT NULL,
"deleted_at" datetime DEFAULT NULL
);

How to insert round brackets in mysql table using AWK script?

I have created a table with two integer columns and wanted to insert square of each number in the table. The output should be like this..
(1,1)
(2,4)
AWK script for creating database table...
BEGIN {
system("echo 'create table square (col1 INT, col2 INT);' | mysql -u root -D database")
}
I want to display the output with brackets and commas into two integer columns and want to write the AWK script to insert the data into the above table
This is my AWK script.
BEGIN {
for (i=1; i<=10; i++)
print "(" i "," i*i ")";
}
Maybe you forgot the "values" key before the bracket??
echo "line" > file.dat
awk 'BEGIN{
system("echo 'create table square (col1 INT, col2 INT);' | mysql -u root -D database")
for (i=1; i<=10; i++)
{
print " insert into square values (" i "," i*i ");"
}
}' file.dat > querys.sql
cat querys.sql
the insert into SQL method is suposed to have the values key http://www.w3schools.com/sql/sql_insert.asp
EDIT some improvements to make it in just one command:
> file.dat
> querys.sql
awk 'BEGIN{
system("echo \"create table square (col1 INT, col2 INT);\" | mysql -u root -D database")
for (i=1; i<=10; i++)
{
print " insert into square values (" i "," i*i ");" >> "querys.sql"
}
}
END{
system("mysql -u root -D database < querys.sql")
}' file.dat
Hope it helps

how to get value from a field in table

I have a field named 'parameter' in table with below format
utf8: "\xE2\x9C\x93"
id: "805265"
plan: initial
acc: "123456"
last: "1234"
doc: "1281468479"
validation: field
commit: Accept
how to query 'acc' value from the table?
database :sequelPro
> str = 'utf8..... your string here...'
> puts str
utf8: "\xE2\x9C\x93"
id: "805265"
plan: initial
acc: "123456"
last: "1234"
doc: "1281468479"
validation: field
commit: Accept
=> nil
> str.match(/^acc: "(\d+)"/).captures.first
=> "123456"
If I understood correctly, In PostgreSQL use split_part
See this EXAMPLE
create table star (param text);
insert into star values ('utf8: "\xE2\x9C\x93"'),
('id: "805265"'),
('plan: initial'),
('acc: "123456"'),
('last: "1234"'),
('doc: "1281468479"'),
('validation: field'),
('commit: Accept');
and use function split_part in SELECT query to get value of acc: like this
select col2
from (
select split_part(param, ' ', 1) col1,
split_part(param, ' ', 2) col2
from star
) t where col1='acc:'
Note: if you want to split your field by : then use select split_part(param, ':', 1) and split_part(param, ':', 2) col2 so the WHERE clause should be where col1='acc'
sqlfiddle-demo

Inserting data into the mysql database from perl

I am trying to insert data into a MySQL database:
$response = $client->fql->query(
query => '
SELECT name, email, birthday, username, first_name, last_name, pic
FROM user
WHERE uid = me()
',
);
print join "\n Name:", sort map { $_->{name} } #$response;
$dbh->do("
INSERT INTO Users(SNo,Name,Email,Birthday,UserName,FirstName,LastName)
VALUES(1,
sort map { $_->{name} } #$response,
'imm\#gmail.com',
'1987/12/10',
'imm',
'imm',
'Dee')
");
$dbh->disconnect();
used the mysql query in one line.
This above print statement is printing the name correctly but why the above sql insert statement is not working?
I connect the db and after that i am receiving the value and printing in the browser is working.
Why does the mysql statement not accept the value?
When inserting the database is not working?
You should have a look at the official doc
and specially this :
# INSERT some data into 'foo'. We are using $dbh->quote() for
# quoting the name.
$dbh->do("INSERT INTO foo VALUES (1, " . $dbh->quote("Tim") . ")");
# Same thing, but using placeholders
$dbh->do("INSERT INTO foo VALUES (?, ?)", undef, 2, "Jochen");