How do I cleanly extract MySQL enum values in Perl? - mysql

I have some code which needs to ensure some data is in a mysql enum prior to insertion in the database. The cleanest way I've found of doing this is the following code:
sub enum_values {
my ( $self, $schema, $table, $column ) = #_;
# don't eval to let the error bubble up
my $columns = $schema->storage->dbh->selectrow_hashref(
"SHOW COLUMNS FROM `$table` like ?",
{},
$column
);
unless ($columns) {
X::Internal::Database::UnknownColumn->throw(
column => $column,
table => $table,
);
}
my $type = $columns->{Type} or X::Panic->throw(
details => "Could not determine type for $table.$column",
);
unless ( $type =~ /\Aenum\((.*)\)\z/ ) {
X::Internal::Database::IncorrectTypeForColumn->throw(
type_wanted => 'enum',
type_found => $type,
);
}
$type = $1;
require Text::CSV_XS;
my $csv = Text::CSV_XS->new;
$csv->parse($type) or X::Panic->throw(
details => "Could not parse enum CSV data: ".$csv->error_input,
);
return map { /\A'(.*)'\z/; $1 }$csv->fields;
}
We're using DBIx::Class. Surely there is a better way of accomplishing this? (Note that the $table variable is coming from our code, not from any external source. Thus, no security issue).

No need to be so heroic. Using a reasonably modern version of DBD::mysql, the hash returned by DBI's column info method contains a pre-split version of the valid enum values in the key mysql_values:
my $sth = $dbh->column_info(undef, undef, 'mytable', '%');
foreach my $col_info ($sth->fetchrow_hashref)
{
if($col_info->{'TYPE_NAME'} eq 'ENUM')
{
# The mysql_values key contains a reference to an array of valid enum values
print "Valid enum values for $col_info->{'COLUMN_NAME'}: ",
join(', ', #{$col_info->{'mysql_values'}}), "\n";
}
...
}

I'd say using Text::CSV_XS may be an overkill, unless you have weird things like commas in enums (a bad idea anyway if you ask me). I'd probably use this instead.
my #fields = $type =~ / ' ([^']+) ' (?:,|\z) /msgx;
Other than that, I don't think there are shortcuts.

I spent part of the day asking the #dbix-class channel over on MagNet the same question and came across this lack of answer. Since I found the answer and nobody else seems to have done so yet, I'll paste the transcript below the TL;DR here:
my $cfg = new Config::Simple( $rc_file );
my $mysql = $cfg->get_block('mysql');
my $dsn =
"DBI:mysql:database=$mysql->{database};".
"host=$mysql->{hostname};port=$mysql->{port}";
my $schema =
DTSS::CDN::Schema->connect( $dsn, $mysql->{user}, $mysql->{password} );
my $valid_enum_values =
$schema->source('Cdnurl')->column_info('scheme')->{extra}->{list};
And now the IRC log of me beating my head against a wall:
14:40 < cj> is there a cross-platform way to get the valid values of an
enum?
15:11 < cj> it looks like I could add 'InflateColumn::Object::Enum' to the
__PACKAGE__->load_components(...) list for tables with enum
columns
15:12 < cj> and then call values() on the enum column
15:13 < cj> but how do I get dbic-dump to add
'InflateColumn::Object::Enum' to
__PACKAGE__->load_components(...) for only tables with enum
columns?
15:20 < cj> I guess I could just add it for all tables, since I'm doing
the same for InflateColumn::DateTime
15:39 < cj> hurm... is there a way to get a column without making a
request to the db?
15:40 < cj> I know that we store in the DTSS::CDN::Schema::Result::Cdnurl
class all of the information that I need to know about the
scheme column before any request is issued
15:42 <#ilmari> cj: for Pg and mysql Schema::Loader will add the list of
valid values to the ->{extra}->{list} column attribute
15:43 <#ilmari> cj: if you're using some other database that has enums,
patches welcome :)
15:43 <#ilmari> or even just a link to the documentation on how to extract
the values
15:43 <#ilmari> and a willingness to test if it's not a database I have
access to
15:43 < cj> thanks, but I'm using mysql. if I were using sqlite for this
project, I'd probably oblige :-)
15:44 <#ilmari> cj: to add components to only some tables, use
result_components_map
15:44 < cj> and is there a way to get at those attributes without making a
query?
15:45 < cj> can we do $schema->resultset('Cdnurl') without having it issue
a query, for instance?
15:45 <#ilmari> $result_source->column_info('colname')->{extra}->{list}
15:45 < cj> and $result_source is $schema->resultset('Cdnurl') ?
15:45 <#ilmari> dbic never issues a query until you start retrieving the
results
15:45 < cj> oh, nice.
15:46 <#ilmari> $schema->source('Cdnurl')
15:46 <#ilmari> the result source is where the result set gets the results
from when they are needed
15:47 <#ilmari> names have meanings :)

Related

decode_json and return first key in hash

JSON string input: https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=MSFT&apikey=demo
I am trying to return just the first key (current day) in the hash but have been unable to do so. My code looks like the following
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
use Data::Dumper;
use JSON;
my $html = get("https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=AMD&apikey=CMDPTEHVYH7W5VSZ");
my $decoded = decode_json($html);
my ($open) = $decoded->{'Time Series (Daily)'}->[0]->{'1. open'};
I keep getting "Not an ARRAY reference" which I researched and got more confused.
I can access what I want directly with the below code but I want to access just the first result or the current day:
my ($open) = $decoded->{'Time Series (Daily)'}{'2017-12-20'}{'1. open'};
Also if I do something like this:
my ($open) = $decoded->{'Time Series (Daily)'};
print Dumper($open);
The output is as follows:
$VAR1 = {
'2017-09-07' => {
'1. open' => '12.8400',
'5. volume' => '35467788',
'2. high' => '12.9400',
'4. close' => '12.6300',
'3. low' => '12.6000'
},
'2017-11-15' => {
'3. low' => '10.7700',
'4. close' => '11.0700',
'2. high' => '11.1300',
'5. volume' => '33326871',
'1. open' => '11.0100'
},
'2017-11-30' => {
'1. open' => '10.8700',
'2. high' => '11.0300',
'5. volume' => '43101899',
'3. low' => '10.7600',
'4. close' => '10.8900'
},
Thank you in advance for any help you can provide a noob.
Problem 1: { denotes the start of a JSON object, which gets decoded into a hash. Trying to derefence an array is going to fail.
Problem 2: Like Perl hashes, JSON objects are unordered, so talking about the
"first key" makes no sense. Perhaps you want the most recent date?
use List::Util qw( maxstr );
my $time_series_daily = $decoded->{'Time Series (Daily)'};
my $latest_date = maxstr #$time_series_daily;
my $open = $time_series_daily->{$latest_date}{'1. open'};
You are picking among hashref keys, not array (sequential container) elements. Since hashes are inherently unordered you can't index into that list but need to sort keys as needed.
With the exact format you show this works
my $top = (sort { $b cmp $a } keys %{ $decoded->{'Time Series (Daily)'} } )[0];
say $decoded->{'Time Series (Daily)'}{$top}{'1. open'};
It gets the list of keys, inverse-sorts them (alphabetically), and takes the first element of that list.
If your date-time format may vary then you'll need to parse it for sorting.
If you will really ever only want the most-recent one this is inefficient since it sorts the whole list. Then use a more specific tool to extract only the "largest" element, like
use List::Util qw(reduce);
my $top = reduce { $a gt $b ? $a : $b }
keys %{ $decoded->{'Time Series (Daily)'} };
But then in your case this can be done simply by maxstr from the same List::Util module, as shown in ikegami's answer. On the other hand, if the datetime format doesn't lend itself to a direct lexicographical comparison used by strmax then the reduce allows use of custom comparisons.

Powershell array value in MySQL insert is outputting entire array

First off I am new to Powershell, as in this is my first ever attempt. I'm doing the I can program in other languages so I can hack through this project. I have spent a few days now trying to solve this one, and I know is something stupid. I'm just stuck.
# Insert into master table
$query = "INSERT INTO ``master``(``diff``, ``key``) VALUES ('$diff', '$key')"
Invoke-MySqlQuery -Query $query
$query
This works fine the test output displays:
INSERT INTO `master`(`diff`, `key`) VALUES ('248', 'k000002143200000000000000680006080005500900030082670009461000500000091000000000000')
It also inputs into the MySQL DB just fine.
This following the exact same format is not working for me. Like I said at the top, I know it is some stupid formatting thing I'm missing.
# Insert into answers table
$query = "INSERT INTO ``puzzles``(``id``, ``p1``, ``p2``, ``p3``, ``p4``, ``p5``, ``p6``, ``p7``, ``p8``, ``p9``, ``p10``, ``p11``, ``p12``, ``p13``, ``p14``, ``p15``, ``p16``, ``p17``, ``p18``, ``p19``, ``p20``, ``p21``, ``p22``, ``p23``, ``p24``, ``p25``, ``p26``, ``p27``, ``p28``, ``p29``, ``p30``, ``p31``, ``p32``, ``p33``, ``p34``, ``p35``, ``p36``, ``p37``, ``p38``, ``p39``, ``p40``, ``p41``, ``p42``, ``p43``, ``p44``, ``p45``, ``p46``, ``p47``, ``p48``, ``p49``, ``p50``, ``p51``, ``p52``, ``p53``, ``p54``, ``p55``, ``p56``, ``p57``, ``p58``, ``p59``, ``p60``, ``p61``, ``p62``, ``p63``, ``p64``, ``p65``, ``p66``, ``p67``, ``p68``, ``p69``, ``p70``, ``p71``, ``p72``, ``p73``, ``p74``, ``p75``, ``p76``, ``p77``, ``p78``, ``p79``, ``p80``, ``p81``) VALUES ('$id', '$p[0]', '$p[1]', '$p[2]', '$p[3]', '$p[4]', '$p[5]', '$p[6]', '$p[7]', '$p[8]', '$p[9]', '$p[10]', '$p[11]', '$p[12]', '$p[13]', '$p[14]', '$p[15]', '$p[16]', '$p[17]', '$p[18]', '$p[19]', '$p[20]', '$p[21]', '$p[22]', '$p[23]', '$p[24]', '$p[25]', '$p[26]', '$p[27]', '$p[28]', '$p[29]', '$p[30]', '$p[31]', '$p[32]', '$p[33]', '$p[34]', '$p[35]', '$p[36]', '$p[37]', '$p[38]', '$p[39]', '$p[40]', '$p[41]', '$p[42]', '$p[43]', '$p[44]', '$p[45]', '$p[46]', '$p[47]', '$p[48]', '$p[49]', '$p[50]', '$p[51]', '$p[52]', '$p[53]', '$p[54]', '$p[55]', '$p[56]', '$p[57]', '$p[58]', '$p[59]', '$p[60]', '$p[61]', '$p[62]', '$p[63]', '$p[64]', '$p[65]', '$p[66]', '$p[67]', '$p[68]', '$p[69]', '$p[70]', '$p[71]', '$p[72]', '$p[73]', '$p[74]', '$p[75]', '$p[76]', '$p[77]', '$p[78]', '$p[79]', '$p[80]')"
$query
$p[0]
$p[1]
$p[2]
$p[3]
$p[4]
$p[5]
Invoke-MySqlQuery -Query $query
This is the output I get on one of the runs:
INSERT INTO `puzzles`(`id`, `p1`, `p2`, `p3`, `p4`, `p5`, `p6`, `p7`, `p8`, `p9`, `p10`, `p11`, `p12`, `p13`, `p14`, `p15`, `p16`, `p17`, `p
18`, `p19`, `p20`, `p21`, `p22`, `p23`, `p24`, `p25`, `p26`, `p27`, `p28`, `p29`, `p30`, `p31`, `p32`, `p33`, `p34`, `p35`, `p36`, `p37`, `p
38`, `p39`, `p40`, `p41`, `p42`, `p43`, `p44`, `p45`, `p46`, `p47`, `p48`, `p49`, `p50`, `p51`, `p52`, `p53`, `p54`, `p55`, `p56`, `p57`, `p
58`, `p59`, `p60`, `p61`, `p62`, `p63`, `p64`, `p65`, `p66`, `p67`, `p68`, `p69`, `p70`, `p71`, `p72`, `p73`, `p74`, `p75`, `p76`, `p77`, `p
78`, `p79`, `p80`, `p81`) VALUES ('2596', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[0]', '000002143
200000000000000680006080005500900030082670009461000500000091000000000000[1]', '0000021432000000000000006800060800055009000300826700094610005
00000091000000000000[2]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[3]', '0000021432000000000000006
80006080005500900030082670009461000500000091000000000000[4]', '00000214320000000000000068000608000550090003008267000946100050000009100000000
0000[5]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[6]', '00000214320000000000000068000608000550090
0030082670009461000500000091000000000000[7]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[8]', '00000
2143200000000000000680006080005500900030082670009461000500000091000000000000[9]', '000002143200000000000000680006080005500900030082670009461
000500000091000000000000[10]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[11]', '0000021432000000000
00000680006080005500900030082670009461000500000091000000000000[12]', '0000021432000000000000006800060800055009000300826700094610005000000910
00000000000[13]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[14]', '00000214320000000000000068000608
0005500900030082670009461000500000091000000000000[15]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[1
6]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[17]', '000002143200000000000000680006080005500900030
082670009461000500000091000000000000[18]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[19]', '0000021
43200000000000000680006080005500900030082670009461000500000091000000000000[20]', '0000021432000000000000006800060800055009000300826700094610
00500000091000000000000[21]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[22]', '00000214320000000000
0000680006080005500900030082670009461000500000091000000000000[23]', '00000214320000000000000068000608000550090003008267000946100050000009100
0000000000[24]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[25]', '000002143200000000000000680006080
005500900030082670009461000500000091000000000000[26]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[27
]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[28]', '0000021432000000000000006800060800055009000300
82670009461000500000091000000000000[29]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[30]', '00000214
3200000000000000680006080005500900030082670009461000500000091000000000000[31]', '00000214320000000000000068000608000550090003008267000946100
0500000091000000000000[32]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[33]', '000002143200000000000
000680006080005500900030082670009461000500000091000000000000[34]', '000002143200000000000000680006080005500900030082670009461000500000091000
000000000[35]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[36]', '0000021432000000000000006800060800
05500900030082670009461000500000091000000000000[37]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[38]
', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[39]', '00000214320000000000000068000608000550090003008
2670009461000500000091000000000000[40]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[41]', '000002143
200000000000000680006080005500900030082670009461000500000091000000000000[42]', '000002143200000000000000680006080005500900030082670009461000
500000091000000000000[43]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[44]', '0000021432000000000000
00680006080005500900030082670009461000500000091000000000000[45]', '0000021432000000000000006800060800055009000300826700094610005000000910000
00000000[46]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[47]', '00000214320000000000000068000608000
5500900030082670009461000500000091000000000000[48]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[49]'
, '000002143200000000000000680006080005500900030082670009461000500000091000000000000[50]', '000002143200000000000000680006080005500900030082
670009461000500000091000000000000[51]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[52]', '0000021432
00000000000000680006080005500900030082670009461000500000091000000000000[53]', '0000021432000000000000006800060800055009000300826700094610005
00000091000000000000[54]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[55]', '00000214320000000000000
0680006080005500900030082670009461000500000091000000000000[56]', '00000214320000000000000068000608000550090003008267000946100050000009100000
0000000[57]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[58]', '000002143200000000000000680006080005
500900030082670009461000500000091000000000000[59]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[60]',
'000002143200000000000000680006080005500900030082670009461000500000091000000000000[61]', '0000021432000000000000006800060800055009000300826
70009461000500000091000000000000[62]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[63]', '00000214320
0000000000000680006080005500900030082670009461000500000091000000000000[64]', '00000214320000000000000068000608000550090003008267000946100050
0000091000000000000[65]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[66]', '000002143200000000000000
680006080005500900030082670009461000500000091000000000000[67]', '000002143200000000000000680006080005500900030082670009461000500000091000000
000000[68]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[69]', '0000021432000000000000006800060800055
00900030082670009461000500000091000000000000[70]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[71]',
'000002143200000000000000680006080005500900030082670009461000500000091000000000000[72]', '00000214320000000000000068000608000550090003008267
0009461000500000091000000000000[73]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[74]', '000002143200
000000000000680006080005500900030082670009461000500000091000000000000[75]', '000002143200000000000000680006080005500900030082670009461000500
000091000000000000[76]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[77]', '0000021432000000000000006
80006080005500900030082670009461000500000091000000000000[78]', '0000021432000000000000006800060800055009000300826700094610005000000910000000
00000[79]', '000002143200000000000000680006080005500900030082670009461000500000091000000000000[80]')
0
0
0
0
0
2
In the $query variable the $p[X] does not work like the $p[X] below. It's spitting out the entire array. Obviously it's not working when it sends it to MySQL, I get the ID correct and that's it.
Thank you all for the help in advance, thumping my head on the keyboard hurts after awhile!
Ian
Bonus question! Any tips on this would be great as well. Any of the 0s don't need to be there, I actually want them as NULL. Any tips on a clean way to do that would be wonderful. Full disclosure I have not even looked into this part yet!
EDIT - In response to #Mathias R. Jessen
I removed all the other code not having to do with $p, to make it easy on the eyes.
# Set the key variable
$key = (Get-Content Puzzles.txt)[$loop] # ....73.146...8.......6......5..2..8.8..5.9..1.923..........273.........8..3..1.46 #Extreme
$key = $key.Replace(".", "0") # 000073014600080000000600000050020080800509001092300000000002730000000008003001046 #Extreme
$key = -join('k', $key.Substring(0,81)) # k000073014600080000000600000050020080800509001092300000000002730000000008003001046
$check = (Invoke-MySqlQuery -Query "SELECT COUNT(*) FROM ``master`` WHERE ``key`` = '$key'")
$check = $check.'COUNT(*)'
# Check in master table if key exists, i.e. not a new puzzle
IF ($check -eq 0) {
$p = $key.Substring(1,81) # 000073014600080000000600000050020080800509001092300000000002730000000008003001046
# Insert into answers table
$query = "INSERT INTO ``puzzles``(``id``, ``p1``, ``p2``, ``p3``, ``p4``, ``p5``, ``p6``, ``p7``, ``p8``, ``p9``, ``p10``, ``p11``, ``p12``, ``p13``, ``p14``, ``p15``, ``p16``, ``p17``, ``p18``, ``p19``, ``p20``, ``p21``, ``p22``, ``p23``, ``p24``, ``p25``, ``p26``, ``p27``, ``p28``, ``p29``, ``p30``, ``p31``, ``p32``, ``p33``, ``p34``, ``p35``, ``p36``, ``p37``, ``p38``, ``p39``, ``p40``, ``p41``, ``p42``, ``p43``, ``p44``, ``p45``, ``p46``, ``p47``, ``p48``, ``p49``, ``p50``, ``p51``, ``p52``, ``p53``, ``p54``, ``p55``, ``p56``, ``p57``, ``p58``, ``p59``, ``p60``, ``p61``, ``p62``, ``p63``, ``p64``, ``p65``, ``p66``, ``p67``, ``p68``, ``p69``, ``p70``, ``p71``, ``p72``, ``p73``, ``p74``, ``p75``, ``p76``, ``p77``, ``p78``, ``p79``, ``p80``, ``p81``) VALUES ('$id', '$p[0]', '$p[1]', '$p[2]', '$p[3]', '$p[4]', '$p[5]', '$p[6]', '$p[7]', '$p[8]', '$p[9]', '$p[10]', '$p[11]', '$p[12]', '$p[13]', '$p[14]', '$p[15]', '$p[16]', '$p[17]', '$p[18]', '$p[19]', '$p[20]', '$p[21]', '$p[22]', '$p[23]', '$p[24]', '$p[25]', '$p[26]', '$p[27]', '$p[28]', '$p[29]', '$p[30]', '$p[31]', '$p[32]', '$p[33]', '$p[34]', '$p[35]', '$p[36]', '$p[37]', '$p[38]', '$p[39]', '$p[40]', '$p[41]', '$p[42]', '$p[43]', '$p[44]', '$p[45]', '$p[46]', '$p[47]', '$p[48]', '$p[49]', '$p[50]', '$p[51]', '$p[52]', '$p[53]', '$p[54]', '$p[55]', '$p[56]', '$p[57]', '$p[58]', '$p[59]', '$p[60]', '$p[61]', '$p[62]', '$p[63]', '$p[64]', '$p[65]', '$p[66]', '$p[67]', '$p[68]', '$p[69]', '$p[70]', '$p[71]', '$p[72]', '$p[73]', '$p[74]', '$p[75]', '$p[76]', '$p[77]', '$p[78]', '$p[79]', '$p[80]')"
Invoke-MySqlQuery -Query $query
}
You need to use the subexpression operator $() to evaluate the collection at that index during string interpolation. Without this operator, the content of your entire collection is being printed, as well as your literal index syntax.
Your first example works as expected because you're only interpolating simple variables, without doing any additional work to them.
Here's a simple example from the command line:
C:\> $arr = 1,2,3,4
Outside of a string:
C:\> $arr[0]
1
During string interpolation, without the subexpression operator:
C:\> "$arr[0]"
1 2 3 4[0]
During string interpolation, with the subexpression operator:
C:\> "$($arr[0])"
1
This means that your example would become something like this:
...VALUES ('$id', '$($p[0])', '$($p[1])'...
Note that $id is working correctly because it is a simple variable. You only need to use the subexpression operator for additional work like evaluating indexes, properties, etc.
This concept is also sometimes called variable expansion, if you would like to research further.

Parse any number of subkeys in a perl hash

I have a perl hash that is obtained from parsing JSON. The JSON could be anything a user defined API could generated. The goal is to obtain a date/time string and determine if that date/time is out of bounds according to a user defined threshold. The only issue I have is that perl seems a bit cumbersome when dealing with hash key/subkey iteration. How can I look through all the keys and determine if a key or subkey exists throughout the hash? I have read many threads throughout stackoverflow, but nothing that exactly meets my needs. I only started perl last week so I may be missing something... Let me know if that's the case.
Below is the "relevant" code/subs. For all code see: https://gitlab.com/Jedimaster0/check_http_freshness
use warnings;
use strict;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use DateTime;
use DateTime::Format::Strptime;
# Verify the content-type of the response is JSON
eval {
assert_valid_json ($response->content);
};
if ( $# ){
print "[ERROR] Response isn't valid JSON. Please verify source data. \n$#";
exit EXIT_UNKNOWN;
} else {
# Convert the JSON data into a perl hashrefs
$jsonDecoded = parse_json($response->content);
if ($verbose){print "[SUCCESS] JSON FOUND -> ", $response->content , "\n";}
if (defined $jsonDecoded->{$opts{K}}){
if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $jsonDecoded->{$opts{K}}, "\n";}
NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $jsonDecoded->{$opts{K}})));
} else {
print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
exit EXIT_UNKNOWN;
}
}
sub DATETIME_LOOKUP {
my $dateFormat = $_[0];
my $dateFromJSON = $_[1];
my $strp = DateTime::Format::Strptime->new(
pattern => $dateFormat,
time_zone => $opts{z},
on_error => sub { print "[ERROR] INVALID TIME FORMAT: $dateFormat OR TIME ZONE: $opts{z} \n$_[1] \n" ; HELP_MESSAGE(); exit EXIT_UNKNOWN; },
);
my $dt = $strp->parse_datetime($dateFromJSON);
if (defined $dt){
if ($verbose){print "[SUCCESS] Time formatted using -> $dateFormat\n", "[SUCCESS] JSON date converted -> $dt $opts{z}\n";}
return $dt;
} else {
print "[ERROR] DATE VARIABLE IS NOT DEFINED. Pattern or timezone incorrect."; exit EXIT_UNKNOWN
}
}
# Subtract JSON date/time from now and return delta
sub DATETIME_DIFFERENCE {
my $dateInitial = $_[0];
my $deltaDate;
# Convert to UTC for standardization of computations and it's just easier to read when everything matches.
$dateInitial->set_time_zone('UTC');
$deltaDate = $dateNowUTC->delta_ms($dateInitial);
if ($verbose){print "[SUCCESS] (NOW) $dateNowUTC UTC - (JSON DATE) $dateInitial ", $dateInitial->time_zone->short_name_for_datetime($dateInitial), " = ", $deltaDate->in_units($opts{u}), " $opts{u} \n";}
return $deltaDate->in_units($opts{u});
}
Sample Data
{
"localDate":"Wednesday 23rd November 2016 11:03:37 PM",
"utcDate":"Wednesday 23rd November 2016 11:03:37 PM",
"format":"l jS F Y h:i:s A",
"returnType":"json",
"timestamp":1479942217,
"timezone":"UTC",
"daylightSavingTime":false,
"url":"http:\/\/www.convert-unix-time.com?t=1479942217",
"subkey":{
"altTimestamp":1479942217,
"altSubkey":{
"thirdTimestamp":1479942217
}
}
}
[SOLVED]
I have used the answer that #HåkonHægland provided. Here are the below code changes. Using the flatten module, I can use any input string that matches the JSON keys. I still have some work to do, but you can see the issue is resolved. Thanks #HåkonHægland.
use warnings;
use strict;
use Data::Dumper;
use LWP::UserAgent;
use Getopt::Std;
use JSON::Parse 'parse_json';
use JSON::Parse 'assert_valid_json';
use Hash::Flatten qw(:all);
use DateTime;
use DateTime::Format::Strptime;
# Verify the content-type of the response is JSON
eval {
assert_valid_json ($response->content);
};
if ( $# ){
print "[ERROR] Response isn't valid JSON. Please verify source data. \n$#";
exit EXIT_UNKNOWN;
} else {
# Convert the JSON data into a perl hashrefs
my $jsonDecoded = parse_json($response->content);
my $flatHash = flatten($jsonDecoded);
if ($verbose){print "[SUCCESS] JSON FOUND -> ", Dumper($flatHash), "\n";}
if (defined $flatHash->{$opts{K}}){
if ($verbose){print "[SUCCESS] JSON KEY FOUND -> ", $opts{K}, ": ", $flatHash>{$opts{K}}, "\n";}
NAGIOS_STATUS(DATETIME_DIFFERENCE(DATETIME_LOOKUP($opts{F}, $flatHash->{$opts{K}})));
} else {
print "[ERROR] Retreived JSON does not contain any data for the specified key: $opts{K}\n";
exit EXIT_UNKNOWN;
}
}
Example:
./check_http_freshness.pl -U http://bastion.mimir-tech.org/json.html -K result.creation_date -v
[SUCCESS] JSON FOUND -> $VAR1 = {
'timestamp' => '20161122T200649',
'result.data_version' => 'data_20161122T200649_data_news_topics',
'result.source_version' => 'kg_release_20160509_r33',
'result.seed_version' => 'seed_20161016',
'success' => 1,
'result.creation_date' => '20161122T200649',
'result.data_id' => 'data_news_topics',
'result.data_tgz_name' => 'data_news_topics_20161122T200649.tgz',
'result.source_data_version' => 'seed_vtv: data_20161016T102932_seed_vtv',
'result.data_digest' => '6b5bf1c2202d6f3983d62c275f689d51'
};
Odd number of elements in anonymous hash at ./check_http_freshness.pl line 78, <DATA> line 1.
[SUCCESS] JSON KEY FOUND -> result.creation_date:
[SUCCESS] Time formatted using -> %Y%m%dT%H%M%S
[SUCCESS] JSON date converted -> 2016-11-22T20:06:49 UTC
[SUCCESS] (NOW) 2016-11-26T19:02:15 UTC - (JSON DATE) 2016-11-22T20:06:49 UTC = 94 hours
[CRITICAL] Delta hours (94) is >= (24) hours. Data is stale.
You could try use Hash::Flatten. For example:
use Hash::Flatten qw(flatten);
my $json_decoded = parse_json($json_str);
my $flat = flatten( $json_decoded );
say "found" if grep /(?:^|\.)\Q$key\E(?:\.?|$)/, keys %$flat;
You can use Data::Visitor::Callback to traverse the data structure. It lets you define callbacks for different kinds of data types inside your structure. Since we're only looking at a hash it's relatively simple.
The following program has a predefined list of keys to find (those would be user input in your case). I converted your example JSON to a Perl hashref and included it in the code because the conversion is not relevant. The program visits every hashref in this data structure (including the top level) and runs the callback.
Callbacks in Perl are code references. These can be created in two ways. We're doing the anonymous subroutine (sometimes called lambda function in other languages). The callback gets passed two arguments: the visitor object and the current data substructure.
We'll iterate all the keys we want to find and simply check if they exist in that current data structure. If we see one, we count it's existence in the %seen hash. Using a hash to store things we have seen is a common idiom in Perl.
We're using a postfix if here, which is convenient and easy to read. %seen is a hash, so we access the value behind the $key with $seen{$key}, while $data is a hash reference, so we use the dereferencing operator -> to access the value behind $key with $data->{$key}.
The callback needs us to return the $data again so it continues. The last line is just there, it's not important.
I've used Data::Printer to output the %seen hash because it's convenient. You can also use Data::Dumper if you want. In production, you will not need that.
use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
my $from_json = {
"localDate" => "Wednesday 23rd November 2016 11:03:37 PM",
"utcDate" => "Wednesday 23rd November 2016 11:03:37 PM",
"format" => "l jS F Y h:i:s A",
"returnType" => "json",
"timestamp" => 1479942217,
"timezone" => "UTC",
"daylightSavingTime" =>
0, # this was false, I used 0 because that's a non-true value
"url" => "http:\/\/www.convert-unix-time.com?t=1479942217",
"subkey" => {
"altTimestamp" => 1479942217,
"altSubkey" => {
"thirdTimestamp" => 1479942217
}
}
};
my #keys_to_find = qw(timestamp altTimestamp thirdTimestamp missingTimestamp);
my %seen;
my $visitor = Data::Visitor::Callback->new(
hash => sub {
my ( $visitor, $data ) = #_;
foreach my $key (#keys_to_find) {
$seen{$key}++ if exists $data->{$key};
}
return $data;
},
);
$visitor->visit($from_json);
p %seen;
The program outputs the following. Note this is not a Perl data structure. Data::Printer is not a serializer, it's a tool to make data human readable in a convenient way.
{
altTimestamp 1,
thirdTimestamp 1,
timestamp 1
}
Since you also wanted to constraint the input, here's an example how to do that. The following program is a modification of the one above. It allows to give a set of different constraints for every required key.
I've done that by using a dispatch table. Essentially, that's a hash that contains code references. Kind of like the callbacks we use for the Visitor.
The constraints I've included are doing some things with dates. An easy way to work with dates in Perl is the core module Time::Piece. There are lots of questions around here about various date things where Time::Piece is the answer.
I've only done one constraint per key, but you could easily include several checks in those code refs, or make a list of code refs and put them in an array ref (keys => [ sub(), sub(), sub() ]) and then iterate that later.
In the visitor callback we are now also keeping track of the keys that have %passed the constraints check. We're calling the coderef with $coderef->($arg). If a constraint check returns a true value, it gets noted in the hash.
use strict;
use warnings;
use Data::Printer;
use Data::Visitor::Callback;
use Time::Piece;
use Time::Seconds; # for ONE_DAY
my $from_json = { ... }; # same as above
# prepare one of the constraints
# where I'm from, Christmas eve is considered Christmas
my $christmas = Time::Piece->strptime('24 Dec 2016', '%d %b %Y');
# set up the constraints per required key
my %constraints = (
timestamp => sub {
my ($epoch) = #_;
# not older than one day
return $epoch < time && $epoch > time - ONE_DAY;
},
altTimestamp => sub {
my ($epoch) = #_;
# epoch value should be an even number
return ! $epoch % 2;
},
thirdTimestamp => sub {
my ($epoch) = #_;
# before Christmas 2016
return $epoch < $christmas;
},
);
my %seen;
my %passed;
my $visitor = Data::Visitor::Callback->new(
hash => sub {
my ( $visitor, $data ) = #_;
foreach my $key (%constraints) {
if ( exists $data->{$key} ) {
$seen{$key}++;
$passed{$key}++ if $constraints{$key}->( $data->{$key} );
}
}
return $data;
},
);
$visitor->visit($from_json);
p %passed;
The output this time is:
{
thirdTimestamp 1,
timestamp 1
}
If you want to learn more about the dispatch tables, take a look at chapter two of the book Higher Order Perl by Mark Jason Dominus which is legally available for free here.

A shorter non-repeating alphanumeric code than UUID in MySQL

Is it possible for MySQL database to generate a 5 or 6 digit code comprised of only numbers and letters when I insert a record? If so how?
Just like goo.gl, bit.ly and jsfiddle do it. For exaple:
http://bit.ly/3PKQcJ
http://jsfiddle.net/XzKvP
cZ6ahF, 3t5mM, xGNPN, xswUdS...
So UUID_SHORT() will not work because it returns a value like 23043966240817183
Requirements:
Must be unique (non-repeating)
Can be but not required to be based off of primary key integer value
Must scale (grow by one character when all possible combinations have been used)
Must look random. (item 1234 cannot be BCDE while item 1235 be BCDF)
Must be generated on insert.
Would greatly appreciate code examples.
Try this:
SELECT LEFT(UUID(), 6);
I recommend using Redis for this task, actually. It has all the features that make this task suitable for its use. Foremost, it is very good at searching a big list for a value.
We will create two lists, buffered_ids, and used_ids. A cronjob will run every 5 minutes (or whatever interval you like), which will check the length of buffered_ids and keep it above, say, 5000 in length. When you need to use an id, pop it from buffered_ids and add it to used_ids.
Redis has sets, which are unique items in a collection. Think of it as a hash where the keys are unique and all the values are "true".
Your cronjob, in bash:
log(){ local x=$1 n=2 l=-1;if [ "$2" != "" ];then n=$x;x=$2;fi;while((x));do let l+=1 x/=n;done;echo $l; }
scale=`redis-cli SCARD used_ids`
scale=`log 16 $scale`
scale=$[ scale + 6]
while [ `redis-cli SCARD buffered_ids` -lt 5000 ]; do
uuid=`cat /dev/urandom | tr -cd "[:alnum:]" | head -c ${1:-$scale}`
if [ `redis-cli SISMEMBER used_ids $uuid` == 1]; then
continue
fi
redis-cli SADD buffered_ids $uuid
done
To grab the next uid for use in your application (in pseudocode because you did not specify a language)
$uid = redis('SPOP buffered_ids');
redis('SADD used_ids ' . $uid);
edit actually there's a race condition there. To safely pop a value, add it to used_ids first, then remove it from buffered_ids.
$uid = redis('SRANDMEMBER buffered_ids');
redis('SADD used_ids ' . $uid);
redis('SREM buffered_ids ' . $uid);

Quick easy way to migrate SQLite3 to MySQL? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
Anyone know a quick easy way to migrate a SQLite3 database to MySQL?
Everyone seems to starts off with a few greps and perl expressions and you sorta kinda get something that works for your particular dataset but you have no idea if it's imported the data correctly or not. I'm seriously surprised nobody's built a solid library that can convert between the two.
Here a list of ALL the differences in SQL syntax that I know about between the two file formats:
The lines starting with:
BEGIN TRANSACTION
COMMIT
sqlite_sequence
CREATE UNIQUE INDEX
are not used in MySQL
SQLite uses CREATE TABLE/INSERT INTO "table_name" and MySQL uses CREATE TABLE/INSERT INTO table_name
MySQL doesn't use quotes inside the schema definition
MySQL uses single quotes for strings inside the INSERT INTO clauses
SQLite and MySQL have different ways of escaping strings inside INSERT INTO clauses
SQLite uses 't' and 'f' for booleans, MySQL uses 1 and 0 (a simple regex for this can fail when you have a string like: 'I do, you don't' inside your INSERT INTO)
SQLLite uses AUTOINCREMENT, MySQL uses AUTO_INCREMENT
Here is a very basic hacked up perl script which works for my dataset and checks for many more of these conditions that other perl scripts I found on the web. Nu guarantees that it will work for your data but feel free to modify and post back here.
#! /usr/bin/perl
while ($line = <>){
if (($line !~ /BEGIN TRANSACTION/) && ($line !~ /COMMIT/) && ($line !~ /sqlite_sequence/) && ($line !~ /CREATE UNIQUE INDEX/)){
if ($line =~ /CREATE TABLE \"([a-z_]*)\"(.*)/i){
$name = $1;
$sub = $2;
$sub =~ s/\"//g;
$line = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";
}
elsif ($line =~ /INSERT INTO \"([a-z_]*)\"(.*)/i){
$line = "INSERT INTO $1$2\n";
$line =~ s/\"/\\\"/g;
$line =~ s/\"/\'/g;
}else{
$line =~ s/\'\'/\\\'/g;
}
$line =~ s/([^\\'])\'t\'(.)/$1THIS_IS_TRUE$2/g;
$line =~ s/THIS_IS_TRUE/1/g;
$line =~ s/([^\\'])\'f\'(.)/$1THIS_IS_FALSE$2/g;
$line =~ s/THIS_IS_FALSE/0/g;
$line =~ s/AUTOINCREMENT/AUTO_INCREMENT/g;
print $line;
}
}
Here is a list of converters (not updated since 2011):
https://www2.sqlite.org/cvstrac/wiki?p=ConverterTools (or snapshot at archive.org)
An alternative method that would work nicely but is rarely mentioned is: use an ORM class that abstracts specific database differences away for you. e.g. you get these in PHP (RedBean), Python (Django's ORM layer, Storm, SqlAlchemy), Ruby on Rails (ActiveRecord), Cocoa (CoreData)
i.e. you could do this:
Load data from source database using the ORM class.
Store data in memory or serialize to disk.
Store data into destination database using the ORM class.
Here is a python script, built off of Shalmanese's answer and some help from Alex martelli over at Translating Perl to Python
I'm making it community wiki, so please feel free to edit, and refactor as long as it doesn't break the functionality (thankfully we can just roll back) - It's pretty ugly but works
use like so (assuming the script is called dump_for_mysql.py:
sqlite3 sample.db .dump | python dump_for_mysql.py > dump.sql
Which you can then import into mysql
note - you need to add foreign key constrains manually since sqlite doesn't actually support them
here is the script:
#!/usr/bin/env python
import re
import fileinput
def this_line_is_useless(line):
useless_es = [
'BEGIN TRANSACTION',
'COMMIT',
'sqlite_sequence',
'CREATE UNIQUE INDEX',
'PRAGMA foreign_keys=OFF',
]
for useless in useless_es:
if re.search(useless, line):
return True
def has_primary_key(line):
return bool(re.search(r'PRIMARY KEY', line))
searching_for_end = False
for line in fileinput.input():
if this_line_is_useless(line):
continue
# this line was necessary because '');
# would be converted to \'); which isn't appropriate
if re.match(r".*, ''\);", line):
line = re.sub(r"''\);", r'``);', line)
if re.match(r'^CREATE TABLE.*', line):
searching_for_end = True
m = re.search('CREATE TABLE "?(\w*)"?(.*)', line)
if m:
name, sub = m.groups()
line = "DROP TABLE IF EXISTS %(name)s;\nCREATE TABLE IF NOT EXISTS `%(name)s`%(sub)s\n"
line = line % dict(name=name, sub=sub)
else:
m = re.search('INSERT INTO "(\w*)"(.*)', line)
if m:
line = 'INSERT INTO %s%s\n' % m.groups()
line = line.replace('"', r'\"')
line = line.replace('"', "'")
line = re.sub(r"([^'])'t'(.)", "\1THIS_IS_TRUE\2", line)
line = line.replace('THIS_IS_TRUE', '1')
line = re.sub(r"([^'])'f'(.)", "\1THIS_IS_FALSE\2", line)
line = line.replace('THIS_IS_FALSE', '0')
# Add auto_increment if it is not there since sqlite auto_increments ALL
# primary keys
if searching_for_end:
if re.search(r"integer(?:\s+\w+)*\s*PRIMARY KEY(?:\s+\w+)*\s*,", line):
line = line.replace("PRIMARY KEY", "PRIMARY KEY AUTO_INCREMENT")
# replace " and ' with ` because mysql doesn't like quotes in CREATE commands
if line.find('DEFAULT') == -1:
line = line.replace(r'"', r'`').replace(r"'", r'`')
else:
parts = line.split('DEFAULT')
parts[0] = parts[0].replace(r'"', r'`').replace(r"'", r'`')
line = 'DEFAULT'.join(parts)
# And now we convert it back (see above)
if re.match(r".*, ``\);", line):
line = re.sub(r'``\);', r"'');", line)
if searching_for_end and re.match(r'.*\);', line):
searching_for_end = False
if re.match(r"CREATE INDEX", line):
line = re.sub('"', '`', line)
if re.match(r"AUTOINCREMENT", line):
line = re.sub("AUTOINCREMENT", "AUTO_INCREMENT", line)
print line,
I usually use the Export/import tables feature of IntelliJ DataGrip.
You can see the progress in the bottom right corner.
[]
If you are using Python/Django it's pretty easy:
create two databases in settings.py (like here https://docs.djangoproject.com/en/1.11/topics/db/multi-db/)
then just do like this:
objlist = ModelObject.objects.using('sqlite').all()
for obj in objlist:
obj.save(using='mysql')
Probably the quick easiest way is using the sqlite .dump command, in this case create a dump of the sample database.
sqlite3 sample.db .dump > dump.sql
You can then (in theory) import this into the mysql database, in this case the test database on the database server 127.0.0.1, using user root.
mysql -p -u root -h 127.0.0.1 test < dump.sql
I say in theory as there are a few differences between grammars.
In sqlite transactions begin
BEGIN TRANSACTION;
...
COMMIT;
MySQL uses just
BEGIN;
...
COMMIT;
There are other similar problems (varchars and double quotes spring back to mind) but nothing find and replace couldn't fix.
Perhaps you should ask why you are migrating, if performance/ database size is the issue perhaps look at reoginising the schema, if the system is moving to a more powerful product this might be the ideal time to plan for the future of your data.
I've just gone through this process, and there's a lot of very good help and information in this Q/A, but I found I had to pull together various elements (plus some from other Q/As) to get a working solution in order to successfully migrate.
However, even after combining the existing answers, I found that the Python script did not fully work for me as it did not work where there were multiple boolean occurrences in an INSERT. See here why that was the case.
So, I thought I'd post up my merged answer here. Credit goes to those that have contributed elsewhere, of course. But I wanted to give something back, and save others time that follow.
I'll post the script below. But firstly, here's the instructions for a conversion...
I ran the script on OS X 10.7.5 Lion. Python worked out of the box.
To generate the MySQL input file from your existing SQLite3 database, run the script on your own files as follows,
Snips$ sqlite3 original_database.sqlite3 .dump | python ~/scripts/dump_for_mysql.py > dumped_data.sql
I then copied the resulting dumped_sql.sql file over to a Linux box running Ubuntu 10.04.4 LTS where my MySQL database was to reside.
Another issue I had when importing the MySQL file was that some unicode UTF-8 characters (specifically single quotes) were not being imported correctly, so I had to add a switch to the command to specify UTF-8.
The resulting command to input the data into a spanking new empty MySQL database is as follows:
Snips$ mysql -p -u root -h 127.0.0.1 test_import --default-character-set=utf8 < dumped_data.sql
Let it cook, and that should be it! Don't forget to scrutinise your data, before and after.
So, as the OP requested, it's quick and easy, when you know how! :-)
As an aside, one thing I wasn't sure about before I looked into this migration, was whether created_at and updated_at field values would be preserved - the good news for me is that they are, so I could migrate my existing production data.
Good luck!
UPDATE
Since making this switch, I've noticed a problem that I hadn't noticed before. In my Rails application, my text fields are defined as 'string', and this carries through to the database schema. The process outlined here results in these being defined as VARCHAR(255) in the MySQL database. This places a 255 character limit on these field sizes - and anything beyond this was silently truncated during the import. To support text length greater than 255, the MySQL schema would need to use 'TEXT' rather than VARCHAR(255), I believe. The process defined here does not include this conversion.
Here's the merged and revised Python script that worked for my data:
#!/usr/bin/env python
import re
import fileinput
def this_line_is_useless(line):
useless_es = [
'BEGIN TRANSACTION',
'COMMIT',
'sqlite_sequence',
'CREATE UNIQUE INDEX',
'PRAGMA foreign_keys=OFF'
]
for useless in useless_es:
if re.search(useless, line):
return True
def has_primary_key(line):
return bool(re.search(r'PRIMARY KEY', line))
searching_for_end = False
for line in fileinput.input():
if this_line_is_useless(line): continue
# this line was necessary because ''); was getting
# converted (inappropriately) to \');
if re.match(r".*, ''\);", line):
line = re.sub(r"''\);", r'``);', line)
if re.match(r'^CREATE TABLE.*', line):
searching_for_end = True
m = re.search('CREATE TABLE "?([A-Za-z_]*)"?(.*)', line)
if m:
name, sub = m.groups()
line = "DROP TABLE IF EXISTS %(name)s;\nCREATE TABLE IF NOT EXISTS `%(name)s`%(sub)s\n"
line = line % dict(name=name, sub=sub)
line = line.replace('AUTOINCREMENT','AUTO_INCREMENT')
line = line.replace('UNIQUE','')
line = line.replace('"','')
else:
m = re.search('INSERT INTO "([A-Za-z_]*)"(.*)', line)
if m:
line = 'INSERT INTO %s%s\n' % m.groups()
line = line.replace('"', r'\"')
line = line.replace('"', "'")
line = re.sub(r"(?<!')'t'(?=.)", r"1", line)
line = re.sub(r"(?<!')'f'(?=.)", r"0", line)
# Add auto_increment if it's not there since sqlite auto_increments ALL
# primary keys
if searching_for_end:
if re.search(r"integer(?:\s+\w+)*\s*PRIMARY KEY(?:\s+\w+)*\s*,", line):
line = line.replace("PRIMARY KEY", "PRIMARY KEY AUTO_INCREMENT")
# replace " and ' with ` because mysql doesn't like quotes in CREATE commands
# And now we convert it back (see above)
if re.match(r".*, ``\);", line):
line = re.sub(r'``\);', r"'');", line)
if searching_for_end and re.match(r'.*\);', line):
searching_for_end = False
if re.match(r"CREATE INDEX", line):
line = re.sub('"', '`', line)
print line,
http://sqlfairy.sourceforge.net/
http://search.cpan.org/dist/SQL-Translator/
aptitude install sqlfairy libdbd-sqlite3-perl
sqlt -f DBI --dsn dbi:SQLite:../.open-tran/ten-sq.db -t MySQL --add-drop-table > mysql-ten-sq.sql
sqlt -f DBI --dsn dbi:SQLite:../.open-tran/ten-sq.db -t Dumper --use-same-auth > sqlite2mysql-dumper.pl
chmod +x sqlite2mysql-dumper.pl
./sqlite2mysql-dumper.pl --help
./sqlite2mysql-dumper.pl --add-truncate --mysql-loadfile > mysql-dump.sql
sed -e 's/LOAD DATA INFILE/LOAD DATA LOCAL INFILE/' -i mysql-dump.sql
echo 'drop database `ten-sq`' | mysql -p -u root
echo 'create database `ten-sq` charset utf8' | mysql -p -u root
mysql -p -u root -D ten-sq < mysql-ten-sq.sql
mysql -p -u root -D ten-sq < mysql-dump.sql
I wrote this simple script in Python3. It can be used as an included class or standalone script invoked via a terminal shell. By default it imports all integers as int(11)and strings as varchar(300), but all that can be adjusted in the constructor or script arguments respectively.
NOTE: It requires MySQL Connector/Python 2.0.4 or higher
Here's a link to the source on GitHub if you find the code below hard to read: https://github.com/techouse/sqlite3-to-mysql
#!/usr/bin/env python3
__author__ = "Klemen Tušar"
__email__ = "techouse#gmail.com"
__copyright__ = "GPL"
__version__ = "1.0.1"
__date__ = "2015-09-12"
__status__ = "Production"
import os.path, sqlite3, mysql.connector
from mysql.connector import errorcode
class SQLite3toMySQL:
"""
Use this class to transfer an SQLite 3 database to MySQL.
NOTE: Requires MySQL Connector/Python 2.0.4 or higher (https://dev.mysql.com/downloads/connector/python/)
"""
def __init__(self, **kwargs):
self._properties = kwargs
self._sqlite_file = self._properties.get('sqlite_file', None)
if not os.path.isfile(self._sqlite_file):
print('SQLite file does not exist!')
exit(1)
self._mysql_user = self._properties.get('mysql_user', None)
if self._mysql_user is None:
print('Please provide a MySQL user!')
exit(1)
self._mysql_password = self._properties.get('mysql_password', None)
if self._mysql_password is None:
print('Please provide a MySQL password')
exit(1)
self._mysql_database = self._properties.get('mysql_database', 'transfer')
self._mysql_host = self._properties.get('mysql_host', 'localhost')
self._mysql_integer_type = self._properties.get('mysql_integer_type', 'int(11)')
self._mysql_string_type = self._properties.get('mysql_string_type', 'varchar(300)')
self._sqlite = sqlite3.connect(self._sqlite_file)
self._sqlite.row_factory = sqlite3.Row
self._sqlite_cur = self._sqlite.cursor()
self._mysql = mysql.connector.connect(
user=self._mysql_user,
password=self._mysql_password,
host=self._mysql_host
)
self._mysql_cur = self._mysql.cursor(prepared=True)
try:
self._mysql.database = self._mysql_database
except mysql.connector.Error as err:
if err.errno == errorcode.ER_BAD_DB_ERROR:
self._create_database()
else:
print(err)
exit(1)
def _create_database(self):
try:
self._mysql_cur.execute("CREATE DATABASE IF NOT EXISTS `{}` DEFAULT CHARACTER SET 'utf8'".format(self._mysql_database))
self._mysql_cur.close()
self._mysql.commit()
self._mysql.database = self._mysql_database
self._mysql_cur = self._mysql.cursor(prepared=True)
except mysql.connector.Error as err:
print('_create_database failed creating databse {}: {}'.format(self._mysql_database, err))
exit(1)
def _create_table(self, table_name):
primary_key = ''
sql = 'CREATE TABLE IF NOT EXISTS `{}` ( '.format(table_name)
self._sqlite_cur.execute('PRAGMA table_info("{}")'.format(table_name))
for row in self._sqlite_cur.fetchall():
column = dict(row)
sql += ' `{name}` {type} {notnull} {auto_increment}, '.format(
name=column['name'],
type=self._mysql_string_type if column['type'].upper() == 'TEXT' else self._mysql_integer_type,
notnull='NOT NULL' if column['notnull'] else 'NULL',
auto_increment='AUTO_INCREMENT' if column['pk'] else ''
)
if column['pk']:
primary_key = column['name']
sql += ' PRIMARY KEY (`{}`) ) ENGINE = InnoDB CHARACTER SET utf8'.format(primary_key)
try:
self._mysql_cur.execute(sql)
self._mysql.commit()
except mysql.connector.Error as err:
print('_create_table failed creating table {}: {}'.format(table_name, err))
exit(1)
def transfer(self):
self._sqlite_cur.execute("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'")
for row in self._sqlite_cur.fetchall():
table = dict(row)
# create the table
self._create_table(table['name'])
# populate it
print('Transferring table {}'.format(table['name']))
self._sqlite_cur.execute('SELECT * FROM "{}"'.format(table['name']))
columns = [column[0] for column in self._sqlite_cur.description]
try:
self._mysql_cur.executemany("INSERT IGNORE INTO `{table}` ({fields}) VALUES ({placeholders})".format(
table=table['name'],
fields=('`{}`, ' * len(columns)).rstrip(' ,').format(*columns),
placeholders=('%s, ' * len(columns)).rstrip(' ,')
), (tuple(data) for data in self._sqlite_cur.fetchall()))
self._mysql.commit()
except mysql.connector.Error as err:
print('_insert_table_data failed inserting data into table {}: {}'.format(table['name'], err))
exit(1)
print('Done!')
def main():
""" For use in standalone terminal form """
import sys, argparse
parser = argparse.ArgumentParser()
parser.add_argument('--sqlite-file', dest='sqlite_file', default=None, help='SQLite3 db file')
parser.add_argument('--mysql-user', dest='mysql_user', default=None, help='MySQL user')
parser.add_argument('--mysql-password', dest='mysql_password', default=None, help='MySQL password')
parser.add_argument('--mysql-database', dest='mysql_database', default=None, help='MySQL host')
parser.add_argument('--mysql-host', dest='mysql_host', default='localhost', help='MySQL host')
parser.add_argument('--mysql-integer-type', dest='mysql_integer_type', default='int(11)', help='MySQL default integer field type')
parser.add_argument('--mysql-string-type', dest='mysql_string_type', default='varchar(300)', help='MySQL default string field type')
args = parser.parse_args()
if len(sys.argv) == 1:
parser.print_help()
exit(1)
converter = SQLite3toMySQL(
sqlite_file=args.sqlite_file,
mysql_user=args.mysql_user,
mysql_password=args.mysql_password,
mysql_database=args.mysql_database,
mysql_host=args.mysql_host,
mysql_integer_type=args.mysql_integer_type,
mysql_string_type=args.mysql_string_type
)
converter.transfer()
if __name__ == '__main__':
main()
I recently had to migrate from MySQL to JavaDB for a project that our team is working on. I found a Java library written by Apache called DdlUtils that made this pretty easy. It provides an API that lets you do the following:
Discover a database's schema and export it as an XML file.
Modify a DB based upon this schema.
Import records from one DB to another, assuming they have the same schema.
The tools that we ended up with weren't completely automated, but they worked pretty well. Even if your application is not in Java, it shouldn't be too difficult to whip up a few small tools to do a one-time migration. I think I was able to pull of our migration with less than 150 lines of code.
Get a SQL dump
moose#pc08$ sqlite3 mySqliteDatabase.db .dump > myTemporarySQLFile.sql
Import dump to MySQL
For small imports:
moose#pc08$ mysql -u <username> -p
Enter password:
....
mysql> use somedb;
Database changed
mysql> source myTemporarySQLFile.sql;
or
mysql -u root -p somedb < myTemporarySQLFile.sql
This will prompt you for a password. Please note: If you want to enter your password directly, you have to do it WITHOUT space, directly after -p:
mysql -u root -pYOURPASS somedb < myTemporarySQLFile.sql
For larger dumps:
mysqlimport or other import tools like BigDump.
BigDump gives you a progress bar:
Based on Jims's solution:
Quick easy way to migrate SQLite3 to MySQL?
sqlite3 your_sql3_database.db .dump | python ./dump.py > your_dump_name.sql
cat your_dump_name.sql | sed '1d' | mysql --user=your_mysql_user --default-character-set=utf8 your_mysql_db -p
This works for me. I use sed just to throw the first line, which is not mysql-like, but you might as well modify dump.py script to throw this line away.
There is no need to any script,command,etc...
you have to only export your sqlite database as a .csv file and then import it in Mysql using phpmyadmin.
I used it and it worked amazing...
Ha... I wish I had found this first! My response was to this post... script to convert mysql dump sql file into format that can be imported into sqlite3 db
Combining the two would be exactly what I needed:
When the sqlite3 database is going to be used with ruby you may want to change:
tinyint([0-9]*)
to:
sed 's/ tinyint(1*) / boolean/g ' |
sed 's/ tinyint([0|2-9]*) / integer /g' |
alas, this only half works because even though you are inserting 1's and 0's into a field marked boolean, sqlite3 stores them as 1's and 0's so you have to go through and do something like:
Table.find(:all, :conditions => {:column => 1 }).each { |t| t.column = true }.each(&:save)
Table.find(:all, :conditions => {:column => 0 }).each { |t| t.column = false}.each(&:save)
but it was helpful to have the sql file to look at to find all the booleans.
This script is ok except for this case that of course, I've met :
INSERT INTO "requestcomparison_stopword" VALUES(149,'f');
INSERT INTO "requestcomparison_stopword" VALUES(420,'t');
The script should give this output :
INSERT INTO requestcomparison_stopword VALUES(149,'f');
INSERT INTO requestcomparison_stopword VALUES(420,'t');
But gives instead that output :
INSERT INTO requestcomparison_stopword VALUES(1490;
INSERT INTO requestcomparison_stopword VALUES(4201;
with some strange non-ascii characters around the last 0 and 1.
This didn't show up anymore when I commented the following lines of the code (43-46) but others problems appeared:
line = re.sub(r"([^'])'t'(.)", "\1THIS_IS_TRUE\2", line)
line = line.replace('THIS_IS_TRUE', '1')
line = re.sub(r"([^'])'f'(.)", "\1THIS_IS_FALSE\2", line)
line = line.replace('THIS_IS_FALSE', '0')
This is just a special case, when we want to add a value being 'f' or 't' but I'm not really comfortable with regular expressions, I just wanted to spot this case to be corrected by someone.
Anyway thanks a lot for that handy script !!!
This simple solution worked for me:
<?php
$sq = new SQLite3( 'sqlite3.db' );
$tables = $sq->query( 'SELECT name FROM sqlite_master WHERE type="table"' );
while ( $table = $tables->fetchArray() ) {
$table = current( $table );
$result = $sq->query( sprintf( 'SELECT * FROM %s', $table ) );
if ( strpos( $table, 'sqlite' ) !== false )
continue;
printf( "-- %s\n", $table );
while ( $row = $result->fetchArray( SQLITE3_ASSOC ) ) {
$values = array_map( function( $value ) {
return sprintf( "'%s'", mysql_real_escape_string( $value ) );
}, array_values( $row ) );
printf( "INSERT INTO `%s` VALUES( %s );\n", $table, implode( ', ', $values ) );
}
}
echo ".dump" | sqlite3 /tmp/db.sqlite > db.sql
watch out for CREATE statements