Perl read from xls insert to mysql - mysql

I'm am pretty new to Perl and I'm writing a Perl script to read data from xls and insert the results to MySQL DB but i have problem...
here is my code:
#!/usr/local/bin/perl
use strict;
use warnings;
use diagnostics;
use Spreadsheet::ParseExcel;
use DBI;
use Data::Dumper qw(Dumper);
my $parser = Spreadsheet::ParseExcel->new();
my $workbook = $parser->parse('test.xls');
my $dbh = DBI->connect("dbi:mysql:parser", "root", "123qwe", { RaiseError => 1}) or die $DBI::errstr;
my $query = 'INSERT INTO parser (Name,Country) VALUES (?,?)';
my $sth = $dbh->prepare($query) or die "Prepare failed: " . $dbh->errstr();
if ( !defined $workbook ) {
die $parser->error(), ".\n";
}
for my $worksheet ( $workbook->worksheets() ) {
my ( $row_min, $row_max ) = $worksheet->row_range();
my ( $col_min, $col_max ) = $worksheet->col_range();
for my $row ( $row_min .. $row_max ) {
for my $col ( $col_min .. $col_max ) {
my $cell = $worksheet->get_cell( $row, $col );
next unless $cell;
my $results = $cell->value();
open(my $fh, '>>', "test");
print $fh "$results\t";
close $fh;
}
}
}
open my $fh, "<", "test" or die $!;
while (<$fh>)
{
chomp;
my #vals = split;
$sth->execute(#vals);
}
close $fh;
So when i execute the script it ends with the following error:
DBD::mysql::st execute failed: called with 6 bind variables when 2 are needed at ./parser.pl line 39, <$fh> line 1.
Uncaught exception from user code:
DBD::mysql::st execute failed: called with 6 bind variables when 2 are needed at ./parser.pl line 39, <$fh> line 1.
Which is natural because i have indeed 6 variables in the output:
John Smith USA Ognyan Penkov Egypt
So the problem is that i cant seem to find a way to split the results from every column/row and put them in the MySQL tables because the Spreadsheet::ParseExcel reads all the date as 1 row.(For example the names must go to table Name and the country to table country)
My XLS file looks like this:
A B
1. John Smith USA
2. Ognyan Penkov Egypt
...etc...
My MySQL tables:
+--------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------+--------------+------+-----+---------+----------------+
| id | int(6) | NO | PRI | NULL | auto_increment |
| Name | varchar(255) | YES | | NULL | |
|Country | varchar(255) | YES | | NULL | |
+--------+--------------+------+-----+---------+----------------+

You are opening a file called test for append writing for every single cell in the document. You then add the current cell's value, followed by a tabulator \t character. Afterwards you open and read that same file line by line (but there is only one line), chomp off a line ending that is not there (because you didn't put one) and split on whitespace, because split without a delimiter uses \s, which is a single whitespace.
If omitted, PATTERN defaults to a single space, " " , triggering the previously described awk emulation.
And that's exactly the problem, because your file looks like this:
John\tSmith\tUSA\tOgnyan\tPenkov\tEgypt
The \t are single whitespaces, so you end up with #val being all of those. And if you pass that to your query, it fails.
Since you do not, as you say, have a table for names and a table for countries, but instead all you do is put the data from the Excel file into a single MySQL table row by row, you can just do that in your $row loop.
for my $worksheet ( $workbook->worksheets() ) {
my ( $row_min, $row_max ) = $worksheet->row_range();
my ( $col_min, $col_max ) = $worksheet->col_range();
for my $row ( $row_min .. $row_max ) {
my #values;
for my $col ( $col_min .. $col_max ) {
my $cell = $worksheet->get_cell( $row, $col );
next unless $cell;
push #values, $cell->value();
}
$sth->execute(#values) or die $dbh->errstr;
}
}

Related

MySQL Query: Concatenate horizontally based on selected ids

Say I have a table 'table_name' with two columns like this:
table_name
------------
my_id | data
------|-----
id1 | 312
id1 | 523
id1 | 128
id2 | 239
id2 | 479
id2 | 121
id3 | 639
id3 | 429
id3 | 131
id4 | 473
id4 | 872
id4 | 662
id4 | 174
id4 | 272
I tried around a while and I now found a way to select the ids I want to use:
SELECT DISTINCT my_id from table_name WHERE (my_id REGEXP 'id[234]')
This gives me the ids 'id2', 'id3' and 'id4'. How would I formulate a query that gives me my desired output, based on the selected ids as below?
id2 | id3 | id4
---------------
239 | 639 | 473
479 | 429 | 872
121 | 131 | 662
NaN | NaN | 174
NaN | NaN | 272
If your table really only has those two columns and there is no related data that can be used to order the data column like you have shown, then there is no good way to achieve the ordering you're looking for.
Relational data doesn't have any intrinsic "order" to it, even if you've inserted that data in a particular order or displayed it in a certain way here. If your table really only has those two columns, it will be difficult to match the "first" id2 value with the "first" id3 value and so on to get to your desired output. We could potentially rank them in a query and match them based upon that rank (ascending or descending) or even provide some random order capability, but there is no documented (i.e. reliable) way to get the order that you've provided.
While a table does have a physical order to the underlying bits in storage, and some products allow you to access them via a physical rowid (Oracle), no RDBMS that I know of (including MySql) guarantees the order in which the rows will be returned unless an order by clause is used in your select statement. This is literally by design and is considered a "feature" that came out of E.F. Codd's research.
First make a function that select distinct of the id's
public function get_id_of_table_name() {
$sql = 'SELECT DISTINCT my_id ';
$sql .= 'FROM table_name ';
$query = $this->db->query($sql);
$result = $query->row();
foreach ($result as $key => $value)
{
$result->data = $this->get_data($result->my_id);
}
return ( $query->num_rows() ) ? $result : FALSE;
}
And create the second function that will gather all the data's base on what id's
public function get_data($my_id) {
$params = [];
$sql = 'SELECT data ';
$sql .= 'FROM table_name ';
$sql .= 'WHERE my_id = ? ';
$params[] = $my_id;
$query = $this->db->query($sql, $params);
$result_data = $query->result();
$data = [];
foreach ($result_data as $result)
{
array_push($data, $result->data);
}
return $data;
}
You can put a where clause in the first function if you want.
$sql .= 'WHERE my_id = ? ';
and add the params
$params[] = $my_id;
You can also add parameters my_id as array in the first function so that you can get what id you want to get
$id = ["id2","id3","id4"];
public function get_id_of_table_name($id) {
$params = [];
$sql = 'SELECT DISTINCT my_id ';
$sql .= 'FROM table_name ';
$sql .= 'WHERE 1 ';
foreach ($id as $my_id)
{
$sql .= 'OR (my_id = ?) ';
$params[] = $my_id;
}
$query = $this->db->query($sql, $params);
$result = $query->row();
foreach ($result as $key => $value)
{
$result->data = $this->get_data($result->my_id);
}
return ( $query->num_rows() ) ? $result : FALSE;
}

Perl DBI where IS NULL with undefined variable

I have a perl DBI mysql query that looks like this:
my $rows = get_rows(
"select id from table where column1=?, column2=? and column3=?",
$var1,$var2,$var3
);
sub get_rows {
my $sql = shift;
my #vars = #_;
my $sth = $dbh->prepare($sql);
$sth->execute(#vars);
my $rows = $sth->fetchall_arrayref();
$sth->finish();
return $rows;
}
I'm running this to check if a particular row exists containing those vars. The problem I have however is with dealing with NULL values. These have to be selected as IS NULL rather than =?, which very often then misses rows that contain NULL values. For example, if vars 1, 2 and 3 contain '12', '23' and undef and table contains 12, 23 and NULL then the query returns no results. Is there a simple way to transform undef values into IS NULL values with DBI?
This is documented in DBI under NULL Values.
$sql_clause = defined $age? "age = ?" : "age IS NULL";
$sth = $dbh->prepare(qq{
SELECT fullname FROM people WHERE $sql_clause
});
$sth->execute(defined $age ? $age : ());

Perl MySQL DBI - update and get last value

I am using the Perl DBI module with MySQL and trying to get the initial value before adding 1 to it when updating a row.
If the current value was 1000 I need to return the value of 1000 and then add 1 to the value.
I use this statement in perl to use one transaction...
update TABLE_NAME set ID = (\#cur_value := ID) + 1
I know I can do a select then an update as two statements or lock the tables manually but transactions happen so fast on our platform that it may cause inconsistencies and this is the fastest way to do it.
However I simply cannot find a way to return the original value before the increment using this statement.
It works fine in ASP as below:
qry = "update V15_TRACKING set TRACKING_ID = (#cur_value := TRACKING_ID) + 1 where TRACKING_TYPE='ABC'"
Set oRS = oConn.Execute(qry)
qry = "select #cur_value"
if not oRS.EOF then
while not oRS.EOF
CurrTrackingID = oRs.Fields("#cur_value")
oRS.movenext
wend
oRS.close
end if
Please can someone advise me what I need to do to return the original value in Perl as I have searched everywhere and tried all sorts of solutions.
A snippet to show what you're actually doing in perl, and your result would help diagnose what is going on in your script.
I tried this trivial example:
The DB:
CREATE DATABASE TEST;
CREATE TABLE foo (
id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
val int(11) NOT NULL
);
INSERT INTO foo (val) VALUES (1);
And the Perl
#!/bin/env perl
use strict;
use warnings;
use Data::Dumper;
use DBI;
my $dbh = DBI->connect('DBI:mysql:database=test', 'dbuser', 'dbpass');
my $select = $dbh->prepare('SELECT * FROM foo WHERE id=?');
my $select_old_val = $dbh->prepare('SELECT #old_val');
my $update = $dbh->prepare('UPDATE foo SET val=(#old_val := val) + 1 WHERE id=?');
$update->execute(1);
$select_old_val->execute();
$select->execute(1);
while (my $row = $select_old_val->fetchrow_hashref) {
print Dumper $row;
}
while (my $row = $select->fetchrow_hashref) {
print Dumper $row;
}
And after a few goes:
$ perl select_and_update.pl
$VAR1 = {
'#old_val' => '10'
};
$VAR1 = {
'id' => '1',
'val' => '11'
};

mysql - finding string from array

Database table:
id| p1 | p2 | notes
1 | 1 | a | cat, mouse, dog
2 | 1 | a | cat, horse, dog
I now need to run a query that selects the row where "notes" does not contain a string defined in the $exclusions array. I have tried the LIKE '%mouse%' operator, but that gave an error.
$exclusions = array ("mouse");
if($row['p1'] == 1 && $row['p2'] == "a" && $row['notes'] not like '%mouse%') {...}
Thank you.
Looks like you are doing the logic in a mix of PHP code and sql. To do it in php, you can do
!strstr($row['notes'], 'mouse')
That says "If there is no occurrence of "mouse" in $row['notes']"
The ! will make it return true if there is no occurrence.
if($row['p1'] == 1 && $row['p2'] == "a" && $row['notes'] not like '%mouse%') {...}
This is not MySQL syntax. It looks like PHP, and in PHP you can not use LIKE. Try a string comparison operator like strstr. http://php.net/manual/en/function.strstr.php
Mysql Style
A query to get all rows without mouse could be this:
SELECT * FROM `tablename`
WHERE `notes` NOT LIKE '%mouse%';
Or to get the exclusions from a php array:
$condition = "WHERE ";
$first = true;
foreach($exclusions as $ex) {
if(!$first) $condition .= " AND "; // or use OR
$condition .= "`notes` NOT LIKE \"%$ex%\"";
$first = false;
}
$complete_query = "SELECT * FROM `tablename` $condition;";

How to get available values for SET field?

Is there any way to get available values for SET field in table?
Thank you.
You can retrieve the possible values for a SET field using DESCRIBE myTableName mySetColumn or SHOW COLUMNS FROM myTableName LIKE mySetColumn:
mysql> DESCRIBE myTableName mySetColumn;
+-------+-------------------------------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------------------------------------+------+-----+---------+-------+
| myset | set('Travel','Sports','Dancing','Dining') | YES | | NULL | |
+-------+-------------------------------------------+------+-----+---------+-------+
Informative article here, manual here.
SELECT `COLUMN_TYPE` FROM `information_schema`.`COLUMNS`
WHERE `TABLE_SCHEMA` = 'my_database_name'
AND `TABLE_NAME` = 'my_table_name'
AND `COLUMN_NAME` = 'my_set_column';
gives you only the type, e.g.
set('Travel','Sports','Dancing','Dining')
you'll still have to extract the set value on a textual base, but there's less clutter around it this way.
Full implementation from #Andy's link to the documentation:
/**
* #return array
* #param table DB table
* #param column Column name
* #desc Return an array of the possible values for a SET
*/
function get_set($table,$column)
{
$sql = "SHOW COLUMNS FROM $table LIKE '$column'";
if (!($ret = mysql_query($sql)))
die("Error: Could not show columns");
$line = mysql_fetch_assoc($ret);
$set = $line['Type'];
// Remove "set(" at start and ");" at end.
$set = substr($set,5,strlen($set)-7);
// Split into an array.
return preg_split("/','/",$set);
}
The above methods weren't exactly working for me, but I ended up with this and it works great. Posting in case this helps anyone else in the same situation. You'll just have to trim the string based on your results, as it will include your column name alongside the word 'set'.
<?php>
include ('database.php');
$query = "SHOW COLUMNS FROM Products LIKE 'Genre'";
$stmt = $con->prepare( $query );
$result = $stmt -> execute();
if ($result) {
$row = $stmt -> fetch(PDO::FETCH_ASSOC);
$genres = implode($row);
$genres = substr($genres,10,strlen($genres)-14);
//echo $genres;
$genres = preg_split("/','/",$genres);
//this is to populate my select box options with set values
echo "<select name = 'genre'>";
echo "<option>Select...</option>";
foreach ($genres as $key=>$value) {
echo "<option name = '$value' value = '$value'>$value</option>";
};
echo "</select>";
}
?>
Here is how to get the possible values of SET using PDO extension.
function get_set($table, $column)
{
global $db; //PDO DB Handler
$sql = "SHOW COLUMNS FROM $table LIKE :column";
$stmt = $db -> prepare($sql);
$stmt -> bindParam(":column", $column, PDO::PARAM_STR, 50);
try {
$result = $stmt -> execute();
$row = $stmt -> fetch(PDO::FETCH_ASSOC);
$set = $row['Type'];
$set = substr($set,5,strlen($set)-7);
// Split into an array.
return preg_split("/','/",$set);
} catch (PDOException $e) {
echo $e -> getMessage();
return false;
}
}
[Source]