Perl parse csv file and iterate curl - csv

Question

I am trying to parse a csv file and iterate through it with curl. The following is my data set:
Act No. 2,Sep/1900/28
Act No. 3,Sep/1900/28
Act No. 10,Oct/1900/28
I have followed this Stackoverflow question: CSV into hash to basically create hash for my data set.
Here is my code:
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV_XS;
use IO::File;
use WWW::Curl::Easy;
my $url = "https://elibrary.judiciary.gov.ph/thebookshelf/docmonth/";
#my $filestoprocess = 'list_acts.csv';
# Usage example:
my $hash_ref = csv_file_hashref('toharvest_og_sourcing.csv');
foreach my $key (sort keys %{$hash_ref}){
my $urlcomplete = "$url"."#{$hash_ref->{$key}}";
#start the curl
my $user_agent = "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20140319 Firefox/24.0 Iceweasel/24.4.0";
my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_USERAGENT, $user_agent);
$curl->setopt(CURLOPT_FOLLOWLOCATION, 1);
#$curl->setopt(CURLOPT_SSL_VERIFYPEER, 1L);
#$curl->curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L);
$curl->setopt(CURLOPT_SSL_VERIFYPEER, 0);
$curl->setopt(CURLOPT_URL, $urlcomplete);
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
# Starts the actual request
my $retcode = $curl->perform;
# Looking at the results...
if ($retcode == 0) {
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
my $curledurldate = $response_body;
our ($issuancelink) = $curledurldate =~ /a href='(https.*?)'>.*?<STRONG>$key/s;
#print "$issuancelink\n";
if (defined $issuancelink) {
my $user_agent = "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20140319 Firefox/24.0 Iceweasel/24.4.0";
#my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_USERAGENT, $user_agent);
$curl->setopt(CURLOPT_FOLLOWLOCATION, 1);
#$curl->setopt(CURLOPT_SSL_VERIFYPEER, 1L);
#$curl->curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L);
$curl->setopt(CURLOPT_SSL_VERIFYPEER, 0);
$curl->setopt(CURLOPT_URL, $issuancelink);
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
# Starts the actual request
my $retcode = $curl->perform;
# Looking at the results...
if ($retcode == 0) {
# print("Transfer went ok\n");
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
my $curledsource = $response_body;
our ($ogsourcing) = $curledsource =~ /<br>\s+(\w+.*?)\s+?<CENTER>.*?H2/s;
my $filename = 'ogsourcingharvested.txt';
open (FH, '>>', $filename) or die("Could not open file. $!");
#print "Error processing ".$fh."$_\n";
print FH $ogsourcing."|"."{$key}\n";
close (FH);
}
else {
# Error code, type of error, error message
print("An error happened: $retcode ".$curl->strerror($retcode)." ".$curl->errbuf."\n");
}
} else {
# Error code, type of error, error message
print("An error happened: $retcode ".$curl->strerror($retcode)." ".$curl->errbuf."\n");
}
}
}
# Implementation:
sub csv_file_hashref {
my ($filename) = #_;
my $csv_fh = IO::File->new($filename, 'r');
my $csv = Text::CSV_XS->new ();
my %output_hash;
while(my $colref = $csv->getline ($csv_fh))
{
$output_hash{shift #{$colref}} = $colref;
}
return \%output_hash;
}
Basically, the code iterates through the second column, add that to the end of a URL, then that URL is curled. Afterwards, the content of the curled URL is searched for a particular content:
our ($issuancelink) = $curledurldate =~ /a href='(https.*?)'>.*?<STRONG>$key/s;
When that link shows up in search, that is put into a variable ($issuancelink) and then that variable $issuancelink is curled. Then a particular text in the curled file is searched, after which that particular text is captured and saved to a text file. However, my code is good if the second column (Sep/1900/28, Oct/1900/28 in this case) aren't repeated. However, if it's repeated, that is where I am having a problem, it seems the first iteration is the one being captured. So in my case, the link for Act No. 3 which has same originating URL (https://elibrary.judiciary.gov.ph/thebookshelf/docmonth/Sep/1900/28) as Act No. 2 (https://elibrary.judiciary.gov.ph/thebookshelf/docmonth/Sep/1900/28), the link for Act No. 2 is instead the one captured.
Thanks in advance!

Answer 1

However, my code is good if the second column (Sep/1900/28, Oct/1900/28 in this case) aren't repeated.
When you store values in a hash, the hash keys are unique. That means that if you have identical key names, they will overwrite each other.
This part of your code:
while(my $colref = $csv->getline ($csv_fh))
{
$output_hash{shift #{$colref}} = $colref;
}
Seems to be responsible. What you can do is to save the values in an array instead of a scalar (in this case, in an array ref).
I would do something like this:
while(my $colref = $csv->getline ($csv_fh))
{
my ($key, $value) = #$colref;
push #{$output_hash{$key}}, $value; # store values in array
}
Another benefit of this is that the values are copied. In your code, the array ref is copied. You are saved from problems by the limited scope of your variable my $colref, but generally speaking copying the values will save you from problems.
To access the array values you will probably need to loop for each hash key. Something like
for my $key (sort keys %$hash_ref) {
for my $values (#{$hash_ref{$key}}) {
# do stuff...