I want to set up a cron job and do scheduled imports from a particular .csv file that I will upload/update via ftp.
I wonder if there is an easy way to set up a product import for X-Cart 5 using linux console command?
There is not default way to do import via linux console. But you can create simple console script and run it via cron.
Example of code(only concept, not solution for your case):
#!/usr/bin/env php
<?php
if ('cli' != PHP_SAPI) {
exit (1);
}
require_once __DIR__ . DIRECTORY_SEPARATOR . 'top.inc.php';
XLite::getInstance()->run(true);
// Initialize importer
// See all possible options in classes/XLite/Logic/Import/Importer.php __construct()
$importer = new \XLite\Logic\Import\Importer(
array(
'warningsAccepted' => true,
'delimiter' => ',',
'ignoreFileChecking' => true,
'files' => array(
'/full/path/to/xcart/var/import/products.csv',
'/full/path/to/xcart/var/import/categories.csv'
)
)
);
// Verifiaction step
while ($importer->getStep()->valid()) {
$importer->getStep()->current()->process();
$importer->getStep()->next();
}
// Check warnings & errors after verification and save to log file
if($importer->hasWarnings()) {
$warnings = \XLite\Core\Database::getRepo('XLite\Model\ImportLog')
->findBy(array('type' => \XLite\Model\ImportLog::TYPE_WARNING));
\XLite\Logger::logCustom('import_warnings', var_export($warnings, true));
//Clear warning messages
\XLite\Core\Database::getRepo('XLite\Model\ImportLog')
->deleteByType(\XLite\Model\ImportLog::TYPE_WARNING);
}
if($importer->hasErrors()) {
$errors = \XLite\Core\Database::getRepo('XLite\Model\ImportLog')
->findBy(array('type' => \XLite\Model\ImportLog::TYPE_ERROR));
\XLite\Logger::logCustom('import_errors', var_export($errors, true));
}
// Import/proccess quick data for products/resize images
// This loop wont'b executed if ($importer->hasWarnings() == true && warningsAccepted == false)
// or ($importer->hasErrors() == true)
while ($importer->isNextStepAllowed()) {
$importer->getOptions()->step = $importer->getOptions()->step + 1;
$importer->getOptions()->position = 0;
while ($importer->getStep()->valid()) {
$importer->getStep()->current()->process();
$importer->getStep()->next();
}
}
Also you can use scheduled task in X-Cart 5. To use it you should create your own module with class witch will extends abstract class classes/XLite/Core/Task/Base/Periodic.php
You can find example of code in file classes/XLite/Module/CDev/XMLSitemap/Core/Task/GenerateSitemap.php
Run tasks registered in X-Cart 5: php console.php --target=cron
Related
I have a .csv file with more than 690 000 rows.
I found a solution to import data that works very well but it's a little bit slow... (around 100 records every 3 seconds = 63 hours !!).
How can I improve my code to make it faster ?
I do the import via a console command.
Also, I would like to import only prescribers that aren't already in database (to save time). To complicate things, no field is really unique (except for id).
Two prescribers can have the same lastname, firstname, live in the same city and have the same RPPS and professional codes. But, it's the combination of these 6 fields which makes them unique !
That's why I check on every field before create a new one.
<?php
namespace AppBundle\Command;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Helper\ProgressBar;
use AppBundle\Entity\Prescriber;
class PrescribersImportCommand extends ContainerAwareCommand
{
protected function configure()
{
$this
// the name of the command (the part after "bin/console")
->setName('import:prescribers')
->setDescription('Import prescribers from .csv file')
;
}
protected function execute(InputInterface $input, OutputInterface $output)
{
// Show when the script is launched
$now = new \DateTime();
$output->writeln('<comment>Start : ' . $now->format('d-m-Y G:i:s') . ' ---</comment>');
// Import CSV on DB via Doctrine ORM
$this->import($input, $output);
// Show when the script is over
$now = new \DateTime();
$output->writeln('<comment>End : ' . $now->format('d-m-Y G:i:s') . ' ---</comment>');
}
protected function import(InputInterface $input, OutputInterface $output)
{
$em = $this->getContainer()->get('doctrine')->getManager();
// Turning off doctrine default logs queries for saving memory
$em->getConnection()->getConfiguration()->setSQLLogger(null);
// Get php array of data from CSV
$data = $this->getData();
// Start progress
$size = count($data);
$progress = new ProgressBar($output, $size);
$progress->start();
// Processing on each row of data
$batchSize = 100; # frequency for persisting the data
$i = 1; # current index of records
foreach($data as $row) {
$p = $em->getRepository('AppBundle:Prescriber')->findOneBy(array(
'rpps' => $row['rpps'],
'lastname' => $row['nom'],
'firstname' => $row['prenom'],
'profCode' => $row['code_prof'],
'postalCode' => $row['code_postal'],
'city' => $row['ville'],
));
# If the prescriber doest not exist we create one
if(!is_object($p)){
$p = new Prescriber();
$p->setRpps($row['rpps']);
$p->setLastname($row['nom']);
$p->setFirstname($row['prenom']);
$p->setProfCode($row['code_prof']);
$p->setPostalCode($row['code_postal']);
$p->setCity($row['ville']);
$em->persist($p);
}
# flush each 100 prescribers persisted
if (($i % $batchSize) === 0) {
$em->flush();
$em->clear(); // Detaches all objects from Doctrine!
// Advancing for progress display on console
$progress->advance($batchSize);
$progress->display();
}
$i++;
}
// Flushing and clear data on queue
$em->flush();
$em->clear();
// Ending the progress bar process
$progress->finish();
}
protected function getData()
{
// Getting the CSV from filesystem
$fileName = 'web/docs/prescripteurs.csv';
// Using service for converting CSV to PHP Array
$converter = $this->getContainer()->get('app.csvtoarray_converter');
$data = $converter->convert($fileName);
return $data;
}
}
EDIT
According to #Jake N answer, here is the final code.
It's very very faster ! 10 minutes to import 653 727 / 693 230 rows (39 503 duplicate items!)
1) Add two columns in my table : created_at and updated_at
2) Add a single index of type UNIQUE on every column of my table (except id and dates) to prevent duplicate items with phpMyAdmin.
3) Add ON DUPLICATE KEY UPDATE in my query, to update just the updated_at column.
foreach($data as $row) {
$sql = "INSERT INTO prescripteurs (rpps, nom, prenom, code_prof, code_postal, ville)
VALUES(:rpps, :nom, :prenom, :codeprof, :cp, :ville)
ON DUPLICATE KEY UPDATE updated_at = NOW()";
$stmt = $em->getConnection()->prepare($sql);
$r = $stmt->execute(array(
'rpps' => $row['rpps'],
'nom' => $row['nom'],
'prenom' => $row['prenom'],
'codeprof' => $row['code_prof'],
'cp' => $row['code_postal'],
'ville' => $row['ville'],
));
if (!$r) {
$progress->clear();
$output->writeln('<comment>An error occured.</comment>');
$progress->display();
} elseif (($i % $batchSize) === 0) {
$progress->advance($batchSize);
$progress->display();
}
$i++;
}
// Ending the progress bar process
$progress->finish();
1. Don't use Doctrine
Try to not use Doctrine if you can, it eats memory and as you have found is slow. Try and use just raw SQL for the import with simple INSERT statements:
$sql = <<<SQL
INSERT INTO `category` (`label`, `code`, `is_hidden`) VALUES ('Hello', 'World', '1');
SQL;
$stmt = $this->getDoctrine()->getManager()->getConnection()->prepare($sql);
$stmt->execute();
Or you can prepare the statement with values:
$sql = <<<SQL
INSERT INTO `category` (`label`, `code`, `is_hidden`) VALUES (:label, :code, :hidden);
SQL;
$stmt = $this->getDoctrine()->getManager()->getConnection()->prepare($sql);
$stmt->execute(['label' => 'Hello', 'code' => 'World', 'hidden' => 1);
Untested code, but it should get you started as this is how I have done it before.
2. Index
Also, for your checks, have you got an index on all those fields? So that the lookup is as quick as possible.
I'm currently using a PDO class that works on MySQL perfectly. But when it comes to MSSQL , I get an error when I try to insert data via the bindValue() function.
I'm using this method for data binding:
bindValue(":param",$value)
Step 1 - Create an array for the table fields in the query
$counter = 0;
foreach($fields as $cols)
{
$fieldBind[$counter] = ":".$cols;
$new_f = $new_f ."". $cols;
$counter ++;
if($counter!=count($fields))
{
$new_f = $new_f.",";
}
}
output : (
[0] => :field1
[1] => :field2
[2] => :field3
)
Step 2 - Create an array for the data of the fields in the query
$counter2 = 0;
foreach($data as $cols)
{
$dataBind[$counter2] = $cols;
$new_d = $new_d."'".$cols."'";
$counter2 ++;
if($counter2!=count($data))
{
$new_d = $new_d.",";
}
}
output : ( [0] => value1 [1] => value2 [2] => value3 )
Step 3 - Prepare the query via the query function
parent::query("INSERT INTO $table($new_f) VALUES($new_d)");
Step 4 - Bind the Parameters and Values
for($i=0;$i<count($data);$i++){
parent::bind($fieldBind[$i],$dataBind[$i]);
}
The query looks like this:
INSERT INTO table(field1,field2,field3) values(':value1',':value2',':value3')
Step 5 - Execute the Query
try {
parent::execute();
return parent::rowCount();
}
catch(PDOException $e) {
echo $e->getMessage();
}
This method works perfectly on MySQL, but when I try to execute this on SQL Server, I get this error:
SQLSTATE[IMSSP]: Tried to bind parameter number 0. SQL Server supports a maximum of 2100 parameters.
Try removing the apostrophe ''
from :
INSERT INTO table(field1,field2,field3) values(':value1',':value2',':value3')
to the following:
INSERT INTO table(field1,field2,field3) values(:value1,:value2,:value3)
Update: A fresh install did not have the problem. I gave up on trying to fix the upgraded Ubuntu.
Orignal question:
I'm using the perl CPAN package JSON to convert a hasref to json using the to_json function.
This worked fine on Ubuntu 14.04 with perl version 5.18.2, but after ugrading to Ubuntu 16.04 with perl version 5.22.1 I get the error message:
hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this)
The orignal code was this:
my $lang = {
'connection_lost' => 'Network connection was lost',
'connection_lost_more' => 'Please refresh this page to fix this problem'
};
my $json_lang = to_json($lang);
I checked with warn ref($lang) the type of $lang which returned 'HASH', so it should be an hashref?
I tried to change it to this:
my %lang;
$lang{'connection_lost'} = 'Network connection was lost';
$lang{'connection_lost_more'} = 'Please refresh this page to fix this problem';
my $json_lang = to_json(%lang);
and this:
my %lang;
$lang{'connection_lost'} = 'Network connection was lost';
$lang{'connection_lost_more'} = 'Please refresh this page to fix this problem';
my $json_lang = to_json(\%lang);
Both failed.
Then I tried the allow_nonref switch:
my $lang = {
'connection_lost' => 'Network connection was lost',
'connection_lost_more' => 'Please refresh this page to fix this problem'
};
my $jsonnonref = JSON->new->allow_nonref;
my $json_lang = $jsonnonref->to_json($lang);
which resulted in the error message to_json should not be called as a method
How do I get this to work?
Absolute minimal code that does not work for me:
package Handlers::test_handlers;
use strict;
use warnings;
use Apache2::Const -compile => qw(OK);
use Apache2::Request;
use JSON;
sub handler {
my $lang = {
'connection_lost' => 'connection_lost',
'connection_lost_more' => 'connection_lost_more'
};
#my $json_lang = 'Hello world';
my $json_lang = to_json($lang);
print $json_lang;
return Apache2::Const::OK;
}
1;
Using the 'Hello world'-line works, while the to_json-line does not.
I now had the same issue with a fresh install, and decided to completely remove JSON::XS / libjson-xs-perl from the project, which instantaneously fixed this problem
I'm just trying out dropzone for the first time and would need assistance to set it up. I am getting error 500 thus:
Failed to load resource: the server responded with a status of 500 (Internal Server Error)
This is the controller:
public function storeDocument(Request $request){
$file = $request->file('file');
$fileName = uniqid().$file->getClientOriginalName();
$file->move('gallery/images', $fileName);
$productVerificationValidation = ProductVerificationValidation::findOrFail($request->productDocumentNameId);
$documentsUploaded = $productVerificationValidation->productDocumentUpload()->create([
'gallery_id' => $request->productDocumentNameId,
'user_id' => Auth::user()->id,
'company_id' => $request->company_id,
'product_id' => $request->product_id,
'file_name' => $fileName,
'file_size' => $file->getClientSize(),
'file_mime' => $file->getClientMimeType(),
'file_path' => 'gallery/images'. $fileName
]);
}
This is the route to store the image credentials:
Route::post('product/document/upload/save', array('before'=>'csrf', 'uses'=>'ProductVerificationValidationController#storeDocument'));
This is the form for Dropzone:
<form action="{{url('product/document/upload/save' )}}"
class="dropzone first-input-div" id="addImages">{{csrf_field()}}
{!!Form::hidden('productDocumentNameId', $productVerificationValidation->id)!!}
{!!Form::hidden('product_id', $productVerificationValidation->product_id)!!}
{!!Form::hidden('company_id', $productVerificationValidation->company_id)!!}</form>
This is the model setup:
public function productDocumentUpload(){
return $this->hasMany('App\ProductDocumentUpload');
}
I'm yet to understand what I need to do right. Please help out.
Which would a the best and most flexible process for creating formatted PDF's of Avery labels on a linux machine with perl?
The labels need to include images and will have formatting similar to spanned rows and columns in an html table. And example would be several rows of text on the left hand side and an image on the right hand side that spans the text.
These are my thoughts but if you have additional ideas please let me know.
perl to PDF with PDF::API2
perl to PS with ??? -> PS to PDF with ???
perl to HTML w/ CSS formatting -> HTML to PDF with wkhtmltopdf
Has anybody done this and have any pointers, examples or links that may be of assistance?
Thank You,
~Donavon
They are all viable options.
I found wkhtmltopdf to be too resource intensive and slow. If you do want to go down that route there are existing html templates already which can be found by a quick google search.
PDF::API2 performs very well, and I run it on a server system with no problems. Here's an example script I use for laying out elements in a grid format;
#!/usr/bin/env perl
use strict 'vars';
use FindBin;
use PDF::API2;
# Min usage, expects bank.pdf to exist in same location
render_grid_pdf(
labels => ['One', 'Two', 'Three', 'Four'],
cell_width => 200,
cell_height => 50,
no_of_columns => 2,
no_of_rows => 2,
);
# Advanced usage
render_grid_pdf(
labels => ['One', 'Two', 'Three', 'Four'],
cell_width => 200,
cell_height => 50,
no_of_columns => 2,
no_of_rows => 2,
font_name => "Helvetica-Bold",
font_size => 12,
template => "blank.pdf",
save_as => "my_labels.pdf",
# Manually set coordinates to start prinding
page_offset_x => 20, # Acts as a left margin
page_offset_y => 600,
);
sub render_grid_pdf {
my %args = #_;
# Print data
my $labels = $args{labels} || die "Labels required";
# Template, outfile and labels
my $template = $args{template} || "$FindBin::Bin/blank.pdf";
my $save_as = $args{save_as} || "$FindBin::Bin/out.pdf";
# Layout Properties
my $no_of_columns = $args{no_of_columns} || die "Number of columns required";
my $no_of_rows = $args{no_of_rows} || die "Number of rows required";
my $cell_width = $args{cell_width} || die "Cell width required";
my $cell_height = $args{cell_height} || die "Cell height required";
my $font_name = $args{font_name} || "Helvetica-Bold";
my $font_size = $args{font_size} || 12;
# Note: PDF::API2 uses cartesion coordinates, 0,0 being
# bottom. left. These offsets are used to set the print
# reference to top-left to make things easier to manage
my $page_offset_x = $args{page_offset_x} || 0;
my $page_offset_y = $args{page_offset_y} || $no_of_rows * $cell_height;
# Open an existing PDF file as a templata
my $pdf = PDF::API2->open("$template");
# Add a built-in font to the PDF
my $font = $pdf->corefont($font_name);
my $page = $pdf->openpage(1);
# Add some text to the page
my $text = $page->text();
$text->font($font, $font_size);
# Print out labels
my $current_label = 0;
OUTERLOOP: for (my $row = 0; $row < $no_of_columns; $row++) {
for (my $column = 0; $column < $no_of_columns; $column++) {
# Calculate label x, y positions
my $label_y = $page_offset_y - $row * $cell_height;
my $label_x = $page_offset_x + $column * $cell_width;
# Print label
$text->translate( $label_x, $label_y );
$text->text( $labels->[$current_label]);
# Increment labels index
$current_label++;
# Exit condition
if ( $current_label > scalar #{$labels}) {
last OUTERLOOP;
}
}
}
# Save the PDF
$pdf->saveas($save_as);
}
Great you have found an answer you like.
Another option, which may or may not have suited you, would be to prepare the sheet that will be printed as labels as an open/libreoffice document, with pictures, layout, non-variant text ... (and can do all your testing runs through open/libreoffice).
Then:
use OpenOffice::OODoc;
then: read you data from a database
then:
my $document = odfDocument( file => "$outputFilename",
create => "text",
template_path => $myTemplateDir );
then:
for (my $r = 0; $r < $NumOfTableRows; $r++ ) {
for (my $c = 0; $c < $NumOfTableCols; $c++) {
:
$document->cellValue($theTableName, $r, $c, $someText);
# test: was written properly ?
my $writtenTest = $document->cellValue($theTableName, $r, $c);
chomp $writtenTest;
if ($someText ne $writtenTest) {
:
}
}
}
then:
$document->save($outputFilename );
# save (convert to) a pdf
# -f format;
# -n no start new listener; use existing
# -T timeout to connect to its *OWN* listener (only);
# -e exportFilterOptions
`unoconv -f pdf -n -T 60 -e PageRange=1-2 $outputFilename `;
# I remove the open/libreoffice doc, else clutter and confusion
`rm $outputFilename `;
As a quick overview of the practical issues:
name the layout tables
place your nice, correct open/libreoffice doc in "/usr/local/lib/site_perl/myNewTemplateDir/" say. You will need this (I think that this is the default, but I pass it anyway as $myTemplateDir
gotcha: the modules routines to wait for Open/Libreoffice to start (for the unoconv converter to start) do NOT work - unoconv will still not work after they say it will. I create dummy pdfs until one actually works - exists and has a non-zero size.