I am using Laravel 8 and I want to update billions of Products Prices. I add this code its work fine but it is not efficient it add load on server.
try {
$priceCsvs = PriceCsv::whereStatus(PriceCsv::PENDING)->get();
foreach ($priceCsvs as $price) {
dump($price->name." is started");
$csvData = fopen($price->file_url, 'r');
$firstline = true;
while (($data = fgetcsv($csvData, 555, ',')) !== false) {
if (!$firstline && !empty($data)) {
dump($data);
}
$firstline = false;
}
fclose($csvData);
dump($price->name." is End");
}
} catch (\Exception $ex) {
dump($ex->getMessage());
}
Point is: Is there any way to add sleep after 1000 entry from CSV file or any other efficient way.
If you are looking to schedule jobs then take a look at
Task Scheduling
Basically define some console (artisan) commands and then schedule them in the Console Kernel file. The scheduler is run with artisan and the recommendation is to run the command every minute (define a cron job). In the event that the kernel schedule item definition lines up with the minute that the 'php artisan schedule:work' command runs then your jobs will execute.
Related
I'm writing my own test runner for Google Apps Script. I want to put all my tests into one (or more) files. Then I will tell the runner which file(s) have my tests, and it should go through each file and run all the functions in it.
However, in order to do that, I need to:
Identify which file(s) contains the tests.
Iterate through each identified file and find all the functions.
Run those functions.
I've tried the following code to identify functions, but its this object seems to refer to my entire GAS code base, so it returns all the functions in my entire code base:
function functions() {
for(var i in this) {
if((typeof this[i]).toString()=="function" && this[i].toString().indexOf("native")==-1){
Logger.log(this[i].name);
}
}
}
Any suggestions?
I created a solution based on #TheMaster's suggestion. All of my test functions are named with the prefix test_.
It uses a try...finally block because if any of the tests throw an exception and cancel the test run, I want the finally block to report.
// Run all tests that begin with "test_" from any file.
function run_tests() {
Logger.log("------ RUNNING TESTS ------", 1);
try {
var successes = 0, failures = 0;
// 'this' refers to the global GAS environment
for(var i in this) {
// Find all functions whose names begin with test_
if(
(typeof this[i]).toString()=="function" &&
this[i].toString().indexOf("native")==-1 &&
this[i].name.indexOf("test_")==0
)
{
// Run a found test.
var result = eval(this[i].name + "()");
// Increment the count of successes and failures.
(result) ? successes++ : failures++;
Logger.log(`Ran test ${this[i].name} with result ${result ? 'SUCCESS' : 'FAILURE'}`);
}
}
} catch(e) {
failures++;
} finally {
var testcount = successes + failures;
Logger.log(`----- Ran ${testcount} tests with ${successes} successes and ${failures} failures. :alarm_clock:`, (failures) ? 0 : 1);
}
}
I have a large json file (not in size, but in elements). It has 30000 JSON elements and I am trying to produce Entities from it as it reads it.
So far I have it reading the file with Guzzle, and it ends up producing about 1500 entities before it crashes. I feel that I must be doing this the wrong way.
Here is my code:
public function generateEntities(Request $request, $number)
{
$client = new \GuzzleHttp\Client();
$request = new \GuzzleHttp\Psr7\Request('GET', 'http://www.example.com/file.json');
$promise = $client->sendAsync($request)->then(function ($response) {
$batchSize = 20;
$i = 0;
foreach (json_decode($response->getBody()) as $entityItem) {
$entity = new Entity();
$entity->setEntityItem($entityItem->string);
$em = $this->getDoctrine()->getManager();
$em->persist($entity);
if (($i % $batchSize) === 0) {
$em->flush(); // Executes all updates
}
$i++;
}
$em->flush();
});
$promise->wait();
return $this->redirect($this->generateUrl('show_entities'));
}
I worked out from research that I should be clearing the Entity Manager frequently, so I added in batch sizing etc to flush it every 20 entities created. This did help but is not enough to load the full 30000 files.
Maybe I am completely wrong and should be handling it a different way?
Is it possible for someone to please point me in the right direction, I am happy to work it out on my own I am just a little unsure where to proceed from here.
Thank you!
You could improve your process act in two ways:
1) increment the time limit of the execution of the controller action with the function set_time_limit, so put this as first line of the controller:
public function generateEntities(Request $request, $number)
{
set_time_limit(0); // set to zero, no time limit is imposed
2) Free much memory is possible for each interaction flush the data to the database and detach/free memory as follow:
$em->persist($entity);
$em->flush($entity);
$em->detach($entity);
$em->clear($entity);
unset($entity);
Hope this help
The script is running out of memory because every one of the 30000 entities is managed in memory. You need to detach the entities from the manager periodically to make sure they are "garbage collected". Use $em->clear(); in your batch flushing block to ensure memory isn't exhausted. See the Doctrine page on batch operations for more information.
Keep in mind though, that $em->clear() will detach all entities from the manager, not just those you are using in this loop.
I am trying to use gulp as an installer for complex system that involves creating folder, copying files around and runnin compliation scripts.
Presently I have the following gulp tasks:
// Some tasks skipped that set sessionFolder
gulp.task('default', function () {
// Main
runSequence('prepare_comedi', 'compile_comedi');
});
gulp.task('prepare_comedi', function () {
// Copies comedi files into build folder
gulp.src(['../comedi/**/*']).pipe(gulp.dest(sessionFolder));
});
gulp.task('compile_comedi', function () {
var logfile=this.currentTask.name+'.log';
gutil.log(gutil.colors.green(this.currentTask.name), ": building and installing COMEDI, logging to "+logfile);
var cmd= new run.Command('{ ./autogen.sh; ./configure; make; make install; depmod -a ; make dev;} > ../'+logfile+ ' 2>&1', {cwd:sessionFolder+'/comedi', verbosity:3});
cmd.exec();
});
When I run gulp, it becomes obvious that the processes start in background and gulp task finishes immediately. The first task above should copy source files, and second one compile them. In practice, second task hits the error, as the first task is not ready with copying when second task (almost immediately) starts.
If I run second task alone, previosuly having all files from first task copied, it works ok, but I have output like this:
[19:52:47] Starting 'compile_comedi'...
[19:52:47] compile_comedi : building and installing COMEDI, logging to compile_comedi.log
$ { ./autogen.sh; ./configure; make; make install; depmod -a ; make dev;} > ../compile_comedi.log 2>&1
[19:52:47] Finished 'compile_comedi' after 6.68 ms
So it takes 6.68 millisec to leave the task, while I want gulp to leave it only after all compilations specified in the task are finished. I then would run another compile process that uses built binaries from this step as a dependency.
How I can run external commands in such a way, that next gulp task starts only after first task complete execution of an external process?
You should make sure that the task prepare_comedi is finalized prior to start compile_comedi. In order to do so, since you're using regular streams on the prepare task, simply return the stream:
gulp.task('prepare_comedi', function () {
// !!! returns the stream. Gulp will not consider the task as done
// until the stream ends.
return gulp.src(['../comedi/**/*']).pipe(gulp.dest(sessionFolder));
});
Since these tasks are interdependent and require certain order, you might also want to consider refactoring your code to actually create two methods and call them normally. Take a look at this note.
Update
Addressing your question in the comment below, if you want to hold a task until some asynchronous job has been completed, you have pretty much three choices:
return a stream (case above)
returning a promise and fulfilling it when you're done (using Q in this example):
var Q = require('Q');
gulp.task('asyncWithPromise', function() {
var deferred = Q.defer();
// anything asynchronous
setTimeout(function() {
Q.resolve('nice');
}, 5000);
return deferred.promise;
});
Receiving a callback function and calling it
gulp.task('asyncWithPromise', function(done) {
setTimeout(function() {
done();
}, 5000);
});
These approaches are in the docs.
The requirement is to execute SSIS package, when a file is arrived at a folder,i do not want to start the package manually .
It is not sure about the file arrival timing ,also the files can arrive multiple times .When ever the files arrived this has to load into a table.I think, some solution like file watcher task ,still expect to start the package
The way I have done this in the past is with an infinite loop package called from SQL Server Agent, for example;
This is my infinite loop package:
Set 3 Variables:
IsFileExists - Boolean - 0
FolderLocation - String - C:\Where the file is to be put in\
IsFileExists Boolean - 0
For the For Loop container:
Set the IsFileExists variables as above.
Setup a C# script task with the ReadOnlyVariable as User::FolderLocation and have the following:
public void Main()
{
int fileCount = 0;
string[] FilesToProcess;
while (fileCount == 0)
{
try
{
System.Threading.Thread.Sleep(10000);
FilesToProcess = System.IO.Directory.GetFiles(Dts.Variables["FolderLocation"].Value.ToString(), "*.txt");
fileCount = FilesToProcess.Length;
if (fileCount != 0)
{
for (int i = 0; i < fileCount; i++)
{
try
{
System.IO.FileStream fs = new System.IO.FileStream(FilesToProcess[i], System.IO.FileMode.Open);
fs.Close();
}
catch (System.IO.IOException ex)
{
fileCount = 0;
continue;
}
}
}
}
catch (Exception ex)
{
throw ex;
}
}
// TODO: Add your code here
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
What this will do is essentially keep an eye on the folder location for a .txt file, if the file is not there it will sleep for 10 seconds (you can increase this if you want). If the file does exist it will complete and the package will then execute the load package. However it will continue to run, so the next time a file is dropped in it will execute the load package again.
Make sure to run this forever loop package as a sql server agent job so it will run all the time, we have a similar package running and it has never caused any problems.
Also, make sure your input package moves/archives the file away from the drop folder location.
As others have already suggested, using either WMI task or an infinite loop are two options to achieve this, but IMO SSIS is resource intensive. If you let a package constantly run in the background, it could eat up a lot of memory, cpu and cause performance issues with other packages depending on how many other packages you've running. So other option you may want to consider is schedule an Agent job every 5 minutes or 10 minutes or something and call your package in the job. Configure the package to continue only when a file is there or quit otherwise.
You can create a Windows service that uses WMI to detect file arrival and launch packages. Details on how to are located here: http://msbimentalist.wordpress.com/2012/04/27/trigger-ssis-package-when-files-available-in-a-folder-part2/?relatedposts_exclude=330
What about the SSIS File Watcher Task?
When using the Background Transfer API we must iterate through current data transfers to start them again ahen the App restarts after a termination (i.e. system shutdown). To get progress information and to be able to cancel the data transfers they must be attached using AttachAsync.
My problem is that AttachAsync only returns when the data transfer is finished. That makes sense in some scenarios. But when having multiple data transfers the next transfer in the list would not be started until the currently attached is finished. My solution to this problem was to handle the Task that AttachAsync().AsTask() returns in the classic way (not use await but continuations):
IReadOnlyList<DownloadOperation> currentDownloads =
await BackgroundDownloader.GetCurrentDownloadsAsync();
foreach (var downloadOperation in currentDownloads)
{
Task task = downloadOperation.AttachAsync().AsTask();
DownloadOperation operation = downloadOperation;
task.ContinueWith(_ =>
{
// Handle success
...
}, CancellationToken.None, TaskContinuationOptions.OnlyOnRanToCompletion,
TaskScheduler.FromCurrentSynchronizationContext());
task.ContinueWith(_ =>
{
// Handle cancellation
...
}, CancellationToken.None, TaskContinuationOptions.OnlyOnCanceled,
TaskScheduler.FromCurrentSynchronizationContext());
task.ContinueWith(t =>
{
// Handle errors
...
}, CancellationToken.None, TaskContinuationOptions.OnlyOnFaulted,
TaskScheduler.FromCurrentSynchronizationContext());
}
It kind of works (in the actual code I add the downloads to a ListBox). The loop iterates through all downloads and executes StartAsync. But the downloads are not really started all at the same time. Only one is runninng at a time and only if it finishes the next one continues.
Any solution for this problem?
The whole point of Task is to allow you to have the option of parallel operations. If you await then you are telling the code to serialize the operations; if you don't await, then you are telling the code to parallelize.
What you can do is add each download task to a list, telling the code to parallelize. You can then wait for tasks to finish, one by one.
How about something like:
IReadOnlyList<DownloadOperation> currentDownloads =
await BackgroundDownloader.GetCurrentDownloadsAsync();
if (currentDownloads.Count > 0)
{
List<Task<DownloadOperation>> tasks = new List<Task<DownloadOperation>>();
foreach (DownloadOperation downloadOperation in currentDownloads)
{
// Attach progress and completion handlers without waiting for completion
tasks.Add(downloadOperation.AttachAsync().AsTask());
}
while (tasks.Count > 0)
{
// wait for ANY download task to finish
Task<DownloadOperation> task = await Task.WhenAny<DownloadOperation>(tasks);
tasks.Remove(task);
// process the completed task...
if (task.IsCanceled)
{
// handle cancel
}
else if (task.IsFaulted)
{
// handle exception
}
else if (task.IsCompleted)
{
DownloadOperation dl = task.Result;
// handle completion (e.g. add to your listbox)
}
else
{
// should never get here....
}
}
}
I hope this is not too late but I know exactly what you are talking about. I'm also trying to resume all downloads when the application starts.
After hours of trying, here's the solution that works.
The trick is to let the download operation resume first before attacking the progress handler.
downloadOperation.Resume();
await downloadOperation.AttachAsync().AsTask(cts.Token);