Luigi : Error while Parallelizing luigi task by using build function - function

Question

I was trying luigi multiprocessing capability by utilizing luigi.build method.
but i'm getting some library error while executing.
for next in self._add(item, is_complete):
File "/home/manoj/anaconda2/lib/python2.7/site-packages/luigi/worker.py", line 604, in _add
self._validate_dependency(d)
File "/home/manoj/anaconda2/lib/python2.7/site-packages/luigi/worker.py", line 622, in _validate_dependency
raise Exception('requires() must return Task objects')
here is piece of code i tried to achieve my objective.
import luigi
class TaskOne(luigi.Task):
custid= luigi.Parameter()
def requires(self):
pass
def output(self):
return luigi.LocalTarget("logs/"+str(self.custid)+"_success")
def run(self):
with self.output().open('w') as f:
f.write("%s\n" % '')
class TaskTwo(luigi.Task):
def requires(self):
customersList = ['A','B', 'C', 'D', 'E', 'F', 'G', 'H', 'I']
yield luigi.build([TaskOne(custid=cust_id) for cust_id in customersList], workers=2)
def output(self):
return luigi.LocalTarget("logs/overall_success.txt")
def run(self):
with self.output().open('w') as f:
f.write("%s\n" % "success")
if __name__ == '__main__':
luigi.run()
========================================================================

Answer 1

Why do you think you need to build in requires?
class TaskTwo(luigi.Task):
def requires(self):
customersList = ['A','B', 'C', 'D', 'E', 'F', 'G', 'H', 'I']
return [TaskOne(custid=cust_id) for cust_id in customersList]
If you want multiple workers, you can specify this at the command line when you start your pipeline.
luigi --module your_module TaskTwo --workers 2

Answer 2

requires() must return a luigi.Task object, or a list of luigi.Task objects. However, luigi.build() doesn't return anything. You don't need to call luigi.build to explicitly run the tasks, because Luigi handles running requirements on its own. The example task outlined in https://luigi.readthedocs.io/en/stable/tasks.html shows the basic paradigm of how it's supposed to work.
Also, you should omit requires() from TaskOne. If it has no dependencies, then there is no need to define it.