The main roles of Scavenger are:
Find Completed tasks and archive them
Find Expired tasks (ones invoked, but not successfully completed by Workers) and either retry or mark them as failed.
- class sosw.scavenger.Scavenger(*args, **kwargs)
Scavenger main class performes the following operations:
- archive_tasks(labourer: sosw.labourer.Labourer)
Read from sosw_tasks the ones successfully marked as completed by Workers and archive them.
- get_db_field_name(key: str) str
Could be useful if you overwrite field names with your own ones (e.g. for tests).
- move_task_to_retry_table(task: Dict, labourer: sosw.labourer.Labourer)
Put the task to a Dynamo table sosw_retry_tasks, with the wanted delay: labourer.max_runtime * attempts. Delete it from sosw_tasks table.
- retry_tasks(labourer: sosw.labourer.Labourer)
Read from dynamo table sosw_retry_tasks, get tasks with retry_time <= now, and put them to sosw_tasks in the beginning of the queue (with greenfield of a task that will be invoked next).