The main roles of Scavenger are:
- Find Completed tasks and archive them
- Find Expired tasks (ones invoked, but not successfully completed by Workers) and either retry or mark them as failed.
Scavenger main class performes the following operations:
Read from sosw_tasks the ones successfully marked as completed by Workers and archive them.
get_db_field_name(key: str) → str¶
Could be useful if you overwrite field names with your own ones (e.g. for tests).
move_task_to_retry_table(task: Dict, labourer: sosw.labourer.Labourer)¶
Put the task to a Dynamo table sosw_retry_tasks, with the wanted delay: labourer.max_runtime * attempts. Delete it from sosw_tasks table.
Read from dynamo table sosw_retry_tasks, get tasks with retry_time <= now, and put them to sosw_tasks in the beginning of the queue.