Scavenger

The main roles of Scavenger are:

  • Find Completed tasks and archive them
  • Find Expired tasks (ones invoked, but not successfully completed by Workers) and either retry or mark them as failed.
View Licence Agreement

class sosw.scavenger.Scavenger(custom_config=None, **kwargs)[source]

Scavenger main class performes the following operations:

  • archive_tasks(labourer)
  • handle_expired_tasks(labourer)
  • retry_tasks(labourer)
archive_tasks(labourer: sosw.labourer.Labourer)[source]

Read from sosw_tasks the ones successfully marked as completed by Workers and archive them.

get_db_field_name(key: str) → str[source]

Could be useful if you overwrite field names with your own ones (e.g. for tests).

move_task_to_retry_table(task: Dict, labourer: sosw.labourer.Labourer)[source]

Put the task to a Dynamo table sosw_retry_tasks, with the wanted delay: labourer.max_runtime * attempts. Delete it from sosw_tasks table.

retry_tasks(labourer: sosw.labourer.Labourer)[source]

Read from dynamo table sosw_retry_tasks, get tasks with retry_time <= now, and put them to sosw_tasks in the beginning of the queue.