As an agency user, I want the broker to run jobs efficiently and in the correct order and to always return feedback so that I can validate my data.
We added this job as a catch-all to revisit the broker's process for managing the various jobs and processing the comprise a submission.
Our current database table that manages job dependencies was implemented before we began using celery/rabbitmq to handle the file-level validations. Now that we have a better idea of the overall workflow (for example, how cross-file validations work, when D, E, and F files get generated), it is time to revisit job flow from a high-level.
Also, during testing we discovered a need to incorporate tools to monitor the queue and its status.
this is resolved since moving to SQS