Not doing SKIP LOCKED will make it basically single threaded, no? I’m of the opinion that you should just use Temporal if you don’t need inter-job order guarantees
I explicitly want single-threaded execution within a tenant. After writing the original post, I figured out how to parallelize across tenants. The trick is to limit the query to only look at the next pending job for each tenant, which then allows for using SKIP LOCKED to process tenants in parallel:
SELECT job.job_id, rm.data
FROM qbo.transmit_job job
JOIN resource_mutation rm USING (tenant_id, resource_mutation_id)
WHERE job.state = 'pending'
AND job.job_id = (
SELECT qj2.job_id
FROM qbo.transmit_job qj2
WHERE job.tenant_id = qj2.tenant_id
AND qj2.state = 'pending'
ORDER BY qj2.create_time
LIMIT 1
)
ORDER BY job.create_time
LIMIT 1 FOR NO KEY UPDATE OF job SKIP LOCKED
> you should just use Temporal
Do you mean the Airflow-esque DAG runner? I prefer Postgres because it's one less moving part, I like transactional guarantees, my volume is tiny, and I can tweak the queue logic like above with a simple predicate change.
Skip locked main utility is to provide work queue capability to PG. Even documentation of PG is referring to skip locked as a main driver for work queues