Process a table from many consumers

#1

Hi,

I have a “to do” table, into which I insert tasks which has to be done.
Is it possible to process this in parallel from multiple consumers (processors) guaranteeing ?
I know this is something which is generally done with a message queue, but I have attached tables, and solving this in cockroach db makes it possible to do atomic modifications in all of them, and not requiring another component, making this easier.

The number of consumers are generally stable, but can change into any direction occasionally.

Is there a good advice on how to do this in cockroach?

(Alex Narayan) #2

This seems like a usecase for Redis or something must this be done in any relational database let alone cockroach? We have a similar setup that we might be migrating away from where workers take tasks from a queue but that queues state and the workers state is persisted.

Cockroach makes for a great database but you have to be smart about the usecase and use the best tool for the job. I think in this the strict isolation level of serializiable everything might cause you some pain with either readers being blocked by writers unless this table is strictly an append only log of sorts then that might be ok. Again I understand why you don’t want to introduce yet another system for this but it’s more or less the canonical use case for a Redis