How Cockroach avoids Cross-node Joins?

General
#1

Hey Folks,

I have watched a few talks and one thing that I’m curious about is how do you guys avoid cross node joins? Let me explain: One of the issues with a distributed relational is that you might end up with your data in multiple nodes, and once you need it back, you might need to go through many nodes to get everything you need.

How do you guys avoid, let’s say, the User Data (which normally is stored using at least +5 tables) to be spread between multiple nodes? is there some kind of sharding based on the data structure (not on the fields)

Best,

#2

Hey Denis,

Thanks for asking the question, and apologies for the delayed response. (By the way, I believe we monitor CockroachDB slack - cockroa.ch/slack - more actively, so feel free to ping there if you see that your question goes unanswered here for too long.)

You’re totally right that in a distributed setting when performing a join we might need to perform cross-node communication (depending on the data placement), and CockroachDB doesn’t actually attempt to avoid the cross-node joins. In fact, we take a somewhat opposite approach - we will plan table readers on all nodes that have the relevant data, and then we place the processors (hash or merge joiners) on the same set of nodes, and the join operation is executed in a distributed fashion (we will route all the data from table readers based on the hash of the equality columns of the join). Thus, we do have a lot of cross-node data “flying” between nodes, but the execution of the join itself is distributed too.

I did a quick search of whether we have some documentation about it, and nothing turned up. I’m happy to provide more details if you’re interested, but there is also a way to examine the physical plan of a query - by using EXPLAIN (DISTSQL) (https://www.cockroachlabs.com/docs/stable/explain.html).

The sharding of the data is actually done automatically. Each shard is called “range” in CockroachDB, and it is the unit of replication (https://www.cockroachlabs.com/docs/v20.2/architecture/overview.html).