I’m running in to the
transaction is too large to complete; try splitting into pieces error. I initially tried to fix the problem as suggested in the documentation by breaking a large
INSERT INTO ... SELECT FROM statement into multiple statements inside a single transaction. However, this doesn’t resolve the issue and I still get the same error even if each statement is only a few hundred bytes. After reading the documentation more carefully I noticed an important detail: “manually page through the data you want to insert using separate transactions” (emphasis mine).
So it sounds like I have to break apart the transaction into multiple transactions rather than break it up into multiple statements within the transaction. This should probably be spelled out much more clearly in the documentation because this is the type of error that you would only encounter in the wild after deploying something. I only found it after running some load tests with very large input data.
This is complicated further by the claims made in the same section which the average reader would incorrectly assume applied to this problem:
“A single statement can perform at most 64MiB of combined updates”
“There’s only a size limit, and it applies independently to each statement”
I’m enjoying using CockroachDB but the size limitations are challenging to deal with.