Sparodical errors after compaction phase

Hi all.
We are migrating our app from postgresql to cockroachdb. One of success criteria of migration is passing of existing integration tests. Those tests run ok on postresql app version.

Migration is completed but some tests fail time to time. It’s about 90 tests and from 0 to 5 of them fail time to time.

When tests fail they always fail with exception like
org.postgresql.util.PSQLException: ERROR: batch timestamp 1530086895.900492711,0 must be after GC threshold 1530086908.229701890,0

Around the same time I see messages like in cocroach db log like
I180627 08:07:59.514805 263525 sql/distsqlrun/hashjoiner.go:217 [client=,user=amp,n1,HashJoiner] buffer phase error batch timestamp 1530086869.326088922,0 must be after GC threshold 1530086869.470778632,0
Messages in cocroachdb log are appearing after compaction phase.

We run official docker image cockroach:v2.0.2 with default configuration in standalone mode. What could be a reason of those errors in cocroachdb?

Hey Vladimir,

That error indicates that a transaction had passed the garbage collection threshold.

By default, this is 25 hours, but the timestamps appear to be from this morning. What do you see when you run cockroach zone get <database>.<table> from the command line?

Hey Tim,
Thank you for the hint. That was the case.
I’ve set earlier ‘ttlseconds: 10’ and forgot about it.

Do I understand correct that some of transactions lasted longer than 10sec and was garbage collected?

Yup, that’s consistent with the error. If you could send over the results of a show create table statement for the tables in the test, and an explain statement for the query, we can see if there’s anything we can do to improve performance.