Index corruption(?) on fresh installed cluster with low traffic usage

General
#1

My company has decided to give a try to CRDB, after a small amount of time in testing we have decided to deploy a 3 node cluster to support a low traffic api with one database and two tables (~5q/s with peaks @ 100qps caused by certain taks), recently I was committed to execute a dump of the main database 1.6GB when I stumbled upon an error:
Error: pq: internal error: Non-nullable column with no value! Index scanned was “primary” with the index key columns () and the values

This error prevents me to create a dump and is casting a shadow on the process of evaluation for the adoption of this system.

Strange fact is that a full table scan (SELECT * FROM) returns same error, but, SELECT + LIMIT (whatsoever limit, may it be less or more than the actual rows) complete successfully.

#2

This does not look expected.

Can i ask what your DDL is?

What version of CockroachDB are you running?

Can you incrementally select with LIMIT to identify where there may be a NULL value where there shouldn’t?

Are you able to retrieve the whole table with LIMIT <# of rows in table>?

Thank you,
Matt

#3

Does your DML include on conflict clauses?

#4

Hi, deployed version is v19.1.5 on 3 nodes CentOS Linux release 7.7.1908 (Core) on GCP.
No on conflict syntax in query.

I can select with whatever limit, never arise the problem that happens when I not use limit or execute a dump, though I have not tried every single limit for each of the ~550k records I have.

Schema is:
CREATE TABLE push_user (
uid UUID NOT NULL DEFAULT gen_random_uuid(),
token STRING NOT NULL,
country STRING NOT NULL,
domain INT8 NULL,
dt_created TIMESTAMP NOT NULL,
dt_last_proc TIMESTAMP NULL,
sent_interval INT8 NOT NULL DEFAULT 1:::INT8,
ip INET NULL,
useragent STRING NULL,
source STRING NULL,
os STRING NULL,
browser STRING NULL,
CONSTRAINT “primary” PRIMARY KEY (uid ASC),
UNIQUE INDEX push_users_token_key (token ASC),
INDEX push_user_dt_last_proc_country_domain_idx (dt_last_proc ASC, country ASC, domain ASC),
FAMILY f1 (uid, token, country, domain, dt_created, sent_interval, ip, useragent, source, os, browser),
FAMILY f2 (dt_last_proc),
CONSTRAINT check_domain CHECK (domain > 0)
)