When I use loadgen to do tpcc test,there are mant errors occur

General
#1

Hi:
I have import the csv data into my Cockrocah cluster,and install the tools Loadgen on one machine of my cluster.When I use the loadgen to test the cluster,many errors occur.

[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.102:26257/tpcc?sslmode=disable
2017/11/08 06:13:14 error in stockLevel: sql: no rows in result set
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.101:26257/tpcc?sslmode=disable
2017/11/08 06:13:20 error in stockLevel: sql: no rows in result set
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.103:26257/tpcc?sslmode=disable
2017/11/08 06:13:29 error in stockLevel: sql: no rows in result set
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.103:26257/tpcc?sslmode=disable
panic: runtime error: index out of range

goroutine 9 [running]:
main.orderStatus.run.func1(0xc420492000, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:122 +0xf70
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx.func1(0xc420492000, 0x777dc0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0x31
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteInTx(0x777dc0, 0xc4200160a0, 0x777840, 0xc420492000, 0xc42033e140, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:86 +0x122
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx(0x777dc0, 0xc4200160a0, 0xc4200d6140, 0xc420126c08, 0xc42016c000, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0xe0
main.orderStatus.run(0xc4200d6140, 0x18e, 0x4d7057, 0xc4200c6180, 0x3e8, 0x18e)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:77 +0x191
main.(*orderStatus).run(0x7b1480, 0xc4200d6140, 0x18e, 0x0, 0x0, 0x0, 0x0)
:1 +0x48
main.(*worker).run(0xc4200c69f0, 0xc4200c41e0, 0xc420016310)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/worker.go:151 +0x104
created by main.main
/root/go/src/github.com/cockroachdb/loadgen/tpcc/main.go:136 +0x33f

[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.101:26257/tpcc?sslmode=disable
2017/11/08 06:15:09 error in stockLevel: sql: no rows in result set
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.102:26257/tpcc?sslmode=disable
panic: runtime error: index out of range

goroutine 16 [running]:
main.orderStatus.run.func1(0xc420510280, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:122 +0xf70
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx.func1(0xc420510280, 0x777dc0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0x31
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteInTx(0x777dc0, 0xc4200160a0, 0x777840, 0xc420510280, 0xc42037c580, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:86 +0x122
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx(0x777dc0, 0xc4200160a0, 0xc4200d6140, 0xc42028cc08, 0xc4202ca000, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0xe0
main.orderStatus.run(0xc4200d6140, 0x209, 0x4d7057, 0xc4200c6180, 0x3e8, 0x209)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:77 +0x191
main.(*orderStatus).run(0x7b1480, 0xc4200d6140, 0x209, 0x0, 0x0, 0x0, 0x0)
:1 +0x48
main.(*worker).run(0xc4200c7470, 0xc4200c41e0, 0xc420016310)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/worker.go:151 +0x104
created by main.main
/root/go/src/github.com/cockroachdb/loadgen/tpcc/main.go:136 +0x33f
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.102:26257/tpcc?sslmode=disable
2017/11/08 06:23:13 error in stockLevel: sql: no rows in result set
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.103:26257/tpcc?sslmode=disable
2017/11/08 06:23:26 error in stockLevel: sql: no rows in result set
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.102:26257/tpcc?sslmode=disable
panic: runtime error: index out of range

goroutine 32 [running]:
main.orderStatus.run.func1(0xc4202d6000, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:122 +0xf70
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx.func1(0xc4202d6000, 0x777dc0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0x31
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteInTx(0x777dc0, 0xc4200160a0, 0x777840, 0xc4202d6000, 0xc42079c0e0, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:86 +0x122
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx(0x777dc0, 0xc4200160a0, 0xc4200d6140, 0xc420122c08, 0xc42019f400, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0xe0
main.orderStatus.run(0xc4200d6140, 0x241, 0x4d7057, 0xc4200c6180, 0x3e8, 0x241)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:77 +0x191
main.(*orderStatus).run(0x7b1480, 0xc4200d6140, 0x241, 0x0, 0x0, 0x0, 0x0)
:1 +0x48
main.(*worker).run(0xc420320b10, 0xc4200c41e0, 0xc420016310)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/worker.go:151 +0x104
created by main.main
/root/go/src/github.com/cockroachdb/loadgen/tpcc/main.go:136 +0x33f
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.103:26257/tpcc?sslmode=disable
panic: runtime error: index out of range

goroutine 22 [running]:
main.orderStatus.run.func1(0xc420320000, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:122 +0xf70
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx.func1(0xc420320000, 0x777dc0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0x31
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteInTx(0x777dc0, 0xc4200160a0, 0x777840, 0xc420320000, 0xc420490020, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:86 +0x122
github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb.ExecuteTx(0x777dc0, 0xc4200160a0, 0xc4200d60a0, 0xc420086c08, 0xc420115320, 0x0, 0x0)
/root/go/src/github.com/cockroachdb/loadgen/vendor/github.com/cockroachdb/cockroach-go/crdb/tx.go:52 +0xe0
main.orderStatus.run(0xc4200d60a0, 0x191, 0x4d7057, 0xc4200c6180, 0x3e8, 0x191)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/order_status.go:77 +0x191
main.(*orderStatus).run(0x7b1480, 0xc4200d60a0, 0x191, 0x0, 0x0, 0x0, 0x0)
:1 +0x48
main.(*worker).run(0xc4200c7bf0, 0xc4200c41e0, 0xc420016310)
/root/go/src/github.com/cockroachdb/loadgen/tpcc/worker.go:151 +0x104
created by main.main
/root/go/src/github.com/cockroachdb/loadgen/tpcc/main.go:136 +0x33f
[root@ceph4 tpcc]# ./tpcc -concurrency=64 -warehouses=1000 --duration=5m postgresql://root@172.16.50.103:26257/tpcc?sslmode=disable
2017/11/08 06:31:19 error in stockLevel: sql: no rows in result set

I don’t know what the trouble is.Thanks.

#2

Hi @going.

Our tpcc load generator assumes that it was first run with the -load flag
to initialize the data. It sounds like the data in your cluster doesn’t
match what the tool is expecting. What csvs did you import?

It’s probably easiest to empty your cluster and try again using ./tpcc -concurrency=64 -warehouses=1000 -duration=5m -load

#3

Hi @dan:

Thanks for your help. You are right. The data in my cluster is generated by the tool benchmarksql and it doesn’t match the tool loadgen. Then I use the tool loadgen and the command you told me to regenerate the data on my cluster. Then I use the tool loadgen to do TPC-C test on my cluster.
When I set the parameter -concurrency=32 or -concurrency=64,the test run normally. But when the -concurrency is set 128, errors occurred.

The command:

[root@ceph4 tpcc]# ./tpcc -concurrency=128 -warehouses=100 -duration=5m postgresql://172.16.50.103:26257/tpcc?sslmode=disable

The result:

28s newOrder 92.0 84.7 671.1 1275.1 1677.7 10200.5
29s newOrder 67.0 84.1 738.2 1946.2 2550.1 3892.3
2017/11/09 21:16:26 error in payment: select by last name fail: pq: restart transaction: HandledRetryableTxnError: TransactionAbortedError: txn aborted “sql txn” id=7f5561b8 key=/Table/51/1/98/0 rw=true pri=0.03538397 iso=SERIALIZABLE stat=ABORTED epo=1 ts=1510280185.403832112,107 orig=1510280185.403832112,107 max=1510280183.481292605,72 wto=false rop=false seq=6

I don’t know why this happen. And what I should to do to finish the TPC-C test?

#4

I responded to @going in another thread, but for anyone that finds this
later, the answer was that there was a bug in our tpcc load generator: