Trouble starting second node

this works occasionally but i cannot start it this morning:

cockroach start --insecure --background ;
CockroachDB node starting at 2017-04-11 17:07:19.176502227 +0000 UTC
build: CCL beta-20170330 @ 2017/04/07 14:45:36 (go1.8)
admin: http://cockroachdb3:8080
sql: postgresql://root@cockroachdb3:26257?sslmode=disable
logs: cockroach-data/logs
store[0]: path=cockroach-data
status: restarted pre-existing node
clusterID: aa1f606f-f67a-4de5-a24e-83cde7a82294
nodeID: 1

cockroach start --insecure --background --store=hello-2 --port=26258 --http-port=8081 --join=localhost:26257 --logtostderr
I170411 17:07:28.126498 1 cli/start.go:330 CockroachDB CCL beta-20170330 (linux amd64, built 2017/04/07 14:45:36, go1.8)
I170411 17:07:28.227449 13 cli/start.go:367 starting cockroach node
W170411 17:07:28.227527 13 server/server.go:158 [n?] running in insecure mode, this is strongly discouraged. See --insecure.
W170411 17:07:28.227645 13 gossip/gossip.go:1178 [n?] no incoming or outgoing connections
W170411 17:07:28.230484 13 server/config.go:352 soft open file descriptor limit 4096 is under the recommended limit 15000; this may decrease performance
please see https://www.cockroachlabs.com/docs/recommended-production-settings.html for more details
I170411 17:07:28.230677 13 storage/engine/rocksdb.go:374 opening rocksdb instance at "hello-2"
I170411 17:07:28.244353 29 gossip/client.go:131 [n?] started gossip client to localhost:26257
I170411 17:07:28.269762 13 server/config.go:486 1 storage engine initialized
I170411 17:07:28.270054 13 server/server.go:641 [n?] sleeping for 460.174629ms to guarantee HLC monotonicity
I170411 17:07:28.733004 13 server/node.go:450 [n2] initialized store [n2,s2]: {Capacity:10725883904 Available:8566120448 RangeCount:7 LeaseCount:0}
I170411 17:07:28.733048 13 server/node.go:341 [n2] node ID 2 initialized
I170411 17:07:28.733136 13 gossip/gossip.go:293 [n2] NodeDescriptor set to node_id:2 address:<network_field:“tcp” address_field:“cockroachdb3:26258” > attrs:<> locality:<>
I170411 17:07:28.733227 13 storage/stores.go:296 [n2] read 1 node addresses from persistent storage
I170411 17:07:28.733384 13 storage/stores.go:312 [n2] wrote 2 node addresses to persistent storage
I170411 17:07:28.733408 13 server/node.go:599 [n2] connecting to gossip network to verify cluster ID…
Error: cockroach server exited with error: node 2 belongs to cluster “d27fb4d2-5f88-435d-a7a3-afe763dd3f13” but is attempting to connect to a gossip network for cluster "aa1f606f-f67a-4de5-a24e-83cd
e7a82294"
Failed running "start"
Error: exit status 1
Failed running “start”

is there something particular obvious i am missing?

There’s data in the hello-2 directory from a previous incarnation of the cluster (which had the ID d27fb4d2...). When you want to start from scratch, you have to delete all the data directories (cockroach-data and hello-2).

If you didn’t want to start from scratch, but wanted to simulate the failure of node 1 and rejoin a new node to cluster in hello-2, you’d specify node 2’s address in the --join parameter when restarting node 1.