Usage of locality-advertise-addr flag

I am try to deploy a multi-cloud cluster, I have a locality setting like this

locality=cloud=gcp,region=us-east1,datacenter=us-east1-c

When I try to set

--locality-advertise-addr=cloud=gcp,region=us-east1,datacenter=us-east1-c@192.168.1.1

I would get this error

Error: invalid argument “cloud=gcp,region=us-east1,datacenter=us-east1-c@192.168.1.1” for “–locality-advertise-addr” flag: invalid value for --locality-advertise-address:
Failed running “start”

Is this intented?

Hi @leokwan! Our docs are not clear enough here, but based on the cockroach start --help output, I think you need to include @address at each level of locality you’re specifying:

--locality-advertise-addr localityList
                                                            List of ports to advertise to other CockroachDB nodes for intra-cluster
                                                            communication for some locality. This should be specified as a commma
                                                            separated list of locality@address. Addresses can also include ports.
                                                            For example:
                                                            "region=us-west@127.0.0.1,datacenter=us-west-1b@127.0.0.1"
                                                            "region=us-west@127.0.0.1:26257,datacenter=us-west-1b@127.0.0.1:26258"

So for your example, I’d try:

--locality-advertise-addr=cloud=gcp@192.168.1.1,region=us-east1@192.168.1.1,datacenter=us-east1-c@192.168.1.1

Does that work? I’m not sure whether you need all levels of locality here. I’ll ask around.

Best,
Jesse

I am able to start the node with this setting.

Is that mean in each level of the locality, we can have different advertise address?
Thanks,
Leo

Although I can start all nodes with the above settings, some nodes in the cluster are trying to connect to other node in the same locality by the public ip address.

./cockroach start --cache=25% --max-sql-memory=35% --locality=cloud=gcp,region=us-east1,datacenter=us-east1-c --store=path=/mnt/d1,attrs=ssd,size=90% --log-dir=log --certs-dir=certs --max-disk-temp-storage=100GB --locality-advertise-addr=cloud=gcp@172.28.0.36,region=us-east1@172.28.0.36,datacenter=us-east1-c@172.28.0.36 --join=172.28.0.29,172.28.0.30,172.28.0.31,172.28.0.32,172.28.0.33,172.28.0.34,172.28.0.35 --advertise-addr=[public ip8]

I have similar settings for all other nodes, except ip addresses changed. How can I make the node to use private ip address for nodes in the same locality?

thanks,

Leo

I think there is some issue on this flag and I’ve open a ticket to follow up .

Thanks for opening that issue, @leokwan. The behavior you mention doesn’t sound right to me either:

some nodes in the cluster are trying to connect to other node in the same locality by the public ip address.

Let’s see if someone with more knowledge can chime in on the GitHub issue.

Best,
Jesse