Bind: cannot assign requested address

Hi all,

Today I set up a secure cluster on my local machine - worked a charm. V. impressed
with the whole setup. So, I spent most of today setting up VMs - one on AWS, one
on Google and one on Azure - all fine and dandy! I can SSH into my VMs.

Then, I followed the docco here - install on AWS.

However, when I try and run

cockroach start
–certs-dir=/home/ubuntu/certs
–host=34.248.163.119
–join=34.248.163.119:26257,35.196.198.37:26257,52.169.87.119:26257
–cache=25%
–max-sql-memory=25%

I get the following message:

==============

ubuntu@ip-172-31-22-158:~/roach/maggot$ cockroach start --certs-dir=/home/ubuntu/certs --host=34.248.163.119 --join=34.248.163.119:26257,35.196.198.37:26257,52.169.87.119:26257 --cache=25% --max-sql-memory=25%
*

  • ERROR: cockroach server exited with error: consider changing the port via --port: listen tcp 34.248.163.119:26257: bind: cannot assign requested address

Failed running "start"
ubuntu@ip-172-31-22-158:~/roach/maggot$

======================================

I also get the same message on Google and Azure - verbatim!

I believe that I followed the instructions carefully. However, I’m thinking that
because I get the same error on all three VMs, I may have made some
horrible newbie blunder - can anyone help me out please?

I did manage to get PostgreSQL (compiled from source) working on Azure
and MariaDB (not source) running and access remotely - is it the port numbers
are too high or something?

Should you require any further information, please don’t hesitate to
ask - no question/point is too obvious - at least not for me! :blush:

Looks like the address you’re specifying in --host is the public IP. On most clouds, you cannot bind to this address, having to use the internal IP instead.
You probably want to leave --host empty, meaning that cockroach will bind to all interfaces (internal IP and localhost).

Yes, removing --host as a parameter appears to have resolved my issue! Thanks for
that! I can now successfully connect to 1 node - but see my new question. I can
connect to one and only one node - still, an improvement :grinning:, but kinda defeats
the raison d’être of the Cockroach system :disappointed_relieved:.

I’m not sure which new questions you’re referring to, so I’ll take a guess that you’re asking how to distribute your connections across all nodes.

You are correct that the ideal scenario is for your client to connect to any of the running cockroach nodes.
Most postgresql drivers do not support multiple server addresses (there are a few exceptions, but I’m not sure which).

In general, we recommend running a load balancer to distribute connections across cockroach nodes.
Each of the documents under our deploy docs has steps to properly configure a load balancer.

Some examples:

Once a load balancer is setup, you can use the load balancer address in the connection string. It will then take care of forwarding the connection to a running cockroach node.

Thanks for your reply - but this no longer appears to be the issue, at
least from a first glance perspective!

Initially, I started out young, dumb and full of, errr, optimism!

I tried to set up three servers: 1 Google, 1 AWS and 1 Azure - all Linux.

Now that I’m older (in terms of stress years! :slight_smile: ), I’ve brought my expectations
down.

I’m now at a point where I’m looking at setting up servers on AWS in Europe only,
so I get the same VPC, the same Security Groups, same Firewall rules - the
same everything!

I manage to launch the cluster on 1 machine but the others don’t appear to
connect - in the logs it says

============
*

  • WARNING: The server appears to be unable to contact the other nodes in the cluster. Please try
    • starting the other nodes, if you haven’t already
    • double-checking that the ‘–join’ and ‘–host’ flags are set up correctly
    • not using the ‘–background’ flag.
  • If problems persist, please see https://www.cockroachlabs.com/docs/v1.1/cluster-setup-troubleshooting.html.

=====================

But I can telnet ($> telnet ip.add.re.ss 26257 - get “connected” response),
but for some reason, the cluster won’t boot! I can’t ping from machine to
machine however - should I put in their internal IP addresses - from AWS
to AWS machine - perhaps will work better:

I’ve been starting my nodes with

======
cockroach start
–certs-dir=/home/ubuntu/certs
–join=34.248.163.119:26257,52.211.164.21:26257,35.154.52.51:26257
–cache=25%
–max-sql-memory=25%

=============

The one I can connect to - after “init” - I can run SQL, but again, the whole
raison d’être is to have multiple servers!

Anway, I’m trying again (with my 3 European servers - everything the same. If I’m
still stuck on that, I’ll have another question! Fingers crossed! :wink:

I’m using small instances - is this an issue?

Thanks for the update. Trying to bring up a cluster within the same VPC is definitely a good idea, networking gets tricky across regions, and even more so across clouds.

More details should be available in the logs, probably pointing us in the right direction.

A few things to try:

  • test your setup with --insecure. If this works, it’s most likely a certificates issue. Node certificates must include the list of all addresses used to reach the node
  • make sure 26257 is allowed in the firewall rules (even in the same VPC, if you’re using public IP addresses, the connection will go through the firewall)

Now, about addresses: cockroach nodes need two-way communication between all nodes for things to work properly. Each server advertises an address for other nodes to use. This means that this address must be reachable by other nodes.

If all nodes are in the same VPC, using internal addresses in the --join flag should be sufficient, and you can leave --host and --advertise-host unspecified.

If nodes are in different networks (internal IP addresses can’t be used), then you need to specify public addresses in the --join flag. Additionally, the nodes probably can’t advertise the proper address, so they need to be told explicitly using --advertise-host=<my public address> for each node. The --host flag can still be left unspecified.

Hi Marc, and thanks - I really appreciate you taking the time
to give me feedback and input!

I can try the insecure approach, but that defeats the whole purpose
of the exercise!

What I’m going to try next is the following: (if it fails, I’ll at least do the
insecure thing for Monday and get back to you - I don’t mind writing up
a bit of docco it it will help others on my path!)

==================================
cockroach cert create-node
52.51.129.17
localhost
127.0.0.1
–certs-dir=certs
–ca-key=my-safe-directory/ca.key

 cockroach cert create-node \
<node1 internal IP address> \
<node1 external IP address> \
<node1 hostname>  \
<other common names for node1> \
localhost \
127.0.0.1 \
<load balancer IP address> \
<load balancer hostname>  \
<other common names for load balancer instances> \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key

==========================

I have kept my cert create-node to a minimum - should
I be putting in more stuff? I do have the AWS internal
IPs. Do I put in $HOSTNAME also?
What about the big long AWS DNS name?
Are localhost and 127.0.0.1 necessary, or is
it a case of more is better?

I’ll try the kitchen sink this time!

I’ve noticed that a lot (but not all) of the node keys
are identical in size - is this to be expected? I haven’t
diff-ed them to check?

=========

cockroach start \
--certs-dir=/home/ubuntu/certs \
--join=52.51.129.17:26257,52.211.164.21:26257,35.154.52.51:26257 \
--cache=25% \
--max-sql-memory=25%

==========

So, you’re saying that here I should put something like

cockroach start \
--certs-dir=/home/ubuntu/certs \
--join=<my_three_INTERNAL_IP_address_list> \
--advertise-host=<my_individual_server_exteral_IP> \
--cache=25% \
--max-sql-memory=25% 

Final question - if I’m kicking the process off on server 1, to I put server 1’s
own IP in the list? Or do I just have 2 for every startup that I do for
a 3 node cluster?

IMHO, this is not well explained here:

https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-on-aws.html

but, as stated, I’ll do up a docco once I have it working!

Thanks again (avec mes remerciements si tu es francophone!)

I reformatted my post - I see that it’s like
dba.stackexchange.com - I’ll know in future! :relaxed:

One reason I was suggesting trying --insecure is to make sure the networking setup is correct before turning on secure mode. I definitely do not recommend running --insecure for real, it’s just a useful step in trying to figure out configure issues.

An important rule about certificates: any address you use to reach the node (private IP for the node to talk to itself, public IP if you’re going across VPCs, load balancer address if you have one) must be included in the node certificate. If it isn’t, the logs will show an error like Error: x509: certificate is valid for 127.0.0.1, not <requested address>

All nodes inside the same VPC

If all your nodes are in the same VPC, then you should be able to do the following:

Create node certs:

for each node, run:

cockroach cert create-node <my node internal IP address> localhost 127.0.0.1

Start the first node

Start it without the join flag to initialize the cluster. If you use the cockroach init method, you can pass the same join flag as the other nodes.

cockroach start --certs-dir=certs

Start the second node

At the very least, the --join flag needs an existing node (the first one). But you can specify as many as you want. It can include itself.

cockroach start --certs-dir=certs --join=<node 1 internal ip>,<node 2 internal ip>,<node 3 internal ip>

Starting the third node is the same thing as starting node 2.

Client connections

Now, this should give you a working 3 node cluster using only internal IP addresses.
This does mean that you won’t be able to talk to the cluster from outside the VPC.

If you want to connect to the cluster from outside. eg: cockroach sql --host <some node's public IP>, you will need:

  • the node’s public IP addresses included in the cockroach cert create-node command
  • firewall rules allowing anyone outside the VPC to reach :26257

Nodes in separate VPC

Now it gets tricky, partially because:

  • private IP addresses don’t mean anything to other nodes
  • nodes can’t determine their own public IP addresses, so we have to tell them

Create node certs

Here you will definitely need the public IP address as that is used by other nodes.

cockroach cert create-node <node private address> <node public address> localhost 127.0.0.1

Start the first node

The important part here is to tell other nodes which address they should use to reach us. This is done using the --advertise-host flag.

cockroach start --certs-dir=certs --advertise-host=<node 1 public address>

Start the other nodes

Each node needs to specify the --advertise-host flag. The addresses in the --join flag need to be public addresses.

cockroach start --certs-dir=certs --advertise-host=<node 2 public address> --join=<node 1 public address>,<node 2 public address>,<node 3 public address>

Client connections

Since public addresses are already in the node certificates and are allowed by the firewall, an external client can just use:

cockroach sql --host <some node public address>

Thank you again so much for your patience and assistance. I won’t post again today,
(it’s 20:00 here in Dublin!) so I’ll get back to you with progress (hopefully positive! :grinning:) on Monday - after lunch US Eastern.

Enjoy the rest of your weekend!

Pól…

Dragam,

By way of encouragement. Don’t give up. I managed to get three secure roaches running on three Google clound instances in three different data centers.

First I made sure I could connect from anywhere to anywhere on the right ports by running netcat as a listener.

Getting those certs and keys right confused me but in the end the instructions here https://www.cockroachlabs.com/docs/stable/manual-deployment.html turned out to be clear and accurate.

By default, you won’t be able to access the 8080 or 26257 ports in a GCE instance, so you need to open up traffic to those ports in the Firewall Rules section of the GCP project.

I recommend using a label named “cockroach” for a firewall rule with the value “tcp:8080;tcp:26257” under the “Protocols and ports” field.

That label then needs to be applied to the instances.

NOTE: This is only if you’re wanting to connect to nodes directly (not for production).

The complete docs for managing firewall rules for a project:
https://cloud.google.com/vpc/docs/using-firewalls