Accessing a cluster externally

General
#1

Hello,

I’m currently running an Apache webserver on Ubuntu and I’m trying to setup a database server as well. I have a secure CockroachDB cluster running on this Ubuntu server, but It’s only listening on the local 127.0.0.1 address.

I can view the management panel on the localhost port 8080 and I can reach the cluster on the localhost port 26257. I’m trying to open these ports externally, so I can view/reach them from my own PC. I’ve opened the TCP ports on the Ubuntu firewall and opened them on my hosting provider’s firewall. In order to make these ports view-able externally, I would have to make the cockroachDB cluster not just listen on localhost?

The http-addr for the Ubuntu cluster is currently set as 127.0.0.1:8080 and the listen-addr is currently set as 127.0.0.1:26257.

Any idea what I would change these listening addresses to in order to make them accessible externally? This way I can view the database and admin UI on my own PC.

Thank you!

#2

Hey Brian,

I believe you will need to connect to your webservers external IP address.

#3

Matt,

Thank you for the reply. I’m a bit new to the sysadmin side of things.

I understand that to connect to/reach the cockroachDB cluster externally, I would need to connect to the webserver’s external IP and port 26257. The issue seems to be with my initial cockroach start commands.

When I attempt to connect to my webserver’s external IP on port 26257, the port is closed/unreachable. I can only connect to the admin UI and database on the localhost webserver itself. If my understanding is correct, I believe this is because cockroach is only listening on the localhost?

Currently, my start node commands are:
Node 1: cockroach start --certs-dir=certs --listen-addr=localhost
Node 2: cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257

Therefore I would have to change the --listen-addr and the --http-addr IP values in order to have the web server listen for external connections?

I’ve tried changing these to 0.0.0.0 and also to the external IP of the webserver the nodes are running on, but the port is still unreachable externally.

Or, is this more of an issue on the webserver itself? I’ve opened the ports 26257 and 8080 on the Ubuntu firewall and on the hosting provider’s firewall, but perhaps something else is blocking it…

#4

I’ve figured it out!

For anyone else having issues in the future, I created the cert for the secure cluster as follows:
cockroach cert create-node 0.0.0.0 $(hostname) --certs-dir=certs --ca-key=my-safe-directory/ca.key
The IP address here is changed from the default localhost to 0.0.0.0.

Then the start node commands are changed slightly to listen on 0.0.0.0 instead of the default localhost.
Node 1: cockroach start --certs-dir=certs --listen-addr=0.0.0.0
Node 2: cockroach start --certs-dir=certs --store=node2 --listen-addr=0.0.0.0:26258 --http-addr=0.0.0.0:8081 --join=localhost:26257

Then opening the ports 26257 and 8080 on the Ubuntu firewall and my hosting provider’s firewall.

The issue with trying this before was not updating the cert for the secure cluster from localhost to 0.0.0.0.

#5

Awesome!

Glad you were able to figure it out.

Let me know if you have any other questions!

Thanks,
Matt

#6

Well, I had the cluster running on the Linux server and was able to access it externally with DBeaver with the generated SSL certs. However, I started moving some things around and must have messed something up. I’ve tried removing all Cockroach data from my server to start from scratch, but no luck.

Error: ERR_INVALID_HTTP_RESPONSE
Whenever I try to reach my cluster externally from the IP and port 26257 it fails to connect. If I enter the cluster’s external IP and port in a browser (to see if the port is open), I get the following error: <EXTERNAL IP> sent an invalid response. ERR_INVALID_HTTP_RESPONSE I can still reach the admin UI externally on port 8080 without any issues.

Now, I know it won’t show much when accessing the cluster address in a browser, but when it was working correctly it showed little boxes like this [][][][][][] on the webpage and didn’t throw an error. It also shows these little boxes on the webpage when I run a browser window on the localhost.

The strange thing is I can reach the external cluster IP in the Microsoft Edge browser, and the lsof command in Linux will show my IP connected to the cockroach db address/port. I tried flushing my local DNS records and reaching the cluster on other devices and IP addresses as well. I figured something strange may be happening on the Linux server, I made sure there was not additional instances running as well.

Attempt on new, second Linux server:
I then tried to start an additional, new Linux VM to narrow it down to some sort of configuration issue on the main server. I followed the same tutorial here, with the only change being adding 0.0.0.0 in the create-node certificate and start commands instead of localhost. This is so that the cluster listens for external connections. But I can’t connect to this cluster node either, the connection attempt will timeout in DBeaver. The same ERR_INVALID_HTTP_RESPONSE is shown on the webpage if I try to access the second, new cluster address in a browser window, but the Admin UI is reachable as usual.

The cluster is running, I’ve started it with the command:
cockroach start --certs-dir=certs --listen-addr=0.0.0.0
The console shows it’s running properly.

Any idea what could be causing these issues? If it was some sort of configuration issue on the main Linux server, the second test server should still work…

#7

Hi Brian,

Here is some insight into those flags and how they work in the absence of others.

I believe ERR_INVALID_HTTP_RESPONSE is expected behavior when trying to access the listen-addr from the browser.

So if i understand this correctly, you are able to connect to your cluster via the console, and are able to connect to the cluster via the admin UI but not able to connect on DBeaver?

Let me know.

Thanks,
Matt

#8

Matt,

Correct, the cluster is running on my Linux VM. I can access it locally and see it running properly. I can access the Admin UI externally, but am not able to connect to the database externally from DBeaver.

DBeaver simply says that the connection attempt has timed out. Port 25565 is open and accessable externally. If this was a SSL certificate/authentication issue, my guess would be that DBeaver would display an authentication error of some sort.

#9

Brian,

Where are you getting port 25565 from?

Have you converted the client.root.key to client.root.pk8?

#10

My bad, port 26257 there was a typo in the previous post.

Yes, I followed the DBeaver tutorial to convert the client.root.key to client.root.pk8 using openSSL.

#11

No worries.

Can you provide me the full cockroach start commands you are using including the cluster initialization, and the cockroach cert commands too?

I think a good place to start with debugging this is to set it up in insecure mode to get the connections working properly first.

Do you recall what you were moving around when it stopped working?