Which method is best for starting a cluster?

Based on what I have been able to gather from the docs, there are two ways to start a cluster:

  1. First method is to start a group of nodes with join to a list of 3-5 nodes. Then running a one time init from a desktop. I have noticed this method is flaky when running start with join to a list of 6 nodes, and then running the init. Often 1 or 2 nodes out of 6 would not join the cluster. Wondering if this is because the required list of hosts must be 3-5, not more (or less)?

  2. Second method is to start 1 node without the join option, and then join all subsequent hosts to the first host (or any of the already joined hosts). This method seems to work without issues. Is this the preferred method for production?

Hi @fat0,

Yes, those are currently the 2 ways to start a cluster, but the first (start initial nodes with --join and then run one-time init command) is actually preferred for production. All of our production deployment tutorials take that approach.

The list of hosts in the --join flags can be more than 6. If a node isn’t joining, it’s likely due to some networking issue. If you haven’t already, please review the relevant deployment doc (linked above) for setup tips. If that does’t help, please share the commands you’re running and any responses you’re getting.

Thanks,
Jesse

Hi Jesse, those were exactly the docs I followed, in particular the “insecure” method and in AWS. After multiple attempts I finally gave up and went with method #2. I was never able to figure out why some nodes would not join the cluster upon init. I can try again and send the logs if that helps.

I just noticed this in the v2.0.2 Release Notes:
Fixed a scenario in which a node could deadlock while starting up. #24831

Will try upgrading and see if the issue goes away.

@fat0, I’d love to know if this problem went away after upgrading to 2.0.2? If not, please open a github issue with a description of the problem and the logs and one of our developers will look into it for you.

Best,
Jesse

Hi @jesse, is it possible to start 1 set of nodes with joins to each other, and another set of nodes with join to each other (they are not aware of the nodes in the first set), and then join these two clusters into one? Probably not but thought I’d ask anyway.

Hi @fat0,

I’m pretty sure it’s not possible to join two independent clusters into one, but I’d like confirmation from @bdarnell. Each node stores the unique ID of the cluster to which it belongs and will give you an error if you try to join it to a different cluster.

Can you shed more light on what you are looking to accomplish with such a scenario?

Jesse

No, it’s not possible to merge two clusters that were once independent.

Thanks, makes sense. The scenario has to do with setting up a multi region cluster and nodes in one region having to know about nodes about in the other one in order to bootstrap. For now they will have to query all the regions :slight_smile: