Cockroach on the Raspbery Pi 3 - 64 bit

The new Raspberry Pi 3 is a four core 64 bit machine with 1Gb memory. I have a few of them. I’m evaluating cockroachdb. Obviously the thing to do is make a cockroach cluster of Pi 3.

I checked the Cockroach on Raspberry Pi instructions here: https://www.cockroachlabs.com/blog/run-cockroachdb-on-a-raspberry-pi/

But that is getting a bit old and targeted at older 32 bit Pi with less memory. There are some hacks in there that I hope are not required for a 64 bit build on the Pi 3.

So, muddling along I installed a 64 bit Debian on my Pi 3 and managed to build a cockroach for it that seems to work.

The procedure is below. It would be great if any roach guys have comments on it. If it looks sensible the next step is a cluster of three Pi 3 built like that.

Install a 64 bit cockroachdb on your Raspberry Pi 3.

Install pi64, a 64 bit version of Debian for the Pi 3, to your Pi 3 using the instructions here: https://github.com/bamarni/pi64

When you have that done, on your Pi 3 do the following:

Be sure Debian is up to date

  $ sudo apt-get update
  $ sudo apt-get upgrade

Install git

 $ sudo apt-get install git

Install vim

 $ sudo apt-get install vim

Install wget

 $ sudo apt-get install wget

Install curl

 $ sudo apt-get install curl

Install make

 $ sudo apt-get install make

Install cmake

 $ sudo apt-get install cmake

Install gcc

 $ sudo apt-get install gcc

Install g++

 $ sudo apt-get install g++

Install the tinfo developer libraries:

 $ sudo apt-get install libtinfo-dev

Install node.js`. Not necessary but I want to test cockroach from node.js later.

  $ wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash
  $ source  /home/pi/.profile
  $ nvm install 9.3.0
  $ node -v

Install go for arm64

  $ wget https://storage.googleapis.com/golang/go1.9.2.linux-arm64.tar.gz
  $ sudo tar -C /usr/local -xzf go1.9.2.linux-arm64.tar.gz 

Add go to your PATH:

 $ sudo vim ~/.profile     # and add the line "export PATH=$PATH:/usr/local/go/bin"
 $ source ~/.profile
 $ echo $PATH
 $ go version

Install autoconf

  $ sudo apt-get install autoconf`

Install the yarn package manager

   $ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
   $ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
   $ sudo apt-get install apt-transport-https
   $ sudo apt-get update
   $ sudo apt-get install yarn

Make a nice big swap file, building cockroach will need it.

  $ sudo dd if=/dev/zero of=/swap bs=1024 count=1M
  $ sudo chmod 600 /swap 
  $ sudo mkswap /swap
  $ sudo swapon /swap

Build and install cockroachdb

  $ cd $GOPATH 
  # Or
  $ mkdir ~/go; cd ~/go  if you have no GOPATH

  $ go get -d github.com/cockroachdb/cockroach
  $ cd src/github.com/cockroachdb/cockroach
  $ make build -j1

That takes about an hour and half…

  $ ls -lh          # should see cockroach executable there

Install it on your PATH and check it runs:

  $ sudo cp cockroach /usr/local/bin/
  $ cockroach version
  $ cockroach start --insecure --host=0.0.0.0

You should now be able to browse to the cockroach dashboard on localhost:8080

If all is well it’s time to check the documentation at https://www.cockroachlabs.com/docs/stable/

4 Likes

Hi heater!

thank you so much for looking into it! One of my colleagues was already attempting some of these steps a month ago, but we are glad that you were able to summarize the steps in such a succinct format!

Would you be interested to publish these instructions and your results as a guest article on our blog?

Gosh. I would be honoured to appear on your blog.

Might take a little while, holidays going on and all.

I’ll have to run through that procedure again to be sure it’s accurate. Most of what is there is recovered from my command history.

Also, I managed to blow up my 3rd Pi 3 here so I can’t actually make a cluster at the moment.

It is kind of freaky though, having 3.5 Terabyte of storage for a database connected to a humble Pi.

I’ll be in touch.

Roach died.

So I left my new 64 bit roach on pi64 running doing nothing for a few hours. When I came back to it it was dead.

It was started with:

$ cockroach start --insecure --host-0.0.0.0 --background

It died with SIGILL : illegal instruction

The log file is attached.

illegal-instruction.pdf (119.7 KB)

It was built with go 1.9.2 and GCC 6.3.0

We’ve seen similar crashes on x86 when compiling with SSE support for processors that don’t support it. I don’t know what an analogy would be in the arm world, but perhaps @benesch has an idea.
Or perhaps @nathan would know if other people building for ARM had to jump through any hoops?

The equivalent SIMD architectures in ARM is called NEON.
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html

As far as I know NEON exists on the Pi3 and is useable if you have GCC 6 or up with compile flags -mfpu=neon

You’re building and running on the same Pi3 device, right? I think all the SIGILL errors we’ve seen in the past have resulted from building on one machine and copying the binary to another.

Can you try to collect a core dump following these instructions? ulimit -c unlimited before cockroach start, then after the crash gdb cockroach <corefile>, thread apply all backtrace full, and disassemble $pc,+32 (I’m not 100% sure the gdb instructions will work for a go binary, but at least once we get the core file and binary we can analyze them without waiting hours for a reproduction)

Yes, I built cockroach on the same Pi 3 and OS I’m running it on. I don’t like to mess with cross-compiling unless I really have to.

OK. It’s running now as per instructions. Let’s see what happens…

I’m not sure the TAGS option I used in my build does what I guessed it should:

$ make build -j1 TAGS=‘v1.1.3’

Produces a version that calls itself:

up-434-gbd7e5fbf7

Yeah, that’s for setting “build tags”, which is how Go allows for limited conditional compilation. If you want to build v1.1.3:

$ git checkout v1.1.3
$ make build PORTABLE=1

The PORTABLE=1 might solve some of your problems. No promises, though. Getting the core dump as @bdarnell detailed will give us a better sense of what went wrong.

*** variable ‘PORTABLE’ is not recognized by this Makefile. Stop

I think you might have misspelled PORTABLE :slight_smile:

Yes, I did. Here not there.

Can’t build version 1.1.3. I did this:

$ git checkout v1.1.3
$ make clean
$ make build -j1

And it fails with:

c++: error: unrecognized command line option ‘-msse3’

If we try to build v1.1.3 then I’d expect we’d run into https://github.com/cockroachdb/cockroach/issues/18319, which is what it looks like you’re seeing. On master, we shouldn’t have any issues building on aarch64.

We’re not aware of any issues when running on ARM at the moment. SSE support should be disabled and I don’t think any of our dependencies use NEON instructions. From the logs and stacktrace you posted, this seems like it may be coming from inside mallocgc (https://golang.org/src/runtime/slice.go line 147). I think our best bet is to follow the instructions @bdarnell posted above.

It may also be telling to see whether we can reproduce this in other ways. For instance, if there is an unsupported instruction in mallocgc then we should be able to find out pretty easily with a few synthetic go tests. For now, could you try running make test in the cockroach directory and check if all tests pass?

make test failed. cannot allocate memory.

I think I have to call it quits for the night.

I set up a 4Gb swap file on a USB hard drive and ran “make test -j1” again.

This time it spent ages building something, then proceeded to scroll millions of lines of what looked like test output to the screen.

Finally ended with:

Makefile:761: recipe for target ‘test’ failed

In the lines prior to that I see a lot of “ok” results interspersed with “?” results.

For example the last such line:

? github.com/cockroacdb/cockroac/pkg/util/uuid [no test files]

Sorry I can’t present all the output, it’s sitting an a virtual terminal screen.

I ran make test -j1 again. This time capturing the output to file.

I see a lot of fails in there.

The test output is available for download from here if anyone is interested:

https://fits-server.rsm-technologies.com/serial-tunnel/test.log.gz

That test output actually looks good for the most part. The two failures are:

--- FAIL: TestOfficializeAddr (0.24s)
        server_test.go:451: lookup pi64 on 192.168.8.1:53: no such host

which looks like some issue with your network setup and:

--- FAIL: TestConcurrentBatch (57.66s)
        rocksdb_test.go:532: write took 20.9s

which just demonstrates how slow the IO on the Raspberry PI is. The “?” results are harmless and just mean that there are no unit tests in certain packages. So it actually looks like the tests succeeded for all intents and purposes.

Have you tried building with PORTABLE=1 on master?

thanks for the instructions.

Building on pi64.

During build(v2.0-alpha.20180116-671-ge0307f658
), I needed to install go-bindata

after that, things worked great