Failing make build of 20.1.0 source on Ubuntu 20.04 / ARM64

I am attempting to compile the new 20.1.0 source, but I’m getting issues when doing the build, using this command:

time make buildoss 2>&1 | tee compile.txt`
  • I’m using buildoss to make a non-commercial executable; make build gets same results

Issue 1 - CMAKE VERSION REQUIREMENT

https://www.cockroachlabs.com/docs/stable/install-cockroachdb-linux.html

  • The above states that CMake Versions 3.81+ are known to work
  • but the newest cmake version available is 3.17.2 according to https://cmake.org/download/
  • Is documentation wrong?
  • My versions are:
Ubuntu 20.04 / aarch64
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
autoconf (GNU Autoconf) 2.69
GNU Make 4.2.1
cmake version 3.16.3
g++ (Ubuntu 9.3.0-10ubuntu2) 9.3.0
go version go1.13.10 linux/arm64
git version 2.25.1

Issue #2 - make buildoss FAILS TO BUILD

The build fails with this output:

make -C src/github.com/cockroachdb/cockroach buildoss
make[1]: Entering directory '/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach'
Running make with -j4
GOPATH set to /home/ubuntu/cockroach-v20.1.0
go install -v ./pkg/cmd/prereqs
runtime
fatal: not a git repository (or any of the parent directories): .git
runtime
# runtime
/usr/local/go/src/runtime/unaligned1.go:11:6: readUnaligned32 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:321:40
/usr/local/go/src/runtime/unaligned1.go:15:6: readUnaligned64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:329:40
# runtime
/usr/local/go/src/runtime/unaligned1.go:11:6: readUnaligned32 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:321:40
/usr/local/go/src/runtime/unaligned1.go:15:6: readUnaligned64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:329:40
make[1]: *** [Makefile:354: bin/.bootstrap] Error 2
make[1]: Leaving directory '/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach'
make: *** [Makefile:6: buildoss] Error 2

real    0m1.524s
user    0m2.714s
sys     0m1.402s

My previous attempts on the same machine building 19.2.6 did not get these errors (OK, it got different errors, but not these errors :slight_smile: )…

Any helpful hints appreciated. Thanks!

Rob Bohn

P.S. Is this not the correct venue for questions like these?

Hey Rob, I’m not positive but I think this might point to problems with your Go installation. Note that the errors are coming from failing to compile the Go runtime, or something like that. Is your GOPATH set up correctly? Another thing you could try is running without make -C, while having cd’d into the source directory (which should be properly ensconced in $GOPATH/src/github.com/cockroachdb/cockroach).

Thank you for your quick response.

You are correct (my bad/thank you) - I overlaid the newer go1.13.10 on top of a 1.12.X version. I’m not familiar with the inner workings of GO, so that error message was not leading me anywhere useful.

To fix that, I removed the old go files and reloaded the new go version from the tar file again, and checked my path for correctness.

I’m now trying my make command again after a make clean:

time make buildoss TAGS='stdmalloc' 2>&1 | tee compile.txt

44 minutes - not too bad for a Rasberry Pi 4 with 4GB and an SSD

I do see three issues/warnings:

1. A warning that occurs in both 19.2.6 and 20.1.0 compiles.

    [ 85%] Building CXX object CMakeFiles/libprotobuf.dir/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc.o
    /home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc: In static member function 'static google::protobuf::util::Status google::protobuf::util::converter::ProtoStreamObjectSource::RenderDuration(const google::protobuf::util::converter::ProtoStreamObjectSource*, const google::protobuf::Type&, google::protobuf::StringPiece, google::protobuf::util::converter::ObjectWriter*)':
    /home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc:386:13: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'google::protobuf::int64' {aka 'long int'} [-Wformat=]
      386 |       "%s%lld%ss", sign.c_str(), seconds,
          |          ~~~^                    ~~~~~~~
          |             |                    |
          |             long long int        google::protobuf::int64 {aka long int}
          |          %ld

2. I am getting one protobuf ERROR that I was getting in my 19.2.6 compiles, such as:

[87%] Building CXX object CMakeFiles/libprotobuf.dir/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/type_info.cc.o
    /home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc: In static member function 'static google::protobuf::util::Status google::protobuf::util::converter::ProtoStreamObjectSource::RenderDuration(const google::protobuf::util::converter::ProtoStreamObjectSource*, const google::protobuf::Type&, google::protobuf::StringPiece, google::protobuf::util::converter::ObjectWriter*)':
    /home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc:386:13: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'google::protobuf::int64' {aka 'long int'} [-Wformat=]
      386 |       "%s%lld%ss", sign.c_str(), seconds,
          |          ~~~^                    ~~~~~~~
          |             |                    |
          |             long long int        google::protobuf::int64 {aka long int}
          |          %ld
    [ 90%] Building CXX object CMakeFiles/libprotobuf.dir/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/type_info_test_helper.cc.o
    [ 90%] Building CXX object CMakeFiles/libprotobuf.dir/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/utility.cc.o

3. I am getting this defined but not used warning that I was also getting in my 19.2.6 compiles (this appears to be a known issue for ARM64):

[ 62%] Building CXX object CMakeFiles/rocksdb.dir/util/delete_scheduler.cc.o
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:426:13: warning: 'bool rocksdb::crc32c::isPCLMULQDQ()' defined but not used [-Wunused-function]
  426 | static bool isPCLMULQDQ() {
      |             ^~~~~~~~~~~
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:412:13: warning: 'bool rocksdb::crc32c::isSSE42()' defined but not used [-Wunused-function]
  412 | static bool isSSE42() {
      |             ^~~~~~~
[ 62%] Building CXX object CMakeFiles/rocksdb.dir/util/dynamic_bloom.cc.o

So, the make now gets a LOT further than my 19.2.6 attempts.

BUT it still fails with this error, most likely due to the above:

make: *** [Makefile:6: buildoss] Error 2

Rob Bohn

Nice. It can be kind of hard to track down root cause of build errors because our Makefile runs in parallel by default. Could you try rerunning (no need to make clean first) with make -j1 buildoss or the like? I suspect that will do a better job of putting the blocking error front and center.

Hmmm…

  • Git error? git is installed but not used by me on this machine. git version 2.25.1 then again this might just be from the section determining git worktree
  • Also seems to have a -j4 buried somewhere.
ubuntu@picr01:~/cockroach-v20.1.0$ time make -j1 buildoss TAGS='stdmalloc' 2>&1 | tee compile.txt
make -C src/github.com/cockroachdb/cockroach buildoss
make[1]: Entering directory '/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach'
Running make with -j4
GOPATH set to /home/ubuntu/cockroach-v20.1.0
fatal: not a git repository (or any of the parent directories): .git
GOFLAGS= go build -o cockroachoss -v  -tags 'stdmalloc gss make aarch64_linux_gnu_stdmalloc' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v20.1.0" -X "github.com/cockroachdb/cockroach/pkg/build.rev=9d456b9ec82cbf9a740a092c0d9f56da48779689" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=aarch64-linux-gnu" -X "github.com/cockroachdb/cockroach/pkg/build.channel=source-archive"  -X "github.com/cockroachdb/cockroach/pkg/build.utcTime=2020/05/14 17:56:09"' ./pkg/cmd/cockroach-oss
github.com/cockroachdb/cockroach/pkg/storage
# github.com/cockroachdb/cockroach/pkg/storage
/usr/bin/ld: /home/ubuntu/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb_stdmalloc/librocksdb.a(crc32c.cc.o): in function `rocksdb::crc32c::ExtendARMImpl(unsigned int, char const*, unsigned long)':
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:476: undefined reference to `crc32c_arm64(unsigned int, unsigned char const*, unsigned int)'
/usr/bin/ld: /home/ubuntu/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb_stdmalloc/librocksdb.a(crc32c.cc.o): in function `rocksdb::crc32c::IsFastCrc32Supported[abi:cxx11]()':
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:495: undefined reference to `crc32c_runtime_check()'
/usr/bin/ld: /home/ubuntu/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb_stdmalloc/librocksdb.a(crc32c.cc.o): in function `_GLOBAL__sub_I__ZN7rocksdb6crc32c13ExtendARMImplEjPKcm':
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:1227: undefined reference to `crc32c_runtime_check()'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:911: cockroachoss] Error 2
make[1]: Leaving directory '/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach'
make: *** [Makefile:6: buildoss] Error 2

real    0m12.918s
user    0m13.407s
sys     0m5.141s

I did do a make help as mentioned in the Makefile, but it gets an error:

ubuntu@picr01:~/cockroach-v20.1.0$ make help
make: *** No rule to make target 'help'.  Stop.

Looks like new goodies in that Makefile - someone was working it!

Rob

You have to run the make -j1 buildoss or make help from src/github.com/cockroachdb/cockroach. That’s where the main Makefile is.

It can’t find the GOPATH when I go there. I’ll have to put this aside and look further later this evening…

ubuntu@picr01:~/cockroach-v20.1.0$ cd /home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach
ubuntu@picr01:~/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach$ echo $PATH
/home/ubuntu/.vscode-server/bin/d69a79b73808559a91206d73d7717ff5f798f23c/bin:/home/ubuntu/.vscode-server/bin/d69a79b73808559a91206d73d7717ff5f798f23c/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin
ubuntu@picr01:~/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach$ echo $GOPATH
/home/ubuntu/go
ubuntu@picr01:~/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach$ time make -j1 buildoss TAGS='stdmalloc' 2>&1 | tee compile.txt
Makefile:240: *** Current directory "/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach" is not within GOPATH "/home/ubuntu/go".  Stop.

real    0m0.049s
user    0m0.003s
sys     0m0.057s
ubuntu@picr01:~/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach$ 

You have to put your Go source in your GOPATH to let Go work properly. This would mean moving your src/github.com/cockroachdb/cockroach tree to your $GOPATH. Have a look at the Go docs for more information on this: https://golang.org/doc/gopath_code.html#Organization

I think the GOPATH is a canary, a false lead. And -j1 may be as well.

The delivered Makefile in ~/cockroach-v20.1.0 already sets the GOPATH and then calls the Makefile in src/github.com/cockroachdb/cockroach. I have always run my makes in ~/cockroach-v20.1.0.

Going into the src/github.com/cockroachdb/cockroach directory and then doing a make from there means GOPATH would not be set, and moving source around means I would no longer match the official delivered source from Cockroach labs.

So, I instead modified the top-level Makefile to have a -j1, changing the line

	$(MAKE) -C src/github.com/cockroachdb/cockroach $@

to be

	$(MAKE) -j1 -C src/github.com/cockroachdb/cockroach $@

I also did a change to the src/github.com/cockroachdb/cockroach/Makefile, since lines 169-174 determine the -j number based on number of CPUs; I changed line 172 from

MAKEFLAGS += -j$(NCPUS)

to be

MAKEFLAGS += -j1 ## MAKEFLAGS += -j$(NCPUS)

Then I did my normal make in the ~/cockroach-v20.1.0 folder

time make -j1 buildoss TAGS='stdmalloc' 2>&1 | tee compile.txt

with this result

make -j1 -C src/github.com/cockroachdb/cockroach buildoss
make[1]: Entering directory '/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach'
GOPATH set to /home/ubuntu/cockroach-v20.1.0
fatal: not a git repository (or any of the parent directories): .git
GOFLAGS= go build -o cockroachoss -v  -tags 'stdmalloc gss make aarch64_linux_gnu_stdmalloc' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v20.1.0" -X "github.com/cockroachdb/cockroach/pkg/build.rev=9d456b9ec82cbf9a740a092c0d9f56da48779689" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=aarch64-linux-gnu" -X "github.com/cockroachdb/cockroach/pkg/build.channel=source-archive"  -X "github.com/cockroachdb/cockroach/pkg/build.utcTime=2020/05/15 20:12:59"' ./pkg/cmd/cockroach-oss
github.com/cockroachdb/cockroach/pkg/storage
# github.com/cockroachdb/cockroach/pkg/storage
/usr/bin/ld: /home/ubuntu/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb_stdmalloc/librocksdb.a(crc32c.cc.o): in function `rocksdb::crc32c::ExtendARMImpl(unsigned int, char const*, unsigned long)':
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:476: undefined reference to `crc32c_arm64(unsigned int, unsigned char const*, unsigned int)'
/usr/bin/ld: /home/ubuntu/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb_stdmalloc/librocksdb.a(crc32c.cc.o): in function `rocksdb::crc32c::IsFastCrc32Supported[abi:cxx11]()':
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:495: undefined reference to `crc32c_runtime_check()'
/usr/bin/ld: /home/ubuntu/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb_stdmalloc/librocksdb.a(crc32c.cc.o): in function `_GLOBAL__sub_I__ZN7rocksdb6crc32c13ExtendARMImplEjPKcm':
/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:1227: undefined reference to `crc32c_runtime_check()'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:911: cockroachoss] Error 2
make[1]: Leaving directory '/home/ubuntu/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach'
make: *** [Makefile:6: buildoss] Error 2

Since it is getting undefined reference errors, I’m going to do a make clean and try make build again.

Rob

@robbohn were you able to succeed in getting the build to finish? I’m also having the same problem of

undefined reference to `crc32c_arm64(unsigned int, unsigned char const*, unsigned int)'

No; the make clean did not resolve the issue. I still get the undefined references. Are you on Ubuntu 20.04?

I’m also getting this error for a long long:

[ 87%] Building CXX object CMakeFiles/libprotobuf.dir/home/ubuntu/cockroach-v19.2.6/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/type_info.cc.o
/home/ubuntu/cockroach-v19.2.6/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc: 
In static member function 
'static google::protobuf::util::Status google::protobuf::util::converter::ProtoStreamObjectSource::RenderDuration(const google::protobuf::util::converter::ProtoStreamObjectSource*, const google::protobuf::Type&, google::protobuf::StringPiece, google::protobuf::util::converter::ObjectWriter*)':
/home/ubuntu/cockroach-v19.2.6/src/github.com/cockroachdb/cockroach/c-deps/protobuf/src/google/protobuf/util/internal/protostream_objectsource.cc:386:13: 
warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'google::protobuf::int64' {aka 'long int'} 
[-Wformat=]
  386 |       "%s%lld%ss", sign.c_str(), seconds,
      |          ~~~^                    ~~~~~~~
      |             |                    |
      |             long long int        google::protobuf::int64 {aka long int}
      |          %ld

I’ve set it aside for awhile.

Ubuntu 18.04, I don’t get the error with the protobuf long long - just the complaint about ```
undefined reference to `crc32c_arm64

1 Like

For the above, I am seeing RocksDB has a github issue with crc32c_arm64

RocksDB also had an an issue with the errors I was getting concerning PCLMULQDQ, SSE42

github
github
possibly helpful

Both of these are from 2019, but I don’t know if the fixes are in the RocksDB version in the Cockroach source.

My Raspberry Pi 4s are remote and accidentally turned off due to a power blip, so I can’t get on them yet to see where the RocksDB source is contained in the CockroachDB source to check versions.

Rob

I switched over from armbian to a ubuntu 20.04 VM with an emulated cortex-53 and I think I am getting a similar error as you do, right?

github.com/cockroachdb/cockroach/pkg/storage

/usr/bin/ld: /home/benutzer/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb
/librocksdb.a(crc32c.cc.o): in function rocksdb::crc32c::ExtendARMImpl(unsigned int, char const*, unsigned long)': /home/benutzer/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:476: undefined reference to crc32c_arm64(unsigned int, unsigned char const*, unsigned int)’
/usr/bin/ld: /home/benutzer/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb/librocksdb.a(crc32c.cc.o): in function rocksdb::crc32c::IsFastCrc32Supported[abi:cxx11]()': /home/benutzer/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:495: undefined reference to crc32c_runtime_check()’
/usr/bin/ld: /home/benutzer/cockroach-v20.1.0/native/aarch64-linux-gnu/rocksdb/librocksdb.a(crc32c.cc.o): in function _GLOBAL__sub_I__ZN7rocksdb6crc32c13ExtendARMImplEjPKcm': /home/benutzer/cockroach-v20.1.0/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/util/crc32c.cc:1227: undefined reference to crc32c_runtime_check()’
collect2: error: ld returned 1 exit status

Yes, those are the same errors I’m getting. I’ve not yet had a chance to investigate the versions of rocksdb and crc32c versus those used by cockroach. I think it is due to #ifndef and/or #ifdef bypassing needed code since arm64 has hardware crc instructions.

Here’s my PR, I hope it gets approved and merged upstream: https://github.com/cockroachdb/rocksdb/pull/85

This will allow cockroachdb’s rocksdb to build without issue on Arm64

After I applied the linked PR cockroachdb built! I’ll make a PR against the rocksdb fork

Awesome! I’ve had other projects pull me away from this, so I’m glad to see you are making progress!