Tesla CCS candidate check - Round 7

Denver Gingerich denver at sfconservancy.org
Fri May 18 17:54:58 UTC 2018

These are the results of the CCS check we did on the Tesla candidate 
source code made available at https://github.com/teslamotors/buildroot 
in May 2018.  This is the seventh source candidate we have received from 

In this check, per the GPL's requirements, we are seeking to both (a) 
confirm that we are able to build binaries from the source that 
correspond to the binaries Tesla ships and (b) verify that we are able 
to replace the binaries Tesla ships with our own (possibly modified) 
binaries, i.e. using installation instructions that Tesla might provide.

Below are the binaries Tesla ships in its Model S that we are comparing 
to built binaries.  These comparison binaries were provided to us by a 
Model S car owner who was one of the many to report this violation to us:
* firmware_2.12.126 - a filesystem provided by Parrot
* 2.15.50.img.bz2 - a filesystem provided by Nvidia (presumed to match 
the now-named Tesla Autopilot 2 platform - we'll refer to this as the 
"main board" system in this report)

First, here are the details on how we downloaded and prepped the source 
candidate Tesla provided to us:

We obtained the source candidate using the following steps (the checkout 
step is necessary to ensure the reader is using the same version of the 
code that we did - it was the latest commit at the time of this writing):

$ git clone https://github.com/teslamotors/buildroot.git
$ cd buildroot
$ git checkout fb9fa1f65c0ffb50695d620d74249b3ca10f7583

Note that the above buildroot repository automatically retrieves via 
reference the repository at https://github.com/teslamotors/linux , which 
is analyzed below in the "linux (part 1)" section.

== Sources for one of the Linux installs found in vehicle binaries 
appear to be completely missing ==

It appears that this buildroot repository contains only code intended to 
match the main board binaries.  Email to users seen at 
specifically mentioned Nvidia in relation to this release, and that 
"Work is underway on preparing sources in other areas as well".  While 
it does not appear that this repository contains any code that 
corresponds to the Parrot binaries, we expect Tesla will provide it 
later.  Tesla has previously privately provided to us two separate build 
trees in its past source candidates, one for "Nvidia" and one for 
"Parrot" (each containing its own copy of Linux and associated userspace 

When provided, we expect these sources to contain a separately 
configured, different copy of Linux, and include the source code for the 
88w8688_uap.ko and 88w8688_uap_mlan.ko files that appear in the Tesla 
Model S.

== Check summary of sources provided ==

We were able to generate some binary files after making several changes 
to the build scripts.  We have already publicly provided pull requests, 
since the availablity of these repositories on GitHub indicate that is 
an appropriate method for receiving improvements for compliance of these 
sources.  These pull requests are at 
https://github.com/teslamotors/buildroot/pulls , and we hope to provide 
more as this work continues.

However, even with these changes, the candidate does not appear to be 
complete because it is both (a) missing installation instructions and 
(b) missing a few details on how to build the initramfs.

It appears that the latter could be resolved with a few extra notes in 
the README.Tesla file (see below), especially pertaining to the 
ap-hw2i_defconfig file (see the "initramfs" section below).  This could 
even be provided by someone more familiar with buildroot, and may not 
require a Tesla employee.

However, the former (installation instructions) can likely only be 
provided by Tesla, since the methods for installing software onto Tesla 
hardware is likely highly specific to that particular hardware.

Also, we would recommend that Tesla host all the files required by its 
buildroot repository rather than having buildroot download them from 
third-party locations, as those sources may become unavailable in the 
future, which would cause further, future compliance problems for Tesla.

The main board binaries we checked against do not match what was built 
by the source repository (see the last section below).  So there need to 
be corrections and/or additions made to future candidates to ensure the 
source does indeed correspond to the binaries.

== Initial build process ==

We found a README file in the root directory of the repository, as well 
as a README.Tesla file.  It was clear that the README file contained 
generic buildroot instructions so we then read the README.Tesla file, 
which stated:

"Current branch is buildroot-ap, it contains default configs to build 
for the Tesla Autopilot 2 platforms:

Since we found we were on the builtroot-ap branch already (prior to the 
"git checkout", "git branch -a" showed the current branch to be 
builtroot-ap), we then looked for the above _defconfig files.  We found 
them in the "configs" directory, and then copied 
configs/ap-hw25_defconfig to .config so that we could use that 
configuration for the build process.  (Note that the ap-hw2_defconfig 
and ap-hw25_defconfig configuration files differ only in which tag they 
checkout from the kernel tree.)

Note that the README files did not specify any particular operating 
system that one should use to build the project, so we used the 
operating system we had on-hand, which was an amd64 version of Debian 9.

When we then ran "make", we were prompted with an ncurses-rendered 
screen entitled "Buildroot 2016.05-00004-gfb9fa1f65 Configuration" with 
some configuration options.  Since we wanted to use the defaults from 
ap-hw25_defconfig, we chose the "Exit" option and hit Enter (answering 
Yes to "Do you wish to save your new configuration?").

Since that "make" invocation had completed, we ran "make" again in order 
to start building the project using the configuration we just saved. 
This did build a few packages without issue, but we then ran into a 
number of packages that required fixes in order to build correctly:

== Specific build/package issues ==

=== host-gcc-final-5.3.0 ===

The first time we attempted the build, we received this error:

build: In file included from ../../gcc/cp/except.c:1023:0:
build: cfns.gperf: In function 'const char* libc_name_p(const char*, 
unsigned int)':
build: cfns.gperf:101:1: error: 'const char* libc_name_p(const char*, 
unsigned int)' redeclared inline with 'gnu_inline' attribute
build: cfns.gperf:26:14: note: 'const char* libc_name_p(const char*, 
unsigned int)' previously declared here

A bit of research showed that this error could be solved by the 
following patch:


We first removed the ChangeLog portion of the patch since it was not 
needed to correct the functional issue, then applied the patch, which 
applied cleanly (inside both the output/build/host-gcc-final-5.3.0 and 
output/build/host-gcc-initial-5.3.0 directories).

=== metis-5.1.0 ===

The build continued after the above patches, but was once again halted:

1: [100%] Built target cmpfillin
1: make[3]: Leaving directory 
1: .../buildroot/output/host/usr/bin/cmake -E cmake_progress_start 
.../buildroot/output/build/metis-5.1.0/build/Linux-x86_64/CMakeFiles 0
1: make[2]: *** No rule to make target 'w'.  Stop.
1: make[2]: Leaving directory 

We found after some research that the following patch solved it:

--- a/output/build/metis-5.1.0/Makefile	2013-03-30 12:24:45.000000000 -0400
+++ b/output/build/metis-5.1.0/Makefile	2018-05-17 22:02:00.965427439 -0400
@@ -64,7 +64,7 @@
  	@if [ ! -f $(BUILDDIR)/Makefile ]; then \
  		more BUILD.txt; \
  	else \
-	  	make -C $(BUILDDIR) $@ $(MAKEFLAGS); \


=== initramfs ===

With the above patch, the build continued, but was then halted again:

 >>> linux-initramfs 1 Syncing from source dir 
 >>> linux-initramfs 1 Configuring
 >>> linux-initramfs 1 Building
cp .../buildroot/output/build/linux-initramfs-1/../../Makefile 
cp: cannot stat 
'.../buildroot/output/build/linux-initramfs-1/../../Makefile': No such 
file or directory
package/pkg-generic.mk:195: recipe for target 
'.../buildroot/output/build/linux-initramfs-1/.stamp_built' failed

This was somewhat expected, as README.Tesla included this text:

"There's also a separate defconfig used to build an initramfs used with 
the kernel
that the above configs use:

However, when we tried to build the buildroot project using 
ap-hw2i_defconfig as the .config, we received the following error:

   Makefile:472: *** util-linux is in the dependency chain of lvm2 that 
has added it to its _DEPENDENCIES variable without selecting it or 
depending on it from Config.in.  Stop.

We were unable to quickly determine a resolution to this problem, so we 
omitted the initramfs build by running "touch 

=== protobuf3 ===

After removing the initramfs package from the build per above, we ran 
into this error next time we ran make:

buildroot-dl-wrapper: 2018-05-18 07:36:57 (7.55 MB/s) - 
saved [4051540]
buildroot-dl-wrapper: ERROR: protobuf3-v3.1.0.tar.gz has wrong sha256 hash:
buildroot-dl-wrapper: ERROR: expected: 
buildroot-dl-wrapper: ERROR: got     : 
buildroot-dl-wrapper: ERROR: Incomplete download, or man-in-the-middle 
(MITM) attack
buildroot-dl-wrapper: exit: 1 [1s elapsed, 0 pending]

It appeared from our subsequent research that this issue was benign, so 
we edited the hash in package/protobuf3/protobuf3.hash to match the 
"got" hash above and then re-ran "make".

=== linux (part 1) ===

This time the build failed with the following:

buildroot-dl-wrapper: PWD: .../buildroot
buildroot-dl-wrapper: Starting: ['support/download/dl-wrapper', '-b', 
'git', '-o', 
'--', 'git at github.com:teslamotors/linux.git', 
buildroot-dl-wrapper: Doing shallow clone
buildroot-dl-wrapper: Cloning into 
buildroot-dl-wrapper: fatal: Could not read from remote repository.
buildroot-dl-wrapper: Please make sure you have the correct access rights
buildroot-dl-wrapper: and the repository exists.

It seemed fairly clear from the above that the path to the GitHub URL 
was not correct for someone downloading it without a GitHub account (due 
to the "access rights" comment) so we changed the .config file as follows:

-BR2_LINUX_KERNEL_CUSTOM_REPO_URL="git at github.com:teslamotors/linux.git"

=== linux (part 2) ===

We then ran into this:

./scripts/gen_initramfs_list.sh: Cannot open 
usr/Makefile:73: recipe for target 'usr/initramfs_data.cpio.gz' failed
linux-internal-055213a7808f31ee04299c2d119b15a080b631f6+Image: make[2]: 
*** [usr/initramfs_data.cpio.gz] Error 1
Makefile:963: recipe for target 'usr' failed

Per the initramfs issues above, this is somewhat expected, so we just 
ran "touch output/images/initramfs.cpio.xz" and continued (see above 
"initramfs" section for details).

== Final result and missing installation instructions ==

After the above change was made, the build finished, but we weren't able 
to find any installation instructions.  We did find a few files in the 
output/images directory, specifically:

* Image
* rootfs.squashfs

The above files appeared to contain the results of the Linux build (the 
strings of the Image file included "Linux version 3.18.69-rt75 (...) 
(gcc version 5.3.0 (Buildroot 2016.05-00004-gfb9fa1f65-dirty) ) #1 SMP 
PREEMPT RT ...") and the rest of the build (the rootfs.squashfs 
contained BusyBox and other binaries).

While it is possible we could install the above files onto Tesla 
hardware with some guessing, it would be very difficult to do, and since 
the GPL requires installation instructions anyway, this omission does 
still require resolution by someone at Tesla with adequate expertise.

== Comparison with Model S binaries ==

We compared the binaries built above with those we found in the binaries 
shipped on the Tesla Model S and discovered that they did not match.  In 
particular, we found the following issues:

* BusyBox v1.24.2 was provided via the buildroot project
* BusyBox v1.19.2 (built 2016-02-23) was shipped on the Model S

So we don't appear to have a source candidate that correponds to files 
shipped on the Model S.

We chose not to compare other binaries inside the 2.15.50.img.bz2 binary 
file (for the main board system) since the BusyBox version was already 
so different.  In order for the source candidate to be compliant, the 
binaries that it builds (including but not limited to BusyBox binaries) 
must correspond to the binaries that Tesla ships.

We do note that Tesla has indicated these sources are for their 2018.12 
build.  We are aware that we may be comparing binaries from an earlier 
build, and that explains the mismatches.  We ask Tesla to indicate more 
comprehensively their plans for compliance on past distributions of 
older versions of the software.  We are willing to help figure out what 
compliance paths are feasible, and on our part may be willing to forgive 
past violations for older versions that may never resolved, provided 
that Tesla can attest that distribution of those older binaries has 
definitely ceased.

(For example, we're particularly concerned about what binaries may be in 
the physical cars that have not received online updates yet.  Compliance 
for those binaries, which as far as we can tell would match the binaries 
we have in our possesion, is an important issue.)

Denver Gingerich
Software Freedom Conservancy

More information about the ccs-review mailing list