From denver at sfconservancy.org Fri May 18 17:54:58 2018 From: denver at sfconservancy.org (Denver Gingerich) Date: Fri, 18 May 2018 13:54:58 -0400 Subject: Tesla CCS candidate check - Round 7 Message-ID: These are the results of the CCS check we did on the Tesla candidate source code made available at https://github.com/teslamotors/buildroot in May 2018. This is the seventh source candidate we have received from Tesla. In this check, per the GPL's requirements, we are seeking to both (a) confirm that we are able to build binaries from the source that correspond to the binaries Tesla ships and (b) verify that we are able to replace the binaries Tesla ships with our own (possibly modified) binaries, i.e. using installation instructions that Tesla might provide. Below are the binaries Tesla ships in its Model S that we are comparing to built binaries. These comparison binaries were provided to us by a Model S car owner who was one of the many to report this violation to us: * firmware_2.12.126 - a filesystem provided by Parrot * 2.15.50.img.bz2 - a filesystem provided by Nvidia (presumed to match the now-named Tesla Autopilot 2 platform - we'll refer to this as the "main board" system in this report) First, here are the details on how we downloaded and prepped the source candidate Tesla provided to us: We obtained the source candidate using the following steps (the checkout step is necessary to ensure the reader is using the same version of the code that we did - it was the latest commit at the time of this writing): $ git clone https://github.com/teslamotors/buildroot.git $ cd buildroot $ git checkout fb9fa1f65c0ffb50695d620d74249b3ca10f7583 Note that the above buildroot repository automatically retrieves via reference the repository at https://github.com/teslamotors/linux , which is analyzed below in the "linux (part 1)" section. == Sources for one of the Linux installs found in vehicle binaries appear to be completely missing == It appears that this buildroot repository contains only code intended to match the main board binaries. Email to users seen at https://teslamotorsclub.com/tmc/threads/tesla-releases-some-gpl-opensource-bits.115545/ specifically mentioned Nvidia in relation to this release, and that "Work is underway on preparing sources in other areas as well". While it does not appear that this repository contains any code that corresponds to the Parrot binaries, we expect Tesla will provide it later. Tesla has previously privately provided to us two separate build trees in its past source candidates, one for "Nvidia" and one for "Parrot" (each containing its own copy of Linux and associated userspace tools). When provided, we expect these sources to contain a separately configured, different copy of Linux, and include the source code for the 88w8688_uap.ko and 88w8688_uap_mlan.ko files that appear in the Tesla Model S. == Check summary of sources provided == We were able to generate some binary files after making several changes to the build scripts. We have already publicly provided pull requests, since the availablity of these repositories on GitHub indicate that is an appropriate method for receiving improvements for compliance of these sources. These pull requests are at https://github.com/teslamotors/buildroot/pulls , and we hope to provide more as this work continues. However, even with these changes, the candidate does not appear to be complete because it is both (a) missing installation instructions and (b) missing a few details on how to build the initramfs. It appears that the latter could be resolved with a few extra notes in the README.Tesla file (see below), especially pertaining to the ap-hw2i_defconfig file (see the "initramfs" section below). This could even be provided by someone more familiar with buildroot, and may not require a Tesla employee. However, the former (installation instructions) can likely only be provided by Tesla, since the methods for installing software onto Tesla hardware is likely highly specific to that particular hardware. Also, we would recommend that Tesla host all the files required by its buildroot repository rather than having buildroot download them from third-party locations, as those sources may become unavailable in the future, which would cause further, future compliance problems for Tesla. The main board binaries we checked against do not match what was built by the source repository (see the last section below). So there need to be corrections and/or additions made to future candidates to ensure the source does indeed correspond to the binaries. == Initial build process == We found a README file in the root directory of the repository, as well as a README.Tesla file. It was clear that the README file contained generic buildroot instructions so we then read the README.Tesla file, which stated: "Current branch is buildroot-ap, it contains default configs to build packages for the Tesla Autopilot 2 platforms: ap-hw2_defconfig ap-hw25_defconfig" Since we found we were on the builtroot-ap branch already (prior to the "git checkout", "git branch -a" showed the current branch to be builtroot-ap), we then looked for the above _defconfig files. We found them in the "configs" directory, and then copied configs/ap-hw25_defconfig to .config so that we could use that configuration for the build process. (Note that the ap-hw2_defconfig and ap-hw25_defconfig configuration files differ only in which tag they checkout from the kernel tree.) Note that the README files did not specify any particular operating system that one should use to build the project, so we used the operating system we had on-hand, which was an amd64 version of Debian 9. When we then ran "make", we were prompted with an ncurses-rendered screen entitled "Buildroot 2016.05-00004-gfb9fa1f65 Configuration" with some configuration options. Since we wanted to use the defaults from ap-hw25_defconfig, we chose the "Exit" option and hit Enter (answering Yes to "Do you wish to save your new configuration?"). Since that "make" invocation had completed, we ran "make" again in order to start building the project using the configuration we just saved. This did build a few packages without issue, but we then ran into a number of packages that required fixes in order to build correctly: == Specific build/package issues == === host-gcc-final-5.3.0 === The first time we attempted the build, we received this error: build: In file included from ../../gcc/cp/except.c:1023:0: build: cfns.gperf: In function 'const char* libc_name_p(const char*, unsigned int)': build: cfns.gperf:101:1: error: 'const char* libc_name_p(const char*, unsigned int)' redeclared inline with 'gnu_inline' attribute build: cfns.gperf:26:14: note: 'const char* libc_name_p(const char*, unsigned int)' previously declared here A bit of research showed that this error could be solved by the following patch: https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=ec1cc0263f156f70693a62cf17b254a0029f4852 We first removed the ChangeLog portion of the patch since it was not needed to correct the functional issue, then applied the patch, which applied cleanly (inside both the output/build/host-gcc-final-5.3.0 and output/build/host-gcc-initial-5.3.0 directories). === metis-5.1.0 === The build continued after the above patches, but was once again halted: 1: [100%] Built target cmpfillin 1: make[3]: Leaving directory '.../buildroot/output/build/metis-5.1.0/build/Linux-x86_64' 1: .../buildroot/output/host/usr/bin/cmake -E cmake_progress_start .../buildroot/output/build/metis-5.1.0/build/Linux-x86_64/CMakeFiles 0 1: make[2]: *** No rule to make target 'w'. Stop. 1: make[2]: Leaving directory '.../buildroot/output/build/metis-5.1.0/build/Linux-x86_64' We found after some research that the following patch solved it: --- a/output/build/metis-5.1.0/Makefile 2013-03-30 12:24:45.000000000 -0400 +++ b/output/build/metis-5.1.0/Makefile 2018-05-17 22:02:00.965427439 -0400 @@ -64,7 +64,7 @@ @if [ ! -f $(BUILDDIR)/Makefile ]; then \ more BUILD.txt; \ else \ - make -C $(BUILDDIR) $@ $(MAKEFLAGS); \ + make -C $(BUILDDIR) $@ MAKEFLAGS=$(MAKEFLAGS); \ fi uninstall: === initramfs === With the above patch, the build continued, but was then halted again: >>> linux-initramfs 1 Syncing from source dir .../buildroot/package/linux-initramfs >>> linux-initramfs 1 Configuring >>> linux-initramfs 1 Building cp .../buildroot/output/build/linux-initramfs-1/../../Makefile .../buildroot/output/build/linux-initramfs-1/output cp: cannot stat '.../buildroot/output/build/linux-initramfs-1/../../Makefile': No such file or directory package/pkg-generic.mk:195: recipe for target '.../buildroot/output/build/linux-initramfs-1/.stamp_built' failed This was somewhat expected, as README.Tesla included this text: "There's also a separate defconfig used to build an initramfs used with the kernel that the above configs use: ap-hw2i_defconfig" However, when we tried to build the buildroot project using ap-hw2i_defconfig as the .config, we received the following error: Makefile:472: *** util-linux is in the dependency chain of lvm2 that has added it to its _DEPENDENCIES variable without selecting it or depending on it from Config.in. Stop. We were unable to quickly determine a resolution to this problem, so we omitted the initramfs build by running "touch output/build/linux-initramfs-1/.stamp_built". === protobuf3 === After removing the initramfs package from the build per above, we ran into this error next time we ran make: buildroot-dl-wrapper: 2018-05-18 07:36:57 (7.55 MB/s) - '.../buildroot/output/build/.protobuf3-v3.1.0.tar.gz.L3ywCw/output' saved [4051540] buildroot-dl-wrapper: ERROR: protobuf3-v3.1.0.tar.gz has wrong sha256 hash: buildroot-dl-wrapper: ERROR: expected: 0a0ae63cbffc274efb573bdde9a253e3f32e458c41261df51c5dbc5ad541e8f7 buildroot-dl-wrapper: ERROR: got : fb2a314f4be897491bb2446697be693d489af645cb0e165a85e7e64e07eb134d buildroot-dl-wrapper: ERROR: Incomplete download, or man-in-the-middle (MITM) attack buildroot-dl-wrapper: exit: 1 [1s elapsed, 0 pending] It appeared from our subsequent research that this issue was benign, so we edited the hash in package/protobuf3/protobuf3.hash to match the "got" hash above and then re-ran "make". === linux (part 1) === This time the build failed with the following: buildroot-dl-wrapper: PWD: .../buildroot buildroot-dl-wrapper: Starting: ['support/download/dl-wrapper', '-b', 'git', '-o', '.../buildroot/output/dl/linux-internal-055213a7808f31ee04299c2d119b15a080b631f6.tar.gz', '--', 'git at github.com:teslamotors/linux.git', 'internal-055213a7808f31ee04299c2d119b15a080b631f6', 'linux-internal-055213a7808f31ee04299c2d119b15a080b631f6'] (buildroot-dl-wrapper) buildroot-dl-wrapper: Doing shallow clone buildroot-dl-wrapper: Cloning into 'linux-internal-055213a7808f31ee04299c2d119b15a080b631f6'... buildroot-dl-wrapper: fatal: Could not read from remote repository. buildroot-dl-wrapper: Please make sure you have the correct access rights buildroot-dl-wrapper: and the repository exists. It seemed fairly clear from the above that the path to the GitHub URL was not correct for someone downloading it without a GitHub account (due to the "access rights" comment) so we changed the .config file as follows: -BR2_LINUX_KERNEL_CUSTOM_REPO_URL="git at github.com:teslamotors/linux.git" +BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/teslamotors/linux.git" === linux (part 2) === We then ran into this: linux-internal-055213a7808f31ee04299c2d119b15a080b631f6+Image: ./scripts/gen_initramfs_list.sh: Cannot open '.../buildroot/output/images/initramfs.cpio.xz' linux-internal-055213a7808f31ee04299c2d119b15a080b631f6+Image: usr/Makefile:73: recipe for target 'usr/initramfs_data.cpio.gz' failed linux-internal-055213a7808f31ee04299c2d119b15a080b631f6+Image: make[2]: *** [usr/initramfs_data.cpio.gz] Error 1 linux-internal-055213a7808f31ee04299c2d119b15a080b631f6+Image: Makefile:963: recipe for target 'usr' failed Per the initramfs issues above, this is somewhat expected, so we just ran "touch output/images/initramfs.cpio.xz" and continued (see above "initramfs" section for details). == Final result and missing installation instructions == After the above change was made, the build finished, but we weren't able to find any installation instructions. We did find a few files in the output/images directory, specifically: * Image * rootfs.squashfs The above files appeared to contain the results of the Linux build (the strings of the Image file included "Linux version 3.18.69-rt75 (...) (gcc version 5.3.0 (Buildroot 2016.05-00004-gfb9fa1f65-dirty) ) #1 SMP PREEMPT RT ...") and the rest of the build (the rootfs.squashfs contained BusyBox and other binaries). While it is possible we could install the above files onto Tesla hardware with some guessing, it would be very difficult to do, and since the GPL requires installation instructions anyway, this omission does still require resolution by someone at Tesla with adequate expertise. == Comparison with Model S binaries == We compared the binaries built above with those we found in the binaries shipped on the Tesla Model S and discovered that they did not match. In particular, we found the following issues: * BusyBox v1.24.2 was provided via the buildroot project * BusyBox v1.19.2 (built 2016-02-23) was shipped on the Model S So we don't appear to have a source candidate that correponds to files shipped on the Model S. We chose not to compare other binaries inside the 2.15.50.img.bz2 binary file (for the main board system) since the BusyBox version was already so different. In order for the source candidate to be compliant, the binaries that it builds (including but not limited to BusyBox binaries) must correspond to the binaries that Tesla ships. We do note that Tesla has indicated these sources are for their 2018.12 build. We are aware that we may be comparing binaries from an earlier build, and that explains the mismatches. We ask Tesla to indicate more comprehensively their plans for compliance on past distributions of older versions of the software. We are willing to help figure out what compliance paths are feasible, and on our part may be willing to forgive past violations for older versions that may never resolved, provided that Tesla can attest that distribution of those older binaries has definitely ceased. (For example, we're particularly concerned about what binaries may be in the physical cars that have not received online updates yet. Compliance for those binaries, which as far as we can tell would match the binaries we have in our possesion, is an important issue.) Denver Gingerich Software Freedom Conservancy