Build system: SSL certificates for Python pip

Entropy512 · July 20, 2020, 6:04pm

One of my use cases for OpenWRT is customized wireless bridges that support diagnostic testing at work. As such, one of my development machines is behind one of those wonderful classic SSL MITM firewalls (Fortigate) for which you need to install the root CA certiificates in order for anything to work.

I've done this on my Ubuntu 18.04 host system, and all browsers running on that system (e.g. pip on the host runs fine), but the OpenWRT build system appears to be using its own version of pip that is obtaining SSL certificates from somewhere other than the build host's CA store. End result is:

  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
ERROR: Exception:
Traceback (most recent call last):
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connection.py", line 352, in connect
    self.sock = ssl_wrap_socket(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/adodd/gitrepos/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/adodd/gitrepos/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/adodd/gitrepos/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 159, in run
    requirement_set = resolver.resolve(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 467, in prepare_linked_requirement
    local_file = unpack_url(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 255, in unpack_url
    file = get_http_url(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 129, in get_http_url
    from_path, content_type = _download_http_url(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 277, in _download_http_url
    download = downloader(link)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/network/download.py", line 189, in __call__
    resp = _http_get_download(self._session, link)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/network/download.py", line 135, in _http_get_download
    resp = session.get(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_internal/network/session.py", line 421, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/cachecontrol/adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pip/_vendor/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
Traceback (most recent call last):
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
    subprocess.check_call(cmd)
  File "/home/adodd/gitrepos/openwrt/staging_dir/target-aarch64_cortex-a53_musl/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/bin/python3.8', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/home/adodd/gitrepos/openwrt/tmp/tmpri059w21', '--quiet', 'wheel']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 216, in <module>
    setup(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/setuptools/__init__.py", line 143, in setup
    _install_setup_requires(attrs)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/setuptools/__init__.py", line 138, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/setuptools/dist.py", line 695, in fetch_build_eggs
    resolved_dists = pkg_resources.working_set.resolve(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pkg_resources/__init__.py", line 781, in resolve
    dist = best[req.key] = env.best_match(
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1066, in best_match
    return self.obtain(req, installer)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1078, in obtain
    return installer(requirement)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/setuptools/dist.py", line 754, in fetch_build_egg
    return fetch_build_egg(self, req)
  File "/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/lib/python3.8/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
    raise DistutilsError(str(e))
distutils.errors.DistutilsError: Command '['/home/adodd/gitrepos/openwrt/staging_dir/hostpkg/bin/python3.8', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/home/adodd/gitrepos/openwrt/tmp/tmpri059w21', '--quiet', 'wheel']' returned non-zero exit status 2.
Makefile:39: recipe for target '/home/adodd/gitrepos/openwrt/build_dir/target-aarch64_cortex-a53_musl/pypi/PyNaCl-1.4.0/.built' failed
make[3]: *** [/home/adodd/gitrepos/openwrt/build_dir/target-aarch64_cortex-a53_musl/pypi/PyNaCl-1.4.0/.built] Error 1
make[3]: Leaving directory '/home/adodd/gitrepos/openwrt/feeds/packages/lang/python/python-pynacl'
time: package/feeds/packages/python-pynacl/compile#2.03#0.12#10.04
package/Makefile:111: recipe for target 'package/feeds/packages/python-pynacl/compile' failed
make[2]: *** [package/feeds/packages/python-pynacl/compile] Error 2
make[2]: Leaving directory '/home/adodd/gitrepos/openwrt'
package/Makefile:107: recipe for target '/home/adodd/gitrepos/openwrt/staging_dir/target-aarch64_cortex-a53_musl/stamp/.package_compile' failed
make[1]: *** [/home/adodd/gitrepos/openwrt/staging_dir/target-aarch64_cortex-a53_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/adodd/gitrepos/openwrt'
/home/adodd/gitrepos/openwrt/include/toplevel.mk:233: recipe for target 'world' failed
make: *** [world] Error 2

How do I add additional CA certificates that are used by the OpenWRT build system when fetching things?

Entropy512 · August 6, 2020, 7:50pm

I've made some progress on this: It is specifically a problem with packages in the feeds that use HostPython3/PipInstall, which is called if HOST_PYTHON3_PACKAGE_BUILD_DEPENDS is set and possibly under other circumstances.

I've worked around this issue temporarily by adding:

--trusted-host pypi.org --trusted-host files.pythonhosted.org

to the pip call at https://github.com/openwrt/packages/blob/master/lang/python/python3-host.mk#L56 based on https://stackoverflow.com/questions/25981703/pip-install-fails-with-connection-error-ssl-certificate-verify-failed-certi

But ideally I'd like to figure out a way to use the host system's SSL certificates (to which I've already added my company's Fortigate CA certs to) instead of just blindly trusting those particular hosts and/or making it a configurable option in menuconfig to explicitly trust these hosts due to (anticipated) corporate firewall MITMing.

Any suggestions as to how to go forward with fixing this in a clean manner that could be upstreamed?