[Python] Configuring pip-Specific Proxy Settings in Containerized Environments with...

Configuring pip-Specific Proxy Settings in Containerized Environments


In containerized environments e.g. Databricks job clusters, I needed to install a Python wheel (e.g., my_package-1.0.0-py3-none-any.whl) with pip-resolved dependencies through a proxy. The proxy should apply only to pip server, not all traffic (e.g., Spark jobs, S3 data access or container networking), but default approaches [setting http_proxy] affected everything, and forces all the traffic via proxy.

Context

• Environments: Docker, Kubernetes, Databricks, etc.
• Use Case: Resolve wheel dependencies via pip using a proxy (e.g., http://{host}:{port}), excluding hosts like <no-proxy-list>.
• Requirement: Isolate proxy settings to pip in ephemeral setups.

Ways to Solve


Use pip-specific environment variables (PIP_PROXY and PIP_NO_PROXY) to configure the proxy cleanly:

General Configuration


PIP_PROXY=http://{host}:{port} # e.g., http://proxy.example.com:3128
PIP_NO_PROXY=<no-proxy-list> # e.g., localhost,127.0.0.1

Examples

Docker


docker run --rm \
-e PIP_PROXY=http://{host}:{port} \
-e PIP_NO_PROXY=<no-proxy-list> \
python:3.9 pip install my_package-1.0.0-py3-none-any.whl

Kubernetes


apiVersion: v1
kind: Pod
metadata:
name: pip-pod
spec:
containers:
1. name: python
image: python:3.9
command: ["pip", "install", "my_package-1.0.0-py3-none-any.whl"]
env:
- name: PIP_PROXY
value: "http://{host}:{port}"
- name: PIP_NO_PROXY
value: "<no-proxy-list>"

Databricks [My use-case]

1. In Advanced Options > Spark > Environment Variables:

PIP_PROXY=http://{host}:{port}
PIP_NO_PROXY=<no-proxy-list>

1. Install libraries via UI
2. Restart cluster.

Challenge with Default HTTP Proxy


Using HTTP_PROXY (e.g., HTTP_PROXY=http://{host}:{port}) routes all HTTP traffic through the proxy, including Spark jobs or container networking, disrupting non-pip operations. For example, in Databricks, this misroutes data traffic, while in Docker, it impacts container runtime calls. PIP_PROXY avoids this by scoping the proxy to pip alone.

Note


Alternatively, an init script (e.g., /databricks/scripts/set_pip_proxy.sh) can export PIP_PROXY=http://{host}:{port} and PIP_NO_PROXY=<no-proxy-list> or can run the command pip config set global.proxy http://{host}:{port} to get this done.
However, this adds complexity compared to directly setting environment variables in the cluster configuration.

Continue reading...