What are the steps of how to use Selenium WebDriver with a proxy in Python?

1. Setting Up Proxies for Chrome WebDriver

To set up a proxy for Chrome, you need to use the --proxy-server argument in ChromeOptions.

Example: Chrome WebDriver with Proxy

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options

# Define the proxy server
proxy = "http://123.456.78.90:8080"  # Replace with your proxy IP and port

# Configure ChromeOptions
chrome_options = Options()
chrome_options.add_argument(f"--proxy-server={proxy}")

# Set up the WebDriver with the configured options
service = Service("path/to/chromedriver")  # Specify the path to chromedriver
driver = webdriver.Chrome(service=service, options=chrome_options)

# Test the proxy configuration
driver.get("https://www.whatismyip.com/")  # Check the visible IP

# Close the browser
driver.quit()

2. Setting Up Proxies for Firefox WebDriver

For Firefox, proxies are set using webdriver.FirefoxProfile.

Example: Firefox WebDriver with Proxy

from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
# Define the proxy server
proxy_host = "123.456.78.90"
proxy_port = "8080"
# Configure Firefox profile
firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("network.proxy.type", 1)  # Enable manual proxy
firefox_profile.set_preference("network.proxy.http", proxy_host)
firefox_profile.set_preference("network.proxy.http_port", int(proxy_port))
firefox_profile.set_preference("network.proxy.ssl", proxy_host)
firefox_profile.set_preference("network.proxy.ssl_port", int(proxy_port))
# Set up WebDriver
service = Service("path/to/geckodriver")  # Specify the path to geckodriver
driver = webdriver.Firefox(service=service, firefox_profile=firefox_profile)
# Test the proxy configuration
driver.get("https://www.whatismyip.com/")  # Check the visible IP
driver.quit()

3. Handling Proxy Authentication

For proxies requiring authentication (username/password), Selenium does not natively support this. Instead, you can create a browser extension or use webdriver.ChromeOptions to handle it.

Example: Chrome WebDriver with Authentication

Here is a simplified way using a Chrome Extension:

import zipfile
# Proxy details
proxy_host = "123.456.78.90"
proxy_port = "8080"
proxy_user = "username"
proxy_pass = "password"
# Create a proxy authentication plugin
manifest_json = """
{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": ["proxy", "tabs", "unlimitedStorage", "storage", "", "webRequest", "webRequestBlocking"],
    "background": {
        "scripts": ["background.js"]
    }
}

"""
background_js = f"""
var config = {{
    mode: "fixed_servers",
    rules: {{
        singleProxy: {{
            scheme: "http",
            host: "{proxy_host}",
            port: parseInt({proxy_port})
        }},
        bypassList: ["localhost"]
    }}
}};
chrome.proxy.settings.set({{value: config, scope: "regular"}}, function() {{}});
chrome.webRequest.onAuthRequired.addListener(
    function(details) {{
        return {{
            authCredentials: {{
                username: "{proxy_user}",
                password: "{proxy_pass}

            }}
        }};
    }},
    {{urls: [""]}},
    ["blocking"]
);
"""
# Create the extension
pluginfile = 'proxy_auth_plugin.zip'
with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
# Configure ChromeOptions with the proxy plugin
chrome_options = Options()
chrome_options.add_extension(pluginfile)
# Set up WebDriver
service = Service("path/to/chromedriver")
driver = webdriver.Chrome(service=service, options=chrome_options)
# Test the proxy with authentication
driver.get("https://www.whatismyip.com/")
driver.quit()

Key Points:

1. Proxy IP and Port: Always ensure the proxy details are accurate.
   
2. Authentication Handling: Use browser extensions for authenticated proxies in Chrome; Firefox profiles may handle this natively.
   
3. Testing the Configuration: Visit IP-detection websites like https://www.whatismyip.com/ to confirm the proxy is working.
   

These steps will allow you to set up proxies for both Chrome and Firefox WebDriver in Python, including handling authentication if required.