What are the steps of how to use Selenium WebDriver with a proxy in Python?

1. Setting Up Proxies for Chrome WebDriver

To set up a proxy for Chrome, you need to use the --proxy-server argument in ChromeOptions.

Example: Chrome WebDriver with Proxy

from selenium import webdriver

from selenium.webdriver.chrome.service import Service

from selenium.webdriver.chrome.options import Options



# Define the proxy server

proxy = "http://123.456.78.90:8080"  # Replace with your proxy IP and port



# Configure ChromeOptions

chrome_options = Options()

chrome_options.add_argument(f"--proxy-server={proxy}")



# Set up the WebDriver with the configured options

service = Service("path/to/chromedriver")  # Specify the path to chromedriver

driver = webdriver.Chrome(service=service, options=chrome_options)



# Test the proxy configuration

driver.get("https://www.whatismyip.com/")  # Check the visible IP



# Close the browser

driver.quit()

2. Setting Up Proxies for Firefox WebDriver

For Firefox, proxies are set using webdriver.FirefoxProfile.

Example: Firefox WebDriver with Proxy

from selenium import webdriver

from selenium.webdriver.firefox.service import Service

from selenium.webdriver.firefox.options import Options

# Define the proxy server

proxy_host = "123.456.78.90"

proxy_port = "8080"

# Configure Firefox profile

firefox_profile = webdriver.FirefoxProfile()

firefox_profile.set_preference("network.proxy.type", 1)  # Enable manual proxy

firefox_profile.set_preference("network.proxy.http", proxy_host)

firefox_profile.set_preference("network.proxy.http_port", int(proxy_port))

firefox_profile.set_preference("network.proxy.ssl", proxy_host)

firefox_profile.set_preference("network.proxy.ssl_port", int(proxy_port))

# Set up WebDriver

service = Service("path/to/geckodriver")  # Specify the path to geckodriver

driver = webdriver.Firefox(service=service, firefox_profile=firefox_profile)

# Test the proxy configuration

driver.get("https://www.whatismyip.com/")  # Check the visible IP

driver.quit()

3. Handling Proxy Authentication

For proxies requiring authentication (username/password), Selenium does not natively support this. Instead, you can create a browser extension or use webdriver.ChromeOptions to handle it.

Example: Chrome WebDriver with Authentication

Here is a simplified way using a Chrome Extension:

import zipfile

# Proxy details

proxy_host = "123.456.78.90"

proxy_port = "8080"

proxy_user = "username"

proxy_pass = "password"

# Create a proxy authentication plugin

manifest_json = """

{

    "version": "1.0.0",

    "manifest_version": 2,

    "name": "Chrome Proxy",

    "permissions": ["proxy", "tabs", "unlimitedStorage", "storage", "", "webRequest", "webRequestBlocking"],

    "background": {

        "scripts": ["background.js"]

    }

}

"""

background_js = f"""

var config = {{

    mode: "fixed_servers",

    rules: {{

        singleProxy: {{

            scheme: "http",

            host: "{proxy_host}",

            port: parseInt({proxy_port})

        }},

        bypassList: ["localhost"]

    }}

}};

chrome.proxy.settings.set({{value: config, scope: "regular"}}, function() {{}});

chrome.webRequest.onAuthRequired.addListener(

    function(details) {{

        return {{

            authCredentials: {{

                username: "{proxy_user}",

                password: "{proxy_pass}"

            }}

        }};

    }},

    {{urls: [""]}},

    ["blocking"]

);

"""

# Create the extension

pluginfile = 'proxy_auth_plugin.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:

    zp.writestr("manifest.json", manifest_json)

    zp.writestr("background.js", background_js)

# Configure ChromeOptions with the proxy plugin

chrome_options = Options()

chrome_options.add_extension(pluginfile)

# Set up WebDriver

service = Service("path/to/chromedriver")

driver = webdriver.Chrome(service=service, options=chrome_options)

# Test the proxy with authentication

driver.get("https://www.whatismyip.com/")

driver.quit()

Key Points:

1. Proxy IP and Port: Always ensure the proxy details are accurate.
   
2. Authentication Handling: Use browser extensions for authenticated proxies in Chrome; Firefox profiles may handle this natively.
   
3. Testing the Configuration: Visit IP-detection websites like https://www.whatismyip.com/ to confirm the proxy is working.
   

These steps will allow you to set up proxies for both Chrome and Firefox WebDriver in Python, including handling authentication if required.