Using a Proxy Server with Python requests

Using a Proxy Server with Python requests Cover image

In this tutorial, you’ll learn how to use the Python requests library to make HTTP requests behind a proxy server. This has numerous benefits, including staying anonymous and secure and preventing having an IP address blocked. You’ll learn how to set HTTP, HTTPS, and FTP proxies.

By the end of this tutorial, you’ll have learned:

Table of Contents

How to Use Proxy Servers with Python requests

In order to use proxies in the requests Python library, you need to create a dictionary that defines the HTTP, HTTPS, and FTP connections. This allows each connection to map to an individual URL and port. This process is the same for any request being made, including GET requests and POST requests.

It’s important to note that while the connections map to individual URLs and ports, they can actually point to the same URL and port.

Let’s see how you can define a set of proxies for the Python requests library:

# Setting up Proxies with the requests Library import requests proxy_servers = < 'http': 'http://proxy.sample.com:8080', 'https': 'http://secureproxy.sample.com:8080', >response = requests.get('sample.abc', proxies=proxy_servers)

Let’s break down what we did above:

  1. We imported the requests library
  2. We defined a dictionary, proxy_servers , which mapped URLs and ports to HTTP and HTTPS connections
  3. We then made a GET request and passed in our dictionary into the proxies= argument

How to Authenticate when Using a Proxy Server with Python requests

In order to add authentication to a request made with Python requests , you can follow normal request authentication methods. This allows you to use different types of authentication methods, including basic authentication.

Let’s take a look at how you can use basic HTTP authentication with proxy servers when making a request:

# Authenticating Requests with Proxy Servers import requests proxy_servers = < 'http': 'http://proxy.sample.com:8080', 'https': 'http://secureproxy.sample.com:8080', >auth = ('username', 'password') response="requests.get('sample.abc', proxies hl-hl">auth=auth)

We can see in the GET request we made above, that we passed in authentication information with the auth= parameter.

How to Use Sessions with a Proxy Server with Python requests

In some cases, you’ll want to use sessions when accessing data via an HTTP request. In these cases, using proxies works a little differently. We first need to instantiate a Session object and then assign our proxies using the .proxies attribute.

Let’s see how this can be done:

# Using Proxy Servers with Python requests Sessions import requests proxy_servers = < 'http': 'http://proxy.sample.com:8080', 'https': 'http://secureproxy.sample.com:8080', >s = requests.Session() s.proxies = proxy_servers response = s.get('sample.abc')

Let’s break down what we did in the code above:

  1. We imported the requests library
  2. We defined our proxy servers dictionary as before
  3. We then created a Session object, s
  4. We assigned proxies using the .proxies attribute and assigned our dictionary
  5. We then performed a GET request which automatically applied our proxies

Conclusion

In this tutorial, you learned how to use proxy servers when making HTTP requests using the Python requests library. Using proxies can help make your requests more secure or anonymous, as well as prevent your IP from being blocked when scraping websites.

You first learned how to use proxies when making requests using the requests library. Then, you learned how to use authentication with proxy servers. Finally, you learned how to use requests Sessions to handle proxy servers.

Additional Resources

To learn more about related topics, check out the tutorials below:

Nik Piepenbreier

Nik is the author of datagy.io and has over a decade of experience working with data analytics, data science, and Python. He specializes in teaching developers how to use Python for data science using hands-on tutorials.View Author posts