• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
PythonForBeginners.com

PythonForBeginners.com

Learn By Example

  • Home
  • Learn Python
    • Python Tutorial
  • Categories
    • Basics
    • Lists
    • Dictionary
    • Code Snippets
    • Comments
    • Modules
    • API
    • Beautiful Soup
    • Cheatsheet
    • Games
    • Loops
  • Python Courses
    • Python 3 For Beginners
You are here: Home / Requests / Requests In Python

Requests In Python

Author: PFB Staff Writer
Last Updated: August 25, 2020

What is Requests

The Requests module is a an elegant and simple HTTP library for Python.

What can I do with Requests?

Requests allow you to send HTTP/1.1 requests.

You can add headers, form data, multipart files, and parameters with simple Python dictionaries, and access the response data in the same way.

Note, the notes in this post are taken from Python-Requests.org : http://docs.python-requests.org/en/latest/

Requests Installation

To install requests, simply:
$ pip install requests

Or, if you absolutely must:
$ easy_install requests

Make a Request

Begin by importing the Requests module:
>>> import requests

Now, let’s try to get a webpage.

For this example, let’s get GitHub’s public timeline

>>> r = requests.get('https://github.com/timeline.json')

# Now, we have a Response object called r. We can get all the information we need from this object.

To make a HTTP POST request

>>> r = requests.post("http://httpbin.org/post")

You can also use other HTTP request types, like PUT, DELETE, HEAD and OPTIONS

>>> r = requests.put("http://httpbin.org/put")

>>> r = requests.delete("http://httpbin.org/delete")

>>> r = requests.head("http://httpbin.org/get")

>>> r = requests.options("http://httpbin.org/get")

Response Content

We can read the content of the server’s response.

Consider the GitHub timeline again:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.text
'[{"repository":{"open_issues":0,"url":"https://github.com/...

Requests will automatically decode content from the server.

Most unicode charsets are seamlessly decoded.

When you make a request, Requests makes educated guesses about the encoding of
the response based on the HTTP headers.

The text encoding guessed by Requests is used when you access r.text.

You can find out what encoding Requests is using, and change it, using the
r.encoding property:

>>> r.encoding
'utf-8'
>>> r.encoding = 'ISO-8859-1'

If you change the encoding, Requests will use the new value of r.encoding
whenever you call r.text.

Binary Response Content

You can also access the response body as bytes, for non-text requests:

>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...

JSON Response Content

There’s also a builtin JSON decoder, in case you’re dealing with JSON data:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...

In case the JSON decoding fails, r.json simply returns None.

Custom Headers

If you’d like to add HTTP headers to a request,
simply pass in a dict to the headers parameter.

For example, we didn’t specify our content-type in the previous example:

>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> headers = {'content-type': 'application/json'}

>>> r = requests.post(url, data=json.dumps(payload), headers=headers)

Response Status Codes

We can check the response status code:

>>> r = requests.get('http://httpbin.org/get')
>>> r.status_code
200
# Requests also comes with a built-in status code lookup object for easy reference:
>>> r.status_code == requests.codes.ok
True
# If we made a bad request (non-200 response), 
# we can raise it with Response.raise_for_status():
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404

Response Headers

We can view the server’s response headers using a Python dictionary:

>>> r.headers
{
    'status': '200 OK',
    'content-encoding': 'gzip',
    'transfer-encoding': 'chunked',
    'connection': 'close',
    'server': 'nginx/1.0.4',
    'x-runtime': '148ms',
    'etag': '"e1ca502697e5c9317743dc078f67693f"',
    'content-type': 'application/json; charset=utf-8'
}

HTTP Headers are case-insensitive, so we can access the headers using any
capitalization we want:

>>> r.headers['Content-Type']
'application/json; charset=utf-8'

>>> r.headers.get('content-type')
'application/json; charset=utf-8'

# If a header doesn’t exist in the Response, its value defaults to None:
>>> r.headers['X-Random']
None

Cookies

If a response contains some Cookies, you can get quick access to them:

>>> url = 'http://httpbin.org/cookies/set/requests-is/awesome'
>>> r = requests.get(url)

>>> r.cookies['requests-is']
'awesome'
# To send your own cookies to the server, you can use the cookies parameter:

>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')

>>> r = requests.get(url, cookies=cookies)

>>> r.text
'{"cookies": {"cookies_are": "working"}}'

Basic Authentication

Many web services require authentication.

There are many different types of authentication, but the most common is HTTP
Basic Auth.

Making requests with Basic Auth is extremely simple:

from requests.auth import HTTPBasicAuth
requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))


# Due to the prevalence of HTTP Basic Auth, 
# requests provides a shorthand for this authentication method:

requests.get('https://api.github.com/user', auth=('user', 'pass'))

Providing the credentials as a tuple in this fashion is functionally equivalent to the
HTTPBasicAuth example above.

Digest Authentication

# Another popular form of web service protection is Digest Authentication:

>>> from requests.auth import HTTPDigestAuth

>>> url = 'http://httpbin.org/digest-auth/auth/user/pass'

>>> requests.get(url, auth=HTTPDigestAuth('user', 'pass'))

Redirection and History

Requests will automatically perform location redirection while using the GET
and OPTIONS verbs.

GitHub redirects all HTTP requests to HTTPS.

We can use the history method of the Response object to track redirection.

Let’s see what Github does:

>>> import requests
>>> r = requests.get("http://github.com")
>>> r.url
u'https://github.com/'
>>> r.status_code
200
>>> r.history
[]
>>>

The Response.history list contains a list of the Request objects that were
created in order to complete the request.

The list is sorted from the oldest to the most recent request.

If you’re using GET or OPTIONS, you can disable redirection handling with the
allow_redirects parameter:

>>> r = requests.get('http://github.com', allow_redirects=False)
>>> r.status_code
301
>>> r.history
[]

Timeouts

You can tell requests to stop waiting for a response after a given number of
seconds with the timeout parameter:

>>> requests.get('http://github.com', timeout=0.001)
Traceback (most recent call last):
  File "", line 1, in 
requests.exceptions.Timeout: Request timed out.

Errors and Exceptions

In the event of a network problem (e.g. DNS failure, refused connection, etc),
Requests will raise a ConnectionError exception.

In the event of the rare invalid HTTP response,
Requests will raise an HTTPError exception.

If a request times out, a Timeout exception is raised.

If a request exceeds the configured number of maximum redirections,
a TooManyRedirects exception is raised.

All exceptions that Requests explicitly raises inherit from
requests.exceptions.RequestException.

You can refer to Configuration API Docs for immediate raising of
HTTPError exceptions via the danger_mode option or have Requests
catch the majority of requests.exceptions.

RequestException exceptions with the safe_mode option.

Related

Recommended Python Training

Course: Python 3 For Beginners

Over 15 hours of video content with guided instruction for beginners. Learn how to create real world applications and master the basics.

Enroll Now

Filed Under: Python On The Web, Requests Author: PFB Staff Writer

More Python Topics

API Argv Basics Beautiful Soup Cheatsheet Code Code Snippets Command Line Comments Concatenation crawler Data Structures Data Types deque Development Dictionary Dictionary Data Structure In Python Error Handling Exceptions Filehandling Files Functions Games GUI Json Lists Loops Mechanzie Modules Modules In Python Mysql OS pip Pyspark Python Python On The Web Python Strings Queue Requests Scraping Scripts Split Strings System & OS urllib2

Primary Sidebar

Menu

  • Basics
  • Cheatsheet
  • Code Snippets
  • Development
  • Dictionary
  • Error Handling
  • Lists
  • Loops
  • Modules
  • Scripts
  • Strings
  • System & OS
  • Web

Get Our Free Guide To Learning Python

Most Popular Content

  • Reading and Writing Files in Python
  • Python Dictionary – How To Create Dictionaries In Python
  • How to use Split in Python
  • Python String Concatenation and Formatting
  • List Comprehension in Python
  • How to Use sys.argv in Python?
  • How to use comments in Python
  • Try and Except in Python

Recent Posts

  • Count Rows With Null Values in PySpark
  • PySpark OrderBy One or Multiple Columns
  • Select Rows with Null values in PySpark
  • PySpark Count Distinct Values in One or Multiple Columns
  • PySpark Filter Rows in a DataFrame by Condition

Copyright © 2012–2025 · PythonForBeginners.com

  • Home
  • Contact Us
  • Privacy Policy
  • Write For Us