• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
PythonForBeginners.com

PythonForBeginners.com

Learn By Example

  • Home
  • Learn Python
    • Python Tutorial
  • Categories
    • Basics
    • Lists
    • Dictionary
    • Code Snippets
    • Comments
    • Modules
    • API
    • Beautiful Soup
    • Cheatsheet
    • Games
    • Loops
  • Python Courses
    • Python 3 For Beginners
You are here: Home / Python On The Web / Fetching data from the Internet

Fetching data from the Internet

Author: PFB Staff Writer
Last Updated: August 27, 2020

What is Urllib2?

urllib2 is a Python module for fetching URLs.

What can it do?

It offers a very simple interface, in the form of the urlopen function.

Urlopen is capable of fetching URLs using a variety of different protocols like
(http, ftp, file).

It also offers an interface for handling basic authentication, cookies, proxies
and so on.

These are provided by objects called handlers and openers.

HTTP Requests

HTTP is based on requests and responses, in that the client makes requests and
the servers send responses.

This response is a file-like object, which means you can for example call .read()
on the response.

How can I use it?

import urllib2
response = urllib2.urlopen('http://python.org/')
html = response.read()

User Agents

You can also add your own headers with urllib2.

Some websites dislike being browsed by programs.

By default urllib2 identifies itself as Python-urllib/x.y (where x and y are
the major and minor version numbers of the Python release,
which may confuse the site, or just plain not work.

The way a browser identifies itself is through the User-Agent header.

See our Urllib2 User Agent post that describes how to use that in a program.

Get HTTP Headers

Let’s write a small script that will get the HTTP headers from a website.

import urllib2
response = urllib2.urlopen("http://www.python.org")
print "-" * 20
print "URL : ", response.geturl()

headers = response.info()
print "-" * 20
print "This prints the header: ", headers
print "-" * 20
print "Date :", headers['date']
print "-" * 20
print "Server Name: ", headers['Server']
print "-" * 20
print "Last-Modified: ", headers['Last-Modified']
print "-" * 20
print "ETag: ", headers['ETag']
print "-" * 20
print "Content-Length: ", headers['Content-Length']
print "-" * 20
print "Connection: ", headers['Connection']
print "-" * 20
print "Content-Type: ", headers['Content-Type']
print "-" * 20
Will give an output similar to this:

——————–
URL : http://www.python.org
——————–
This prints the header: Date: Fri, 12 Oct 2012 08:09:40 GMT
Server: Apache/2.2.16 (Debian)
Last-Modified: Thu, 11 Oct 2012 22:36:55 GMT
ETag: “105800d-4de0-4cbd035514fc0”
Accept-Ranges: bytes
Content-Length: 19936
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

——————–
Date : Fri, 12 Oct 2012 08:09:40 GMT
——————–
Server Name: Apache/2.2.16 (Debian)
——————–
Last-Modified: Thu, 11 Oct 2012 22:36:55 GMT
——————–
ETag: “105800d-4de0-4cbd035514fc0”
——————–
Content-Length: 19936
——————–
Connection: close
——————–
Content-Type: text/html
——————–

Related

Recommended Python Training

Course: Python 3 For Beginners

Over 15 hours of video content with guided instruction for beginners. Learn how to create real world applications and master the basics.

Enroll Now

Filed Under: Python On The Web, urllib2 Author: PFB Staff Writer

More Python Topics

API Argv Basics Beautiful Soup Cheatsheet Code Code Snippets Command Line Comments Concatenation crawler Data Structures Data Types deque Development Dictionary Dictionary Data Structure In Python Error Handling Exceptions Filehandling Files Functions Games GUI Json Lists Loops Mechanzie Modules Modules In Python Mysql OS pip Pyspark Python Python On The Web Python Strings Queue Requests Scraping Scripts Split Strings System & OS urllib2

Primary Sidebar

Menu

  • Basics
  • Cheatsheet
  • Code Snippets
  • Development
  • Dictionary
  • Error Handling
  • Lists
  • Loops
  • Modules
  • Scripts
  • Strings
  • System & OS
  • Web

Get Our Free Guide To Learning Python

Most Popular Content

  • Reading and Writing Files in Python
  • Python Dictionary – How To Create Dictionaries In Python
  • How to use Split in Python
  • Python String Concatenation and Formatting
  • List Comprehension in Python
  • How to Use sys.argv in Python?
  • How to use comments in Python
  • Try and Except in Python

Recent Posts

  • Count Rows With Null Values in PySpark
  • PySpark OrderBy One or Multiple Columns
  • Select Rows with Null values in PySpark
  • PySpark Count Distinct Values in One or Multiple Columns
  • PySpark Filter Rows in a DataFrame by Condition

Copyright © 2012–2025 · PythonForBeginners.com

  • Home
  • Contact Us
  • Privacy Policy
  • Write For Us