• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
PythonForBeginners.com

PythonForBeginners.com

Learn By Example

  • Home
  • Learn Python
    • Python Tutorial
  • Categories
    • Basics
    • Lists
    • Dictionary
    • Code Snippets
    • Comments
    • Modules
    • API
    • Beautiful Soup
    • Cheatsheet
    • Games
    • Loops
  • Python Courses
    • Python 3 For Beginners
You are here: Home / Python On The Web / Browsing in Python with Mechanize

Browsing in Python with Mechanize

Author: PFB Staff Writer
Last Updated: December 2, 2020

Browsing with Mechanize

The mechanize module in Python is similar to perl WWW:Mechanize.

It gives you a browser like object to interact with web pages.

Here is an example on how to use it in a program.

import mechanize
br = mechanize.Browser()
br.open("http://www.example.com/")

Follow second link with element text matching regular expression

response1 = br.follow_link(text_regex=r"cheeses*shop", nr=1)
assert br.viewing_html()
print br.title()
print response1.geturl()
print response1.info()  # headers
print response1.read()  # body

To get the response code from a website, you can the response.code

from mechanize import Browser
browser = Browser()
response = browser.open('http://www.google.com')
print response.code

Get all forms from a website

import mechanize
br = mechanize.Browser()
br.open("http://www.google.com/")
for f in br.forms():
    print f

I found this post at http://stockrt.github.com that very accurate describes how
to emulate a browser in Python using mechanize.

Browsing with Python (written of Drew Stephens)

#!/usr/bin/python
import re
from mechanize import Browser
br = Browser()

Ignore robots.txt

br.set_handle_robots( False )

Google demands a user-agent that isn’t a robot

br.addheaders = [('User-agent', 'Firefox')]

Retrieve the Google home page, saving the response

br.open( "http://google.com" )

Select the search box and search for ‘foo’

br.select_form( 'f' )
br.form[ 'q' ] = 'foo'

Get the search results

br.submit()

Find the link to foofighters.com; why did we run a search?

resp = None

for link in br.links():
    siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )

    if siteMatch:
        resp = br.follow_link( link )
        break

Print the site

content = resp.get_data()
print content

The script above is split up to make it easier to read

Related

Recommended Python Training

Course: Python 3 For Beginners

Over 15 hours of video content with guided instruction for beginners. Learn how to create real world applications and master the basics.

Enroll Now

Filed Under: Mechanzie, Python On The Web Author: PFB Staff Writer

More Python Topics

API Argv Basics Beautiful Soup Cheatsheet Code Code Snippets Command Line Comments Concatenation crawler Data Structures Data Types deque Development Dictionary Dictionary Data Structure In Python Error Handling Exceptions Filehandling Files Functions Games GUI Json Lists Loops Mechanzie Modules Modules In Python Mysql OS pip Pyspark Python Python On The Web Python Strings Queue Requests Scraping Scripts Split Strings System & OS urllib2

Primary Sidebar

Menu

  • Basics
  • Cheatsheet
  • Code Snippets
  • Development
  • Dictionary
  • Error Handling
  • Lists
  • Loops
  • Modules
  • Scripts
  • Strings
  • System & OS
  • Web

Get Our Free Guide To Learning Python

Most Popular Content

  • Reading and Writing Files in Python
  • Python Dictionary – How To Create Dictionaries In Python
  • How to use Split in Python
  • Python String Concatenation and Formatting
  • List Comprehension in Python
  • How to Use sys.argv in Python?
  • How to use comments in Python
  • Try and Except in Python

Recent Posts

  • Count Rows With Null Values in PySpark
  • PySpark OrderBy One or Multiple Columns
  • Select Rows with Null values in PySpark
  • PySpark Count Distinct Values in One or Multiple Columns
  • PySpark Filter Rows in a DataFrame by Condition

Copyright © 2012–2025 · PythonForBeginners.com

  • Home
  • Contact Us
  • Privacy Policy
  • Write For Us