Oct. 17, 2012

Browsing in Python with Mechanize

Browsing with Mechanize

The mechanize module in Python is similar to perl WWW:Mechanize.

It gives you a browser like object to interact with web pages.
Here is an example on how to use it in a program.
import mechanize
br = mechanize.Browser()
Follow second link with element text matching regular expression
response1 = br.follow_link(text_regex=r"cheeses*shop", nr=1)
assert br.viewing_html()
print br.title()
print response1.geturl()
print response1.info()  # headers
print response1.read()  # body
To get the response code from a website, you can the response.code
from mechanize import Browser
browser = Browser()
response = browser.open('http://www.google.com')
print response.code
Get all forms from a website
import mechanize
br = mechanize.Browser()
for f in br.forms():
    print f
I found this post at http://stockrt.github.com that very accurate describes how
to emulate a browser in Python using mechanize.

Browsing with Python (written of Drew Stephens)
import re
from mechanize import Browser
br = Browser()
Ignore robots.txt
br.set_handle_robots( False )
Google demands a user-agent that isn't a robot
br.addheaders = [('User-agent', 'Firefox')]
Retrieve the Google home page, saving the response
br.open( "http://google.com" )
Select the search box and search for 'foo'
br.select_form( 'f' )
br.form[ 'q' ] = 'foo'
Get the search results
Find the link to foofighters.com; why did we run a search?
resp = None

for link in br.links():
    siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )

    if siteMatch:
        resp = br.follow_link( link )
Print the site
content = resp.get_data()
print content
The script above is split up to make it easier to read

Share this article

Recommended Python Training – DataCamp

For Python training, our top recommendation is DataCamp.

Datacamp provides online interactive courses that combine interactive coding challenges with videos from top instructors in the field.

Datacamp has beginner to advanced Python training that programmers of all levels benefit from.


Download Our Free Guide To Learning Python

* indicates required

Read more about:
Disclosure of Material Connection: Some of the links in the post above are “affiliate links.” This means if you click on the link and purchase the item, I will receive an affiliate commission. Regardless, PythonForBeginners.com only recommend products or services that we try personally and believe will add value to our readers.