Python: Crawling Static Pages

As a software developer assigned to test and create crawler scripts for a few months... or years.. I would like to share to you how to scrape the page source from a static webpage.

Here's what we need:

• IDLE (Python GUI) (I am using 2.7.5)

The code is simple as:

#imports
import urllib2

# variable declaration
url = "http://techno-ob.blogspot.com"

# get page source
data = urllib2.urlopen(url).read()

# print source
print data

That's it. ;)



Other helpful links:

• Difference of urllib and urllib2

Read more ...