Question

Develop a crawler that collects the email addresses in the visited web pages. You can write a function emails() that takes a document (as a string) as input and returns the set of email addresses (i.e., strings) appearing in it. You should use a regular expression to find the email addresses in the document. Explain the code and how it works.

Answer 1

the code is written in the python file

It will be very easy to make you understand the python code. Just follow the below code and execute it as shown below.

import urllib.request#it is imported because we have to make a request to a site

import re# we have to use the regular expression therefore imported regular expression lib

def email(str):#function email has an argument as str which is your site url

    text=urllib.request.urlopen(str).read().decode(errors='replace')

    regex=re.compile(r'[\w.-]+@[\w.-]+')

    # \w means all characters[A-Z][a-z] and followed by it @ and after that . or domains

    # Examples. .com , .uk , .co.in

    #+ indicates that they should occur atleast once

    emaillist=re.findall(regex,text)#Findall Occurences

    print("All emails are kept in a list format shown below\n",emaillist)

    #Print all emails in the list format

    f=open("email.txt","a+")#Append all emails to text file a+

    f.write("Title:-"+str+"\n")# from which site you got email will be appended as title

    for i in emaillist:

        f.write(i+" ")

url=input("Enter your site :-")

email(url)

Execution

site containing all emails for sample data

Site url:- http://e-mailid.blogspot.com/

Create a file email.txt in the same directory of the python file

Output written to email.txt