Question

Could you please do this in Python to run in Spark.

For your second assignment I would like you to write code that does a word and character count on some input:

URL

Input File

The word count should not only count all occurrences of an individual word, the total number of representations of that word, and a frequency analysis of that word versus the total document.

Occurence are considered for spelling, without care of capitalization

for example -- "word Word word WoRd apple:

word 80% 4 total occurrences, 3 representations {word, Word, WoRd,}

apple 20% 1 total occurence, 1 representation {apple}

The character count similar analyzes the occurrences, representations and frequency analysis of character in the document.

Answer 1

Before running this code, make sure that validators, requests packages are installed.

validators is used to check whether the given file_name is url or not.

remove_word(): this function is used to remove the specific word from the list and returns the modified list

along with the list of different representation of the specific word.

import os

import validators

import requests as req

# DELIMITERS is used to split the string based on the spacial characters it contains.

# Update DELIMITERS as per your needs

DELIMITERS = ['.',',','\t','\n']

def remove_word(data, word):

    # this list is used to store the different representations of the word

    representations = []

    for d in data:

        if d.lower() == word:

            if d not in representations:

                representations.append(d)

            data.remove(d)

    return data, representations

def word_stats(data: str):

    # Replace all delimiters with space

    for char in DELIMITERS:

        data = data.replace(char,' ')

    # coverting the string into list of words by splitting the string

    words = data.split()

    words_copy = data.lower()

    total_words = len(words)

    while len(words) != 0:

        word = words[0].lower()

        word_count = words_copy.count(word)

        words, representations = remove_word(words, word)

        print("{0}:{1}%, {2} total occurrences, {3} representations {4}".format(word,(word_count/total_words)*100,word_count, len(representations), str(representations)))

if __name__ == "__main__":

    file_name = input()

    data = None

    if validators.url(file_name):

        data = req.get(file_name).text

    else:

        data = open(file_name, mode = 'r').read()

    

    word_stats(data)

Output: