Question

bigrams = text_file.flatMap(lambda line: line.split(".")) \
                   .map(lambda line: line.strip().split(" ")) \
                   .flatMap(lambda xs: (tuple(x) for x in zip(xs, xs[1:])))

bigrams.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x + y)

This code gives me a list of bigrams from my file. Can this be edited so it gives, unigrams/trigrams/quadgrams etc?

Answer 1

def getNGrams(n):
    grams = text_file
        .flatMap(lambda line: line.split(".")) \
        .map(lambda line: line.strip().split(" ")) \
        .flatMap(lambda xs: (tuple(x[i:i+n]) for i in range(len(xs) - n + 1)))
        
return grams.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x + y)

**************************************************

I have created a generic function, which you can call with values n=1,2....n for Ngrams. Like for bigrams, use 2.

Thanks for your question. We try our best to help you with detailed answers, But in any case, if you need any modification or have a query/issue with respect to above answer, Please ask that in the comment section. We will surely try to address your query ASAP and resolve the issue.

Please consider providing a thumbs up to this question if it helps you. by Doing that, You will help other students, who are facing similar issue.