Question

Question about spark with python.

I have a dataset named example.csv, here is data in CSV file below: [0] name [1] name of fruits [2] number of fruits

apple banana Jack Bob Diana apple 7 Eric orange frank 2 mango 0LO C N

1. use Pyspark code to plot the number of each fruit. sort the result based on the number of fruits

the x-axis is the name of fruits, the y-axis is num of total fruits

the output should be apple 10 banana 5 orange 3 mango 2

2. use Pyspark code to plot a number apple that each person have

the x-axis is the name, y is num of apple each person have

the output should be jack 3 Diana 7

Answer 1

PySpark doesn't have any plotting functionality (yet). If you want to plot something, you can bring the data out of the Spark Context and into your "local" Python session, where you can deal with it using any of Python's many plotting libraries

there you can use the following code:

1)

import matplotlib.pyplot as plt
import csv

the_dict = dict()

with open('example.csv','r') as csvfile:
    plots = csv.reader(csvfile, delimiter=',')
    for row in plots:
        if row[1] in the_dict:
            the_dict[row[1]]+=int(row[2])
        else:
             the_dict[row[1]] = int(row[2])

sorted_dict = sorted(the_dict.items(), key=itemgetter(1),reverse=True)

x = []

y=[]

for i in sorted_dict:

    x.append(i[0])

    y.append(i[1])

plt.xlabel('name of fruits')
plt.ylabel('num of total fruits')

plt.plot(x,y, label='fruits')

plt.legend()
plt.show()

2)

import matplotlib.pyplot as plt

import csv

x=[]

y =[]

with open('example.csv','r') as csvfile:

    plots = csv.reader(csvfile, delimiter=',')

        for row in plots:

   if row[1]=='apple':

   x.append(row[0])

   y.append(row[2])

plt.xlabel('name of the person')
plt.ylabel('number of apples')

plt.plot(x,y, label='apples')

plt.legend()
plt.show()