FreqDist.plot not using the most common words - nltk

Question

I am following the code in http://www.nltk.org/book/ch01.html specifically section 3.2.
from __future__ import division
import nltk
from nltk.book import *
import dateutil
import pyparsing
import numpy
import six
import matplotlib
fdist1 = FreqDist(text1)
fdist1.plot(50,cumulative=True)
is the script that I am running (the future import is leftover from some other stuff). However the plot I get does not match the one in the book. It has uncommon words such as funereal. I am running 32bit python 2.7 on windows. My friend who is running it on his mac runs the same commands and gets the plot from the book. I am at a complete loss as to what the difference may be. Thanks!

Answer 1

I've recently had the same problem.
NLTK builds it FreqDist upon the Counter class from python's collections package. The plot() (and the tabulate() function) extract the samples to use as follows:
samples = list(islice(self, *args))
Source: http://www.nltk.org/_modules/nltk/probability.html
I guess they're doing it that way to enable plotting of sub-sequences like the range between the 10th and the 40th sample, but it assumes that the Counter object (which basically is a dict) is sorted. Though that sometimes is the case, it obviously does not always hold, as the python documentation explicitely states. The correct replacement for the above line in the NLTK source would be:
samples = [item for item, _ in self.most_common(*args)]
That correct version can be also found in the most recent NLTK code on GitHub. It has been fixed in NLTK 3.0.0, so make sure that you aren't using an older version (like an alpha or beta version of NLTK 3).
If you do not want to change the source of NLTK and can not update it, you could easily adapt the plot() function yourself:
def plot_freqdist(fd, num = 0, cumulative = False, title = None):
import pylab
# Set up parameters
if num <= 0:
num = fd.B
# Get samples and frequencies
samples, freq, accu = [], [], 0
for s, f in fd.most_common(num):
accu = accu + f if cumulative else f
samples.append(s)
freq.append(accu)
# Create plot
pylab.grid(True, color = 'silver')
if title:
pylab.title(title)
pylab.plot(freq, linewidth = 2)
pylab.xticks(range(len(samples)), samples, rotation = 90)
pylab.xlabel('Samples')
pylab.ylabel('Cumulative Counts' if cumulative else 'Counts')
pylab.show()