In-class activity: Trying out NLTK

The goal of this in-class activity is to experiment with the NLTK library.

Below you will find a series of suggested exercises. You are free to pick and choose and modify these, and, to a reasonable extent, explore other questions. Keep a record of what you attempt and what you discover (such as a Python file annotated with comments) to turn in to count toward your participation grade.

Resources:

• Python and NLTK:
  • Official reference for Python 2.7
  • NLTK book
  • NLTK API. That's actually for a new version than what's (currently) installed in the lab, but it should be pretty close.
• Sample code:
  • /homes/tvandrun/Public/cs384/code/tryout.py -- what we did in class Monday
  • /homes/tvandrun/Public/cs384/code/simple_util.py -- a few helpful functions
• Data sets:
  • /homes/tvandrun/Public/cs384/texts/children-lit -- (folder containing) the texts we used in class Monday
  • /homes/tvandrun/Public/cs384/texts/proverbs -- (folder containing) the book of Proverbs
  • Several interesting texts are available in the data that comes with NLTK; just do from nltk.book import *
  • Project Gutenberg has lots of texts, for example plain text of James Barrie's Peter and Wendy (also known as Peter Pan)

Suggested exercises

1. In class we considered calculating a measure of a text's lexical diversity: the average number of times each type is used, or the ratio of tokens to types. (A lower score indicates a richer vocabulary). Can you detect any patterns in lexical diversity? Is it consistent for an author? Do different genres tend to have noticeably different lexical diversities? There are some problems with this measure used by itself (someone may have noticed this in class yesterday): the size of the text will skew the result, since smaller texts will have lower scores than longer texts. Can you think of a variation on this measure that would improve its accuracy?

2. NLTK has some features for plotting graphical representations of data about texts. These include a dispersion plots and plots of frequency distributions. A dispersion plot will show the density of use of a type or set of types over the length of the text. To see the density of the types Alice, Queen, and curious in the wonderland books, we can use

text.dispersion_plot(["Alice", "Queen", "curious"])

To see the relative frequencies of the top fifty most common types, use

freq_dist.plot(50, cumulative=False)

By setting cumulative to True, you can see a graph of the cumulative frequencies (ie, the 10th most common type is plotted against the combined number of occurrences of the top ten types).

Try these on a few texts. What can we learn? For example, can the dispersion plot be used to identify sections of a text?

3. Compare common collocations (bigrams) of different authors or texts. Are there any pairs that reveal something interesting about the content?

4. Using the POS tagger, write a function that calculates the frequency of personal pronouns. One would expect personal pronouns to be more common in narrative fiction. What do you find?

5. Similarly to the previous, write a function that calculates the density of adverbs. Is adverb use consistent for a given author? Does it vary greatly between authors or between genres?

6. The sci-fi/adventure series Tom Swift was sometimes mocked for relying heavily on the pattern of "proper noun said adverb". This spawned a kind of pun known as a Tom Swifty, which follows the same pattern but where the adverb puns what the character says. Some examples:

"Oops, I pierced your cheek by mistake," he said mysteriously.

"I'm making pancakes," he said flippantly.

"I use the Bourne Again Shell," he said bashfully.

Find how commonly this pattern (not necessarily as puns) occurs in various texts.

7. This problem comes from the NLTK book, problem 7 of Section 2.8:

According to Strunk and White's Elements of Style, the word however, used at the start of a sentence, means "in whatever way" or "to whatever extent", and not "nevertheless". They give this example of correct usage: However you advise him, he will probably do as he thinks best. (http://www.bartleby.com/141/strunk3.html) Use the concordance tool to study actual usage of this word in the various texts we have been considering. See also the LanguageLog posting "Fossilized prejudices about 'however'".

I would suggest not only using the concordance tool but also the tagger to see how it identifies however, as either an adverb RB (I, however, like ice cream) or as a (subordinating) conjunction IN (I will help however I can). Actually, in my quick experiments, I can't find an example where NLTK's pos tagger identifies any use of however as a conjunction.

At the end, turn in a record of what you did (and learned) to a turn-in folder /cslab.all/ubuntu/cs384/(your id)/ica1