This week’s assignment was

Get some XML from a web service. Extract some interesting information from the XML and use it as input to one of your homework assignments from previous weeks.

You could do this either by piping the output of your XML parsing program to the input of your previously implemented program, or by using a class (e.g., instantiate and use the MarkovChain class).

So I thought, cool, I’ll scramble the New York Times Best Seller titles using their shiny new API. Okay, so, I got an API key and managed to pull in lists of titles using—

import org.dom4j.Document;
import org.dom4j.io.SAXReader;
import org.dom4j.Element;
import java.util.List;

public class BestSeller
{
  public static void main(String[] args) throws Exception
  {

/*  Input should be one of the NY Times Best Seller list categories.
    Available categories:
    hardcover-fiction
    hardcover-nonfiction
    hardcover-advice
    paperback-nonfiction
    paperback-advice
    trade-fiction-paperback
    picture-books
    chapter-books
    paperback-books
    series-books
    mass-market-paperback
*/
    String listName = args[0];

    SAXReader reader = new SAXReader();
    EasyHTTPGet getter = new EasyHTTPGet(
      "http://api.nytimes.com/svc/books/v2/lists/" + listName + ".xml?api-key=f25de92cf1d2615621b68d2d31f81b63:4:1703697"
    );

    Document document = reader.read(getter.responseAsInputStream());
    List bookTitle = document.selectNodes("//results/book/book_details/book_detail/title");

//  If I wanted to scramble the authors, . . .
//  List bookAuthor =
//  document.selectNodes("//results/book/book_details/book_detail/author");

    for (Object o: bookTitle)
    {
      Element elem = (Element)o;
      String text = elem.getText();
      System.out.println(text);
    } // end for
  } // end main
} // end class

I had tried to make an array of category names and then loop through them, but I got lost after the first bit—

import org.dom4j.Document;
import org.dom4j.io.SAXReader;
import org.dom4j.Element;
import java.util.List;

public class BestSellerAll
{
  public static void main(String[] args) throws Exception
  {

/*  There are 11 categories of NY Times Best Seller lists. I want to download all of them, but the Times makes you download them individually.
*/

    String [] book_cats = new String [] {
    "hardcover-fiction", "hardcover-nonfiction", "hardcover-advice", "paperback-nonfiction", "paperback-advice", "trade-fiction-paperback", "picture-books", "chapter-books", "paperback-books", "series-books", "mass-market-paperback"};

//    String listName = args[0];

    SAXReader [] reader = new SAXReader() [];
    EasyHTTPGet [] getter = new EasyHTTPGet [];
    for (int i = 0; i < 11; i++)
    {
        getter[i] = "http://api.nytimes.com/svc/books/v2/lists/" + book_cats[n] + ".xml?api-key=f25de92cf1d2615621b68d2d31f81b63:4:1703697";
    } // end for i

// I got sadly confused somewhere in this block:
    Document [] document = new Document [];
    for (int j = 0; j < 11; j++)
    {
        document[j] = reader.read(getter[j].responseAsInputStream());
        List bookTitle = document.selectNodes("//results/book/book_details/book_detail/title");
    } // end for j

//  If I wanted to scramble the authors, . . .
//  List bookAuthor =
//  document.selectNodes("//results/book/book_details/book_detail/author");

    for (Object o: bookTitle)
    {
      Element elem = (Element)o;
      String text = elem.getText();
      System.out.println(text);
    } // end for bookTitle
  } // end main
} // end class

So, semimanually, then, I ran this on one category list at a time and used cat to agglomerate them. So then I had my list of all the current best sellers’ titles:

HANDLE WITH CARE
CORSAIR
THE ASSOCIATE
THE HOST
RUN FOR YOUR LIFE
PROMISES IN DEATH
DEAD SILENCE
HEART AND SOUL
ONE DAY AT A TIME
NIGHT AND DAY
THE GUERNSEY LITERARY AND POTATO PEEL PIE SOCIETY
WHITE WITCH, BLACK CURSE
PATHS OF GLORY
TERMINAL FREEZE
FOOL
THE HELP
DON’T LOOK TWICE
FAULT LINE
TRUE COLORS
STORM FROM THE SHADOWS
OUTLIERS
HOUSE OF CARDS
THE YANKEE YEARS
OUT OF CAPTIVITY
DEWEY
THE LOST CITY OF Z
A LION CALLED CHRISTIAN
A BOLD FRESH PIECE OF HUMANITY
MY BOOKY WOOK
THE UNFORGIVING MINUTE
INSIDE THE REVOLUTION
MELTDOWN
ARE YOU THERE, VODKA? IT’S ME, CHELSEA
JESUS, INTERRUPTED
JOKER ONE
NO ANGEL
LORDS OF FINANCE
THE NEXT 100 YEARS
MULTIPLE BLESSINGS
PICKING COTTON
ACT LIKE A LADY, THINK LIKE A MAN
THE LAST LECTURE
THE POWER OF SOUL
THE SECRET
THE ULTRAMIND SOLUTION
FLAT BELLY DIET!
THE GREAT DEPRESSION AHEAD
PEAKS AND VALLEYS
UNCOMMON
THE SURVIVORS CLUB
MAGNIFICENT MIND AT ANY AGE
THE TOTAL MONEY MAKEOVER
EMOTIONAL FREEDOM
THE 4 DAY DIET
FIGHT FOR YOUR MONEY
THREE CUPS OF TEA
THE MIDDLE PLACE
I HOPE THEY SERVE BEER IN HELL
DREAMS FROM MY FATHER
THE TIPPING POINT
EAT, PRAY, LOVE
THE AUDACITY OF HOPE
MY HORIZONTAL LIFE
90 MINUTES IN HEAVEN
TEAM OF RIVALS
SAME KIND OF DIFFERENT AS ME
BLINK
MARLEY & ME
THE FORGOTTEN MAN
THE OMNIVORE’S DILEMMA
THE ZOOKEEPER’S WIFE
BEAUTIFUL BOY
A WHOLE NEW MIND
ANIMAL, VEGETABLE, MIRACLE
INFIDEL
THE LOVE DARE
WHAT TO EXPECT WHEN YOU’RE EXPECTING
EMERGENCY
SUZE ORMAN’S 2009 ACTION PLAN
NATURALLY THIN
TWILIGHT
SKINNY BITCH
THE FIVE LOVE LANGUAGES
THE POWER OF NOW
HAPPY FOR NO REASON
THE PURPOSE-DRIVEN LIFE
A NEW EARTH
THE BIGGEST LOSER 30-DAY JUMP START
HE’S JUST NOT THAT INTO YOU
THE BIGGEST LOSER FAMILY COOKBOOK
THE SHACK
THE READER
FIREFLY LANE
AMERICAN WIFE
SUNDAYS AT TIFFANY’S
PEOPLE OF THE BOOK
A THOUSAND SPLENDID SUNS
TAKE ONE
THE ALCHEMIST
SARAH’S KEY
THE MIRACLE AT SPEEDY MOTORS
THE BRIEF WONDROUS LIFE OF OSCAR WAO
REVOLUTIONARY ROAD
WATER FOR ELEPHANTS
THE WHITE TIGER
STILL ALICE
THE ELEGANCE OF THE HEDGEHOG
LOVING FRANK
THE KITE RUNNER
LUSH LIFE
THE HOUSE IN THE NIGHT
THE COMPOSER IS DEAD
BLUEBERRY GIRL
LISTEN TO THE WIND: THE STORY OF DR. GREG AND “THREE CUPS OF TEA”
LADYBUG GIRL AND BUMBLEBEE BOY
GALLOP!
CAT
SWING!
NAKED MOLE RAT GETS DRESSED
ALL IN A DAY
MILES TO GO
THE GRAVEYARD BOOK
THIRTEEN REASONS WHY
SCAT
THE HUNGER GAMES
FADE
THREE CUPS OF TEA
3 WILLOWS
SEEKERS: GREAT BEAR LAKE
THE MYSTERIOUS BENEDICT SOCIETY AND THE PERILOUS JOURNEY
EVERMORE
THE BOY IN THE STRIPED PAJAMAS
THE BOOK THIEF
THREE CUPS OF TEA: YOUNG READERS EDITION
TWEAK
WICKED: WITCH AND CURSE
CORALINE
THE MYSTERIOUS BENEDICT SOCIETY
THE TALE OF DESPEREAUX
SLAM
THE TWILIGHT SAGA
HOUSE OF NIGHT
DIARY OF A WIMPY KID
THE 39 CLUES
THE CLIQUE
PERCY JACKSON & THE OLYMPIANS
HARRY POTTER
NIGHT WORLD
KISSED BY AN ANGEL
INKHEART
THE WHOLE TRUTH
HOLD TIGHT
BONES
THE GRAND FINALE
PLAGUE SHIP
LOST SOULS
MONTANA CREEDS: DYLAN
DANGER IN A RED DRESS
THE APPEAL
THE MACKADE BROTHERS: RAFE AND JARED
MAVERICK
SMALL FAVOR
ANGELS AND DEMONS
THE READER
SECRETS
FIRST COMES MARRIAGE
CHASING DARKNESS
SHADOW COMMAND
TEMPTATION RIDGE
CONFESSIONS OF A SHOPAHOLIC

And then, all I wanted to fucking do was run the Markov code from week 5 and make some crazy new titles. I had it working earlier in this process, when I was working with just the hardcover fiction list, and came up with these completely uninteresting results:

TERMINAL FREEZE
TERMINAL FREEZE
TERMINAL FREEZE
THE ASSOCIETY
ONE DAY AT A TIME
NIGHT AND SOUL
ONE DAY AT A TIME
FOOL
STORM FROM THE HELP
FOOL
TERMINAL FREEZE
CORSAIR
ONE DAY AT A TIME
TRUE COLORS
FOOL
ONE DAY AT A TIME

Clearly, the set of words was too small to do anything really interesting with, which is when I decided to make one file of all the lists, to get more words. But then . . . I couldn’t get the Markov code to work anymore. Kept getting this stupid error:

java.lang.StringIndexOutOfBoundsException: String index out of range: 4
at java.lang.String.substring(String.java:1765)
at Markov.feedLine(Markov.java:21)
at MarkovFilter.eachLine(MarkovFilter.java:12)
at com.decontextualize.a2z.TextFilter.internalRun(TextFilter.java:326)
at com.decontextualize.a2z.TextFilter.run(TextFilter.java:208)
at MarkovFilter.main(MarkovFilter.java:6)

And it took me one million years to determine that this was because the code was choking on the colon in

LISTEN TO THE WIND: THE STORY OF DR. GREG AND “THREE CUPS OF TEA”

and then the quotation marks, and then something else, and then I still don’t know what. It will not work if I include any lines after “GALLOP.” So, with that frustrating exception noted, here, finally, are the new titles:

I HOPE THE STORM FROM MY FATHERE, VODKA? IT’S ME, CHELSEA
THE FIVE LANE
MY HORIZONTAL LIFE OF DIFFERENT MIND THE LOVE DARE
THE BRIEF WONDROUS LIFE OF THERE, VODKA? IT’S WIFE
INSIDE THE REVOLUTION AHEAD
THE 4 DAY JUMP START
MAGNIFICENT AS ME, CHELSEA
THE ELEGANCE OF SOUL
THE ELEPHANTS
THE UNFORGOTTEN TO THE YANKEE YEARS
THE HEDGEHOG
THE COLORS
THE BOY
PROMISES IN HEAVEN
I HOPE THE TOTAL LIFE OF Z
HE’S DILEMMA
PROMISES IN THE TIPPING MIND SPLENDID SUNS
THE ZOOKEEPER’S JUST NOT THAT TO EXPECT WHEN YOUR LIFE OF NOW
TWILIGHT ANY AGE
A THOUSE OF DIFFERENT MINUTES IN HEAVEN
A BOLD FRESH PIECE OF OSCAR WAO
THE KIND OF HUMANITY
OUT OF GLORY
SAME KITE TIGER
FIGHT ANY AGE
MAGNIFICENT MIND SOUL
SAME KIND OF HUMANITY
THE FIVE LANGUAGES
THE MIDDLE WITH CARDS
TERMINAL FRESH PIE SOCIATE
90 MINUTES IN DEAD SILENCE
HE’S 2009 ACTIONARY AND SOLUTIONAL FRESH PIECE OF DIFFERENT MIND SOUL
NIGHT FOR ELEGANCE OF THERE, VODKA? IT’S WIFE
STORM FROM THERE, VODKA? IT’S JUST NOT THAT INTO YOUR LIFE OF THE LOVE DAY JUMP START
THE NIGHT ANY AGE

As you may recall, for my midterm project, I got stumped on several seemingly simple tasks. One of those—the most important, since upon it depends my semester-long assignment for Mainstreaming Information—was figuring out a way to compare one list of words to another and pull out the words that were unique to one of those lists. In my head, I can see very easily how this would be done. Given my special way of haphazardly flailing through code, however, I just couldn’t get it to work.

Until today!

In fiddling with the Bayesian comparison code for this week’s homework, I finally pulled out a list of unique words. Of course, this is a completely perverse misuse of that code—like using a steamroller to kill a pillbug—but as long as it works, I don’t fucking care.

So, here’s what I did. In BayesClassifier.java, I replaced the last two for loops with the following:

for (String word: uniqueWords)
    {
      for (BayesCategory bcat: categories)
      {
        double wordProb = bcat.relevance(word, categories);
        if (wordProb < 1)
        {
        println(word);
        }
        else {}
      } // end for bcat
    } // end for word

    for (BayesCategory bcat: categories)
    {
      double score = bcat.score(uniqueWords, categoryWordTotal);
      println("---The following words were not found in " + bcat.getName());
    } // end for bcat

And in BayesCategory.java I replaced the percentage and relevance blocks with

 public double percentage(String word)
  {
    if (count.containsKey(word))
    {
      return count.get(word);
    } // end if
    else
    {
      return 0.001;
    } // end else
  } // end percentage

  public double relevance(String word, ArrayList<BayesCategory> categories)
  {
    double percentageSum = 0;
    for (BayesCategory bcat: categories)
    {
      percentageSum += bcat.percentage(word);
    } // end for bcat
    return percentage(word);
  } // end relevance

So now, if I run the command

$ java BayesClassifier A2_unique.txt < B1_unique.txt | sort >results.txt

I get a list of words that are in B1_unique.txt (The Masada Scroll by Paul Block and Robert Vaughan, 2007) but not in A2_unique.txt (Zuleika Dobson or, An Oxford Love Story by Max Beerbohm, 1911). For example,

Akbar, Allah, Allahu, Apostolic, Ariminum, Arkadiane, Asmodeus, Astaroth, Barabbas, Beelzebub, Bellarmino, Blavatsky, Brandeis, Breviary, Byzantine, Caiaphas, Calpurnius, Catacombs, Charlemagne, Clambering, DNA, Diavolo, Franciscan, Freemasons, GPS, Gymnasium, Haddad, Hades, IDs, IRA, Jettisoning, Kathleen, Lefkovitz, MD, MRI, Masada, Masonic, Muhammad, Muhammadan, Nazarene, Nazareth, Olympics, Orthodoxy, Palatine, Palazzi, Palestine, Palestinian, Palestinians, Petrovna, Pleasant, Plenty, Plunge, Pocketing, Pontiff, Pontifical, Pontius, Praetorian, Prissy, Professors, Protestants, Rasulullaah, Ratsach, Revving, Rosicrucians, Satan, Scrolls, Seder, Shakespeare, Syracuse, Tacitus, Theosophical, Torah, Trastevere, Turkish, USB, Uzi, VAIO, VCR, Yeah, Yechida, Yeetgadal, Yiddish, adrenalin, agita, airliner, airport, ankh, awesome, bitch, bomb, bookstores, braked, breastplate, briefcase, broadsword, broiler, brotherhood, bulrushes, cellular, checkpoint, chuckling, chutzpah, combatant, computer, dashboard, database, departmental, desktop, divorce, dysentery, electricity, enabling, entrepreneurs, firearms, firestorm, fishtailed, flagon, forensics, goatskin, groggily, gunfire, gunman, gunshots, handbag, handball, handbrake, handgun, helicopter, helmets, highwaymen, hijinks, homeland, homeless, homespun, hometown, innkeeper, internship, journalist, kebob, kidnappers, kilometers, lab, laptop, lyre, mawkish, monitor, muezzin, nickname, nightfall, nonbeliever, northeaster, notebook, notepad, notepaper, numerology, paganism, password, pastries, phone, photo, photocopies, photocopy, photograph, photos, pig, pigeons, pistol, playback, police, quintessentially, recycles, redialed, roadblock, roadway, sandwich, screensaver, site, sites, submachine, superheating, synagogue, taped, taxi, terrorism, terrorist, terrorists, thousandfold, thrashing, toga, tortured, trigonometry, universe, unto, vegetables, vehicles, video, videotape, vinegar, violence, warehouses, waterfall, welfare, wholeheartedly, whoosh, whore, windshield, worker, workstation, worldwide, yardstick, yarmulkes, yeetkadash, zooming

And if I run the comparison in the opposite direction, I come up with words such as

Abernethy, Abiding, Abimelech, Abyssinian, Academically, Academy, Accidents, Achillem, Adam, Adieu, Admirably, Age, Agency, Agents, Alas, Albert, Alighting, America, Atlantic, Australia, Balliol, Baron, Baronet, Britannia, Broadway, Brobdingnagian, Colonials, Cossacks, Crimea, Devon, Dewlap, Duchess, Duke, Dukedom, Earl, Edwardian, Egyptians, Elizabethan, Englishmen, Englishwoman, Europe, Holbein, Ireland, Iscariot, Isis, Japanese, Kaiser, Liberals, London, Madrid, Meistersinger, Messrs, Monsieur, Napoleon, Novalis, Papist, Parnassus, President, Prince, Professor, Prussians, Romanoff, Segregate, Slavery, Socrates, Switzerland, Tzar, Victoria, Wagnerian, Waterloo, Whithersoever, Zeus, absinthes, acolyte, adventures, affrights, affront, afire, afoot, aforesaid, aggravated, album, analogy, anarchy, ankle, ape, aright, aristocracy, ataraxy, automatically, avalanche, avow, balustrade, bandboxes, bank, beastliest, beau, beauteous, billiards, biography, bodyguard, bosky, boyish, broadcast, bruited, bulldog, businesslike, bustle, calorific, casuistry, catkins, chaperons, chidden, cigarettes, clergyman, cloven, comet, compeers, coquetry, cricket, crinolines, custard, dandiacal, dapperest, decanter, devil, dialogue, diet, dipsomaniacal, disemboldened, disinfatuate, drunken, ebullitions, equipage, exigent, eyelashes, eyelids, farthingales, female, femininity, fishwife, fob, forefather, forerunners, freemasonry, furbelows, gallimaufry, goodlier, gooseberry, gorgeous, gypsy, haberdasher, halfpence, handicapped, handicraft, handiwork, handwriting, hearthrug, helpless, hip, hireling, honeymoon, housemaid, housework, hoyden, hussy, idiotic, impertinent, impudence, inasmuch, incognisant, insipid, insolence, insouciance, item, keyboard, landau, legerdemain, loathsome, luck, maid, maidens, manhood, manumission, matador, maunderers, model, mushroom, nasty, newspaper, noodle, nosegay, novel, oarsmen, omnisubjugant, ostler, otiose, parasol, pinafore, poetry, poltroonery, postprandially, prank, prestidigitators, propinquity, queer, romance, sackcloth, salad, sardonic, saucy, schoolmaster, seraglio, sex, skimpy, skirt, snuff, socialistic, streetsters, surcease, surcoat, swooned, teens, telegram, telegraphs, thistledown, thither, thou, threepenny, tomboyish, toys, tradesmen, treacle, ugly, uncouthly, unvexed, vassalage, waylay, welter, wigwam, witchery, withal, woe, woebegone, womanly, womenfolk, wonderfully, wonderingly, wretchedness, wrought, yacht, yesternight, zounds

Exciting!

For our Context Free assignment, I chose to make a recipe generator. Mmm, don’t you want to cook this up right now?

1. Toss 1/3 cup thinly sliced baking soda and 1 quart egg yolks with a whisk until just blended.
2. Knead 5 cups cubed light corn syrup with a wooden spoon until golden brown.
3. In a cast iron Dutch oven, pour 1/4 pinch egg yolks, 1/4 cup whole milk, and 2 pounds sifted egg.
4. Simmer 3 pounds light corn syrup with 1 cup thinly sliced brown sugar and 2 teaspoons sifted heavy cream.
5. In a small saucepan, boil 3 ounces thinly sliced egg, 4 cups peeled water, and 3/4 tablespoon water until incorporated.
6. Whisk together 1 pound ginger and 2 cups candied egg whites until golden brown.
7. Beat 2 quarts marzipan with a wooden spoon until the pieces are pea-sized.
8. Grate 1/2 teaspoon sifted unsweetened cocoa powder with 2 quarts baking powder and 3/4 ounce ginger.
9. Beat 5 pounds thinly sliced buttermilk, 5 quarts sifted whole milk, and 4 tablespoons thinly sliced egg with a wooden spoon until slightly warm.
10. In a large bowl, boil 5 cups candied kosher salt, 2 cups diced lemon zest, and 1/2 cup sifted buttermilk until incorporated.
11. In a shallow pan, simmer 5 tablespoons thinly sliced ginger until golden brown.
12. Beat 2/3 teaspoon brown sugar, 2 ounces diced egg yolks, and 3 teaspoons unsalted butter with 3 teaspoons light corn syrup, 1/2 tablespoon thoroughly washed lemon zest, and 3/4 quart peeled whole milk.
13. Brush 2 tablespoons grated kosher salt, 2/3 tablespoon sugar, and 4 pounds diced baking powder until it forms a thick syrup.

Here’s the grammar I used:

# procedures
S -> In a V Action IP with a Utensil until When
S -> In a V Action IP until When
S -> In a V Action IP .
S -> Action IP with IP .
S -> Action IP with a Utensil until When
S -> Action IP until When
IP -> IN
IP -> IN and IN
IP -> IN , IN , and IN
IN -> SQuantity SUnit Prep Ingredient
IN -> SQuantity SUnit Ingredient
IN -> PQuantity PUnit Prep Ingredient
IN -> PQuantity PUnit Ingredient
V -> Size Quality Vessel
V -> Size Vessel
V -> Quality Vessel
V -> Vessel

# components
SQuantity -> 1/4 | 1/2 | 1/3 | 3/4 | 2/3 | 1
PQuantity -> 2 | 3 | 4 | 5
SUnit -> pinch | teaspoon | tablespoon | ounce | cup | pound | quart
PUnit -> pinches | teaspoons | tablespoons | ounces | cups | pounds | quarts
Ingredient -> all-purpose flour | water | egg whites | egg yolks | heavy cream | whole milk | almond paste | marzipan | sugar | baking powder | baking soda | kosher salt | ginger | unsalted butter | egg | buttermilk | lemon zest | light corn syrup | bittersweet chocolate | unsweetened cocoa powder | brown sugar
Prep -> grated | cubed | chilled | sifted | diced | candied | thoroughly washed | peeled | thinly sliced | melted
Vessel -> bowl, | saucepan, | pot, | Dutch oven, | pan, | ramekin, | skillet,
Quality -> separate | airtight | cast iron | heatproof | nonstick | heavy-bottomed | shallow | buttered
Size -> large | small | medium
Action -> combine | whisk together | knead | brush | sprinkle | pour | stir | beat | boil | simmer | saute | brown | grate | mash | toss | blend
Utensil -> wooden spoon | whisk
When -> combined. | the pieces are pea-sized. | incorporated. | just blended. | it forms a thick syrup. | translucent. | golden brown. | slightly warm. | completely cooled.

Yummers!

Apparently, I have learned absolutely nothing all semester, because what seemed like a very straightforward project proved to be completely beyond my abilities.

The overarching goal is to generate data for the visualization I’m making for Lisa Strausfeld and Christian Marc Schmidt’s Mainstreaming Information class. Click the image above to see some slides (PDF, 128 KB) explaining the gist of the project, provisionally called Vocabulap (vocabulary + overlap; not a handsome coinage). My specific goals for the A2Z midterm were as follows (with subsequent comments in all caps):

For A2Z midterm
===============
Prep
—-
* Remove all blank lines
DONE
* Remove all extra spaces
DONE
* Break all lines – DONE
* Rename all to number consecutively: A01, A02, . . . A10 (for old books); B01, B02, . . . B10 (for new books)
DONE

Compare major sets
——————
* Extract the text from between the body tags in each file. Dump it out as a new file with the extension body.txt in the folder ../body.
THIS IS HARDER THAN IT LOOKS (FOR ME, AT LEAST). EASIER TO JUST CUT THEM OFF BY HAND.
* Concatenate all the files in each set.
DID THIS FROM THE COMMAND LINE, USING CAT
* Make a list of unique words in each concatenated set, with the number of times the word appears.
CAN GET THE UNIQUE WORDS, BUT NOT THE COUNT.
* Strip out all words beginning with numerals.
DONE BY HAND
* Create the following lists:
– Words shared by both major sets, with frequency counts
– Words unique to set 1, with frequency counts
– words unique to set 2, with frequency counts
I APPARENTLY CANNOT DO ANY OF THIS.

Find unique words in each book
——————————
For each book:
* Concatenate all the files in that major set *except* the file for that book.
* Make a list of the unique words, with frequency counts, in
– the current book
– the set of all books except the current one
* Make three lists:
– Words shared by all books in the major set, with frequency counts
– Words that appear only in the current book, with frequency counts
– Words that appear only outside the current book, with frequency counts

Return lines surrounding specific words
—————————————
For each word in a given list:
* Get the line numbers on which it appears.
For each appearance,
* Print the line above
* Print the line with the word, replacing it with itself wrapped in span tags to apply color
* Print the line below

The most essential piece of code that I could not get working is the comparison doodad. It almost worked for, like, five seconds, but it was generating a huge file of every unique word times however many words were in the document, or something like that. When I tried to fix it, it completely stopped working. The offending code is as follows:

/*  1. Takes in a file name from the command line.
    2. Makes a string array out of the hard-coded comparison file.
    3. Imports the contents of the file whose name was passed in.
    4. For each line of the input file (i.e., each word), changes it to
       lowercase and checks to see if it's contained in the comparison file.
    5. If it's not in the comparison file, checks to see if it's in a hashset of
       unique words.
    6. If the word's not in the hashset, add it.
    7. Print the contents of the hashset.
*/

import java.util.ArrayList;
import java.util.HashSet;
import com.decontextualize.a2z.TextFilter;

public class CompareUnique extends TextFilter
{
    public static void main(String[] args)
    {
        new CompareUnique().run();
    } // end main

    private String filename = "body/unique/allB_uci.txt";
    private HashSet uniqueWords = new HashSet();
    private HashSet lowercaseWords = new HashSet();

    // make a String array out of the contents of the comparison file
    String[] checkAgainst = new TextFilter().collectLines(fromFile(filename));

  public void eachLine(String word)
  {
    String wordLower = word.toLowerCase();
    for (int i = 0; i < checkAgainst.length; i++)
    {
        if (checkAgainst[i] != null && checkAgainst[i].contains(wordLower))
		{} // end if
		else if (checkAgainst != null)
		{
            if (lowercaseWords != null && lowercaseWords.contains(wordLower))
            { } // end if
            else if (lowercaseWords != null)
            {
                uniqueWords.add(wordLower);
                lowercaseWords.add(wordLower);
            } // end else
 		} // end else
    } // end for
  } // end eachLine

  public void end()
  {
    for (String reallyunique: uniqueWords) {
      println(reallyunique);
    } // end for
  } // end end

} // end class

I know, it seems very simple, but you have no idea how long it took me to get this far.

So, basically, for the midterm I’ve got bupkis—just a big pile of text files, and a list of unique words for each.

Oh, honestly.

For once I actually set out to be a slacker on the homework. The assignment began,

Modify, augment, or replace one of the in-class examples. A few ideas, in order of increasing complexity:

• Make Unique.java insensitive to case (i.e., “Foo” and “foo” should not count as different words).
• Modify WordCount.java to count something other than just words (e.g., particular characters, bigrams, co-occurrences of words, etc.). . . .

So I thought, “Today I feel like doing the easy thing. I’ll just take that first option.”

Yeah, right. Many hours later, after trying several extremely complicated methods of doing this really fucking simple thing, I finally found the rat-simple method that I’d been looking for all along and had all but given up hope of. Goddamnit.

The original code was this:

import java.util.HashSet;
import com.decontextualize.a2z.TextFilter;

public class Unique extends TextFilter {
  public static void main(String[] args) {
    new Unique().run();
  }

  private HashSet<String> uniqueWords = new HashSet<String>();

  public void eachLine(String line) {
    String[] tokens = line.split("\\W+");
    for (String t: tokens) {
      uniqueWords.add(t);
    }
  }

  public void end() {
    for (String word: uniqueWords) {
      println(word);
    }
  }
}

and what I came up with after way too much beating my head against the desk is this:

import java.util.HashSet;
import java.util.regex.*;
import com.decontextualize.a2z.TextFilter;

public class UniqueCI extends TextFilter
{
  public static void main(String[] args)
  {
    new UniqueCI().run();
  } // end main

  private HashSet<String> uniqueWords = new HashSet<String>();
  private HashSet<String> lowercaseWords = new HashSet<String>();

  public void eachLine(String line)
  {
    String[] tokens = line.split("\\W+");
    for (String t: tokens)
    {
    	// If hashset that's all lowercased contains t all lowercased, then don't add anything.

		String tLower = t.toLowerCase();

		if (lowercaseWords != null && lowercaseWords.contains(tLower))
		{
		} // end if
		else if (lowercaseWords != null)
		{
			uniqueWords.add(t);
			lowercaseWords.add(tLower);
 		} // end else
    } // end for
  } // end eachLine

  public void end()
  {
    for (String word: uniqueWords) {
      println(word);
    } // end for
  } // end end
} // end class UniqueCI

If you put this in,

It is a truth universally acknowledged, that a single man in possession of a large fortune must be in want of a wife.
It Is A Truth Universally Acknowledged, That A Single Man In Possession Of A Large Fortune Must Be In Want Of A Wife.

you get this out:

of
possession
wife
truth
be
large
It
fortune
universally
single
that
acknowledged
man
a
must
want
is
in

Big whoop. I wish I could say I learned a lot from this, but I think all I learned is that I’m much more lost than I thought I was.

Photo: The Downward Knife by Jill Greenseth; some rights reserved.

This week’s A2Z assignment was as follows:

Make a program that creatively transforms or performs analysis on a text using regular expressions. The program should take its input from the keyboard and send its to the screen (or redirect to/from a file). Your program might (a) filter lines from the input, based on whether they match a pattern; (b) match and display certain portions of each line; (c) replace certain portions of each line with new text; or (d) any combination of the above.

Sample ideas: Replace all words in a text of a certain length with a random word; find telephone numbers or e-mail addresses in a text; locate words within a word list that will have a certain score in Scrabble; etc.

Bonus challenge 1: Use one or more features of regular expression syntax that we didn’t discuss in class. Reference here.

Bonus challenge 2: Use one or more features of the Pattern or Matcher class that we didn’t discuss in class. Of particular interest: regex flags (CASE_INSENSITIVE, MULTILINE), “back references” in replaceAll. Matcher class reference here.

So, what I made is a program that tries to find the proper names in the input text (my preferred input being the first scene of Pride and Prejudice) and replace them with names of people in our class. For my purposes, a proper name is any capitalized word that (a) follows an honorific, such as Mr., Mrs., Sir, Lady, or Miss, or (b) does not immediately follow a carriage return or chunk of terminal punctuation (i.e., a period, exclamation point, or question mark, with or without a closing quotation mark thereafter), and (c) is neither a day of the week, a month of the year, nor the name of a holiday (though the only holiday it’s really looking for is Michaelmas, as that’s what appears in my P&P extract).

I tried for hours to do this in a compact way, by looking for the absence of certain words or characters (terminal punctuation, honorifics, days of the week, holidays) combined with the presence of other patterns, but it just would not fly. So the result requires a lot of intermediate steps, to avoid changing words that have been identified as not likely to be names. In a vain attempt to make testing my regular expressions quicker, I worked them out in BBEdit first, using the grep option in the search and replace panel. Saved me from having to recompile the thing fifty thousand times. And the expressions worked in BBEdit, for the most part. They do not work so well translated into Java, unfortunately—the success rate at finding names is much lower, and after ten straight hours, I just didn’t have the patience to troubleshoot it any more. But these glitches just make the results more funny, which is, of course, the point.

So, here is the code:

import java.util.regex.*;
import com.decontextualize.a2z.TextFilter;

public class RegexNames extends TextFilter
{
  public static void main(String[] args)
  {
    new RegexNames().run();
  } // end main(String[] args)

  private String search1  = "(Mr\\.|Mrs\\.|Lord|Lady|Sir) ([A-Z]\\w+)";
  private String replace1 = "##NAME##";

  private String search2  = "(\\.” |\\?” |!” |\\. |\\? |! |\\r|\\r“)([A-Z]\\w+)";

  private String search3  = "( |“|\\r)(Mr\\.|Mrs\\.|Lord|Lady|Sir|Miss|Monday|Tuesday|Wednesday|Friday|Saturday|Sunday|January|February|March|April|May|June|July|August|September|October|November|December|Michaelmas)";

  private String search4  = "( |“)([A-Z][a-z]+)";

  private String [] class_names = new String [] {
    "Adam", "Alejandro", "Andrew", "Bryan", "Caroline", "Dimitris", "Jonathan", "Joseph", "Martin", "Michael", "Ozge", "Sanjay", "Steven"       };
  public void begin()
  {
    println("* * *");
  }

  public void eachLine(String line)
  {
    String line_new;

    Pattern p1 = Pattern.compile(search1);
    Matcher m1 = p1.matcher(line);
    if (m1.find())
    {
      line = line.replaceAll(m1.group(2), replace1);
    }

    line_new = line;

    Pattern p2 = Pattern.compile(search2);
    Matcher m2 = p2.matcher(line_new);
    if (m2.find())
    {
      line_new = line_new.replaceAll(m2.group(2), "##" + m2.group(2) + "##");
    }

    Pattern p3 = Pattern.compile(search3);
    Matcher m3 = p3.matcher(line_new);
    if (m3.find())
    {
      line_new = line_new.replaceAll(m3.group(2), "##" + m3.group(2) + "##");
    }

    Pattern p4 = Pattern.compile(search4);
    Matcher m4 = p4.matcher(line_new);
    if (m4.find())
    {
      line_new = line_new.replaceAll(m4.group(2), "##NAME##");
    }

    line_new = line_new.replace( "##NAME##", class_names[(int)(Math.random() * 13)] );

    line_new = line_new.replace( "##", "" );

    println("\t" + line_new);
  } // end eachLine(String line)

  public void end()
  {
    println("* * *");
  }

} // end RegexNames extends TextFilter

And here is the output (you can find the original text at Project Gutenberg):

* * *
     It is a truth universally acknowledged, that a single man in possession of a large fortune must be in want of a wife.
     However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered the rightful property of someone or other of their daughters.
     “Adam dear Mr. Adam,” said his lady to him one day, “have you heard that Netherfield Park is let at last?”
     Mr. Bryan replied that he had not.
     “Steven it is, returned she; ”for Mrs. Steven has just been here, and she told me all about it.
     Mr. Michael made no answer.
     “Dimitris you not want to know who has taken it?” cried his wife impatiently.
     “YOU want to tell me, and I have no objection to hearing it.”
     This was invitation enough.
     “Steven, my dear, you must know, Mrs. Steven says that Netherfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise and four to see the place, and was so much delighted with it, that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and some of his servants are to be in the house by the end of next week.”
     “Joseph is his name?”
     “Adam.”
     “Dimitris he married or single?”
     “Bryan! Single, my dear, to be sure! A single man of large fortune; four or five thousand a year. What a fine thing for our girls!”
     “How so? How can it affect them?”
     “Bryan dear Mr. Bryan,” replied his wife, “how can you be so tiresome! You must know that I am thinking of his marrying one of them.”
     “Ozge that his design in settling here?”
     “Caroline! Nonsense, how can you talk so! But it is very likely that he MAY fall in love with one of them, and therefore you must visit him as soon as he comes.”
     “I see no occasion for that. You and the girls may go, or you may send them by themselves, which perhaps will be still better, for as you are as handsome as any of them, Mr. Michael may like you the best of the party.”
     “Michael dear, you flatter me. I certainly HAVE had my share of beauty, but I do not pretend to be anything extraordinary now. When a woman has five grown-up daughters, she ought to give over thinking of her own beauty.”
     “Jonathan such cases, a woman has not often much beauty to think of.”
     “Michael, my dear, you must indeed go and see Mr. Michael when he comes into the neighbourhood.”
     “Steven is more than I engage for, I assure you.”
     “Jonathan consider your daughters. Only think what an establishment it would be for one of them. Sir Jonathan and Lady Lucas are determined to go, merely on that account, for in general, you know, they visit no newcomers. Indeed you must go, for it will be impossible for US to visit him if you do not.”
     “Sanjay are over-scrupulous, surely. I dare say Mr. Sanjay will be very glad to see you; and I will send a few lines by you to assure him of my hearty consent to his marrying whichever he chooses of the girls; though I must throw in a good word for my little Lizzy.”
     “I desire you will do no such thing. Lizzy is not a bit better than the others; and I am sure she is not half so handsome as Michael, nor half so good-humoured as Lydia. But you are always giving HER the preference.”
     “Andrew have none of them much to recommend them,” replied he; “they are all silly and ignorant like other girls; but Lizzy has something more of quickness than her sisters.”
     “Mr. Bryan, how CAN you abuse your own children in such a way? You take delight in vexing me. You have no compassion for my poor nerves.”
     “Caroline mistake me, my dear. I have a high respect for your nerves. They are my old friends. I have heard you mention them with consideration these last twenty years at least.”
     Mr. Dimitris was so odd a mixture of quick parts, sarcastic humour, reserve, and caprice, that the experience of three-and- twenty years had been insufficient to make his wife understand his character. HER mind was less difficult to develop. Dimitris was a woman of mean understanding, little information, and uncertain temper. When she was discontented, she fancied herself nervous. The business of her life was to get her daughters married; its solace was visiting and news.
* * *

This week’s A–Z assignment:

Create a program (using, e.g., the tools presented in class) that behaves like a UNIX text processing program (such as cat, grep, tr, etc.). Your program should take text as input (any text, or a particular text of your choosing) and output a version of the text that has been filtered and/or munged. Your program should use at least one method of Java’s String class that we didn’t discuss in class.

Be creative, insightful, or intentionally banal. Optional: Use the program that you created in tandem with another UNIX command line utility

Sounds easy-peasy, except that despite our handy review session, I’ve completely forgotten anything I ever knew about Java or Processing. Not good.

Finally, after hours of beating my head against the couch and various O’Reilly books, I came up with two Java versions of the vowel shift–inducing UNIX command tr 'AEIOUaeiou' 'EIOUAeioua'. The first loops through the input letter by letter, checking each to see if it belongs to an array of vowels:

import com.decontextualize.a2z.TextFilter;

public class Tr extends TextFilter
{
    public static void main(String[] args)
    {
        Tr t = new Tr();
        t.run();
    } // end main(String[] args)
    public void eachLine(String line)
    {
        char [] vowels_in = new char [] { 'A', 'a', 'E', 'e', 'I', 'i', 'O', 'o', 'U', 'u' };
        char [] vowels_out = new char [] { 'E', 'e', 'I', 'i', 'O', 'o', 'U', 'u', 'A', 'a' };
        for (int i = 0; i < line.length(); i++)
        {
            char currentletter;
            char currentvowel;
            char printletter;
            currentletter = line.charAt(i);
            printletter = currentletter;
            for (int v = 0; v < 10; v++)
            {
                currentvowel = vowels_in[v];
                if (currentletter == currentvowel)
                {
                    printletter = vowels_out[v];
                } // end if
            } // end for v loop
            print(printletter);
        } // end for i loop
    } // end eachLine(String line)
} // end Tr extends TextFilter

The second rather tediously searches for and replaces each vowel in turn:

import com.decontextualize.a2z.TextFilter;

public class Tr2 extends TextFilter
{
    public static void main(String[] args)
    {
        Tr2 t = new Tr2();
        t.run();
    } // end main(String[] args)
    public void eachLine(String line)
    {
        String line_new;
        line_new = line.replace( "a", "[1]" );
        line_new = line_new.replace( "e", "[2]" );
        line_new = line_new.replace( "i", "[3]" );
        line_new = line_new.replace( "o", "[4]" );
        line_new = line_new.replace( "u", "[5]" );
        line_new = line_new.replace( "A", "[6]" );
        line_new = line_new.replace( "E", "[7]" );
        line_new = line_new.replace( "I", "[8]" );
        line_new = line_new.replace( "O", "[9]" );
        line_new = line_new.replace( "U", "[10]" );
        line_new = line_new.replace( "[1]", "e" );
        line_new = line_new.replace( "[2]", "i" );
        line_new = line_new.replace( "[3]", "o" );
        line_new = line_new.replace( "[4]", "u" );
        line_new = line_new.replace( "[5]", "a" );
        line_new = line_new.replace( "[6]", "E" );
        line_new = line_new.replace( "[7]", "I" );
        line_new = line_new.replace( "[8]", "O" );
        line_new = line_new.replace( "[9]", "U" );
        line_new = line_new.replace( "[10]", "A" );
        print(line_new + "\n");
    } // end eachLine(String line)
} // end Tr2 extends TextFilter

I tried to rewrite the latter using arrays and a loop, as well—

import com.decontextualize.a2z.TextFilter;

public class TrB extends TextFilter
{
    public static void main(String[] args)
    {
        TrB t = new TrB();
        t.run();
    } // end main(String[] args)
    public void eachLine(String line)
    {
        String line_new;
        String find_string;
        String replace_string;

        char [] vowels_in = new char [] { 'A', 'a', 'E', 'e', 'I', 'i', 'O', 'o', 'U', 'u' };
        char [] vowels_out = new char [] { 'E', 'e', 'I', 'i', 'O', 'o', 'U', 'u', 'A', 'a' };

        line_new = line;

        for (int i = 0; i < vowels_in.length; i++)
        {
            find_string = "[" + i + "]";
            line_new = line_new.replace( vowels_in[i], find_string );
        }
        for (int j = 0; j < vowels_in.length; j++)
        {
            replace_string = "[" + j + "]";
            line_new = line_new.replace( replace_string, vowels_out[j] );
        }
        print(line_new + "\n");
    } // end eachLine(String line)
} // end Tr2 extends TextFilter

but I just couldn’t get past this error message:

rB.java:28: cannot find symbol
symbol  : method replace(char,java.lang.String)
location: class java.lang.String
            line_new = line_new.replace( vowels_in[i], find_string );
                               ^
TrB.java:33: cannot find symbol
symbol  : method replace(java.lang.String,char)
location: class java.lang.String
            line_new = line_new.replace( replace_string, vowels_out[j] );
                               ^
2 errors

What stupid, obvious thing am I doing wrong?

Sample Input

Nelson was enjoying a little quiet at his house in the pretty village of Merton, in Surrey, when he learnt that the French and Spanish fleet, joined by the Ferrol squadron, had succeeded in entering Cadiz. His resolution was quickly taken. He went to the Admiralty and offered his services, which were joyfully accepted. Nelson was full of hope. “Depend on it,” he said to captain Blackwood, “I shall yet give M. Villeneuve a drubbing.” He formed his plans of attack during the short time of preparation, when the Victory had to be refitted, and other ships were to be got ready to accompany him. Lord Sidmouth told Mr. Eush, the American ambassador, that in the course of a visit he had received from Nelson, three weeks before the battle of Trafalgar, he described the plan of it, with bits of paper on a table, as it was afterwards fought. Yet he had a presentiment of his own fate. The coffin which was made out of the mast of l’Orient was deposited at an upholsterer’s. He desired its history to be engraved on its lid, saying that he should probably want it on his return. When he arrived at Portsmouth on the 14th of September, the enthusiasm of the people reached that height which sometimes gives a character of sublimity to the movements of multitudes acting with one heart. They wept; they blessed him; they even knelt as he passed along. The cheer which went up from thousands of voices as his barge pushed off to his flag-ship, -was the Godspeed of hia country. He waved his hat—a last farewell to England.
—The Popular History of England: An Illustrated History of Society and Government from the Earliest Period to Our Own Times by Charles Knight (Boston: Estes and Lauriat, 1874)

Sample Output

Nilsun wes injuyong e lottli qaoit et hos huasi on thi pritty vollegi uf Mirtun, on Sarriy, whin hi liernt thet thi Frinch end Spenosh fliit, juonid by thi Firrul sqaedrun, hed sacciidid on intirong Cedoz. Hos risulatoun wes qaockly tekin. Hi wint tu thi Edmorelty end uffirid hos sirvocis, whoch wiri juyfally ecciptid. Nilsun wes fall uf hupi. “Dipind un ot,” hi seod tu cepteon Bleckwuud, “O shell yit govi M. Volliniavi e drabbong.” Hi furmid hos plens uf etteck darong thi shurt tomi uf priperetoun, whin thi Voctury hed tu bi rifottid, end uthir shops wiri tu bi gut riedy tu eccumpeny hom. Lurd Sodmuath tuld Mr. Iash, thi Emirocen embessedur, thet on thi cuarsi uf e vosot hi hed riciovid frum Nilsun, thrii wiiks bifuri thi bettli uf Trefelger, hi discrobid thi plen uf ot, woth bots uf pepir un e tebli, es ot wes eftirwerds fuaght. Yit hi hed e prisintomint uf hos uwn feti. Thi cuffon whoch wes medi uat uf thi mest uf l’Uroint wes dipusotid et en aphulstirir’s. Hi disorid ots hostury tu bi ingrevid un ots lod, seyong thet hi shuald prubebly went ot un hos ritarn. Whin hi errovid et Purtsmuath un thi 14th uf Siptimbir, thi inthasoesm uf thi piupli riechid thet hioght whoch sumitomis govis e cherectir uf sablomoty tu thi muvimints uf maltotadis ectong woth uni hiert. Thiy wipt; thiy blissid hom; thiy ivin knilt es hi pessid elung. Thi chiir whoch wint ap frum thuasends uf vuocis es hos bergi pashid uff tu hos fleg-shop, -wes thi Gudspiid uf hoe cuantry. Hi wevid hos het—e lest feriwill tu Inglend.
—Thi Pupaler Hostury uf Inglend: En Ollastretid Hostury uf Sucoity end Guvirnmint frum thi Ierloist Piroud tu Uar Uwn Tomis by Cherlis Knoght (Bustun: Istis end Learoet, 1874)