Matthew Kellogg – Data Scraping

I downloaded the text version of the IMDb (Internet Movie Database). From this I hope to illustrate a pattern of movie genre popularity vs movie age. For instance, I would illustrate a difference in popularity of current Sci-Fi films in comparison to older Sci-Fi films.

 

Here is a sketch of what this may look like.

data-sketch

An example of some data from the database is the following list of the some of the films Adam Sandler has been in.

Sandler, Adam (I)	101 Most Unforgettable SNL Moments (2004) (TV)  (archive footage)  [Himself]
			12-12-12 (2013)  [Himself]
			12-12-12: The Concert for Sandy Relief (2012) (TV)  [Himself]
			1996 MTV Movie Awards (1996) (TV)  [Himself]
			1999 Blockbuster Entertainment Awards (1999) (TV)  [Himself]
			1999 MTV Movie Awards (1999) (TV)  [Himself]  
			2001 MTV Movie Awards (2001) (TV)  [Himself]
			2002 MTV Movie Awards (2002) (TV)  [Himself - Presenter]  
			2004 MTV Movie Awards (2004) (TV)  [Himself - winner]
			2005 MTV Movie Awards (2005) (TV)  [Himself - Presenter]
			2006 MTV Movie Awards (2006) (TV)  [Himself]  
			2010 MTV Movie Awards (2010) (TV)  [Himself - Presenter]
			2012 MTV Movie Awards (2012) (TV)  [Himself - Presenter]
			2013 MTV Movie Awards (2013) (TV)  [Himself - Presenter]
			2014 MTV Movie Awards (2014) (TV)  [Himself - Cameo #14]  
			50 First Dates (2004)  [Henry Roth]  
			A Celebration of Bob Barker's 50 Years in Television (2007) (TV)  (uncredited)  [Himself]
			A Day with the Meatball (2002)  [Adam Sandler]
			Adam Sandler Goes to Hell (2001) (V)  [Himself]
			Adam Sandler: What the Hell Happened to Me? (1996) (TV)  [Himself]
			Airheads (1994)  [Pip]  
			America: A Tribute to Heroes (2001) (TV)  [Himself]
			Anger Management (2003)  [Dave Buznik]  
			Bedtime Stories (2008)  [Skeeter Bronson]  
			Big Daddy (1999)  [Sonny Koufax]  
			Billy Madison (1995)  [Billy Madison]  
			Blended (2014)  [Jim]  
			Blossoms & Blood (2003) (V)  [Barry Egan]  
			Bulletproof (1996)  [Moses]  
			Ceremonia de inauguración - 56º festival internacional de cine de San Sebastián (2008) (TV)  (archive footage) (uncredited)  [Himself]
			Chabad: To Life Telethon (2008) (TV)  [Himself - Video Guest]
			Click (2006)  [Michael Newman]  
			Comedy Central Presents: The Commies (2003) (TV)  [Himself - Presenter]
			Comedy Club Shoot-Out: Vol. 1 (2006) (V)  (archive footage)  [Himself]  
			Coneheads (1993)  [Carmine]  
			Couch (2003) (TV)  [Couch Testing Man]  
			Cómo conseguir un papel en Hollywood (2007) (TV)  (archive footage)  [John Clasky]
			Der Weg des Ruhms (2011)
			Deuce Bigalow: European Gigolo (2005)  (uncredited)  [Javier Sandooski]
			Deuce Bigalow: Male Gigolo (1999)  (voice) (uncredited)  [Robert Justin]
			Diet Coke with Lemon Celebrates 40 Years of Laughter: At the Improv (2002) (TV)  [Himself]
			Dirty Work (1998)  (uncredited)  [Satan]
			Eddie Murphy: One Night Only (2012) (TV)  [Himself]
			Eight Crazy Nights (2002)  (voice)  [Davey/Whitey/Eleanore/Deer]  
			ESPY Awards (1996) (TV)  [Himself]
			Finding Sandler (2015)  [Himself]

This data is available for non-profit use via FTP and is in a fairly usable format, so I did not write a scraper/parser.

I spent a large amount of time on Temboo, trying to find something that sparked my interest. Whenever something came up I would try to use the api, but the supported API would not be as Temboo described it. An example of this is the SchoolFinder api. I was able to obtain a key, but the rate was not limitted to 2400 requests per day, but 200. Also, temboo could not make the requests as there was a one IP per api key rule enforced, so the API had to be used directly.