Category Archives: 12-datascraping

Amanda Watson — Data Scraping

For this assignment, I was interested in looking at how people’s desires compared to their necessities.  I wanted to scrape tweets off of Twitter and look at the usage of “I want to” versus “I have to”.  What did people want, and what did they think was holding them back?

I collected about 1500 unique posts for each phrase using Temboo.  Looking at the output, I thought it might also be interesting to look at the wants-to’s and have-to’s people were prescribing others.  I collected another set of posts containing the phrase “they want to” and “they have to”

My ideas for visualization don’t really involve visualizing the sheer amount of data — I think it would be kind of interesting to use these phrases to construct sentence for narrative.  Display a randomly selected set of posts of the form “I want to…”, “but I have to…”, and examine how desires and necessities relate.  It might also be interesting to switch around the word “want” and “have” in the phrases, and see if the sentences remain plausible.

 

“i want to be harry’s friend so i can try to make him smile his beautiful smile and laugh so much his eyes crinkle shut every single day”

“I want to do something different….think i may sign up for some kickboxing classes”

“I want to eat but I’m not even hungry”

“@nina__schubert I want to see it ASAP!!! I’m so excited aha”

“i want to devote my time to individually messaging all 85k+ who liked that post and informing them of their poor life choice”

“I want a follow from @RickyPDillon but it’s 1:20 am so I think I have to sleep..”

“ugh now I have to go to track practice”

“So I’m at my last class of the day but then I remembered that I have to go see my advisor…”

“@PrairieHedge as an ex nurse I have to say hope!”

“Those words of reassurance is pretty much all I have to calm myself atm -A-”

“THERES SO MUCH SNOW OUT AND I HAVE TO SHOVEL IT ALL”

“Can’t wait to leave college so I can get away from fucking idiots I have to see every day”

“Lol at all the reading I have to do before tomorrow”

“for art i have to do active research so i would appreciate if you could RT this if any of these have happened to you http:\\/\\/t.co\\/VnzGdDDoXV”

“I have to get approval on certain things because it’s already hard to decipher tone thru text & me being as forward as I am it can go wrong.”

“Errbody wants to be rich but I rarely hear ppl say they want to be wealthy..#BigDifference…”

“they want to be accepted for shit that they actually do.\\” – OG Maco”

“Everyone is tweeting about how they want to go home and I’m just chillin in my bed :-)”

“@LaryssaBee exactly. People know where to go if they want to read the article. At least TG posts his blog but also summarizes in tweets”

“@chriscantrellOF fr they want to make it more of a hitters game prob”

“@abreumegan lol yes they will you’ll guide them down the right path \\ud83d\\udc4f you better let them smoke if they want to though”

“First impressions are big with #Virgos. They will know whether or not they want to be around you after just one conversation.”

“It’s Detroit v.s Everybody till one of them get where they want to go then its Us v.s the asshole who is doing fine.”

“\\”@TotoroHearts: @amsoulking they want to feel the music so deeply that their eardrums could explode \\n\\nthat’s me alright\\”\\n\\n#THIS”

“\\”Motivation is the art of getting people to do what you want them to do because they want to do it.\\”\\n\\n#PSFforSenate”

“RT @MaralaScott: When someone has something they want to share with you it is not because you are entitled but because you are blessed. \\u2013Ma\\u2026”

“They want to close”

“RT @KrIsTiNeOtErSoN: See they want to buy my pride”

“@Rathgrith027_SL @TheFartRises @b00nes They want to speak at the world but not have the world speak back.”

“@DigiRanger1994 @TheChaosBlue — They HAVE to cancel it.\\nHAVE TO.\\nThere’s a TRAVEL BAN as of 9PM tonight.”

“Girls who think they have to compete for a guy are just dumb if he likes you he’ll show quit trying”

“When people decide they want to start the unfriended movie in real life http:\\/\\/t.co\\/d9atJi4Lo9”

“They want to go to Disneyland Feb 19-22..-.”

“#Hijackraw WWE @TripleH They want to play with the fans well then lets play with Them”

“Is the word \\”dyke\\” disrespectful to lesbians orrrr no? Like do they want to be called that?”

“Why do ex’s give the guilty trip when they want to make something right?”

“\\u201c@aquuuarius: #Aquarius tend to push people away when they’re moody or upset. They only’ll talk about it if they want to.\\u201d \\ud83d\\ude02\\ud83d\\ude02 so me”

“@ihateprospects There is a better chance of that than there is of him becoming beloved by the fans. They want to boo him after last night”

“Just cuz someone is single that doesn’t mean they want to be accosted in their DMs lol \\”@TucoFbaby: If you single why are your DM’s broke?\\””

“RT @leYOWard: We can’t beg someone to stay when\\nthey want to leave and be with someone else.”

“RT @akhbarlarh: @OhJujAi brooo please struggle sikit for your future . You know you mum and dad . They want to see you succed”

Fblikes predictor_Priya Ganadas

This is my attempt to make a pun at people who are obsessed with approvals, fb likes being a measurable factor in social media. If you had a fblikes predictor, would you improve your chances of getting liked by people?

For my data scraping assignment, I want to use my facebook account. I plan to scrape data of fb likes by my friends.
The Process-
1. List all the status messages posted.
2. List all the people who liked particular status message.
3. Assign labels to the status message(tagging in terms of keywords).
4. Draw a graph of keywords vs Fb likes.

This graph would show possibility of getting a like by people if certain keywords are used or if the status expresses emotions of certain kind.

Further: I would like to track 50 of my friends to visualise their pattern of liking my status message.

Depending on this data, the graph would also extrapolate or predict. If a future status message is typed in, ideally, I should know % chance of likelihood that who of those 50 friends would end up hitting ‘like’ to my status message.

 

Code-
This code is built off Ben Grosser’s examples for temboo.

import com.temboo.core.*;
import com.temboo.Library.Facebook.Reading.*;
 
// Create a session using your Temboo account application details
TembooSession session = new TembooSession("ypag","myFirstApp","");
 
String accessToken = ""; // INSERT YOUR FACEBOOK TOKEN HERE
 
ArrayList friendNames;
ArrayList friendIDs;
 
void setup() {
 
  // grab (and print) your friend data from FB. fills 2 ArrayLists:
  // friendNames 
  // friendIDs
  getFriendIDs();
 
  // get your own statuses and names of who liked them
  getStatuses("me");
 
  // WARNING WARNING WARNING
  // This will grab data for EVERY friend! This can be a lot of data
  // you can just call getStatuses("00010202309202") for whichever friend you 
  // want data from instead of getting data from all of them
   
//  run through all of your friends and get their statuses and who liked them
//  for(int i = 0; i < friendIDs.size(); i++) {
//    String friendID = friendIDs.get(i).toString();
//    String friendName = friendNames.get(i).toString();
//    println("\n---------------------------------------------");
//    println("STATUSES FOR: "+friendName);
//    getStatuses(friendID);
//  }
   
}
 
// gets all your friends and returns an array with their IDs
void getFriendIDs() {
  Friends friendsConnector = new Friends(session);
  friendsConnector.setAccessToken(accessToken);
  FriendsResultSet friendsResults = friendsConnector.run();
  //println(friendsResults.getResponse());
  JSONObject response = parseJSONObject(friendsResults.getResponse());
  JSONArray data = response.getJSONArray("data");
   
  friendIDs = new ArrayList();
  friendNames = new ArrayList();
   
  for(int i = 0; i < data.size(); i++) {
    JSONObject friend = data.getJSONObject(i);
    String name = friend.getString("name");
    String id = friend.getString("id");
     
    friendIDs.add(id);
    friendNames.add(name);
     
    //println(name + " : " + id);
  }
}
 
void getStatuses(String profileID) {
  // Create the Choreo object using your Temboo session
  Statuses statusesChoreo = new Statuses(session);
 
  // Set inputs
  statusesChoreo.setAccessToken(accessToken);
   
  if(profileID != "me") {
    statusesChoreo.setProfileID(profileID);
  }
   
  // Run the Choreo and store the results
  StatusesResultSet statusesResults = statusesChoreo.run();
 
  // want to see what you're getting? print out statusesResults.getResponse(), 
  // copy output, and then paste into a JSON beautifer, like http://jsonformatter.curiousconcept.com/
  // println(statusesResults.getResponse());
   
  // extract and printout status info  
  printStatusInfo(statusesResults);
 
}
 
// looks at a statusResult and extracts some info
// needs to be customized to meet your needs if they're different than this 
// https://temboo.com/processing/parsing-json can be helpful trying to figure it out
void printStatusInfo(StatusesResultSet s) {
   
  // create a JSONObject from the StatusesResultSet
  JSONObject response = parseJSONObject(s.getResponse());  
 
  // 'data' holds all of our response data  
  JSONArray data = response.getJSONArray("data");
   
  // print raw data
  //println(data);
     
  // run through the objects in data (our statuses)
  for(int i = 0; i < 10; i++) {
     
    // extract a status from the data
    JSONObject status = data.getJSONObject(i);
     
    // see if there's a message (sometimes there's not)
    // extract the message string if there is one
    String message;
        
    try {
      message = status.getString("message");
    } catch (Exception e) {
      message = null;
    }
     
    // if we don't have one, skip to the next for() iteration
    if(message == null) continue;
     
    // print the message string (e.g. the status)
    println("\n\n"+message);
     
    // check for and get all the likers of this status
    // if there are any
    JSONObject likes;
     
    // see if there are any likes (there may not be)
    // use try/catch as if there aren't any it throws 
    // an error --- this lets us get by it if there is
    try {
      likes = status.getJSONObject("likes");
    } catch (Exception e) {
      likes = null;
    }
 
    // if there are some likes, process them
    if(likes != null) {
       
      // get an array of all the likers
      JSONArray likers = likes.getJSONArray("data");
       
      // we have some, so printout a header
      println("  Liked by:");
       
      // run through all of the likes, extract/print name
      for(int j = 0; j < likers.size(); j++) {
        String liker = likers.getJSONObject(j).getString("name");
        println("   -"+liker);
        if(liker.equals("Raj Lakshmee"))
            { println (" wohoo!!" );
            }
      }
       
    } // end like process
     
  } // end of status for()
   
}

Visualisation- Would consist of keywords in fb status where the size of region is proportional to the number of likes.

datascrape

sejalpopat

27 Jan 2015

My original goal was to come up with a clear question regarding metaphors and their usage by people online in different contexts (i.e. from social media to mathematics) and I spent a lot of time researching this but came to realize that…metaphor detection is hard.  So, while I did stumble on various hand curated databased of metaphors, there were no automated methods to detect these (which makes sense), and my research led me to papers discussing possible techniques etc. But I’m still attached to the idea and hopeful that I can find other data that is related to metaphor and using language to connect unlikely concepts together. I found a paper that described and listed general categories of “source” and “target” domains for metaphors. This, I think, could be useful if I can find an api that gives me instances of these general categories. Then given this information, in conjunction with the already curated metaphor databases I could have a good base of information to work with.

Possible visualizations that I would like to explore are graphs (networks) of concepts. These would be collapsible (clusters of similar concepts which are “instances of larger categories” could maybe be collapsed into that larger category). Another, maybe more interesting/interactive option that also takes into account not just the structure(concept relations) in metaphors but the social context and usage, is to have two columns (one for source and one for target) that the user could choose from each and connect, then the system would search for metaphors that connect these and instances of these occurring in conversations online.

Here is a link to the full data: https://github.com/spopat/metaphor-network/blob/master/metaphors.json

And here is a sample:

{
  "views": {
    "view": [
      {
        "name": "Body Is A Container",
        "source_domain": "Container",
        "target_domain": "Body",
        "origin": "mml",
        "timestamp": "2002-07-25T15:53:49"
      },
      {
        "name": "Emotions Are Entities Within A Person",
        "source_domain": "Entities",
        "target_domain": "Emotions",
        "parent": "Body Is A Container",
        "examples": { "example": "I was filled with rage" },
        "origin": "mml",
        "timestamp": "2002-07-25T15:56:14"
      },
      {
        "name": "Abilities Are Entities Inside A Person",
        "source_domain": "Entities",
        "target_domain": "Abilities",
        "parent": "Body Is A Container",
        "examples": { "example": "he doesn't have it in him to be polite" },
        "origin": "mml",
        "timestamp": "2002-07-25T15:57:50"
      }

Ron

27 Jan 2015

I’ve always enjoyed the Dilbert comic strip, and out of curiosity, I discovered that the entire strip’s dialogue over the last 25+ years is embedded in the page source of each comic that’s posted online on Dilbert.com. I thought it might be interesting to examine the conversation word content over 11,700 comic strips.

I learned writing in Python, which I was previously unfamiliar with. And I also learned to use BeautifulSoup for this exercise. Since each strip’s URL ends with the date, I wrote a Python script to iteratively fetch a given date’s comic strip’s dialogue (as well as the date that the comic strip was published) and work backwards until the comic’s first strip in April 1989. Below is a sample (past 60 days) of the Dilbert comic strip dialog that I scraped:

2015-01-24 * Coworker: You know how studies always say one out of ten people have a particular problem. I'm always that guy. Statistically speaking, I keep nine people safe just by existing. Dilbert: That's not how statistics work. Coworker: And... everyone else in the department knows that?
2015-01-23 * Boss: A global survey says only 13% of employees feel engaged at work. Dilbert: If you're wondering which one of your employees is engaged, it's this guy. Boss: We need ten more just like him. Dilbert: I think I just figured out what's wrong with the rest of us.
2015-01-22 * Wally: I heard you invented a selfie camera for car steering wheels. Dilbert: Not exactly. I invented an ion-powered flying car, but all anyone cares about is the selfie camera in the steering wheel. Wally: Can you share the selfies on Facebook? Dilbert: Gaaa!
2015-01-21 * Dilbert: I designed a flying car that harvests ions from the air to power itself. We can build them for only $3,000 apiece. CEO: There's no market for ion-powered flying cars. Dilbert: I can put a selfie camera in the steering wheel. CEO: Much better. And let' say the car does not fly.
2015-01-20 * CEO: The board is getting on me for not having a succession plan. Find me a loser who is so incompetent that the board won't want to fire me. Boss: It's an honor to even be considered! Catbert: I was going to say that!
2015-01-19 * Boss: Are you interested in becoming a manager? Dilbert: That would never work. Managers are supposed to hire people who are smarter than they are. That's easy for you, but how would I ever find anyone to hire? Boss: I don't understand. Dilbert: And that makes you the perfect manager.
2015-01-18 * Dilbert: That isn't in my job description. Boss: What?! You should never tell your boss that a task isn't in your job description! It makes you sound like a whiney quitter instead of a resourceful entrepreneur. And don't forget all the personal growth that comes from taking on new challenges. Think outside the box. That is the key to greatness. Dilbert: So, according to you, the best way to achieve greatness is by assigning the wrong people to tasks? Are there any other dumb things I need to do to achieve greatness or is one thing enough? Catbert: Did you find someone to mow your lawn yet? Boss: Almost. He's putting up a fight.
2015-01-17 * Boss: We're looking for employees who are fierce! Applicant: Should I punch you or something? I don't know how to play this. Boss: Try acting normal, but angrier.
2015-01-16 * CEO: The quality we look for in our employees is "super pumpedness." Alice: Blechhhh!!! I told you to bring a bag.
2015-01-15 * Boss: I keep saying leader-ish things, but the employees are still terrible. Catbert: Try using this hickory branch as a wand. Boss: Qualitos improvimentus! Wally: I'll be on the Harry Potter.
2015-01-14 * Coworker: Wally, did you finish the... Wally: Get off my back! Why can't you just trust me to do my work on time?!?! Coworker: Sorry... Dilbert: Who were you yelling at? Wally: Beats me. It didn't seem important.
2015-01-13 * Boss: I forget why I walked over here. Dilbert: Were you planning to spew empty jargon in my direction to create the illusion of leadership? Boss: Oh, right. But now it feels as if the moment has passed.
2015-01-12 * Catbert: Your co-workers are complaining that your tube clothes distract them from work. Dilbert: I can make some modifications so I'm less sexy. That should cut down on the distraction. Catbert: We might not be on the same page. Dilbert: I could wear a corduroy tube. That's a libido killer.
2015-01-11 * Wally: Asok, the key to winning at your job is the taper. Asok: Taper? Wally: At the start of any new job, you want to put in long hours and create a good first impression. Then you should start to gradually taper off your effort. But be sure you taper slowly. You don't want to be obvious. Boss: Wally, is it my imagination, or are you working slightly less every day? Wally: It only looks that way because I'm working smarter, not harder. Just the way you taught me. Boss: Okay, that sounds right. Wally: Always keep that round in the chamber. Asok: You scare me, but in a good way.
2015-01-10 * Alice: I only slept three hours last night. Topper: That's nothing! I used kunk fu to divert an asteroid that was on a collision course with Earth. Alice: Topping needs to be in the same category! Topper: Only if you're bad at it! Hoo-ha!
2015-01-09 * Woman: I want a guy who loves me for me, and not for the way I look... or the things I do. Dilbert: That doesn't leave me much to work with. Can I love you for your money and your vanilla-scented body lotion? Woman: You could love me for my mind. Dilbert: That might have worked two minutes ago.
2015-01-08 * Dilbert: I discovered that I can wear my tube clothing in the shower! It's like doing laundry and taking a shower at the same time! I can add one more efficiency, but I'd need to replumb the shower drain. Tina: Please stop talking!!!
2015-01-07 * Dilbert: Tube clothes! I invented that style! Did it reduce the number of decisions you need to make every day? Man: I'm just a guy wrapped in a carpet. Dilbert: Oh, I thought you were like me. Man: I don't have to take these insults.
2015-01-06 * Woman: I want to be in a serious relationship that can lead to marriage. Dilbert: Marriage is a financial contract. How much money do you have? Woman: All I have is me. Dilbert: So... more vendor than spouse?
2015-01-05 * Woman: Have you been dating a lot? Dilbert: I call it A-B testing. I go on dates and then compare the analytics to see who I click with most.Woman: What exactly are you measuring? Dilbert: Asks too many questions.
2015-01-04 * Boss: That doesn't sound like a work call. Carol: It isn't I don't have time to do my personal stuff on my own time. I have to do it on work time. Boss: I pay you to do work stuff, not personal stuff. Carol: Then how would I get all of my personal stuff done? Boss: That's not my problem. Carol: Then why did you bring it up. Boss: Because I need you to do work. Carol: I told you I can't get all of my personal stuff done if I do your work! Boss: Okay, okay. I probably could have handled that better.
2015-01-03 * Dilbert: I'm starting to realize that my market value as a single guy is higher than I thought. Men such as you have set the bar so low that all I need to do is have a job and be taller than most women in heels. I thought they were turned on by my tube clothing, but they actually like me for me. Wally: You're welcome.
2015-01-02 * Woman: Mom, this my date, Dilbert. He only wears tube clothes. Dilbert: For the efficiency. Whoa! Before I touch that paw, have you been to any Ebola hot spots lately? Woman: He has a job. Dilbert: My time has come!
2015-01-01 * Dilbert: I noticed you eyeing my tube clothes. You're thinking I am a man who values substance over style and it turns you on. Woman: No, I'm thinking I'll date anything that has a job. Dilbert: I have one of those!
2014-12-31 * Dilbert: I got 9,752 responses on this dating site and I haven't even completed my profile. All I said is that I'm six feet tall, I have hair and a job. Meanwhile, everywhere: Women: Hair... height... job! Triple threat!
2014-12-30 * Dilbert: I only wear tube clothes now because it reduces my number of daily decisions. Tina: You mean decisions such as... where to go on a date? Because I don't see that coming up. Dilbert: I'm still fine-tuning the system.
2014-12-29 * Dilbert: I call my invention "tube clothes." The idea is to eliminate as many daily decisions as possible, the way Mark Zuckerberg does with his gray t-shirt. I like to understand what makes people successful. Dogbert: And you narrowed it down to his shirt?
2014-12-28 * CEO: You didn't promptly respond to my email last night. Dilbert: You sent that email at 1 a.m. CEO: I expect my employees to be checking email at all times. Dilbert: Sleep is necessary for normal brain function. Anyone who succumbs to your leadership on this topic will turn into a dysfunctional moron in 48 hours. CEO: I don't see where you're going with this. It's all so confusing to my brain. So tired... can't stay awake... Dilbert: I don't usually win conversations this decisively.
2014-12-27 * Dilbert: We found the "bliss point" for immersive 3-D headgear. The product is so good that 87% of our customers starved to death while using it. CEO: We never get the customer retention part right.
2014-12-26 * Dilbert: Hey, Ted, how do you like our new 3-D immersive goggles you've been testing for two weeks? Some people say the experience is so good that you forget to eat. You're dead, aren't you...
2014-12-25 * Boss: I only have enough in the budget to hire an employee who is incompetent half of the time. But if I don't use the budget, I will lose those funds next year. Employee: And I am proud to say that I'm 75% competent. Boss: I wish I could afford that.
2014-12-24 * Tina: What are you talking about? It sounds interesting. Dilbert: We were saying that our smartphone technology has caused us to speed-evolve into a meta-organism that is the sum of our connected parts. Tina: In my defense, from across the room it looked interesting.
2014-12-23 * Carol: Warren Buffett says my career will be better if I show passion for my job. I'll have to fake the passion because everything I do in this job is mindless and boring. Later. Carol: Woo-hoo! I forwarded an email!
2014-12-22 * Boss: You need to show more leadership on your project. Dilbert: How do you know my leadership is a problem? Maybe the team members are bad followers. Boss: It's your job to fix it either way. Dilbert: The way you just fixed me with your leadership?
2014-12-21 * Coworker: I should warn you that I'm a perfectionist. Dilbert: I appreciate the warning. Do you have any other psychological problems or just the one? Coworker: I don't think of it as a "problem." Dilbert: I guess that's what makes it so bad. I see a lot of other psychological problems in your writing. Cognitive dissonance, unrealistic optimism, and some projection bias. But I can see why you think your perfectionism is the worst part. Coworker: You're a jerk. Dilbert: ...and here come the anger issues.
2014-12-20 * Tina: I heard you had a freak accident while vacuuming your house naked. Dilbert: That's a rumor. I don't know how that stuff spreads. Tina: Now I feel a little bad that I told thirty people.
2014-12-19 * Carol: Dilbert called to say he'd be late for your meeting. He said something about having a freak accident while vacuuming naked. Dilbert: Did you tell him traffic was bad? Carol: More or less.
2014-12-18 * The Marketing Guy. Marketing Guy: I don't see why engineers get paid more than marketing professionals. Dilbert: Maybe because engineers designed and built every important part of modern civilization and all you did was misrepresent it. Marketing Guy: My point is that you need both. Dilbert: You really don't.
2014-12-17 * Wally: I'm thinking of getting into the strategic planning game. If I understand the job description, you basically hallucinate about the future and then something different happens. Dilbert: You also have to pretend it's useful. Wally: Really? That sounds hard.
2014-12-16 * Financial Advisor. Advisor: Convertible notes... preferred stock... municipal bonds... covered call options. These are things you can never hope to understand. So trust me and try to forget that my only career ambition is to drain your account like a giant mosquito. Boss: That sounds reasonable. Advisor: I'm always surprised at how easy this is.
2014-12-15 * Financial Advisor. Dilbert: You kept me waiting in the lobby for five minutes. So I used that time to learn everything that matters in the field of personal investing. Did you know that you don't know much? Financial Advisor: I did not know that.
2014-12-14 * Wally: What's it like to work hard? I'm curious because the reward for hard work seems to be identical to the reward for pretending to work. It seems as if it would be demotivating to work so hard for no incremental benefit. If I had to pick one word to describe my day, it would be "relaxing." But you took the harder path, and for that, you have my respect. Alice: I don't want the respect of a loser! Wally: If it makes you feel any better, I don't actually respect hard work.
2014-12-13 * Boss: Hold on, I have a text message that is probably more important than you. I will demonstrate my power over you by handling a text message while you sit there, waiting for a pellet of my attention. Stop texting me! Dilbert: Mmm... pellet.
2014-12-12 * Wally: Here's the analysis you asked me to do. Coworker: Oops. I forgot to tell you, I don't need this anymore. Wally: What? I did all of that work for nothing? Coworker: Wait... this is nothing but... gibberish. Wally: That is a separate issue.
2014-12-11 * Wally: I would love to help you, Alice, but Dilbert says everything you are doing is misguided. Alice: What? I will whip him to death with his own intestines! Wally: Can you either do that quietly or wait until after my nap time?
2014-12-10 * Wally: I added air bags to my underpants to avoid accidental assignments. Boss: Hey, Wally, I need you to... Maybe I'll ask someone else. Wally: The system works!
2014-12-09 * Dilbert: I don't know how you juggle work plus a family. Carol: Spending time with my family is like fighting porcupines in a salt mine. I come here just to get away from them. Dilbert: So... you like your job? Carol: No, but at least I can go home to get away from it.
2014-12-08 * Wally: If you need me, I'll be in the cloud fixing a software issue. There's no cell coverage in the cloud, so it might seem to you as if I am at home doing nothing. If you need me, I'll be at home doing nothing. Dilbert: Why would anyone need you?
2014-12-07 * Woman: I like dancing and... Dilbert: I'm out. I avoid any relationship that has a risk of dancing. Woman: You're rejecting me because I like to dance? Dilbert: Yeah, it would start out all innocent... but two months into it you'd be guzzling wine and dragging me toward live music. Then you'd start doing all this... and this... and some of this... Woman: I also enjoy doing yoga to release my chakra energy. Does that bother you? Dilbert: I think I'm getting hives.
2014-12-06 * Dilbert: Your rule is that no more than eight people should attend a meeting, so I can't let you sit down. CEO: When did I say that? Dilbert: It was in a book you co-wrote. CEO: I knew I should have skimmed that thing. Dilbert: Your unknown co-author is quite wise.
2014-12-05 * Dinosaur: Ha ha! I am now the coolest member of the household because I have a smartwatch. Hello, watch. What time is is? Watch: This is the anthropocene epoch. Dinosaur: Wow, that carbon dates me.
2014-12-04 * Dinosaur: All I need is my flip phone, my Windows XP, and my basic cable television. Did I sound like a big, dumb dinosaur that time? Dogbert: Pretty much. Dinosaur: Wow... there is no cool way to describe my life.
2014-12-03 * Dinosaur: I don't own a smartphone. I use a flip phone because it does everything I want. Alice: Why are you proud of being a big, dumb dinosaur with a terrible phone? Dinosaur: I"m sensing a lot of judgement in that question. Alice: Wait until you hear my follow-ups.
2014-12-02 * Boss: Do you want some constructive criticism? Dilbert: No, but I would love some under-informed opinions about things you don't understand. Boss: That took a lot of fun out of it.
2014-12-01 * Boss: I couldn't open the document you sent with your project update. Wally: Try erasing your hard disk and reinstalling the operating system. Boss: I guess I don't need it that badly. Wally: After all the work I put into making that document?
2014-11-30 * Boss: Would you like some feedback on your performance? Dilbert: No. Boss: You're supposed to appreciate feedback because it makes you feel valued. Dilbert: How does listening to you belittle me about things you don't understand make me feel valued? Boss: Well, I don't know. It must be an indirect thing. Maybe we should just try it and see how it feels. Dilbert: Whatever. Boss: I don't actually watch you work, so I'm mostly guessing about the things you do wrong. I accuse you of being slow and disorganized! Is it working yet? Dilbert: Yes. If that makes you go away.
2014-11-29 * Alice: I feel so rested and relaxed after my vacation. I wonder how long this afterglow will last because... excuse me while I check this text message. Dilbert: Is it too late to start timing it? [Alice is on fire]
2014-11-28 * Dilbert: Let's skip to the part where I tell you how you should have engineered it instead of whatever you did. Coworker: You don't know what I did. Dilbert: I have five minutes to pretend that matters. Coworker: That's all I need.
2014-11-27 * Dilbert: If our business plan fails, do we have a plan for that? Boss: Only losers plan for failure. Dilbert: I think we're on the same page, but for some reason you're angry about it.
2014-11-26 * Boss: Are you going to meet the launch deadline? Dilbert: No. There were unexpected problems. Boss: I need solutions, not unexpected problems! Did that mean anything? Dilbert: Almost. Good try.
2014-11-25 * Dilbert: And I plan to launch the beta version in two months. Group: Ha ha ha ha ha ha ha ha ha ha ha ha ha! Dilbert: I have a credibility problem. Dogbert: And I should believe that?
2014-11-24 * Dilbert: The good news is that we had the winning bid for the project. The less-good news is that we don't make the product we just sold, nor could we make it for the price we bid. My plan is to put out an RFP to secretly subcontract the work to a bigger liar. CEO: That could work.
2014-11-23 * Wally: Is it okay if I telecommute on "Bring Your Kid To Work" Day? I'll show my kid how I work from home. Boss: How would I know you were working? Wally: What??!! That is exactly the sort of distrust that corrodes the motivation of employees! How can I feel good about my job in this toxic atmosphere? Boss: Okay, okay. You can work from home on "Bring Your Kid To Work" Day. Dilbert: You don't have a kid. Wally: I hear they ruin your naps.

I originally wanted to examine the most frequently-used words over the course of the strip, to see how the comic’s conversation topics evolved over the years. Unfortunately, I discovered that the dialogue isn’t separated by the panel of the strips, and the attribution of the dialog isn’t consistent, creating some challenges. The recent strips have the character name followed by a colon, followed by the dialogue (Dilbert:”Did you finish the TPS report?”), but in the older strips, this is done differently (Dilbert replies, “Did you finish the TPS report?”), so finding a way to filter this content was challenging. Moreover, looking over the frequently-used words in this data was far less interesting than expected, even after accounting for common words. I’m not sure exactly how I’ll visualize these results, so I don’t have a sketch. I may have to examine the findings in a different way.

https://github.com/ronkim/homework12

 

Matthew Kellogg – Data Scraping

I downloaded the text version of the IMDb (Internet Movie Database). From this I hope to illustrate a pattern of movie genre popularity vs movie age. For instance, I would illustrate a difference in popularity of current Sci-Fi films in comparison to older Sci-Fi films.

 

Here is a sketch of what this may look like.

data-sketch

An example of some data from the database is the following list of the some of the films Adam Sandler has been in.

Sandler, Adam (I)	101 Most Unforgettable SNL Moments (2004) (TV)  (archive footage)  [Himself]
			12-12-12 (2013)  [Himself]
			12-12-12: The Concert for Sandy Relief (2012) (TV)  [Himself]
			1996 MTV Movie Awards (1996) (TV)  [Himself]
			1999 Blockbuster Entertainment Awards (1999) (TV)  [Himself]
			1999 MTV Movie Awards (1999) (TV)  [Himself]  
			2001 MTV Movie Awards (2001) (TV)  [Himself]
			2002 MTV Movie Awards (2002) (TV)  [Himself - Presenter]  
			2004 MTV Movie Awards (2004) (TV)  [Himself - winner]
			2005 MTV Movie Awards (2005) (TV)  [Himself - Presenter]
			2006 MTV Movie Awards (2006) (TV)  [Himself]  
			2010 MTV Movie Awards (2010) (TV)  [Himself - Presenter]
			2012 MTV Movie Awards (2012) (TV)  [Himself - Presenter]
			2013 MTV Movie Awards (2013) (TV)  [Himself - Presenter]
			2014 MTV Movie Awards (2014) (TV)  [Himself - Cameo #14]  
			50 First Dates (2004)  [Henry Roth]  
			A Celebration of Bob Barker's 50 Years in Television (2007) (TV)  (uncredited)  [Himself]
			A Day with the Meatball (2002)  [Adam Sandler]
			Adam Sandler Goes to Hell (2001) (V)  [Himself]
			Adam Sandler: What the Hell Happened to Me? (1996) (TV)  [Himself]
			Airheads (1994)  [Pip]  
			America: A Tribute to Heroes (2001) (TV)  [Himself]
			Anger Management (2003)  [Dave Buznik]  
			Bedtime Stories (2008)  [Skeeter Bronson]  
			Big Daddy (1999)  [Sonny Koufax]  
			Billy Madison (1995)  [Billy Madison]  
			Blended (2014)  [Jim]  
			Blossoms & Blood (2003) (V)  [Barry Egan]  
			Bulletproof (1996)  [Moses]  
			Ceremonia de inauguración - 56º festival internacional de cine de San Sebastián (2008) (TV)  (archive footage) (uncredited)  [Himself]
			Chabad: To Life Telethon (2008) (TV)  [Himself - Video Guest]
			Click (2006)  [Michael Newman]  
			Comedy Central Presents: The Commies (2003) (TV)  [Himself - Presenter]
			Comedy Club Shoot-Out: Vol. 1 (2006) (V)  (archive footage)  [Himself]  
			Coneheads (1993)  [Carmine]  
			Couch (2003) (TV)  [Couch Testing Man]  
			Cómo conseguir un papel en Hollywood (2007) (TV)  (archive footage)  [John Clasky]
			Der Weg des Ruhms (2011)
			Deuce Bigalow: European Gigolo (2005)  (uncredited)  [Javier Sandooski]
			Deuce Bigalow: Male Gigolo (1999)  (voice) (uncredited)  [Robert Justin]
			Diet Coke with Lemon Celebrates 40 Years of Laughter: At the Improv (2002) (TV)  [Himself]
			Dirty Work (1998)  (uncredited)  [Satan]
			Eddie Murphy: One Night Only (2012) (TV)  [Himself]
			Eight Crazy Nights (2002)  (voice)  [Davey/Whitey/Eleanore/Deer]  
			ESPY Awards (1996) (TV)  [Himself]
			Finding Sandler (2015)  [Himself]

This data is available for non-profit use via FTP and is in a fairly usable format, so I did not write a scraper/parser.

I spent a large amount of time on Temboo, trying to find something that sparked my interest. Whenever something came up I would try to use the api, but the supported API would not be as Temboo described it. An example of this is the SchoolFinder api. I was able to obtain a key, but the rate was not limitted to 2400 requests per day, but 200. Also, temboo could not make the requests as there was a one IP per api key rule enforced, so the API had to be used directly.