Category Archives: 12-datascraping

mileshiroo

27 Jan 2015

Google Image search results for “beautiful”

 

I chose to collect the 781 images that result from searching “beautiful” on Google Images. I am interested in the way that these images consistently violate my personal sense of beauty, despite being among the most powerful (visible) beautiful images on the web. My first attempt used the Google Image Search API, but I soon discovered that this API limited me to 64 images, when simple inspection of the results page showed many more images. I then tried downloading the .html of a given results page, and scraping that file for image urls. What I didn’t realize at the time was that a given .html file only shows 100 image urls at a time. David Newbury explained to me that the image urls are dynamically loaded as the user scrolls, so it isn’t possible to download them in one go. Together, we inspected the network activity of the results page, and found requests of the form:

https://www.google.com/search?q=beautiful&safe=off&sa=X&biw=1595&bih=894&tbm=isch&ijn=1&ei=7xzHVKugCMyvggT7xIDgAw&start=100

From there, I modified the values in ‘n=1’ and ‘start=100’ eight times to download 781 total image urls. I used the ‘cat’ unix command to combine these 8 .txt files into one file, and then used Python to scrape this file for the urls.

View code on GitHub.

Here is a sketch that identifies categories of normative beauty by grouping related images.

EDIT: What I’d like to do with these images is display them in a grid, and use the Eyetribe eye tracker to log the time spent looking at the images. Then I will change their scale to match gazing time. This way a person can imprint their subjectivity onto the collection of images. (I’m writing this here so I remember to do it.)

chen

27 Jan 2015

I’m always unsatisfied with China’s website banning policy, so my topic is about the great firewall. The most difficult part for me is to decide which part I should critique and what kind of data I should collect.

I intended to collect data about which website is banned and how long since it was banned. But it’s hard to tell the exact date — we don’t have this kind of data.

Then I thought maybe collecting data of posts over time which were pinned inside China in some banned websites, like Instagram in this case, might be a good idea.

I used a lot of time to label the latitude and longitude of a precision of 0.001 inside China mainland, and after I spent 1 hour on this work, I realized it was too trivial. So at last I just collected data from three major cities — Beijing, Shanghai and Guangzhou.

It is the first time for me to do data collection and scraping work, and it’s really interesting to decide the topic. And through this project, I feel I have more insights of data I collected, and I’d love to present and visualize it in the future.

Here are some positions I collected using Instagram (most of them are just restaurants:) ):

羊坊镇
阳坊胜利涮羊肉(胜利总店)
后白虎涧村
Yangfang, Beijing, China
阳坊大都涮羊肉(大都总店)
阳坊大都饭店 Yang Fang Da Du Hotel
白虎涧 Baihujian
Great Wall of China
Isle of Skye,Scotland
沙河水庫
沙河基地科技园区
百晨生态园
沙河大坝
先力片場
Mobil Ave
北亚花园
北亚小区
物美超市
China Beijing International Acupuncture Training Center
Grill 29
北七家
Braja ashram
八仙宾馆影视基地
Beiqijia
俄罗斯餐厅
Chateau Laffitte Hotel
Garden Hotel
虎鳄农场
福寿斋
福寿斋清真烤鸭店
顺丰优选北京仓
Casa da vovó Alzira
佰吉天园
beijing air harbor hotel
上井日式料理
海港一号
中国东方航空营销委员工食堂
沪渝高速
海港一號
帕缇欧香苑北门
Shanghai Qingpu
梦丹苑
Qing Pu Paladset
崧泽古文化遗址
Home
重庆家富富侨足浴
Longde Road Station
Longde Road Station
Longde Road Station
英倫風尚
DHJ Interlining (Chargeurs Interlining) | 迪志衬布(上海)
Ai Mei Chinese Restaurant
ZF China (Investment) Co.
阔客仕
好又多
Big Shot Beef

The not so interesting data(like position IDs and posts amount over month) can be found on github.

Here is my github repo.

My visualization plan includes a Chinese map, with the color showing the user post amount.

oie_QwSYsiVfeCP7

pedro

27 Jan 2015

compressed

In this project I want to analyse visually the landscape surrounding streets, avenues and highways. In the end of the 1960s, Robert Venturi, Denise Scott Brown and Steve izenour took many students from Yale to understand the emerging urban spaces of Las Vegas. With different resources such as diagrams, maps and pictures they tried to decipher the logic underlying the urban patterns and structures.

Well, with Google large data base it is possible to access a large set of photos based on its coordinates and angles.  With Temboo I scraped the data of some directions in Google maps. The resulting file contains coordinates and distances from each node of the path. The next step is to connect these points and distances with the streetview image, what is possible with Google API.

},
{
“distance” : {
“text” : “0.5 km”,
“value” : 549
},
“duration” : {
“text” : “1 min”,
“value” : 67
},
“end_location” : {
“lat” : -23.5654915,
“lng” : -46.7118095
},
“html_instructions” : “Continue onto \u003cb\u003eBR-116\u003c/b\u003e”,
“polyline” : {
“points” : “ntxnCfmb|Gh@Ax@CnBEb@AjAAhAArCCjACjACnACnAA~@El@APA”
},
“start_location” : {
“lat” : -23.5605573,
“lng” : -46.71203879999999
},
“travel_mode” : “DRIVING”
},

At first sight, there are some challenges, as how to get intermediate coordinates with precision, how to select the angles, etc. In a ordinary trip inside a city, the file contains hundreds of lines and just dozens of nodes. So, I’ll have to choose between creating many paths inside the same city or choosing very long trips between urban centres.

In the end, I would like to use morphing techniques and visual analysis to compress series of images of certain paths in a single image, to understand both the distribution of the buildings, the interface between public and private and also the predominant colours of the environment.

LValley

27 Jan 2015

In the year 2005, stay at home mother Stephanie Meyer published the first book in a series that would quickly become a global phenomenon. Twilight — the word, the vampire connotation, the fandoms — has swept the world like a fungal disease (only with less fungus).

Fan culture is something I am very interested in, and I thought it would be revealing to see how the fandom, more specifically the “Team Edward” cult associated with the series, has evolved.

Before the release of the movie in 2008, the series had already become a sensation, but on sites like youtube, fans were limited without movie clips to create Twilight content with.

For the project, I scraped Youtube videos with the keyword “Team Edward” from both before and after 2008 (the release date of the first movie) to see how the Edward fandom has progressed.

DS_Diagram

Currently, I am playing with the idea of either taking elements of these pre/post Twilight Movie videos and splicing them together to create one super video.

Another idea I had was to order these videos by length or similarity in title and playing them side by side.

Screen Shot 2015-01-26 at 11.52.30 PM Screen Shot 2015-01-26 at 11.48.49 PM               (Team Edward Video Pre-2008)                             (Team Edward Video Post-2008)

 

Info for pre-2008 Videos:

 

“channelTitle”: “alyandaj1010”,
“description”: “This is a Jonas Brothers S.O.S. Please come before the Year 3000. I’ll Hold On for just A Little Bit Longer. Youre my One Man Show. When I see you I’m going …”,
“url”: “https://i.ytimg.com/i/6zy1d7JiVCByOg-PNykGOA/1.jpg”
“channelTitle”: “ElvenWolf91”,
“liveBroadcastContent”: “none”

“title”: “teamedward”,
“description”: “”,
“url”: “https://i.ytimg.com/i/w9EgiXpz9GtaRTXZ5X6igQ/1.jpg”
“channelTitle”: “teamedward”,

“title”: “I Dislike Jacob Black! Team Edward Cullen!!!!”,
“description”: “i really dont likfe jacob black! some of you make hate mail me or say crap but ur not gonna change whats been already done! Jacob Black is an annoying puppy …”,
“url”: “https://i.ytimg.com/vi/hnVPcCEQk9w/default.jpg”
“channelTitle”: “leftxmyxspritxcoldx”,

“title”: “dora1569”,
“description”: “I love Michael jackson he is so awsome i would love to have meet him in person.”,

“title”: “MsLissy”,
“description”: “”,
“thumbnails”: {
“default”: {
“url”: “https://i.ytimg.com/i/aP3FA2SRKFTUdv7qwS8QVA/1.jpg”

“title”: “EDWARDCULLENSOFiNE”,
“description”: “”,
“url”: “https://i.ytimg.com/i/tRC5LZkzwTdk3RHuT4_Wzw/1.jpg”

“title”: “idefixy7”,
“description”: “”,
“url”: “https://i.ytimg.com/i/iZ4atcUSQXFQuNub3cOMeA/1.jpg”

“title”: “GaaraGourd21”,
“description”: “Hello, I’m , Noah. I’m a fashion addict! You may know my sister FruitsBasketIsCute, a.k.a. FB-Chan. If you like her than you’ll love me! X3 I’m always open t…”,
“url”: “https://i.ytimg.com/i/wB8vqjpxaF4-fsoEMIAPUw/1.jpg”

“title”: “0Luna0uchiha0”,
“description”: “º¤ø„¸¸„ø¤º°¨¸„ø¤º°¨ ¨°º¤ø„¸ ♥Adam♥ ¸„ø¤º°¨copy and paste ¸„ø¤º°¨ ♥Lambert♥ `°º¤ø„¸if you think Adam ¸„ø¤º°¨¸„ø¤º°¨¨°º¤ø„¸¨°º¤ø„ is the BeSt!! Its a Mad Wor…”,
“url”: “https://i.ytimg.com/i/3JowzzFrVii0m6wNxh8now/1.jpg”

“channelId”: “UCRX2Yra_JsyESZF8bHzN0aQ”,
“title”: “Twilight Team Edward, Team Jacob, and Team Bella? Spoiler Alert!”,
“description”: “Twilight Team Edward Team Jacob and Team Bella? Music: http://freeplaymusic.com (c)2008 DuskTilDawnFilms.”,
“url”: “https://i.ytimg.com/vi/zPOdZQTAyQU/default.jpg”

“title”: “omgjobros”,
“description”: “http://youtube.com/stream?u=omgjobros&i=0 im agianst team jacob jk btw my new website is www.gofish4fun.webs.com.”,
“url”: “https://i.ytimg.com/i/8dexLoXyQHB-Ks5yxPaf1Q/1.jpg”

“title”: “Team Jacob and Team Edward Breaking Dawn Party Merritt Books”,
“description”: “Team Jacob and Team Edward Breaking Dawn Party Merritt Bookstore Millbrook August 1, 2008. Which team won? Great shirts made with metalic paint.”,
“url”: “https://i.ytimg.com/vi/6oCN83EOoEY/default.jpg”

“title”: “omgjoeishot”,
“description”: “Guess what? I’m going back to Camp!! I can’t wait! I’m gonna have more videos and I’ll post the actual show this year if I’m lucky!Videos probably won’t be r…”,
“url”: “https://i.ytimg.com/i/4qMPWKtiyqceoV53cLV9Pg/1.jpg”

“channelId”: “UCILXMyrprqJuUJFWftFwPEw”,
“title”: “jerseygirlpeacelover”,
“description”: “dance chica!!!! funky, punky style!!!!! wacky, fun, weird !!!!! :) :) :):) im a dancer i dance with dancenergy i have 2 scholarships ! TWILIGHT SAGA!!!!!!! i…”,
“url”: “https://i.ytimg.com/i/ILXMyrprqJuUJFWftFwPEw/1.jpg”

“title”: “allisson56”,
“description”: “hey there; my name is allisson(lol) i live by the east coast i love TWILIGHT AND THE REST OF THE BOOKS OF COURSE I AM 100%(EVEN MORE) TEAM …”,
“url”: “https://i.ytimg.com/i/vP0FkgY1Pz9KSdb7lEYSqg/1.jpg”

“title”: “Team Edward or Jacob. *SPOILERS*”,
“description”: “This is a video I made so you can vote for which team you are on in Twilight. If you are neutral or really don’t care, you can be Switzerland. Cast your vote…”,

“channelId”: “UCEoxzcSQxY3TvIklNG2V4Mg”,
“title”: “Team Edward Forever – Edward Cullen Fans”,
“description”: “So… one day… I decided I FINALLY did want to make a video. and of course, i did it on Edward Cullen :D I love him (: It is dedicated to all of the Edward…”,
“url”: “https://i.ytimg.com/vi/dAScmHnL9Vs/default.jpg”

“title”: “twilightvampiregirl”,
“description”: “i’m 12 years old and i love twilight!!!!! and new moon and eclipse!!!! and the host!!!!! and no doubt!!!!!! i saw stephenie meyer on may 20th and it was amaz…”,
“url”: “https://i.ytimg.com/i/NFzxf7Hj8gk3UwKr2f6R8w/1.jpg”

“title”: “Team Edward or Team Jacob?”,
“description”: “Team Edward or Team Jacob? which side are u on?”,
“url”: “https://i.ytimg.com/vi/Y7WQ2V9rYUM/default.jpg”

“title”: “Team Edward!!!”,
“description”: “This is a sample of Team Edward (aka fans of Stephenie Meyer’s gorgeous vampire Edward Cullen). Edward and Bella is TRUE LOVE! :D.”,
“url”: “https://i.ytimg.com/vi/0w7eIfMPAEY/default.jpg”

“title”: “Team Jacob vs. Team Edward”,
“description”: “this was some of the debate for team edward and team jacob at the breaking dawn release party!”,
“url”: “https://i.ytimg.com/vi/H6HvOxfgaHs/default.jpg”

“title”: “Team Edward~Edward & Bella~A Love Story”,
“description”: “Fan-vid for Twilight Saga. Team Edward!”,
“url”: “https://i.ytimg.com/vi/rhBdSpH01fw/default.jpg”

“title”: “Twilight Collab : TEAM EDWARD!”,
“description”: “Don’t forget to check out little miss vintageortackys Team Jacob look! http://www.youtube.com/watch?v=L4EE40-qy3Y I used : NYX jumbo pencil in Pots & Pans …”,

“title”: “Twilight Team Edward or Team Jacob?”,
“description”: “this is my first movie so it’s pretty bad. i didnt make or draw any of the pictures exept the one with Edward and the diamonds. the song is ‘its not just mak…”,
“url”: “https://i.ytimg.com/vi/Ik9LkYHaHP4/default.jpg”

“title”: “Shelbell99”,
“description”: “Thanks for visting my channel! Feel free to check out my videos and to subscribe! I am more busy now so you won’t really get new vidoes from me, but I will t…”,
“url”: “https://i.ytimg.com/i/zl8V5XRmKeXLRYFHuoZGjg/1.jpg”

“title”: “mrsorlando33”,
“description”: “Crazy? No…. Crazy about Twilight? YES. ~*♫`~*♫`~*♫`~*♫`~*♫`~*♫`~*♫`~ * When life hands u lemons, throw them back and yell \”I WANT ROBERT …”,
“url”: “https://i.ytimg.com/i/1U7ysJ583p2SuAn5MySAmQ/1.jpg”

“title”: “venus344”,
“description”: “90% percent of teens would have a breakdown if miley cyrus, was standing on the edge of a tower ready to jump, copy and paste if your in the 10% that would y.”,
“url”: “https://i.ytimg.com/i/2A7-uGoma3Mk9JHZNgb6ig/hq1.jpg”

“channelId”: “UCsMffDp8VFLh36t7T02IRpg”,
“title”: “alyandaj1010”,
“description”: “Edward♥Bella TEAM EDWARD ! ╔═╗ Edward Cullen is my …”,
“url”: “https://i.ytimg.com/i/sMffDp8VFLh36t7T02IRpg/1.jpg”

“channelTitle”: “alyandaj1010”,
“description”: “This is a Jonas Brothers S.O.S. Please come before the Year 3000. I’ll Hold On for just A Little Bit Longer. Youre my One Man Show. When I see you I’m going …”,
“url”: “https://i.ytimg.com/i/6zy1d7JiVCByOg-PNykGOA/1.jpg”

“channelTitle”: “ElvenWolf91”,
“title”: “teamedward”,
“description”: “”,
“channelTitle”: “teamedward”,

“title”: “I Dislike Jacob Black! Team Edward Cullen!!!!”,
“description”: “i really dont likfe jacob black! some of you make hate mail me or say crap but ur not gonna change whats been already done! Jacob Black is an annoying puppy …”,
“thumbnails”: {
“default”: {
“url”: “https://i.ytimg.com/vi/hnVPcCEQk9w/default.jpg”
“channelTitle”: “leftxmyxspritxcoldx”,

“title”: “dora1569”,
“description”: “I love Michael jackson he is so awsome i would love to have meet him in
“url”: “https://i.ytimg.com/i/rExsR66K4HGSVt-VNoVhzw/1.jpg”
“channelTitle”: “dora1569”,

“title”: “MsLissy”,
“description”: “”,
“url”: “https://i.ytimg.com/i/aP3FA2SRKFTUdv7qwS8QVA/1.jpg”
“channelTitle”: “MsLissy”,

“title”: “TeamEdwardCullen”,
“description”: “Team Edward Cullen v. Team Jacob Black To Join Team Edward Cullen (TEC): 1) Subscribe to me to keep updated with information about the movie, books, etc.”,
“url”: “https://i.ytimg.com/i/fIOikEGRBDh9y81LZP3o7A/1.jpg”
“channelTitle”: “TeamEdwardCullen”,

“title”: “outsidersarah101”,
“description”: “FAV GREASERS IN ORDER: -PONYBOY CURTIS -SODAPOP CURTIS -JOHNNY CADE -DALLAS WINSTON -TWOBIT MATTHEWS(Im soo sorry i forgot how to …”,
“url”: “https://i.ytimg.com/i/h2qMmmXFFF3bKqsJ8UmNcw/1.jpg”
“channelTitle”: “outsidersarah101”,

“title”: “EDWARDCULLENSOFiNE”,
“url”: “https://i.ytimg.com/i/tRC5LZkzwTdk3RHuT4_Wzw/1.jpg”
“channelTitle”: “EDWARDCULLENSOFiNE”,

“title”: “idefixy7”,
“url”: “https://i.ytimg.com/i/iZ4atcUSQXFQuNub3cOMeA/1.jpg”
“channelTitle”: “idefixy7”,

“title”: “GaaraGourd21”,
“description”: “Hello, I’m , Noah. I’m a fashion addict! You may know my sister FruitsBasketIsCute, a.k.a. FB-Chan. If you like her than you’ll love me! X3 I’m always open t…”,
“url”: “https://i.ytimg.com/i/wB8vqjpxaF4-fsoEMIAPUw/1.jpg”
“channelTitle”: “GaaraGourd21”,

“title”: “0Luna0uchiha0”,
“description”: “º¤ø„¸¸„ø¤º°¨¸„ø¤º°¨ ¨°º¤ø„¸ ♥Adam♥ ¸„ø¤º°¨copy and paste ¸„ø¤º°¨ ♥Lambert♥ `°º¤ø„¸if you think Adam ¸„ø¤º°¨¸„ø¤º°¨¨°º¤ø„¸¨°º¤ø„ is the BeSt!! Its a Mad Wor…”,
“url”: “https://i.ytimg.com/i/3JowzzFrVii0m6wNxh8now/1.jpg”
“channelTitle”: “0Luna0uchiha0”,

“title”: “Twilight Team Edward, Team Jacob, and Team Bella? Spoiler Alert!”,
“description”: “Twilight Team Edward Team Jacob and Team Bella? Music: http://freeplaymusic.com (c)2008 DuskTilDawnFilms.”,
“url”: “https://i.ytimg.com/vi/zPOdZQTAyQU/default.jpg”
“channelTitle”: “DuskTilDawnFilms”,

“title”: “omgjobros”,
“description”: “http://youtube.com/stream?u=omgjobros&i=0 im agianst team jacob jk btw my new website is www.gofish4fun.webs.com.”,
“url”: “https://i.ytimg.com/i/8dexLoXyQHB-Ks5yxPaf1Q/mq1.jpg”
“channelTitle”: “omgjobros”,
“title”: “Team Jacob and Team Edward Breaking Dawn Party Merritt Books”,
“description”: “Team Jacob and Team Edward Breaking Dawn Party Merritt Bookstore Millbrook August 1, 2008. Which team won? Great shirts made with metalic paint.”,
“url”: “https://i.ytimg.com/vi/6oCN83EOoEY/mqdefault.jpg”
“channelTitle”: “MerrittBookstore”,

“title”: “omgjoeishot”,
“description”: “Guess what? I’m going back to Camp!! I can’t wait! I’m gonna have more videos and I’ll post the actual show this year if I’m lucky!Videos probably won’t be r…”,
“url”: “https://i.ytimg.com/i/4qMPWKtiyqceoV53cLV9Pg/mq1.jpg”
“channelTitle”: “omgjoeishot”,

“title”: “jerseygirlpeacelover”,
“description”: “dance chica!!!! funky, punky style!!!!! wacky, fun, weird !!!!! :) :) :):) im a dancer i dance with dancenergy i have 2 scholarships ! TWILIGHT SAGA!!!!!!! i…”,
“url”: “https://i.ytimg.com/i/ILXMyrprqJuUJFWftFwPEw/1.jpg”
“channelTitle”: “jerseygirlpeacelover”,
“title”: “allisson56”,
“description”: “hey there; my name is allisson(lol) i live by the east coast i love TWILIGHT AND THE REST OF THE BOOKS OF COURSE I AM 100%(EVEN MORE) TEAM …”,
“url”: “https://i.ytimg.com/i/vP0FkgY1Pz9KSdb7lEYSqg/mq1.jpg”
“url”: “https://i.ytimg.com/i/vP0FkgY1Pz9KSdb7lEYSqg/hq1.jpg”
“channelTitle”: “allisson56”,

“title”: “Team Edward or Jacob. *SPOILERS*”,
“description”: “This is a video I made so you can vote for which team you are on in Twilight. If you are neutral or really don’t care, you can be Switzerland. Cast your vote…”,

“url”: “https://i.ytimg.com/vi/dP91Oop-LsY/hqdefault.jpg”
“channelTitle”: “theLionandtheLamb500”,

Github Link:

https://github.com/LValley/Twi-Finder

Thomas Langerak

27 Jan 2015

For this assignment I have taken a random artist (the Eagles in this case). Looked at the five most similar artists according to LastFM. For this five I looked again at the most similar. I did this for a thousand artists in total. To summarize 1000 artists were analyzed for their five most similar bands and their match value (a value between 0-1 on how well they matched). Giving a total of 10000 points.
For this assignment I have made use of the processing software, Temboo libraries and the lastFM API.

I have learned a lot during this assignment. Next to the usual planning learning point I seem to have, I again have taken a different look on how processing can be used. I never worked with this large amount of data acquisition and I think the knowledge this exercise gave me can be definitely helpful in the future.

I am not sure were to with the visualization. While running the gathering I noticed that it was very interesting how several bands had the same connections. I will try to do something with this. The most logical would be to make a spiderweb, yet I feel this is kind of old and done. I am aiming for something more interactive.

A sample from my data:
Capture

the code can be found at:
https://github.com/tlangerak/temboo_data/blob/master/IACD_assignment1.pde