Back in 1997 I had a short stint with a company called Excite.com. You may remember Excite, in the late 90’s they were one of the largest search engines, up there with Yahoo and pre-Google. My job was to catalog the Internet.
Yup, categorize and describe every website I came across for 5 hours a night.
I had a swing shift and started work at 6 in the evening. Every weeknight I drove to a large, steel-sided warehouse-looking building in Sausalito California. Upon entering this industrial hangar I was met with modern furnishings, an oversized front desk manned by a security guard (during earlier shifts there was a receptionist), and multiple levels of office space divided by glass walls and fabric covered cubicle partitions.
I was free to take any empty workstation, the chair usually still warm from the previous occupant. The computer was already on and I navigated to a group of folders organized by categories such as restaurants, auto dealerships, children’s toys, travel, etc.
Within each folder were subfolders of subcategories. I would keep clicking away until I reached the lists. The lists were collections of URLs for me to visit on the Internet. I don’t know where these lists came from or who made them, I only knew it was my job to visit all the websites on these lists.
When I visited a website from a list I entered information about the site into a cataloging application. I would verify the category, perhaps add a subcategory, and write a short description of what I thought the website was supposed to be about.
I assume the data we were cataloging was somehow being used in the Excite search engine. At the time I heard that most search engines employed a mix of algorithm generated and human reviewed search results. But I wasn’t really sure and no one else I worked with knew. We were all just drones being paid to surf the web for five hours. My manager may have known what was going on, but I think only saw him in the building once the entire time I was employed.
I was paid about $10 an hour to catalog every night for about five or six hours. It doesn’t sound like much money, but way back in ‘97 this was a very nice supplement to my day income. Plus it was fun and super mindless.
After about three or four months the plug was pulled. Without warning the project was declared over, at least for our office. I really didn’t get a chance to get to know anyone, only Steve, the security guard, who actually helped train me because my manager wasn’t around.
When the project was shut down we weren’t given a reason, but, being the smart guy that I am, I knew we had not cataloged the entire web yet. So it wasn’t because we ran out of websites, hhmmm…
Either Excite ran out of funding for the project, or someone realized what we were doing was pretty lame.
Yeah, I was disappointed when it was over. I really liked this gig, but I have to admit I thought I what we were doing was a waste. It was fun looking in all these different websites, and for some of them I’m sure I provided good descriptions and insight for the catalog. But for many sites I had no idea what they were about, I left the description blank or offered a best guess.
I know other people were filling the catalog with nonsense. There were times I picked up a list that had already been worked on. Apparently the company’s list queuing and workflow system was lacking (I don’t even know if there was a system for completed lists). I could see many people had entered garbage descriptions. I rewrote many entries, though I wasn’t asked to do so. I’m not sure why I did this; I know many of my own descriptions were halfhearted drivel.
It just goes to show the amount of time and money being spent probably did not yield the results Excite was hoping for. Who knows how much they spent on this project, I know in my office we had maybe 60 people cataloging (counting both day and night shifts), each being paid $10 an hour. So, two shifts, that’s about 14 hours times $10, by 5 days a week, plus the manager, plus the equipment, plus T-1 lines, plus a building lease, plus insurance, plus Steve.
And we were only one office. I wonder how many other offices like ours existed. Factor in executive planning, project management and software development. It could not have been cheap cataloging the entire Web.
Is cataloging the Web by humans a stupid idea? I thought so, until I learned about StumbleUpon.com a few years ago.
StumbleUpon is achieving what we were trying to do it at Excite (and what other search engines were trying to do at the time), a large catalog of human reviewed websites.
How many websites does StumbleUpon have in its index? Well, at time of this writing StumbleUpon as about 9,000,000 members, some members submit 100’s of reviews. I think I’ve reviewed about 80 sites so far myself.
And guess what, StumbleUpon doesn’t have to pay for website reviews, people do it for free. And I’m sure the quality of reviews is much higher at StumbleUpon compared to the reviews that Excite was paying for.
That’s pretty amazing if you think about it. Back then if I told Excite executives of a way to make their catalog a 1000 times bigger, of higher quality, all for free, they would have accused me of smoking the office plants.
It shows the power of online communities. This is nothing new to people that are into Open Source technologies. We have many amazing, free-to use, tools available thanks to the communities that build them.
Then there’s Facebook, which everyone knows makes it easy to interact with friends in different parts of the world, but it also has the power to connect the present to your past. I have dozens and dozens of friends who I thought I would never see or hear from for the rest of my life until I die.
In 1997 no one could imagine a StumbleUpon, a Facebook or a Twitter. It makes me wonder what the next big community on the Internet is going to produce for us.
Maybe it’s lifestyle design, which is what Tim Ferriss, author of the 4-hour Work Week, talks about, where we can have access to things we want without needing money.
Perhaps it’s a health community that connects people who have overcome illness and disease to people who are currently fighting the same afflictions.
Maybe you doubt we will have websites helping people overcome diseases, or learning how to get everything you want in life without money, but I certainly believe these things are coming, and things we haven’t even imagined yet too.
What do you think next big online community will do for us?
