It's Rishi

People Who Read This Article Also Read… by Greg Linden of Microsoft Live Labs (and formerly of Findory.com) is a comprehensive review of the uses of recommendation systems on the Web and their implementations. Recommendation systems is a topic that I love and Greg’s descriptions of systems such as that of Google News was very educational.

I’m a huge proponent of the idea that the newspaper, with it’s one-size-fits-all news, is dead. I discussed this in my prior post, Ok, I admit it one size fits all news will die. In this prior post, I discussed the fact that I consume most of my news today using my RSS reader. I’ve added several news feeds, from many topic areas, that I respect and enjoy to my reader and I check it every few hours. I have found that over the past couple years, my awareness of current events in topic areas that I am interested in has risen considerably.

However, there are limitations to the RSS reader. “Rolling” your own news feed takes time to create and maintain. I don’t expect that many will do this. More importantly, though, the scope of the news that is available to me is bounded by the content of those news feeds which I have explicitly included. I don’t doubt that every day I miss news stories that would be of high interest to me because they originate from news sources that I am not following. A news application that can show me news from both my explicitly chosen news sources as well as news stories that come by leveraging recommendation technologies (e.g. “Story X is similar to news stories which Rishi typically reads” and “Story X is being read by many people who have similar news tastes to Rishi”) will be the ultimate solution for me. What’s exciting is that I expect such a news application to be available very soon…

The goal of any news delivery medium is to provide maximum signal-to-noise ratio to its target audience. “Signal” is the set of news items that is of interest to a person. “Noise” is everything else. The reality is that an infinitesimally small percentage of news is interesting to any given person. And that percentage is shrinking every day because more news is being created on a daily basis: more frequently are more people documenting more people who are doing more newsworthy stuff every day.

In order to keep SNR high, news mediums need to focus on the news interests of their audiences more intensely than ever before. However, trying to create a single focus for a group of individuals, each of whose interests differ somewhat, is not a long-term solution. Sites like PerezHilton.com, a leading Hollywood gossip blog, and TechMeme, a leading (especially here in the SV) tech news aggregator, provide a certain segment of the news to an audience specifically interested in that segment. However, over time, the amount of news created in the news segment grows and the the segment bulges. The news publisher either must choose to further narrow their segment, which will alienate some of their existing audience, or publish a higher volume of news, which ultimately lowers the SNR to any given audience member. Either of these options is not a good choice.

Long-term, the only news deliver medium which is viable is the roll-your-own news concept. Geeks here this and start throwing out terms like RSS and OPML but the bottom line is that you don’t have to know technology in order to determine whether a piece of news is interesting to you. Over the past months, I’ve found myself going to news sites, including TechMeme, less and instead refreshing Google Reader more. I’ve added many feeds and the news that arrives is astonishingly interesting to me. Most importantly, my Reader is astonishingly uninteresting to most other people. This kind of relevance is ultimately impossible to achieve by any news publisher that tries to appeal to more than a handful of people.

I don’t want you to conclude from this that I think the penultimate solution is the RSS Reader. The concept of explicitly adding feeds to a reader is just not going to fly with mainstream folks. So what is the perfect news medium that allows you to roll your own news but doesn’t require any tech savvy? Attempts have been made (NewsVine, etc..) but I think we have yet to see the killer news app.

Oh how mainstream media has changed over the past decades. Back in the 1960’s, during JFK’s presidency, news outlets wouldn’t publish any stories about the president’s infidelities. News editors had a sense of responsibility towards upholding the values and code of our society. There was no need to blemish the president’s name for little good would have come from it. Back then, news was controlled by a handful of agencies. Not only did these agencies have control over what news was received by citizens across the nation, but also when they received news. There were no 24-hour cable news channels and of course there was no Internet.

The landscape of news exchange/delivery today could not be more different. Major news outlets source and publish news around the clock and around the world. Americans are able to receive news wherever and whenever. News is no longer thought of as a single collection of headlines that you consume at once. Instead, news is a continuous flow of stories and headlines that is streaming whether you’re there to catch it or not. The consumption of news went from being a 30-minute event each morning or evening to being a virtually constant activity. How did this happen? Where is all this news coming from?? From two places:

The world shrank – Digital information networks enables news to efficiently travel across the globe in an instant. Now only can data travel at the speed of light, but there is a connected path from the news source to the news consumer. Often, very little human intervention is involved.

Citizen journalism – Digital cameras/videocams, camera phones, laptops, and wireless connectivity allow every one of us to capture the events of the world. I would venture a guess that the majority of Americans under the age of 30 now have atleast one device capable of digital capture with them at all times. We then take this digital information and disseminate it to the world via social-networking sites, blogs, online photo albums/streams, YouTube, message boards, etc.. An average citizen doesn’t have the reach of NBC or CNN, but as is seen every day on the Web, viral citizen media can spread like wildfire and ultimately achieve the same or greater reach as a mainstream media broadcast.

With so much news being created and so many new ways by which news can be spread, there is tremendous competition for people’s attention. I’m not suggesting that the big media companies are going to be extinct any time soon, but I am suggesting that their role in society is. Let’s face it, NBC was thrilled when Cho’s package arrived in their mailroom. NBC said they spent hours deciding whether to air footage from Cho’s videos on the air. There’s little doubt in my mind that they were going to broadcast it. How could they not? The fact that Cho chose to send the package to NBC affirms NBC’s stature as a dominant media outlet. The only issue that they may have been wrestling with was whether to air it and get a backlash from the public, politicians, or special interest groups who might denounce NBC for sensationalizing the Va Tech shooter. However, if NBC didn’t air the footage, they would have no doubt posted it on their news website, MSNBC.com. I’m sure the NBC execs realized that if they didn’t release it, eventually the material would at some point get leaked and in this case, NBC wouldn’t get the limelight for having the scoop.

If Cho would have simply posted all his videos to a MySpace page or YouTube, he would have demonstrated that the big media companies are simply becoming irrelevant. But, whether he knew it or not, what he did was smart. He knew that NBC would whore out the video footage as much as it possibly could since they would have the exclusive and others would inevitably do the job for him of ensuring that the video got on MySpace, YouTube, etc… The reach of his videos was maximized as a result.

Unlike 40 years ago during JFK’s presidency, the media companies a) can’t afford to ignore stories which will garner them attention and b) simply have little to no control over what stories make it to the public. If they don’t cover a story, someone else will. AOL Time Warner realized this a couple years ago and launched TMZ.com. TMZ.com is a hollywood news/gossip site that basically runs stories that AOL Time Warner couldn’t on their mainstream sites. TMZ.com stories often lack the journalistic integrity that a mainstream news organization would want to uphold. AOL Time Warner knew that this segment of news was too much in demand and too lucrative to ignore. And they were right: TMZ.com has been enormously successful and one of the fastest growing blogs on the Web. Moreover, TMZ.com relies heavily on citizen- captured stories, photos and videos and not a dedicated news team. TMZ.com is an example of an old media giant embracing the fact they are losing control of the news rather than trying to combat this fact. There can be little doubt that other media giants will follow suit with sites of their own which embrace citizen media.

A big part of being a trusted news source is providing comprehensive information. Increasingly, this means relying on sources beyond a dedicated news team. Dedicated news teams simply will not be able to scale to meet the volume of news consumption in the future. News sites like TMZ.com, which rely on citizen journalism, can scale and will be a crucial strategy for the big media companies to maintain their significance.

Hmm I know I’ve got some more thoughts on this but enough for now… =)

I was composing an e-mail reply to someone (the person reading this will know who he is) and what I intended to be a short e-mail on the topic of conversation in the blogosphere ended up sprouting into this long rambling. I realized that I wanted to throw it on my blog for viewing by anyone who might find it interesting:

On any given Monday morning, thousands of people gather around water coolers at offices around the country to chat about the “Desperate Housewives” episode that aired the night before. On a daily basis, radio shows around the country host discussions covering the same events in news and politics. In these examples, because of physical limitations, the number of people that can engage in any one of these conversations is limited. That’s why many people flock to the Internet to discuss these same topics with a broader scope of people. Ultimately, the “perfect” conversation is when everyone interested in a topic, can engage in a single, dynamic conversation. It is often the case, however, that in the blogosphere, at any given moment in time, many blogs will be covering the exact same topic. The result is that there are many, duplicate conversations going on – just like what happens in the offline-world, as I described above.

A new service that made some commotion over the weekend, coComment, helps to facilitate conversation on a single blog. Also, services like Memeorandum help connect blog posts by finding memes thru back-links and track-backs. While services like these help to connect opinions, the problem is far from solved. Person a who comments on blog A may be stimulated by a comment from person b on blog B, but the two people are likely to never read each other’s thoughts.

As we all know, blogs compete for attention. More specifically, each conversation is competing for attention from other conversations about the same item. Every blogger would rather have a comment posted on his blog, and see his own comment thread grow, rather than that happen on another blog.

The blogosphere isn’t the only form of discussion on the Web. Far from it actually. There are many other discussion environments, most which are centralized. The best example of centralization are message boards (and if you think message boards are on the decline, do yourself a favor and check out some stats at Big Boards) where the entire conversation (both people and content) is centralized.

Message board culture is very different from blogosphere culture. People who visit and post in message boards do so because they like to be part of a community and for the entertainment value that is had by engaging in intelligent conversation about things of interest to that person. The blogosphere, which is sort of the opposite of message boards in the sense that people and content are decentralized, has a somewhat different culture. For many bloggers, their online identity is their blog (and the content that they publish on it). Most bloggers blog for the purpose of promotion of their identity – whether it’s their social identity or professional identity. The key advantage of a blog, in terms of building an indentity, is the very fact that all the content a blogger writes is explicitly connected to his blog, and thus his identity. This is not to say that bloggers don’t care about intelligent conversation, it’s just that bloggers have this additional motive of building identity.

By building our identity, we can increase the attention that we garner from our peers, and thus increase our value amongst our peers. This is true in both the message board case and the blogosphere case, except that in the former, the community of peers is small and isolated so building identity in this case has, in a sense, limited and finite value.

So the big question is, does blogger greed inhibit the unifying of conversation? Are bloggers so hung up on building attention that they’d rather own their conversations instead of joining their conversations together for the benefit to those involved in the conversation?

Anyways, I’m not sure I brought my points together as well as I wanted but it’s 5AM and I need to sleep. But I’m really hoping you read this post and tell me your thoughts.

At 4:02PM (Eastern), Google posts their Q4 earning results on the Business Wire. The big, big news (definitely the biggest news out of the valley for today) is that their numbers fell short of consensus estimates. At 5:11PM Reuters posts their summary of this news item and at 5:29PM, AP does the same. At around 5:30PM, this news cluster lands on Google News under the Business section. At 5:45PM, a CNN (via CNNMoney) writer has published an article covering this news.

It’s 6PM (Eastern), a full 2 hours since this news landed, and no sign of it on Memeorandum. This exhibits a limitation of pure algorithm-based aggregators is that in their attempt to maintain a high signal-to-noise ratio, they have a hard time grabbing big stories that are just breaking. However, I know several people that consider Memeorandum to be the best source for real-time Tech-business news. Clearly in this case, it is not.

What’s somewhat amusing about this is that the first news organization to post a follow-up to Google’s own announcement was not even US-based. It was The Financial Times, a London-based publication.

UPDATE: At around 6:15PM (Eastern), the news hits Memeorandum. The head story is the AP article and it has a couple posts from the blogosphere connected to it. I’m guessing what happens is that since there’s tons of news items posted by AP every day, there’s no way to isolate immediately which few are actually big news. Big news publications, in this case like CNN or TheStreet.com, publish fresh copy on the news and do not generally back-link. So, unless you are clustering news by relevance, you’re not going to be able to figure out what’s big until bloggers, for which back-linking is common practice, start posting about it.
Also, one could certainly argue that for 99.9% of people, a 2 hour delay is totally justifiable especially if it means keeping a high signal-to-noise ratio. I know for myself, this would usually be my preference as well.

It’s approaching 3AM right now and I’m not asleep. In fact, over the past year, my sleeping time has gotten later and later and later. Why you ask? Partly it’s because I’ve been busy working on my startup Dontbuyjunk and I’m often working late into the night until I’m satisfied with the progress that I’ve made for the day. But, I’m increasingly finding that what really is preventing me from getting to bed is information overload courtesy of the Internet. Let me explain.

I’ve been spending hours per day on the Internet for several years now. The big difference though is that recently the time I spend is shifting away from entertainment (mindless chatting on message boards, gaming, etc.) to information exchange activities such as reading/writing in the blogosphere. Every night, after I’m done working, I do one last catch up with my RSS reader and almost without fail, I end up spending a couple hours bouncing from one blog to the next and then to aggregators like del.icio.us and memeorandum.

Today, publishing (via the Web) is essentially free. And when I say “free” I mean that it both has no cost and is without rules or barriers. Furthermore, the second you publish your content, it is instantly accessible to a billion people. Because of all this, the rate at which information id created and disseminated is astonishing. So this is a good thing right?

Well…sure. enabling people to express and share both knowledge and opinions is great for society in countless ways. The problem that develops is that with so much publishing going on, how can I keep track of that tiny subset of information that is relevant, unique (remember that the majority of content published everyday is either syndication or basically duplicate) and valuable in my world? It’s getting harder by the day. Further exacerbating my problem is the wanting to not just read the facts behind a topic/news bit, but also read the opinions and participate in the many insightful discussions that branch from it.

So what’s the solution to my problem? Lunesta? Maybe. The next-generation of aggregators? Bingo.
One big trend that we are starting to see develop and I believe will be a major area of focus in the years to come is in information filtering and aggregation. Search engines like Google and centralized information sources like ESPN and Wikipedia allow me to pull in specific pieces of information when I am actively seeking it. However, their limitation stems from the fact that most of the information I absorb on a daily basis is new and could not have been searched for. In other words, if I didn’t know the information existed, how could I have searched for it? Instead, I must rely on my set of trusted sources to push this new information to me. Information aggregations, either human-derived (digg, reddit, del.icio.us) or algorithmic (memeorandum, blogniscient, Google News), are a step in the right direction. But aggregators have a long way to go before they truly are accurate and encompassing tools for information.

Anyways, it’s now 4:30AM and I’m basically just blabbing. Aggregators is an area that I’m becoming increasingly interested in myself and I have some of my own ideas brewing in my head about what the perfect aggregator would be and how it would work. I’ll be thinking and blogging about it in the coming weeks.

For some more discussions on aggregators, check out a blog post on memeorandum I was reading earlier that I found insightful:

http://mashable.com/2005/11/08/hacking-memeorandum-more-proof-that-algorithms-dont-work/

Be sure to read the comments thread.