Wingedpig.com - Mark Fletcher's Blog

The concept of aggregation is increasingly important on the Internet, as the sheer number of information resources increases. The average user wants to track more and more things on the Internet; an aggregator quickly becomes necessary as one's bookmark list grows to infinity. The first aggregators, what I call 'general purpose' aggregators, like Bloglines, Google Reader, and Newsgator, are focused on tracking blogs and news feeds, making it easy to subscribe to whatever blogs the user came across.

The new service FriendFeed has been getting a lot of attention the past couple of weeks. It's the latest in the line of what I call 'individual aggregators,' services that aggregate all the distributed parts of a person's on-line presence in one place. A person may have a blog, a Twitter account, a Flickr photostream. These services combine all of these items in one place. This trend started with Facebook's newsfeed, continued with Plaxo's Pulse, and then several other services, including Tumblr can do most of what the individual. These services are different than the general purpose aggregators in that they're focused on tracking individuals, not feeds. But the general purpose aggregators can do what the individual aggregators can do, because the underlying technology, RSS, is the same. It's really just a matter of user interfaces and a key bit of information.

The Problem

The individual aggregators collect a list of all of the distributed parts of a person's on-line presence. They ask each user to list their Twitter account, their Flickr account, their YouTube account, their blog. This list doesn't exist anywhere in a way that's machine readable. Each of the individual aggregators has to deduce this information and then maintain it. Or more specifically, each user has to maintain this information on each of the individual aggregators. Wouldn't it be better if this list existed somewhere under direct control of the user in a way where it wasn't siloed in a centralized, proprietary service? That way, every aggregator could take advantage of it and users would only have to update the list in one place.

A Modest Proposal

This problem is actually a general purpose version of a problem already solved by something called RSS Autodiscovery. In order to make it easier for general purpose aggregators to find RSS feeds to subscribe to, many publishers included a special line of text in the headers of their HTML. I have one on my blog:

<link rel="alternate" type="application/rss+xml" title="RSS" href="http://www.wingedpig.com/index.rdf" />

Aggregators know to look for this line, which tells them where the RSS feed for that blog exists. Can't we just extend this to include a list of all the other aspects of a person's identity? Have one line for each service the person uses, and change the title accordingly. So, I could include:

<link rel="alternate" type="application/rss+xml" title="Flickr Feed" href="http://api.flickr.com/services/feeds/photos_public.gne?id=35034347955@N01&lang=en-us&format=rss_200" />

for my Flickr feed. This doesn't have to only apply to services that publish RSS feeds. I could even do something like:

<link rel="alternate" type="application/twitter" title="Twitter" href="wingedpig" />

to indicate my Twitter account.

By doing this, the list of all the parts of a person's on-line presence is kept under the control of the person, associated with their blog. It's distributed, open, and easy to implement.

How To Make It Work

For this to work, a couple things need to happen. Blog publishing software has to be modified to ask for and then insert this information into the headers of a person's blog. Then,aggregators need to be modified to look for this information, and to periodically recheck it. The general purpose aggregators need to augment their interfaces to allow people to subscribe to these new feeds. But none of these things are terribly difficult to do.

Posted by Mark Fletcher at 09:42 AM | Permalink | Comments (4)

My friends over at Bloglines just launched a new version tailored to the Apple iPhone, called iBloglines. I've been using it the past couple of days, and it's great. Back when Bloglines first started, I got my first smartphone, a Treo 600. I developed Bloglines mobile so that I could use my smartphone to read feeds. It's great that there's an even better solution for iPhone users now.

Posted by Mark Fletcher at 11:58 AM | Permalink | Comments (1)

Well all, sadly it is time for me to move along from Bloglines. The service is in great hands with Ask and I am confident it will live long and prosper under their watch.

But "why" you cry? According to my ever-present Ask PR twin, "to spend more time with my family." And it's true; my cats have begun to run with a bad crowd. Beard Papa has some new stores that need visiting. And there are American Idol tryouts just around the corner and I need to get gussied up.

But the real reason is once a start up guy, always a geek (I know, say it ain't so!). So what's next? At some point I'll start another company; that's a difficult habit to break. But I'm also going to focus more of my time helping other startups and newbie entrepreneurs, something I'm finding increasingly rewarding. I haven't really discussed that aspect of my career much here on the blog, but I've recently been involved with two startups. When Plaxo first started, I gave them some technical help. More recently, I've been serving on the board of directors of One True Media, a startup that provides some great on-line tools to create and edit videos. I've also really enjoyed my recent speaking engagements, where I've been able to talk about the process of starting a company and some of the lessons I've learned.

In the meantime, Bloglines is in the capable hands of a great team at Ask, first and foremost led by Jim Lanzone, and including Robyn, Paul, Alan, Ben, Rob, Ryan, Andrew, Doug and Scott. Over the past year and a half, they've become the driving force behind the service and it is time to let them run with it. During that time, some of the major updates to Bloglines included: Package tracking, HotKeys, drag-and-drop feed management, Ping tracking, as well as vast and numerous improvements to the Bloglines infrastructure. And if that wasn't enough, just look at the new Ask.com Blog & Feed Search as an indication that there are lots of great Bloglines innovations to come.

I'm not disappearing. I'll be one of the speakers at this Saturday's Techdirt Greenhouse conference. I hope to see you there.

Posted by Mark Fletcher at 07:11 AM | Permalink | Comments (20)

This evening, we rolled out an industry leading blog search engine for Bloglines. In addition, there's a new 'Blogs & Feeds' tab on the main Ask site. We started working on the new search engine soon after Bloglines was acquired by Ask last year, and the engine is the result of a lot of hard work by both the Ask search team in New Jersey and the Bloglines team in California. Generating an index with over 1.5 billion archived blog posts, and keeping it updated with new blog posts (within 5 minutes of us receiving the posts) is non-trivial. Please check it out, it's quite an impressive piece of work.

Posted by Mark Fletcher at 10:48 PM | Permalink

Anyways, thanks again to everyone involved. Why did I have to leave early? To meet with my cat sitter. Sigh, what a life I lead. Anyways, there are more pictures of the gathering up on Flickr.

Posted by Mark Fletcher at 02:57 PM | Permalink

Also See Parts One, Two, Three, Four.

The move itself went almost perfectly. At 2pm, we took the crawlers off-line and started copying many of the databases. At 4pm, we took the entire site down and started copying the remaining databases. Around 5pm, wandering around barefoot, I broke my toe, but that didn't affect things (other than my toe). Around 7:30pm, it became clear that we would require an extra half an hour to complete things, so we updated the plumber page with the new estimate. Around 8:20pm, everything was back up and we completed testing the site. We took the plumber down at 8:30pm.

After the site came back up, we found a couple of small things that we didn't discover during testing, but nothing major. And over the past couple of weeks, we've continued to tweak and tune the service. The one scare we had happened the Thursday after the move, when fully half of our database machines decided to freeze up, all within half an hour of each other. The site still functioned ok, but it definitely scared us. The biggest unknown with the move, at least for me, were the new machines themselves. How many of the new machines would fail when put under load? We had done a lot of stress testing before the move, but we weren't able to completely test everything. Having half of the database machines fail almost simultaneously brought that fear to the foreground. Luckily it was a one time occurance, although we're still tracing the problem. Update: This happened again yesterday. For those interested, we think it's an issue with the ACPI support in the Redhat Enterprise Linux 4 Update 2 kernel on the Dell 1850s that we use.

So after all this work, what do we have? The extra hardware allows us to crawl every feed in the system twice an hour now. Also, the website is much more responsive. And we have room to grow. I think it was worth the broken toe.

Posted by Mark Fletcher at 07:50 PM | Permalink

Unfortunately, at a point after that, it became clear that we wouldn't hit the December 16th date; we'd most likely be ready two days later on Sunday December 18th. Ask Jeeves has a winter shutdown, which this year started on December 23rd and runs through January 2. We had a couple of options at that point: do the move on Sunday or one of the weekdays before December 23, or push the move out to the new year, most likely to January 6th. In my experience, user-based Internet services have two slow periods during the year: July/August and the last half of December. Because of this, and because moving to the new datacenter would greatly improve the user experience, we decided to push for the move to happen in December. We targeted Monday, December 19th to give us an extra day past when we estimated we'd be ready. And we decided to start the process at 2pm, which would hopefully let us finish without extending too far in the evening, while still avoiding the peak time of traffic to the site.

On Sunday, December 18th we put up a blog post announcing the upcoming downtime, and also inserted a link at the top of every page on the site alerting users to the downtime. One of the last things we did before the move was to have Ben, our UI/graphics guru, modify the Bloglines Plumber, giving him a pirate makeover. It was going to be a special downtime, and we wanted to make sure he looked good (we're fans of both Talk Like a Pirate Day and the Flying Spaghetti Monster).

At this point, I want to get a little technical. Don't worry, it'll only last a paragraph and you won't be quizzed. When planning a move like this, where a site will end up with a new IP address, you need to take some DNS issues into consideration. DNS is like the white pages of the Internet. It maps domains like www.bloglines.com to IP addresses, which are the actual machines. Each DNS record has a Time To Live, or how long the record is valid for (and how long you can cache the record before asking for it again). DNS records are cached all over the Internet, and many of these caches are broken. When planning this move, we did a couple of things:

1. A week before the move, we turned the TTLs down to 5 minutes.
2. Before the move itself, we put the Bloglines Plumber downpage up at the new datacenter.
3. To take down the site, we configured the webservers at the old datacenter to proxy to the new datacenter.
4. We then changed the DNS records to point to the new datacenter.

Ok, enough of the nerd lesson. I'll wrap this up next time.

Posted by Mark Fletcher at 11:53 AM | Permalink

The replication code started to stabilize, and we began copying blog articles from the old datacenter to the new one. This happened in fits and starts as we debugged the code. The fact that it happened without us having to take the site down was a great advantage. We also continued to test the Bloglines installation at the new datacenter.

Concurrently, we started working out the datacenter move checklist, enumerating all the items that had to be completed, and at which point. The blog articles were being copied in the background, but all the other databases in the system could only be copied when we could be assured that they wouldn't be updated (ie. they were operating in read-only mode). With Bloglines, we could "cheat" a little. By turning off the crawlers in Redwood City, we could assure that many of the databases in the system would not be modified, while still keeping the site alive. We could then start copying these databases, and the total amount of downtime would be reduced further. So our move checklist was divided up into the following sections:

1. Tasks that had to be completed before the day of the move
2. Tasks to do after the crawlers were turned off
3. Tasks to do after the site was taken down
4. Verification steps after everything was moved to the new datacenter
5. Tasks to do after the site was back up at the new datacenter

Tomorrow, I'll talk about the joys of broken DNS caches and pirates.

Posted by Mark Fletcher at 10:17 AM | Permalink

I'm curious if you considered a zero-downtime move at all, where you would keep the "old bloglines" still running while data was transferred to the new bloglines datacenter, and then switch over to the "new bloglines" via DNS after the new site was up? And either users have data loss of several hours (arguably better than downtime of the same amount), or you replicate transaction logs for user-critical data.

Those aren't the only things to think about when considering downtime. Bloglines is a free service and isn't currently monetized. If Ask Jeeves generated a significant amount of revenue from us, that would have factored into our thinking. But even then, the 'net is littered with examples of sites like eBay. I haven't checked recently, but eBay, at least in the past, had a policy of regular, scheduled downtimes.

In any event, with scheduled downtime, it's important to communicate with your users. For us, that meant a post to the Bloglines blog 24 hours in advance. The blog post had specific times listed, along with links to a site that converted the times into all other timezones. We also added a link at the top of every page on the site alerting users to the downtime. During the downtime, we displayed a page that explained exactly when we'd be back on-line. And we updated that page when we went half an hour over our scheduled time. Finally, people appreciate humor. If you're going to have downtime, make the down page fun. For the datacenter move, for example, we gave the Bloglines plumber a pirate makeover.

I'll continue my tale of the Bloglines datacenter move tomorrow, with a post that could be titled 'Rho Rho Rho Your Boat.'

Posted by Mark Fletcher at 11:40 AM | Permalink

The simplest (and safest) way to move a site is to take it completely down, copy all the data to the new machines, and then bring the site back up at the new datacenter. We could have done that, but the length of downtime required would have numbered in the days, and we didn't want to do that. Actually, an even simpler way to move a site is to physically take the machines and move them to the new datacenter. Going across country, that still would have required probably 24 hours of downtime, factoring in the time to pull the machines from Redwood City, pack them, put them on an airplane, unpack them, reinstall them in Bedford, and reconfigure them for their new network environment. And after a journey like that, chances are some of the machines wouldn't come back up. So our only real option was to create a system that would copy at least a large amount of our data to the new datacenter in the background, while Bloglines was still live and operating.

The Bloglines back-end consists of a number of logical databases. There's a database for user information, including what each user is subscribed to, what their password is, etc. There's also a database for feed information, containing things like the name of each feed, the description for each feed, etc. There are also several databases which track link and guid information. And finally, there's the system that stores all the blog articles and related data. We have almost a trillion blog articles in the system, dating back to when we first went on-line in June, 2003. Even compressed, the blog articles consist of the largest chunk of data in the Bloglines system, by a large margin. By our calculations, if we could transfer the blog article data ahead of time, the other databases could be copied over in a reasonable amount of time, limiting our downtime to just a few hours.

We don't use a traditional database to store blog articles. Instead we use a custom replication system based on flat files and smaller databases. It works well and scales using cheap hardware. One possibility for transferring all this data was to use the unix utility rdist. We had used rdist back at ONElist to do a similar datacenter move, and it worked well. However, instead, we decided to extend the replication system so that it'd replicate all the blog articles to the new datacenter in the background, while keeping everything sync'ed up. This was obviously a tricky bit of programming, but we decided it was the best way to accomplish the move, and it would give us functionality that we would need later (keeping multiple datacenters sync'ed up, for example).

As the new machines were being built out at Bedford, work started on the blog article replication improvements. In the meantime, we still had a service to run. All growing database-driven Internet services have growing pains. All growing database-driven Internet services have scaling issues. That's just a fact of life. So, in the midst of all this, we couldn't stop working on improving the existing Bloglines site. It made for an interesting juggling effort.

Posted by Mark Fletcher at 12:48 PM | Permalink

One week ago, we moved the Bloglines service from the AT&T datacenter in Redwood City, CA to MCI in Bedford, Massachusetts. This was a challenging and complex undertaking that required months of preparation by many groups. Now that the dust has settled, over the next couple of days I'll explain some of the process involved.

We had been at AT&T since Bloglines first went on-line in June, 2003, and had been very happy with them. AT&T is a tier 1 colocation facility. They aren't the cheapest, but we never had to worry about power outages or other issues that can crop up with other facilities. After we were acquiried by Ask Jeeves in February, we started talking about moving the Bloglines service to the main Ask facility, which is in Massachusetts. This made sense for a number of reasons: it would be easier for operations, it would be easier for us to quickly expand in the future, and it would be easier for us to tie into other parts of Ask Jeeves.

Once the decision was made to move, we had two tasks: figure out how many machines to build out in Bedford, and figure out how to do the move with the minimum amount of downtime. In my experience, estimating how much hardware you'll need at some point in the future can be difficult, especially when you're growing quickly and you don't have a lot of history to use in estimating. I believe in the concept of overwhelming firepower (when in doubt, double or triple it), so we overestimated everything. In the end, the new system has 3 times the number of machines that we were running in Redwood City, and each of those machines is probably twice as fast as any of the old boxes. Once operations had the configurations, they set about ordering, installing, and configuring the machines. That left us with having to figure out how to move the site across the country while minimizing downtime.

I'll continue that part of the story tomorrow.

Posted by Mark Fletcher at 04:00 PM | Permalink

I'll be at the Always On conference this Thursday, July 21, on the Open or Closed Web panel, moderated by that shrinking wallflower Marc Canter. I'll be arguing for a Closed Web. Of course, I am kidding with those last two statements. As I am continually reminded, sarcasm doesn't translate well to the written page.

On August 8th, I'll be on the Vox Populi: Understanding the role of consumer-generated content panel of the SES San Jose conference.

Most likely unrelated to the above two events, this Saturday I'll be trying my hand at flying a 1945 Boeing PT18 Stearman, which is an open-cockpit WWII era bi-plane. While I do not have the leather cap and goggles, I do have a scarf and jacket, so I'll be at least somewhat dressed for the occasion. I can't wait!

Posted by Mark Fletcher at 08:16 PM | Permalink

On Thursday Google launched their My Yahoo competitor. It's not really called My Google, but it might as well be. As has been pointed out by many others, this marks a reversal for Google and their 'no portal' policy. Honestly, who couldn't have seen this coming for some time now?

The surprising thing, at least to me, is that they're trying to copy My Yahoo, which is the wrong thing to do. As many people have found out, the My Yahoo metaphor of a customizable page displaying static information doesn't scale. It may have worked in the mid-1990s. But in this particular century, with millions of blogs and other sites of interest, you need a different interface paradigm to deal with all that information.

My Yahoogle doesn't track what information you've already read, and what bits are new. So, each time you visit your My Yahoogle page, it takes time to scan the page to see if there's new information. This is a complete waste. If you only show new things, the amount of information that needs to be displayed decreases greatly. There's less information, and it's all new. It's a much more efficient way of dealing with many information sources.

Another flaw in the My Yahoogle model is the idea of placing everything on one page. Besides forcing the user to become a web page designer (should I place this information source in the right corner, or left?), this again reduces the number of information sources that can be followed, to a number that can be reasonably placed on a single web page.

The Bloglines user interface was developed partially in response to these flaws. Only show new articles. Provide a mechanism (the tree display in the left pane) that allows you to easily select a subset of information sources to display at any one time.

I have over 200 subscriptions in my Bloglines account (many of which you can see in my blogroll on the left). There's no way I could follow that many sites in My Yahoogle. Sometimes I'm asked if I consider My Yahoogle competition. There's no way that they can compete without completely changing their interface.

Posted by Mark Fletcher at 07:32 PM | Permalink

I'm back on the West Coast again, at least for a few weeks. I moderated a panel at the Syndicate Conference in NYC this past Wednesday. We arrived Tuesday evening, in time to catch a fun blogger dinner at Gallagher's Steak House in Times Square. For this particular carnivore, it was like heaven. Large slabs of cow and other mammals all over the place. Very yummy.

Wednesday morning, I had a meeting with Business Week's Stephen Baker, as described here. Stephen asked some great questions, and I hope I was able to give coherent answers (thanks to jet lag, I was operating on about 2 hours of sleep).

After that, I moderated a panel at Syndicate. The panelists did a great job, and I thank them for making the panel a success. Unfortunately my flight was at 6pm, so I had to scoot right after the panel ended. With a seat in the last row of the plane, and after having the plane sit on the tarmac for an hour due to weather delays at SFO, I finally got back home around midnight.

Posted by Mark Fletcher at 07:04 PM | Permalink

As part of our ongoing Universal Inbox strategy, we launched weather forecasts in Bloglines this evening. Simply click the 'Add' link from your My Feeds display, and then select the 'Weather' link. You can add a weather forecast by city/state or zipcode. Forecasts are available from throughout the world, and like all of Bloglines, this new feature is localized in the 8 different languages that Bloglines supports.

Posted by Mark Fletcher at 12:54 AM | Permalink

• Early this week, we hired Garrett Rooney. Garrett's super smart and we're really excited to have him join the team. And most importantly, he's another Sony PSP player, and we need fresh meat for our Ridge Racer and Wipeout Pure games.
• On Wednesday, Bloglines announced the ability to track UPS, USPS and FedEx packages. We were working on this when the Askquisition happened, and I'm really happy that we've finally been able to release it, as it's my favorite new feature. Testing this feature required the ordering of many items (ie. toys) from Amazon and other places. Sigh, the things I have to do for my company.
• On Friday, we announced that Bloglines had been translated into Klingon. Yes, this was for April Fools, but Paul actually did translate the site into the Warrior Tongue. For those keeping score, Paul also did this for ONElist back in 1999.
• Also on Friday, I made my acting debut in one of the videos promoting the Jeeves9000. I'm in the video titled 'Integration'. No, I have no plans on quitting my day job to become a master thespian.
• Finally, I've been having terrible luck with cars lately. Don't let me drive your car; it's guaranteed to break down.

Posted by Mark Fletcher at 11:19 PM | Permalink

Bloglines is looking for help as we continue to innovate and lead the news aggregation space. We're looking for a Senior Product Manager, several Software Engineers and several Senior Software Engineers. All jobs will be based out of the Los Gatos, Ca office of Ask Jeeves. To apply, you must fill out the on-line application. Other jobs with Ask Jeeves are available here.

The environment is fast paced; we're looking for self-starters who thrive in an environment where they have a lot of freedom along with a lot of responsibility. We've got a lot of exciting things planned; help us continue to innovate!

Posted by Mark Fletcher at 03:21 PM | Permalink

This past week has been hectic. I'm gratified by the overwhelmingly positive response to Bloglines' acquisition by Ask Jeeves. As I've said, we've got great plans ahead, and once the dust settles a bit, we'll be able to start rolling out new features again. We will start hiring in the next couple of weeks. As a first step, we'll be hiring a couple of engineers and a product manager to augment our existing team. I'll post a pointer when that happens

I'd also like to extend thanks to everyone that emailed me congratulations over the past week. I'm sorry if I haven't responded, I'm suffering email overload at the moment. So if you don't hear from me, please know that I do appreciate the emails.

Posted by Mark Fletcher at 10:17 PM | Permalink

1. I won't be going anywhere. I'm fully committed to Bloglines, and we've got great things in store.
2. The Bloglines web site will keep on going, business as usual. One of the things we liked about Ask Jeeves is their multi-brand strategy, and we’ll be operating as one of their independent brands.
3. So what will change?
   We'll have a lot more resources available to us. For example, we'll be integrating Ask's killer Teoma search engine technology within Bloglines. This will vastly improve our blog search capabilities. We don't think that world-class blog search exists yet; with Teoma and Bloglines that will happen.

Speaking from experience, I know that the acquisition of a service that you use and depend upon can be unnerving. I also know that after acquisition some services wither.I am confident that won't happen here. There is a shared passion and vision for Bloglines, and I'm very excited about the future.

Update: Here's Jim Lanzone's take on the acquisition. Also, thanks for all the great emails and comments! I will try to respond to them over the next couple of days.

Posted by Mark Fletcher at 09:21 PM | Permalink

Seperately, a lot can be gleaned from the live traffic stats that BoingBoing publishes. Kirk Scott ran the numbers, and he came up with the following. BoingBoing is the 3rd most popular feed on Bloglines, with 13,533 subscribers as of this morning. In the "Connect to site from" section of BoingBoing's stats report, the stats show that 90% of all traffic is directly linked from a bookmark or URL. Of the remaining 10% that is referred to BoingBoing, Bloglines is delivering more than 2.7x the number of hits than Google.

Referrals from search engines:

Referrals from other sites: (Bloglines has 13 entries in this list)

What this tells us is that Google, through search, delivers more page views on BoingBoing than Bloglines, but Bloglines audience share is 2.7x larger than Google's in the number of overall impressions.

The hit count (images retrieved) show that many more people are reading the content on Bloglines, and relatively few clicking through to the site. Even so, our clickthrough rate is higher than any other site with the exception of Google and Yahoo! search.

Nutshell: Bloglines is the 3rd largest source of referring pageviews on BoingBoing, and the largest generator of referring hits to BoingBoing content, larger than Yahoo! and Google -- and all other search engines -- combined. Bloglines generates 30% of all referring hits to BoingBoing.

In another post, Richard MacManus points to a blog post by Jupiter Research analyst Eric Peterson based on a conversation Eric and I had last week. Eric was interested in the business model behind Bloglines. Not accepting my usual stock answer of "Volume!", I detailed that we will integrating highly targetted contextual advertising into Bloglines next year, or "Adwords on Steroids" as Eric puts it (I like that description!). To reiterate what I told Eric, when we do start to roll out advertising, we will be very sensitive to user feedback, and we will be looking to our users to help guide us in this area.

Back to Richard's commentary on Eric's post, Richard says "Bloglines currently has only a fraction of the quantity of users that Google has". True enough. But if you look at the number of hits BoingBoing is seeing (as detailed above), Bloglines has the #1 share of any site. We certainly drive fewer single-visit unique visitors, but based on these numbers, we are the largest source of repeat visitors, which are the most valuable kind of audience to have. These numbers highlight just how sticky an aggregator is; we have incredibly high page view per user numbers and active unique numbers.

It's fascinating to examine all of these stats, and we're grateful that BoingBoing provides a live glimpse into the traffic on one of the most popular blogs on the Internet. On our end, Bloglines currently only provides subscriber counts to publishers. What other statistics, demographics or bits of information do publishers want from us about the traffic we're aggregating?

Posted by Mark Fletcher at 10:55 PM | Permalink

Posted by Mark Fletcher at 11:31 AM | Permalink

The idea of podcasting, while having been around for a couple of years, seems to have really taken off in the past few weeks. What's really exciting for me is seeing people use the Bloglines Web Services API to create new and interesting podcasting clients. There's Doppler Radio which has been out for a couple of days. And I just saw an announcement of the Bloglines Enclosure Download Script. We'll be adding these to the Bloglines Web Services section of the web site. Are we missing any other programs?

Posted by Mark Fletcher at 11:17 PM | Permalink

The last couple of weeks I've been busy with travel and work. And I had the misfortune of coming back from New York with a nasty cold. But that doesn't mean that things aren't progressing with Bloglines. We're in the middle of one of our hardware upgrade cycles - we generally have to add machines to the Bloglines cluster every few months to keep up with growth. This past month, the API announcement and all the press has accelerated the latest upgrade cycle. Always a good problem to have. We're also completing some back-end architecture work that I'm really excited about. Just because an Internet service is scalable, that doesn't mean you're not always working on it to continue to scale it. Scalability is not an endpoint, it's definitely a journey.

On Friday, a great article on RSS and Aggregators appeared in Business Week. Here's the first paragraph:

Sarah Houghton used to get her online news the traditional way: by visiting 10 of her favorite sites several times a day. But since joining the free online service Bloglines a year ago, the San Rafael (Calif.) librarian surfs no more. Now the news comes to her. Using software known as RSS -- for Really Simple Syndication -- Bloglines pulls together regular updates from a variety of sites. Houghton can check them each time she logs on to the service. "RSS is the primary tool I use to get news," says Houghton, 27. "It's all delivered to me; I don't have to go searching anymore."

Posted by Mark Fletcher at 02:14 PM | Permalink

I continue to be amazed at the activity surrounding our announcement last week of the Bloglines Web Services. Several libraries have been quickly developed to interface with the BWS, and several aggregators have been working on supporting the services. I think I've even seen a Bloglines IM notifier client.

We just opened up a set of forums for developers to talk about the BWS.

In other Bloglines news, there was a really interesting article in the Wall Street Journal today talking about RSS and Bloglines. And today we rolled out a new feature, Show Only Updated Feeds. When enabled, this option (found under the Feed Options of the Account page) changes the left-pane of the My Feeds display so that it only shows subscriptions that have unread items. It's a great way to simplify the display of the My Feeds page.

Posted by Mark Fletcher at 08:47 PM | Permalink

All I can say is "Wow!" We thought people would be interested in the BWS, but the overwhelmingly positive response was beyond our expectations. Thanks!

Here are just a few items on the announcement:

• Eweek : Bloglines Tackles RSS Bandwidth Issue


• O'Reilly : The New Bloglines Web Services, which was then Slashdotted


• Someone has already written a BWS Perl Module. This is fantastic!


• Links to Blogllines, the announcement and the new services section of the web site dominate today's Top Gaining Links

Posted by Mark Fletcher at 09:03 AM | Permalink

Tomorrow morning around 5am Pacific Time, a press release with the title New Bloglines Web Services Selected by FeedDemon, NetNewsWire and Blogbot to Eliminate RSS Bandwidth Bottleneck will go out, but I'm so excited I'm going to blog about it now.

So what is this? First, continuing a tradition we started with the notifiers, we're augmenting the data that you can pull out of Bloglines programmatically. We're calling the new functions the Bloglines Web Services and we've launched a whole new part of the web site to document them. The new functionality lets a program pull Bloglines subscription data as well as blog entries, using the OPML and RSS formats.

So what does this mean? If you're a desktop aggregator developer, you can use Bloglines to provide a sync'ing capability for your users. And you don't have to worry about supporting the different RSS and Atom formats (and various imperfect feeds), because Bloglines normalizes all data.

If you're a publisher thinking about entering the world of RSS, you don't have to worry about thousands of desktop aggregators pummeling your servers into oblivion. With the Bloglines Web Services, Bloglines acts as a feed cache, insulating content providers from bandwidth problems.

What makes this announcement extra special, of course, is that the leading desktop aggregators are announcing support for the Bloglines Web Services. FeedDemon has a beta version available now with support built-in, and NetNewsWire and BlogBot will be launching new versions soon.

We've been working on this for awhile now, and I've gotta say that Nick, Brent and Dru are great people to work with. I'll have more to say later, but I wanted to be "First Post" with the news.

Posted by Mark Fletcher at 10:20 PM | Permalink

We pushed out a couple of cool new features last night on Bloglines. First is 'Keep New', which lets you mark individual blog entries as unread. The second is 'Related Feeds', which are a list of feeds that are similar to the feed you're reading. This compliments the Bloglines Recommendations, which are personalized for each user.

Also, there's a great article on us in the San Jose Mercury News today (Yahoo link because the Merc changes URLs and puts things behind registration after a day).

Posted by Mark Fletcher at 10:14 AM | Permalink

Really great quote from The RSS Weblog on a survey done of aggregator users:

Bloglines is on fire. A ton of people use it and love it enough to evangelize it.

Posted by Mark Fletcher at 10:07 PM | Permalink

In Analysis of an artificial meme, Greg analyzed the data generated by the GoMeme experiment launched by Nova Spivak. It doesn't appear that the experiment yielded much info on how memes propagate, but what was interesting to me was the raw data, which can be found in an XML file at the end of the article. It contains the raw responses of the participants. One bit of information people were asked to include was which aggregator they used. I did a quick counting and came up with the following. A total of 154 people included the aggregator question in their response. The top aggregators were:

Bloglines with 64 responses for 42% of the market
NetNewsWire with 20 responses for 13% of the market
SharpReader with 9 responses for 6% of the market
Newsgator with 9 responses for 6% of the market
Feed Demon with 6 responses for 4% of the market
Firefox RSS Extension with 6 responses for 4% of the market

Posted by Mark Fletcher at 10:17 AM | Permalink

Dave asks why Bloglines uses a 3-pane interface. I guess Dave has never tried Bloglines, because we don't use a 3-pane interface. I happen to completely agree with Dave that a 3-pane interface is the wrong interface for an aggregator. My guess is that if Dave tried Bloglines, he'd really like it.

Bloglines uses a two-pane interface, which gives a little more flexibility than the single-page interface Dave likes. But here's the thing. If you organize your feeds into folders, you can click on a folder to see all the new items within that folder at once. And, if you click on the top link in the left pane (the one with the count of your subscriptions), then all unread items for all your feeds are displayed. Just like a single-page interface. Best of both worlds. And if you still don't like the two-frame interface, you can get to a single-page interface by just using the following link:

http://www.bloglines.com/myblogs_display?all=1

Dave, I ask that you give Bloglines a try.

Posted by Mark Fletcher at 06:12 AM | Permalink

Posted by Mark Fletcher at 01:03 PM | Permalink

Of course, if you ask my family, they'll tell you I've had an ego problem for awhile now. But now, with this piece from Jon Udell over at Infoworld, I'm just going to be completely unbearable to be around. :)

Since last fall, I've been recommending Bloglines to first-timers as the fastest and easiest introduction to the subscription side of the blogosphere. Remarkably, this same application also meets the needs of some of the most advanced users. I've now added myself to that list. Hats off to Mark Fletcher for putting all the pieces together in such a masterful way.

Posted by Mark Fletcher at 09:26 PM | Permalink

With the launch of the new Bloglines user interface a couple of weeks ago, we started publishing subscriber numbers for every blog in the system. Previously, we had only listed subscribers who had public profiles enabled, which is a small percentage of the subscriber base. Now, we include a total subscriber count with each feed. Richard MacManus has some interesting thoughts as he analyzes this information.

Posted by Mark Fletcher at 10:02 AM | Permalink

Chad Dickerson has a really interesting article up at Infoworld in which he talks about the problem with desktop RSS aggregators:

Fast forwarding to the present, InfoWorld.com now sees a massive surge of RSS newsreader activity at the top of every hour, presumably because most people configure their newsreaders to wake up at that time to pull their feeds. If I didn’t know how RSS worked, I would think we were being slammed by a bunch of zombies sitting on compromised home PCs. Our hourly RSS surge has all the characteristics of a distributed DoS attack, and although the requests are legitimate and small, the sheer number of requests in that short time period creates some aggravating scaling issues.

Because Bloglines has a vested interest in increasing RSS (in the generic sense) adoption, we're looking at ways we can help. We are working on a couple of projects right now, and we're of course open to suggestions.

Posted by Mark Fletcher at 09:55 AM | Permalink

There's a great article in the Wall Street Journal's Personal Technology section this morning talking about blogs and aggregation. From the article:

Overall, I had the best experience with a service called Bloglines, and I recommend it, especially for beginners. Bloglines (www.bloglines.com) works as a Web service, which means there's no software to install and you can catch up with your blogs from any Web browser.

Fantastic!

Update: I've changed the link to point to the free version of the article.

Posted by Mark Fletcher at 08:51 AM | Permalink

By now you may have heard about or seen the changes we made to Bloglines last night. We upgraded the user interface and added several new features, including the ability to sort subscriptions. Also, every Bloglines user now gets his or her own blog, and the tools to easily post items to it from their subscriptions.

With such large scale changes as these, there are always little issues that come up, and we're working to address them. We will be adding back the font size preference soon, for example. But overall it appears that our users are very happy with the upgrades, and that's the most important thing. We're getting great feedback so far (200 emails just this morning). We're very lucky to have such passionate, helpful users.

I'd like to thank everyone involved in this launch. In no particular order: Joe, Cathy, Kirk, Chad, Joseph, Greg, and our advisory group. I consider myself very lucky to have the opportunity to work with such talented people.

Posted by Mark Fletcher at 10:44 AM | Permalink

Posted by Mark Fletcher at 02:36 PM | Permalink

Bloglines has been listed as one of TIme Magazine's 50 Best Websites. Does that not rock? Yes, that does indeed rock. Fully.

Posted by Mark Fletcher at 06:43 AM | Permalink

Apparently several thousand blogs hosted at weblogs.com are now off the air (see this for more information). Bloglines archives all blog entries that we crawl (over 70 million at last count). If you have one of the affected blogs, and your blog was crawled by Bloglines, then we have at least some of your site archived (everything from the time we started crawling your site). We can dump the entries out into a text file, which will hopefully help in migration. Don't know if your site was ever in the Bloglines system? To find out, try subscribing to your site if you already have a Bloglines account, or you can search the Bloglines archives for your blog.

If your blog was crawled by us, and you want an export of everything we have for it, send an email to support (at) bloglines.com, and include the URL of your blog and/or the URL of your feed.

Posted by Mark Fletcher at 04:17 PM | Permalink

Thursday morning, I'll be a guest on the RSS Weekly webcast. The topic this week is 'How are news readers evolving? What will they become?''. You'll hear me talk about how Bloglines will evolve into a cure for world hunger, a method to end global warming, and a floor wax as well as a dessert topping. Oh, and an end to root canals and other dental maladies.

I could talk about all of that and more. Or I could talk about how we are looking into ways of addressing the 'information overload' problem and adding all sorts of cool new features. Most likely, that's what I'll talk about, because, well, I'm boring like that.

Posted by Mark Fletcher at 05:03 PM | Permalink

This is fantastic. PCWorld's latest issue has a review of some of the top aggregators, News on Demand. They gave Bloglines the highest ranking of any of the aggregators, tied with Feeddemon. They didn't talk about some of the features that are completely unique to Bloglines, like personalized recommendations, email subscriptions, and references, but it's still a great review.

Posted by Mark Fletcher at 09:40 AM | Permalink

• Dave Winer is asking for reviews of aggregators. If you're a Bloglines user, would you consider writing a review?
• Speaking of Dave, I sincerely appreciate the effort that he is leading to clarify the RSS 2.0 spec in regards to escaping practices, especially as they currently exist. Clarity is good.
• Finally, I really appreciate the efforts of Sam Ruby and Mark Pilgrim in building test cases for Atom and RSS. Writing test cases is absolutely no fun, but having them makes our lives as aggregator developers much easier. Thanks guys!

Posted by Mark Fletcher at 03:19 PM | Permalink

A Scan of the Headline Scanners:"Bloglines is free, powerful and intuitive, making it a good choice for those new to RSS. You can subscribe to hundreds of feeds without slowing the service (or your computer). It handles categories of feeds beautifully and has a strong search feature." Thanks Ryan!

Posted by Mark Fletcher at 11:10 AM | Permalink

As announced this morning, we introduced several new Top Links features. In addition to seeing the overall list of most popular links for the past day, you can now view just the top gaining links or just the top declining links for the past day. These listings are a great way to find out what people are talking about every day.

In addition, you can now use Bloglines or another reader to subscribe to RSS feeds of these pages. Each page has a 'Subscribe' button to subscribe using Bloglines, and the individual pages are:

• The Overall Most Popular Links
• The Top Gaining Links
• The Top Declining Links

• http://www.bloglines.com/rss/toplinks - for the 25 most popular links
• http://www.bloglines.com/rss/toplinks_inc - for the top 25 gaining links
• http://www.bloglines.com/rss/toplinks_dec - for the top 25 declining links

As a reminder, you can subscribe to the Bloglines News RSS feed, which is at:

http://www.bloglines.com/rss/about/news

Posted by Mark Fletcher at 12:41 PM | Permalink

Harold Check's interview with me has been posted on the RSS Weblog. Harold asked some great questions and I appreciate the opportunity to talk about Bloglines and the world of syndication.

Posted by Mark Fletcher at 07:21 PM | Permalink

This morning, we announced the Bloglines Mozilla Toolkit. I've been playing with a development version of this for the past couple of weeks, and I think it's fantastic. The toolkit extends Mozilla, embedding a Bloglines notifier into it, and adding several features to the right-click menu. Some of the things it makes easy to do:

• Find references for the page you're viewing
  
• Find references to a link within the page you're viewing
  
• Search for a term by just highlighting it
  
• Subscribe to the page you're viewing (like the easy subscribe bookmarklet)
  

The toolkit was developed by Bloglines user Chad Everett and he did a great job. We have an amazing group of users.

Posted by Mark Fletcher at 11:17 AM | Permalink

This morning we introduced a new Bloglines feature. If another feed has linked to an item that you're reading, there will be a 'References' link under the item with a count of the number of references to that item. Clicking on that will bring up a list of all the items from other feeds that have linked to the item you're reading. It's a great way to follow conversations and see different viewpoints.

This is another feature based on the link database work that we've been doing over the past two months.

Posted by Mark Fletcher at 11:00 AM | Permalink

Tom Biro of The Media Drop has posted an interview he did with me recently. He asked some great questions, and it was a pleasure to do the interview. I talk about Bloglines and some of my earlier experiences. Thanks Tom!

Posted by Mark Fletcher at 10:49 AM | Permalink

Sorry for being quiet here lately. We've been hard at work on several things, the first of which we announced this morning. The new Top Links listing is a ranking of the most popular links of the past 24 hours from the blogs that we index. It's based on a a new links database backend that you can search using the Citations page.

As Google has proven, links are the currency of the web. There are a lot of interesting features we can provide our users that involve link analysis. But we first had to create an entirely new and seperate backend to index all the links from all the blogs that we index. That's tens of millions of links right now, and it has to scale to much more than that over time. It was not a small job, and we're not entirely done yet. So watch for more features to be rolled out over the next several weeks and months.

Posted by Mark Fletcher at 10:17 AM | Permalink

Wow, a flurry of Bloglines related activity today. First up, Jeremy Zawodny posts that he's switching to Bloglines. Thanks! If you or any other user has any suggestions for how we can improve the service, please tell us, by either using the comment submission form on the web site or by sending email to support (at) bloglines.com.

Next, we're mentioned briefly in the latest Wired magazine, in an article on RSS.

Also, an interview I did for nPost has been published. I was interviewed by Nathan Kaiser and I'm very happy with how it came out.

Finally, I have to point to this review of Bloglines by the Tao Of Dowingba. If I may quote from the review:

Bloglines = wicked. Understand? Bloglines = wicked. Just in case you didn't catch it, one more time: Bloglines = wicked.

I couldn't have said it better myself. :-)

Posted by Mark Fletcher at 12:50 PM | Permalink

For kicks, I just figured out how much usable disc space Bloglines has between all the machines that run the service. With the recent addition of a couple of servers to support some upcoming features, I believe we've passed the 1 terabyte mark. That's in usable disc space, not raw. All of our various databases run on RAIDs, so the raw disc space number is actually much higher than that. Of course not all of that disc space is being used right now. We have a couple of backup machines that are pretty much empty and disc usage varies widely by what the particular server is being used for.

I continue to be amazed at how far technology has progressed in just my lifetime.

Posted by Mark Fletcher at 09:20 PM | Permalink

Just wanted to quickly pop in here and say thanks for the spat of recent Bloglines mentions. Scoble mentioned us a few times when he was talking about aggregator adoption rates. Steve Gillmor also mentioned us in his response to Scoble. I could probably write all day about my thoughts about RSS and Aggregation adoption rates, but I'm really busy doing what I can to increase those rates.

Things have been quiet on the Bloglines new features front. That doesn't mean we're not working hard on new features. We generally try to release something new every week, but with some of the bigger projects we're working on, we sometimes can't make that schedule. And I don't like talking about features haven't been released yet.

I can tell you that a new version of the Bloglines Windows notifier will be released this week, provided testing goes well. This is mainly a bug-fix release, for people with proxies that require authentication. But it will also have the ability to specify longer intervals between pings, and the oft-requested feature that clicking on the system tray icon always brings up your Bloglines account in your browser. In the current version, that only happens when you have new items to read.

Oh, and speaking of notifiers, one of our users has written a Unix/KDE version of the Bloglines notifier. It just hit Sourceforge, and you can download the code here. We'll be posting a note on the Bloglines site about it soon.

Ok, guess this wasn't such a quick note.

Posted by Mark Fletcher at 06:05 PM | Permalink

This afternoon we released an upgrade to the Bloglines email subscription feature. You can now reply to emails received with your email subscriptions.

From the announcement:

Email subscriptions make it easy to receive your mailing list messages and other email within your Bloglines account. You can create an unlimited number of special @bloglines.com email addresses. Email sent to those addresses show up in your Bloglines account, as if they were blog entries.

Posted by Mark Fletcher at 03:28 PM | Permalink

This evening I attended the SDForum Web Services SIG down at the Microsoft campus in Mountain View. I normally don't go to these things, because I'm really busy and I'm not a very good networker. But tonight's meeting was on RSS and Tom Gieselmann from Bertelsmann Ventures was one of the speakers. I met Tom back when they first invested in ONElist back in 1998. They're now looking at the RSS space.

Anyways, both Tom and Chris Pirillo, who was presenting via webcast from LA, said some very nice things about Bloglines during their respective presentations. Thanks guys! I really appreciate it.

Posted by Mark Fletcher at 10:43 PM | Permalink

Sigh. Looks like the main Bloglines user database corrupted itself around 2:30am this morning. It did so in such a way so that the database still 'ran', for some definition of that. Which means that none of our automated monitoring picked up on this fact. We will change that.

We take snapshots hourly, and we're recovering to the most recent good snapshot. Unforuntately, that means that if you registered with Bloglines after around 2am Pacific time or so (and that's a lot of people), you will need to register again.

What caused the corruption? We're not sure yet, although we've been having issues with the JFS filesystem that we're using. So it may have been that.

Update: The system is back on-line, based on the 2:05am userdb snapshot. We now need to go through the bad database files and find out what really went wrong. We are also bulking up our monitoring system to specifically detect this type of problem in the future.

Update 2: We think we've pinpointed the problem. It only affected a small number of users and it had nothing to do with the filesystem, which is good. It was a bug in one of our programs, which is bad. In a few cases, a batch process would delete some site records from the database without removing any subscriptions that referenced those sites. This ended up looking like database corruption, but it wasn't. The batch process in question runs nightly. The change we made was for it to delete some sites that were invalid (for a couple of definitions of invalid). This is part of our ongoing attempts to make sure our crawler only crawls valid RSS feeds. Yes, we did test the program before we pushed it to the site, but our testing didn't uncover this particular behavior.

Posted by Mark Fletcher at 10:00 AM | Permalink

We have rolled out two new Bloglines features, send subscriptions and a user directory.

Send subscriptions makes it easy to transfer your Bloglines subscriptions to others. Simply enter the email address of the person, select which of your subscriptions you'd like to send, and add a short message. An email will be sent to the person which contains a link back to Bloglines. If the user is not already a Bloglines user, clicking the link will automatically create a new Bloglines account and pre-populate it with the subscriptions you've sent. It's a very easy way to introduce your friends to aggregation.

The other new feature announced is the user directory. When viewing a blog in Bloglines, or when looking through the blog directory, there is now a 'Subscribers' link to the user directory of that blog. The user directory lists all the Bloglines users with public profiles who are subscribed to that blog. It's a great way to see who else is subscribed to a blog, and what other blogs they're subscribed to.

Both features were a direct result of user requests. As Bloglines gains in popularity, we are getting more and more great suggestions for how to improve the service. We're doing what we can to keep up!

Posted by Mark Fletcher at 02:34 PM | Permalink