# | Visit Site »

ShareThis

Here are the quick bits:

• Last Wednesday, I sold my share of the company to Scott and he is continuing to run the service.
• I am no longer involved in anything FeedLounge related (except in a historical sense).
• Scott and I don’t hate each other and never got into any screaming at each other or mud-slinging or anything juicy (sorry rumormongers), we just couldn’t agree on how to move forward from the status quo. I consider Scott to be a friend and continue to tout him as one of the smartest and most capable engineers I’ve ever worked with.

For those who want more details, read on…

A Brief History

When Scott and I first discussed creating a feed reading service (in January 2005), we decided on a simple and proven model for the business: offer a basic free version and upsell a portion of our customers to a premium version that has a monthly fee.

Along the way, we discovered a whole mess of challenges in scaling the service and eventually decided to go live with only the paid service to start. This was a tough choice, and it annoyed a number of folks who were anxiously waiting for the doors to open.

In retrospect, I don’t know if this was a good idea or not. Once we were live as a paid service, maintaining the service and trying to scale it became job number one (slowing down our ability to innovate and execute new features). And even so, we struggled.

Reality

Sometimes life gets in the way of what you want to do. Scott and I both had to take some time away from FeedLounge over the last year and a half. We both have mortgages and Scott just recently became a father, giving him a whole slew of new responsibilities.

When you’re making your living as an independant developer, you’ve got to make sure you get your bills paid. While we had attracted a number of loyal users, FeedLounge was far from paying 2 salaries plus operating costs.

Trouble Moving Forward

About a month ago, I approached Scott about changing the FeedLounge status quo. I didn’t feel that we were providing the level of service necessary for us to attract and retain users, and we were continuing to struggle with a variety of issues. I didn’t feel like we were moving forward as much as we were treading water.

In fact, I proposed a number of options that included temorarily halting billing, turning off registration and even shutting down the service entirely to allow us to address some of these issues.

I felt that we needed to:

1. Fix the bugs
2. Improve stability and uptime
3. Move forward quickly (and perhaps in different directions on the back-end) to offer a free version as originally planned

I believe that the little guy can beat the big guy, but has to do so by delivering a superior product and user experience. Small wins born from execution provide momentum, and you can grow from there. Failing to execute is something the little guy just can’t afford, and ultimately I feel it is where we came up short.

Unfortunately, Scott and I had a fundamental disagreement over where our major failings (as a business) were - and as a result, what needed the most attention. The more we discussed things, the more they boiled down to this single point of disagreement. If we couldn’t agree on how to move forward, how were we to move forward?

While I was willing to shut things down, Scott believes he can get things turned around with time. I hope that he does. FeedLounge is the best user interface and user experience I’ve ever built, and I’ve got a lot of blood, sweat and tears invested in it.

Conclusion

Ultimately, the choice for me to leave was very hard, but one I felt had to be made. Despite hard work and the best of intentions, the level of service we were providing to our FeedLounge customers was just not something I was comfortable putting my name behind.

Looking Ahead

For the time being, I’m going to concentrate on making the Tasks Pro™ 1.7 and Tasks 2.7 releases the best they can be (folks love tagging!) and get them out as soon as possible. But I still believe that the RSS/Atom/Feed space is one that is ripe with potential. I’m discussing a variety of possibilities with a variety of folks and weighing a number of options - don’t be surprised to see me playing in the RSS space again at some point.

ShareThis

We ended up ripping it out because the cross-platform compatibility issues were killing us and we had a bunch of higher priority features to implement¹.

When Yahoo!² announced their Design Patterns, including a forthcoming auto-complete library, I thought it made sense to wait and see what they released rather than continuing to develop our version and banging our way through the zoo that is cross-browserland.

Yesterday I spent many hours experimenting with and implementing the Yahoo! auto-complete library and it is very good. In fact, I’ve got everything working as I want it to in Tasks Pro™ with a single exception: I found a cross-platform bug.

Oh the irony!

Thankfully I have a friend from a “former life” that is a front-end engineer at Yahoo!, so I was able to get the issue in front of someone who can actually do something about it. I am optimistic that a solution can be found quickly, and that the Yahoo!s will make iteritive patches available.

The good news, of course, is that when this is fixed the Yahoo! auto-complete library will be quickly implemented for tagging in Tasks Pro™, Tasks and FeedLounge. Thanks Yahoo!

1. Though users have asked for it. [back]
2. Was a link really necessary here? [back]

ShareThis

Are you telling me your server literally burned up?

Yes, the fan that was supposed to keep the CPU cool stopped working. The CPU overheated and burned a little CPU-sized crater in the motherboard.

So FeedLounge was running on this server?

No, neither FeedLounge or the feedlounge.com web site were running on the server that burned up.

Umm, then why can’t I get to the ‘Lounge or the feedlounge.com web site?

The dead server was the primary DNS server for feedlounge.com - it was the magic beans that told the rest of the internet where to go when you typed ‘feedlounge.com’ into your browser.

Wow, this box was responsible for making feedlounge.com accessible to the rest of the internet and you only had one of these?

Actually, we had three of them. We had the box that burned, and two other separate boxes hosted by Austin Web Development.

I don’t get it. If there was more than one box, then why didn’t the back up boxes kick in?

About a week ago, Austin Web Development re-configured their DNS servers. They gave us warning beforehand and we checked through all the sites on the now-crispy server to make sure we weren’t relying on the Austin DNS for any of them.

Why didn’t you also check this for feedlounge.com?

Unfortunately, the answer is really simple - we forgot. We moved the feedlounge.com web site to a dedicated server in a data center in New Jersey last summer around the same time we move the ‘Lounge onto our big servers in our rack space in San Francisco.

Since feedlounge.com was no longer on boxes at the Austin data center, we didn’t think to check the DNS records for feedlounge.com.

Do you now feel that was monumentally stupid?

Um, yeah. And then some.

Are you saying that if this had happened a week ago, it wouldn’t have caused any trouble?

Most likely, yes. We’d have replaced the fried box just like we’ve done, but the backup DNS servers would have shouldered the load while we did so.

Sounds like you guys should have paid more attention to this.

Agreed. Lesson learned - the hard way.

So why is it taking so long for FeedLounge to become available again?

DNS is a bit complicated. Scott has written up a great post on this on the FeedLounge blog. The root of the problem is that a DNS change can take 24-48 hours to propagate throughout the internet.

That’s ridiculous, can’t you speed that up?

Unfortunately, no. I really wish we could.

Why can’t you put up a message saying that the server is down - at least an explanation for people?

That’s the real problem. The DNS information is what allows your ‘feedlounge.com’ request to go to the proper server. If we could get you routed to a server to see the message, we could just serve up the ‘Lounge and the feedlounge.com web site to you. The problem is getting you to the right place, not that FeedLounge is down.

How can you say that FeedLounge isn’t down? If I can’t get to it, it is down!

I can certainly understand how you could feel that way, but there is a little bit of a difference. The way the DNS…

Stop blaming this on DNS! I don’t care about the technical details - I care that I can’t get to FeedLounge.

Ok, let’s try an analogy.

Let’s say FeedLounge is a car (maybe a BMW). You drive to a nice restaurant and give the car and the keys to the valet - not thinking much of it. After dinner when you go get your keys, the valet tells you that there was an accident and unfortunately your key was broken.

“No problem” you think, “I gave my buddy my spare set of keys when I bought the car - I can just get the spare set.” Unfortunately, when you call him you find that your buddy left last week for vacation on a remote island - he’s not going to be able to help you.

Now which is more accurate in this situation?

• My car is broken.
• I can’t get into my car.

This is where we are with FeedLounge. The service isn’t down or broken, the problem is getting to it.

So are you saying this isn’t your fault?

Of course not - it’s definitely our fault. We weren’t diligent enough with our backup DNS servers and got burned¹ because of it.

It’s the same way you’d be responsible for not having the forethought to make sure your spare set of keys was available “just in case” when you valet’ed your car.

So if the service isn’t down, there has to be some way for me to get to it.

If you’re having trouble accessing the server, you can add a DNS server listing that has the proper information for FeedLounge to your computer’s list of DNS servers. That DNS server is “65.90.218.228″, however we’ve gotten lots of reports from folks who are already back in the ‘Lounge already so hopefully it won’t be long now for anyone still affected.

I think it’s totally unreasonable to ask me to do this.

I wish there was another way, but this will give you access to FeedLounge now. The alternative is to wait for the DNS changes to reach your DNS servers. This is how DNS works.

So what do you do now?

Besides offering an apology to our users, there isn’t much we can do. We have to wait for the changes we’ve already made to take effect.

Can you at least promise this won’t happen again?

Unfortunately, no. In fact (as I’ve said before), I’m quite sure it will happen again at some point. What we can promise is that we’ll continue to take responsability and to be open and honest about what is going on - even when things go wrong and being honest makes us look stupid.

As hard as we try to make sure they don’t, web services go down or are unreachable at times. This time was our fault for not having working backup DNS entries. While I think it’s highly unlikely this particular problem will surface again, the last time folks had trouble accessing FeedLounge was due to a DDOS attack on Live Journal that affected our entire data center; unfortunately that is hardly something we can promise won’t happen again.

That is totally unacceptable for a service I’m paying for!

I’m sorry you feel that way, and it’s not an uncommon response whenever there is a problem with FeedLounge or any other paid service.

We work really hard to keep the service up and running all the time, and I feel our track record in our first 4 months of service is excellent. However, the fact of the matter is that this is a service funded by Alex and Scott, and run by Alex and Scott. There is no Yahoo! or Google or VC with deep pockets to pay for mirrored data centers, or 24 hour IT staff managing the servers.

This is how bootstrapping works and we’ve tried hard to be transparent and upfront about this since the beginning. I hope that it doesn’t come as a surprise to anyone.

I’m not satisfied or happy about this.

Trust me, we’re not either.

Ok, now what?

It was a bad day for FeedLounge. With apologies to our users, we fix the problem and move forward. As we do, we’ll continue to work hard to make FeedLounge as reliable as possible and continue building the features our users are asking for.

1. Pun intended. [back]

ShareThis

As I noted in the forum thread, this is the only time in the last month this has happened, and I’m pretty sure¹ it’s the only time in the last couple months. Regardless, it shouldn’t happen and we’re adding yet another layer of watchdog processes to help ensure it doesn’t in the future.

Back in mid-February, when we were upgrading to the Eirik release, we experienced a day of downtime which riled up a number of folks and we had a spirited conversation about it. In the forum discussion one of our users pointed out that a day of downtime was only 17 cents of the $5 monthly subscription cost.

A developer friend of mine noticed the hubbub from folks complaining about missing a day of service and sent me a keep your head up e-mail with the following point:

If people are more than 17 cents worth of upset at missing a day of FeedLounge, then they obviously feel that missing a day of FeedLounge is worth more than 17 cents to them. If this is the case, then they are obviously not paying what the service is worth to them and they should be happy that they are getting the service at such a great discount!

This made me smile.

A comment in this morning’s thread reminded me of this exchange.

This morning’s queue slowdown wasn’t a full service outage, as the FeedLounge interface was up the entire time and folks had access to all of the items fetched before this queue lull. I’m also not sure of the exact length of the slowdown, but the “7 hours without new items” quoted in the forums is probably pretty accurate. A few quick calculations in LeanCalc show that a full outage of 7 hours is less than a nickel of service (per user), so a half outage is somewhere around 2-3 cents.

Since we went live in January, FeedLounge has experienced the following service outages:

• 2006-01-27: 4 hours
• 2006-01-31: 25 minutes
• 2006-02-15: 22 hours (planned downtime for upgrades)
• Today: 7 hours (Queue slowness only)

This works out to 98.8% of uptime (98.7% if you count the queue slowness this morning) in the first 3 months of a hosted web service - pretty darn good! Since our 22 hours of scheduled downtime in February, we’ve had 100% uptime for the ‘Lounge itself and 99.6% uptime for the Queue.

While we understand and welcome the high expectations of our customers, it’s nice to step back and look at the big picture view as well.

1. Our bandwidth stats page doesn’t give me easy access to previous months. [back]

ShareThis

I’m looking forward to see what people do with these, and which APIs are requested next. This also knocked 2 highly ranked items off our feature voting list. Progress is good.

ShareThis

ShareThis

This was my first podcast, I think I probably stumbled on my words more than you’re supposed to - oh well.

ShareThis

• The ability to pass your FeedLounge item tags on to del.icio.us with TagThru™.
• The ability to sort items from oldest to newest.

We’ve also upped the 3 hour tour to a 24 hour tour - check it out.

ShareThis

It’s nice to be included in the comparison, and also nice to see comments on the post from some of our users.

ShareThis

I spent most of my Friday upgrading WordPress installations (this blog, my Golf blog, the FeedLounge site and the King Design site). For the most part, everything went smoothly - however there were a few notable exceptions.

I was upgrading all of these sites from 1.5.2 (plus security patches) except for the Golf blog. I had no trouble to speak of upgrading this blog or the golf blog; then I got to the FeedLounge site.

1. I ran into some caching issues right away on the test server and had to disable cache.
2. The upgrade plus the cache issue also seemed to convert me back to the old FeedLounge theme - possibly due to the theme name having a space in it (”FeedLounge v2″ vs. “FeedLounge”).
3. The changes in how the rewrite rules are handled violently disagreed with the permalink structure I’d chosen for feedlounge.com. The main problem was me wanting /blog to be both the root of the blog and the base for the categories. I was finally able to track down the specific problem and fix it - more on this later.
4. Then feedlounge.com went away for a bit. Good times!

The King Design site was pretty smooth, with the only issues being:

1. get_the_excerpt() no longer appears to work for pages.
2. The new rewrite rule handling broke some of my is_single() checks.

For being such a major upgrade, I have to say WordPress did a pretty good job keeping things compatible. With a project used in such a wide variety of ways, it must be a huge challenge to keep moving forward while trying not to break existing installations.

I plan to report the issues I ran into¹ with test cases so that folks can decide if they are worth fixing or not. Details of the issues and work-arounds to follow when I have more time.

UPDATE: Oops, found one other issue. It seems that WP now loads the execute-pings.php page via an internal GET call (in the spawn_pinger() function). If you’ve password protected your /wp-admin directory (as I think is a wise idea), sending pings and trackbacks will not happen unless you manually load the execute-pings.php page. I’ll try fixing this via plugin if I can.

1. Actually, I’ve reported some already. [back]

ShareThis

UPDATE: We’re back up.

ShareThis

I fixed most of the issues, but we still have one issue outstanding before the FeedLounge OPML validates. When a node has an attribute other than ‘text’, a ‘type’ attribute is also required. We include the title of the element in both the ‘text’ and ‘title’ attribute for compatibility. The inclusion of the ‘title’ attribute necessitates the inclusion of a ‘type’ attribute as well - according to the OPML Validator.

This is no problem for the subscription elements themselves, as a ‘type’ of ‘rss’ is appropriate here. However, for our tag nodes (nodes that indicate what tags or categories a feed is in) are not representing feeds, and a type of ‘rss’ seems inappropriate here. I tried setting a type of ‘tag’, but got a warning that told me to use only a known type.

So, I’m in a bit of a quandry. I want the OPML to validate, but I also want to maximize compatibility. To validate, it appears that my only choice is to remove the ‘title’ attribute from the tag outline element. To maximize compatibility, I need to leave the ‘title’ attribute in, but cannot make it validate due to the rigidness of the ‘type’ attribute.

For now, I’ve left the ‘title’ attribute in on these nodes, and not set the ‘type’ attribute¹. What should I be doing here? What is the proper solution?

1. Partly because this is also how NNW does it. [back]

ShareThis

Special thanks to: my travel partner Steve, Adam, DeWitt, Anil, Niall, Davey, Tim and everyone else I had the pleasure of learning from over the past 2 days.

Other posts of note:

• tow.com - Mashup Camp Day 2 (Sharpcast looks very cool, but they should really have full content feeds¹ in their blog)
• Baus on MashupCamp and identity
• tecosystems - Report from MashupCamp
• The 20″ MacBook Pro
• c|net coverage of MashupCamp

1. This is a perfect example of one of the many reasons why partial content feeds are bad: Sharpcast is a new company to me, one that I know little about (other than Adam is working there - cool!). I’m willing to subscribe to their feed to follow what they’re up to. However, since I’m merely watching from the sidelines with miid interest and am not yet invested, I’m very unlikely to click through to any of their posts - thus by choosing to offer a partial content feed they have lost an opportunity to have me read their content where it is convenient to me while I’m navigating through my unread items, which might get me invested.

   Another bad choice is making the RSS .92 feed the linked feed on the blog. Not having a date published for the items in the feeds is bad. Luckily, they use WordPress, so you can find their RSS 2 feed or Atom feed pretty easily. [back]

ShareThis

The sessions I attended today:

• API Best Practices - we had already planned to use REST APIs for FeedLounge, a decision reinforced by many strong opinions.
• Identity Management - this was very interesting (perhaps the wiki page will be updated later), OpenID and LID seem quite intriguing. I especially like the fact that Six Apart already has a big OpenID implementation in the wild. WordPress should strongly consider supporting OpenID.
• Creative Commons - As I’d already been told, Larry Lessig is an excellent public speaker.
• Search Mashups - this was fairly free form, but there were some interesting ideas discussed.

My attendance was documented by others.

Funny side note: we had a little trouble finding the hotel we’re staying due to an (apparently) ambiguous address given to Google Maps. It turns out that the hotel is blocks from my old apartment and about a mile from my old condo. Yes, I drove past it every day for 4 years - and had no idea it was there.

ShareThis