The Stellar Status Blog Tumblr. Thingie.

The Stellar Status Blog Tumblr. Thingie.

In his landmark television series Cosmos, Carl Sagan stated: “If you wish to make an apple pie from scratch, you must first invent the universe.” Sagan was goofing on the absurdity of “from scratch” when talking about physics, referring to the reducibility of all matter down to elementary particles, and reveling in the connection of everything in the universe back to the Big Bang. He wasn’t, to my knowledge, talking about software development.
I’ve been pretty quiet here since introducing the Fun Pass three months ago. The idea, after the Fun Pass launched, was to do something seemingly simple for Stellar: adding Instagram likes as a full source. Turns out, like baking an apple pie from scratch, that wasn’t as easy as it seems. When Stellar launched, it had four sources that you could connect: Twitter, Flickr, Vimeo, and YouTube. Instagram will be the fifth. Because I didn’t know what I was doing almost four years ago when I started hacking away on Stellar**, the code wasn’t eager to accommodate any additional sources without a significant amount of refactoring, which is a fancy word for ripping apart and rewriting code.
So that’s what I’ve been doing, rewriting the stickiest part of Stellar’s guts, aka reinventing the universe. But I’ve been testing the Instagram integration for the past week, alpha testers are playing with it now, and hopefully you’ll be able to use it within another week or so. And then onward to other things that won’t take so much effort. (I hope.)
** I still don’t know what I’m doing, but I can do it so much quicker now. Progress?

In his landmark television series Cosmos, Carl Sagan stated: “If you wish to make an apple pie from scratch, you must first invent the universe.” Sagan was goofing on the absurdity of “from scratch” when talking about physics, referring to the reducibility of all matter down to elementary particles, and reveling in the connection of everything in the universe back to the Big Bang. He wasn’t, to my knowledge, talking about software development.

I’ve been pretty quiet here since introducing the Fun Pass three months ago. The idea, after the Fun Pass launched, was to do something seemingly simple for Stellar: adding Instagram likes as a full source. Turns out, like baking an apple pie from scratch, that wasn’t as easy as it seems. When Stellar launched, it had four sources that you could connect: Twitter, Flickr, Vimeo, and YouTube. Instagram will be the fifth. Because I didn’t know what I was doing almost four years ago when I started hacking away on Stellar**, the code wasn’t eager to accommodate any additional sources without a significant amount of refactoring, which is a fancy word for ripping apart and rewriting code.

So that’s what I’ve been doing, rewriting the stickiest part of Stellar’s guts, aka reinventing the universe. But I’ve been testing the Instagram integration for the past week, alpha testers are playing with it now, and hopefully you’ll be able to use it within another week or so. And then onward to other things that won’t take so much effort. (I hope.)

** I still don’t know what I’m doing, but I can do it so much quicker now. Progress?

In February, I spoke at the Webstock conference in Wellington, New Zealand. My talk was called “I built a web app (& you can too)” and was about how I built Stellar.

In the final third of the talk, I discussed the future of the site and the difficult time I was having with my motivation. At the time, I honestly didn’t know if I would continue developing for the site or even hosting it. The process of giving the talk was very helpful in helping me figure out that, yes, I did want to keep Stellar going. My first code check-in in several months occurred just a week or two after I got back from NZ and I’ve been working steadily on it ever since.

ps. Webstock is a wonderful conference. I don’t know if they’re doing it next year or not, but if they do, you should go.

pps. Oh man, I am not a good public speaker. I’m a little embarrassed watching this, even beyond the usual “that’s what my voice sounds like?” reaction. I feel like I had a compelling story to tell, I just didn’t tell it very well. Next time — if there is a next time — I will do better.

Introducing the Stellar Fun Pass

Today, I’m happy to announce the availability of the Stellar Fun Pass, a way for people to support the ongoing development of the site (and gain additional features) through the purchase of an annual membership. Right now, there are three membership “tiers” and three additional Fun Pass-only features: 1) you can filter by source in your flow (e.g. no Twitter or Vimeo), 2) your faves, flows, and best-ofs go back much further, and 3) only members can invite new users to the site. Oh, and you get a little Fun Pass icon next to your name on the site. There will be more Fun Pass-only features in future, so stay tuned. And you non-Fun Pass holders, don’t you worry…I will always love you and your Stellar experience will continue to be top-notch.

So, that’s the “what”. You might be wondering “why?” The plan for Stellar, right from the first few conversations I had with people about the idea, was that if it worked, the site would need to include a revenue component. The site has worked and I’ve put more than a little thought into how to make the revenue part happen. I’ve had conversations with investors about building a company, I’ve had interest from publishers wanting to build publications around Stellar, I’ve chatted with companies about integrating their faves into the site, I’ve had discussions with ad networks, I’ve thought about Kickstarter. But the idea I keep coming back to is direct financial support from the people who actually use the service, which support goes right back into improving the site for them. Ads or funding or sponsored sources might come into play at a later time, but for now, a closed symbiotic loop of Stellar and its users seems like the way to go.

Direct support of creators and their projects is an idea that’s very much in the air right now. Kickstarter, Tugboat Yards, 29th Street Publishing’s stable of publications, and Pinboard are all exploring different aspects of the idea, offering a spectrum of membership benefits, tangible products, and donation opportunities. This isn’t a new thing of course (Kevin Kelly wrote about 1000 True Fans back in 2008 and experimentation around this idea is as old as the web), but the relatively high receptivity among current web/app users seems novel. Kickstarter and the App Store have done much to get people used to the idea of paying for ideas and apps. Users are tired of seeing VC-backed sites shutter because they run out of money, close down after being acqui-hired by Facebook, or make user-hostile business decisions in order to pump revenue up to please current and potential investors. Direct-funded projects certainly aren’t immune from these issues, but when I think about Stellar and the Fun Pass, it’s in terms of growing more organically and thinking longer-term w/r/t site features than hockey-sticking growth on some bullshit metric for the next round of funding.

So if that sounds at all interesting to you, head on over to the Fun Pass page on Stellar and check out the options. Questions, comments? Send em along.

P.S. I got the Fun Pass name from Caine’s Arcade. Some rejected names were: Plus, Pro, Extra, First Class, Premier, Cartographer (Trek joke, yo), Shooting Stars, Star Alliance Members, Astrologers Club, and Liker Gang.

And now, what I would assume might be a few frequently asked questions:

What’s the current status of Stellar?

Stellar is currently a project, not a startup or a company. When I started working on this project three years ago, I spent at least 20 hours a week on it, sometimes 40-50 hours. That figure was down to 2-4 hours a week a few months ago but it’s trending back upward over the past three months and I hope that trend continues.

Platform-wise, Stellar is stable. It works without intervention…for the most part, it just sits on the server and goes. [knock on wood emoji] From time to time, there’s an issue that crops up and the site will see some slowness or downtime, but that’s rare. There was a major screw-up recently where the entire database was almost lost (man, even writing that still turns my stomach right into a knot of dread and shame), but I have much more robust backup procedures in place now (larger DB backup window at AWS, periodic offsite DB backups, etc.) that will hopefully make that a one-time occurrence. [knock on wood emoji x1000]

Site growth is deliberately slow. There are currently about 15,000 users (~3000 of those are active) and over 25 million faved items in the DB. I recently turned off invite requests to keep things nice and stable for the near future. New users can still be invited by Fun Pass holders, so a small amount of word-of-mouth growth will occur.

Feature development is ongoing, continuing alongside the effort to fortify the site’s infrastructure. In particular, I am looking at adding more sources, upgrading the design (particularly on mobile devices and on retina displays), and making more past faves available on the site.

What’s up with Twitter’s API? Are they going to shut you down?

I hope not, but that’s the nature of the beast. Stellar doesn’t exist without APIs provided by the likes of Google and Twitter. In the future, they may reconsider providing those APIs to me or others. (For instance, Google shut down Reader and its API before I even had a chance to add GR likes to Stellar.) Right now, there’s no way to pay any of these companies money for the data Stellar needs. I wish there were…that way, I could pay them for guaranteed access and then sell Fun Passes and ads to offset the costs. It would remove a lot of the uncertainty I have, that’s for sure.

On the other hand, I am certain there will always be a number of available APIs full of interesting data from companies that see sites like Stellar as helpful in spreading their content around the web rather than as competition or whatever.

So what doesn’t a Fun Pass get me?

So many things, starting with a stay in the Royal Suite at the George V hotel in Paris’ swank 8th arrondissement. But also: prompt customer support (I will do my best though), specific uptime guarantees, input into future product features, ability to run a Stellarbot forever and ever, API access, t-shirts, a cool laminated Fun Pass that I can put into my wallet, and so forth. Which are all reasonable requests! And I hope to be able to offer some of them in the future. But not just yet.

Why didn’t you go with Tugboat Yards?

The Fun Pass has been under development since before TY launched their offering, so I was past the point of no-return when their alpha launched. Plus, Stripe is so pleasant to work with and gives me 75% of what I need. That and I am a roll-your-own control freak. (Did I just say that out loud?)

Any other questions? Drop me a line!

Backups and security update

Since the recent unpleasantness, I have been busy working on shoring up the Stellar’s backend. The RDS backup retention period at Amazon has been increased. (RDS is Amazon’s Cloud Relational Database Service, i.e. the thing I use for storing Stellar’s data.) Periodic offsite backups of the database are in effect…in fact, there’s one happening as I type this. The size, power, and speed of the RDS instance has been increased to make sure that the DB doesn’t run out of memory anytime soon. Amazon has a service called CloudWatch which allows you to set alarms that will email you if something happens (say the crawler server pegs at 100% CPU for five minutes)…I now have several of those set up for the DB and web server instances.

A big part of the site’s operation are the crawlers and streams on the backend that are constantly harvesting faves from the various sources. Almost 25,000 crawls are done each hour, along with almost 8500 accounts streaming faves in from Twitter. Those processes give the server a good workout and need to be restarted periodically. The process by which this happens has been updated, which has increased the stability of the crawler server and guards against missing data.

I’ve recently purchased a SSL certificate for the site and will soon make the sign-in process secure. Or maybe I’ll make the whole site secure…that seems to be heading towards best practice status these days.

And as always, Stellar’s source code is stored on my local machine, on a code repository in the cloud, and on multiple servers at Amazon.

I am optimistic (but remain vigilant!) these improvements will keep Stellar running smoother, faster, and more secure in the future.

A narrowly averted disaster

Stellar was down for most of yesterday. The outage happened due to the confluence of at least three mistakes, A Perfect Storm of Idiocy, if you will, for which I take full responsibility. I sincerely apologize for the outage and am beyond apologetic about what almost happened to all of your faves…even now, the thought of it makes my stomach churn. Here’s what happened.

I have a friend named Adam. I’ve known Adam since the mid 90s when we were on a email list together. Upon returning from a long weekend with my family late on Monday night, I noticed that Adam had faved quite a lot of items on Stellar. 7.8 million of them, actually. Upon further investigation, I determined that every single thing ever faved by any Stellar user was now, instead, faved by Adam. OH. SHIT.

But let me back up. After I got home, I checked my email right before bed and got a notification that Stellar was acting weird. And it was…people’s faves were all out of whack and messed up. When I went to check it out, the first thing I noticed was that my RDS instance at Amazon was running out of memory. Assuming that was the cause of the issue, I raised the memory allocation from 20 GB to 25 GB. That triggered a backup of the database. A backup full of bad all-Adam-faves data that replaced the backup full of good data made earlier in the day. Or so I thought.

Around this time, I messaged my friend Greg. Greg has helped me a bunch with Stellar’s back-end setup. We found the millions of Adam-faves, worked through a bunch of scenarios, read a bunch of docs, and determined that 1) a rogue UPDATE query had changed all those DB entries, and 2) because the memory allocation had triggered a backup, the good data was gone. For good.

I didn’t sleep at all that night. I cried. I nearly threw up. The thought of something I’d worked so hard on for more than three years disappearing, of being responsible for everyone’s data vanishing, of it being all my own stupid fault…it was all too much for me to sleep. By around 8am the next morning, I felt like that raccoon that Kevin Rose threw down the stairs.

After running an essential errand, I sat down to start writing this post (or, rather, the Bizzaro universe version of this post where Stellar is dead), to explain to everyone what had happened and what the scenarios were for moving forward. I was interrupted by an email from my friend Mark offering his help. After I explained the situation and how, with no backup, there was likely no hope, he offered the assistance of Buzzfeed’s ops team. I mean, I say “team” but they are actually ninjas. BF, being a large operation with many many servers, has AWS contacts and resources that a little site like Stellar does not. Raymond got in contact with those resources and eventually an answer came back: regardless of the bad all-Adam-faves backup, you should be able to restore the database back to a point in time within the past 24 hours, perhaps even further. [I don’t want to ding Amazon here, because I love AWS and their support people were very helpful in resolving this matter, but the docs could be clearer on this point. Everything I read (or remember reading at 2am after a long day of hiking and driving) gives the impression that, with the backup settings I was using, the data was unrecoverable.]

After a few tries (and after tricking RDS into thinking it had a longer backup window than it actually did), we successfully restored the DB back to before the favepocalypse. With help from Adam in retracing his steps on Stellar the evening before, I located the bug, which turned out to be a small line of code I forgot to check regarding The Great YouTube Debacle of 2013. I fixed the code, pushed it to the server, pointed it at the recovered database, and got the site back up again.

So, the worst thing that happened in the end was that everything anyone did on Stellar for about 12 hours (from noon to midnight on Monday) was discarded. That’s still a serious thing and I am sorry that it happened. If you followed someone, changed your information, or authed/deauthed an account during that time, you’ll need to do it again. In addition, some faves were probably missed while the site was down. Many of the missing faves have already been picked up by the crawlers, so hopefully the impact will be minimal.

What happens now? Stellar’s backend infrastructure must improve. By a wide margin, the biggest mistake I made was not having an adequate backup plan in place. This was a total idiot maneuver on my part and is in the process of being corrected, big time. The plan is better as of this writing (longer backup window, frequent snapshots) and will improve significantly (daily offsite backups, etc.) by the end of the week. There are other aspects of the site’s workings that I’ve been a little lazy about…those will be improved upon as well.

And that’s only the beginning…lots more to come. The idea of learning from your mistakes has been flogged to beyond cliche, but in this case, I really feel hopeful this will make Stellar better. It’s also reinforced how much I love doing the site and how important it is to me. Having so many people reach out to help and express concern and offer their best wishes was really touching. So thanks for that, extra-special thanks to Raymond, Greg, Mark, & Eugene, and I’ll work harder on screwing up less spectacularly. Fave on, you crazy diamonds.

A note on invites

Stellar is an invite-only site. But the mechanism for signing up for an invite is currently closed and has been for a couple of months. There are also a few hundred people languishing on the waiting list after signing up more than two months ago. The only way to get into the site right now is through an invite from current users who have invites available, and I haven’t been replenishing those.

So, why is Stellar basically closed and members-only right now? Plastics. Focus. I’m only one person with lots of potential Stellar-related tasks to perform and for now, I’ve chosen to focus my attention on making the experience for existing users better. This involves making the site faster, making it easier to scale, building new features, reorganizing, strengthening what’s working, and jettisoning what’s not. All this is easier when I don’t have to worry about new users.

My hope is that in a month or two, I’ll be able to work through the waiting list and open the site up to invites again. Thanks for your patience and your interest.