On Podcast Server Downtime, Scars from General Motors, and Taking Ownership of My Situation — Lean Blog
Ever since I started podcasting in 2006, I’ve been using a company called “Hipcast” to be the “hosting” company for the podcast audio files and feeds. For those of you not familiar with podcasting, that’s the website where I upload podcast audio files… and they then serve those files when requested by services like Apple Podcasts or Spotify.
This has been an aggravating week or so, since Hipcast has been having a lot of technical issues going back to last Tuesday or so. Their website has been down, the podcast feeds (RSS feeds) have been broken. People haven’t been able to listen to my podcasts unless they had already downloaded the episodes through their app.
One of my regular listeners pulled the virtual “ andon cord” and emailed me to let me know there was a problem. I wasn’t thrilled because I had just published an excellent episode with Joy Furnival, and I was disappointed that the “launch” of that was now messed up.
Hipcast was up, then down. They posted messages about things being fixed, and then it all broke again.
I felt frustrated and powerless. All I could do is wait… but I’m impatient. I pictured me and their other customers looking like this (except I was working from home and not in a suit).
Questions like that are hard to answer, but remind me of…
Scars from General Motors
My impatient waiting reminded me of working at General Motors in 1995… under the first plant manager, or the one who was very much the traditional style of GM leader. This was before the NUMMI-trained plant manager came in to start fixing the culture.
I was the industrial engineer for an engine block machining line. The actual value-added work of cutting metal was done by machines. Those machines were run and monitored by UAW employees and received support from a process engineer, whose role it was to be in the details of the machines, working with the vendors, etc. while I was focused more on the overall flow and management of the area — the bigger picture vs. the details.
Often, one of those giant expensive machines would go down for some reason, meaning the flow of the engine block line was interrupted. Sometimes, the downtime was very much due to bad management decision making.
A UAW worker (and maybe the engineer) would say, “It’s time to stop and change one of the cutting tools, because this is the planned frequency.”
A senior management would basically say, “No, you can’t stop the machine to change the tools… because we’re behind on production and I’m getting chewed out, so we can’t afford to stop now. Those tools have a longer lifespan anyway, they’re overengineered.”
Sometimes, that would end up biting them in the rear. Instead of taking a short amount of time for a tool change (was it under an hour? I forget), there would be some catastrophic failure that would really mess up the machine. That made the “behind on production” problem even worse.
I used to joke that you could tell how long a machine had been down, based on which level of management was standing there, arms crossed, looking mad (and not being able to do anything about the situation but wait). The longer the downtime, the higher the level.
If the machine had been down for a few shifts or days, the plant superintendent would really start yelling at people.
Of course, Bob would yell and scream instead of (at least outwardly) reflecting on the decision that he had made that led to this problem. I used to joke that we could have just bought a cardboard cutout of Bob… put him far enough away and people might not have known the difference… the message of “work faster” would have been the same?
I’m sure similar things might happen in software companies, I don’t know. Management makes a bad decision, then something bad happens. Management gets mad and pressures the people who know how to fix it… and those people resent having to work 24/7 to resolve a problem that could have been avoided (the machine going down). The glares and the yelling… not helpful.
Taking Ownership of my Situation
But, now back to the podcast host situation. I firmly believe that a company doesn’t get to blame a supplier or a vendor for bad performance or bad behavior. Leaders shouldn’t blame employees for bad performance when leaders are the ones more responsible for the system (and if you’re hiring bad employees… why are you doing that??)
Unlike the GM managers, I couldn’t directly pressure anybody to fix the situation with my podcast hosting any faster. I could post a whiny tweet or a complaint on their Facebook page… but that wasn’t really helping.
It’s one of those situations where stopping to give a status update or to respond to some “when will it be fixed?” message then ends up SLOWING DOWN the efforts.
I was happy with Hipcast for almost 13 years. I’m not happy about the downtime. I’m not happy about the lack of communication on their part. But, I chose the supplier. This is on me.
Earlier this year, I was thinking, “Why is Hipcast never on anybody’s lists of top podcast hosts? Did I make a bad choice? Should I move to one of the “Top 10” podcast hosts or somebody who is always recommended by others?
But, I wasn’t proactive. I thought about switching, but I didn’t take action. I wondered about their low $9.99 monthly price… appreciating that it had never gone up over time (then again, hardware and bandwidth costs in this domain have only fallen). I wondered about their business model… what happened if Hipcast went out of business?
Well, I wish I had acted on my “what if something went wrong?” hunch.
I’ve spent a lot of time over the last five days researching other podcast hosting companies and working to do what I can to get things up and running at other companies.
And, I say “companies” because I’ve decided, for now, to not put all of my podcast eggs in the same basket. I have six different podcasts now and I should have recognized the risk of them all being at the same company. I mean, it had been convenient and cost effective to have them all in the same account.
I’ve gotten my flagship podcast “ Lean Blog Interviews “ back up and running on a company that is considered one of the top tier hosting companies. I’ll pay about $29 a month for that.
I also put the new “ Lean Whiskey “ podcast there for the initial launch after planning to use Hipcast. That created additional work for me, but assured a smooth launch last Friday.
I put the “ Practicing Lean “ podcast on a different platform, one that will be free since it’s got fewer listeners and now that the full audiobook is published there, I probably won’t be adding new episodes.
Part of my thought process was to experiment and try out different companies. I could read reviews and look at the pros and cons… but the only way to know for sure was to try them. I tried a few podcast hosts that ended up not going with for one reason or another. One had great features, but seemed like too small of a company. Others just didn’t import the old podcasts in as well from Hipcast (and the ability to port the podcasts over was hit or miss because it didn’t work during the periods of downtime).
The rest of my podcasts
are on a relatively new company called Anchor.fm. I was skeptical about them being new (and being free) but they were acquired earlier this year by Spotify, which gives me hope that they will be a major player in the podcast space.
I know there is some risk there… but I don’t have all of my digital audio eggs in the same server basket. I did also import Lean Blog Interviews into Anchor.fm. I haven’t decided yet if that’s an additional backup or if I’ll use that service going forward.
Status Update for Listeners
So, if you’re a listener, here’s where I think things stand.
All of the “RSS feeds” should be restored now. If you’ve been a subscriber, you shouldn’t have to take any action. Please let me know if you have problems with podcast episodes on your smart phone or other podcast devices.
One known problem still is with “embedded players” for various episodes. Those single-episode players (which are installed individually on each podcast episode blog post) are from hipcast and those currently aren’t working. Moving the RSS feed and podcast hosting doesn’t fix those players.
Update: Hipcast says, as of Monday morning, that their service is fully restored. I think the embedded players are working again.
A customer asked about “root cause analysis” and the CEO is going to post something, they claim:
With Hipcast getting things back and running, I’ll probably leave most of these embedded players there (on pages like this)… continuing to pay $9.99 a month vs. putting in the time (or paying someone) to copy-and-paste code for the new hosts’ players is something I have to consider. I apologize for the ones that aren’t working.
But, recent episodes ( like the one with Joy Furnival) have the new host’s player installed so that works. Thanks for your patience… or if you’ve been impatient with me being impatient with Hipcast… at least I think I have things on a better path (or better paths) now.
Originally published at https://www.leanblog.org on April 29, 2019.
