Ship Happens, Week 2: A/B testing, software engineering in the age of AI, advancement wisdom and Brian Eno
A/B testing, software engineering in the age of AI, advancement wisdom and Brian Eno.
Happy Friday everyone! Welcome back to Ship Happens, your weekly product manager newsletter.
I’m using this newsletter to share at least three things I’ve come across this week to help you build better product. Subscribe so you don’t miss these when they come out:
Before we head into the main content, a short story. My son started playing basketball last Saturday. He’s 5, so I wasn’t expecting dunks or three-pointers. I expected lots of standing around and confusion in his first basketball game ever.
I definitely got that from my son’s team, which is what you’d expect from kindergarteners. What I did not anticipate was how good his opponent, the dreaded Blue Team was. Running plays and assigning positions…THESE ARE KINDERGARTENERS.
At first, it feels like these kids are just super good at basketball. But one of the dads and I were comparing notes, and it was clear that the other team had spent a lot of time together learning skills, positions and plays. Much of the high performance we see, we think is raw talent. But underneath is lots of hard work and practice, and I hope the below helps you build that success brick by brick.
On to the newsletter!
1. Required reading: Trustworthy Online Controlled Experiments
If you are running A/B tests in your product, now or in the future, save yourself lots of pain and read Trustworthy Online Controlled Experiments.
I wish I had knowledge of this book when I had just started out as a baby product manager. It is very dense, but it is the gold standard guide for properly experimenting on digital products.
You might think yourself “look millennial product manager guy, I don’t have time to read, my brain is mush from TikTok videos JUST LIKE YOURS and I just wanna work, go home and hug a pillow, not read a dense-af experimentation book”…heard. It’s a tough sell. But the temporary pain is worth it. The alternative is not doing A/B testing correctly, and then ending up locked in a room for months with your senior leadership trying to figure out how the hundreds of sloppy tests you ran tanked your revenue. This is exactly what happened to yours truly.
So make the investment. Some other things you might learn about A/B testing from reading this book:
A/B testing correctly is pretty hard. Most people are doing it wrong. I wish it were simply about splitting traffic and seeing which variant makes more money, but there’s so much more science and effort that goes into getting the best results. Speaking of which…
It might not be worth running A/B tests at all. If you’re a startup with 100 users a month using your app, A/B testing for you is probably a waste of time. You’re probably better off shipping features and measuring their impact to confirm or deny your hypothesis.
Your win/loss rates for testing are going to be pretty low. This varies pretty wildly depending on the maturity of your product, but your win rate will likely be in the range of 8-15% if you have a mature product, and 20-30% otherwise. But even in the best case, that means only 1 out of every 3 experiments will win!
Got your wheels turning? Good. Read the book! Save yourself lots of future you pain.
2. The Pragmatic Engineer strikes again: AI & the state of software engineering
To me, if you’re a product manager or aspiring product manager in tech, this is more required reading on the state of software engineering in the age of AI:
lays out the case for AI augmenting software engineers, not replacing them and I think that’s right. From my personal experience writing code with them and building things like my resume reviewer, AI is really great as a thought partner. But I do find myself checking it’s work a lot, which is why this quote from the article resonated with me:
The reality is that AI is like having a very eager junior developer on your team. They can write code quickly, but they need constant supervision and correction. The more you know, the better you can guide them.
I’ve also found this quote accurate as well from
, where the the polish and the cohesiveness of products is best managed by human beings, for human beings at this point in time:Creating truly self-serve software, the kind where users never need to contact support, requires a different mindset: Obsessing over error messages, Testing on slow connections, Handling every edge case gracefully, Making features discoverable, Testing with real, often non-technical users…
This kind of attention to detail (perhaps) can't be AI-generated. It comes from empathy, experience, and deep care about craft.
Often when disruptive technologies come around, we humans default to “how is this going to destroy my job?” The better question almost always is “how can I do a much better job with this new tool?”
The next 10 years will probably be less engineers losing their jobs to AI, and more AI helping engineers and non-technical people build more great things…
Great read! Highly recommend making the investment. We’re in the early innings of AI, more to come but the future is bright.
3. "What got you here won't get you there"
We’re in performance review season, so I am definitely in the mode of dropping nuggets of wisdom for product managers. Here’s one that I’ll share that I think is broadly relevant to most people.
When you’re an Associate Product Manager, often your success is measured on execution. Did you hit your OKRs that you set out for your team? Are the features that you decided to build well-researched and thought through in partnership with your customers and stakeholders? Did your team execute on time and appreciate your leadership throughout that process?
Once you start getting more towards those Senior Product Manager and Principal Product Manager levels, things start to change. The job is less about your personal execution, and more about how you can influence and enable others in your organization to move in a direction that makes sense based on your insight on strategy, their trust in your leadership, and your higher order execution through others.
The expectations rise even higher when you’re getting into the Director and Senior Director levels! Not only are you now managing and mentoring other product managers, but you now only get the stickiest of problems from your teams, and are expected to coordinate and work through many teams and people to drive a much bigger and more durable outcome.
Each level and situation within that level requires different tools and a different mindset. What I find with product managers generally is that success in the struggle to get to the next level lies in learning that what got you here won’t get you there. To get to Senior or Principal, you’ll need new tools beyond executing flawlessly by yourself. To get to Director, you’ll need tools around organization and communication. If your business is in trouble, you’ll need different tools in wartime than in peacetime.
I’m not sure if Marshall Goldsmith was the first person to write that, but it’s a good line!
4. Hidden Gem: The Flow State newsletter
I was Googling around to find great Substack newsletters. Here’s one that’s great for when you’re in deep focus called Flow State. You get a playlist every week to help you get into, well, flow state.
This week’s playlist features Brian Eno and the sountrack from his new movie, worth a subscribe and listen:
That’s it for this week! 🚀
If you’ve found this helpful, please consider showing your support by subscribing:
And please consider becoming a paid subscriber at $5/month or $50/year. I’m going to start sharing things like my resume template, resume reviewer for paid subscribers only. I’m also thinking about opening up chat to paid subscribers so you can ask me questions and get some 1:1 coaching. So consider going paid to get some of those great benefits. 🤝
Finally, share with someone else that you think would find this useful:
I’ll be back around this time next week with more useful product manager things!