Peter Zaitsev, known to all as Percona’s founder and CEO, asked me to pen a few reminisces as I retire. How do I sum up 22 years in the MySQL world? How do I even begin? I best start with gratitude. I’ve had the amazing privilege of being on the inside of the leadership teams of two highly impactful software startups, MySQL and Percona.
I now find myself as Percona’s first-ever retiree. So I write as PZ requested of me, with the caveat that these reflections are wholly my own. They do not necessarily reflect Percona policy or even Percona’s current conditions, as I’ve been less in the front lines in recent times.
Tom, Monty Widenius (MySQL Founder), Heikki Turri (InnoDB Creator), & Peter, at the 2nd ever MySQL User Conference, 2004
I became a MySQL DBA in late 1999. My then-boss in Maryland agreed to buy the highest tier support offered by the fledgling MySQL company in Finland. A $12,000 annual payment got you the personal telephone numbers of everyone in the company, then maybe a half dozen persons, and quick answers to your emails often direct from company founder Monty Widenius. The passionate intensity of MySQL support was unlike anything I’d ever experienced. I loved it, while the speed and simplicity of MySQL delighted me.
Monty Widenius and co-founder David Axmark pioneered the now-familiar pattern of open source software employment – no in-person job interviews, work at home from anywhere via the internet, then fly all over the world for meetings and conferences. But when Monty invited me to join MySQL Ab in 2001, this was still pretty weird stuff, at least in the USA. My wife was incredulous that I’d take a job from some guy in Finland whom I’d never met and who gave away his software for free. As I once wrote in 2007:
I had a much better Oracle DBA offer from a large bank, and six kids (ages 2 to 13) plus my wife at home. To her the bank was a sure bet and a virtual boss in Finland an absurd risk. But MySQL struck me as an exciting company that kept its promises. What to do? I asked a trusted (Catholic) priest for his opinion. He surprised me and voted for MySQL. I borrowed a bedroom from the kids as my office and joined MySQL as employee #11.
Monty, Vadim, & Peter Share Dinner, 2008
Monty appointed me Director of Support. This ultimately grew to be a team of 60 high-end experts. This was another of Monty’s innovations – technical support was not an entry-level role but a senior-level one, a prestigious career destination for top experts.
Twenty years seems not that long ago. Yet so much that’s considered ordinary now wasn’t then. Neighbors assumed I was really unemployed until I started to travel to Europe pretty regularly. Everything inside MySQL was transacted by email. No Zoom, no Slack, no Confluence, nor any other of today’s common tools were then known. Maybe within a year of my joining MySQL, we began using Internet Relay Chat, or IRC. Having real-time chat among a global team was breakthrough technology. Yet, the IRC command line was too geeky for admin staff, so it never became a company-wide tool.
Another novelty though now commonplace, was a 100% global workforce. I had been in my mid-30s when the Cold War ended in 1991. Throughout my formative years, Russians, Ukrainians, Estonians, Bulgarians, and Serbs lived trapped behind the Iron Curtain, far distant from ordinary Americans like me. Now they had become my daily co-workers and personal friends. For me, it was another bit of cultural whiplash of the MySQL era.
MySQL experts were then extremely scarce. In 2002 Monty called me excitedly. A Russian MySQL prodigy had just accepted his job offer. His name was Peter Zaitsev, and Monty wanted me to be his manager. That year the entire MySQL company fit into one bus and rode from Helsinki to St. Petersburg to meet our new Russian colleagues. Peter looked to me like a teenager. In reality, he was age 20, married with one child, had a Master’s Degree in Computer Science, and was already a serial entrepreneur. He’d built his Russian startup Spylog around InnoDB, making himself one of the world’s first experts on this now ubiquitous storage engine.
Peter & Monty at the MySQL User Conference, 2008
I created the “High-Performance Group” within MySQL Support as a home for Peter to run. He was in demand all over the globe to troubleshoot difficult MySQL cases. He badly needed a deputy, and this is how Vadim Tkachenko entered my life. It also became the first of many business lessons I would learn from Peter. Peter is young enough to be my son. But regarding entrepreneurship, it has been the reverse, Peter the father teaching me the son.
One lesson personified in Vadim was how demanding Peter was in his hiring standards. He disqualified many applicants who struck me as very well qualified. Peter painstakingly probed and tested Vadim’s coding expertise. But the fruit of that care has shown in their long partnership and the technical excellence that Percona is known for. More troubling is that when I approved Vadim’s hiring, I had no warning of his wry sense of humor or that I would ever become its victim. How often I have paid for this oversight in the ensuing years.
In 2006 Peter and Vadim left MySQL to launch Percona and invited me to join them. MySQL had by now become well established, with 400+ staff and strong VC backing. Peter had only Vadim and a good reputation, but no money. Peter argued loudly with me – he is known for this – that Percona would become the future of MySQL. Peter said trends were shifting in his favor, and if I had any sense, I’d see it and get onboard.
I was then comfortable and established in MySQL Ab. I saw no need at age 53 to risk everything (again) on a two-person startup. But in 2008, Sun acquired MySQL for $1 billion. Sun’s culture was big corporate America and so unlike the freewheeling Scandinavian culture of MySQL. And being the CEO’s friend at a tiny startup seemed a better place to live than the anonymous middle tiers of a downsizing megafirm.
So I quit Sun and became Percona’s first COO, a title I held for seven years. Later, Peter named me Percona’s Chief of Staff, a role I held until my retirement. In total, I was Peter’s boss for four years, and then he was my boss for 13 years. Vadim went from my second-level report to being my second boss. The arrangements of fate are indeed curious.
So what have I witnessed at Percona, and what have I learned from it all?
Foremost is that in a certain sense, Percona is a web of friendship. It’s a nexus of skilled people who cooperate, communicate, help, and labor hard as friends around a common endeavor. Seeing Percona as a family is an exaggeration, but seeing it as a community of highly interdependent friends is not. The enjoyment of friendships I’ve seen among staff, especially at conferences and meetings, has been deep and real.
Tom jumps off a cliff with Percona colleagues in Cancun, 2011
My memories of Percona include events in California, Texas, New York, North Carolina, Quebec, Mexico, the Dominican Republic, Ireland, the United Kingdom, the Netherlands, Portugal, Switzerland, Estonia, Ukraine, Croatia, and Montenegro. These remain precious memories, and I’m certain I’m not alone in this. The relationships nurtured in these events build up reservoirs of trust and goodwill. This is what oils the progress made during the long months of working in isolation from home.
Percona also welcomed families to most Percona events. My wife and I shared seven trips with Percona to attractive locales. She made Percona friends of her own with whom she’d enjoy reunions. Kids came too, a few times. We also hosted Percona guests in our home.
Vadim with Tom & his wife Kathleen at their home in Maryland, 2007.
Consciously making Percona a family affair was another of Peter’s ideas and a very good one for business. That my family’s welfare was directly tied to Percona’s success was understood by each of us. When I had to upend our schedule for the Percona crisis du jour, family complaints were few. My wife and children’s connectedness with Percona also made working remotely less isolating for me.
I think the interruption of professional friendships has been the worst consequence of the Covid pandemic for Percona. Friendships can be built online too, but it’s much harder when done among those who’ve never once met. There’s something about relating in person that’s unique. The virtual world can imitate but not replace it.
When friends are dealing with friends, all problems get resolved without managerial escalation or bureaucratic morass. All sorts of improvements get implemented with minimal friction. Peter usually fosters debate and invites dissenting opinions, as did Monty and MySQL CEO Marten Mickos before him. Yet honest debate and credible decisions require trust. And trust is best fostered within an atmosphere of authentic friendships.
This dynamic was at play inside MySQL Ab, and I think a big factor in powering MySQL to prominence. Percona, at its best, has continued with this and many other MySQL traditions. For instance, Percona’s monthly All Company Zoom calls are where the entire staff is trusted with batches of corporate metrics, both good and bad, and staff can directly question the CEO.
Yet friendship is not indulgence. Another PZ business lesson I’ve imbibed is that deserved firing is an act of friendship towards the company as a whole. I didn’t fully grasp this pre-Percona. Not everyone makes a worthy friend, and some need to depart before they drag down the whole. Friendship is also not indiscreet. Transparency has proper limits, and some things cannot be explained or even acknowledged.
Another PZ lesson is that what (or who) got you here isn’t what will get you where you need to go. At a certain point, Percona outgrew me as COO. I had the work ethic but not the skills or experience for Percona’s larger corporate stage. Peter helpfully explained this to me at one of our less agreeable meetings. Needful directness is another of his business attributes.
A few years ago, a high-priced consultant flattered our Executive Team by explaining that only 0.4% of all startups reach Percona’s age and size. That means 99.6% of startups don’t survive at all or end up far smaller than Percona if they endure.
The absolute necessity of constant adaptation to a swirl of change became another lesson. I saw in Peter a constant lookout for what’s the next opportunity, the next danger, the process to strengthen, etcetera. My temperament favors stability and predictability, but I learned its limits in a competitive marketplace.
Seizing an unexpected opportunity was part of this lesson. It’s the lesson of taking action now when an opportunity appears, not when it’s convenient. It’s hard to believe, but Percona software wasn’t, at first, part of any grand master plan. Percona did consulting, period. But MySQL Ab had let InnoDB development languish for tangled reasons. Desperate for a workaround, some experts inside MySQL privately asked us to release all of the InnoDB performance patches Percona had accumulated.
These patches were at first bundled as XtraDB, a drop-in replacement for the InnoDB engine. Later this grew into the Percona Server for MySQL, a drop-in replacement for MySQL. Ultimately it grew into a full-fledged engineering team with Percona versions of MySQL, MongoDB, and PostgreSQL, plus the Percona Monitoring and Management (PMM) dashboard. But in a certain sense, Percona’s entire future sprung from seizing one unexpected and seemingly small opportunity.
Peter awards Tom a Percona University Ph.D.!
What does the CEO’s life consist of? What I witnessed includes – Endless meetings. Upset customers. Aggressive competitors. Flaming disagreements. Sudden resignations. Regulatory swamps. No money for payroll. Paying others but not yourself. Failing products. Whispering critics. Continual interruptions.
Crisis after crisis gets lobbed at the CEO, often several at once. It takes guts just to persevere and to improvise when no visible solution exists. I recall an early Executive Team meeting that Peter ran for sixteen straight hours trying to deal with everything; then he reconvened us for more of the same after a few hours of sleep.
I hope Peter and Vadim get rich. Guts should have its reward. I might not have thought this pre-Percona. But life in its trenches convinced me. Expert tech opinion bet heavily against Percona early on. Voices said there’s no profit in consulting, you’re too niche to survive, VC money is your only hope, and similar refrains. It takes some guts to stand alone and not cave into the doubters, especially when there’s a faction on your own leadership team chanting “give up now.” Simply surviving is underappreciated for the victory it represents.
How do you explain business risk to those who’ve never lived it? What does it feel like to have hundreds of families depend on your payroll? How do you cope knowing that other people pay hard for your own mistakes? Percona once misjudged and had to quickly lay off 20 people. It was painful not only for them, since Peter as CEO bore conspicuous responsibility.
I, too, dreamed of VC money as easier than the painful austerity of a bootstrap. Peter was a fanatic about keeping Percona independent. Only gradually did I see how Percona’s freedom from VC interference was paying off. A near-term VC exit strategy looks very different from a lifetime venture strategy. Perhaps VC oversight would have given Percona more discipline and consistency inside or had other benefits. But I doubt it would have been worth it.
My career ended as it began. My boss told me he’d hired a Russian prodigy and asked me to become his manager. The boss this time was Peter, and the employee was Daniil Bazhenov. Daniil barely spoke English, but Peter assured me he was a quick learner and that we’d find a way. We did, along with some amusement. In another bit of cultural whiplash, Daniil’s worked for me from Ulyanovsk, Lenin’s hometown.
Tom & Daniil on the beach in Punta Cana, 2020
I could ramble on and on. But I close my blog and professional life with images of so many wonderful people, which flood my mind.
I think of Monty, who opened the door for my MySQL future, and of my wife Kathleen, who trusted me to walk through it. I think of those very first MySQL employees, Sinisa Milivojevic, Jani Tolonen, Tim Smith, Matt Wagner, Jeremey Cole, and Indrek Siitan, who first wooed me with MySQL’s beauty and excellence.
I think of the MySQL Support Team leaders to whom I owe so much: Dean Ellis, Salle Kermidarski, Lachlan Mulcahay, Bryan Alsdorf, and Miguel Solorzano. Joined with them are Mark Leith, Tonci Grgin, Hartmut Holtzgraffe, Victoria Reznichenko, Todd Farmer, Geert Vanderkelen, Domas Mituzas, Harrison Fisk, Hartmut Holzgraefe, Kolbe Kegel, Matt Lord, Shawn Green, Ligaya Isler-Turmelle, and many others to whom I owe a great debt, including Ulf Sandberg, my boss.
Finally, from my MySQL years, I think of David Axmark, Marten Mickos, Kaj Arno, Zack Urlocker, Edwin Desouza, Brian Aker, Boel Larsen, and other key players who, with Monty, navigated MySQL to stunning success. Please forgive me if the fading of time means I’ve omitted a name I should never forget.
And of my Percona years, how can I begin to name with gratitude everyone I ought with whom my Percona life has intersected. I can only say thank you and again ask forgiveness for any omissions.
First, I think of those who were part of my teams over the years or whom I helped recruit: Mark Sexton, RIP. Svetlana Prozhogin. Kortney Runyan. Natalie Kesler. Andrey Maksimov. Agustin Gallego. Colin Charles. Drew Sieman. Lorraine Pocklington. Daniil Bazhenov. Laura Byrnes. Aleksandra Abramova. Fredel Mamindra. Jana Carmack.
I think of those Percona experts with whom I had the chance to closely interact at different times: Michal Coburn. George Lorch. Alexander Rubin. Ovais Tariq. Marcos Albe. Alkin Tezuysal. Marco Tusa. Liz van Dijk. Kenny Gryp. Dimitry Vanoverbeke. Przemek Malkowski. Lenz Grimmer. Ibrar Ahmed. Tate McDaniel. Yves Trudeau. Yura Surokin. Mykola Marhazan.
I think of those EMT colleagues with whom I shared so many long meetings: Baron Schwartz. Bill Schuler. Ann Schlemmer. John Breitenfeld. Matt Yonkovit. Sam Duffort. Bennie Grant. Jim Doherty. Plus Keith Moulsdale of Whiteford, Taylor, Preston, Percona’s expert legal counsel for many years.
I think of those who’ve moved on but remained friends of Percona: Peter Farkas. Ignacio Nin. Bill Karwin. Aurimas Mikalauskas. Sasha Pachev. Raghu Prabhu. Peter Schwaller. Roel van de Paar. Evgeniy Stepchenko. Morgan Tocker. Brian Walters. Ewen Fortune. Ryan Lowe.
Tom thanks Vadim for his gift of retirement guidance!
And Vadim Tkachenko, how could I almost forget you?!
And Peter Zaitsev. I am so glad we met.
2022 is coming! While we look forward to it, it is time to summarize the results of this year to find out what content was the most viewed, and raised the most interest, during the past months.
Complete the 2021 Open Source Data Management Software Survey to share your knowledge and experience, and help inform the open source database community.
This article is meant to provide a high-level overview of how a web-based application is commonly structured nowadays. Keep in mind the topic presented is very simplified. It is meant as an introduction only and hopefully encourages the reader to investigate some of the concepts in more depth.
Have a burning question about MongoDB?