When I was looking for a new home, I found a house that had a double-wide garage being used as company headquarters for a software start up. When I learned what they were building, it sent a shiver down my spine.
Packed with several desks, some beanbags and an espresso machine, this garage was home to a handful of engineers making safety-critical software for cars. No doubt the engineers were building clever software. After all, the wee company was moving because it had been bought by a big company in the auto industry and needed more space for more engineers so they could build even more software. The question was, were they building good software? There’s quite a difference between clever software and good software.
Good software works well and doesn’t break. Not all clever software is good software.
In my experience, groups of engineers working on clever software seldom understand or value the importance of sometimes slowing down and making sure that quality management experts get involved. Even CEOs of large, well-established companies often make the same mistake – to their peril.
When Startups Skip the Formalities
I started my consulting career deep in the bowels of a software testing lab and, over time, began to consult at the enterprise level to help companies improve their overall product quality. At times, my large corporate clients would call me to help fix issues at the start ups they’d purchased. These startups were certainly building very clever software but within a year or two of being bought out, their software reliably failed, over and over again. It wasn’t that the engineers weren’t top notch. The problem was very often that the smaller companies were led by very bright engineers who’d built the prototype for the very clever software and had a disdain for “process” and “rules” and felt that quality management techniques just slowed them down. Interestingly, at these growing little companies, their staff inevitably were crying out for help in the form of clear and precise development processes, quality assurance and formal testing.
So it’s natural that my first instinct is to never trust the quality of anything built by engineers without a solid Quality Assurance team supporting them. It might be a clever product, but it typically isn’t robust, scalable, or necessarily safe for the long run.
Plenty of startups push forward with a great idea only to find it flounder because building a foundation that supports robust quality and security was never the focus of the founder. Over time, the software becomes strained and things break.
When you realize that even the leadership teams of large companies across most industries seldom understand what needs to happen so that their teams are able to build secure, sound, good quality software, it’s not a stretch to assume that a handful of engineers working in a garage typically don’t, either.
At Big Companies, It’s the CEOs Who Pay the Price for Bad Software
This point was hammered home recently when the CEO of Volkswagen, Herbert Deiss, resigned because of buggy car software. Mr. Deiss had survived the fact that he’d known of the “cheating” (and illegal) emissions software and had done nothing about it. Regulators fined VW and the company moved on. But he couldn’t recover from years of launch delays due to so many software defects in the company’s new vehicles. Reading about the ongoing issues, it seemed to me that Mr. Deiss didn’t really understand what his people needed from the company’s leadership team in order to build safe, sound, bug-free software.
At first, VW decided to launch the ID.3 on schedule even though there were 22 known software defects plus a battery issue being delivered along with each car. They’d just fix the problems after the launch, as so many companies prefer to do. But customers started finding even more problems and so VW halted delivery. Again and again. One customer said that:
His ID.3’s rear camera would randomly turn off, the location of his car in the navigation system and both displays would freeze, the speaker system would make a crackle every time he connected his smartphone to Bluetooth, the keyless system would work intermittently, and a message asking him to take the car to the dealer would pop up every once in a while. We counted 30 glitches so far.
Inside EVs, Oct 30, 2020
VW’s CEO Had Been Warned of the Software Bugs
Volkswagen’s union had earlier warned that many more product defects and delays were in the mix because the “root cause of the issue is reportedly [a] ‘too hastily’ developed all-new basic architecture of the software” plus a lack of programmers. That message should have stopped Mr. Deiss his tracks. But it didn’t. The company ploughed forward, pushing for bug fixes and delivery ASAP without getting to the technical root of the problem: the problem was that the foundational architecture on which the software was built would simply lead to more and more defects as time goes along.
Forget the lack of programmers for a bit – no matter how many programmers you have, and no matter how good they are, if the “all-new” basic architecture was slapped together too quickly, software bugs will flourish and keep on flourishing. Testers will be catching swarms of software bugs for eternity, or at least until the company steps back and takes the time to solve a root cause of problem. In this case, VW needed to redefine a solid new architecture with safety, security, and sound software development practices as the priority.
If you were building a house and discovered that its architecture was fragile, you wouldn’t keep building. You’d stop, take stock of the situation, strengthen the architecture, rebuild the poorly built bits, and then build the rest of the house. It seems that in this situation, VW didn’t take a full, hard stop to fix the architecture.
Foundational Facts
When a company finds that there is an ever-increasing number of software defects, ploughing forward is never the right call. There are some basic “laws” of quality assurance and the first, most important one is this: You don’t test quality into a product. You design and build it in.
In my experience, not enough executives understand the foundational principles to build a solid software solution. And that’s too bad because the backbone of companies across virtually all industries is now software. Whether building cars or financial products or pizza delivery systems, software can make or break a company and a relentless stream of software defects can make or break the career of a CEO. Mr. Deiss recognized that building buggy software would continue to affect many other companies.
Before resigning, Mr. Deiss explained to staff that:
“Where we are struggling through,” Diess said, “that will be the same for everybody else.”
The comment was a nod to issues many VW owners are having with their vehicles, from freezing touchscreens to buggy driver-assistance functions, customers are opting to just turn off. Trouble at the company’s software unit Cariad is delaying crucial new models and one of the major reasons Diess was forced to hand over the keys . . .
Bloomberg, Sept 14, 2022
I hope the new CEO at VW takes a hard look under the hood of his company’s car software development processes. His people have been warning executives for years that the underlying architecture needs fixing – that the root cause is not bad coding but that people were pushed to develop the architecture too quickly.
The Non-Technical Root Cause (Source of the Problem)?
In my experience, there is almost always another, non-technical “root cause” to many software quality issues. I suspect that in VW’s case, the cause of the software problems was sitting right there with Mr. Deiss on the executive floor. Executives didn’t seem to understand that you can’t build better software by moving faster and faster. Moving fast and breaking things, as the Mark Zuckerberg adage goes, is not what any company that’s building cars should ever be doing.
Here’s the Key
Great software starts with a solid, “bulletproof” architecture which sets the foundation for a design that prevents bugs before they get coded and delivered to the customer. That takes time, full quality reviews, and even boring quality assurance audits. Sometimes moving slowly and wisely at the outset gives your company a brilliant head start. Why?
Because as your competitors sink under the weight of their own sloppy software, your bulletproof solution helps shorten the delivery cycle and you’ll have fewer issues. This advice applies to big companies like Volkswagen just as much as it applies to anyone with big dreams building clever software in a garage.
