The Internet, as anyone who works in the trenches can tell you, is not a smooth, well-oiled machine.
It's a messy patchwork that's been pieced together over decades and is held together by the digital equivalent of duct tape and chewing gum. Much of it relies on open source software thanklessly maintained by a small army of volunteer programmers who fix bugs, patch holes, and ensure that the entire rickety contraption, responsible for trillions of dollars in global GDP, continues to function. .
Last week, one of these programmers may have saved the Internet from huge problems.
His name is Andrés Freund. He is a 38-year-old software engineer who lives in San Francisco and works at Microsoft. His work involves developing open source database software known as PostgreSQL, the details of which would probably bore you to tears if I could explain them properly, which I can't.
Recently, during a routine maintenance task, Mr. Freund inadvertently found a hidden backdoor in software that is part of the Linux operating system. The backdoor was a possible prelude to a major cyberattack that experts said could have caused enormous damage if successful.
Now, in a Hollywood twist, technology leaders and cybersecurity researchers are hailing Mr. Freund as a hero. Satya Nadella, CEO of Microsoft, praised his “curiosity and mastery”. An admirer I called him “the silverback gorilla of nerds.” The engineers circulated an old webcomic, popular among programmers, about how all modern digital infrastructure is based on a project run by some random guy in Nebraska. (According to them, Mr. Freund is the random guy from Nebraska.)
In an interview this week, Mr. Freund — who is actually a soft-spoken German programmer who declined to be photographed for this story — said becoming an Internet folk hero had been disorienting.
“I find it very strange,” he said. “I'm a pretty private person who just sits in front of the computer and hacks code.”
The saga began earlier this year, when Mr Freund was returning from a visit to his parents in Germany. While examining an automated test log, he noticed some error messages that he didn't recognize. He was suffering from jet lag and the messages didn't seem urgent, so he filed them away in his memory.
But a few weeks later, while running more tests at home, he noticed that an application called SSH, used to access computers remotely, was using more processing power than normal. He traced the problem to a set of data compression tools called xz Utils and wondered if it was related to previous errors he had encountered.
(Don't worry if these names sound Greek to you. All you need to know is that these are all little pieces of the Linux operating system, which is probably the most important piece of open source software in the world. The vast majority of the world's servers , including those used by banks, hospitals, governments, and Fortune 500 companies, run on Linux, making its security a matter of global importance.)
Like other popular open source software, Linux is continually updated, and most bugs are the result of innocent mistakes. But when Mr. Freund looked closely at the xz Utils source code, he saw clues that it had been intentionally tampered with.
In particular, he discovered that someone had inserted malicious code into the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user's SSH connection and secretly run their code on that user's computer.
In the world of cybersecurity, a database engineer who inadvertently finds a backdoor in a core Linux feature is a bit like a baker who smells a freshly baked loaf of bread, senses that something is wrong, and correctly deduces that someone has tampered with the entire global yeast supply. . It's the kind of intuition that requires years of experience and obsessive attention to detail, as well as a healthy dose of luck.
Initially, Mr. Freund doubted his findings. Had he really discovered a backdoor in one of the most closely scrutinized open source programs in the world?
“It seemed surreal,” he said. “There were times when I thought, I must have had a bad night's sleep and had some feverish dreams.”
But his research continued to uncover new evidence, and last week Freund sent his findings to a group of open source software developers. The news set the technology world alight. Within hours, a solution was developed, and some researchers credited it with preventing a potentially historic cyberattack.
“This may have been the most widespread and effective backdoor ever installed in any software product,” said Alex Stamos, chief trust officer at SentinelOne, a cybersecurity research firm.
If it had gone undetected, Stamos said, the backdoor would have “provided its creators with a master key to any of the hundreds of millions of computers around the world running SSH.” That key could have allowed them to steal private information, install crippling malware, or cause major disruptions to infrastructure, all without getting caught.
(The New York Times is suing Microsoft and its partner OpenAI over claims of copyright infringement involving AI systems that generate text.)
Nobody knows who planted the backdoor. But the plot appears to have been so elaborate that some researchers believe only a nation with formidable hacking skills, such as Russia or China, could have attempted it.
According to researchers who went back and examined the evidence, the attacker appears to have used a pseudonym, “Jia Tan,” to suggest changes to xz Utils as early as 2022. (Many open source software projects are governed via hierarchy; developers suggest changes to a program's code, so more experienced developers known as “maintainers” must review and approve the changes.)
It appears that the attacker, using the name Jia Tan, spent several years slowly gaining the trust of other xz Utils developers and gaining more control over the project, eventually becoming a maintainer and eventually inserting the code with the backdoor hidden early on of this year. (The new hacked version of the code had been released, but was not yet widely used.)
Mr. Freund declined to guess who might be behind the attack. But he said whoever it was had been sophisticated enough to try to cover their tracks, including by adding code that made the backdoor harder to detect.
“He was very mysterious,” she said. “They clearly spent a lot of effort trying to hide what they were doing.”
Since his findings became public, Freund said, it has helped teams who are trying to reverse engineer the attack and identify the culprit. But he's been too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is due out later this year, and he's trying to make some last-minute changes before the deadline.
“I don't really have time to go out for a drink to celebrate,” he said.