By Jack Lindsay.
Recently there was a massive vulnerability found in a critical Java ecosystem package. When fully weaponized the vulnerability allows attackers to manipulate Java servers into executing arbitrary code that is fetched from an LDAP server. It is an entirely reasonable solution to a real-world problem and accidentally created a massive hole in countless networks.
This microcosm of the Open source ecosystem demonstrates why the Open source community may soon vanish.
If you want me to make you useful software, pay me.
The ‘Log4j’ project is so beneficial it is essentially in the standard library for Java users.
As of December 11, 2021, there was three sponsors for the author’s work. A day later, after it compromised a significant number of systems, the number had increased to 14.
It is the yet unresolved – and possibly unresolvable – conundrum of Open source. Developers are put in a situation where software they create as a passion project on the side can end up as a critical component within a company’s stack. Ultimately, as is the case with Log4j, these developers find themselves in the precarious position where they’re not being paid for their efforts but are suddenly held responsible for significant disruption.
The idea that Open source is a pool of free labour – or ‘leech culture,’ as it has become known – has accelerated technologists well beyond the efficiency levels capable previously. It has also put the sector in a vulnerable state where a random smattering of volunteers have become critical for the business continuity of some of the largest enterprises in the world.
It makes sense. The Open source community build valuable software. The vulnerability that created the #Log4j twitter storm is actually a great way to get a shell whenever you want. Which is why it is used so widely. As a result, there is a strong push within the community for companies that rely on Open source projects to recognise their ‘moral obligation’ and support projects by making donations.
The fallacy of Open source software development
“Open source software is developed in a decentralised and collaborative way, relying on peer review and community production. Open source software is often cheaper, more flexible, and has more longevity than its proprietary peers because it is developed by communities rather than a single author or company” says Redhat when explaining what open source software is. This is the essence of open source, of which all projects are merely imitations. Something the community aspires to achieve but ultimately falls short.
A recent example of this is the code-js debacle. This JavaScript library gives JavaScript’s standard library a lot more core primitives that make it so programmers do not need to reach out to other libraries. Of greater significance, it is a core dependency of React.
In early 2020, this project had a single administrator, was contributed to by this person alone and had 25 million weekly downloads. The author is infamous for letting programmers know they are looking for a new job every time you install the project in CI.
The author was then incapacitated for one and a half years, causing chaos in the community.
The project is key to the success of almost all JavaScript companies and was maintained by a single developer. It is one of countless Open source projects of its kind that form landmines scattered throughout the information technology industry.
Open source is to information technology as Wikipedia is to academia
Confronting, yes, but this shouldn’t come as a surprise to anyone.
The market rate for a developer who would maintain a project like Apache Log4j is between $200,000 and $300,000/yr. Meanwhile, the most you are likely to see any Open source developer rack up on Patreon and/or GitHub is ~$1,000/month.
Understandably, the developers then choose to spend their time working on changes that are of most interest to them. Not running sprints of bug fixes following a thorough security audit.
This, at a time when the first rule of being a good programmer is “don’t reinvent things.” Instead, re-use code libraries and packages of previously written code that can be used in your own program to accomplish a task. It’s the rational thing to do when building something complex.
Developers have been commoditized; And with every commodity the key to profitability is velocity. The culture this has created within the information technology industry is one that encourages, demands, and rewards developers for grabbing tools like Log4j as an easy way to handle a problem as someone else has already done the work. It is in this world of copy-paste programming, where policymakers have struggled to keep pace and layers of management demanded higher returns, that the Open source community has thrived.
Log4j has shown the world what this ecosystem can do. Corporations understandably encourage cost-saving and efficiency. But given the dangers, strict policies requiring rigorous analysis and proof of safety and security are necessary. Unless these are created and enforced, there is simply too great a risk that financial pressures will systematically bias decision making in favour of time- and cost-savings. Of course, some decisions should have shorter timelines than others, and a full-blown security risk review is not always practicable. However, without protections, in the world of copy-paste programming, it is inevitable that development teams will globally distribute CVEs.
In practice, Open source projects are volunteer-run. Issues remain unnoticed for days, weeks, months or even years. It should be expected that these projects will compromise systems. It is evident why a government auditor would question any project maintained by a random person in Nebraska within your stack. Suddenly, the great ideas and passion projects that once formed the pool of free labour technologists used to stitch their software stacks together represent vulnerabilities.
Expecting projects – typically maintained by a single developer – to maintain a professional level of DevSecOps is unrealistic. It would require a significant philosophical shift in the way developers approach Open source projects. Nonetheless, if companies want to include an Open source project in their stack, they need to be capable of evidencing the appropriate controls.
Regulating against software vulnerabilities
Countries are taking steps to ensure these controls are in place. Australia’s Critical Infrastructure (Amendment) Bill 2020 is the nation’s most significant step toward such controls. Similar regulation from the United States and United Kingdom is also being implemented. Affirming that governments will intervene to ensure IT practices meet necessary safety and security controls for the industry. The most well-established attempt is the Cybersecurity & Infrastructure Security Agency’s (CISA) assessment evaluation and standardisation (AES) program. A newly introduced federal government initiative that is training ‘assessors’ nation-wide to standardise assessment and introduce a performance baseline. These regulatory ‘assessors’ are tasked with evaluating and reporting on the workforce, operational resilience, cybersecurity practices, organisational management of external dependencies, and other key elements of a robust and resilient cyber framework.
These assessors will ensure a day comes where it is no longer possible for an organisation to take software someone wrote for free, and put it into production, without robust security controls. Changing the essence of Open source from collaborative software development to a community of ideas that require industrialisation. At the same time making the developer dream of working full-time on a community funded Open source project almost impossible.
About the Author
Jack Lindsay’s primary focus is on management, sales, and technology issues in industry focusing on software and security. Jack brings expertise in learning, coaching, and software options at every level to ensure companies are successful at people, strategy, execution, and finance. Jack is co-founder of Upward Spiral, an innovative solution to the recruitment issues facing the infosec community. Their mission is to help 1 million cyber security professionals find their dream job and measurably improve the job-seeker and hiring experience. When Jack isn’t working, he is a Board member at the women’s international cycling union (The Cyclists’ Alliance), contributor to various cycling websites, hockey player in the Bundesliga, and involved in various InfoSec and FinTech conferences
