Everything we do is based on the work of others. From sharing stories, knowledge, methods of observing and interpreting to making and sharing tools, it is how humanity has been able to thrive, evolve and survive. It does not matter if you are constructing a building, trying to solve a math problem or cooking a dish with a list of ingredients. You are reusing the work of others where the only reason why tools and methods are reused is their proven track record together with a certain level of quality or function. With quality comes quality control, and with control comes ownership and a free market of competition that drives our innovation forward. A luxury – but also a hidden challenge – we all experience with this concept is that we generally are no longer focused on individual components. Rather we focus on convenience and end-product. We all love the convenience of fast food you can just grab on the way home, or the availability of ready-made roof tiles for constructing your house. We do not have to worry about quality control because for a lot of these things there are regulatory or governmental quality control watchdogs in place to make sure that certain expectations will be consistently met. Even more, any kind of quality anomaly or deviation can be identified and backtracked in order to limit its impact and address the root cause.
This is where the modularity analogy starts to break down and deviate from what we observe in the world of software development and how we benefit from the convenience of software. The reason why anyone can write software in a modular way and use the hundreds of thousands of software packages already out there is also the reason why we are where we currently are with the cloud, IoT, mobile applications, operating systems, games, and any other software.
Free has a price
The world of software development and maintenance, and especially the world of free and open-source software (FOSS), follows the exact same modular principle of building on top of each other’s work. Different kinds of software packaging conventions have been worked out over time to make the inclusion of other people’s work as easy and modular as possible. At least, that was the idea. In the last 20-30 years, we have seen an explosion of FOSS frameworks providing package managers, collaboration tools and other software maintenance tools. This is the reason we all have mobile devices in our pockets filled with long lists of mobile applications and computers everywhere: in our TVs at home, in our cars, in airports and in any part of the modern society. Millions of computers, embedded systems and devices, all running FOSS software or at least some FOSS components. The plethora of different and individual FOSS projects, but also just the idea itself as a principle has been a revolution in the way we traditionally looked at software: instead of paying for software, you get it for “free”. Of course, you pay for it with giving support, adapting it to your specific needs and with giving the adapted and improved source code back to the community. This is something different from when software first emerged and started proving its worth in government and corporate environments and later in our households and the education sector. The FOSS movement 30-40 years ago was a push against the monopolization attempts of a number of big software players determining the beat of the market and setting the incentives by only allowing expensive subscription models. As home computers (then called “microcomputers”) were not a thing yet, those companies had the monopoly of services. Most software existed locally, and low performance devices were used to access the central repository of software services. Could this feel familiar when thinking of how we use cloud services today?
The “journey” and the hidden risk
Every new FOSS project, no matter how large or small, always starts out with a goal in mind to be able to accomplish something that has not been done yet or can be achieved in a different way. Unfortunately, this goal is only the visible part of the story and the only one which is currently in the focus for the software industry. A key problem is that a lot of FOSS is made by volunteers and organizations with widely varying incentives for the maintenance and support of the software. The following challenges require attention:
Time cost – Any software development takes time, and that time needs to be paid – by being on someone’s payroll, by getting funds from donations or grants, by working in the open source model at a company, or by being a volunteer and sacrificing your time in lieu of something else. Any software project has a limited number of days.
Maintenance cost – Just like for an Olympic games event, infrastructure, stadiums and villages, is built in a short time (with a lot of enthusiasm and fanfare). Only the things that are required are built and only towards specific finite specifications. But we often see the Olympic infrastructure of the past, with its swimming pools, stadia and Olympic villages, in a state of trees and weeds growing out of it. There is no incentive or need for maintenance, at least not in the eyes of the creators. And most of the infrastructure was created for such a specific purpose that it is hard, almost impossible, to repurpose it. FOSS is very similar to this and makes one feel that it is easier to create something than to maintain it. As maintaining software means being up to speed with what is going on in different software eco-systems, looking at what has been developed recently and whether it is time to start jumping trains which are faster and easier in the software creation and maintenance process. This means that a lot of software in the FOSS world is “as is”, abandoned or not even functional anymore in certain cases. If there is nobody paying for the maintenance time (see above), then we are left with the current situation: critical and widely used pieces of software rely on the time and effort of a small number of volunteers that may not be available anymore next year or next month. Any ingredient or building material has an expiration date to check before we use it, but software modules do not and are often left to languish. Unfortunately, software ages like milk, not like wine.
Responsibility – When it comes to embedded systems, appliances and other software integrated for a certain outcome, it is difficult to determine where the ultimate ownership and responsibility actually lies. Who is responsible for making sure the software and all its individual dependencies still perform their core functionality and the level of security is maintained through patch management efforts, access control adjustments, etc.? A lot of software is used in an OEM fashion where software is integrated by a number of parties around the world, for many of which no real tracking exists.
Ownership –Software will always require update mechanisms for patching vulnerabilities, handling security problems, adopting new standards, improving performance, and fixing bugs. Software supply chains are chains and are only as strong as their weakest link. If software developers, including FOSS volunteers, are no longer able to check their e-mail, DNS or other project hosting information, then a potential loose thread is created within the software eco-system. Attackers can scoop up the DNS domains of developers which are no longer paid for but for which e-mail infrastructure still exists. Such attackers can then gain access to a developer account of what might look like a small and ordinary software package and commit malicious code, which can lead to a compromise of behemoth-sized commercial software projects using that package.
Side effects
“We are in the golden age of the loss of ownership”
The same challenges that were experienced during the 70s and 80s are vital today, though sometimes for different reasons. It makes sense for a software development company to centralise services and sell access to those, but the consumer advantages only go so far. We are in the golden age of the loss of ownership when it comes to software and content. It seems nowadays companies let us rent, loan, borrow, subscribe, digitally download everything we consume, instead of spending a lot of money, time and effort for keeping software aligned and updated on customer’s devices and investing in Digital Rights Management (DRM) to prevent theft or loss of the projected revenue. Nowadays everything is a service that needs to be subscribed to. This has advantages for the suppliers and creators of software, but the advantages for consumers that just have to face newly raised monthly rates are waning. It is hard to prevent things like vendor lock-in, when the vendor is the only one creating the service or product your business depends on. It is a new kind of monopoly that sweeps through every company and household. While this helps with managing the supply chain challenges in many cases, this comes at a cost of no longer being able to own anything and makes our planning for continuity and recovery more difficult.
The need to regain control
“Companies have effectively lost most of the control of their software stacks, especially when it comes to FOSS integrations or commercial software in an appliance form”
When it comes to modular software design and the larger FOSS supply chain, the genie has been out of the bottle a long time ago. The current situation of FOSS and commercial software is where convenience has taken precedence over ownership and control. Risks are transferred to the vendors that supply and integrate the software services but ultimately for a lot of organizations it is a bet with extremely high stakes, which are mostly not even understood. We have given up our requirements or desire to know what is inside the software that businesses and governments depend on.
This has multiple implications that can impact individuals, whole countries and global supply chains:
- Vulnerabilities and security weaknesses go undetected until it is too late
- Lack of ownership, problematic ownership and ownership changes are not addressed
- Core or dependency software packages can be hijacked, without a chance for timely recourse
- Lack of insight, lack of interest and lack of governance, risk and compliance frameworks mean the lowest bidder of software wins
- Companies have effectively lost most of the control of their software stacks, especially when it comes to FOSS integrations or commercial software in an appliance form
We will now present several recommendations that can help mitigate software supply chain challenges.
Transparency
We may not be able to stop or change the way software is developed, integrated or used. But at the very least, we can outline Software Bill of Materials (SBOM) requirements for software developers. Having a list of the software components used in a given product or system allows organisations to track the origins, authors and package management and regain a bit of control to hedge their risk and threat model accordingly. If this can be integrated as part of the DevOps practices and the CI/CD[1] infrastructure of larger software houses, then at least the software end-users will have a fighting chance to follow the comings and goings of the authors of the most critical software dependencies. Software buyers demanding to see the SBOM for the software on their networks is, of course, a prerequisite for this.
Pick the right architecture
“The total cost of software ownership is not just the buying price: buying or developing software is a lot cheaper than maintaining it”
Some FOSS frameworks are more adaptive and better supported than others and some software frameworks are picked mainly because the maintenance, support and bespoke development can be outsourced at a lower cost. One should always check the number of published vulnerabilities for the software frameworks they are evaluating and question and challenge their software developers, third party developers and DevOps partners on what frameworks they adopted. Some software frameworks look cheaper in their perceived cost of development and maintenance, but the budget saved can also vanish quickly if a software framework is prone to a high number of vulnerabilities per month, which will require threat analysis on a regular basis to calculate the impact on your business.
Expect security problems and compartmentalise
Any software is bound to have bugs, vulnerabilities and security weaknesses, so one should be prepared to security problems. Expect that the “black box” commercial software product, appliance or FOSS project you procured will have ownership issues, exploitable vulnerabilities, or in some cases trojans[2]. Expect that the company that made the software will go out of business or will get acquired by another company that may drop the software / project support within a year or sooner. Ensure that your overall architecture allows for such a software or its eco-system to be segregated from the rest of the organisation with access control, and only allow communication and functionality for which there are clear documented business requirements.
Detect, respond and test regularly
Gather relevant threat intelligence, model relevant risks, and produce a threat model to see what controls can be put in place for each threat scenario to ensure that:
- Potentially undesirable behavior of software can be acted upon as part of a strategy for quick detection and containment of malicious behavior
- Purple teaming and threat simulation exercises can be run to test your detection and response capabilities for relevant supply chain-related threats
Ensure continuity as part of your security assurance plan
If a software you are procuring is mission critical, then business continuity planning (and budgeting) is required:
- Make sure you have your “plan B” and you know how long and with what expectations you can conduct your business without the software in question
- Analyse what can be achieved through a redundancy strategy, e.g., using software of different vendors
Conclusion
In this blog post, we shared our thoughts on the current software supply chain challenges and ways of addressing those. As stated in Internet Organised Crime Threat Assessment (IOCTA) 2021 report:
“The previous IOCTA reported on the potential threat that attacks can pose on IT supply chains. During the reporting period, we have seen these concerns come to fruition through the examples of the SolarWinds, Kaseya and Microsoft Exchange Server attacks. Criminals have realised how much potential there is to compromise digital supply chains …”
Fully recognising the scale of the problem, CYBERSPACE places it high on the project agenda and expects to contribute with further research and recommendations.
[1] Continuous Integration and Continuous Delivery
[2] Modules or programs designed to breach the security of a computer system while pretending to perform innocuous functions
Author: Tom Van de Wiele (WithSecure)