devopsy.kinsta.cloud

Tag: embedded systems

  • Embedded Connectivity Solves the Right (and Left) Problem

    Embedded Connectivity Solves the Right (and Left) Problem

    In Steven Bellovin’s book, “Thinking Security: Stopping Next Year’s Hackers,” (2015) he wrote:

    “Companies are spending a great deal on security, but we read of massive computer-related attacks. Clearly, something is wrong.

    The root of the problem is two-fold: We’re protecting (and spending money on protecting) the wrong things, and we’re hurting productivity in the process.

    InfoSec is hurting productivity in the organization it is supposed to be protecting.”

    Bellovin clarified what he meant by “the wrong things.” He observed that security measures tend to stick around even when the threat they were intended to address has faded into the past. Those security costs have only increased since 2015 when the book was published, and today represent a trillion-dollar drag on the global economy despite the emergence of security-integrated models of software delivery such as DevSecOps, shift left and shift right.

    The 2021 IBM Security Threat Intelligence Index reports the No. 1 attack vector was a surveillance technique known as scan-and-exploit. Defenders face an asymmetric engagement with such network threats. The successful attacker needs only to exploit a single vulnerability whereas defenders must secure 100% of the network’s ports of entry 100% of the time. The fundamental problem is the anonymous and permissionless nature of the internet. This makes it both useful and easily abused. Anyone anywhere may conduct reconnaissance and launch attacks at a massive scale with little cost or risk. How can we make it prohibitively difficult or costly to successfully surveil and attack our own applications? What if we could reliably defend against the entire category of active scanning tactics?

    There are no secure networks, only isolated networks. Taking this back to first principles, a network’s purpose is to convey packets of data from sender address to recipient address. The network has no built-in ability to identify either party or to discern the legitimacy of any activity. Active scanning techniques identify potential targets that are then probed by scripts that may be intrusive or stealthy or both. We could simply reject all incoming network traffic with a firewall, but application servers must be continually reachable in order to listen for incoming requests. Any interruption of availability causes errors like “connection refused” and “service unavailable” for the user. 

    This necessary network exposure leaves the upper layers of the network and the application server itself vulnerable to intrusion and denial of service attacks. “Private” enterprise applications are similarly exposed to the famously-vulnerable group that are VPN users’ devices, and so are by no means immune to this problem. This presents the problem of automatically discerning which incoming connections are legitimate, continually, forever. Are we trying to solve the wrong problem?

    Ideally, we would identify the sender before they connect to the application server. That way, we can know whether they have permission to connect at all. This is something we simply can not do when the sender is known only by a network address. We can’t ask for a credential because that introduces yet another application layer with all of the same problems. This is precisely why VPNs do not go far enough: A superficial barrier is erected for users that prevents the application server from checking for permission until after surveillance and subsequent attacks have occurred. 

    Embedded connectivity-as-code solves the right problem. This represents a shift in thinking about the relationship between the application and the network. Now, the application brings its own point-to-point overlay with a verifiable identity on both ends. This has significant implications for the security of the application that can be summarized in terms of shifting security left in the development timeline and shifted right in the life span after deployment.  

    Shift left: OpenZiti makes embedding connectivity-as-code convenient for developers and transparent for auditors and users. As open source software, it fits into DevSecOps processes that scan the source code. The management side of the OpenZiti network is cloud-native and API-driven and fits into the DevOps paradigm of continuous integration testing. For example, a network-as-code may be continually created and destroyed by a pipeline to exercise any imaginable configuration of endpoints, services and policies, or to facilitate dynamic scanning of the data edge and fabric APIs. 

    Shift right: OpenZiti allows us to shift right security to measure and monitor applications after development, in production, with a novel ease and without compromising security. Operators maintain perfect visibility of deployed applications and need not cope with the complexity of hopping through security gateways or bastions or worrying about external malicious actors and inbound connections. This best-of-both-worlds reality is a natural effect of  an access control scheme that is based upon verifiable cryptographic identity, not a weak identifier like an IP address. Importantly, this means it is always knowable which identity is accessing what services, when they were accessed and to what extent.

    OpenZiti gives applications superpowers! To name a few: Identity-driven addressability, private domain name system, shelter from unauthorized connections, portable and programmable micro-segmentation, end-to-end encryption, solves the IPv4 shortage without IPv6, smart routing to choose optimal routes across the internet and is transparent to end users.

    Learn more about OpenZiti by checking out our Quickstarts or engaging in conversation on Discourse. 

  • Engineering Applications for DevOps (Part 5)

    Engineering Applications for DevOps (Part 5)

    This blog series explains how to engineer applications for DevOps.

    Topics covered in this blog series:

    • Factors to decide whether an application is a good candidate for DevOps.
    • Practices to engineer designs for DevOps.
    • DevOps applied to enterprise apps and software services.
    • DevOps applied to COTS systems.
    • DevOps applied to manufactured (embedded) systems software.
    • Five levels of application maturity.

    This blog covers DevOps applied to manufactured (embedded) systems software. You can read part one here, part two here, part three here and part four here.

    DevOps Applied to Manufactured (Embedded) Systems Software

    In the late 1970s (yes, forty years ago!) as a young engineer, I was given an assignment to figure out how to accelerate the software development and delivery cycles for an innovative packet switch called SL-10, (predecessor to the more recent model called Data Packet Node [DPN]) manufactured by Northern Telecom. Each SL-10 “node” consisted of millions of lines of original embedded software source code and custom hardware configured with many interconnected processors and protocols interfacing with other nodes, host devices, and terminal devices.

    Given the complexity of the system and lack of automated build, test and deployment tools, it was taking more than twelve months to generate each release of the system, and there typically were lots of bugs even after the release. Without knowing much about DevOps (the word “DevOps” was not invented for another twenty-nine years), we proceeded to implement a continuous delivery system that included automated build, test and delivery tools and workflows.

    The result was remarkable. Lead times for releases to manufacturing were reduced more than tenfold, from twelve months to one month, and quality, measured as defects found by customers per release, also reduced tenfold. This dramatic shift in performance and capabilities made it possible for the engineering and manufacturing teams to accelerate the rate of innovation and trample the competition. The SL-10 product, and its rebranded DPN version, quickly dominated the connection-oriented packet switch market, and it became the pride of the Nortel corporation and the Canadian nation. Even cooler, I got a nifty president’s pin and promotion which paved the way for my career that led to writing a book and this series of blogs! If this real-life example is not sufficient to demonstrate that DevOps can be applied to manufactured embedded software products, then I don’t know what is.

    There have been many other examples published that show how DevOps practices applied to embedded manufactured systems have reaped similar benefits. A notable example was published in 2013 by Gary Gruver in A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware.

    That is not to say that DevOps applied to manufactured software embedded systems is the same as applying DevOps to enterprise applications or COTS systems. A paper published on the CloudBees blog, Five Aspects that Make Continuous Delivery for Embedded Different, explained some of the key DevOps practices used with embedded systems.

    DevOps Benefits for Embedded Systems

    Benefits of DevOps for embedded systems are summarized as follows:

    • Retrofitting the huge code base typically found in legacy embedded software systems into a DevOps model is not easy. It is likely to be expensive, but it is almost guaranteed to be a worthwhile effort, especially if you have a longer-term vision and intend for your product to stay competitive in the future.
    • Developers of embedded software need access to massive amounts of computer resources for testing their real-time embedded code on powerful hardware simulators.
    • Developers and testers need access to physical product hardware to run their binaries for system tests that cannot be run on simulators alone. Lab-as-a-service (LaaS) systems that orchestrate testing topologies of real devices can improve sharing of critical resources. LaaS systems provide a test framework that can be integrated through APIs into a continuous delivery pipeline.
    • Long build and test processes may need to be reengineered to reduce serialized lead times into quicker build times made possible by scaling build and test resources.
    • Long compliance and conformance test processes for embedded systems need to be accelerated with automation and resource scaling.
    • Focus on continuous delivery rather than continuous deployments in which shippable product versions are frequently delivered to manufacturing for verification, but the frequency of deployments to end customers is determined by other business factors.

    What This Means

    This blog is one of a series explaining how to engineer applications to be most suitable for DevOps. This blog series explains how to engineer applications for DevOps including:

    • Factors to decide whether an application is a good candidate for DevOps.
    • Practices to engineer designs for DevOps.
    • DevOps applied to enterprise apps and software services.
    • DevOps applied to COTS systems.
    • DevOps applied to manufactured (embedded) systems software.
    • Five levels of application maturity.

    This blog covers DevOps applied to manufactured (embedded) systems software. You can read part one here, part two here, part three here and part four here.

    For more information regarding how to engineer applications for DevOps refer to my book Engineering DevOps.

  • Functional Safety in Embedded Design Starts with Code Development

    Functional Safety in Embedded Design Starts with Code Development

    With embedded software likely to become more complex, now is the time to start thinking about functional safety

    As the world has become increasingly more dependent on embedded systems in functional safety markets—such as rail, automotive, aerospace and medical—ensuring the safety of the software in those systems has become even more imperative. That assurance must begin with software development.

    As we all know, software development is the point at which many vulnerabilities are introduced, potentially leading to future performance, security and safety issues. The challenge for many embedded software developers is that they are often dealing with highly complex environments. In general, these complex environments are built using C and C++ programming languages.

    While C and C++ provide developers with the flexibility and scope for the innovation that embedded environments demand, they also introduce risk. Code can have undefined or unspecified behavior, or may not behave the same way when run on different hardware. Even the most skilled developer may make decisions that inadvertently lead to errors.

    Embedded Systems and Functional Safety: Cultural Appreciation

    There are ways to reduce software development risks. From a cultural perspective, safety and security need to be baked into the software development process, right from the very start and continuing throughout the entire DevOps life cycle. Of course, this is a practice that should be happening across all software markets, not just within embedded design.

    At the same time, there needs to be a greater focus on enforcing compliance with functional safety standards. This practice requires strategies to ensure the safety and security of systems. In some industries, such as rail and automotive, the use of these standards is mandated as part of approval and compliance processes.

    Functional Safety Standards

    Functional safety standards require performing a hazard analysis and risk assessment of the system. Based on the results, safety functions are then implemented to reduce the risk to an acceptable level. Functional safety standards set out appropriate software development and verification actions to mitigate known risks.

    Some of the main functional safety standards in use today include IEC 61508, which covers electrical, electronic and programmable electronic safety systems. First published more than 20 years ago, IEC 61508 is also the basis for several other functional safety standards including ISO 26262 for automotive, EN 50128 for rail, IEC 62304 for medical devices and ISO 25119 for agricultural machinery.

    Similar to other standards, functional safety standards do not always receive a warm reception because there is a perception that they require additional work and cost. In fact, by requiring consistent, high-quality coding practices, functional safety standards can reduce time-to-market and cost because they introduce best practice efficiencies into the development process.

    Coding Standards

    Most functional safety standards require or recommend the use of coding guidelines. For example, in the automotive industry, ISO 26262 mandates the use of coding guidelines and highlights specific areas that must be covered, such as the use of a safe language subset and naming conventions.

    Typically, a coding standard is made up of the collective knowledge of many people and organizations, such as manufacturers, consultants and industry associations. This collective knowledge is then used to provide developers with a set of rules to ensure that the code they are developing is safe, secure and compliant. Coding standards also engender better quality coding practices by making code more consistent, readable and understandable within a team.

    Depending on the project, organizations often choose to implement an external coding standard either fully or partially. They may also decide to create their own in-house coding standard based on an external standard or their own experience.

    While MISRA C and MISRA C++ were originally created for the automotive industry, the coding standards are now widely used across multiple industries. AUTOSAR C++14 Coding Guidelines were developed to address the requirements of modern C++ development environments, including autonomous vehicles and in the future will be used as the basis of a new MISRA C++ standard.

    Another popular coding standard used across multiple markets is CERT (coordinated by the CERT division of the Software Institute at Carnegie Mellon University), which focuses on security. With increasing awareness of cybersecurity risks, the use of CERT in embedded software development is likely to increase.

    Successful Implementation

    Regardless of the approach, successful implementation of coding standards depends on several factors. As there may be resistance to the implementation, teams need to be involved in the selection process of a coding standard and supporting tools. By being included in the decision process, it helps the team to understand why the requirement is critical.

    In addition, while it is possible to carry out manual code reviews against a coding standard, any project of significant size will need an automated static code analysis tool.

    Static code analysis tools can detect thousands of defects quickly and accurately, which frees up the developers to consider the implementation of the design. These tools work by both analyzing an individual developer’s code changes before they are committed and then scanning throughout the entire project as part of continuous integration (CI). For that reason, static code analyzers harmonize well within DevOps-led development environments.

    With embedded software likely to become more complex—especially against a backdrop of AI, ML and IoT—now is a good time to start thinking about putting in place an effective functional safety strategy, which includes tools such as static code analyzers and coding guidelines.

  • Wind River Launches CD Platform for Embedded Systems

    Wind River Launches CD Platform for Embedded Systems

    Wind River is fostering the adoption of DevOps processes for embedded systems by making available a continuous delivery (CD) platform that its customers can download.

    Glenn Seiler, vice president for open source strategy for Wind River, said the CD platform is based on an open source instance of the Jenkins continuous integration/continuous delivery (CI/CD) platform. The company envisions organizations downloading the platform as source code that Wind River will update regularly for use on top of the Wind River Linux platform, he said.

    Wind River is taking advantage of containers, Kubernetes, the Puppet IT automation framework and a repository to ease deployment of its CD platform, added Seiler. The company is committed to providing updates to that platform, which Wind River uses internally, at least every three weeks, said Seiler.

    Wind River expects most organizations will already have some form of a CI process in place. The CD platform provided by Wind River will focus on accelerating the delivery of applications built using whatever CI process organizations have in place today, said Seiler.

    In the embedded systems space, Seiler said the adoption of best DevOps practices is still in its very early days. However, with the rise of edge computing and the internet of things (IoT), organizations will need to find a way to securely deploy and update highly distributed application environments. To achieve that goal, many organizations will apply the same DevOps principles enterprise companies have been applying to their embedded systems, he said.

    To ensure the integrity of the software supply chain on which the Wind River CD platform is delivered, Wind River has also been certified under the ISO 9001:2015 specification for management processes, as well as the OpenChain specification for open source compliance.

    Wind River also has the Wind River Workbench for Linux LTS, based on the Eclipse integrated development environment (IDE), along with various system analysis tools including a trace visualization tool from Percepio AB to identify the root cause of anomalies.

    While it’s taken more than 10 years to drive the adoption of best DevOps practices in the enterprise, adoption in embedded systems environments should be faster now that the journey has begun. Organizations of all sizes are pushing more application workloads to the edge at faster rates. Traditional approaches to building and deploying applications on standalone embedded systems will not be able to keep pace. Organizations that build applications for embedded systems will be challenged by first acquiring the right tools and then adapting their internal culture to the complexities that arise when they need to build and deploy applications faster.

    Wind River is making the case for a curated open source CD platform that should enable organizations to spend less time on managing the underlying DevOps platform in favor of focusing more time and energy on changing their internal culture.

    — Mike Vizard