Tag: code

  • AT&T Democratizes Acumos Code for AI

    AT&T Democratizes Acumos Code for AI

    AT&T and India IT outsourcing company Tech Mahindra announced that code the companies jointly developed to drive artificial intelligence (AI) applications is being contributed to the Linux Foundation.

    Mazin Gilbert, vice president of advanced technology for AT&T Labs, says the goal of Acumos is to not only democratize access to AI software, but also consume it as a microservice with a standard set of DevOps processes.

    Gilbert says AT&T decided to contribute Acumos to the Linux Foundation because it’s become clear the proprietary implementations of AI platforms will prove to be too difficult and costly to support. AT&T is addressing this issue because it currently has 10 proprietary AI platforms attached to its network and was looking at having hundreds attached within a matter of months.

    Making Acumos available as an open-source project should go a long way to reducing that complexity by providing a standard interface to integrating various AI platforms, says Gilbert. Acumos includes a standard mechanism for downloading AI code from an app store, bringing AI functionality to where data resides instead of requiring developers to load data into a proprietary cloud service to access AI functions. In the case of AT&T, that integration is accomplished via the company’s Indigo software-defined network (SDN).

    Gilbert notes that much of what passes for advanced AI today has been around for decades. Many of the algorithms being used to drive AI applications are now over 50 years old. The only real differentiation is how these algorithms are being combined to drive applications to access enough data to make the algorithms useful. Because of that simple fact, AT&T doesn’t believe giving Acumos to the Linux Foundation represents any major loss of intellectual property.

    As AI becomes more democratized, DevOps teams will manage massive amounts of distributed data pipelines. AI applications require access to massive amounts of data that must be consumed via a consistently applied information architecture. Sometimes referred to as DataOps, the pipelines created by operations teams enable the algorithms embedded in a developer’s application to continually learn about changes that impact the recommendations being surfaced by the AI engine.

    At this point it’s now only a matter of time before every application is infused with some form of AI. In many cases, the front end to those AI capabilities will be a voice-enabled digital assistant trained to optimize a specific processor or answer a broad range of capabilities. Applications that don’t provide these capabilities soon will be deemed archaic. The result from a DevOps perspective should be a massive series of rolling upgrades to infuse AI capabilities into existing applications or replace them with one that provide those capabilities. Regardless of how that is accomplished, without some form of integrated DevOps processes in place there can be no data for AI applications to consume.

    — Mike Vizard

  • Building a Services and IoT Platform with DevOps, Part 3

    Building a Services and IoT Platform with DevOps, Part 3

    Business needs and IoT demands are taking new dimensions in connecting millions of devices, people, process, tools and systems. With endless possibilities of creating new IoT solutions, DevOps teams are the key players who handle the challenges in successfully rolling out a IoT solution while handling the business, development, operations and infrastructure demands in the background. This article focuses on some of the DevOps best practices to adopt during IoT development, enabling faster time to market.

    DevOps practices to adopt during IoT development

    1. Adopt Security by design – Teams need to be educated and made aware of security threats. Use threat models mitigation techniques such as using authentication; storing secrets separately and protecting those secrets; using tamper-resistant protocols, digital signatures, timestamps, audit trails, encryption, authorization policies, filtering; and running services with least privileges during security design and development.
    2. Shift left security analysis and security testing – Integrate security analysis tools, customize security compliance rules and run security testing as part of continuous integration pipeline, generate security threat reports on vulnerabilities and act on these threat reports to fix issues early in the development phase and minimize the security risks.
    3. Follow security standards during development such as security development life cycle (SDL), which has guidelines and measures for building the software for devices and application components of IoT platform.
    4. Perform code reviews for device software, application software, PaaS rules and configurations, as quality assurance can become a huge risk given the complexity of the infrastructure, millions of devices connecting and dynamic environment configurations. Code review tools need to be integrated to the CI pipeline with pull request code review.
    5. Use the right branching policies in a version control repository, including precommit/preflight builds, pull requests for merging the code changes and access control for maintaining branches and using the right branching strategies, as there will be multiple fragmented development and deployment pipelines for devices and application developments to reduce the overhead. Use distributed repository/central repository with traceability and auditability based on the requirements.
    6. Automate deployments and create deployment pipelines for device development, application development, infra-automation scripts development with build and deployment pipelines targeting for different target endpoints.
    7. Customize the builds and release pipelines to pull the packages from multiple build and deployment pipelines to integrate and deploy as units of deployment.
    8. Plan for frequent release cadences—say, monthly—for application rollouts. Release cadences for firmware update rollouts can be planned for three to six months, as device updates cannot be pushed so often.
    9. Continuously monitor deployments – Use a release management tool for orchestrating the releases and release coordination activities, with gated checks and approvals with a view of what is getting deployed to each environment, as well as traceability and audit.
    10. Enable feedback loops and automated alerts if firmware deployments or application updates fail.
    11. DevOps teams can define key tenets, criteria to be met and standards to be followed for the devices; infrastructure to be provisioned; device protocols usage and standards in connecting to existing/new devices and services. This can reduce the complexity in development and deploying to multiple varied devices by following the same protocols for connectivity and communication.
    12. Train the team of developers, QA and hardware developers on how to do continuous integration for IoT development, follow agile and lean practices and ensure security criteria for all layers of the application and in process.
    13. Adopt user-centric test automation and end-to-end testing approaches and build robust test cases in perspective of testing a complete flow, with devices/services that may not be in your scope but will be part of your overall IoT platform ecosystem.
    14. Create/utilize container based test environments and device simulators coupled with service virtualization for functional testing.
    15. Automated infrastructure provisioning – Use the right configuration management tools to automate the infrastructure along with dependencies to create environments through code instead of provisioning basic VMs.
    16. Use container and cluster management tools such as Docker Swarm or Kubernetes for service discovery, auto-scaling, self-healing and scheduling.
    17. Use RBAC, service accounts and security access controls in the infrastructure templates/scripts that are created instead of individual user credentials to limit the risk of unauthorized users provisioning the environments using scripts. Build security in devices, cloud security, application security and infrastructure automation templates. Use secrets, key vaults, encryption methods and certificates to secure the credentials and access control.
    18. Configure individual build and deployment pipeline jobs for application development and infra-automation development, as there will be a huge development effort involved in automating the infrastructure.
    19. Create infrastructure verification tests with scripting tools such as Python or use testing frameworks such as Pester or any other open-source tools and integrate the infrastructure tests with the CI/CD pipeline. Ensure to run these tests every time the infrastructure is provisioned.
    20. Version-control the infrastructure templates, scripts, user profiles, environment configurations and device configurations apart from source code.
    21. Set up a centralized dashboard of monitoring tools to monitor the clusters, nodes, firewall rules, VMs, applications, devices and services. Feedback should flow back to developers/QA/DevOps teams.
    22. Configure applications to provide logging about exceptions, memory leakages and response times and push the logs to the monitoring tools or the dashboards in monitoring tools.
    23. Use the telemetry from the applications and devices to derive the analytics graphs/trends and set up KPIs/metrics to measure application performance/availability, as well from derive business metrics to set SLAs for DevOps and operations, to improve productivity.
    24. Ensure DevOps teams have close coordination with dev/QA teams of both software and hardware teams, as managing IoT platforms need more skill and expertise in hardware and software in the IoT domain.
    25. Manage legacy and cloud infrastructure with multicloud management platforms, and monitor and optimize the infrastructure usage.
    26. Create a DevOps governance plan/guide, with clearly defined access control policies, access rights for devices, control limits on infrastructure provisioning in different clouds, limits on PaaS/IaaS services to be used, etc., and standardize the process to be followed in DevOps/development/testing/operations, tools, etc.

    Following DevOps practices for application development has become quite streamlined in recent years. However, in IoT, where traditional methodologies and practices are followed in industrial enterprises, adapting to agile methodologies and following DevOps practices coupled with right DevOps-skilled team, tools and expertise can streamline and simplify IoT developments.

    About the Author / Lavanya Subbarayalu

    Lavanya Subbarayalu is Senior Architect working with Technology Office in HCL Technologies. She has expertise in IOT, Azure, DevOps consulting & Microsoft technologies. She is associated with DevOps COE, working on design and Development of DevOps solutions and consulting tools. Connect with her on LinkedIn.

  • Tips for Creating Unit Tests for Mobile Apps

    Tips for Creating Unit Tests for Mobile Apps

    Mobile app testing ensures your application is working perfectly without any freezes, bugs or crashes. There are unit, auto, system and acceptance tests, and in this article we will talk about unit testing for mobile apps.

    First, let’s discuss what unit tests are in general. Unit testing checks whether certain parts of the code work correctly. In other words, it helps us produce more robust and maintainable code. Each one small test is dedicated to a small feature. All of them are connected, however, so you get a kind of chain. If any part of this chain fail, the whole chain will fail as well.

    As a part of mobile application testing unit tests may be written at any phase of development or even before development (test-driven development, or TDD, is one of the iOS unit testing best practices). With this kind of development, you let your tests drive the development process. They indicate what you do, what you should do next and what the result should be. Testing controls the design.

    Thus, unit testing is the first step to perfectly working code. But it also is the primary means for your software quality assurance. In this article we’ll explain how to do mobile apps unit testing on an iOS example and give you a small iOS unit testing tutorial.

    Do I Need to Conduct Unit Testing?

    Does everyone need to do unit testing? The short answer is no. The long answer is that it  is a very good tool, though it isn’t necessary in all situations.

    You don’t need unit testing if:

    1. your app is very small and simple. If your app is small, it will be faster to check everything manually because there’s not that much logic to be checked.
    2. you don’t plan your app to work in the long term and you need it only for a demonstration.
    3. you’re a super-human who writes perfect code without any mistakes.

    If you don’t fall into any of these categories, then unit testing is what you need. At first, this kind of testing may seem too time-consuming. Lack of time is mentioned as one of the most serious problems related to testing, but actually performing unit testing can save a lot of time and improve development speed. In addition, it can serve as a kind of documentation for your library. We at Mobindustry spend up to 30 percent of our development time on testing.

    Unit tests have two main purposes:

    1. Reduce the number of bugs by making sure that our class/module/component is working as expected with all possible inputs.
    2. Reduce the number of regressions by making sure that our new functionality, bug fixes or refactoring hasn’t broken any existing functionality.

    How Do Unit Tests Work?

    How to create iOS unit testing? Xcode has built-in support for unit tests and works as an iOS unit test framework. You can use Xcode by pressing cmd+U or via the menu by selecting Product -> Test. xCode will run all active tests and generate a report with results (passed/failed). All failed assertions are indicated. Code coverage data is also generated in this framework, which allows you to get to any class and check which lines of code were called and how many times they were called.

    Types of Unit Tests

    Unit test for iOS can either check the UI or the logic of the application. Logic ones have a few subtypes:

    • Regular ones are used to check straightforward functionality.

    func testExample() {

    let account = Account()

    account.amount = 100.0

     

    account.withdraw(20.0)

     

    XCTAssert(account.amount == 80.0, "Account should have current amount of 80.0 but has

    (account.amount)")

    }

    This is used to simply check if the withdraw() method is working correctly.

    • Asynchronous tests use XCTestExpectation to check for expected outcomes.

    func testExample() {

    let url = URL(string: "http://www.google.com")

    let expectation = XCTestExpectation(description: "expectation_description")

     

    let task = URLSession.shared.dataTask(with: url) { (data, response, error) in

    if let error = error {

    XCTFail("Error: (error.localizedDescription)")

    return

    } else if let statusCode = (response as? HTTPURLResponse)?.statusCode {

    if statusCode == 200 {

    expectation.fulfill()

    } else {

     

    XCTFail("Request failed with status code (statusCode)")

    expectation.fulfill()

    }

    }

    }

    task.resume()

     

    waitForExpectations(timeout: 10) { (error) in

    XCTFail("Expectation (expectation.description) has failed")

    }

    }

     

    This one will check if Google’s website is available (i.e. if it returns status code 200).

    XCTestExpactation is used to wait for the async task to finish. An Expectation will either finish successfully by calling the fulfill() method or will fail if the timeout time has passed but fulfill() hasn’t been called.

    • Measure ones check heavy logic:

     

    func testExample() {

    self.measure {

    let lotOfObjects = []

    for obj in lotOfObjects {

    obj.makeCalculations()

    }

    }

    }

     

    You can create this test to measure how much time and how many resources are spent on particular calculations inside the measure() block.

    Tips on Unit Testing for Mobile Apps

    • Remember the structure

    The common AAA approach to structuring is the most common and convenient way to write tests. AAA stands for:

    1. Arrange – the part that allocates all necessary objects and sets up initial parameters
    2. Act – the part that executes the logic
    3. Assert – the part that checks if the obtained results match the expected results

    Сode written this way is easy to read and use, which means it’s easier to maintain.

    • Cover only necessary things with tests

    Some people think it’s necessary to cover every single part of your code with a unit test. This approach is actually time-consuming, complex and completely useless.

    There are four types of code you may have in your project. Let’s look at them starting with the most extreme options:

    1. Simple code without any dependencies: Probably everything is already clear with this kind of code and there’s no point in assessing it.
    2. Complex code with lots of dependencies: If you have this kind of code, it’s a good idea to refactor. There’s no need to cover this code because we’ll rewrite it, which means new classes will appear and method signatures will change. Why write something that won’t be used? Of course, such code also needs quality assessment, only on a higher level.
    3. Complex code without dependencies: This code contains algorithms or business logic. This is the perfect kind of code for testing because these are very important parts of the system.
    4. Moderately complex code with dependencies: Such code links different components together. Tests are very important here; they tell how exactly components must interact.

    Usually, they cover the following areas of code:

    • Common functionality, core models, classes and their interaction
    • Main UI workflows
    • Crucial logic
    • Bug fixes

    Unit tests may be applied not only for these areas. They can actually be used for testing almost any functionality.

    Don’t cover trivial functions and system functions.

    • Test only one thing at a time

    The idea is to divide your application into different independent modules that are able to work by themselves without interacting with other parts of the application. This allows you to work on each module separately and simplify the code. If you check several things at the same time, sooner or later your code will become overloaded and too complex. It will become difficult to support it and use.

    • Name your tests logically

    It’s very important to name your tests properly and assign them to the correct parts of your project. Many projects have tests that no one is running because of inconvenient placement and naming. Here are some tips for logical structure:

    1. Choose a logical location in your version control system: They have to be part of your version control. Depending on your needs they can be organized differently, but here’s a common recommendation—if your app is monolithic, put all of them in a Tests folder; if it has many components, store relevant tests in each component’s folder.
    1. Choose a convention for naming: One of the best ways to do this is to add a test project to each project you build.
    1. Use the same convention for naming test classes that you use for naming project classes: If you have a class ProblemResolver, it’s a good idea to add a class called ProblemResolverTests. Each class should work only on one thing; otherwise, your tests will be very hard to manage.
    • Work with mocking objects

    You can use different fake objects to make the testing process easier. There’s no need to allocate and manage a huge class or component just to use a single method from it. Such fake objects can be divided into two groups:

    • Stubs
    • Mocks

    Stubs are fake methods for returning a predetermined response. The main use for stubs is when interacting with an API. The idea is not to wait for the actual connection to finish but to return predefined data or a predefined error instantly.

    Let’s assume we have an app where you can add a friend connection. There’s a FriendListViewController that displays a list of all your friends. This friend list can be downloaded from the backend, so there’s an ApiManager class with a method like this:

    ApiManager.shared.getListOfFriends(withCompletion: { (friends) in

    let friendsListVC = FriendListViewController.init(friends: friends)

     

    //Present friendsListVC

    }) { (error) in

    //Display error

     

    }

    This method has a completion handler, and we need to wait till it finishes.

    But we actually don’t need to wait because we don’t need any APIs or even an internet connection. We need to make sure FriendListViewController is working correctly, and all we need for that is some list of friends; it doesn’t matter where they come from. So we’ll introduce an ApiManagerStubs class that will duplicate methods from the actual ApiManager but will not perform any actual API requests.

    For this particular test, we’ll write a method for getting a list of friends:

    func getListOfFriends() -> [Friend] {

    let friend1 = Friend.init(name: "Bob", surname: "Barker", age:34);

    let friend2 = Friend.init(name: "John", surname: "Doe", age:30);

     

    return [friend1, friend2];

    }

    After that we can easily allocate and test FriendListViewController.

    func testExample() {

    let friends = ApiManagerStubs.shared..getListOfFriends()

     

    let friendsListVC = FriendListViewController.init(friends: friends)

     

    //Testing friendsListVC.tableView

    }

    Mocks are fake objects that partially duplicate real objects, allowing you to check if a method was called or a property was set after certain events.

    In the example above, when a friend is deleted we need to clear the chat history for this friend. We have some class like ChatManager for this functionality. But again, for this, we don’t need to actually delete any data or call any APIs; we just need to check if the corresponding callbacks are triggered. So we’ll introduce the mock object ChatManagerMock, which behaves like ChatManager, only instead of deleting data it just remembers which callbacks were triggered.

    class ChatsManagerMock {

    var deleteFriendWasCalled = false

     

    func clearChatHistoryForFriend(_ friend: Friend) {

    deleteFriendWasCalled = true

    }

    }

    Now we can implement our test as follows:

    func testExample() {

    let friends = ApiManagerStubs.shared.getListOfFriends()

     

    let friendsListVC = FriendListViewController.init(friends: friends)

     

    let mock = ChatsManagerMock()

    friendsListVC.chatsManager = mock

     

    //Trigger Delete action for first friend in the list

    friendsListVC.deleteFriendButtonAction(nil, index: 0)

     

    XCTAssert(mock.deleteFriendWasCalled, "deleteFriendWasCalled should have been called")

    }

    Conclusion

    Unit tests are a very powerful tool to help developers avoid bugs and spot them quickly when they do appear. They help you maintain your code, but at the same time they need maintenance as well. The big problem of many projects is tests that no one uses, and to avoid it you need to stick to some rules. Your tests must:

    • be reliable;
    • not depend on the environment they’re run in;
    • be easy to maintain;
    • be easy to read and understand (even a beginner QA has to be able to understand what’s going on);
    • have a standard naming convention; and
    • be launched regularly and be automated.

    Following these basic rules will help you get a working system. Each test will check only one thing. Your tests will be specifications for class method; it will tell all about what input they expect and what other system components expect as the output. There are very few systems like this. Such systems have an actual specification, and there’s usually not very much text in it—only basic features, server schemes and a “getting started” guideline. The life of such projects no longer depends on people; developers may come and go while the system is reliably tested and tells all about itself with the help of its own tests.

    About the Author / Svetlana Cherednichenko

    Svetlana Cherednichenko is a content writer. Currently she is a part of Mobindustry marketing team. She is an active member of the debate movement and she is a winner of two regional tournaments. Now she is in the process of improving her skills in graphic design and learning new frameworks. She graduated from the Dnipropetrovsk National University. Svetlana likes to travel, read about tech news and play video games. Connect with her on LinkedIn.

  • Standardizing a Process that Drives Secure-By-Design

    Standardizing a Process that Drives Secure-By-Design

    In “Building Security into Code + Culture,” I described how our culture of coding is predicated on chaos, and the critical challenge for organizations is to establish the right mix of processes and tools that support a smarter, risk-based approach to development. A key aspect of this strategy is, of course, to implement a sense of order within the chaos that enhances security without hindering the speed of innovation.

    I overviewed a number of different recommendations to approach secure by design, such as making code more mature and baking in security measures and privacy/contractual compliance from the start. I also emphasized that organizations should tackle the rules, tools and fears pillars that help shape the culture, as each of these can be transformative in helping people use the right approach or method. Making the rules and defining fears is relatively straightforward, and with the right degree or training and awareness your organization can adapt.

    But what about tools? The right tools can do wonders to “force” people to use better, smarter, more secure methods—that is, if such tools exist. Though I have not done a deep dive into whether there are any robust development tools integrated with security software, I’m a realist and will go out on a limb to say none of them have reached maturity just yet. In light of this, it’s essential that organizations focus on standardizing the overarching process that will enforce privacy and security by design, development and deployment.

    The first thing to do is set the business drivers behind the process from a legal standpoint. Gather and research the privacy regulations and compliance requirements that affect your organization. Analyze and then translate these using a risk-based approach to security controls requirements, and then publish these as an awareness catalogue available to all developers across your organization. You should also ensure that training is available, on-demand and on a just-in-time basis, for any developer on any desired control.

    Next, set the technology drivers by prescribing to secure coding practices defined by the industry, such as the Web Application Security Consortium (WASC) – Threat Classification system (based on Open Web Application Security Project [OWASP] Top 10) and SysAdmin, Audit, Network and Security (SANS) Top 25 (which are based on the Mitre – Common Weakness Enumeration [CWE]).

    At this point, you must ensure that all your applications and systems comply with the security controls requirements catalogue you created. These can be batched into smaller parts that are handled by developers or teams of developers, all of which use your security controls requirements catalogue and industry-standard secure coding practices. Again, you’ll want to make sure that training sessions are available on a just-in-time basis for any developers who need them!

    Upon completion of each section or unit, code must be validated and verified for trust. Peer code review is the first step—have your developers present their work and provide the logic to a different team of developers. You can always employ static code analysis to scan code for any weaknesses, defects or violations of best practices, and subsequently enter the bug fix stage if any are detected.

    Then, you can merge the code branches into the main and compile for run time, so you can conduct live vulnerability scanning and penetration testing. Again, these should be based on the industry-standard secure coding practices adopted across your organization. At this point, you should be able to establish reasonable assurance for the release version of your application or system by having published artifacts for review by entities such as auditors or clients.

    Continuous improvement will require adding new features and subsequent emphasis on the validation and verification process. I would also recommend repeating the entire process from start to finish on a regular basis to reinforce the process, perhaps annually or semi-annually. And of course, if there are any major changes or shifts (as there are likely to be) in your privacy regulations, compliance requirements or secure coding practices, this will absolutely need to be addressed at the organizational level so your process addresses these head-on.

    Privacy and security are such important topics today because they are an artifact of the youth of the computer software industry. If this industry was more mature, then the outcome would be quality code. For some reason, it’s okay to create buggy, weak and inadequate software in a world in which we wouldn’t tolerate buggy, weak or inadequate cars, airplanes, bridges, buildings … and the list goes on. And the reason for this is pretty obvious: People are acutely aware that cars, airplanes, bridges and buildings put their very lives on the line, whereas software “just” makes them susceptible to things such as identity theft and ransomware.

    This is the message we’re sending: It’s not OK to kill people, but it’s perfectly acceptable to ruin their lives. This is not OK! It’s time for organizations to standardize the processes that drive secure-by-design, and embrace necessary and fundamental changes to how we approach development. As developers and IT professionals, the onus is on us to shape a better world driven by quality software.

    About the Author / Robert Hawk

    Robert Hawk is Resident Security Expert at xMatters. He has extensive experience in Information Systems Security, Computer Security, Cyber Security, Information Assurance, as well as Governance, Risk, and Compliance (GRC) Management. He specializes in frameworks and standards from: ISO/IEC, NIST, IEEE, IETF, ITU-T, Common Criteria, AMI-SEC, NERC, CIS, DoD, ANSI, PCI, and ISECOM. Robert is a lifelong researcher, innovator and instructor. Connect with him on LinkedIn.

  • Google Launches Software Supply Chain Initiative

    Google Launches Software Supply Chain Initiative

    Maintaining the integrity of a software supply chain is an issue that has long bedeviled IT organizations. With the rise of microservices based on containers, however, this issue has become more acute. As a result, software supply chains are an issue that Google now plans to tackle.

    Google, along with JFrog, Red Hat, IBM, Black Duck, Twistlock, Aqua Security and CoreOS, is taking the lead on an open-source Grafeas initiative that aims to create a standard application programming interface (API) through which metadata about the software components that make up an IT environment can be more easily captured. The whole software supply chain issue has become even more challenging now that enterprise IT organizations regularly make use of open-source software within larger applications. There are now more dependencies on software modules that are being created and updated outside the IT organization, most of which are not prepared to keep track of the modules’ history.

    One of the first instances of Grafeas is going to manifest itself in the form of Kritis, a policy engine that Google has developed for the Kubernetes container orchestration engine. Because of its long history with Kubernetes, Stephen Elliott, product manager for developer platforms at Google, says Google tends to approach every issue these days through a container prism. But the expectation is that the Grafeas API will be more broadly applied to collect data about and enforce policies across a broad range of types of software components, he says.

    Elliott says that while there is no shortage of tools for managing software life cycle, each of those tools generate their own proprietary metadata. Grafeas promises to provide a consistent means to, for example, identity developers, determine when the code was checked in and built, what vulnerabilities have been detected, and what tests were passed or failed. Trying to manage those issues is at the heart of any approach to DevOps. A standard API would not only go a long way toward simplifying those issues, it also should also serve to make many of the tools DevOps teams rely on today more interoperable. That would also go a long way toward preventing IT organizations from finding themselves locked into a specific set of DevOps tools.

    Obviously, the launching of an initiative represents only the beginning of a process that should extend well into 2018. The number of IT vendors that have a vested interest in software supply chain issues exceeds that number that has signed on to the Grafeas initiative, and when this API issue is addressed by an industry standards body remains to be seen. Google is betting that as IT organizations become more familiar with microservices-based containers, there will be lot more support for addressing these issues now than anytime in the past. In the meantime, IT leaders should take some comfort in the fact that, at the very least, the IT vendor community is starting to better appreciate their software supply chain pain.

    — Mike Vizard

  • GitHub Applies Data Science to Managing Code

    GitHub Applies Data Science to Managing Code

    Over the last several years GitHub has emerged as one of the primary repositories around which application development now revolves. At its recent GitHub Universe 2017 conference, the company revealed how it is extending that central role to provide DevOps teams with the addition of a dependency graph that can be employed to alert DevOps teams when a module of code has been updated and a security alert service that will notify them anytime a patch has been made available for a specific module.

    Miju Han, engineering manager for data science for GitHub, says both new capabilities are examples of how the company is applying advanced algorithms and data science techniques to make it easier to manage DevOps processes that revolve around GitHub.

    In addition, Han says GitHub is making available a news feed through which DevOps teams can track updates to groups of modules stored in the repository, as well as a tool through which they can explore a curated set of modules.

    Han says his company intends to combine a variety of emerging data science tools to simplify the daily workflow that revolves around GitHub. As the repository has become more widely used, the sheer volume of code and related updates a DevOps team is supposed to track has become, in many cases, overwhelming. That’s why the company is now pouring resources in several data science research projects with an eye toward making it simpler to navigate the repository.

    The latest annual edition of an Octoverse report published by GitHub finds that there are now more than 24 million developers in 1.5 million organizations around the world accessing 67 million GitHub repositories. A total of 25 million of those repositories are public. The most popular programming languages in use across GitHub are JavaScript (2.3 million projects); Python (1 million projects); Java (986,000 projects) and Ruby (870,000 projects). In terms of pull requests, GitHub notes that Python surpassed Java in popularity in the last year. The company also notes that half of the largest companies in the U.S. based on revenue are employing GitHub repositories.

    There is no shortage of repositories being employed inside and out of enterprise IT organizations, including an enterprise edition of GitHub that the company claims has been implemented by 45 percent of the Fortune 100 list of the world’s largest companies.

    Arguably, the biggest challenging many organizations now face is trying to reconcile the various code repositories in use. Just about every provider of an application development platform provides access to a code repository. In addition, providers of everything from container frameworks to application lifecycle management (ALM) often provide similar capabilities. Keeping track of what version of any given piece of code resides where at any given time has become a significant challenge.

    The good news is that providers of repositories are starting to address that issue on their own platforms, which means it stands to reason that one day soon tools for managing code across heterogeneous repositories may not be all the far off. In the meantime, DevOps teams can take some comfort in the fact that the management tools they rely to manage all the code are about to become a whole lot smarter.

    — Mike Vizard

  • Building Security into Code and Culture

    Building Security into Code and Culture

    Organizations tend to have a very predictable approach to security: reactive. Most don’t really care about security until something bad happens. But with data regulations and requirements becoming more rigid around the globe and cybercrime only continuing to escalate, the notion of security as an “add on” or an afterthought is quickly becoming outdated. Today, an organization without adequate security measures is like a skydiver without a parachute—sure, the parachute doesn’t guarantee a 100 percent safe experience, but wouldn’t you rather have one than go without?

    A big problem, however, is that the modern approach to security remains inadequate. Most companies simply layer piecemeal measures over their existing infrastructure, not realizing that this isn’t an effective way to mitigate risk. Instead of adding on bits and pieces of security such as firewalls and antivirus programs, organizations should strive to incorporate security principles into the very foundation of their IT, in both the code and the culture.

    This is what Microsoft sought to do with its SD3+C principles, e.g. Secure by Design, Secure by Default, Secure in Deployment and Communications. While all of these are designed to impose better security measures throughout the development process, the first two arguably provide the highest security benefits. This is because they prevent the introduction of vulnerabilities in the first place while minimizing the potential exposure of software—so-called “attack surfaces.”

    Security and safety by design has thus far achieved greatest maturity in the automobile industry. Modern-day cars have more than 100 different safety and security features, including redundancies, and are rigorously tested for verified assurance. While crash tests destroy vehicles and take time to conduct, they ultimately teach us invaluable lessons that make them 100 percent worth the investment.

    Baking security measures into design and development is likewise a better approach when it comes to code, but it presents a problem: Developers, by nature, don’t want to take the time to build safety nets. A major criticism of SD3+C is that it slows down innovation by forcing developers to make more considerations than “make something really cool at lightning speed.”

    This reaction isn’t surprising when you consider the culture of coding, including the process by which we teach new developers. Hacker culture has long carried a certain kind of allure, romanticizing the role of the developer as a creative, clever and adventurous rogue. It elevates those resourceful individuals that can overcome software challenges in the fastest, nimblest, slickest way—not necessarily the securest.

    It’s never been the standard to teach the morality, ethics or laws of coding; instead, we teach people how to code and let them take off running. But the modern digital age has required us to put the brakes on and rein in that spirit. It’s like teaching someone to drive a Ferrari at breakneck speeds but not about the rules of the road, and then later expecting them to steer within the lanes, stop at red lights and not exceed speed limits.

    Bring Order to Code Chaos

    Our culture of coding inherently perpetuates chaos, but now we need to implement some degree of order, and this is a tough challenge. We need to drive a risk-based approach to development that introduces order into chaos without hindering innovation.

    How do we begin to accomplish this? The first thing to consider is changing how you approach code in the first place by making it more robust. Typically, code can’t be easily changed once it’s written; a single line may be easy to manipulate, but at several million lines, you risk severing the ecological connections that make the app work, and changing any one piece could break the entire application. With your code in such a susceptible state, it’s far too resource-intensive to maintain.

    The solution is to make it more mature in the first place. Make your code modular enough so that it can be amended in part when necessary, and then you can easily bake in additional security measures and eliminate dependence on firewalls and antivirus as your first (or only) line of defense.

    Second, consider the privacy law and contractual requirements of your organization, and address them directly in your code from the start. Understand that different types of privacy data will need to be handled differently and require different levels of protection across technical, physical and administrative dimensions. Whether you deal with payment card industry (PCI) or protected health information (PHI), you should be intimately familiar with the standards governing the protection of data your organization deals with.

    Now, what about culture? In its simplest essence, we can think of culture as comprised of three basic factors: rules, tools and fears. Where rules exist, you should apply them; where tools exist, you should use them; and as an organization, you should be “afraid” of the same things, because common fears tend to be strong motivators for groups of people or the masses. The only re-transmission of culture is through awareness, training and immersion.

    An important point I invite you to consider is that human beings won’t do anything without a reason, whether real or imagined. Humans are experienced-based learners not knowledge-based learners. For this reason, we learn valuable lessons from mistakes and failures. An important aspect of this is timely communications. When something has gone wrong we need to be able to get the right people on task to work toward planning mitigations, learning lessons and making things better.

    One way to effectively change or reshape your culture is to enforce the rules—for instance, the new General Data Protection Regulation (GDPR) mandates severe fines and consequences for PII data breaches (which is, in fact, similar to penalties enacted by the auto industry years ago).

    Another is to provide better tools and processes that essentially “force” (or, more kindly, compel) your team to use the right approach or method.

    Finally, you can employ fear, uncertainty and doubt, otherwise known as “FUD.” Though it may sound a bit sinister, FUD can be a very effective way to govern—although on its own, it’s a very tenuous and fragile kind of power. This is why you balance it with rules and tools, so that the overarching objective will resonate in both the hearts and minds of your team: Keep your organization happy and healthy.

    Because this is the reality that modern digital organizations must face: It doesn’t matter how innovative or bleeding-edge you are unless your business stays healthy! A tree that is well-rooted and cultivated to grow properly and upright will weather the worst storms, and ultimately live a long and prosperous life. Remember that Ford once had to introduce critical safety measures to the Model T, and nobody was worried about hydraulic brakes, seat belts and padded dashboards slowing us down.

    About the Author / Robert Hawk

    Robert Hawk is Resident Security Expert at xMatters. He has extensive experience in Information Systems Security, Computer Security, Cyber Security, Information Assurance, as well as Governance, Risk, and Compliance (GRC) Management. He specializes in frameworks and standards from: ISO/IEC, NIST, IEEE, IETF, ITU-T, Common Criteria, AMI-SEC, NERC, CIS, DoD, ANSI, PCI, and ISECOM. Robert is a lifelong researcher, innovator and instructor. Connect with him on LinkedIn.

  • DevOps: Time for Developers to Dig In

    DevOps: Time for Developers to Dig In

     Speaking to developers and project managers at the Gartner Catalyst Show, Jenkins World and Atlassian Summit revealed some insights about how most IT specialist perceive DevOps. Yes, I had a busy end of summer schedule attending events and talking to a wide range of developers and project managers about DevOps.

    two dogs digging together in one hole in the ground

    What I learned is that most people have not yet figured out what is meant by DevOps. What most people told me is that they are doing DevOps explaining they have a CI/CD pipeline. When questioned further about how they perceived DevOps, they believed DevOps was about creating a CI workflow that developers and sometime-testing teams could execute to automate builds, deploys and testing. One of the most ironic of these conversations was project managers making the claim that their CI server performed builds and deploys.

    Well folks, DevOps is not about a CI/CD pipeline. And contrary to popular beliefs, your CI server does not perform builds and deploys any more than it performs automated testing. Your CI/CD pipeline orchestrates build and deploy processes in the same way that it executes an external tool for test automation. The ugly truth is that your CI/CD pipeline is executing one-off build and deploy scripts, the same scripts used in your waterfall practice, but just triggered by your CI/CD process. These scripts cannot easily adapt across the continuous delivery pipeline. Testing and production environments are configured differently. The build and deploy scripts must be tweaked, maintained and debugged in the same way they were during waterfall practices. Once ready, they are then called by your CI/CD workflows. DevOps is different.

    DevOps demands that your developers and operation teams share and collaborate from the earliest phases of a project. In essence, this requires that more of the operational duties be shifted left and directly onto the shoulders of developers. This is precisely were the responsibility and control should be. Operation teams need knowledge and transparency so they can communicate infrastructure requirements early and often to the developers. Eventually what we will see is a decline in the number of operational resources. A “release engineer” who managed production deployments said, “If we define our DevOps process correctly, my job will not exist in the future. Instead, automation will replace the manual Ops scripts, allowing Dev teams to do 70 percent of my work.” And that, my friend, is real DevOps.

    Developers, Dig In

    So it is time for developers to really dig in to DevOps. What I mean is developers must begin walking away from automating most of the processes by a one-off script and begin incorporating tools that perform DevOps tasks from continuous deployment, security scanning and performance monitoring. And different platforms need different tools. Project teams will need the power to select the tools that work for them. The days of “legacy” purchases of a single tool for the enterprise is long gone. High-dollar operational tools are high-risk and incur technical debt when developers choose to use a different solution. DevOps means giving development teams more control and purchasing authority for automation tools that can be managed at a project level, not an enterprise level. These tools must support a process where the developers work on the declarations followed by testing and production consuming the process with a good amount of transparency. Scripts cannot achieve this, even when triggered by CI.

    Our future is becoming more and more complex. Most companies I spoke to still managed over 70 percent of monolithic applications. The writing on the wall is in the 30 percent. With microservices and containers, operational tasks are going to become more complex, not less. The sooner developers can become the core owners of operational tasks, the easier it will be to adopt a microservices architecture. Managing microservices will not be the job for the operations team. It will need to be done by the developers who have the intimate knowledge of their application.

    To mature from CI/CD to DevOps is going to require a close look at the engine under your CI/CD hood. When developers and project managers begin to really dig in and see how the engine is running, it will be clear that achieving DevOps is a bit down the road. And there is urgency in this process. Leaner and better ways of writing and releasing code is upon us. Most of us are just not ready to toss out some of the old habits from waterfall practices to get there.

    Some departing thoughts:

    • Acknowledge that CI/CD is a good place to be, it is just not DevOps.
    • Look carefully at your underlying processes that are called by CI/CD. How well do they support your production environment?
    • Can your CI/CD workflow at Dev support a production deploy? If not take the time to understand why.
    • Are your developers working late to get a code update to production?
    • Do you have a clear continuous feedback loop that shows where a code update is running, and on what exact endpoints?
    • Do your developers know what infrastructure updates are going on in production today?
    • How do your developers communicate infrastructure updates to production teams? Can they do it alone?

    The time is now for self-examination and a clear honest appraisal of just how automated your core operational tasks are. If they are built upon one-off scripts, you should assume you have some work ahead to really begin moving to a DevOps practice that can be declared by developers, consumed by all runtime environments and is transparent to all.

    And remember, microservices are chasing you.

    About the Author / Tracy Ragan

    Tracy Ragan is COO and Co-Founder, DeployHub. She has extensive experience in the development and implementation of business applications. It was during her consulting experiences that Ms. Ragan recognized the lack of build and release management procedures for the distributed platform that had long been considered standard on the mainframe and UNIX.  In the four years leading to the creation of OpenMake Software she worked with development teams in implementing a team-centric standardized build to release process. Connect with her on LinkedIn.

  • NodeSource, AppDynamics Ally for Node.js DevOps

    NodeSource, AppDynamics Ally for Node.js DevOps

    The prevalence of Node.js as the most widely deployed instance of JavaScript is putting a lot of pressure on DevOps teams to find ways to automate the management of Node.js applications. To provide the visibility needed to achieve that goal, NodeSource has announced it has integrated its N|Solid development environment with the application monitoring tools developed by AppDynamics.

    Pravin Halady, product manager for NodeSource, says one of the things that differentiates N|Solid, which is based on Node.js, most is the amount of telemetry data NodeSource is able to collect from Node.js applications. The alliance with AppDynamics provides a mechanism for DevOps teams to now tap into that data to better manage those applications alongside the panoply of existing applications they need to support and manage, says Halady.

    AppDynamics already supports Node.js. But Halady says the alliance with NodeSource provides deeper visibility into, for example, asynchronous activity. Advanced metrics can be used to augment the basic Node.js metrics that AppDynamics currently provides to better ensure service level agreements (SLAs) are both achieved and maintained, Halady says.

    Adoption of Node,js exploded across the enterprise once it became feasible several years ago to deploy JavaScript code on both the client and the server. That capability in many ways fulfilled the early promise of Java in the eyes of many developers. But DevOps teams have been slow to include support for JavaScript applications within their DevOps processes, mainly because of a lack of visibility. Now that the base of Node.js applications in the enterprise has reached a level of critical mass, tools vendors such as AppDynamics are looking to provide that visibility by expanding the scope of support they provide for Node.js.

    Halady says NodeSource chose to partner with AppDynamics in part because in the wake of AppDynamics being acquired by Cisco Systems, it’s clear that NodeSource needs to participate in the end-to-end visibility platforms that larger companies are starting to build. Cisco has already made significant investments in a Cisco Tetration analytics application in addition to committing to imbuing the IT infrastructure environment it provides with machine-learning algorithms. That data will then be combined with data about applications collected by AppDynamics to identify the source of any potential performance issue faster. NodeSource, adds Halady, will look to foster similar agreement with other providers of application monitoring tools.

    In the meantime, DevOps team can take heart in the fact the Node.js Foundation is making providing visibility into DevOps environments a higher priority. The number of Node.js applications in the enterprise is only expected to increase. There’s a high correlation between usage of Node.js and containers that collectively are being used to facilitate the development of microservices. As those microservices bgin to replace monolithic applications developed originally developed in languages such as Java, the size and scope of the DevOps challenge exponentially increases.

    Application development in the enterprise is going to be polyglot well into the next decade. Today there are more programming languages being actively used in the enterprise than any time in history. In fact, the days when IT operations could mandate what technologies developers can employ are long over.

    — Mike Vizard

  • Puppet Makes Development Kit Available to Advance DevOps

    Puppet Makes Development Kit Available to Advance DevOps

    Organizations that embrace DevOps often find themselves moving from having a paucity of IT automation tools to an embarrassment of riches in short order. As a result, many of the teams involved in those DevOps processes wind up making different IT automation choices. Before too long, the organization finds itself trying to bridge too many islands of automation.

    Puppet is working to bridge the automation gap between developers and IT operations teams with the release of a Puppet Development Kit that makes it easier for anyone to generate code that can be executed using the Puppet IT automation framework.

    Michael Olson, senior product marketing manager for Puppet, says all too often developers automate various tasks using their own favorite tool. When it comes time for IT operations teams that have standardized on Puppet to deploy that code, they have to rewrite various routines. Puppet Development Kit makes it possible for developers to now write code that can be shared seamlessly with IT operations teams that have adopted the Puppet IT automation framework.

    In some environments, DevOps has literally become too much of a good thing. Olson says the Puppet Development Kit represents an opportunity to truly unify fragmented processes in a way that developers can easily consume. The Puppet Development Kit includes unit test modules from Windows, OS X and Linux workstations to ensure that Puppet code is creating and managing configuration resources as intended.

    The existence of the Puppet Development Kit should also make it easier for organizations that have not embraced DevOps to get started. A recent survey conducted by Puppet finds that organizations that have embraced DevOps are delivering new applications and updates at rates that far exceed other organizations. The longer it takes for organizations to embrace DevOps the wider that gap grows. By making it easier to include developers in more structured approach to DevOps, many companies that are getting started late with DevOps now have an opportunity to close that gap.

    Developers don’t always have a lot of respect for IT operations. Many of them have been throwing code over the proverbial data center wall because deploying code in a production environment was always someone else’s problem. Tools such as the Puppet Development Kit essentially steal a page from the same playbook Tom Sawyer used in the book by Mark Twain to get the kids in the neighborhood to help him paint a picket fence.

    At the other end of the spectrum, however, are developers who are firmly convinced they can program the IT operations team out of existence now that “infrastructure is code.” In reality, most of those developers quickly discover that the ongoing care and feeding of hundreds of applications deployed in a production environment still requires a fair amount of human intervention. In fact, without some form of IT automation framework in place, deploying applications at that level of scale would not be possible. The challenge is striking the right balance between what can be automated and higher-level tasks that still require human expertise.

    — Mike Vizard