Category: Blogs

  • The Role of APM in Continuous Integration and Continuous Release

    Today’s software-defined and -driven business requires fast changes in business models—a phenomenon permeating successful businesses. What this means is almost every company is learning how to make small, rapid changes and adjustments within their business, especially within software systems. As a result, IT is feeling immense pressure to evolve, and the net result is a major uptick in private cloud environments (especially Apprenda, Pivotal Cloud Foundry and Red Hat OpenShift) and public clouds (especially Amazon and Microsoft) within our install base of large enterprises. These new platforms enable faster development, testing and releasing of software.

    Enterprise customers are getting much smarter by using automation to drive the software life cycle, including continuous integration, and even experimenting with and using continuous release processes. This is no longer a startup scenario; it’s a strategy for our large, enterprise customers. These companies are trying to build agile development and operations teams by changing people, process and technology to enable a DevOps feedback loop:
    image02

    Wikipedia describes the continuous delivery loop as follows:

    DeliveryLoop

    Within this loop, application performance management (APM) can be part of build, unit, regression and load tests to drive the automated acceptance testing before new code is pushed to production. In fact, at the recent AppSphere 2015 user conference we had several great talks on this subject.

    The first talk was presented by our customer The Container Store‘s  August Azzarello, who discussed, “How The Container Store uses AppDynamics in the Development Lifecycle.” The company began with APM in production, in an effort to improve software quality before production. It began the expansion by installing APM within test, integrating it into functional and performance test suites. It also enabled alerting from dev/test so that developers and testers both understood when there were performance deviations and issues before production.

    Key features the company leveraged in pre-production included dashboards such as this one:image03

    Comparative analysis views such as this one:

    image04

    And being able to understand between releases if they were improving or degrading:

    image00

    The Container Store is using open source tools including Selenium for functional testing and Locust.io for performance testing.

    During his discussion, August also explained some of the major benefits the company saw from implementing APM in pre-production. Here are some best practices outlined by August in his discussion:

    • Monitor everything
    • Test continuously
    • Performance test early in development life cycle
    • Empower development & QA team members

    Some benefits The Container Store has seen, according to August, are:

    • Set performance expectations before production deployments – about 40 percent improvement since it started using AppDynamics in test
    • Fine-tune alert and business transaction policies prior to production deployments
    • Identify testing requirements and testing gaps
    • Decrease performance test result reporting to 20 minutes from five hours

    One of my favorite talks (which is why I selected it for the deep dive track at AppSphere) was the talk given by one of our senior sales engineers Steve Sturtevant, who came to AppDynamics from PayPal two years ago. In his talk, titled, “DevOps and Agile: AppDynamics in Continuous Integration Environments,” he explained the integration work he did. He also delved into the questions, Can you scale and auto implement monitoring in dynamic environments? Do you quantify the impact of change? Steve also offered a demo with integration between Puppet, Jenkins, ServiceNow and AppDynamics in his examples, with automation throughout the life cycle.

    One of our partners Electric Cloud specializes in continuous delivery and embraces an open ecosystem of technology providers and open source software. It has put together some great resources for people looking to explore the rest of the feedback loop:

  • Self Forgiveness

    Charlie Breaks the Build
    “How to apologize when you have broken the nightly build.” — http://bit.ly/1REOAOz

    — ROELBOB

  • What it’s Like Working as Quality Assurance in Consultancy Companies

    What it’s Like Working as Quality Assurance in Consultancy Companies

    Quality Assurance (QA) engineers working at consultancy companies should embrace trying new technologies and solving problems. This role is best defined by two words: variety and challenges. QA engineers at consultancy companies must pay great attention to detail and check all of their actions meticulously to make sure everything is being done correctly. The position also requires equal focus on the quality of the software application and client expectations, and constant communication between all individuals and teams involved in the process.

    Thinking about a job in QA? Here are some more details about this demanding yet rewarding position.

    Expectations

    Working as a QA engineer in a consultancy company demands very strong communication skills, flexibility and prioritization skills to help deliver a high-quality product in a timely manner. This role requires continuous improvement of both the process and working methods, which are not limited to testing or preventing defects but also consider the application’s lifecycle and ways to improve the product.

    Challenges can arise from working with myriad intricate software applications and environments. As the complexity of the software and the multitude of targets that can be addressed increase, so do the technical and communication challenges. A good QA engineer should always plan for this complexity in each project and be prepared to handle these challenges.

    Variety

    QA challenges and projects become even more complex and varied at consultancy companies. In a consultancy company engineers work with multiple companies from different cultural backgrounds and different levels of technical experience. Sometimes there are significant cultural differences across projects and within a team, which can have a big impact on the overall quality assurance process. To overcome any obstacles, QA engineers must communicate frequently with each company to answer any questions that may occur during the project lifecycle and to keep it on track.

    For each project, the QA engineer is expected to contribute to various parts of the process, which can include everything from client and internal team communications to delivering technical requirements. In addition, QA engineers must ensure technical requirements are communicated, tracked, planned, scheduled and executed correctly for each team. This is where attention to detail and prioritization skills are key, in making sure the correct communication is reaching the correct teams so the project runs smoothly and successfully.

    In addition to the innate variety that exists within the QA role, there is a fair amount of diversity, with multiple clients of different sizes spread across numerous industries. While some clients write out the project specifications and define clearly the expected outcome of the software application, others don’t, creating another opportunity for a QA engineer to contribute to a new project.

    Finally, QA engineers in a consultancy company constantly work with a variety of platforms and applications, from desktop to mobile projects and web-based to server-side. I see this as a positive—it keeps you on your toes and you constantly are gaining additional technical and architectural experience.

    As a Quality Assurance consultant, you will be expected to work on a wide variety of technologies and communicate effectively under a variety of circumstances. While the position can be challenging, it’s never boring. And knowing the project is successful because of your work is extremely rewarding.

  • Continuous Security Key to Defending Your Data Well

    Continuous Security Key to Defending Your Data Well

    Defending data isn’t easy. If it were, we wouldn’t see new data breach headlines on a weekly—or sometimes even daily—basis. That said, defending data also isn’t rocket science. A new report from Nuix highlights some of the major concerns with protecting data, and also illustrates some of the ways in which consistent vigilance is half the battle.

    The Nuix report, compiled by principal researcher Ari Kaplan, covers a broad spectrum of concerns related to data security. Some of the key findings from the report include:

    • Increased granularity of security budgets: Respondents report there is a growing trend to specify how the security budget should be broken down and allocated.
    • Regulatory impact on spending has doubled: Government and industry compliance mandates continue to take a sizable bite out of security budgets.
    • Greater focus on insider threats: More than 70 percent of respondents have some sort of insider threat policy or program.
    • BYOD continues to play a role in security: 82 percent of respondents have some sort of BYOD (bring your own device) policy in place. A growing number of companies, however, are prohibiting remote access by employees.
    • Cloud adoption slowing: According to the study, about 70 percent of companies are migrating data to the cloud—a number consistent with the previous year—but fewer companies are moving systems or servers to the cloud. More than four out of five respondents agreed that the cloud creates unique cybersecurity concerns, and only 43 percent indicated plans to move systems or servers to the cloud.

    What sorts of unique security concerns does the cloud bring? According to the survey respondents, there are a variety of potential issues, such as:

    • losing visibility into the management of data
    • being at the mercy of the cloud provider’s security skills
    • reduced control over access to data; and
    • confusion about the privacy/legal issues when a government or law enforcement agency wants to access data.

    In a nutshell, none of this is really new or different. Compliance, insider threats, BYOD and concerns over security and privacy in the cloud have been prevalent factors in network and computer security for years—and likely will continue to be for years to come. What has changed, though, is the focus on ways to implement better security and the tools to do it.

    One of the primary issues with traditional computer and network security is that it’s reactionary. Scans provide moment-in-time confirmation that there are no known threats, but do little or nothing to protect against unknown threats or prevent exploits and compromises between scans. DevOps, and continuous security, change the game by enabling organizations to monitor for and detect security issues on a consistent, proactive basis.

    “Security practitioners cannot continue to rely on manual, human controlled processes to defend against attacks,” agreed Andrew Storms, VP of Security Services for New Context. “Automation is a requirement for the next evolutionary phase of security protection.”

    Nick Galbreath, founder and CTO of Signal Sciences, also believes that automation and continuous DevOps principles are key components of effective security. “I have argued that continuous deployment (how rapid the ‘continuous’ is up to the organization and risk-dependent on the nature of the code) is actually the most important security tool since without it, I don’t really know how you make things better rapidly.”

    There are a lot of names tossed around for various aspects of DevOps automation. The actual name is not as important as the actual practice of proactively automating security processes for more effective security. Take a look at the report from Nuix to learn more about the general state of data security, but make sure you embrace DevOps and continuous security if you really want to do something about it.

  • A Guide to Public Cloud Security Tools

    A Guide to Public Cloud Security Tools

    Why is an arrangement of servers—which are constructed of hard metal, tend to run hot and weigh thousands of pounds—called a “cloud”? It only makes sense from the standpoint of an engineering diagram, in which data travels by an undefined pathway from origin to endpoint. In that respect, the cloud refers to the randomized packet transfer protocol that underlies modern computing.

    Although no one can know with certainty where the term “the cloud” originated, it’s clear what it represents. “The cloud is a metaphor for the Internet,” Cloud Camp co-founder Reuven Cohen told the MIT Technology Review in 2011.”It’s a rebranding of the Internet.” But for those with a security-first mindset, that metaphor is a problem. The Internet is a dangerous place that is filled with malicious people who enjoy misappropriating or damaging data. Storing private company and customer data anywhere on the Internet is a potentially hazardous course of action without the proper security controls firmly in place.

    Private vs. Public Cloud
    Private clouds are exclusive to a single enterprise—such as a enterprise company’s internal storage system. They are usually easier to secure but not as flexible in terms of rapid scalability. Private clouds also tend to have higher capital and operational costs. Public clouds, on the other hand, are operated by third parties that usually offer the cloud as service. Some of the most recognizable names are Amazon Web Services, Microsoft Azure and Rackspace in addition to the Google Cloud Platform and IBM’s cloud service.

    Private Cloud Security: Riskier Than You Think
    As Roger A. Grimes notes at InfoWorld, both private and public clouds come with risks. Private clouds may seem safer at first because they are under the enterprise’s tight control, but one of the biggest drawbacks is that many companies cannot afford the same level of security as Amazon or IBM. Even a company’s own employees are risk factors—they might do malicious acts or cause unintentional accidents. Private cloud networks rarely have geographic failover options, so all of its valuable data is more at risk. Public clouds, on the other hand, are supported by data centers all around the world—often making them more secure and efficient.

    Public Cloud Security: Still Facing Challenges
    Still, businesses typically consider these three specific security concerns when deciding whether to use public clouds:

    • Multitenancy: Sharing a cloud with other businesses—especially competitors—is the top concern of many corporate executives. Companies in public clouds must be confident that neither their private data nor their customer data will leak into other users in the space.
    • Virtualization: Exploits are grabbing headlines with increasing frequency. As identities across devices have become more fluid, access and authentication systems are having to work harder to approve the right users.
    • Ownership: Many public cloud providers are silent on the issue of ownership in their user agreements—leaving open the possibility that they could try, for example, to profit from the storage of the data.

    There are many ways to approach these challenges. The U.S. National Highway Traffic Safety Administration, for example, found a way to make the public cloud safe and accessible in less than one month by using some of the latest cloud security tools. The rapidly growing industry of cloud security firms and related software—such as those listed below—is evidence of the fact that cloud security is entering a new era. Businesses looking into a public cloud solutions may want to explore these options as additional ways to protect their data.

    The Top Public Cloud Security Resources

    • AppRiver – looks at messaging security for SaaS-based e-mail and web tools
    • Awareness Technologies – brings its SaaS-based DLP model to analyze mobile and the cloud
    • Barracuda Web Security Service – offers malware protection, URL filtering and application control
    • Bitglass – acts as a cloud-access security broker to secure applications and mobile devices
    • Bitium – handles identity and access management for BYOD and BYOA
    • BitSight Technologies – analyzes data on security behaviors and rates companies on security effectiveness
    • Centrify – centers on identity management across devices and applications
    • CipherCloud – handles encrypting or tokenizing data directly at a business gateway
    • Dome9 – examines firewall rules, IP address tables and ports to look for unusual web traffic
    • Evident.io – provides cloud security in partnership with AWS
    • ForgeRock – protects enterprise, cloud, social and mobile applications by identity access management
    • HyTrust – delivers access control, policy enforcement, hypervisor hardening and logging
    • IntraLinks – protects critical content and enables client control over data
    • Kismet – passively traces cloud machines while leaving no logs or traceable packets
    • Logz.io – users can create proactive alerts on selected events and relevant dashboards to aggregate and view data trends and monitor security threats including password brute force detection, access control and network access
    • Metasploit – takes a cloud IP address and tests penetration to assure that security is in place
    • MyPermissions – sends out alerts whenever apps or services try to access personal data
    • Nessus – operates as an open source vulnerability assessment tool
    • Nmap: the Network Mapper – tests penetration by scanning networks for congestion and latency
    • Netskope – discovers any cloud apps and shadow IT used on your network
    • Okta – manages logins across all cloud applications including Google Apps, Salesforce, Workday, Box, SAP, Oracle and Office 365
    • Proofpoint – focuses specifically on email to protect inbound and outbound data
    • Qualys – scans any and all used web apps for vulnerabilities in SaaS, IaaS and PaaS tools
    • SilverSky – offers email monitoring and network protection for HIPAA and PCI compliance
    • Skyhigh Networks – discovers, analyzes and secures cloud apps with logs from existing firewalls, proxies and gateways
    • SnoopWall – flags and stops access to high-risk data ports such as webcams, microphones, GPS and USB
    • WhiteHat Security – mimics threats to avoid coding vulnerabilities on websites
    • Vaultive – encrypts any data going from a network to an application
    • Zscaler – monitors all the traffic that comes in and out of your network along with protecting iOS and Android devices

    Public cloud security is only going to become more important with each passing year. Just look at these data points from Forbes reporter Louis Columbus:

    • Worldwide spending on infrastructure as a service (IaaS) is expected to reach $16.5 billion in 2015, an increase of almost 33 percent from the prior year
    • Cloud applications will account for 90 percent of global mobile data traffic by 2019, compared to 81 percent at the end of 2014

    So, with these trends in mind, how do you plan to secure your public cloud? I invite your thoughts in the comments below.

  • Chronos, Kairos, & Janus: It’s Time to Seek Balance

    Chronos, Kairos, & Janus: It’s Time to Seek Balance

    We conflate idealism with naïveté and pessimism with ‘just being realistic’. But to deny the existence of perfection is to deny the evidence of our own lives.

    Was 2015 a win or a loss?

    It can be tough calling the winners and losers. The mathematics department I studied in had a habit of calling out the As in the class or saying nothing at all. One professor considered us lucky as his alma mater called out the Fs when handing back papers. It’s interesting how information presented from different perspectives can have such a dramatic impact on the range of emotions it invokes.

    A postmortem is an activity that focuses on the time line leading up to a failure. Etsy has done an excellent job of encouraging blameless postmortems in industry. It’s a matter of when things are going to fail, not if failure will occur, when we’re dealing with complex systems.

    Retrospectives are a similar meeting held at the end of a development sprint where team members try to identify actions they can take to have more successful sprints in the future. Looking back on 2015 I see the need to balance our successes with our failures. When we’re focused on optimization we’re too quick to celebrate our successes. When we’re focused on stability we tend to only see the failures and ignore the wins as we come across them. When we’re well balanced it can be hard to tell the difference between our postmortems and our retrospectives, but we definitely have both.

    No matter how you feel about it 2015 is quickly slipping through our fingers. Chronos & Kairos represent time as we typically think of it, mechanically, and the opportunistic time to take action, respectively. I happen to be a fan of Janus, the god of all beginnings and endings, as he was said to exist through all space and time simultaneously. While I do not have that type of experience or foresight I am capable of looking at this past year as I anticipate the next and try to adjust.

    I can remember students that had trouble making passing grades that would be excited about Cs. I can remember other students that would be disappointed if they made a B and utterly devastated if they failed a test. Odds are you’ve been both of these students as our performance is just as dynamic as the courses we’re faced with over time. While we celebrate As it’s really a sign that a student isn’t being challenged. An F is a sign that they need to slow down or maybe back up a bit, and C students represent those that are being challenged by the curriculum and are successfully making forward progress.

    Cultural Balance

    In early November I gave a talk called “Precision, Accuracy, and Debugging the Bimodal Model” at the devOps Summit during Cloud Expo in Santa Clara. My focus was on the granularity in which we measure things and our tendency to get caught up in sorting things as right or wrong, good or bad, new or old, and into other extremes instead of simply acknowledging them as being different. While boolean categorization can be accurate in some contexts there’s often circumstances that require the precision provided by defining a spectrum between two extremes. These intermediate states often increase our odds of implementing successful change as they provide gradual steps to follow instead of expecting a single instantaneous jump to occur.

    When undeniable failure occurs we find where to learn by following a time line of small successes. At some point our expectations will not accurately line up with the reality we’re faced with and this is where we need to take a closer look.

    I was able to catch Devops Days Silicon Valley while I was in Santa Clara. The conversations I had where intended to gather more feedback around culture hacking and how culture influences the artifacts an organization produces. The theme was “bridging devOps cultures”, which made it easy to get feedback on some models I’ve been working on as it was a general topic of conversation at the event.

    Matter of fact, concerns that cultural change is ultimately what is needed for sincere advancement echoed throughout various sectors of the science and technology industries I found myself in this year. Devops Days Silicon Valley made it a theme, but cultural change seemed to be the hallway track topic at most of 2015’s events. Dave Snowden‘s cynefin framework was also present at most events I attended this year. I was introduced to the model during Jabe Bloom’s talk in NYC. Both J. Paul Reed and myself mentioned it at devOps Summit, but Jason Hand went into it in more detail with his talk, “Flaming Poo and the Human Response” at DevOps Days Silicon Valley.

    Lean Agile Scotland had a lot of discussion around culture and cynefin as well. It was even in the carpet. This worked for me since I made the trip in an effort to contrast the cultural differences in tech communities across US cities I had visited with an international group. Outside of the presentations culture was a common topic among corridor conversations. Lean Agile Scotland had folks from all over the world in attendance. It’s often the little differences, like calling it a corridor instead of a hallway, that we tend to find amusing.

    Conway’s Law links the artifacts an organization produces to its communication structure, which ultimately is an artifact of the organization’s culture itself. I’m starting to look at culture as the context of an organization with respect to how it communicates. Jabe Bloom and Chris Matts both gave presentations on culture in Edinburgh but the videos have not been posted yet. Here’s a picture of Jabe providing a definition of culture during Chris’s keynote.

    culture

    I was really delighted with Chris’s keynote. It was well rehearsed and entertaining but it also provided a missing piece to a vector based model I’ve been developing in an effort to more consistently benchmark organizations. I had hit a wall as I couldn’t see how I could quantify something like culture without injecting my personal bias into the metric, but what Chris shared about Hofstede’s Cultural Dimensions easily drops into my model and satisfies the requirement.

    I’ve received a lot of feedback on my model between Pittsburgh, Edinburgh, and Silicon Valley and it’s all been very encouraging. I’ve seen initial doubt turn to optimism and have had lots of people offer to help me extend my efforts and research this further, which I intend to be a focus for me in 2016. I do look forward to benchmarking organizations next year in an effort to find more paths between safe game states in industry.

    Looking back at the communication I described earlier with professors handing back graded tests, it is clear to see that how we learn is influenced by the emotions that are emphasized when we communicate. Are we focused on failure or are we focused on success? Does the culture we’re in value stability or optimization more? Does your team have postmortems, retrospectives, or both?

    Katherine Kirk advocated “Why Team Happiness Can be the Worst Thing To Aim For”. Insight is enveloped in emotion and equanimity is the tool we need to access it. As emotions are fleeting we should not pursue them. Instead, by anchoring ourselves in equanimity we are prepared for the waves of different emotions that will come and allow them to pass without judgement.

    Richard Sheridan, author of Joy Inc and CEO of Menlo Innovations, offered a different perspective with his keynote on how to “Build A Workplace People Love – Just add Joy“. He describes an open, high trust community that’s centered on learning. This means an environment that’s comfortable with failure when it happens and embraces the opportunity to learn from it. To pursue an emotion may be folly, but when culture is so harbored on stability that the failure ultimately manifests in the human components of the system it is only natural to drastically course correct towards different bearings.

    To Err is Human

    Jonathan Disher‘s talk, “DevOps Army of One” at devopsdays Silicon Valley was an excellent blend of technical and cultural information. We automate in an effort to gain some room for balance but that space can quickly be filled with other work instead.

    Asking for certain kinds of help, like more team members to offset workload, can bring leadership to question an individual’s level of commitment. Fear of retribution may discourage us from taking the personal time we need even if we don’t have adequate depth on the team, but eventually something will give. This is why burn out rate is high in our industry.

    The intention of mapping is to clearly challenge our assumptions. This helps us to realize we’re off course as soon as we checking our bearings. Wardley Mapping is a value chain mapping approach developed by Simon Wardley, appropriately enough. His mapping is quite practical and provides an interesting contrast to Edward Tufte‘s techniques, another data visualization and mapping expert I was pleased to meet this year.

    It is unfortunate that the time needed to give the level of attention required for this type of insight is often a considered a luxury in corporate cultures. Thanks to automation and adaptive visualization frameworks the standards are slowly rising. Will Evans‘s presentation from Lean Agile Scotland provides a solid blend of Wardley Mapping and design as he explains how new practices like these enter our workplaces.

    A Culture of Learning

    As 2015 enters its final moments I feel successful in finding consensus that the cause of our failure is a cultural one. I believe we cannot improve what we do not measure, and therefore my plans for 2016 are to further study, model, and visualize organizations and their equilibrium states.

    I cannot help but form a hypothesis on how we fix the problem before I’ve gained the context to make a reasonable attempt, but mapping is intended to help us challenge our assumptions. I’m assuming the solution lies in our actions as individuals and we must want to improve ourselves if we expect to see improvement around us.

    Often the final state we are striving for is literally not achievable. John Willis introduced me to this concept of “True North” from Toyota Kata, by Mike Rother, and suggested I read it myself to learn more. We’re currently reading Toyota Kata in a devOps book club we started, now that we’ve finished reading Designing Delivery by Jeff Sussna.

    The book club is intended to help us discuss ideas as we move through a book. I’ve been surprised to find how asynchronously a book club can operate. Everyone reading the same book at the same pace is a more typical book club approach, but given we live in different time zones and keep busy schedules we’re adapting to a model that’s a better fit. If you’ve read Designing Delivery, Toyota Kata, or plan to read either book and would like someone to discuss them with feel free to ping me on twitter.

    The improvement kata from Rother’s book maps the scientific method into Plan, Do, Check, and Act. Instead of focusing on processes we should focus on enabling people to improve themselves. We improve when we learn and we learn when we communicate. If your culture does not foster communication can if foster learning? If you’re not open and honest with yourself, how does that effect your environment?

    I could go into more detail about my intentions for next year and may do so at a later date, but my plans are of no use to you as we are different people. I would encourage you to gain awareness of your own situation by challenging the assumptions that have bounded your operation over the past year. You have had successes and failures and both deserve attention. Reviewing your outcomes with equanimity will help balance your actions, and hopefully lead you into a successful 2016. Happy New Year.

  • From Laboratories to DevOps Factories for Continuous Testing

    From Laboratories to DevOps Factories for Continuous Testing

    The prior blog Continuous Testing and Monitoring Essentials for SDN and NFV discussed the importance and suggested practices for testing and monitoring complex systems such as software defined networks and network functions that are virtualized . Whether the product being tested is a complex network system or not, with DevOps the laboratory is no longer just a place where engineers go to test their changes. In a DevOps environment, implemented according to best practices, the laboratory is becoming an on-line operation that is reliable, usable, remotely accessible and shareable by many types of remote users that expect 24/7/365 availability.
    The old “Santa’s workshop” model of development engineering (elves) hand-crafting products in specialized local laboratories and then handing off their products to operations staff (reindeer) for infrequent delivery (on Santa’s sleigh once per year) is being replaced by a fully automated, orchestrated software DevOps laboratory infrastructure akin to a modern factory staffed by robots churning out products at a dizzy pace. Elves and reindeer need to rest between deliveries and simply can’t keep up with today’s continuous delivery expectations. Santa’s “lap-top” has become an app store where users can download products anytime no matter if they have been bad or good (but be good for goodness sake!).

    If a modern DevOps factory-style laboratory is a priority on your Christmas list, here are some suggestions for tool requirements you may want to include on your letter to Santa.

    1. Inventory tools to record and track resources:
      All of the systems used in tests such as test tools, systems under test, and interconnection systems need to be tracked in an on-line inventory database in which each resource and the critical attributes are visible and addressable by administration staff and users of the resources. The ability to discover and import new or changed resources is essential. Without these capabilities it is impossible to keep up with the continuous changes that occur as the DevOps continuous test environment evolves to meet ongoing requirements.
    2. Tools to administer users:
      With a large number of users accessing lab resources features for administration of users and user groups are essential. Large scale DevOps environments require administration of hundreds or thousands of users with varying privileges for resource access according to project priorities and schedules. The ability for lab administrators to assign privileges to individual users or user-groups is essential to control access to some resources which may be restricted for security or other reasons.
    3. Ability to create, and save test configurations:
      While tests may be created to work with a variety of configurations every test assumes something about the test configuration. In a test environment implemented according to best practices the configuration of the test, or at least an abstract version of the test configuration is specified and associated with each test. Configuration data is saved in a database and may be recalled by search attribute matches when a test needs to run.
    4. Ability to reserve tests and test configurations:
      With a large number of lab users in large scale DevOps environments and limited lab resources conflict resolution features are needed. The ability assign real resources to abstract configurations provide the flexibility to use lab resources more efficiently. The ability to reserve test resources and test configurations in advance according to a calendar, or immediately when an event trigger fires are important options to minimize wasted time for both users who may have limited time windows or automated processes that have non-deterministic schedules.

    5. Ability to orchestrate many automated tests in parallel:
      In large scale DevOps environments the number of tests that need to run in a given short DevOps cycle may vary from a few tests to thousands depending on the complexity and nature of the change that are integrated in the cycle. The tools need to be able to scale the tests and run on as many physical and virtual test resources as are available, to minimize the total time for test.

    6. Test Environment Metrics:
      The entire DevOps pipeline depends on reliable, scalable and maintainable continuous testing. Built-in metrics are essential to monitor the environment’s reliability, scalability and maintainability.

    The above is a partial list of suggestions for a state-of-the art DevOps continuous testing laboratory “factories” list. At Spirent we think continuous testing laboratory “factories” are critical for successful DevOps.

    You can read more about our views at Spirent.com/solutions/LabManagement

    What do you think of these suggestions and do you have others that should be mentioned?

    In the meantime we wish Santa, Elves and Rudolf, reindeer friends and you happy holidays and continuous happiness for the New Year!

    Santa's Workshop

  • Merry Christmas, Happy New Year and Happy Holidays from DevOps.com

    Merry Christmas, Happy New Year and Happy Holidays from DevOps.com

    We will be shutting down for the Christmas holiday tomorrow and will only publish lightly next week as we get ready to update the site for the new year. But we wanted to take a moment to share our holiday greetings card with you and wish you a year of continuous good cheer, good times and peace on earth!DevOpsComHolidayStandAlone02