devopsy.kinsta.cloud

Category: DevOps Toolbox

  • Case Study: Pearson Weaves ThreadFix into AppSec, Part 2

    Case Study: Pearson Weaves ThreadFix into AppSec, Part 2

    In part one, we discussed how education content publisher Pearson applied ThreadFix to its AppSec request workflow needs. Here’s the rest of the story.

    The ThreadFix Implementation

    Senior software security engineer Matt Tesauro added ThreadFix to Pearson’s workflow in the following steps:

    1. Using FPM, Tesauro created a Debian Linux package internally to deploy the ThreadFix Tomcat Package (ThreadFix also offers its own Debian package for this).
    2. Tesauro hooked ThreadFix into each of Pearson’s toolsets using the tool scanner file formats ThreadFix understands.
    3. Using the importers built into ThreadFix, Tesauro connected it to the Checkmarx, Veracode, BurpSuite, ZAP, WhiteHat, AppScan and Qualys testing tools Pearson uses.
    4. To make that happen, Tesauro pointed ThreadFix to an API.
    5. Tesauro then set up an API key.
    6. The system was then set to fetch the test results.
    7. If necessary, Tesauro wrote glue code to upload the results to ThreadFix.

    Pearson used ThreadFix to normalize and de-dupe what had proceeded from manual assessments, dynamic scans on applications and static code analyses of source code, directing those outputs into ThreadFix. “So if a static tool and a dynamic tool both found the same cross-site scripting issue, we fed those into ThreadFix, which de-duped those and gave us just one issue to report so that we didn’t bury two teams with fixing the same single coding error,” says Tesauro.

    “ThreadFix became the one source of knowledge for all of the output of our team, which for us is the security vulnerabilities we find in the applications that we either created, bought or had a third party create,” says Tesauro. That enabled Pearson’s AppSec team to feed tickets into JIRA of backlogs from development teams, which made it easier to produce metrics and do reporting. ThreadFix made it easier to feed data into governance, risk, and compliance tools such as RSA Archer. “That enables visibility into work in progress and workflow and into areas of the business that affect AppSec’s work,” Tesauro notes.

    Additional Capabilities with ThreadFix

    Tesauro eventually expanded ThreadFix to other departments beyond AppSec to benefit from its capabilities and the metrics it could produce regarding app progress.

    Tesauro notes a point between the “Bag of Holding” and ThreadFix in the AppSec pipeline that required some work. He used a StackStorm orchestration and layered ChatOps on top to enable new developer capabilities at that point in the process.

    “Now you can go into our private channel and use HipChat to ask the system, for example, to set up Checkmarx for a particular application in a particular business unit. You give it the URL pointer to that unit’s Git repository and off the system goes, setting up the application profile in the Bag of Holding and in ThreadFix and then talking to Checkmarx to set up weekly scans of the code that sits on the master branch. It set that all up in a minute,” says Tesauro.

    HigherEd was the first additional group inside Pearson to benefit from ThreadFix metrics. One such metric was a 44 percent decrease in time-to-fix after adding a JIRA integration into the mix, according to Tesauro. “Once we started producing those numbers, we had a lot of other business units ask, ‘Hey, why does HigherEd get these great time-to-fix numbers? We want them too for our apps.’ It was a great way to get buy-in,” says Tesauro.

    Further Results

    AppSec saw a 5x increase in throughput for the request workflow and went from 44 application assessments in 2014 to fewer than 200 in 2015, Tesauro notes.

    “In 2015, we lost two and a half people and still sped up the process,” he says.

  • Case Study: Pearson Weaves ThreadFix into AppSec, Part 1

    Case Study: Pearson Weaves ThreadFix into AppSec, Part 1

    Pearson is a publisher of education industry content to meet the needs of teachers and students from kindergarten/early learning through higher education and continuing education (for professionals). The company uses a mix of software, from legacy third-party software and “classic ASP apps that are on life support to auto-scaling systems on Amazon,” says Matt Tesauro, senior software security engineer at Pearson. “It’s a very diverse development environment.” Which meant it was no easy task keeping it secure.

    Pearson Before DevOps

    Prior to adopting DevOps, Pearson was bogged down by manual processes, especially in application security. “People sent requests buried in email or, if you were lucky, in a Google spreadsheet. There were very few clear lines and hand-offs between the different activities that were the responsibility of the AppSec group,” explains Tesauro. It was a very manual, custom approach in terms of request handling: someone would respond to a request with an activity, which would require a report that was then emailed.

    At the time, Pearson had no single, well-established methodology or workflow for these activities. There was no particular look and feel or design for creating dynamic, manual or static assessments. There was no list of inputs into the process or outputs to expect at the end of the process. “At best, everything flowed in an ad hoc fashion,” says Tesauro.

    Then, about two years ago, Tesauro and a senior-level peer at Pearson attempted to prioritize those apps the company most needed to assess for security and related issues. They had hoped and planned it would be about 10 apps, to keep the number and the workload manageable. For political reasons, they were able to settle on no fewer than 64 apps that needed appraisal. “We ended up being able to assess only 44 out of the 64 top apps. There are 2,000 and some-odd apps at Pearson, in rough numbers. Just the scale told us we were going to fail in the long run,” says Tesauro.

    Ramping Up to Change

    To resolve this app assessment challenge, Tesauro looked to institute DevOps and Build Pipeline approaches to lay the groundwork for eventually integrating security into the various development teams at Pearson. “I made equivalents of DevOps and Build Pipeline approaches for how we did work in the AppSec program at Pearson,” says Tesauro. These included standard methods of input and intake of service requests similar to how the business produces software development targets for a DevOps shop to hit. That’s when Pearson began to work on instituting and leveraging ThreadFix.

    “ThreadFix was one of the first two parts of that pipeline,” he says. The other part of the AppSec pipeline solution was what Tesauro calls the “Bag of Holding,” a mechanism and repository for tracking all the team-to-team interactions and work assignments. That feeds into ThreadFix.

    During that time, and for about the first half of 2015, Tesauro and a teammate automated a number of manual steps by creating code to perform the “mindless work” so developers could concentrate on their core business instead. “Now when the static code analysis tool Checkmarx finishes its analysis, it generates a report, washes the directory and pushes the report into ThreadFix,” says Tesauro. He also automated a number of other test applications and tools.

    Stay tuned for part two of this case study, coming soon.

  • Performance Testing: The Unexpected Missing Link in DevOps

    Performance Testing: The Unexpected Missing Link in DevOps

    It is no secret that DevOps and continuous delivery goes hand in hand in the enterprise, but how do processes such as performance testing fit in to the equation?

    Many enterprises still harbor the traditional way of thinking—that the QA and performance testing group is completely separate from the development group. A widely held myth is that each has different roles, responsibilities, backgrounds and areas of expertise. However, with DevOps gaining popularity in the enterprise, the distinct separation between testing, development and operations is no longer. In fact, to successfully implement performance testing, the once-siloed departments must work together.

    Creating a Culture for DevOps

    The most common purpose of DevOps is to have an automated pipeline of software and application deployment updates. An enterprise cannot implement DevOps unless all are on board, from the top down. To do this, there must be buy-in from not only the software engineers and technology buyers, but also the C-suite. This ensures DevOps processes and strategies are being implemented appropriately.

    However, it’s not just about getting the tools needed for DevOps; there must be a culture shift toward collaboration for it to be effective. That is when an enterprise gets the most from the shift to DevOps and becomes the most agile.

    Once DevOps is embraced, experts and power users can help facilitate how it is being used to drive maximum benefit and help the organization get more from it. The key way to determine and evaluate this maximum benefit is through continuously testing throughout the development process. While the first thought may be for “functional” testing (to ensure code operates without errors and to provide fast feedback to developers when things go wrong), no organization can afford to release applications that deliver slow user experiences, and so “performance” testing with fast feedback is equally critical. Continuous testing is only scalable and sustainable if it is automated.

    DevOps Makes Load Testing Easier

    A key benefit of DevOps is that automation makes everything—from implementation to functional and performance testing—faster and more predictable. Tool chains ensure that the “right things” are done with every code push. A concern commonly cited by those who have not yet implemented performance testing is that it is “too time consuming” and they will be unable to facilitate different tests at the rate they need to be done to be effective. But surprise! Organizations that already practice DevOps are in a better position to add performance testing to the mix, in a fully automated fashion. A solution like BlazeMeter takes just minutes to get API tests up and running, by just adding the HTTP transaction for specific pages or endpoints to test. Once set up, these tests will continuously run automatically and will notify the team when there are any problems or potential problem areas. When there are known surges in website traffic, more Uniform Resource Identifiers (URIs) can be added to the test, but this will not impact the autonomy of it.

    Another key function of DevOps is scalability through the enterprise, and scalability of user traffic is extremely important. While performance testing would still scale without DevOps, operating on the same principles makes each concept more relevant to the folks who are analyzing the outcome of both. The key is to integrate performance testing directly with an already existing continuous delivery or integration pipeline, such as Jenkins or CircleCI. Here, fast, small-scoped performance tests are able to run in parallel to ensure performance is as expected before promoting a build to the next stage. Tests run in staging or QA environments, such as IBM BlueMix or Chef, can also be automated and integrated.

    Performance testing is also important not just because of its easy integration into DevOps practices and how it fits into the CD pipeline, but the overall customer experience as well. A properly functioning webpage or mobile application is the connection between your brand and your target customer, and if it doesn’t load properly or in a timely manner, you may lose a customer to a competitor. Performance testing ensures the user doesn’t experience slowdowns or the dreaded downtime, which is often the result of a poorly functioning back-end system. Incorporating performance testing into workflows creates a centralized environment where all team members have what they need to design, run or analyze tests, while communicating quickly and clearly with each other and everybody in the enterprise.

    Performance testing and DevOps are not opposing forces; and performance tests run automatically within the tool chain save time, rather than consuming more. The reality is that during this age of rapid software delivery, testing is the glue that keeps the development process on track and high quality. Performance testing should be part of every software delivery workflow, and DevOps is allowing that to happen.

    About the Author/Michael Sage, Chief Evangelist

    unnamedMichael Sage brings a 15-year career of enterprise software pedigree and customer success to BlazeMeter. Prior to BlazeMeter, Michael (better known as “Sage”) was a solutions architect and consultant in software delivery and performance management. He has worked with industry-leading companies including Mercury Interactive, Hewlett-Packard and New Relic, helping teams implement integrated solutions that enhanced the performance and experience of their customers. A self-taught hacker and open source evangelist, Sage always has the pulse on the cultural zeitgeist inflected with a heathy dose of humor.

  • At the Intersection of Git and DevOps

    At the Intersection of Git and DevOps

    Everyone knows Git. It’s a favorite developer tool and for good reason. Developers love its flexibility, speed and ability to branch locally and in place. It also appeals to the attraction to community development.

    But as developers adopt Git within enterprises, IT managers face the challenge of reconciling the needs of developers with the needs of the larger organization, especially when it comes to DevOps. In this article, we will examine some of these challenges and suggest best practices that can alleviate them.

    Performance

    IT organizations often rely on performance and low latency as key indicators of the adoptability of any solution. Speed tests run against Git’s network operations generate impressive results, since the Git protocol for transferring data is highly optimized. Over time, however, an internal implementation reveals limitations that can impact the speed and efficiency of the delivery pipeline.

    First, Git works very well for small projects. Yet, when projects grow larger, many teams will feel the pinch of its all-or-nothing architecture. A developer who needs to work on a project must retrieve (or “clone”) every file associated with it. For enterprise software, this necessity is not always feasible or desirable. Large projects can require a lot of disk space and time to clone, even though much of the repository content may not be relevant to that developer. The all-or-nothing approach of a Git clone also compromises security—users will have the same access to the whole repository, whether they should have it or not.

    Second, Git is optimized for fairly small assets, such as code and configuration files. But many organizations need to version large binary files such as images, videos and CAD designs. Large binary files are also often the product of the build system. Such content can quickly bloat the size of a repository and, by extension, the file systems of contributing developers. A less obvious issue with large files is the fact that some Git functions require the calculation of hash values over the contents of the repository. As a result, large binary assets can inexplicably reduce some operations to a crawl while others execute almost instantly. To address these issues, you should employ an external store for your large binary assets. There are no formal standards for external asset stores, but there are options available, including git-annex and Git LFS.

    The solution that Git offers is to separate components into dedicated repositories or to use Git submodules. Both approaches lead to what is known as Git sprawl, a condition that increases complexity and risk. Git sprawl can be avoided using the narrow cloning approach. Narrow cloning enables developers to retrieve just the subset they need from the contents of a code repository. Although the developer community has long requested this feature, Git currently provides no support for narrow cloning. For these reasons, you should standardize on a Git management system that offers narrow cloning, while using shallow cloning as much as possible to limit file revisions.

    As long as repositories are logically divided along component lines, developers might not experience any immediate problems with Git sprawl. Build and test teams, by contrast, are likely to feel the pain immediately. Trying to find all the correct and related revisions of files from many different repositories is an expensive and risky process. Builds will frequently fail if the input files are not correct. The ideal agile workflow supports build and test on every check-in, especially on the mainline. If builds frequently fail, the build and test system will not be able to keep up with the development pace and will slow down the entire DevOps pipeline.

    Another bad practice that can result from Git sprawl is having inconsistent or conflicting branching models for different teams or repositories. Dealing with the idiosyncrasies of individual developers or teams is needlessly frustrating and inefficient for the build and test teams. But Git provides no support for defining or enforcing branching strategies across multiple development teams. For this reason, it is critical to define an overall branching strategy, socialize it throughout the organization and review practices on an ongoing basis.

    Hosting

    Hosting options for cloud deployments of Git are plentiful. Organizations embracing Git need to evaluate these options carefully because the choice will impact not only information security but also DevOps. Developers generally seek options that make it easy to create new projects, clone or fork and push their work, preferably with great review tools and a simple method to deliver units of work to the master branch.

    The developer’s desire for ease of use is often at odds with DevOps’ need for scalability. Available Git management systems vary significantly in their capacity to support enterprise IT, with capabilities ranging from single-server topology all the way up to clustering and, in some cases, high availability. Note that none of the solutions supports disaster recovery. Therefore, organizations need to identify tools for backup and recovery, while also defining internal processes to ensure the system can be restored after a catastrophic event.

    Security

    Developers’ need for speed and customizability also often run counter to the enterprise’s need to protect the intellectual property locked away in Git. The key issue is that Git lacks a robust and manageable mechanism for fine-grained access control. Git is limited to providing security in the form of authentication; users must present credentials (username/password or SSH key) to access private repositories. Once the user has accessed the repository, Git has no way of controlling what that user does. While this works well for democratized, community-based open source development, it creates security holes and compliance problems for enterprises.

    For these reasons, it is critical to think through the necessary roles and permissions that will enable developers and DevOps to do their jobs, while minimizing the risk incurred by hosted Git solutions. Personnel should always be provided with “least privilege” access to assets—that is, they should be given only the permissions necessary to complete their tasks. For compliance purposes, it is often necessary to provide logs showing who did what when. This requirement is especially true for enterprises in regulated industries such as health care and finance. Git, however, gives users the ability to destroy virtually every form of historical evidence. Again, this is useful for developers, but detrimental to information security and governability.

    In summary, DevOps is penetrating enterprise IT departments at the same time that Git management solutions are proliferating. The differences among competing Git management solutions become most apparent in the context of DevOps. Remember, DevOps has to manage everything across the application lifecycle. Evaluating Git solely from the perspective of development teams, without considering the requirements of DevOps, can be a recipe for the failure of your DevOps initiative.

    About the Author/Mark Warren

    MarkWarren_BW2Mark Warren is Product Marketing Director at Perforce, based in the United Kingdom. Mark has more than 25 years of experience working with development and product management teams in software development tools and configuration management.

  • How do you measure DevOps? What does that even mean?

    What are some of the metrics you should be monitoring to track your DevOps efforts, and which metrics should you abandon?

    (more…)

  • 3 Ways Manual Processes are Risking Your Development Cycle

    3 Ways Manual Processes are Risking Your Development Cycle

    During my two decades in IT, I’ve seen the demand for Agile and its associated methodologies grow. At the same time, however, I’ve witnessed the database being left behind.

    DBmaestro, where I am co-founder and CTO, recently conducted a survey titled, “Database Deployment & Development Risks,” to gain insights into the risks associated with various database development practices. The survey focused on manual processes used throughout the development cycle.

    The results revealed two divergent realities: Despite respondents, from diverse sectors, overwhelmingly recognizing the risks associated with manual process for the database, many organizations continue to use processes and solutions that don’t fully automate the development cycle.

    One area where this was particularly apparent was with the source control. It’s imperative that development teams are able to trust their source control. But even when standard source control solutions are being utilized, 70 percent of those surveyed are still interjecting manual processes.

    What that means is that even though many of these professionals have some type of solution, they still see a clear and present risk because:

    1. When integrating changes, it’s possible for one developer to wrongly override valid changes with older revisions from other development branches, a potential concern that nearly 90 percent of survey respondents associate with risk. Still, more than half (53 percent) of the IT professionals surveyed integrate changes, creating the possibility that such overrides may occur.

    Nearly three-fourths (70 percent) reported having a database source control, but 53 percent reported that they continue to have problems that a more adequate source control solution should resolve. In this case, there is a problem with the source control tool implemented.

    2. The database may not be in sync with the version control repository. Ninety percent of the IT professionals surveyed associated the potential for these sources to be out of sync with risk. What’s more, 72 percent of those surveyed admit that their database may not be in full sync with the version control repository.

    3. Introducing changes without first verifying each database object vs. its version control revision to ensure each database object is up to date is a process that nearly 90 percent of survey participants associate with risk. Despite a widespread awareness that failing to verify accuracy between database objects and database source control before introducing changes is a risky practice, 53 percent continue to do so.

    That means that the remaining 47 percent of respondents do not trust their source control solution regarding the database code and are implementing steps to validate it matches the database before introducing changes.

    While interjecting manual steps can help alleviate the risk associated with the possibility of overriding critical changes in the database, it’s time-consuming, and manual interruptions to the continuous delivery cycle introduce risks of their own. When you can’t place full trust in source control, two developers working on the same database code or objects can easily override or revert critical changes introduced by the other developer when the script is updated in the database—sometimes with disastrous consequences.

    Only a Database Enforced Source Control solution that covers database lifecycle management (DLM) from development through build and deployment can create a “Single Source of Truth” for your database development assets. This allows you to gain trust in your source control and implement the rapid changes today’s development cycles demand—without plugging the process with time-consuming manual steps. Database Enforced Source Control promotes complete trust in your source code, allowing developers to work simultaneously on the same database objects without the risk of overriding important changes when scripts are updated, promoting true synchronization across dispersed development teams.

    Download the complete results of the 2015 “Database Deployment & Development Risks.”

    About the author:
    image001Yaniv Yehuda is the Co-Founder and CTO of DBmaestro, an Enterprise Software Development Company focusing on database development and deployment technologies. Yaniv is also the Co-Founder and the head of development for Extreme Technology, an IT service provider for the Israeli market. He was a captain in Mamram, the Israel Defense Forces computer centers, where he served as a software engineering manager.

  • The Role of APM in Continuous Integration and Continuous Release

    Today’s software-defined and -driven business requires fast changes in business models—a phenomenon permeating successful businesses. What this means is almost every company is learning how to make small, rapid changes and adjustments within their business, especially within software systems. As a result, IT is feeling immense pressure to evolve, and the net result is a major uptick in private cloud environments (especially Apprenda, Pivotal Cloud Foundry and Red Hat OpenShift) and public clouds (especially Amazon and Microsoft) within our install base of large enterprises. These new platforms enable faster development, testing and releasing of software.

    Enterprise customers are getting much smarter by using automation to drive the software life cycle, including continuous integration, and even experimenting with and using continuous release processes. This is no longer a startup scenario; it’s a strategy for our large, enterprise customers. These companies are trying to build agile development and operations teams by changing people, process and technology to enable a DevOps feedback loop:
    image02

    Wikipedia describes the continuous delivery loop as follows:

    DeliveryLoop

    Within this loop, application performance management (APM) can be part of build, unit, regression and load tests to drive the automated acceptance testing before new code is pushed to production. In fact, at the recent AppSphere 2015 user conference we had several great talks on this subject.

    The first talk was presented by our customer The Container Store‘s  August Azzarello, who discussed, “How The Container Store uses AppDynamics in the Development Lifecycle.” The company began with APM in production, in an effort to improve software quality before production. It began the expansion by installing APM within test, integrating it into functional and performance test suites. It also enabled alerting from dev/test so that developers and testers both understood when there were performance deviations and issues before production.

    Key features the company leveraged in pre-production included dashboards such as this one:image03

    Comparative analysis views such as this one:

    image04

    And being able to understand between releases if they were improving or degrading:

    image00

    The Container Store is using open source tools including Selenium for functional testing and Locust.io for performance testing.

    During his discussion, August also explained some of the major benefits the company saw from implementing APM in pre-production. Here are some best practices outlined by August in his discussion:

    • Monitor everything
    • Test continuously
    • Performance test early in development life cycle
    • Empower development & QA team members

    Some benefits The Container Store has seen, according to August, are:

    • Set performance expectations before production deployments – about 40 percent improvement since it started using AppDynamics in test
    • Fine-tune alert and business transaction policies prior to production deployments
    • Identify testing requirements and testing gaps
    • Decrease performance test result reporting to 20 minutes from five hours

    One of my favorite talks (which is why I selected it for the deep dive track at AppSphere) was the talk given by one of our senior sales engineers Steve Sturtevant, who came to AppDynamics from PayPal two years ago. In his talk, titled, “DevOps and Agile: AppDynamics in Continuous Integration Environments,” he explained the integration work he did. He also delved into the questions, Can you scale and auto implement monitoring in dynamic environments? Do you quantify the impact of change? Steve also offered a demo with integration between Puppet, Jenkins, ServiceNow and AppDynamics in his examples, with automation throughout the life cycle.

    One of our partners Electric Cloud specializes in continuous delivery and embraces an open ecosystem of technology providers and open source software. It has put together some great resources for people looking to explore the rest of the feedback loop:

  • What it’s Like Working as Quality Assurance in Consultancy Companies

    What it’s Like Working as Quality Assurance in Consultancy Companies

    Quality Assurance (QA) engineers working at consultancy companies should embrace trying new technologies and solving problems. This role is best defined by two words: variety and challenges. QA engineers at consultancy companies must pay great attention to detail and check all of their actions meticulously to make sure everything is being done correctly. The position also requires equal focus on the quality of the software application and client expectations, and constant communication between all individuals and teams involved in the process.

    Thinking about a job in QA? Here are some more details about this demanding yet rewarding position.

    Expectations

    Working as a QA engineer in a consultancy company demands very strong communication skills, flexibility and prioritization skills to help deliver a high-quality product in a timely manner. This role requires continuous improvement of both the process and working methods, which are not limited to testing or preventing defects but also consider the application’s lifecycle and ways to improve the product.

    Challenges can arise from working with myriad intricate software applications and environments. As the complexity of the software and the multitude of targets that can be addressed increase, so do the technical and communication challenges. A good QA engineer should always plan for this complexity in each project and be prepared to handle these challenges.

    Variety

    QA challenges and projects become even more complex and varied at consultancy companies. In a consultancy company engineers work with multiple companies from different cultural backgrounds and different levels of technical experience. Sometimes there are significant cultural differences across projects and within a team, which can have a big impact on the overall quality assurance process. To overcome any obstacles, QA engineers must communicate frequently with each company to answer any questions that may occur during the project lifecycle and to keep it on track.

    For each project, the QA engineer is expected to contribute to various parts of the process, which can include everything from client and internal team communications to delivering technical requirements. In addition, QA engineers must ensure technical requirements are communicated, tracked, planned, scheduled and executed correctly for each team. This is where attention to detail and prioritization skills are key, in making sure the correct communication is reaching the correct teams so the project runs smoothly and successfully.

    In addition to the innate variety that exists within the QA role, there is a fair amount of diversity, with multiple clients of different sizes spread across numerous industries. While some clients write out the project specifications and define clearly the expected outcome of the software application, others don’t, creating another opportunity for a QA engineer to contribute to a new project.

    Finally, QA engineers in a consultancy company constantly work with a variety of platforms and applications, from desktop to mobile projects and web-based to server-side. I see this as a positive—it keeps you on your toes and you constantly are gaining additional technical and architectural experience.

    As a Quality Assurance consultant, you will be expected to work on a wide variety of technologies and communicate effectively under a variety of circumstances. While the position can be challenging, it’s never boring. And knowing the project is successful because of your work is extremely rewarding.

  • From Laboratories to DevOps Factories for Continuous Testing

    From Laboratories to DevOps Factories for Continuous Testing

    The prior blog Continuous Testing and Monitoring Essentials for SDN and NFV discussed the importance and suggested practices for testing and monitoring complex systems such as software defined networks and network functions that are virtualized . Whether the product being tested is a complex network system or not, with DevOps the laboratory is no longer just a place where engineers go to test their changes. In a DevOps environment, implemented according to best practices, the laboratory is becoming an on-line operation that is reliable, usable, remotely accessible and shareable by many types of remote users that expect 24/7/365 availability.
    The old “Santa’s workshop” model of development engineering (elves) hand-crafting products in specialized local laboratories and then handing off their products to operations staff (reindeer) for infrequent delivery (on Santa’s sleigh once per year) is being replaced by a fully automated, orchestrated software DevOps laboratory infrastructure akin to a modern factory staffed by robots churning out products at a dizzy pace. Elves and reindeer need to rest between deliveries and simply can’t keep up with today’s continuous delivery expectations. Santa’s “lap-top” has become an app store where users can download products anytime no matter if they have been bad or good (but be good for goodness sake!).

    If a modern DevOps factory-style laboratory is a priority on your Christmas list, here are some suggestions for tool requirements you may want to include on your letter to Santa.

    1. Inventory tools to record and track resources:
      All of the systems used in tests such as test tools, systems under test, and interconnection systems need to be tracked in an on-line inventory database in which each resource and the critical attributes are visible and addressable by administration staff and users of the resources. The ability to discover and import new or changed resources is essential. Without these capabilities it is impossible to keep up with the continuous changes that occur as the DevOps continuous test environment evolves to meet ongoing requirements.
    2. Tools to administer users:
      With a large number of users accessing lab resources features for administration of users and user groups are essential. Large scale DevOps environments require administration of hundreds or thousands of users with varying privileges for resource access according to project priorities and schedules. The ability for lab administrators to assign privileges to individual users or user-groups is essential to control access to some resources which may be restricted for security or other reasons.
    3. Ability to create, and save test configurations:
      While tests may be created to work with a variety of configurations every test assumes something about the test configuration. In a test environment implemented according to best practices the configuration of the test, or at least an abstract version of the test configuration is specified and associated with each test. Configuration data is saved in a database and may be recalled by search attribute matches when a test needs to run.

    4. Ability to reserve tests and test configurations:
      With a large number of lab users in large scale DevOps environments and limited lab resources conflict resolution features are needed. The ability assign real resources to abstract configurations provide the flexibility to use lab resources more efficiently. The ability to reserve test resources and test configurations in advance according to a calendar, or immediately when an event trigger fires are important options to minimize wasted time for both users who may have limited time windows or automated processes that have non-deterministic schedules.

    5. Ability to orchestrate many automated tests in parallel:
      In large scale DevOps environments the number of tests that need to run in a given short DevOps cycle may vary from a few tests to thousands depending on the complexity and nature of the change that are integrated in the cycle. The tools need to be able to scale the tests and run on as many physical and virtual test resources as are available, to minimize the total time for test.

    6. Test Environment Metrics:
      The entire DevOps pipeline depends on reliable, scalable and maintainable continuous testing. Built-in metrics are essential to monitor the environment’s reliability, scalability and maintainability.

    The above is a partial list of suggestions for a state-of-the art DevOps continuous testing laboratory “factories” list. At Spirent we think continuous testing laboratory “factories” are critical for successful DevOps.

    You can read more about our views at Spirent.com/solutions/LabManagement

    What do you think of these suggestions and do you have others that should be mentioned?

    In the meantime we wish Santa, Elves and Rudolf, reindeer friends and you happy holidays and continuous happiness for the New Year!

    Santa's Workshop

  • Winning the game with devops at Gameforge

    Winning the game with devops at Gameforge

    Gameforge is a leading game developer of free-to-play massive online multiplayer (MMO) style games.  It provides over 20 games to 450 million registered users on more than 20,000 servers worldwide.  As Gameforge continues to grow, the challenges with server maintenance and communication between developers and system administrators continues to increase.  I recently spoke with Felix Oechsler, Lead Windows System Administrator to look at the ways Gameforge is overcoming those challenges utilizing DevOps tools and methodology.

    I thought it would be interesting to capture the implementation of DevOps at a gaming company since I haven’t seen much on gaming here at devopsy.kinsta.cloud, so I took it upon myself to speak with Felix.

    As a sys admin, there are two main systems at Gameforge, the windows and linux systems.  Felix’s focus is on managing the servers and his job summary includes server upkeep, security, performance management, log data, and to oversee the health of the servers.  When I asked Felix how Gameforge came to use DevOps, he told me that it started 2 years ago on a project they were working on.  Many issues arose because the communication between the system administrators and developers was not efficient in regards to dumps and data logs.  When an error had occurred they needed to inform the developers and then manually hand the data from the server over to the developer team to fix the issues.  This took away valuable time from Felix and his fellow sys admins, and also caused various security issues, since they could not give server access to the developers.  Realizing the teams needed a way to interact more efficiently and effectively they asked the simple question, how can we do this better?

    “We needed a better log management solution to quickly and easily troubleshoot issues before our gamers noticed any problems in the games or systems.  We also wanted an all-in-one solution that would be easy to use and manage across our different departments.”

    -Felix Oechsler, Lead Windows System Adminstrator, Gameforge

    Alas! Devops to the rescue!

    Felix’s team began to implement devops tools and best practices such as Graylog’s centralized log management platform.  This allowed developers to see the logs in one centralized location without involving the sysadmins to comb through the logs to deliver the data to the developers, freeing up valuable time of sysadmins and minimizing security issues.  Graylog helps to deliver 5 beneficial uses including user access management, intuitive search and filtering, visualizations, alerting, and flexible log data extraction.  Explanations on each of these can be found here, a case study done by Graylog.

    “My development team created our own streams and set alerts for the games we were working on. We’re able to get real time alerts on critical errors in games and applications so we can react faster than before. In one case, our mobile game team received an alert that our connection to the Apple payment system was down. We were able to resolve the problem quickly and avoided the bad user experience of a payment not going through.”

    – Dennis Simon, Lead Developer, Gameforge

    Gameforge also begun using key performance indicators (KPIs), Graphite and Grafana, to see how the applications are performing, such as high CPU costs.  Using both Graylog and the KPIs, developers were able to continuous deploy updates to the games without involving the sysadmins, cutting down on time and cost, and effectively creating a better playing environment for their gamers.  Gameforge has developed their own personal, customized continuous deployment platform using C# and powershare to code the mechanism.  They decided to develop the platform in-house to best fit their structure and security policies.

    Since Gameforge has begun implementing devops tools and best practices they have seen great beneficial change.

    “We now monitor our entire server environment with Graylog and have total visibility, enabling us to search for internal application problems across multiple backend servers. Also, our Developers and DevOps people rely on Graylog’s alerting functionality to react to critical errors in our games or applications.  Using Graylog, we are able to provide a better user experience for our gamers. Our reaction time is much faster than before, and our players experience less outages and other error-related problems.”

    – Felix Oechsler, Lead Windows System Administrator, Gameforge

    Using DevOps, Gameforge has freed up valuable time for the system administrators and provided immense visibility to the development teams.  Visibility provides quick access to alerts on errors, allowing problems to be fixed quicker and pushed live via a continuous deployment environment.  It is safe to say that devops is a valuable methodology for system administrators and developers alike.