devopsy.kinsta.cloud

Category: DevOps Practice

  • DevOptimism: The Future of DevOps is Becoming Clearer

    DevOptimism: The Future of DevOps is Becoming Clearer

    DevOps has quickly matured from a fairy tale to a bestseller at most large enterprises. However, we are still at the beginning of the DevOps story, with many exciting chapters still to be written. This article looks at several emerging technologies shaping DevOps today and a handful of developing trends that will shape the ecosystem in the years to come.

    Today’s IT organizations increasingly are moving away from monolithic applications deployed through grand-slam releases with security slapped on after the fact. We are seeing many enterprises transitioning to more atomic applications released in continuous deployments with application security baked in from the beginning. Fueling today’s emerging DevOps is a bevy of new products focused on architectures including microservices, service mesh and function-as-a-service (FaaS); advancements in pipeline automation; and more adaptable security paradigms such as interactive application security testing (IAST) and runtime application self-protection (RASP). While these solutions herald a great step forward, they do demand that legacy applications be refactored into smaller granularity services, layered onto newer architectures and migrated to processes that allow greater automation.

    OK, that’s where we sit today: Great progress creating great opportunities and great challenges. But let’s head a little further down the road to understand where today’s advancements are driving DevOps tomorrow.

    There are five trends consistently appearing on the whiteboards of venture firms and forward-thinking startups that will shape DevOps in the upcoming years. At a macro-level, DevOps must become wider and smarter. As we dig layer deeper, we see five distinct patterns appear in which emerging technologies will make the release pipeline more insightful and proactive while allowing it to handle a wider spectrum of pipeline artifacts beyond just applications:

    • AI for DevOps: Artificial intelligence (AI) is making inroads into all facets of IT and DevOps is no exception. AI techniques soon will make the DevOps pipeline smarter, able to predict the impact and risk of deployments, spot procedural bottlenecks and identify automation shortcuts. Advancements in robotic process automation (RPA) lend themselves directly to optimizing the various handoff and automation points of the release pipeline. AI-based predictive analytics also will allow for higher fidelity operational capacity planning and pre-deployment fault prediction.
    • DevOps for AI: Today we are seeing a handful of new tools, commonly called MLOps, emerging on the market that allow developers and modelers to deploy new AI models rapidly and repeatably. However, these products have yet to collaborate closely with the Ops side of the enterprise. They are generally a development aid, not an operational aid—or said another way, they are all “ML” and very little “Ops.” This space will mature rapidly, however, allowing release of AI models via the DevOps ecosystem so as to include operational aspects. Conversely, tomorrow’s DevOps pipeline will also allow AI models that have been learning from real-world data to be extracted from production and moved back into the development environment.
    • DBOps: Today, new databases, schemas, indexes, triggers and stored procedures typically are deployed manually by DBAs outside the DevOps pipeline, with their operational impact being provided by best-guess capacity planning. This glaring gap soon will be closed by products that allow databases and their associated artifacts to be released through the deployment pipeline along with the applications with which they are coupled. This will not just encompass traditional relational databases, of course, but also support non-structured databases and object stores. The prime objective being that applications and their supporting data repositories will be developed, tested and deployed via the same automated pipeline as applications without the need for specialized database handholding.
    • DataOps: When new apps are deployed, the associated analytics and visualizations are typically brought online by data scientists after the fact through manual deployments. Looking forward, expect data analytics to be introduced into the DevOps pipeline soon after the requirements for the release are defined. Much like DBOps above, the goal is to eliminate specialized manual promotion. But beyond just synchronous deployment, DataOps will allow data governance, data loss prevention (DLP), data lifecycle management and copy data management features to be integrated during each new deployment.
    • Blockchain for DevOps: Today’s and tomorrow’s DevOps evolutions will increase the velocity, variety and volume of pipeline traffic dramatically. While this sounds promising, it undoubtedly will be a nightmare to audit, verify compliance and ultimately control and manage. One technology being eyed to untangle this spaghetti is a distributed consensus mechanism such as blockchain. Stop rolling your eyes—we’re not talking about Bitcoin or ICOs. We are describing the need for a centralized, permissioned, secure, distributed ledger to record, track, authenticate and audit the torrent of pipeline transactions. Blockchain seems to fit the bill nicely and also has the added benefit of enabling digital contracts at each pipeline stage that can be verified at runtime to trigger artifact promotion and also audited long after the fact to ensure that all rules were followed.

    DevOps has already become the standard deployment paradigm in many enterprise IT shops and its adoption will continue to become more pervasive. But this growth requires DevOps products and processes to become smarter, more secure and more controllable as the release pipeline incorporates a wider set of deployment artifacts beyond just applications. This is a daunting challenge to be sure, but there are exciting new products that will emerge to help write the next chapters in the DevOps story.

    — Mark Campbell

  • Software AG Acquires Built.io to Advance Integration Strategy

    Software AG Acquires Built.io to Advance Integration Strategy

    Software AG has acquired Built.io as part of an effort to provide a multitier approach to application integration designed for different classes of developers.

    Subhash Ramachadran, senior vice president for product management at Software AG, said Built.io adds a low-code approach to integration in the form of a platform-as-a-service (iPaaS) environment developed using Node.js to a portfolio of technologies that include the webMethods platform.

    Ramachadran said the low-code approach will appeal more to so-called citizen integrators who are trying to integrate business processes with minimal intervention on the part of an IT team. Many of those capabilities are now expected to be delivered in a software-as-a-service (SaaS) model that citizen developers can invoke on demand. Built.io, for example, already provides built-in support for more than 300 applications and services. Those tools, however, don’t replace the need for a set of integration frameworks aimed at more complex tasks that require professional developers, Ramachadran said.

    Increasingly, there are large numbers of use cases for integration that simply don’t require a professional developer to accomplish, he noted. In fact, the number of use cases that can be addressed by tools such as Built.io might be substantially larger than the use cases requiring a platform the size and scale of webMethods.

    Software AG’s goal is to create a single platform to provide organizations with multiple classes of integration frameworks that can be applied to different personas within an organization, Ramachadran said. A common platform eventually would make it possible for integrations of varying skill levels to collaborate more easily.

    Those frameworks could then be plugged into any continuous integration/continuous development (CI/CD) platform to enable various Software AG integration platforms to be natural extensions of a DevOps environment, he added. The webMethods integration platform already supports Kubernetes clusters and Docker containers as a runtime environment.

    As DevOps processes continue to evolve, Ramachadran said Software AG is betting that organizations will want to make it easier for different classes of developers working with multiple types of tools to collaborate.

    To facilitate that transition, Software AG now enables everything from sharing code via GitHub to deploying micro gateways for managing application programming interfaces (APIs) within the context of a set of microservices, he noted, adding those capabilities will only become more important as organizations begin to build highly distributed internet of things (IoT) applications running on a wide variety of platforms.

    Microservices, however, create something of a paradox when it comes to integration. On the one hand, frameworks such as Built.io will make it simpler for more organizations to integrate multiple microservices and legacy monolithic applications. On the other hand, microservices are making integration more challenging for many professional developers as they continue to expand across the enterprise.

    The challenge facing most organizations will be striking the right balance between microservices and existing monolithic applications, many of which don’t naturally lend themselves to being re-engineered into a series of microservices. In fact, in many cases it may be difficult to distinguish from an integration standpoint what constitutes a microservice versus a monolithic application, so long as both can be invoked using a simple API.

    — Mike Vizard

  • OutSystems Infuses DevOps Processes into Low-Code Platform

    OutSystems Infuses DevOps Processes into Low-Code Platform

    OutSystems has turned its low-code platform into a framework for generating microservices that can run on-premises or in a variety of public clouds.

    Mike Hughes, senior director of product marketing at OutSystems, said applications generated using OutSystems 11 now can be deployed on both platform-as-a-service (PaaS) environments and container-as-a-service (CaaS) environments, including Amazon Web Services public clouds and the Pivotal PaaS based on open source Cloud Foundry software and CaaS platforms such AWS ECS, Azure Container Service, or as a set of Docker containers running on-premises.

    In addition, OutSystems 11 includes a set of monitoring tools capable of identifying issues emanating from within the applications and external dependencies, as well as a set of impact analysis tools that identify issues before an application is deployed. Those tools are critical elements of a concerted effort to apply DevOps practices and principles to the development of applications created using low-code tools, said Hughes.

    OutSystems 11 also now includes an entitlement model for controlling who can create and manage services and who can consume them, using a full entitlement model that extends to all applications and services.

    Finally, OutSystems has added support for automated risk assessments, granular access controls and activity monitoring of supplemental manual processes along with a revamped user interface.

    IT organizations of all sizes are trying to cope with a growing application development backlog by embracing low-code tools that can be employed either by professional developers to accelerate development or so-called citizen developers to create applications on their own. Legacy approaches to application development have created a gridlock issue that organizations are not going to be able to break without embracing a more modern approach to application development that combines the best of low-code application development platforms and DevOps practices, said Hughes.

    Hughes noted that more organizations than ever are trying to accelerate application development at a time when digital process transformation has become a much high priority. The rate at which new applications can be built, tested and deployed has a direct impact on an organization’s ability to compete, he said. The challenge organizations face is that much of the subject matter expertise required to build these applications resides in business units that have limited application development expertise. The central IT organization, conversely, typically has a limited number of developers available to work on applications at any one time. Rather than simply providing citizen developers with low-code tools, OutSystems is making a case for an approach to rapid application development that provides guidelines for citizen developers in a way that doesn’t create the equivalent of a black box that professional developers can’t later drill down into to determine how an application is behaving.

    It may take a while for low-code application development and DevOps processes to meld within enterprise IT organizations. But the more DevOps tools that come included with the application development frameworks being provided, the greater the chances DevOps processes eventually will pervade the enterprise.

    — Mike Vizard

  • Splunk Aims to Trigger DevOps Processes

    Splunk Aims to Trigger DevOps Processes

    Splunk this week at its .conf18 conference continued a campaign to extend the reach of its operational analytics platform into the realm of DevOps through a partnership with Puppet.

    Spunk also announced the 4.0 release of Splunk IT Service Intelligence (ITSI), which extends a platform for managing IT using machine learning algorithms by integrating it with incident management software Splunk gained via its acquisition of VictorOps. Other new capabilities include support for a set of key performance indicators (KPIs) for tracking metrics such as application workloads and the ability to do root cause analysis on any event the machine learning algorithms predict.

    Andi Mann, chief technology advocate for Splunk, said the alliance with Puppet furthers Splunk’s ambitions to be a primary source of data used to inform automation frameworks that are at the heart of any set of DevOps processes. Over time, Splunk will continue to expand its DevOps partnerships with an eye toward being the analytics platform used to trigger automated processes using frameworks developed by Puppet and others, he said.

    At the same time, Splunk continues to develop orchestration capabilities of its own based on technologies it gained via its acquisition of Phantom. While that orchestration software primarily is used today in a security management application, Mann said Splunk is now looking at applying that software more broadly, as it encouraged developers to build applications using a Splunk Developer Cloud now in beta.

    Mann said that when it comes to machine learning in IT operations, otherwise known as AIOps, the most challenging task ahead is establishing trust. IT professionals are not going to sanction a “black box” approach that doesn’t enable them to see how a machine learning framework is determining what to automate. Because of that issue, Mann said, providing a root cause analysis capability in ITSI is critical.

    The next big issue facing organizations, he noted, is improving the quality of the data being used to inform a machine learning framework. One of the benefits of the DevOps movement is that, as developers take responsibility for both building and running applications, more of them are paying attention to how log data is being created. Mann said the notes developers leave within an application on how to resolve issues are becoming more detailed, a process that Splunk refers to as “semantic logging.”

    Mann noted the goal is to create a closed-loop system where every IT issue eventually creates a ticket in a DevOps system for developers to address. The bridge between those processes will be the chat capabilities provided by the VictorOps incident management platform. In fact, Mann said Splunk is moving from being a provider of log analytics software to being a provider of a continuous monitoring solution that analyzes data both stored in logs and in near real-time using a Splunk Data Stream Processor, which is currently in beta.

    It may take a while for that loop to be completely closed. But it’s clear that many of the tools that IT operations teams rely on today are becoming increasingly DevOps-friendly.

    — Mike Vizard

  • DevOps India Summit Sets the Stage for DevOps’ Future in India

    DevOps India Summit Sets the Stage for DevOps’ Future in India

    DevOps India Summit 2018, India’s biggest DevOps conference, concluded with notable audiences and best in industry experts setting the tone for the future of DevOps in India. The summit was held Sept. 1 at the Marriott Hotel, Whitefield, Bengaluru with a pre-event workshop Aug. 31. Around 500 professionals, including corporate leaders and mid-level managers from across the country, who witnessed the best minds in DevOps sharing their valuable insights on all phases of DevOps transformation. The event was designed to create a platform where practitioners of DevOps in India can hear the stalwarts share their experience and share their own journey and experience through calls for papers.

    The world-class summit, organized by Xellentro in partnership with DevOps Institute, Crevise Technology, Confinity Consulting, GitLab, Datamato, Sumologic, Zoomcar, MongoDB, TechGig, The Entrepreneur and PEOPLECERT, focused on the future of DevOps in India. The attendees got an opportunity to connect with the industry pioneers and learned about some of the emerging DevOps practices they can apply in their own organization. Experts from the United States, France, Brazil and India gave a sneak preview of DevOps adoption and how those initiatives and technical practices can help organizations attain the highest levels of performance.

    The topics were well-chosen and balanced, starting from the keynote by John Willis on DevSecOps to Marc Hornbeek speaking about the Continuous Delivery Architecture, Aymen El Amri speaking about Cloud Native and Amanda Mattos sharing her DevOps journey in the organizations she has worked. We also had sessions on Container-based CI Solutions by Archana Joshi, director of Cognizant Technology Solutions; DevOps Assessment Framework by Dr. Shome from TCS; High-Speed DevOps from Zoomcar, the Importance and Use of Database by MongoDB; Gaming in DevOps by Wipro; Scaling DevOps by Padma; DevOps in Automotive Engineering by KPIT; and many more. There were 30 sessions and six lightning talks, all of which were received with great enthusiasm.

    There was a panel discussion with 10 speakers and various topics were discussed from various areas of DevOps including DevSecOps and the need to include DevOps in university curriculum.

    The participants expressed their happiness to be able to hear such varied topics on DevOps and the great learning they had in the event and expressed their desire to attend in future years.

    During the event, Xellentro also awarded the highest Tweeter with Amazon vouchers.

    The pre-event had workshops by Marc Hornbeek on Architecting Continuous Delivery and was attended by a large audience who found this session extremely useful.

    Niladri conducted the G2G3 DevOps Simulation and the participants were intensely involved in this great immersive experience, which taught them the basic principles of DevOps—a great way to start the DevOps transformation and helps in making the participants feel the difference practicing the principles.

    GitLab products were also shown by their representatives.

    “It was a great feeling to attend this summit, as we get to meet new people who have been in this industry for a while,” said John Willis, president, DevOps and Digital Practices at SJ Technologies. I have come across people who are smart, aggressive learners and are eager to know more that they put up questions that was never asked before, it took me a time to navigate to an answer.”

    Marc Hornbeek commented that the event had the feel of DOES Summit but at a more manageable scale, where speakers could mingle with the delegates and take part in more knowledge-sharing among each other.

    At the end of the event, prizes were awarded on behalf of Xellentro. PEOPLECERT announced awards for five participants who were given free examination vouchers for training through Xellentro. DevOps Institute also announced prize to the highest scorer in DevOps Practitioner certifications conducted through Xellentro.

    Thanking all the partners and participants at the event, Niladri Choudhuri, who conceptualized the event said: “It is very encouraging to see people coming together and working as team to promote the real essence of DevOps. I thank all the participants who brought their enthusiasm along and asked so many relevant questions. This is the first ever DevOps event in India in such large scale with prime focus on the core content and quality of speakers who bring value to organisation. This is a start to more such future initiatives and I thank everyone for their support.”

    DevOps India Summit is starting a community of DevOps practitioners to create a platform to share, learn and enjoy the DevOps journey and help each other in the journey. People interested, other than the delegates who are already members of this community, can join this community here.

    — Niladri Choudhuri

  • DevOps Enterprise Summit: Las Vegas 2018: A Sneak Peek

    DevOps Enterprise Summit: Las Vegas 2018: A Sneak Peek

    Wow! It’s hard to believe that just a few months ago, we were wrapping up two days of immersive learning at the DevOps Enterprise Summit London event, and now we are getting ready for our biggest DevOps Enterprise event ever — Las Vegas 2018!

    In this post, I describe the programming objectives for the 2018 DevOps Enterprise Las Vegas event—what our most important focus areas are, how it differs from previous years, and why certain elements remain the same. I shared this with our programming committee early in our planning process, so that we could scrutinize and critique it. As someone once famously noted, “To be able to think clearly, one usually needs to first write it clearly.”

    Our first objective is to have the 2018 Las Vegas event be even better than our 2018 London event, which in turn, I believe, was even better than our 2017 San Francisco event, which was the better than all of our previous events. Aiming to have the next conference to be better than the last is challenging, especially when you think that your last event was the best yet—but without a doubt, stating this as an explicit objectives makes it impossible just to coast.

    But, the good news is, there are plenty of challenges this community faces, and I have confidence that by helping the community overcome those challenges, the result is a better conference.  And I think you’re going to love what we have in store for you!

    (And with that in mind, I hope that when you see this year’s conference speaker lineup, you’ll agree that this will be the best DevOps Enterprise Summit yet — if you haven’t registered yet, please use the special code below to save $300 off your registration price!)

    Here are the other programming objectives for 2018, created with the fantastic Programming Committee:

    Our Programming Objectives:

    • Get great experience reports, and repeat experience reports
    • Elevate focus on Spanning Business/Technology Divide
    • Elevate focus on Next Generation Ops and Infrastructure
    • Ensure conference structure helps attendees achieve their goals
    • Get best subject matter experts from the domains we need

    Great Experience Reports (And Repeat Experience Reports)

    For the past four years, one of the hallmarks of DevOps Enterprise Summit is that we have many repeat speakers presenting their ongoing DevOps transformation experience reports. This has been an awesome way to document their journeys and progress over time—it is especially gratifying to see how their areas of responsibility keep growing. Repeat experience reports make up 15 percent of our programming.

    • Chad Avery, Program Manager, DevOps; Aimee Cardwell, VP, Consumer Product Development – American Express
    • Ross Clanton, Executive Director, Technology Modernization – Verizon
    • Pauly Comtois, VP DevOps – Hearst Business Media
    • Jim Grafmeyer, Director, IT Infrastructure; Jared Speno, Senior Technical Consultant – Nationwide
    • Chris Hill, Head of Systems Engineering – Jaguar Land Rover
    • Olivier Jacques, Principal, DevOps transformation; Joan Watson, Dir. DevSecOps – DXC Technology
    • Bryan Finster, Staff Software Engineer; Dana Finster, Sr. Software Engineer – Walmart

    One potential danger of having repeat speakers is the self-reinforcing loop and creating a self-contained echo chamber, merely repeating what we like hearing.  That’s why we continue to seek out new experience reports—new experience reports are 25 percent of our programming.

    In these new experience reports, we actively seek the largest and most recognized firms across every industry vertical, to give DevOps Enterprise champions more case studies and empirical evidence to make their case internally, and overcome objections.

    I’m delighted that over the years, we’ve had some of the world’s largest banks present, as well as large not-for-profits, government agencies, and military branches—organizations that are often views as risk-averse and conservative.  For these stories, we are interested in organizations at all stages of the journey.  And these stories are a source of fantastic and serendipitous discoveries, as well.

    I’m delighted that this year, we have new speakers:

    • Dan Barker, Chief Architect – National Association of Insurance Commissioners
    • Ann Corrao, Distinguished Engineer and Master Inventor – IBM
    • Scott Havens, Director of Software Engineering – Jet.com and Walmart Labs
    • Simmons Lough, IT Specialist – US Patent & Trademark
    • Keanen Wold, Manager, DevOps Transformation  – Delta Airlines

    Spanning the Business and Technology Divide

    Another goal is to help the community get business leadership onboard—and by that, I don’t mean business leadership silently tolerating DevOps.  Instead, what we are looking for are enthusiastic and grateful business partners, who want to tell the world how all their goals, dreams and aspirations were finally achieved by working together with their technology counterpart.  In London 2018, we had some fantastic talks along these lines (from Nike, Capital One, and Verizon Enterprise).

    In Las Vegas, we’ll be featuring even more talks like this, showing how technology leaders are partnering with not just business leaders, but also other areas of the business, such as product leadership, security/privacy, legal, project management and so forth.

    These unique types of experience reports will represent about 15% of our overall talks, and I’m super-excited about their stories—these are the most difficult talks to find, because they require a technology leader with a great working relationship their business counterpart, and that they’ve created visible successes.

    • Courtney Kissler, Vice President, Nike Digital Platform Engineering – Nike; Ann Bradley, Chief Privacy Officer and Global Counsel – Nike Direct
    • Aimee Bechtle, Senior Manager Advanced Engineering; Abbie Gray, Product Manager in Credit Card Channels Product & Platform – Capital One
    • Levi Geinert, Director of Engineering; Lucas Rettig, Principal Product Owner; Dan Cundiff, Principal Engineer – Target
    • Topo Pal, Senior Director & Sr. Engineering Fellow;  Jamie Specter, Counsel, IP & Technology Legal Group – Capital One
    • Jason Cox, Director, Systems Engineering; Deepti Mutnuru, Senior Project Manager – The Walt Disney Company
    • Alice Raia, Vice President, Digital Presence Technologies; Le Quach, Director, Strategic Design & Implementation – Kaiser Permanente

    Next Generation Ops and Infrastructure

    I’ve written before about how this programming objective came from Jason Cox at Disney (one of our programming committee members), who said to me every year, “Great conference for Dev, but it wasn’t as great for Ops.”

    In 2018, we finally did something about it: Starting in London 2018, we reserved 25 percent of talks for next generation Ops and Infrastructure concerns, and created a separate subcommittee to focus on this area.

    And wow, we were all very, very happy with the results—in fact, I think the elevation of this topic has fundamentally changed the feel of the conference (you can watch a full discussion about this track here). In this track, we really make an effort to help define and clarify what exactly the role of Ops leader is, all the technology and organizational changes there have been in how infrastructure is designed, delivered and operated.

    This year, we are dedicating about 25 percent of the programming to this topic, and we are preserving that proportion for the main stage presentations, as well. I love that these presentations are also a bit more technical, but there’s no doubt that it is extremely relevant for the technology leadership community. (After all, as Mark Schwartz famously observed, “Last I heard, this is still a technical field.”  Haha!)

    Examples of talks in this track include:

    • Jeffrey Snover, Technical Fellow and Chief Architect for Azure Storage & Cloud Edge – Microsoft
    • John Rzeszotarski, SVP, Director of Continuous Delivery and Feedback – KeyBank
    • Cornelia Davis, Senior Director of Technology – Pivotal
    • Damon Edwards, Co-founder and Chief Product Officer- Rundeck
    • Tom Limoncelli, Renowned Author and Speaker, Former Sys Admin for Google, currently SRE Manager – Stack Overflow
    • Shaun Norris, Global Head, Cloud Infrastructure Services – Standard Chartered Bank
    • Charity Majors, former Production Engineering Manager for Facebook, currently CEO, Engineer and Co-founder – Honeycomb.io.

    BONUS – if you haven’t seen the Next Gen Ops panel discussion from the June conference in London, it’s worth a watch here.

    Subject Experts from Areas We Draw Upon

    In our San Francisco 2017 conference, one of the most notable (and for me, memorable) moments was the Safety Culture and Lean Panel we had, featuring three luminaries in the field.  Dr. Sidney Dekker and Dr. Richard Cook (Safety Culture) and Dr. Steven Spear (Lean) and I discussed the areas of mutual reinforcing principles from those two fields, and discussed how lessons learned from those domains could be applied to the DevOps Enterprise community.

    This year, we are honored to have Dr. Christina Maslach speak on her lifetime of research on burnout. She is professor emerita of psychology at the University of California, Berkeley, and her area of research focus includes: burnout and job stress; and individuation and social influence, as well as the positive antithesis of burnout, work engagement, as a better framework for developing interventions.

    My mentor, Dr. Steven Spear, will be presenting as well.  He presented at DevOps Enterprise Summit in 2015, and was part of the panel I mentioned above.  He is currently Senior Lecturer at the MIT Sloan School of Management. Dr. Spear is a Senior Fellow for the Institute of Healthcare Improvement, and was a four-time recipient of the Shingo Prize for Research Excellence. He is the author of “The High Velocity Edge,” and has extended his research beyond manufacturing to R&D, workplace safety, health care, etc.

    (Stay tuned for some exciting news that we’ll be able to share about something else special we have planned!)

    Twenty percent of our programming is dedicated to these types of talks—Dr. Nicole Forsgren (founder and CEO of DevOps Research and Assessment) will be presenting the “2018 Accelerate State of DevOps Findings,” which was done with the help of Google Cloud. Rosalind Radcliffe (Distinguished Engineer, Chief Architect for DevOps for z Systems IBM) will be presenting on her work elevating the state of the practice for the mainframe ecosystem.  And many more!

    Networking and Community Activities

    When I look at the DevOps Enterprise community, it is a community of networkers. I see lasting collaborations and friendships that have lasted for years and the mark of a phenomenal community is one that is actively helping one another. With that desired outcome in mind, we have made some changes to our programming.

    • New – DevOps Confessions! This came about some advice from Dr. Richard Cook who made the observation that the stories presented on stage are the heroic triumph stories, but stories that come out over evening drinks are those about the struggles and failures and near-misses. To bridge that gap, we’re going to set aside 15 minutes every morning where I will read two anonymized stories of confessions from leaders in the DevOps community. Hear more about this project here.

    Furthermore, there will be an hour per day in the programming where there are no talks — instead, this is a time for networking and collaboration and information sharing, with various levels of structure. (Our observation was that formats such as Lean Coffee were extremely popular, but because they were scheduled at the same time as talks, people went to the talks instead because of the “Fear of Missing Out”.  So, now there are no talks to compete with!)

    • Ask the Speakers! If you have questions for our speakers, we’re creating dedicated times for you to meet with them and ask the questions that will dig deeper into what your interests are. We’ll also open up a Slack channel for every attendee to use and submit questions directly to speakers and to other attendees.
    • Lean Coffee Sessions! We want to help attendees connect with and learn from their peers at the conference in another semi-structured format as well. We’re going to produce Lean Coffee sessions, led by Dominica DeGrandis, both days of the conference. This will be comprised of 10 tables big enough for 10 people to talk about similar interests that are relevant to the theme at the table. Hopefully, this provides another opportunity to learn a lot and make some valuable new connections.
    • Birds of a Feather Sessions!  These are more like the DevOpsDays unconference sessions, with less structure than a Lean Coffee format, driven by specific topics that attendees want to discuss and learn about.

    We also will have…

    • Slack Workspace! We will have our very own Slack workspace for the DevOps Enterprise Summit! You’ll be able to join the channel to take part in the conversation, as well as a “Who’s Hiring?” channel to learn about potential opportunities in the space. These channels will stay open after the conference ends to keep the dialogue rolling!
    • Lunch time! There will be a few different lunch options that attendees can take advantage of as well:
      •  Eat lunch with a speaker will be a first come, first serve basis. During these sessions, you will have opportunities to sit down face-to-face with a speaker and again, dive deeper into the subject matter of their presentation and experiences in a more personal setting.
      • Have lunch with fellow attendees at high top tables around the buffet stations where there are pre-written note cards to serve as seed questions for discussion.
    • Industry party! Every good conference comes with a good party, and DOES18 Las Vegas will be no different! Thanks to our friends at Delphix, the industry party is happening on Monday evening following Day 1. We’ll have sponsor giveaways, food, drinks, and another great opportunity to continue conversations with fellow attendees and speakers. I’ve found this time proves to nurture some of the best networking and listening to some amazing stories from in the trenches!

    This year at DOES18 Las Vegas, we’ll also be having Lightning Talks and the IT Revolution Author Book Signing! These will be on Day 2, following closing remarks. There will be refreshments and happy hour snacks as well!

      • Lightning Talks! Presented by our friends at Sonatype, these talks happen from 6:30-7:30 p.m. in Chelsea. These are rapid fire, must-see presentations – just watch the Lightning Talks from 2017 to see what I mean. More information forthcoming about these talks soon and if you’re attending, be on the lookout for the call for presentations as well!
      • IT Revolution Author Book Signing! From 7:15-8 p.m. in the Chelsea Balcony. We are excited to debut the latest title from IT Revolution Press and author Dr. Mik Kersten, “Project to Product,” at the IT Revolution book signing on Tuesday night! I believe this will be a very important book for our generation of technology leaders—the majority of economic value will be created when the largest brands across every industry vertical adopts the better ways of working pioneered by the FANGs (Facebook, Amazon, Netflix, Google), of which the DevOps Enterprise community is leading the charge. Mik’s book describes how project management and other orthodoxies need to be replaced with product-oriented cultures and management paradigms.

    As you can see, there’s a lot that is getting me excited for the conference this year! We hope we will see you at what is sure to be the best DevOps Enterprise Summit yet! Register now (and book your room at the Cosmopolitan) to ensure your spot at DOES 18 Las Vegas!

    — Gene Kim

  • We Still Haven’t Solved the Logging Problem

    We Still Haven’t Solved the Logging Problem

    At a company I previously worked at, I helped build out the initial infrastructure to support a large cloud security product. It was one of the first times I had the opportunity to build something from nearly the very beginning. Like many operations professionals, so much of my career prior to this experience had been spent cleaning up other people’s messes and technical debt that I was finally excited to “do things right” from the onset.

    Looking back, I wish I knew that building our monitoring platform would be hampered without the ability to retain and analyze both warm and cold log data. Here’s why.

    Time Series to the DevOps Rescue 

    Hindsight is almost always 20/20, and today I can proudly say that my team and I built a killer app monitoring platform. My goal was to make it easy for Dev and Ops folks to report time series metrics for their applications and build dashboards and alerts for app health. And we did just that.

    While many ops colleagues would say that monitoring journeys usually start by capturing application logs with open source tools such as Logstash/Elasticsearch, Graylog2, or even buying commercial services from Splunk or Sumo Logic, not all journeys start that way. My team and I took a different approach and optimized for time series metrics versus the more traditional structured log management approach, leveraging newer toolsets such as Librato for real-time operations monitoring and performance analytics. Librato enabled us to scale much faster than with traditional log solutions and gave us almost instant visibility across our containers, cloud servers and the apps themselves.

    Logging is Too Expensive 

    Using hosted services such as Librato to manage our time series metrics worked fine for a while. However, the cost of SaaS monitoring eventually grew, and, since we had the time and energy to bring metrics in house, we deployed Graphite, the open source enterprise-grade monitoring tool. Graphite allowed us to continue enabling our engineers with affordable access to granular time series metrics to help them manage their applications.

    While this allowed the business to respond quickly when an issue arose, time series metrics were only part of the solution. They allowed us to see what was happening, but not always why it happened. While time series metrics and even some of the more advanced tracing technologies including Jaeger and Opentracing are fantastic to help debug complex distributed systems, they fall short when investigating and analyzing causes for problems, issues and hacks.

    Digging Into Your Log History Gets Expensive, Quick

    Tools such as the ELK stack have been booming, with Elastic alone hitting more than 100 million downloads last year, which shows that DevOps teams want more from traditional log management solutions. They’re looking for simplicity, ease of use and affordability, among others. With JSON effectively the de facto standard for logging, most teams take log data with its predefined schema and then index it into Elasticsearch/Lucene. Unfortunately, the Lucene indexing mechanism results in an unintended consequence: a 5 to 10X increase in disk usage. And just like that, ops cloud budgets get blown away when they realize that 10GB of JSON data indexed into Elasticsearch could be 50GB of disk used. You wanted that index to be replicated for disaster recovery as well? Then add 100GB of storage for every 10GB of logs.

    DevOps Normal: Still the Tough Choice Between Retention and Cost

    Unfortunately, most people choose to simply purge data from our “hot” Elasticsearch clusters as soon as we possibly can to minimize cost issues. This is far from ideal.

    Companies moving to microservices-based architectures, deployed on containers such as Kubernetes, are going to be in for a rude awakening when they start building out their centralized logging platforms to keep up with the volume. Security teams are going to feel the burden as well, as they come under increasing pressure to keep data around for long periods of time due to compliance and audit reasons. When the next big web-facing vulnerability comes out, the ability to go back weeks or months will be key to see what IP address accessed a potentially malicious endpoint. With that data available, we could then take all those IPs and correlate them with other access logs.

    Is centralized logging any better than it was 10 years ago? Or has all this data generated exploded so much that the tools we have today are just barely keeping up? One thing is certain, this problem is far from solved, and many companies may find themselves drowning in their own data lakes.

    — Pete Cheslock

  • DevOps & DataOps: Catalysts for Organizational Transformation

    DevOps & DataOps: Catalysts for Organizational Transformation

    Over the past few decades, use of the internet has increased exponentially. Nowadays, the need to build quality web applications and managing its huge data effectively is a huge concern of any organization. Thus, there has been continuous exploration for finding better methods of software development and data management.

    To meet the need of today’s internet users, many technology companies have stepped up and are continuously utilizing their resources to find more efficient ways of developing web applications in less time and with better quality, and are finding more effective ways to manage and analyze its data.

    To achieve the same in application development and data pipelining management, many big organizations have started following a set of practices—DevOps and DataOps respectively.

    In this article, we’ll take a look at what DevOps and DataOps are, and then we will focus on the ultimate question: Are they really worth to implement in an organization, and will following the same set of practices really improve the application development and data management—and consequently the growth—of an organization?

    DevOps and DataOps: An Introduction

    In simple words, DevOps and DataOps are a set of practices for development of software and management of data respectively, for a particular project or to follow at an organization level.

    DevOps emphasizes on automation of processes of various teams—software development team, quality assurance team and IT operational team—so as they can complete the relevant tasks (i.e. plan, code, build, test, release, deploy, operate and monitor) of application development quickly. Also, the these teams will be able to maintain the higher quality of software application by delivering/publishing software in smaller parts with higher deployment frequency while keeping the cost low, thus following the agile methodology more seamlessly.

    DataOps, in comparison, can be defined as DevOps with data analytics.

    As mentioned earlier, we know that DevOps emphasizes the collaboration of software development team (including testers) with system administrators (operational team). In a similar manner, DataOps weighs on collaborating the teams of data engineers, data scientists and other data professionals who are handling the data-related processes (i.e. store data, analyze data, deriving decisions from data, archive data or deliver data).

    The concept emphasizes both communication and collaboration between various teams of the same organization. The combined team can have more visibility of the project and come up with faster results. It also will improve the morale of the organization by proving the fact that people will be always above processes and tools when it comes to management of projects.

    DevOps DataOps
    Automation and Monitoring of Software Development Automation and Monitoring of Data Pipelining/Management
    Combination of

    –       Software Development

    –       Quality Assurance

    –       Technology Operations

    		 Combination of

    –       Data Engineering

    –       Data Integration

    –       Data Quality

    –       Data Security/Privacy
    Collaboration of

    –       Software Engineers (Developers)

    –       System Administrators (Operational Team)

    –       Testers

    		 Collaboration of

    –       Data Engineers

    –       Data Scientists

    –       Data Analysts

    –       Data Professionals
    Typically aims at

    –       Shorter development cycles

    –       Increased deployment frequency

    –       Quickly moving from Dev –  to/from Test – to Production

    –       More dependable releases

    		 Typically aims at

    –       Automation of Storage Workflows

    –       Monitoring the Operational Flow

    –       Optimization of Data Analytics

    –       Creating Infrastructure that will enforce data quality rules

    Now that we’ve gone through the basic details regarding DevOps and DataOps and what they are doing for software development and data management respectively, let’s take a look at some of their potential benefits to see if they can really act as a stimulator for an organizational transformation.

    Benefits of DevOps and DataOps to adapting Organization

    There are numerous benefits of adapting the defined processes of DevOps and DataOps in project execution and running the organization. Here are some of the key benefits for applying the practices of DevOps for applications development in an organization.

    Benefits of DevOps

    • Transparent Processes and Culture across Organization – By continuous communication and collaboration between the development team, QA team and IT operations team for every deployment cycle, each employee will be updated regarding latest information of project and will be more aware regarding the processes of other departments.
    • Quicker and Higher Quality Deployment Cycles – By applying the set of practices of DevOps, especially the shorter development cycle, will boost the frequency of deployment. As the focus of teams will be limited—i.e. on small task/module—for a deployment cycle, quality of each deployment will improve.
    • Faster Resolution of Bugs – As tasks are smaller and continuously deployed on relevant servers, the development team will be given feedback very quickly from the QA team, thus enabling the development team to provide much quicker resolution of any non-working functionality. Application stability will be a byproduct of this process, as the reason behind any error can be identified very easily (because particular deployment cycle of an error is known).
    • Flexibility to Handle Unplanned Work – As agile methodology insists on the handling of the raised change request, teams streamlined with DevOps will be able to handle such unplanned work more efficiently.

    Benefits of DataOps

    • Faster Processing of Data – As various teams are getting collaborated, communicating continuously and working on micro databases rather than all at a time, processing of data at various process stages are done much quicker.

    • Quicker Solution of Problems – Similarly to DevOps, in DataOps data are being processed in bunches rather than at once. This approach can help to identify issues pertaining to a bunch of data more quickly, and their resolution can be derived more quickly, as the focus is limited.
    • Better Data Security – As the data security professionals are in continuous touch with data integrators and data analysts, securing data at each process level becomes easier.
    • Improved Resource Output – When the team adheres to the fundamentals of DataOps, seamless communication and transparency between the various teams will aid in the performance of each team member and will lead to better resource output.

    Implementation Procedure of DevOps and DataOps

    Defining the processes for implementing DevOps and DataOps is simple, but implementing same processes do require tremendous dedication of teams. Let’s focus first on the general steps for implementing DevOps processes:

    • Collaboration, communication/seamless integration between formerly siloed teams, i.e. software development team, quality assurance team and IT operations team
    • Breaking down the major modules/big development tasks in particular smaller tasks
    • Processes for DevOps should be followed for each deployment cycle/each smaller tasks
    • Tools – There are various tools available in the market to accomplish processes of DevOps, and there are some techniques for continuous development and performing agile operations‚ such as Kanban, Scrum, etc., which should be followed for seamless application deployment.

    With DataOps, there are no specific software tools available in the market to implement DevOps. But there are some frameworks and its supporting tools that can assist the DataOps approach for agile-based project management.

    • Data storage and analytics infrastructure should be redefined, i.e. cluster-based or redundant data storage approach is recommended. Also some latest data analytics techniques should be adopted.
    • Use of microservices-based architecture and software that can allow blending of various type of data, i.e. structured and unstructured, are also recommended
    • Implement ‘built for purpose’ database engines – By doing such, accessing and analyzing large quantities of data become much easier (these are the same type of database engines that drive Google Search, YouTube etc.)
    • Supporting tools – Tools that aid to agility of project and collaboration between teams i.e. ETL/ELT tools, data curation and cataloging tools and log analysers should be used. For data blending, software tools such as Map Reduce, HDFS, Kafka, Hive and Spark can be used

    Conclusion

    All IT organizations have been managing software application development and its relevant data for a long time. But right now, we are in an era where managing the same with velocity and quality can make your organization stand above the rest of peer companies. Adopting DevOps and DataOps demand sheer dedication and focus of the teams, but the output after implementation will be very positive and such followed standards will definitely transform the organization constructively.

    — Vikash Kumar

  • 5 Tips for Creating Blockchain IOS and Android Apps

    5 Tips for Creating Blockchain IOS and Android Apps

    Blockchain development and blockchain technology are playing a major role in the creation of various iOS and Android apps. Blockchain technology has seized the attention of many and when it is utilized properly, mobile app developers are able to achieve results that few thought possible.

    There are no shortage of benefits provided by blockchain development. Data security levels are increased and unnecessary intermediates are removed from the process. Financial organizations have been taking an increased level of interest in blockchain technology in recent years and this creates added opportunities for app developers.

    So, how can app developers properly capitalize on the opportunities that are created by blockchain technology and blockchain development? This guide is here to help. Here are some tips and pointers that are designed to help developers create iOS and Android apps that take advantage of blockchain technology and blockchain development.

    Focus on the Interface

    Blockchain development can be used to create all sorts of valuable iOS and Android apps, but for these apps to reach their full potential, developers must remain focused on the interface. Not considering the user’s experience during the development process could lessen the level of impact of the app.

    Is the front-end programming language operational? Is the blockchain application that is necessary for operating this language included? Has the developer considered the importance of an external database? Is the right application control being included during the development process?

    These are all questions that need to be answered so the app developer can create a quality design for the user interface. Identifying the proper analytics is also crucial. Otherwise, it becomes more difficult for developers to identify the right system. Building an admin console is a key step of the process as well.

    Take a Closer Look at the Importance of Architecture

    If a blockchain app is being created to be used on an iPhone or Android device, developers must take a closer look at the design architecture so that they can avoid common mistakes. Unfortunately, many aspects aren’t always considered, and they often involve additional processes that can add extra time to the process.

    For example, those who are planning on creating a hybrid app will need to obtain certain permissions first. The more developers know about the necessary parameters that have been established, the better their chances of creating an app that will be useful to those who rely on blockchain technology daily.

    The processors, size, memory and operational systems must be configured properly, and these configurations all fall under the architectural umbrella. Great design is crucial, and app developers who aren’t willing to consider the most vital components will come up short when it comes to their design architecture.

    Proper Platform Utilization

    An app that relies on blockchain development and technology needs the proper platform to be successful. This platform should be unique yet accessible. When issues come to the forefront (and they are going to, despite your best efforts), developers’ ability to solve these problems will be determined by the chosen platform.

    For issues to be solved, a consensus must be reached, which is easier when the correct platform has been selected. Now that more platforms are openly sourced, this makes life much easier—developers have the chance to select the platform that best aligns with all of their needs and desires.

    Whether a developer decides to choose Ethereum, Quorum or a different platform entirely, it behooves them to take the time to study the advantages and disadvantages of each, especially when blockchain technology is being utilized.

    Recognizing the Importance of Consensus Mechanisms

    Is the application being decentralized? This is one of the main questions that must be answered before this process can be completed. An iOS or Android app that is reliant on blockchain needs a consensus mechanism to work in the proper manner once it has launched.

    Without decentralization and a consensus mechanism, many of the typical problems and issues that arise are not as easy to solve. The system used to connect and provide a link needs a proper consensus. Without a consensus mechanism, the system’s ability to carry out all of the necessary tasks is severely compromised.

    Apps that are related to blockchain must be developed with the right approach. Are the necessary considerations being made? Is the best method being chosen for delivering proofs? Are the basic requirements being met when these proofs are delivered? A developer looking to take advantage of the current climate needs to be able to answer all of these questions.

    Identification of Objectives

    Last but certainly not least, developers must identify their objectives. What is the goal they are trying to accomplish by creating this app? What role is blockchain technology going to play? Are all of the necessary blockchain development principles being adhered to? Is the issue that is going to be solved being made perfectly clear?

    Answering these questions in a timely manner allows a developer to decide on the correct course of action. The same rules that apply during the creation of any other app still come into play when blockchain apps are being developed for iOS and Android users. The problem that is being solved by the creation of the app must be clearly defined.

    Are the issues being caused by data loss? What resources are being provided that were not previously available? To get the most out of the development process, the issues and the associated goals must be examined closely.

    — Harnil Oza

  • Fusing iPaaS with DevOps: Synergistic Potential of Continuous Zero Code Integration

    Fusing iPaaS with DevOps: Synergistic Potential of Continuous Zero Code Integration

    DevOps and integration platform as a service (iPaaS) are two dominant trends that together have cutting-edge use cases. Zero code integration could bring significant improvements to the DevOps life cycle, enabling efficient movement of applications from development to production. Implemented in the right way, both technologies can rewrite the way applications are developed.

    DevOps is a Paradigm Shift 

    DevOps is a development approach based on continuous development and continuous integration to unify development and operations. The power of this approach has enabled behemoths such as Amazon and Netflix to leapfrog other players in their domains and set up an entirely new industry.

    DevOps enables enterprises to manage the complete life cycle of applications from development to production in a faster and efficient way. In this way, teams can significantly accelerate the speed of application development life cycle.

    In traditional ways of software development, IT and operations teams work in silos. To develop something, development needs to build codes, package everything and hand them over to a release manager. In the next stage, the release manager delivers the developed code to a test environment, where QA teams test the code and report the errors back to the development team. Developers again make changes and process them to the test environment. Then, again, the project is handed over to the development environment. This process entails a lot of handovers across organizational departments. Teams show resistance to changes coming down the road. Red-tape administration and the overhead involved in these activities slow down the development process.

    All these hassles can be circumvented with the help of DevOps. In DevOps methodology, the development and testing teams work as one single team for the development of assets. The combined Development and Operations team achieves a balance to ensure that no efficiency is lost during the development process. All aspects of development testing, unit testing, integration testing, acceptance testing and production happen in a single environment. Small iterations are used to deliver usable products at the end of each sprint.

    iPaaS for Transforming Monolithic Applications 

    iPaaS is an ingenious innovation for breaking down silos and ensuring operational efficiency and agility. It is a modern-day approach to automating integration between applications and marshaling them in an order. iPaas platforms bundle connectors, maps, and transformations to enable zero code integrations that are thrice more durable and efficient than conventional integrations. This unique advantage enables business teams to build integrations and lets IT teams to assume the governance role.

    iPaaS solves a critical IT problem holding back organizations from leveraging their digital technologies: IT complexity. This problem was not present 15 or 20 years back, when enterprises had only three or four business systems. Enterprises used developer tools to combine applications and move data between source and target systems. They hand-coded connections for electronic data interchange. However, this method is not ideal for a scenario in which thousands of cloud and on-premises applications run in parallel. A lot of coding is required to build these integrations between these applications. Developers don’t have visibility to what other team members are building, and they create independent “islands of automation” and silos. Another relevant problem is updating integrations that remain in a constant flux.

    All these challenges can be circumvented if integration becomes codeless and manageable by everyone involved. iPaaS provides custom development kits and pre-built accelerators to build ground-to-cloud integrations or cloud-to-ground integrations in a hybrid environment. Without writing a single line of code, teams can build integrations and establish a smoother symbiosis between differentiated applications. Zero code integration of iPaaS helps on four fronts: Integrate, Design, Monitor and Govern.

    Harnessing the Vantage Point: Continuous Zero Code Integration

    Real-time integration resides at the heart of both iPaaS platforms and DevOps methodologies. IPaaS can further strengthen the continuous integration muscle of DevOps. Teams can fuse iPaaS with DevOps for a transformative technological advancement called continuous zero code integration. This step can take continuous integration even one step closer to continuous integration.

    IT and business teams can break down IT silos down the road and turbocharge the development process. Start-to-end integration between DevOps tools for other developers can fast forward the process of code development and code testing. A single intelligent mesh enables developers to integrate codes of developers and run them on a common build. Automated reporting ensures that developers are not stepping on each other’s code. Teams are able to make code updates or deliver things right into production hundreds or thousands of times a day.

    Version control integration can be updated with the command center to provide fully automated provisioning. It further can be used to automate the entire standing. Development can be connected seamlessly with the version control system for a shared development environment in which shippable products can be developed. Individuals get a better look at their assets while working on them. This allows automation to run automation test units for frictionless development.

    A microservices-based DevOps architecture is the most interesting part of DevOps and iPaaS fusion. This model has the potential to accelerate productivity significantly. Teams can deliver solutions much more quickly and access many business benefits downstream. Shorter build and deploy cycles improve deployability. It delivers the scalability to incorporate service-specific security, replication and monitoring in the build process.  Fault-tolerant systems can be set up in which an error in one service affects only that respective system and not all services. Developers don’t have to use brute computational force for releasing new versions of a service. Loosely coupled microservices provide a greater degree of elasticity in the cloud. New frameworks, libraries, data sources and modular components support dynamic discovery and binding.

    Any type of DevOps model can be driven by an iPaaS and their fusion should not be perceived as an enigma. Surely, there can be many more use cases out there. Melding the two for power and positive force is worth a closer look. This fusion can transcend application development in ways that are hard to imagine right now.

    — Jitesh Banga