devopsy.kinsta.cloud

Tag: devops

  • Is DevOps Making Life Better for Data Scientists?

    Is DevOps Making Life Better for Data Scientists?

    The path to success and happiness of the data science team working with big data project is not always clear from the beginning. It depends on the maturity of the underlying platform, their cross skills and DevOps processes around their day-to-day operations.

    The past few years have been like a dream come true for those who work in analytics and big data. There is a new career path for platform engineers to learn Hadoop, Scala and Spark, while Java and Python programmers have a chance to move to the big data world. There they find higher salaries and new challenges and get to scale up to distributed systems.

    But recently, I have started hearing some complaints and dashed hopes from engineers who have spent time working there.

    Their expectations are not always met. Big data engineers spend most of their time in extract, transform and load (ETL), preparing data sets for analysts and coding models developed by data scientists into scripts. They have become cogs between data scientists and big data.

    I see the same feedback from data scientists. Real big data projects differ from Kaggle competitions. Even minor tasks require Python and Scala skills as well as multiple interactions and design sessions with big data engineers and operation teams.

    DevOps engineers get to go no further than just using the basic tools and running through tutorials. They grow weary of being given nothing more to do than deploying the next Hadoop cluster through the Cloudera wizard and semi-automated Ansible scripts.

    In this environment, the big data engineer will never get to work on scalability and storage optimization. Data scientists will never invent a new algorithm. And the DevOps engineer will never put the entire platform on autopilot. The risk is the team’s ideas might not make it beyond proof of concept.

    How do we change this situation? There are three key points:

    1. Tools evolution—the Apache Spark/Hadoop ecosystem is great. But it is not stable and user-friendly enough to just run and forget. Engineers and data scientists should contribute to existing open-source projects and create new tools to fill the gaps in day-to-day operations.
    2. Education and cross skills—when data scientists write code they need to think not just about abstractions but also consider the practical issues of what is possible and what is reasonable. For example, they need to think how long their query will run and whether the data they extract will fit into the storage mechanism they are using.
    3. Improve the process—DevOps might be a solution. Here DevOps does not just mean writing Ansible scripts and installing Jenkins—we need DevOps working in optimal fashion to reduce hand-off and invent new tools to give everyone self-service to make them as productive as possible.

    The Continuous Analytics Environment

    With continuous analytics and self-service, data scientists own the data project from the original idea all the way to production. The more autonomous they are, the more time they can dedicate to producing actual insights.

    Data scientists start from the original business idea and work through data exploration and data preparation. Then they move to model development. Next comes deployment and validating the environment. Finally, there is the push to production. Given the proper tools, they could run this complete iteration multiple times a day without relying on the big data engineer.

    Big Data engineers, in turn, work on scalability and storage optimization, developing and contributing to tools such as Spark, enabling streaming architectures and so on. They provide an API and a DSL for the data scientist.

    Product engineers should be given analytical models developed by data scientists as neatly bundled services. Then they can build smart applications for business users to allow them to use those in self-service.

    Working like this, no one is stuck waiting on the next person. Each person is working with their own abstractions.

    This might sound like a utopia. But I believe this could motivate big data teams to improve their existing big data platforms and processes, with the added benefit of making everyone happy.

    About the Author / Stepan Pushkarev

    Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO of Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He has been responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as hiring, establishing engineering culture and delivery process. Stepan combines strong technology, management skills and entrepreneurial spirit. Connect with him on LinkedIn.

  • The Emergence of UI and UX in DevOps Tools

    The Emergence of UI and UX in DevOps Tools

    Most of the popular DevOps tools on the market such as Puppet, Chef and Jenkins, are open-source and require a high level of proficiency. With the complexity comes the challenge of learning new frameworks and paradigms, a benefit to those who want to challenge themselves and learn new skills.

    But while these tools have become widely adopted, the widespread adoption of DevOps is creating a need for interfaces that cater to all users, from the most technical engineers to those who don’t want to learn, don’t see the value in learning or don’t need some of the top-end power that a complex scripting interface provides.

    It is for this reason that CloudBees developed Blue Ocean, a new interface for its continuous delivery tooling, and Red Hat released a user interface (UI) for Ansible.

    The Right Time for UI?

    Why have there not been UIs before now?

    I would say two reasons:

    1. Users did not demand nor care for one until now. A UI in many circles is seen to been more presciptive/restrictive of what can be achieved, and that only scripting/programming interfaces expose the real power to an engineer. Also, many of the most progressive engineers like scripting; they derive satisfaction from learning new skills, especially those that are the most highly desirable/in demand. And, I think many engineers like things because they are difficult to do; it’s a challenge, they have bragging rights and means they likely are getting paid more/are in demand.
    2. The open-source community is not as good at or geared toward building UIs.

    Why has a UI become important now?

    At MidVision, like many of the other commercial automation tool vendors, we have always believed that user experience and accessibility is critical, so we have provided command-line, scripted, API and GUI interfaces to our tooling. But this has not been the common approach.

    What makes this moment different is the wider adoption of DevOps automation tooling by groups that want at least graphical user interfaces (GUIs) and improvements to user experience such that you don’t need to be super-technical to be able to use them.

    The main groups we see are:

    • Business users within an organization. Continuous delivery/release automation tooling very much promotes collaboration, interaction and self-service, allowing release managers to see the status or pipelines, approve releases or initiate deployments. Managers may want to see dashboards and metric, produce reports or usage data. Other less technical staff, offshore teams, second level support, etc., may want to be able to provision, clone environments or deploy changes.
      There is a gap between the nerds in the basement speaking one language – and the business users who speak another which accessibility, UI and UX hopes to bridge.
    • Operations. There are certain parts of IT Operations that have welcomed the technical script-based tooling with open arms—it was a natural evolution for a UNIX system admin to become a DevOps/automation engineer. They can still script in shell (or are they meant to be doing it all in Python? ?). Again, some of the security tooling, storage, etc., are generally more comfortable down in the weeds. As you move up the stack however, toward the business application deployments, users are much more familiar and comfortable with UIs. Also, at this level dashboard, metrics, access control and reporting are much more important—and all of which are very hard to achieve without a proper UI.
    • Wider industry adoption. Adoption of the tools and techniques that are often advocated in the DevOps world started with the same poster child companies—Netflix, Twitter, Facebook, Google—and other newer companies that are cloud-native, have an open-source-first policy and are unencumbered by heritage technology stacks.

    We have then seen parts of enterprises adopt DevOps practices and tooling. Obviously, it is important to start small, establish value and then scale—running an initiative like you would a startup within an enterprise works well.

    Now that there is a compelling body of evidence to suggest that DevOps practices provide competitive advantage, cost savings, improved time to market, etc., other industries that are heavily dependent on technology have not had yet adopted these practices for varying reasons. Some parts of the finance and healthcare industries are two such areas. These industries have concerns around security and stability that may have slowed their adoption to date.

    Overall, however, this wider adoption means we need to cater not just for the new breed of DevOps engineers that are both system administrators and programmers and can script in Shell, Rudy and Python. We also need to cater to a wider group that either don’t have the skills to use these tools or don’t care for them.

    What change, then, are we likely to see?

    I believe DevOps tooling will evolve to provide new interfaces to cater to business users and self-service, and to make it easier to achieve the same thing without needing years of experience using a tool before being considered a competent user.

    I would compare many of the current tools today to a map and compass. Lovers of orienteering or mountaineering may suggest using a GPS or map on a mobile phone is cheating, cannot be relied upon or won’t provide the same same information as the contours on a 1 to 10,000 map. Or maybe you prefer vinyl to Spotify.

    Neither of these things are wrong; they are just not the only right. If you want to nerd out with your map and compass, vinyl LPs and Python scripts, I say fill your boots. All of which, BTW, I have some passion for! However, if you also want to get in your car, enter a zip code and know that it will get you there (accepting there is maybe a route that is slightly quicker if only you had done your research more thoroughly), this makes perfect sense to me.

    User experience is at the heart of the most of the other products we choose to consume, and I think many new and existing users will be pleased with UX becoming more important as tools like ours and others put this at the forefront of their design.

    About the Author / Dave Sayers

    Dave Sayers is a co-founder and technical services director of MidVision, a provider of Release Automation and Continuous Delivery Software. He works closely with customers and partners on DevOps and Automation initiaitves throughout Europe and North America and is responsible for devising and managing the partner and channel strategy. Connect with him on LinkedIn.

  • Improving DevOps Success with Well-Defined Metrics

    Improving DevOps Success with Well-Defined Metrics

    Collaboration within diverse project teams is often more difficult to achieve when there is no clearly visible aim, the planning is haphazard and the information shared is inadequate. However, these are crucial hurdles to overcome in any organization—and more so in product companies with globally distributed teams working on different parts of the product pipeline.

    Successful product development requires effective and constant communication between various teams through well-defined metrics, not only to reduce the engineering operations overhead but also to improve on the overall engineering deliverables.  These metrics, including  ROI or profitability, mean time to recover and deployment frequency provide a clearer picture to an engineering or business owner on how the Dev and Ops teams are working, their adherence to processes and guidelines and the technology they’re using.

    Every engineering owner likes to know if the technology investment they make is going to yield a significant ROI. It is no different with DevOps. It is important to identify the exact metrics by which the positive impact of DevOps can be measured.  The most effective way to illustrate these metrics in a way that would make sense to senior management is revenue impact.  Here are a few of them which have had proven success on internet of things (IoT) projects.

    Business metrics such as cycle time impacts incremental sales and customer satisfaction. Whereas other business metrics such as product investment and running costs impact profitability, DevOps implementation effectiveness can be measured by both. Cycle time is, in turn, derived from a number of product features delivered, sprint velocity and release planning metrics. The lower the cycle time, the more features get pushed and impacts the go-to-market. It also results in potential incremental revenue in terms of the value brought by reducing the cycle time.

    Another dimension to measure DevOps impact is engineering metrics. Companies today are worried about the cost of downtime in the operational phase as much as they worry about time it takes to bring the product to market. DevOps metrics are valid throughout the end-to-end implementation of a project.

    Build automation and code quality analysis, for example, can provide significant savings in build cycle time and build rollout time. Automating deployment process improves time to deploy and sprint velocity. Continuous testing improves mean time to resolve, and CloudOps monitoring tools can help reduce mean time to recover and uptime of any application. Further collaboration in automation through ChatOps tools can help improve mean time to respond. Once set in motion, all these matrices start improving the product quality, which leads to less rework. Intelligent automation also incurs direct productivity benefits.

    Organizations now measure these codes to release the metrics at the end of the day to get maximum ROI. A lot of these metrics also depend on the DevOps capability maturity. Organizations at the higher end of the maturity spectrum would tend to measure all of these metrics and try to optimize them. Also, the emphasis of these metrics vary with the domain and stage of the product life cycle being worked on. Home automation companies developing their IoT platform, for example, would emphasize build and deployment metrics, whereas companies in industrial automation with large infrastructure deployment would look at CloudOps monitoring metrics more closely. For telematics companies, test automation metrics become important when there is limited time and bandwidth to capture the data.

    DevOps is a cultural transformation and most of the time businesses measure the effectiveness in a very obvious way. However, a change in the way the organization operates should be looked in such a way that the metrics can generate long-term value. Thus, continuous measurement of the DevOps practice should also become one of the major focus areas of DevOps teams.

    About the Author / Urvashi Babaria

    Urvashi Babaria is a Product manager at eInfochips working in new age areas like IoT, DevOps and CloudOps. She is a techno commercial marketer with nine years of experience in product and project management, business analysis and transformation. She has keen interest in areas of data science including visualization, analytics and modeling. You can start a conversation with her at urvashi.babaria@einfochips.com

  • Measuring What Matters in Evaluating Software Performance

    Measuring What Matters in Evaluating Software Performance

     When developing and deploying a new feature or application, performance (speed, reliability) must be closely analyzed. After all, there’s no benefit to delivering a product if it doesn’t work well or users are forced to stare at a spinning wheel. To get a true view of performance, many organizations rely on metrics—a smart move since metrics, much like the scale, do not lie. Right?

    Unfortunately, that’s not always the case. Even when metrics are used, there often are different interpretations of performance, as well as debate over which metrics should be used in the first place. Metrics can be biased and manipulated, but does that mean we should stop using them? Of course not. Instead, we must learn to weed out bias and metric manipulation.

    What is Bias?

    Bias is not a bad thing; everybody is biased to some degree. Bias is defined as a particular tendency, inclination, feeling or opinion, especially one that is preconceived.

    Two common biases are anchoring and confirmation bias. Anchoring is the tendency to rely on a single piece of information, usually the first piece of information we encounter. The statistics presented first will most likely be remembered and drive decisions, responses and reactions.

    Imagine you’re about to launch a new application, but before doing so you run some performance tests to see how fast the application loads. The initial tests show the load time is 10 seconds. That’s too high, so the team looks for ways to improve performance. A few weeks later response times drop to 6 seconds, a 40 percent improvement. That’s great, and everybody celebrates. But what if initial tests showed page load times of 4 seconds and a few months after launch times crept up 40 percent to 5.6 seconds. Would you still be celebrating? No, you would be looking at what went wrong and trying to fix it. The initial anchor influences whether you see a time as positive or negative. 5.6 seconds is better than 6 but the initial anchor makes 5.6 seem worse.

    Confirmation bias is the tendency to interpret and search for information that confirms a pre-existing hypothesis. If you believe a new feature does not add value to an application, you may be more likely to focus on negative statistics rather than positive ones. If testing shows application load times will increase 15 percent with the new feature, you may use that as proof to delay or stop the release of the feature. If, however, you think the feature is a real game-changer, you will find the research about the 20 percent rule, which says people perceive changes in speed when the difference is at least 20 percent faster or slower. A change in 15 percent won’t be perceived by users. At the core, confirmation bias is about ego and not wanting to be wrong.

    The first step to avoiding biases is acknowledging they exist and everybody—yes even you—is susceptible. But there are ways to avoid them, or at least minimize their impact. Create your own anchor. Do research to determine what is an acceptable response time and use that as the anchor point. Search for facts that support and oppose your viewpoint, and don’t stop with a single site or fact; find multiple sources. Foster an environment where disagreement is encouraged in a positive and constructive manner, and listen to the dissenting viewpoints. Ask better questions and be willing to accept the answers even if they aren’t what you were seeking.

    Avoiding Metric Manipulation

    Metrics should be actionable and tied to business goals as opposed to being purely “vanity metrics”—those metrics that may make everybody feel good, but don’t reflect the true user experience and the consequent impact on the business.

    Specific to measuring web performance, there are countless metrics that can be selected to determine whether a user experience is positive or negative. If you choose to measure the time to the onload event as the core metric, you can recode a page to defer items to after the onload event. The metric will go down, teams will celebrate, but did this have any impact on the user experience? Was the goal really to reduce onload time or was it to better satisfy end users through a faster site in their eyes, increasing conversions and time spent on site?

    Here is a quick checklist you can use to choose the right user performance metrics:

    • Do improvements or declines in this metric impact what really matters: the user perception and business performance? Can this correlation be clearly made? How will this metric show progress towards the businesses goals and objectives?
    • Can the metric be gamed or influenced? Once something is being measured, people may look for ways to game the system and show improvement when in actuality there is none.
    • Is the metric “good enough”—i.e., the best one available, although it may not be perfect? Though not a performance-related example, popular website Medium’s choice of top-line metric illustrates this approach. While most websites rely on unique site visitor numbers as the ultimate gauge for their success, Medium, which views high user engagement as their key success factor, instead measures the time users spend reading articles on the site. This may not reflect the totality of user engagement (visitors who come to the site, but may just scan, not read) but it’s good enough and a more realistic indicator than alternatives.

    Whether we like it or not, people are heavily guided by bias and emotion, and they often manipulate reality (even subconsciously) to support their own motivations. Performance metrics are and always will be central to evaluating software performance, but it’s not enough to just lay out the facts. Organizations must frame and present metrics appropriately and sufficiently, and focus on those that are the most relevant and reliable. These are the keys to delivering the most complete, impartial canvas of performance information, helping guide the best decisions in the interest of the business.

    About the Author / Dawn Parzych

    Dawn Parzych is Director of Product and Solution Marketing at Catchpoint. She enjoys researching, writing and speaking about trends related to application performance, user perception, and how they impact the digital experience. In 15+ year career, Dawn has held a wide variety of roles in the application performance space at Instart Logic, F5 Networks and Gomez. Connect with her on LinkedIn.

  • Why DevOps Success Depends on the Right Infrastructure

    Why DevOps Success Depends on the Right Infrastructure

    Do you remember building with LEGOs? You could follow the directions that come in the box down to the letter, or you could go off-script to create something totally your own and evolve and refine it over time. Regardless of your level of creativity, if you had one of those classic green base plates to build from, then LEGO cities were in your grasp.

    In many ways, a DevOps methodology aligns with the block-building process. While the tools and processes can change, a solid foundation is the backbone for fast and repeatable success. Developers and IT operations are beginning to reap the benefits: According to RightScale’s “2016 State of the Cloud report,” DevOps adoption increased to 74 percent in 2016 from 66 percent in 2015.

    But even though adoption is on the rise, some DevOps teams still struggle to extract the full range of benefits from a DevOps model. Why? Because they’re more concerned about the LEGOs without first having considered the base plate. DevOps requires a flexible and modular foundation, a software-defined infrastructure (SDI) built with open-source technologies to be successful. Here’s why.

    Infrastructure: The Path to Agile Business

    The goal of a DevOps approach is to speed the entire application delivery life cycle (initial development, testing, delivery to market), enabling the business to quickly respond to consumer demands (deliver new services, fix bugs, add new features, etc). However, for the DevOps teams to successfully execute this, they need a flexible and modular cloud infrastructure that easily adapts and extends to meet their needs.

    SDI provides easier access to resources via a self-service interface so developers can quickly get the resources they need.  It also gives the DevOps team the capability to standardize services so that applications built in the development environment will easily deploy and run in the test and production environments, speeding their delivery to market. SDI also offers a flexible and modular approach that provides timely and easy access to the latest technologies, such as containers, where innovation is occurring at a rapid pace and easy integration is available via standard open protocols.

    SDI provides the best LEGO base plate to support your DevOps processes that improve application lifecycle management.

    Automation to Support Strategic Collaboration

    Routine deployment and management tasks are the Achilles heel of every IT organization. The best way to improve efficiency, reduce costs and quickly respond to infrastructure demands is with automation.

    As a key component of an SDI, automation and orchestration of the application life cycle including deployment of hardened VM or container templates ensures faster, consistent and repeatable provisioning and configuration. Additionally, the SDI automation and monitoring tools can intelligently sense and respond to infrastructure demands, automatically optimizing and adapting the infrastructure in real time. This frees time for the team to focus on strategic initiatives versus backend tasks.

    The DevOps movement is all about collaboration and constant innovation, which is made possible by having your infrastructure automate the mundane tasks and automatically respond to infrastructure demands.

    Visibility Down to the Base Level

    One challenge facing successful DevOps adoption today is lack of visibility into application delivery. As developers rely on constant innovation through a loosely coupled chain of events, understanding how each link operates is crucial to keeping the chain moving in the right direction.

    In the case of open source, there is no infrastructure more transparent or collaborative. The infrastructure itself is built on openness and exchange of code, leading to a natural transition to its use in a developer environment. As DevOps uses CI/CD to stay refined, so do the open-source projects that contribute to its infrastructure. Constant innovation is what keeps the infrastructure and development tools up to date, supporting the entire DevOps team.

    The Solution to Cultural Challenges

    Successful DevOps adoption doesn’t solely involve changes to existing technology. It also requires a cultural shift within an organization. A software-defined infrastructure that utilizes open-source technologies is built on the same concept: growth through collaboration. Businesses aren’t committed to using a single vendor or propriety solution, giving them an expanded network of resources and tools to optimize their IT needs and business operations. DevOps is a marriage of technology and people, and an SDI environment allows that union to succeed.

    The world has seen many great LEGO projects over the years, both the small-scale private home projects and large-scale public exhibitions. But the common thread for their success is the base plate, allowing innovators everywhere to build LEGO masterpieces more rapidly and effectively. For DevOps professionals, a software-defined infrastructure built using open-source solutions is the base plate for success. SDI environments improve efficiency with automation, refine visibility into application performance, enable innovative and agile business models, and create stronger cultural alignment.

    About the Author / Terri Schlosser

    Terri Schlosser, head of SUSE product and solution marketing, has 20 years of experience in the IT software industry.  She has held roles in marketing, software development, product management and strategy in many different IT software areas including cloud, networking, storage, management and operating systems.  Most recently her focus has been on open-source software defined infrastructure solutions. Follow her on Twitter.

  • How IT is More Like an ATV Than a Sedan

    How IT is More Like an ATV Than a Sedan

    There is an analogy for IT being like a car and it often goes something like this:

    IT should run like a well-oiled machine. I don’t get in my car and think about how it works, I just work some controls and it does what I need it to and it goes where I need it to go. No assembly required!”

    When people think IT, it seems they’re hoping for their IT department to work like a Tesla luxury sedan that drives for you. They like to think of IT in this “it should just work” kind of mentality. The internal dialogue of some may be, “I have this goal, IT is this tool to get it done, and I have paid for it! It should just work by itself! Heaven knows, I have spent enough on it!” They want something that’s comfortable, doesn’t require much interaction and gets them where they need to be.

    The problem with the sedan metaphor is that sedans are not really what companies need.

    Sedans are built for paved roads, made to go to places that others often go. The average sedan can’t really explore much else other than what is well-paved and well-traveled by others. They’re a comfortable ride for drivers who want all the amenities of the home wherever they are. Anything too far off of the beaten path and the car is being torn to pieces by the rough terrain. Sedans perform consistently and reliably in consistent and reliable environments.

    This is not to say that IT should not be reliable. By all means, IT has to be reliable. If IT isn’t operating reliably, much like a broken car it should either be fixed or replaced. Why would anyone pay for something that can’t perform the tasks you need it to do?

    However, organizations know they have to take calculated risks, and the larger the risk, the larger the payout. This means going into unfamiliar territory and going down paths that are not mapped out. Even a Tesla is going to struggle over terrain that isn’t already mapped out.

    On top of that, we know customers and technologies are constantly changing. There is very little consistency from the world outside of the enterprise. Every day brings new fluctuations and challenges that companies need to anticipate and adjust to.

    So why do we expect IT to behave in a consistent way when the environment around us is constantly shifting? We can’t act as though our business can succeed riding around in a comfortable sedan! We need something that can be flexible and can still move us forward.

    Think ATV, Not Sedan

    The ATV fits this kind of unpredictability perfectly. ATVs can adjust to a variety of terrain. They don’t need paved roads nor beaten paths. They can go almost anywhere and can get there quickly. These machines are rugged; they can adjust to a variety of terrain and, most importantly, they can go anywhere—if the driver knows what they’re doing.

    But ATVs are not comfortable. An ATV takes knowledge to drive and it takes engagement. One can’t sit back disengaged and hope to have a smooth ride on an ATV that is going full speed. The places these machines are made to go are not for the complacent or for those seeking comfort.

    The business-IT relationship should look more like an autocross driver and his ATV than a suit in a Lexus. In this analogy, both machine and driver are adjusting in unison to the environment around them. The driver (the business) guiding the ATV (IT)  where to go and pushing through hills, mud and dirt to make sure both arrive safely and quickly despite the dangerous and uncertain environment around them. At the end, the driver and his machine will have pioneered places many feared to tread and will be rewarded appropriately.

    To extend the analogy, this is not a problem only between business and IT; this is the same attitude Dev often takes with Ops. Dev says, “Here’s a build, your environments should be good enough,” and Ops is expected to just make it work. Isn’t this the same disconnect as the IT and the business?

    It is so important to collaborate in a large enterprise. It takes investment of not just money but also time. If we have an agile team with no product owner, we essentially have a car with no driver. If we have Devs who don’t talk Ops, don’t we have a car body with no engine?

    In today’s world of Agile and DevOps, the companies that build that bridges between Devs, Ops and the business are the ones who take the calculated risks and succeed. These companies win bigger market shares and know confidently what their customers need. They can experiment fast with different business ideas, use the latest and the greatest technology and not comprise on security. These same companies can leverage their IT function to accurately monitor spending while minimizing waste.

    When businesses interact with IT, they shouldn’t just toss something at them like an external vendor and expect quality. Collaboration and engagement is the key. When all the necessary members in an organization are talking, communicating and engaged with one another, then we begin to see the true power of IT.

    The power of IT is the ability to go over terrain that would destroy other organizations. It is the ability to set any vision and know that your people will have the tools to tackle whatever comes their way. It is the ability to adapt, change and continuously improve while watching organizational goals become reality.

    About the Author / Caleb Wolfe

    Caleb is an IT professional with a passion for DevOps and transforming the way IT organizations work. As a member of Delta Air Lines, Caleb has a variety of experience from working as a baggage handler, business analyst, software developer and as a technical analyst in IT transformation. Caleb uses his multidiscipline experience to identify and remove constraints, create effective throughput and work to break down barriers both internally and externally. Caleb is a perpetual student of process improvement and IT technology and uses his knowledge for the betterment of Delta’s employees and customers worldwide.

  • Best Practices for Internal Communication in DevOps

    Best Practices for Internal Communication in DevOps

    DevOps has become a widely popular method of collaboration in the IT world. The combination of development, operations and quality assurance, DevOps requires every project have clear communication channels between development teams and individuals. In this article, we’re going to explore some of the best practices for internal communication within the DevOps structure.

    Software development is the game that runs on invention and communication. Therefore, communication methods and channels play a significant role in project success. Better communication between managers, engineers, developers and testers leads to better performance and more satisfying results. Each practice presented can help improve communication within teams and make your work more synchronized and efficient.

    Digital Communication Channels

    Scott W. Ambler notes in his communication diagram, e-mail communication is noted as more effective than any other documentation options, including paper memos, audio tapes and others. E-mail is the best way to share official documents and company memos, making it more suitable for external communication in the DevOps project. But while most individuals today are able to access their e-mail on their smartphones, and e-mail enables users to send large attachments and contact several team members at once, DevOps teams tend to rely on more interactive communication channels.

    Depending on the project needs and the preferences of team members, internal communication can be either through project collaboration apps, an intranet or even the usual communication apps or social media. Since Ambler’s diagram shows that video conversation is by far the most effective form of project communication (aside from face-to-face meetings), most project managers rely on apps with video-calling and videoconferencing functionality.

    Regular Meetings

    Regular meetings are the backbone of the DevOps communication. They can be organized in-person or through videoconferencing apps. Same as with the Agile Project Management structure, the DevOps team should meet at least twice a week. Preferably, the meetings should occur on a daily basis. The Monday or morning meeting can be used for assigning tasks to each team member and the Friday or end-of-day meeting can be used for assessing completed work and for future planning.

    Building Soft Skills

    Unfortunately, HR practices in many industries rely on stereotypes. One of the most prominent stereotypes about IT professionals is their lack of interpersonal skills. Although some engineers and developers fall into this generalization, their knowledge and interests go far beyond IT and technology topics. Still, even the most talkative and assertive members of the development team need to work on their soft skills to improve the team’s communication.

    Improved soft skills enable team members to collaborate more effectively and to develop strong working relationships. The first task of every manager is to find the soft skills already present in your team. This will give the team the chance to find the best opportunities for further development. After the initial skill appraisal, manager can enroll members in courses for assertive communication, negotiation and conflict-solving, depending on their needs and current capabilities.

    Importance of Time Management

    The whole DevOps paradigm strongly relies on effective time management. The implementation of the DevOps model often changes the whole development process and wipes out all the previous scheduling and time-management patterns. A good manager needs to motivate his or her team to change their long-held habits and pace their daily routines so they can keep up with the new organizational patterns.

    We already mentioned how regular meetings are essential for successful DevOps implementation. When organizing the meeting, schedule managers need to leave enough working time so each team member can effectively work on their tasks. Too many interruptions can break up the work process make it difficult for employees to focus. Effective time management relies on both team and individual needs and gives your team the flexibility to work as efficiently as possible.

    Use of Data Insights

    Internal communication on DevOps projects needs to be smart. It needs to be specific, measurable, attainable, realistic and timely, and one of the ways to achieve this is to use accumulated data from current and previous projects. The thorough analysis of accumulated data can give managers more details on:

    • The average number of working hours the team will need to release a new app, make various updates or make patches;
    • How to make the team’s communication more efficient;
    • Identifying the areas where the development strategy is losing its momentum; and
    • Understanding how each task influences the project delivery.

    Constant adaptation and changes in work processes is the base for successful DevOps implementation. This ideal can be achieved only through effective communication between team members. Communication also requires time, so the only way to ensure effective DevOps work is to combine strategic skills, process development, excellent time management and smart data insights.

    About the Author / Nate Vickrey

    Nate Vickrey is a business consultant and an online author. He is mostly engaged in finding the best solutions and latest trends regarding company management and culture. Nate is also the executive editor at one Bizzmark Blog. You can follow Nate’s work on his Twitter page.

  • The Time Is Now for an IT Operations Revolution

    The Time Is Now for an IT Operations Revolution

    There’s a level of frustration starting to build inside IT operations teams that’s about to boil over. Lucas Carlson, vice president of strategy for Automic, a unit of CA Technologies that provides an IT automation platform, says developers are bullying IT operations teams by programmatically invoking application programming interfaces (APIs) exposed by public cloud service providers. As the number of application workloads running in those clouds increases, IT operations professionals are losing control.

    To ensure their continued relevance, IT operations teams need to leverage automation to inject levels of agility into local data center environments, he says. That would make it much less likely developers would feel the need to programmatically invoke IT infrastructure resources.

    Carlson says achieving that level of agility will require IT operations teams to rely more on machine- and deep-learning algorithms to automate low-level tasks. Instead of thinking in terms of managing silos of compute, storage and networking resources, Carlson contends IT operations professionals need to reinvent themselves.

    The artificial intelligence (AI) models based on advanced algorithms must be trained continuously. IT operation professionals could become the trainers of those AI models. In fact, Carlson says IT operations teams are approaching a critical moment in history: They can either wait for AI models to eventually replace them or they can leverage them to take back control of IT infrastructure.

    However, developers tend to make decisions based on what they prefer versus what’s best for the organization. Before too long, multiple duplicate operations management frameworks are in place. Rather than incur all those management headaches and associated technical debt, Carlson says organizations need a single framework that can more adroitly accommodate the multiple roles within the DevOps team. It’s already long past the time when IT teams should have modernized their operations, he says.

    The relationship between developers and IT operations teams has always been contentious. The more belligerent members of the developer community are even making a case for “NoOps.” In that context, all the IT dollars spent on operations are moved into developing applications because the management of IT infrastructure is completely automated. Given the amount of legacy infrastructure that isn’t likely changing anytime soon, achieving that level of automation inside the enterprise won’t happen for a while. But there’s no doubt that IT operations needs to get more agile. Because of that issue, investment in software-defined data center (SDDC) technologies is growing rapidly.

    Instead of resisting that trend, Carlson says teams need to demand that internal IT operations be brought into the 21st century. It’s true that SDDCs infused with AI models will automate most of the lower-level tasks associated with DevOps. But those members of that IT operations teams who become the masters of building and training those AI models will be among the most highly sought-after individuals in IT.

    — Mike Vizard

  • DevOps: Trust, Obligations and Promises

    DevOps: Trust, Obligations and Promises

    Mark Burgess (@markburgess_osl) is a theoretical physicist, but in his keynote at the 2016 All Day DevOps conference, he talked more about economics and human interactives than physics. What does either have to do as the keynote for a conference on DevOps?

    Well, for a little more background, Mark Burgess is also the founder and former CTO of CFEngine, a configuration management and automation framework, and is the author of “Promise Theory.” While at CFEngine, Mark worked to apply a theory of how the autonomous agents in software interact with each other. Promise Theory was born.

    Exploring Promise Theory

    Promise Theory sits counter to obligation theories. Obligation theories view human interactions and behavior under the assumption that people (or agents, in the case of software) choose behavior based on their obligation to follow the rules. Agents are obligated to take a certain action. Promise theory contends the desire to follow the rules is voluntary and that agents can only be responsible for their own behavior, not the behavior of other agents.

    In his presentation, Mark started out looking at human behavior and trends in our society. For instance, with technological advances, our need for one another is decreasing, even as our interdependence through communication is increasing. For example, our smartphones enable us to do almost anything without the direct interaction of someone else.

    Looking at interactions in society, trust is the basis for our exchanges, working communities, and function as a society. He contends, “The promise is the source of intent by which all promises and observers may calibrate their expectations.” Money is an example of this. Money does not have intrinsic value, but we use it because we trust the backer will stand behind its value and it is more convenient than bartering goods. He explains that in the old monetary economy, banks and governments were the provider of trust; in the new service/IT economy, the service provider is the basis of trust.

    How Agents Make Promises

    How do you model this trust? Enter Promise Theory, which describes how agents make promises with each other and how this leads to trust between them. Mark outlines the “ingredients” of Promise Theory:

    1. Agents and super agents make promises in a scaling hierarchy.
    2. An agent can never make a promise about another agent’s behavior (only its own).
    3. An agent’s promise need not be accepted or used by its intended or unintended recipient!
    4. An agent makes its own valuations: what you have is only worth what another is willing to give you for it.
    5. Dependency invalidates promises.

    At the core of understanding this, you have to understand that collaboration is made up of semantics, what words mean and dynamics: all how things work.  And that our ever-increasing service economy is driving us to an explosion of semantics, because words matter so much more than the mechanics. As an example, Mark sketched the hierarchy of promise dependencies in the old and new economies. Looking at the old economy, where we have more dynamic money, the hierarchy is:

    Retail→ Money→ Banks→ Central Banks→ Government.

    In the new economy, where we have more semantic services, the hierarchy is:

    Client→ Network Availability→ Provider→ Internet Authorities→ Company/Service Organization.

    The Need for Cooperation

    None of these systems could work without cooperation at the bottom of it all. So, production is a cognitive process between the producer and the consumer, but also between the agents inside those organizations that have to interact. Often we create pipelines to describe these interactions, but the interdependencies within the systems are often much more complex than the pipeline can model.

     

    This is all just the tip of the iceberg. Mark has three books that dive into it in detail: “Thinking in Promises,” “In Search of Certainty,” and, “Promise Theory.”

    In the end, Mark made the point that technology does an excellent job of both binding us together and pulling us apart. We need to be careful that we don’t take ourselves out of the systems we create, or we could scale ourselves out of existence.

    DevOps.com Supports the All Day DevOps Conference

    If this piqued your interest, you can watch Mark’s full All Day DevOps conference session (just 40 minutes).  When you do, you can view the other 56 presentations from the 2016 conference online and free of charge.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s free and online on Oct. 24.  DevOps.com is a media sponsor for All Day DevOps 2017.

    — Derek E. Weeks

  • 5 Team Skills That Make DevOps a Success

    5 Team Skills That Make DevOps a Success

    When starting their DevOps journey, your team will discover it needs to develop some new skills. Most likely the team is good at typing code and solving issues, but what about the soft skills?

    Many teams have to learn how to deal with issues themselves—as self-supporting teams, there is no longer a manager who will help them out and tell them what to do.

    During the Phoenix Project DevOps business simulation, the majority of teams discover collaboration and communication are skills they lack. I am often surprised how few people can really facilitate standups, retrospectives, Andon-sessions and other communication methods.

    Here are the five team skills that participants recognize as core skills they learn (or could learn) during the simulation.

    The Standup

    The standup is a very powerful way to share knowledge about certain issues. The team will stand together for a short time and discuss an issue that blocks the flow, needs clarification or needs to be high priority. So, who should initiate this standup and who should lead it? I’ve noticed a lot of participants find it very difficult to manage this standup initally, but after a few times the team discovers the key elements of a successful standup:

    • Chose a facilitator. Someone who is leading this session. It doesn’t matter who (it doesn’t have to be a manager).
    • Share the focus: What are we going to talk about?
    • Keep it short (time is ticking).
    • Make sure we all understand the outcome.
    • The importance of listening and engagement.

    Facilitating a Retrospective Session

    DevOps is also about continuous learning and experimenting. One of the important steps in this process is retrospective. This activity explores the past round of work and reviews what went OK and what needs to be improved. Again, during the simulations I discovered that not many participants know how to execute this activity. After being shown the right example, the team often can perform this activity themselves. This is what teams learned during the simulation:

    • Someone from the team must facilitate this process.
    • Make sure the improvement is supported and understood by the whole team.
    • Test the improvement before to applying it.
    • Record the discoveries—this becomes part of the visual management system and part of the team backlog of improvement needs.
    • This team skill is one of the skills that more than one team member should be able to execute.

    Managing the KanBan

    During a project, teams sometimes will have standup moments to discuss the work visualized on their KanBan. Work will move from backlog to doing and done, and new priorities will change the order of work. During the simulation participants will take the lead in this process, but some don’t find this easy. After the sessions they discover:

    • Use standups to discuss the changes on the KanBan, make sure the team is fully involved when moving notes around.
    • Challenge participants to share their thoughts. Saying “yes” doesn’t always mean “yes.”
    • Errors are made, but that doesn’t mean somebody has failed and should be punished. Keep the session safe and open.
    • The KanBan is a form of visual management system (VMS). The VMS is not the goal—it is an instrument and must help the team to visualize flow, limit work in progress, drive productivity through an awareness of what people are working on and continuously improve the process.
    • Make sure at the end of the session, all participants know what’s agreed.

    Managing an ‘Andon’ Session

    ‘Andon’ is a very powerful instrument to explore a situation that stopped the flow. As soon as the flow stops, someone “pulls the rope” and the team “swarms around” to investigate the error and try to continue the flow as quickly as possible. This session should be led by a team member; it doesn’t matter who. During this session, the team discovers the following learning:

    • Make sure the session is short and focused on fixing this blockage and restarting the flow.
    • Make sure we are not blaming each other; it should be about exploring what stopped the flow, fixing it and agreeing how to prevent it from happening again.
    • Make sure the whole teams swarms around and participates.
    • Create a safe environment.

    Collaboration

    Doing the work is not always easy. Sometimes we don’t know what to do. Sometimes we need help, and sometimes we are not happy with the actions of a team member. This is what collaboration is all about.

    One of the first aspects of collaboration is helping each other. But, this is not always effective. Sometimes it’s even disruptive or we just take some work from somebody else if we think he is too busy. Here’s what participants learned about collaboration during the simulation:

    • We have to avoid “compensational behavior.” This is about taking other people’s work with the intention to help the other. Sometimes this person doesn’t know you took the work and he or she does it again, creating unnecessary double work.
    • Collaboration is about shared goals and shared processes. We should all know what we are doing and how.
    • It’s also about clear roles and responsibilities. If you know what your colleague is doing, you know what you have to do. And the other way around.
    • You must have an open team culture in which it is also safe to say you don’t know and to ask for help.
    • Collaboration is fun, and it gives energy if we as a team achieves something.

    Conclusion

    I have visited many global DevOps conferences in the previous years and have listened to numerous speakers declaring, “It’s all about people.” And they were right. Unfortunately, the audience often did not get many useful answers.

    A high-performing collaborative team needs team players who know how to deal with the collaboration and communication issues themselves without needing management intervention.

    Teams will have to go through the stages made famous by Bruce Wayne Tuckman (Forming, Storming, Norming, Performing and Adjourning) to become an effective, collaborative team.

    Continuous learning is a key success factor and a critical competency for any DevOps team. Why not start this learning journey in a safe and challenging environment?

    A simulation such as the Phoenix Project is a very powerful instrument to teach teams how to collaborate and become high-performing. The environment is safe and feels like the real work environment. Participating in this simulation with an actual team allows you to capture and agree on actions to takeaway. Outcomes can easily be transferred to day-to-day work.

    I wish you a lot of soft skills.

    About the Author / Jan Schilt

    Jan Schilt is owner of GamingWorks. His background is Human Resource Development (MSc). Within GamingWorks he is responsible for Business Simulation design, Facilitation and Train the Trainer. He is also an expert in developing learning activities related to organizational change. He builds all the simulations and has delivered more than 300 workshops worldwide.