devopsy.kinsta.cloud

Author: David Geer

  • Continuous Delivery and Deployment: A Tale of Two Models

    Continuous Delivery and Deployment: A Tale of Two Models

    EnerNOC (Energy Network Operations Center) is an energy intelligence software provider with more than 1,300 employees in its Boston headquarters and international locations. The public firm helps global customers to efficiently consume energy. “Our mission is to change how the world uses energy, helping customers to improve profitability and visibility for better decision-making and sustainability by focusing on how they buy energy and when and how they use it,” says James Nichols, principal engineer and cloud architect at EnerNOC.

    A regular user of Splunk (the company uses other DevOps tools and platforms as well, including AWS and Oracle), EnerNOC has fully automated monitoring in its perpetual drive to move past continuous delivery and on to continuous deployment.

    Embedded in EnerNOC’s story of why it uses continuous development and continuous deployment models is a chapter on its move to fully automated monitoring.

    EnerNOC DevOps Models

    EnerNOC’s DevOps model is in constant transition toward full automation. Currently, it has fully automated deployment, fully automated monitoring and semi-automated quality assurance (QA), which pull the company’s continuous development approaches.

    A change to “builds on commit” of smaller applications within the legacy software monolith at EnerNOC came hand in hand with further automation, targeting what the company calls “trustworthy continuous deployment,” Nichols says. That was the crux of EnerNOC extending itself into deployment automation.

    Wherever the model is not fully automated and continuous in deployment, it remains continuous delivery. “Most of the variance between the approaches is due to aspects of the delivery pipeline that still have manual processes; these parts are usually comprised of the remaining manual testing procedures,” says Nichols.

    Change Control Necessitates Two Models

    EnerNOC develops and maintains mission-critical software that must pass through a change management approval process. This process includes communications within and across the company and customers, as well as user acceptance testing. This is where the manual testing comes in. Only at the end of this approval pipeline can EnerNOC pass the software baton into the hands of production.

    Due to EnerNOC’s tightly yoked systems and the manual testing, change control remains, along with a mix of continuous development and semi-automation with fully automated continuous deployment. “DevOps is a cultural movement as well as a technical one, and so the existing model could evolve with the organization’s quality and maturity evolutions,” says Nichols. If the manual testing and the relationships between systems give way to more fully automated testing and more amenable relationships, QA could more closely embrace full automation.

    Monitoring Challenges/Achievements

    EnerNOC’s application diversity includes hybrid software with a foothold in the Amazon cloud, in EnerNOC’s data centers and on-premises. “We use three tiers on top of Oracle in the data center. In AWS, we use a combination of big data, cloud-friendly and serverless technologies,” says Nichols. Business transaction data traverses many machines that belong to service groups’ stakeholders. “The distributed, decentralized architecture makes troubleshooting, performance management, security operations and production operations challenging,” notes Nichols. Better monitoring was the answer.

    EnerNOC wanted to offer nontechnical users easy access to systems-level monitoring data with transparency into the many systems through a single pane-of-glass dashboard with readily available system notices and troubleshooting abilities. “We would use Splunk to do this and then feedback everything we learned from using Splunk into driving our DevOps and engineering processes into the future,” he says.

    The EnerNOC DevOps performance team built the dashboard that accompanied the Splunk service, ensuring it would register the performance metrics that are of value to the organization. The performance, product management and development teams maintain an awareness of the metrics and receive the related alerts. “The development team incorporates the ongoing metrics early in the development life cycle to improve application performance,” says Nichols.

    EnerNOC first applied Splunk to web logs for operational capacity planning, then used Splunk to give the Ops team transparency into the function of its new real-time data product that was working in production. As new products came on board, the organization added them to Splunk to provide Ops teams with dashboards, at the urgent request of Ops team members.

    Current Status: All Systems Good

    EnerNOC applies Splunk to test and production systems for views into consistency and acceleration. “Splunk is our primary alerting infrastructure for low-level system metrics like CPU utilization as well as for complex business transactions that span multiple systems,” says Nichols. Splunk has eyes on application health, cluster state across machines, expected systems tasks and suspect or known-bad system behavior, as well as the presence or lack of any significant event.

  • Building DevOps Careers: One Man’s Journey, Part 2

    Building DevOps Careers: One Man’s Journey, Part 2

    In Part One of this two-part series, we introduced former SlideShare and LinkedIn software engineer Sylvain Kalache, who shared his career growth within the two companies. In this second part, we’ll look at San Francisco-based The Holberton School of Software Engineering, which Kalache co-founded to better prepare students to work in a DevOps world.

    School Founder, Still a Coder

    Kalache still uses much the same coding skills in his new role as he did as an engineer. “I still manage machines, the ones that host our website and intranet and the iMacs that students use,” he says.

    The Holberton School of Software Engineering founders removed every manual step possible, using software to automate the admissions process and any process that they could. Even in the classroom, automation rules: Computers automatically review student work on projects.

    “Computers test that students’ code works as expected by simulating user interaction, checking syntax and assessing how fast the code executes. For infrastructure in code, computers check for stability, response times and security breaches. By using multiple ratings parameters that are linked to specific skills, the computers collect detailed data about the strengths and weaknesses of each student,” says Kalache.

    “We have a lot of our own secret sauce in software automation running the school for us,” he adds. “This requires all the technical skills I used in my coding days including designing, building, configuring, testing, deploying and maintaining software and systems.”

    Teaching Methods

    The Holberton School of Software Engineering engages students in building each skill they will need for careers in software engineering through use of industry-standard teaching methods. “Methods include deep learning in building features into software products and classical teaching in clear, documented code,” Kalache says. “We have a team of 150-plus mentors who make sure our curriculum is up to date and in sync with what the industry actually does and who coach and share their knowledge with students.”

    The Holberton School of Software Engineering uses fully practical methods with no formal teachers or lectures, only mentors. “Mentors are like managers in a workplace. I mentor, coaching students in their learning journey and giving advice when there are issues,” he says.

    Students engage in peer learning, working collaboratively on software projects, which encompass the skills they will need to demonstrate in their careers. Successful completion of projects, including solo projects, is proof of attaining the required skills. “Students must defend their projects before mentors with mentors ensuring that every student can present and explain the technical aspects of the project; if even one student can’t explain it, then the group fails on that project,” says Kalache.

    “We teach problem-solving and how to learn new skills because there will constantly be new technologies and concepts for our students to understand and master in their careers in the tech world,” he adds. “Professionals who cannot adapt and retrain themselves will quickly fall out of date with technology.”

    Readying Students for Careers from Day One

    The Holberton School of Software Engineering prepares students to be ready to work when they start their careers. “Graduates of other programs whom I have spoken with have no idea what it is to ship code to production and run a production environment. They are used to academic projects that teachers correct for them,” Kalache says. “That is not how it works in the industry.

    “Even boot camps and universities provide minimal practical experience that will make job hunting difficult,” he continues. Smaller organizations don’t have the money to pay someone to learn before starting them working in production. Thus, he says, startups need to execute fast using professionals with mature skills. “Though our students will fail and struggle, they will do it at the school, which will have a much smaller impact than doing it on the job,” says Kalache. Students are exposed to creating, shipping and working with code in production for two years before graduation.

    What’s more, real, workday DevOps requires effective communications skills. “Whether it is to introduce the last tool you built, explain what happened during an outage or convince your managers that your solution to a problem is the right one, you need to have oral and written communications skills,” says Kalache. As such, the school runs students through a series of exercises in public speaking, technical writing and documentation, as well as post-mortem communications when dissecting an outage, prepping them even further for their careers and the world beyond the classroom.

  • Building DevOps Careers: One Man’s Journey, Part 1

    Building DevOps Careers: One Man’s Journey, Part 1

    In 2011, Sylvain Kalache was a software engineer at SlideShare, which LinkedIn acquired in 2012. Kalache was a senior systems reliability engineer (SRE) at LinkedIn, doing project management and planning, documenting and sharing knowledge tools, and adapting himself and his work to the LinkedIn organization and its technology.

    Today, Kalache is helping build DevOps careers as co-founder of Holberton School of Software Engineering in San Francisco, where students learn to code through “project-based and peer learning,” according to information from the Holberton School website.

    This is Kalache’s DevOps career story, how he worked in and then transitioned from a career in DevOps to a personal and professional investment in helping students to build their DevOps talents toward a lifetime of coding and contribution in software engineering and, hopefully, successful careers.

    Software Engineering at SlideShare

    “SlideShare was a very small company when I joined. The San Francisco office had only 10 people; I was only the second engineer at that location,” says Kalache. But the DevOps mindset was strong there, he says, and the company had bet everything on DevOps methodologies. In fact, it was the commitment to DevOps by SlideShare and its CTO at the time, Jon Boutelle, from which Kalache learned of the development approach and took to it so quickly.

    “Boutelle was looking for full-stack or aspiring full-stack engineers and I was a junior programmer and eager to learn. In a small company like that where you are encouraged to work on every aspect of the product life cycle, it became very clear that I was in charge of taking care of my mistakes,” says Kalache, adding SlideShare expected him to take ownership of his code from design to production.

    “Because I was eventually in charge of a lot that was going on with the infrastructure at SlideShare, I had to build and maintain a lot of systems that were serving billions of slide views per month at what was one of the top 100 most visited websites in the world,” says Kalache.

    People from the Dev and Ops teams there did not only their own work, but also the work of their companion team. In essence, Dev was both Dev and Ops and, likewise, Ops was both Ops and Dev.

    “The SlideShare people were writing their own configuration management code, setting up monitoring of new services and features as they built them, deploying their own code, and troubleshooting code in the production environment,” says Kalache. Ops people helped developers design sound code for acceleration while also giving their code a second look to ensure fitness for operations.

    Project Skynet

    “The most exciting project I built for SlideShare was Skynet, an automated tool to monitor, scale and auto-heal a system in the cloud,” says Kalache. Skynet was developed to meet several challenges in the day-to-day work of simply being SlideShare and doing what SlideShare does. SlideShare acted on, renamed and translated countless files from one type to another to share them on the site. As with other slide technologies, the tool was a fit for teaching and presentations.

    There was no way to know exactly how much users would upload, access or view any slide content at a given time. “There were peaks and valleys in upload traffic with mornings seeing high traffic and low volumes in the evening and on weekends. The SlideShare infrastructure had to scale quickly to encompass and convert common document files at speed. This required adding many hundreds of machines to the infrastructure,” says Kalache, which were virtual  machines that spun up in minutes in the AWS EC2.

    One purpose of Skynet was to enable precise analysis of log files for transparency into everything from individual file conversions to changes in the entire machine cluster as the environment scaled up or down. The other purpose of Skynet was to use the analyses to auto-heal; in other words, to restore the cluster back to its initial size once the peaks were over.

    With many thousands more employees at LinkedIn than at SlideShare, Kalache left many of his duties at SlideShare to others while he dove much more acutely into specific aspects of his work.

    Co-Founding a School That Builds Careers

    In speaking with his colleagues and representatives from various enterprises, Kalache continually confirmed that organizations using DevOps are in need of lots of engineers but they can’t find candidates who are knowledgeable and experienced. “I was actually recruiting for SlideShare and for LinkedIn. What saddened me was to see a graduate who had just spent half a decade and a lot of money on a master’s degree only to be far from employable,” he says.

    Kalache found it hard to find people who knew basic software engineering, DevOps, and how to be an SRE for LinkedIn. “We instead had to hire early graduates who had coding skills and an interest in operations and train them on being an SRE,” he says.

    Kalache decided to combine an awareness of alternative teaching methods with his passion for the coding and ops communities to find a solution to how to better prepare software engineers for careers. “After mulling it over for two years, I decided to quit my job at LinkedIn and create the Holberton School,” says Kalache.

    In Part Two, we dive headlong into the teaching and learning experiences at the Holberton School of Software Engineering to help students learn and understand the DevOps methodologies and increase their chances for success in their respective careers.

  • Salvador Partners Bridges Global Gaps for DevOps

    Salvador Partners Bridges Global Gaps for DevOps

    Salvador Partners works on a global scale to help its customers grow their business. The company, which has its own set of products, also focuses on assisting startups in making their vision reality. Indeed, it bills itself as “experts at innovation and exploiting innovation.”

    The company started life as a subcontracted electronic components manufacturing company, slowly broadening its lineup to include security products and patents and evolving to become a provider of technology for banking, financial services (fintech), defense and mechanical systems, says Lewis Daniels, founder of Salvador Partners. The company also serves as an incubator and capital arm for new technologies.

    Salvador builds circuit boards and technology products, working across artificial intelligence (AI), augmented reality and blockchain technology, together with its other endeavors. Across its projects, wherever software is part and parcel with its products, Salvador Partners’ international team of developers are hard at work.

    The firm has built a DevOps culture across its organization despite geographical and language barriers. devopsy.kinsta.cloud explores the Salvador Partners devops culture that works in an umbrella fashion among these international developer groups.

    The Global DevOps Culture Challenge

    Salvador Partners works with development teams in Russia, China, India, the U.K. and the United States who build software prototypes, which can sometimes prove tricky: There are challenges to getting foreign teams to plug themselves more deeply in to existing domestic teams for future projects. “There’s an obvious value in their experience from the build of your prototype that you then want to tap. The question is whether they want to be a part of your longer-term vision,” Daniels says.

    The company also grapples with the location question—whether to consider moving some of its operations to those countries, closer to those teams, or move some or all of them to its domestic operations. “Perhaps the world has become small enough through the use of tools like Slack and [project management] tools that you can let the technology bridge any gaps,” says Daniels. The tools so far have helped disparate development teams make the leap, including addressing any time zone and language issues that appear.

    Delineating the Salvador Partners Culture

    Salvador Partners uses an agile approach. “If each client wants a unique tweak, we simply badge up a boutique offering where we can innovate, collaborate and deploy in an agile environment to suit the client requirements,” Daniels says.

    As development teams work together more frequently, they reuse code and proven approaches more often. “We often find tricks that get carried over to the next application or dashboard,” says Daniels.

    Tools are the answer. “Things have changed in recent years. In terms of a culture, I think it’s pretty easy to leverage tools such as Slack and Dropbox to make the team smaller, regardless of where in the world they are positioned,” he says.

    Salvador Partners developers also use Trello for sharing project plans and Skype and Uber conference for team collaboration calls. “We use a secure mobile app we developed internally to communicate on some of the new, high-tech or sensitive topics. WeTransfer is also pretty useful for sending larger files,” says Daniels.

    Though Salvador Partners looks at its DevOps culture as a whole rather than by region, there are some unique, regional aspects. “We have a real good team based in India and they just go get things done, whereas some of their counterparts we use sometimes get caught up in planning the plans. This can make it difficult to pull a number of pieces of a project together,” says Daniels.

    Other Solutions to Challenges

    Locations, languages and distances present other challenges as well. Because some projects work with encryption protocols that by law can’t be shared beyond certain countries, the development work can happens only inside the U.K., the United States, Australia, Canada and some European countries, Daniels notes. In these cases, the company simply keeps the work internal, within the countries permitted.

    Language barriers can also present issues. “Such was the case where Salvador Partners had Canadian companies working on something and had a French speaker cause a few problems for his English-speaking colleagues,” Daniels says. For language challenges, the company simply whiteboards the project or uses translators.

    Time zones can also trip things up. “It creates challenges when you have a New Zealand head office dealing with partners in Singapore and developers in Russia and California. We have had conference calls where participants are going for dinner and at the same time another having their first coffee,” Daniels says.

    However, time zones and geographies are less an issue as the available toolsets to solve communications barriers have evolved.

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    In part one of my conversation with Stefan Thies, Sematext DevOps expert. we spoke about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Let’s continue the conversation.

    David Geer: What makes Logsene centralized? Where is it centralized a typical deployment? What are the advantages of it being centralized?

    Thies: Traditionally, systems store logs from multiple applications on every server. This makes it difficult to search logs for troubleshooting or auditing purposes. To find a relevant log entry, operators must log in to each server, find the right file on this server and search in this file for a relevant message.

    This can easily take several minutes. With systems storing logs on multiple servers, it is difficult to correlate data, so minutes can turn into frustrating multi-hour root cause-hunting sessions, often involving multiple team members. Logsene provides a central full text index for all log data coming from multiple servers and applications. There is no need to log in to multiple servers and search/grep multiple log files. All team members see all data at the same time and can collaborate much more efficiently.

    Having a central place for search makes it much easier to find logs quickly, run analytic queries or centralize alert rules in a single place.

    Geer: What is it about what Logsene does for Docker logging?

    Thies: Sematext Agent for Docker interacts with the Docker API to automate the collection, parsing and processing of logs, metrics and events. It detects containers automatically (no setup or configuration for a new container!) and supports many log formats for applications on Docker Hub such as Nginx, Apache, Elasticsearch, Solr, MongoDB, MySQL and many more.

    Geer: What does your statement mean, “Some DevOps engineers even think of Logsene as ‘ELK Stack on steroids’”?

    Thies: The ELK Stack is a combination of open source tools: Elasticsearch, Logstash and Kibana. These tools are very popular for log shipping (Logstash), indexing and search (Elasticsearch) and visualization (Kibana dashboards). However, these tools are missing many enterprise features like multiuser user interface, role-based access controls, secure transport of logs or out-of-the-box configurations and an easy query language to search logs. This is where Sematext Logsene adds the relevant enterprise features to these open source tools to make them easy to use and securely integrated into the enterprise information technology infrastructure.

    Maintaining Elasticsearch clusters is another problem for many people. It requires a lot of expertise. Using Logsene does not require this sort of expertise. The Logsene SaaS provides a managed solution where users don’t need to know anything about maintaining Elasticsearch clusters or the required infrastructure. Sematext takes care of that for them. Sematext customers can benefit from these managed services and pay a fraction of the cost compared to self-hosted ELK stacks, where storage, computing and networking infrastructure and manpower for highly skilled workers counts.

    Geer: What is Tutum? What is so important about ease of launching SPM and Logsene Docker on Tutum?

    Thies: Docker acquired Tutum in 2015. Tutum provides management of Docker containers as a cloud service. Tutum is a platform made to operate “Dockerized” application stacks. In the case of Sematext Agent for Docker, the Tutum integration deploys Sematext Docker Agent automatically to all cluster nodes in a few seconds. These nodes could run on different cloud platforms such as Amazon, Digitalocean, Rackspace or on premises. Again, a big time-saver for people running larger infrastructures or hybrid clouds. Supporting Docker products and having excellent integration with Docker products is important for Sematext.

    The Sematext Docker Agent runs on all solutions offered by Docker e.g., Docker Swarm and others such as Google Kubernetes, Google Container Engine, RancherOS, CoreOS, Amazon ECS or Hashicorp Nomad.

    Geer: Please discuss the Sematext Docker and Docker image?

    Thies: Sematext Docker image is available on Docker Hub, and we shared the Tutum Stackfile for Sematext Docker on Stackfiles.io. The easiest way to get it is to use Sematext, which generates the stackfiles for you, including tokens.

    Docker Hub is the central registry for public application images including related installation instructions. Docker users can find out quickly how to run a “Dockerized” application with the help of the instructions in Docker Hub. Typically, this requires a little configuration work before a container can be started with the right settings.

    Stackfile.io is the place where people share preconfigured application stacks (e.g., a web service together with the required database). Sematext provides a preconfigured setup for Sematext Docker Agent to monitor all containers in Tutum Cloud. This means one can launch Sematext Docker Agent and monitor all containers running in Tutum Cloud with just a single click for the chosen stack file and by entering a unique identifier (Token) for the application. That is really all it takes.

    When you log in to Sematext SPM or Logsene the general installation instructions have all required configurations displayed. Sematext users can simply copy/paste the relevant commands including unique identifiers (tokens) for their application to get the Sematext Docker Agent deployed. This, too, is a big time-saver for our users and it gets them up and running in just a few minutes.

  • Walmart OneOps Won’t Be One-Upped

    Walmart OneOps Won’t Be One-Upped

    One of the biggest names in retail also is kind of a big deal in DevOps. Walmart has developed its own platform, OneOps, designed to streamline operations and keep its application development schedule in check.

    “OneOps is an open source cloud and application lifecycle management technology platform with benefits including cloud portability so developers can seamlessly move applications, databases or entire cloud environments from one cloud provider to another,” says Brian Johnson, VP of Engineering, Platform, Walmart Labs. The latter feature allows developers to leverage the broad selection of cloud environments and pricing schemes to their advantage.

    The OneOps platform deploys Walmart website adds/changes, launches new mobile features such as Walmart Pay, and creates APIs so third parties such as Hiku can tap into Walmart’s online grocery offerings, at speed and with a lower investment in the cloud, says Johnson. The OneOps platform once only managed the Walmart.com e-commerce site, but now it also serves individual Walmart stores and supports other initiatives.

    According to Johnson, OneOps undergirds and empowers the work of developers as follows:

    • OneOps continuous lifecycle management continuously auto-pilots the process of automating application scaling and repair whenever any unforeseen changes take place.
    • OneOps promotes accelerated innovation so software engineers can spin up new environments for app hosting in minutes, since the DevOps platform skirts the need to specify the intricate details of any one cloud environment by modeling that for the developer.
    • This abstraction of the cloud environment gives developers control over the tools and technologies they select regardless of the cloud provider they use.

    Life Before OneOps

    “Before OneOps, the process of updating Walmart’s website was so tedious that we rolled out updates only every two months. The process used hundreds of manual steps and required significant resources,” Johnson says. “This was time-consuming and created room for manual errors.”

    Before OneOps, A/B testing was expensive and time-consuming, taking months to set up multiple cloud environments and conduct tests. By the time testing was done, the rationale for the new feature had waned and whatever opportunity it was created to take advantage of had passed, he says.

    At that time, creating enough virtual machines (VMs) in time to handle peak shopping events and traffic would have taken teams of people months to complete, at the risk of exorbitant operating costs, Johnson adds.

    The OneOps Afterlife

    OneOps now fully manages the Walmart e-commerce platform such that in a typical day the site sees more than 1,000 new software deployments, which developers execute on demand in a span of minutes on average, says Johnson.

    OneOps serves a variety of customers such as consumers of Walmart.com, who on black Friday found their shopping experience supported by the tens of thousands of VMs OneOps was able to spin up across multiple clouds to cradle the peak traffic, Johnson says. OneOps is the DevOps and cloud platform for SamsClub.com as well.

    Solving the Single Cloud Provider Dilemma

    Walmart targeted OneOps to use multiple cloud providers easily, because of the pitfalls of being locked in to a single provider. “The fear of vendor lock-in is the major challenge to cloud service adoption. Cloud vendors like AWS don’t make it easy for customers to move from one cloud to another. The complexities of cloud service migration mean that many customers stay with a provider that doesn’t meet their needs simply to avoid the cumbersome and expensive process of migration,” says Johnson.

    Further vendor lock-in concerns, he notes, include:

    • Contracts of lengthy duration geared toward ensuring higher server utilization, maximum ROI and profitability for the cloud provider but providing little benefit for the cloud customer.
    • Lack of freedom of choice where storage space, processing power, bandwidth and tools and technologies are concerned.
    • Because it is typically difficult to find any two cloud providers that use the same tools, compatibility is an issue. “You can’t simply port your site to a new provider without rebuilding it using the new provider’s toolsets,” says Johnson.
    • “While you build and test your site on the new cloud platform, you are still paying to maintain it on the old provider, doubling your costs,” says Johnson.

    This is why OneOps abstracts the use of differing cloud providers, making porting easy.

    The Big Question Is …

    OneOps solves a unique set of DevOps challenges related to the cloud. Can you name another platform that does something similar?

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Stefan Thies is DevOps evangelist Sematext Group. I recently spoke with him about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Read on for part one of our conversation.

    David Geer: Technically speaking, for the purposes of this article, what is logging?

    Thies: Logging means to store the output of programs (applications, servers) running on a computer. Typically, log messages that software applications produce are stored in log files. However, software running in Docker containers writes its log messages to a virtual terminal (console). The Docker Daemon (the server process managing the containers) has several facilities to forward log messages via log drivers or the Docker API to external applications, which are responsible for storing the log messages. This is not the traditional way of working with log data. This changes the log collection and storage mechanisms and forces many organizations to change their workflows for log management by introducing new tools.

    Typically, you would use a service called syslog to receive and store log messages on servers, but this requires special configuration for all Docker containers. While Docker made it easier to deploy applications, the new technology made logging and monitoring much harder as you could not use the traditional tools apart from integration with Docker.

    This is where companies like Sematext provide added value, by enabling:

    • Integration into the Docker Ecosystem
    • Cloud storage for log files
    • Indexing for full text search, analytics, visualization and alerting on log data

    Geer: What has Sematext Logsene contributed to logging, technically?

    Thies: Logsene Log Management includes all required workflows and security procedures to handle log data. Logsene provides role-based access controls for access to log data. Logs are collected and shipped to Logsene servers (running in the cloud or on premises) where the data gets stored and analyzed.

    Other important features are:

    • Full text search
    • Analytics
    • Alerts
    • Anomaly detection

    Geer: In what way is Logsene/Sematext available to users and organizations that seek solutions to capture logging for monitoring their Dockerized distributed applications?

    Thies: Logsene is available on premises and as SaaS in the cloud. It fits in hybrid environments as well such as running the Dockerized application in-house on bare metal while shipping logs to Logsene SaaS. In this case, only Sematext Agent for Docker runs on premises.

    Sematext Docker Agent automatically detects running Docker containers and automatically collects logs from the containers. The Sematext Docker Agent can transform unstructured log messages into structured data for analytics. We do this with integrated parsers and a pattern library, which detects log formats to structure logs. The final task of Sematext Docker Agent is to ship the structured logs from all containers securely to Logsene, where indexing for full text search and analytics queries takes place. The Logsene user interface provides all tools for search, visualization, anomaly detection and alerts on log data.

    Geer: What is SPM for Docker? What role does it play?

    Thies: Both Logsene and SPM collect data via the Sematext Docker Agent. SPM is a performance monitoring, anomaly detection and alerting solution for many server applications like Web servers (e.g., Apache, Nginx), databases (e.g., Cassandra, MongoDB, HBase, MySQL, etc.), big data engines (e.g., Spark, Storm, Hadoop, etc.) and full text search engines (e.g., Elasticsearch, Solr). Sematext Agent for Docker collects not only container logs but also performance metrics of all containers at the same time, plus Docker events.

    SPM for Docker provides operational insights into application performance on the Docker platform. It is not limited to performance monitoring—it integrates with Logsene for log management in a single user interface. DevOps teams can find log messages in correlation with potential performance bottlenecks. Most organizations use different tools for performance monitoring and logging. Sematext has recognized that metrics and logs must be available in a single platform to provide detailed operational insights without having to switch between different tools and without overhead for things like integration of multiple tools, multiple user interfaces, logins managing access permissions for multiple tools, and billing or licensing from multiple vendors such as NewRelic or Splunk.

    Watch this space for part two of my conversation with Stefan Thies of Sematext.

  • 7 Causes of Continuous Delirium

    7 Causes of Continuous Delirium

    When we think of CD, we typically think of continuous deployment as a result of continuous delivery, which is a goal of DevOps.

    In this article, I coin the phrase, “Continuous Delirium,” to mean unending confusion or bewilderment in development environments. There are probably several examples that could fit the bill here, but for now I offer seven real-world examples. I bet you’ve seen or heard about at least one of these.

    Now that I have your attention, here are seven causes of Continuous Delirium, each with cures rooted in DevOps:

    No. 1: Insufficient Configuration and Build Management

    Insufficient configuration and build management in software builds leads to the same or similar software impurities rearing their ugly heads again and again. You build, test, discover the defect, fix it—or so you think—build again, retest and boom: The same coding error or software malfunction recurs.

    “You can cure this malady with a rational approach to the build and configuration management process, checking software in and out with established procedures for each software module using a tracking sheet that identifies the revision of the sub-modules for the entire software product,” notes Jon M. Quigley, founder of Value Transformation, a product development consultancy and training firm based in Lexington, North Carolina.

    No. 2: What’s Going On? Who’s Doing What?

    When there is no reasonable understanding of the work, who is doing what work and who is responsible for what—and there is no consistent measure of what is good work or bad work—you have a breeding ground for anarchy.

    “You cure this ill by establishing what constitutes the appropriate quality of work as well as by periodically reviewing the state of the objective with the receiving or depending entity,” says Quigley. Make sure everyone is on the same page with the same definition of what constitutes quality work and make sure the objectives have not changed. Also, ensure these goals are also clearly understood by all.

    No. 3: Confusing Clean Daily Compiles with Testing

    When you do not test the software as early as possible and throughout the development process, you can’t really say it has no bugs. Error-free compiling is not error-free software. Software compilation does not have the same investigative effect as using the software in the customer environment.

    “Test throughout development using complimentary testing approaches such as testing to requirements, combinations of stimulus testing and exploratory testing, for examples,” says Quigley.

    No. 4: Planning Too Far Out to Control

    The further you try to plan things out into the future, the less control you have as you follow the project through to conclusion. “When we treat a plan as though we know everything associated with the project for the next 12 to 18 months, we are fooling ourselves,” says Quigley. By breaking projects into bite-sized increments, you can better assure successful completion of sub-projects and of the project as a whole.

    No. 5: A Lack of Customer Involvement

    Customer involvement is critical to customer satisfaction. “By getting the product or a prototype into the customers’ hands for exploration, you can retrieve feedback for developers,” says Quigley. This may expose the project to risk in the form of the potential for a great deal of change at the customers’ request, but the other side of the coin is the risk that nothing changes and the customer doesn’t use the product at all.

    “Rather than try to deliver one incarnation of the product at some future date, break down the delivery into separate iterations that meet prioritized customer expectations one by one as you refine the product and add features as you go along,” says Quigley.

    No. 6: Isolating Change Management from Customer Involvement

    Customer involvement means there will be change. Change requires change management. There is a given—and correct—assumption that customer involvement will trigger change management at some point. You cannot isolate the one from the other.

    Further, just as you can better address customer requests for changes by breaking the project into smaller subsections, so also can you manage budget through change and change management. Project subsections enable you to see the need for change a little at a time so you can create change in a more targeted and cost-effective manner.

    No. 7: Forget About Collaboration—at Your Own Peril

    It is necessary for DevOps team members to collaborate at all levels, except those where the work on software is so minor that it wouldn’t benefit from shared thought. To maintain development as an expedient process that benefits from all stakeholders, open communications and collaboration must reach and pull from all team members. Otherwise, the project could suffer from a lack of critical input.

  • SaaS-Powered FlightPartner Fueled by DevOps

    SaaS-Powered FlightPartner Fueled by DevOps

    Private jet charter company FlightPartner is reaping the benefits of DevOps to build a powerful software-as-a-service (SaaS) offering to keep its customers flying high with no delays.

    “FlightPartner does not own any aircraft but rather facilitates the travel needs of clients by offering air charter brokers (agents to the client) access to a global network of private jet charter availability,” says Douglas Schmohl, CEO and founder of FlightPartner. By acting as a global distribution network (GDN), FlightPartner enables access to real-time availability of private jets.

    What TravelPort is to airline seat availability, FlightPartner is to private jet availability, Schmohl says.

    FlightPartner’s cloud-based SaaS manages data exchanges and transactions through a web browser as a web-based service, requiring no client software installs for the brokers or operators. The service uses Angular.js, a “structural framework for dynamic Web apps” to pass data from the browser to a PHP back end. A database and integrated systems back end builds real-time pricing quotes for available private jets, while the payment mechanism supports search, booking and payment, completing the transaction, Schmohl says.

    FlightPartner uses a Laravel open-source PHP framework, which was created by Taylor Otwell and was originally targeted to developing Web applications following the model-view-controller (MVC) architectural pattern. “This framework allows for rapid development, testing and deployment of our SaaS solution,” Schmohl notes.

    SaaS Technical Challenges

    Schmohl and company endured some technical issues as they built FlightPartner software and worked to raise it to the skies where it would eventually soar. Schmohl was committed to following the lean startup practices of Eric Reis, author of “The Lean Startup,” which redefined the roadmap for a startup tech company, Schmohl says. According to the book, one of the first steps in approaching a new business as a lean startup is to try to float a “minimum viable product,” or just enough product or service to test the market demand for it. “This was nearly impossible for us due to the complexity of the air charter marketplace,” he says, so Schmohl had to invest in a more technically developed product rather than a simple solution.

    In the private jet charter industry, with so much change, so many variables, so many players needing access and so many systems that FlightPartner would need to access, a minimum viable product still necessitated numerous external integrations and APIs. “Developmental collaboration to achieve this was difficult at best and developers often pushed code to production only to create more issues with each deployment,” Schmohl says.

    A second technical misstep—and in the absence of formal development structure—Schmohl’s team wrote code to solve specific functional problems one by one. However, the team found that a method for developing one function would conflict with a different method that was used to create another function, and so on. “The constant patching of the front and back end consisted of whatever worked. This code became challenging to manage and scale,” he says.

    Because of these two technical challenges, FlightPartner’s attempts at a minimum viable product and at following a lean startup approach soon were tangled in a mess of spaghetti code and unmanageable methods, Schmohl says.

    FlightPartner Development Pipeline

    Today, development at FlightPartner is much improved. The SaaS and its components start with a local build that the system pushes to a development server in AWS. FlightPartner developers use SourceTree, a Git client and the Git versioning system to manage software versions. FlightPartner uses quality assurance to manage development and to test code, and developers fix errant code and deploy it to AWS using Elasticbeanstalk. “This enables rapid builds and continuous deployment,” Schmohl says.

    Using a DevOps approach to development brings structure to an otherwise-highly dynamic marketplace. “Offering a SaaS solution as a marketplace presents challenges unlike a typical marketplace. Our inventory (aircraft) is fluid in that it moves often and the price changes depending upon location. This operational challenge translates to numerous and constant development with changes in software functionality,” Schmohl notes.

    These are the operational needs that drive software requirements; the developers’ goals increasingly are to meet those requirements so the software seamlessly supports operations. Development uses a priority status to identify and escalate the builds that are most critical to continued, smooth operational activity.

    Results

    FlightPartner’s development process has created a workflow process that minimizes the resources necessary to achieve operational results. Thanks in large part to DevOps and the cloud-based SaaS system, FlightPartner’s GDN has aircraft available for charter in six of the seven continents, including 35 percent of available aircraft located outside the United States, says Schmohl.

  • Legacy IT Challenges Continuous Delivery

    Legacy IT Challenges Continuous Delivery

    Legacy information technology infrastructure has a strong effect on the potential success of DevOps. In environments where different teams have always used the same servers, change can be difficult due to inter-team coordination and limited server transparency and application inventories, says David Blank-Edelman, a 30-year systems administration/DevOps/SRE veteran and currently Technical Evangelist at Apcera.

    “Unraveling the Gordian knot to bring DevOps principles and practices into that environment will be quite a challenge,” he says.

    In such a legacy environment, the enterprise walks a topological tightrope stretched between one or more development machine clusters shared by multiple teams with no real isolation between teams and projects, Blank-Edelman notes. “In most cases, there is a second parallel set of machines for production and, if the business is very fortunate, for test/QA as well. There is seldom anything, however, to prevent a production workload from talking to a database server in development by mistake, which can lead to all sorts of fun.”

    How to Move to CD

    One solution is a policy system approach, in which the organization provides for a policy system to migrate a legacy information technology environment to a platform that uses continuous delivery (CD), Blank-Edelman says.

    To create the policy system, he notes, the organization will need to address questions such as:

    1. Are the workloads allocated/using only the resources that they should?
    2. What is inside the workload? What is its provenance? Is it acceptable and in good order from security, coding and business requirements perspectives?
    3. Where is the workload allowed to run? In which cloud? In which environment? In which country? Near what other containers?
    4. Can the workload communicate with only the things it should? If I move it to another location, is this still true?

    “The process of developing a policy system and then enacting it can lead an organization on a journey that brings their existing infrastructure to a much better place. If that platform can also automatically enforce the decisions that organization made during this question and answer discovery process, you can be certain that the answers to these questions stay answered,” says Blank-Edelman.

    Other approaches to moving to continuous delivery include tool-first approaches in which the organization adopts one to several CI/CD/workflow tools and then makes an effort to implement those, possibly by shoehorning existing processes and runbooks into that tool or toolset.

    A better tactic is the people-first or culture-first approach that focuses on how the development and operations personnel will work together to create a product vs. a project, says Blank-Edelman. While tools still have a role to play in the people-first approach, they are not the priority; rather, organizational change comes first. Once the people have made the commitment and the change, everything else will follow.

    Challenges During the CD Move

    It’s not the existing information technology infrastructure that presents moderate challenges to CD that is the issue; it’s the infrastructure that poses either very many challenges or very few that is the hurdle. “If you start with a more challenging situation, this will incentivize people to work very hard to reduce the associated pain points in their life,” Blank-Edelman says.

    On the other hand, he notes, when an organization begins a move to CD with an infrastructure that is already in pretty good shape, you won’t often hear its people say that they should stop working on the product and work on CD all the time instead. People will still claim they have plenty to do already.

    Continuous Deployment Can’t Exist Without CD

    Continuous delivery starts with a code base and takes logical steps through that code base to increase confidence that it is ready for production, using a process that is as smooth, automated and loosely coupled as possible, he says. With confidence in that pipeline comes confidence in moving the result of that process into production automatically, which is continuous deployment, he adds.