devopsy.kinsta.cloud

Category: Features

JFrog, Puppet, Datical & Sendachi – Money Continues to Pour Into DevOps

The stock markets may be bearish, the price of oil is down the drain, but money continues to pour into the DevOps tools and solutions market. Just in the last few days 4 companies have announced significant new funding. Whether this is a last gasp before the doors slam shut, the tech industry thumbing their nose at the rest of the business world or proof that the recent market downtrend is a red herring I don’t know. But as someone who has lived through more than a few bubble bursts in the tech world it bears watching.

For now though, the good times continue to roll! Lets review the new funding news:

JFrog raises an astounding $50m! That amount of money is nothing to sneeze at! I spoke with Shlomi Ben Haim, CEO at JFrog. They have over 120 people working there in offices in Israel, HQ in Santa Clara and France. Their JFrog Artifactory and JFrog Bintray are hosting software for organizations the world over, big and small. Ben Haim and his team, as well as his blue chip investors believe the time is now as DevOps goes mainstream. They want to make sure they have the money to capitalize on the opportunity. Congrats to the JFrog team and we are excited to see what this money translates to for them.

Puppet Labs raises $22m in debt financing. While Puppet continues its phenomenal growth it took the opportunity to grab $22m in debt financing from Silicon Valley Bank. I spoke to Puppet CEO Luke Kanies about the raise and he told me that he is not even sure the company will use this money, as they still have a sizable portion of their last raise in the bank and are getting close to profitability. But at today’s low interest rates who can blame him for taking advantage of available cash? There was a time when debt financing was frowned upon. Why take on debt when you could give some equity for cash. But in today’s low interest rate/ high valuations world, why give valuable equity when you can get money for near free? In addition to the money, Puppet is also expanding their offices by about 37,000 square feet and have appointed financial veteran, Lou Lavigne to their board. This continues their preparation towards and IPO down the road. Of course with the market where it is, they may be better off staying private.

Datical raises $8m Series B round. Databases as part of the DevOps movement are going to have their day and Datical wants to be the company to make it happen. To do that they have raised a Series B led by S3 Ventures. They money will be used to continue development of their agile Database automation tools, as well as sales and marketing. I had a chance to speak with Ben Geller and Robert Reeves of Datical. One of the things they want to use the funds for is to expand beyond the North American market. That is a wise move given the economic environment and the reality of a global marketplace. There was a time when Internet companies could concentrate on the US and thrive, but the demographics just don’t support it anymore.

Finally, today marks the debut of Sendachi. Sendachi is a consulting firm dedicated to helping companies with their DevOps transformation. It was (initially anyway) formed by the merger of Seattle-based Clutch and London based Contino. Along with $30m dollars from Columbia Capital. The money will be used to expand the firms marketing and business teams. Some portion of it though will also probably be used to fund further acquisition. I had a chance to speak with former Clutch and now Sendachi CEO,Steven Anderson and former Contino founder and my friend Ben Wooten. They were both very excited by the potential of this Atlantic spanning venture. Anderson said that the $30m is just the start and if necessary more capital could be available for strategic purposes. My feeling is that we may be seeing the birth of a roll up in DevOps consulting practices in Sendachi.

In other M&A news on this topic, Docker announced it has acquired Unikernel Systems as well.

Congratulations to all of the companies raising capital and buying and selling. Anyone who has had to raise money or have bought and sold a company knows that it is not an easy thing to accomplish. Who will be the next DevOps company raising money? We probably won’t have to wait long to find out.

January 21, 2016
3 Tips to Avoid Painting Your DevOps into a Scalability Corner

When everything is firing on all cylinders, DevOps can be awesome. Teams work together seamlessly. Routine tasks are automated. Continuous delivery is continuously integrated, continuously tested, continuously deployed and continuously monitored. All is well, and right with the world—unless your DevOps needs begin exceeding the capacity of the platforms and tools you depend on. Scalability is a crucial element of DevOps success.

Wikipedia defines “scalability” as the capability of a system, network or process to handle a growing amount of work, or its potential to be enlarged to accommodate that growth.

The need for scalability is one of the primary reasons that DevOps and cloud computing are so integrally linked. While it’s possible to do DevOps on some level using physical hardware or virtual servers running on hardware in a local data center, it can quickly become impractical—if not impossible. Cloud computing using platforms such as AWS, Rackspace or Microsoft Azure enables an organization to increase server capacity exponentially at the push of a button.

“Many of the issues and challenges experienced before the advent of cloud are no longer issues: Engineers now working on cloud implementations remember working at companies that feared the Slashdot effect—a massive influx of traffic that would cause servers to fail,” explains Jason McKay, VP of Engineering for Logicworks, in a blog post.

Tackling Scalability

There are a number of factors impacting scalability in your environment, which could impact the success of your DevOps. Here are three things to keep in mind to ensure your environment is ready to scale:

1. Make sure your tools and apps are scalable

The ability to scale capacity on demand—automatically—is one of the chief advantages of cloud computing, and one why cloud computing is so important to DevOps success. There is more to scalability, however, than just server and network capacity. You must also ensure that the tools you rely on can seamlessly scale with the infrastructure, and that the apps you’re delivering through your DevOps environment are designed to take full advantage of the increased capacity as well.

2. Continuous optimization

Some tools and apps will continue to run as the underlying infrastructure scales up or down, but may not run optimally. Make sure that the tools you use and the apps you deploy are able to continuously adjust to changes in server, bandwidth and storage capacity. The tools and apps should be able to take full advantage of the resources available to ensure the best performance possible at all times.

3. Don’t forget about storage

Most apps have a data component and that means there is data storage that must be scaled as well—either as a function of the broader network infrastructure scaling or just in response to increased demand for data storage.

Dan Leary, VP of Products, Solutions and Alliances at Nimble Storage, believes that scalability is emerging as a differentiating factor between storage solutions. “Due to the high performance offered by many vendors of flash-optimized storage solutions, performance will no longer be the primary differentiating factor when considering one vendor over another. Instead, a much greater focus will be placed on the array’s reliability, data integrity, scalability, manageability and in-built data-protection capabilities. We are now getting to the point where performance in storage has been democratized and will change the way people purchase storage.”

There is one more thing to remember when considering scalability: cost. While it may be easy to automate scalability on cloud platforms, it’s not free, and the DevOps tools you depend on may include licensing fees that could be impacted as your usage scales, so keep that in mind.

January 19, 2016
Kicking Off InterConnect 2016
Our InterConnect 2016 coverage kicks off with online events

Once again this year devopsy.kinsta.cloud will be providing extensive coverage of InterConnect. InterConnect 2016, features hundreds of sessions and tracks around DevOps. We think InterConnect offers more DevOps related tracks than any conference in the world.

We are kicking off our InterConnect 2016 coverage with some great online events leading up to conference which takes place on February 21-25 at Mandalay Bay and MGM Grand in Las Vegas. You don’t have to go to Vegas though to participate and attend these events though. Mark your calendar for these:
- Twitter Crowd Chat, DevOps: Outthink Delivery January 22^nd, 1 pm EST. Join me and some of the clients, partners and IBM experts for this free-form, lively twitter conversation on how to deliver innovation and start-up performance at enterprise scale.
- Google Hangout #1: What Will DevOps Look Like in 3 to 5 Years, January 27^th, 1 pm EST. This is a preview of a panel I am moderating at InterConnect featuring Gene Kim, Dan Berg and Raghunath Raman.
- DevOps Chat: DevOps for multi-speed IT and DevOps & Open Technologies IT, January 21^thand February 8^th. My interview with IBM Distinguished Engineer, Rosalind Radcliffe and Dan Berg. This will be a recorded podcast that unfortunately will not offer audience participation live. However, if you have any questions that you would like me to ask Rosalind or Dan, please leave them in the comments on this article.
- Google Hangout #2: DevOps for multi-speed IT, February 3^rd, 4 pm EST. Join Sanjeev Sharma (IBM ), Carmen DeArdo (Nationwide), Brad Schick (Skytap) and myself as we discuss Bi-Modal IT and DevOps. Link will be available shortly.
- Google Hangout #3: Innovate Like a Startup, Scale for the Enterprise, February 10^th, 1 pm EST. Join Gene Kim, IBM VPs Angel Diaz and David Lindquist and another special guest (still confirming) with me moderating for our last pre-InterConnect event. Link to be provided.
We will also feature more related content leading up to InterConnect in a dedicated section of the site, DevOps@InterConnect.

At the conference itself we will be on scene with live coverage, video interviews, blogging and tweeting, so stay tuned.

In the meantime please join us for these great upcoming events.
January 18, 2016
3 Ways to Develop the Perfect DevOps Culture

DevOps is not a new concept, but it is still maturing.

In the seven years since the first “DevOps Day” took place in Europe, DevOps’ influence has spread to every corner of the world. Many organizations have embraced the concept, and it’s changed the way business is done. These companies have benefited from removing the gulf between development and operation teams—we have seen companies consistently churn out better software, in shorter time periods, to achieve new levels of efficiency.

However, this success has not been replicated for all sectors. Federal agencies, for example, have been slow to adapt to DevOps. For other companies, a rush to implement DevOps has led to ongoing issues that prevent them from realizing the true benefits of DevOps.

Building a DevOps Culture

Many companies are moving in the right direction to implement DevOps successfully, but critical barriers still remain in developing the perfect culture. Here are three ways you can make the vision of DevOps a reality:

1: End the Culture of Fear

Much like previous aversions to cloud, the biggest barrier to implementing DevOps is the fear of change. This is a common problem across sectors, because trying something new can be daunting. This is true for federal agencies protecting national security or private companies attempting an ambitious new project. Successful DevOps begins with buy-in from the highest levels of management—but how do you get them on board?

The best approach is to start small by moving forward with a low-risk, high-reward pilot program. This is an opportunity to get the development and operations teams in the same room and allow them to work together toward a common goal. Once executives see the positive results—reduced time-to-value, efficiency gains and cost savings—they’ll be more apt to double down on establishing a companywide DevOps culture.

2: Find and Establish Talent

A DevOps culture is different and requires a shift in mentality. It also requires talented people with unique skills and experience. Since DevOps is still a relatively new concept—even by technology standards—talented people are scarce and in high demand. But companies can bridge the talent gap through a targeted strategy that incorporates both recruiting and training.

We are starting to see more companies provide educational support programs to develop talent. That must continue if companies want to build the next generation of DevOps leaders. These programs allow employees to grow their skills to better meet the needs of DevOps, which will fuel their career growth while helping their company improve its efficiency.

In parallel with training, companies must aggressively recruit tech talent from top universities that have DevOps classes. Recruitment of tech talent has rarely been more fierce, and it will only intensify as more federal agencies implement DevOps and compete for talent. The commitment from federal agencies to emerging IT strategies, including DevOps and agile development, will make them far more attractive to the best STEM talent.

3: Measure, Measure, Then Measure Again

The ability to deliver impactful ROI metrics is one of the best aspects of DevOps, because it removes all the guesswork whether it is making an impact. The numbers will tell you daily.

The first step is to measure your current metrics, whether you’re struggling to make DevOps work or implementing it for the first time. Companies must know how efficient—or inefficient—existing software development processes are before beginning to change. With the right talent in place with the right frame of mind, it’s time to track the improvement.

Also, understand what results to expect and plan accordingly. Based on our experience, deployment should be eight to 10 times faster with 50 percent uplift in quality (measuring reduction of incidents and incident duration), while software builds should be three to four times quicker.

If you properly set up the perfect DevOps culture, the results can be astounding. We’ve seen a company institute a DevOps structure and reduce release windows from two days to two hours, simply through repeatability and scalability.

The Goal: Break Down Barriers

Despite DevOps rapidly growing in importance, questions remain from those skeptical about its effectiveness. It reminds me of a line I hear frequently at conferences and shows that always gets a laugh: “To err is human. To err on 10,000 servers in 10 minutes is DevOps.”

The truth, however, is that DevOps is a very simple concept about working together and working smarter. It is the realization of a more efficient, more productive organizational model. It’s time for all companies to break down barriers, create a modern culture, and deliver better software in shorter cycles.

About the Author/Raj Ananthanpillai

Mr. Ananthanpillai is the Chairman, CEO and President of InfoZen and has been with the company since January 2004. Previously, he served as the Chief Strategy Officer of ePlus, a business process automation and financial services company, responsible for technology, marketing, analyst relations and business strategy. Prior to that, he was the president and CEO of NetBalance, a venture capital-backed multimillion-dollar software company, which was successfully sold in 2000. You can connect with Raj on twitter or on linkedin.

January 15, 2016
Boston DevOps Leader Pete Cheslock Talks Accomplishments

Pete Cheslock began his leadership role in DevOps in 2009 at cloud-based email archiving company Sonian. The company was founded by friends and former colleagues of his, who called Cheslock for help when they got funding for the Sonian venture. He filled in wherever they needed him—as did everyone else in the first few years of the 10- to 20-person startup.

“As we grew, I saw a need for leadership for Sonian’s five-person operations team. I went to our VP of Engineering about the need for ownership of operations and said I’d love to be the one to give it a shot,” says Cheslock.

Sonian was an early user of Amazon Web Services, deploying production code with customers by 2008. Sonian had been named an AWS Startup Challenge finalist, but despite its forward-thinking approach to the cloud and infrastructure as code, many of its processes were immature.

“The application had scaling issues and we were spending an inordinate amount of time on reactive measures, dealing with application downtime and using short-term fixes for continuing issues when I took over,” says Cheslock. The young DevOps shop wasn’t able to make headway in innovative, future-looking projects.

Smothering Fires, Forging Ahead

Cheslock transitioned its team into two groups—a proactive group for innovative projects with a view to tomorrow, and a reactive team to get the worst recurring fires under control. Cheslock added himself to the latter group while remaining head of Operations. “While we attacked the hairiest issues, the proactive group worked on tooling that we needed in support of CD (continuous development) and CI (continuous integration), which made deployments much easier,” says Cheslock.

It had been taking the Sonian team about a week to deploy software for all its customer environments. The process of deployment affected much of the software stack; it required testing as well as completing and checking off several items on a lengthy deployment list.

Cheslock directed the proactive team to automate much of that checklist. Soon, they had reduced deployment time to 20 customer environments in just three hours, from five three-hour periods over five days.

Weeding Out Costly Cloud Issues

Cheslock and Sonian next began to work on cost issues. “We built an application to analyze our Amazon usage and correlate it with our Chef server, which was our single source of truth for all instances,” says Cheslock. The app revealed unused and abandoned servers that had been used for development and testing, which were still running and costing the company money.

“We found occasional spot instances where we could pay fractions of the retail price for batch workloads to cover these. We leveraged the Amazon Marketplace to make better purchases of infrastructure to save money,” says Cheslock.

Cheslock was able to shrink 800 instances on a single AWS cloud with a monthly spend of $450,000 down to 400 instances across four public cloud environments with a much smaller price tag of $200,000 a month. He accomplished this in three months, even as Sonian’s cloud operations team doubled.

Now senior director of Operations and Support at Threat Stack, Cheslock is nurturing DevOps methodologies and seeing continued growth and success there.

BosOps

A few years ago, Cheslock and a few friends started a Boston-area technical operations meetup called BosOps, inviting people who managed operations at various companies to get together regularly to discuss their challenges and offer support.

“As BosOps grew, we wanted to reach a broader audience and started pulling in more people in operations or DevOps roles to gather on a quarterly basis,” says Cheslock.

Currently, BosOps has about 150 participants, who also participate in online chats using the free tier of Slack.com, a popular team-working messaging app among DevOps folk. “It’s great for people who are new to Boston,” says Cheslock.

BosOps helps by quickly connecting a professional network for people who are new to the industry or to the area, enabling them to meet like-minded people and to secure a better foothold—perhaps even find a job they like more than their current position.

January 14, 2016
Jenkins Founder Kohsuke Kawaguchi Talks DevOps Leadership, Part 2

In part one of this two-part series, we covered the early work of Kohsuke Kawaguchi, founder of Jenkins while he was at Sun (now Oracle) and also how he addressed the project’s challenges to growth by applying extensibility/plugins, incentivizing sharing in the community, and removing barriers that can result from apprenticeships. Now let’s continue as Kawaguchi shares the rest of his story.

Explosive Community & Tooling Growth

Today the Jenkins Project and its open source community boast more than 120,000 members and 1,000 plugins. It was by design that Kawaguchi spurred that growth. Here are the challenges he faced, his solutions and his good fortune.

Though continuous integration (CI) and continuous delivery (CD) have sprouted an immense amount of automation in DevOps, it was necessary to have diversity in the software development environment/toolkit to ensure that each developer and team have the tools best-suited to their goals and uses.

“This is where our extensibility features came in handy—by letting people reuse what they could and build the missing pieces. As developers, we prefer flexible building blocks over rigid software that does things in just one way,” says Kawaguchi. This diversity encouraged community growth by removing a tooling stumbling block and brought even more plugins to life.

Jenkins emerged at a time when IT began embracing open source projects and, as a result, benefited from that momentum. “Companies recognize that it’s better in a lot of ways to rely on open source software than proprietary tools. It gives them more control,” he says.

Jenkins as a Key Component for Build Orchestration

The DevOps movement has helped Jenkins become a key component of helping teams orchestrate their builds. From its roots of using time-based activity triggers and requiring a developer complete “this task, then that one” in a cascading, triggering fashion, Jenkins has moved on to model what people do and then pull that work into Jenkins as automated functions.

The automation capabilities of Jenkins kept expanding and Jenkins entered what Kawaguchi calls the “orchestration-idioms-as-plugins” era, with different people in the community developing different patterns for orchestrating/automating activities. “I created one such plugin called the Promoted Builds plugin, which basically was a generalization of how different teams collaborated within Sun to release software,” says Kawaguchi. By creating these idioms within the model of executable plugins that others can adopt, the Jenkins Project fostered even more efficiency in sharing, adopting and reusing proven orchestrations.

As automation spread from builds to tests, deployments and releases, these orchestrations gained new importance but presented still another challenge, which a plugin also solved. “Idioms were useful but different idioms created by different people weren’t really composable,” Kawaguchi said. “In recognizing that, we’ve built the Workflow plugin, which creates a single framework to describe any orchestration, whether simple or complex. We can share and compose these fragments.”

Kawaguchi’s Leadership Accelerates DevOps Acceptance

As both a leader of the Jenkins open source community and the CTO of CloudBees, Kawaguchi has helped the open source community and enterprise teams alike wade in to DevOps. The open source Jenkins ecosystem helped to move automation ahead. It gave developers a fast track to useful, functional software that would please ops and enter production quickly. At the same time, it enabled developers to innovate swiftly and improve on that same software to see more utility and recurring rewards.

“Now with CloudBees, we have allowed the project to have a number of full-time engineers, which increased its velocity. It also brought commercial support, an enterprise version of Jenkins, professional services and training,” Kawaguchi notes. “It made Jenkins more viable in the mind of large-scale conservative enterprise users who value those things and allowed Jenkins to be deployed at scale in those organizations.”

January 11, 2016
Machine Learning, AI Driving DevOps Evolution

Automation is the fuel that drives DevOps. Automating routine, repeatable tasks is one of the defining characteristics of DevOps culture. As artificial intelligence (AI) and machine learning improve, the scope and complexity of the tasks that can be automated increases, which raises the bar for all of DevOps. The humans behind the DevOps will be freed from even more mundane tasks and able to focus on more innovative and creative endeavors.

It makes sense, if you think about it. The trend from physical services in an on-premise data center, to virtualization, to cloud services, to DevOps has been a steady march of raising the bar for automation. The more applications and entire infrastructures can be developed, deployed, monitored and managed programmatically and through automation, the more efficiently and effectively an organization can function.

As tasks are offloaded to automation, humans can address other challenges. That focus inevitably leads to identifying other new tasks that can be automated. AI and machine learning represent a quantum leap in this evolution, however—with AI and machine learning, the automation can begin to identify and improve itself. When the automation can identify bottlenecks and inefficiencies and automatically adapt to overcome those obstacles, human input and interaction could become almost obsolete.

In a recent interview, Christian Beedgen, CTO of Sumo Logic, shared some thoughts about the impact of machine learning on DevOps. “Machines will enable greater productivity among teams and businesses through the advances that machine learning and automation can offer,” he said. “A necessary component to augment human contributions, machine learning will finally give humans the power to review millions of bits of data.”

Automating routine tasks is crucial, but there’s another factor that plays into the role that AI and machine learning have with the future of DevOps. The reality is that there are some things humans simply can’t do as well or as fast as machines—especially at scale.

Steve Burton stressed in a blog post that humans are capable of observing, digesting and interpreting only so much information at one time. Some feel like the Holy Grail solution is some sort of application performance management (APM) dashboard that can aggregate and correlate real-time data so human IT personnel can monitor metrics and identify issues manually. “This all sounds good in theory, except it’s completely unmanageable in reality when your applications have thousands of components, billions of metrics and hundreds of changes every day,” he wrote. “Humans can no longer cope with this complexity, they now need machines to do the leg work, process this Big Data and provide operational insight into what the f**k is going on.”

Beedgen agrees. He suggested, “Increasingly, businesses will adopt machine learning technology for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”

On some level this all sounds a bit scary. It seems as though the more we succeed at DevOps and the more DevOps evolves, the less humans are even a necessary part of the equation. Conceivably you could be so good at DevOps that you DevOps your way right out of a job or inadvertently launch SkyNet—or both.

That scenario is simultaneously feasible and far-fetched. I suggest that you don’t fear that future or look at it as machines stealing your job or taking over humanity. Embrace the future of DevOps driven by AI and machine learning and find the silver lining. Seek out the new frontiers and horizons, because there always will be tasks that the machines simply can’t do without us—and that is where innovation happens.

January 6, 2016
3 Ways Manual Processes are Risking Your Development Cycle

During my two decades in IT, I’ve seen the demand for Agile and its associated methodologies grow. At the same time, however, I’ve witnessed the database being left behind.

DBmaestro, where I am co-founder and CTO, recently conducted a survey titled, “Database Deployment & Development Risks,” to gain insights into the risks associated with various database development practices. The survey focused on manual processes used throughout the development cycle.

The results revealed two divergent realities: Despite respondents, from diverse sectors, overwhelmingly recognizing the risks associated with manual process for the database, many organizations continue to use processes and solutions that don’t fully automate the development cycle.

One area where this was particularly apparent was with the source control. It’s imperative that development teams are able to trust their source control. But even when standard source control solutions are being utilized, 70 percent of those surveyed are still interjecting manual processes.

What that means is that even though many of these professionals have some type of solution, they still see a clear and present risk because:

1. When integrating changes, it’s possible for one developer to wrongly override valid changes with older revisions from other development branches, a potential concern that nearly 90 percent of survey respondents associate with risk. Still, more than half (53 percent) of the IT professionals surveyed integrate changes, creating the possibility that such overrides may occur.

Nearly three-fourths (70 percent) reported having a database source control, but 53 percent reported that they continue to have problems that a more adequate source control solution should resolve. In this case, there is a problem with the source control tool implemented.

2. The database may not be in sync with the version control repository. Ninety percent of the IT professionals surveyed associated the potential for these sources to be out of sync with risk. What’s more, 72 percent of those surveyed admit that their database may not be in full sync with the version control repository.

3. Introducing changes without first verifying each database object vs. its version control revision to ensure each database object is up to date is a process that nearly 90 percent of survey participants associate with risk. Despite a widespread awareness that failing to verify accuracy between database objects and database source control before introducing changes is a risky practice, 53 percent continue to do so.

That means that the remaining 47 percent of respondents do not trust their source control solution regarding the database code and are implementing steps to validate it matches the database before introducing changes.

While interjecting manual steps can help alleviate the risk associated with the possibility of overriding critical changes in the database, it’s time-consuming, and manual interruptions to the continuous delivery cycle introduce risks of their own. When you can’t place full trust in source control, two developers working on the same database code or objects can easily override or revert critical changes introduced by the other developer when the script is updated in the database—sometimes with disastrous consequences.

Only a Database Enforced Source Control solution that covers database lifecycle management (DLM) from development through build and deployment can create a “Single Source of Truth” for your database development assets. This allows you to gain trust in your source control and implement the rapid changes today’s development cycles demand—without plugging the process with time-consuming manual steps. Database Enforced Source Control promotes complete trust in your source code, allowing developers to work simultaneously on the same database objects without the risk of overriding important changes when scripts are updated, promoting true synchronization across dispersed development teams.

Download the complete results of the 2015 “Database Deployment & Development Risks.”

About the author:
Yaniv Yehuda is the Co-Founder and CTO of DBmaestro, an Enterprise Software Development Company focusing on database development and deployment technologies. Yaniv is also the Co-Founder and the head of development for Extreme Technology, an IT service provider for the Israeli market. He was a captain in Mamram, the Israel Defense Forces computer centers, where he served as a software engineering manager.

January 6, 2016
Why You Need to Embrace Automation

Just in case you hadn’t noticed: The robots are coming.

Well, you probably had noticed. It’s hard to miss. Newspaper articles and blogs everywhere herald their arrival. The stories usually follow the same general formula: A catalogue of amazing things that robots (really, AI systems) can do. Debate about forced unemployment vs. people freed up to do more interesting work. And then some comments about whether it’s for the good or bad. You can generally infer the general tone from the choice of image accompanying the story: a cute Japanese robot with a plastic smile or a more sinister face with the Terminator’s jawline.

Of course, workplace automation has been around for almost as long as we have had workplaces. Each new wave of technology disrupts the employment landscape in its own particular way. However, this time it’s serious: It’s not just manual labor that’s under threat, but increasingly, we’re told, white-collar workers are being supplanted by machines. A recent McKinsey report has generated a wave of headlines: “Even the CEO’s job is susceptible to automation,” wrote Patrick Nelson at Network World; and Ana Swanson at the Washington Post reported, “How robots will even affect the jobs of people we thought were immune.”

Not surprisingly, a closer read of the McKinsey material gives a more nuanced appraisal than a headline can offer. Few of us will actually be replaced by a robot, but many workers will lose some part of their current jobs to automation. The McKinsey report states that up to 45 percent of the activities that employees perform today could be automated, but very few occupations will be automated entirely in the near or medium term. The things that require creativity and emotion remain uniquely in the domain of humans, and they are likely to remain there for quite some time.

It’s against this background that we see a steady increase in what is often referred to as robot journalism: machines writing news stories. So far, these applications rely on scenarios where the general ‘shape’ of a story remains pretty consistent because of the regularity of the data underlying those stories. For example, a report on stock performance is always going to mention the current stock price, what the stock price was at some particular point in the past, the price highs and lows, and perhaps something on volume or analyst sentiment. These reports often are found in the back pages of a newspaper, and are the stories that few journalists actually want to write, since they allow almost no space for creativity and emotion.

The reality is that today’s news-writing software is still a long way from replacing human journalists. Which is not to say that what a robot journalist produces isn’t useful; it surely is. Last month I spoke about robot journalism at the news:rewired conference. It was fascinating to see how most of the major news media are exploring how to get robots on staff. So while we’re going to see more robot journalists, it doesn’t necessarily mean fewer human journalists. A recent study by researchers at Oxford University and Deloitte explored how susceptible various jobs were to automation—journalists ranked 285th out of 366 jobs, with an ‘automation risk’ of just 8 percent.

But if you really want to see where the action is regarding automated writing, just look at it from a different angle. Rather than Natural Language Generation displacing people from their jobs, it’s offering a solution to the problem of shortage: A shortage of workers or expertise. And that story isn’t about news; it’s the Big Data story. You can take issue with the astounding numbers that are thrown around, but even if they’re out by a factor of 10, they’re still significant: by 2020, about 1.7MB of new information will be created every second for each person on earth. But less than 0.5 percent of all data is ever analyzed.

In 2011, McKinsey noted, “The United States alone faced a shortage of 140,000 to 190,000 people with analytical expertise, and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis of big data.” And subsequent reporting points to massive year-on-year leaps in demand.

Regardless of how few or how many people are displaced by robots, they are not all going to become data scientists. The only way this volume of data will be analyzed and explained is if we have machines doing it for us. And that’s the real calling and promise of Natural Language Generation (NLG). Interactive visualizations and dashboards don’t cut it; the meaning in all that data will be lost unless we call in the robots. But you need more than robot journalists, tirelessly churning out one formulaic story after another. You need articulate robots that use sophisticated NLG to construct fluent narratives from idiosyncratic data sets. So, accept the data reality, embrace automation, and find an NLG robot to explain the insights buried in your data.

About the Author:

Dr. Robert Dale, Chief Technology Officer and Chief Strategy Scientist at Arria NLG, is recognized as one of the world’s foremost experts in Natural Language Generation (NLG) research and Development, having authored or edited seven books and 160 papers on computational linguistics. He was a Professor in the Department of Computing, Director of the Centre for Language Technology at Macquarie University. He co-authored the seminal textbook “Building Natural Language Generation Systems” with Arria NLG Chief Scientist and co-founder Professor Ehud Reiter.

January 6, 2016
Big Room Planning: Getting Everyone on the Same Page

Many organizations—or even teams within organizations—are attempting to embrace DevOps. The operational and strategic advantages of DevOps over traditional business methodologies make it a virtual business imperative. One of the prevailing ideas about DevOps is that it’s about a culture shift more than tools and principles—and a big part of that culture shift is making sure everyone is on the same page.

Rally Software—which was acquired by CA in May of 2015—provides software and services to help customers succeed with Agile development. One of the main components of the services side is to educate customers about the importance of working in harmony and guiding customers through a process called Big Room Planning. DevOps is a natural extension of Agile development principles to the rest of the business, so Big Room Planning can be an effective tool for DevOps just as it is for Agile development planning.

I spoke with Steve Wolfe, director of Product Marketing for Rally, who explained the goal of Big Room Planning is to teach customers how to be more collaborative and help them get to a position where they’re able to respond faster to customer demand or market pressure. The idea is to foster an environment of collaborative planning that brings in the whole organization.

“At Rally we’ve learned that delivering the most value from your software usually requires multiple, coordinated development teams—often distributed—working together and with the business,” Wolfe noted in a blog post. “Building software at this level is complicated, with a lot of moving parts, dependencies and competing priorities.”

Many organizations—particularly organizations that already are familiar with Agile development—understand the value of consistent feedback and frequent iterations. Collaboration is a keystone of Agile. However, when you take that to a broader scale with DevOps, it becomes critical that all parties impacting—or affected by—the project be represented in the collaborative effort.

Wolfe noted even in companies that use Agile development effectively, problems arise when teams work in isolation. Teams create plans that they agree on in their group, but aren’t realistic or don’t align with broader business goals. Meanwhile, other teams or departments don’t feel committed to plans they had no part in creating. The result is more like the traditional business model than DevOps: You end up with a situation in which any one team can halt progress and create a bottleneck.

The Big Room Planning approach is intended to address that problem. The entire organization must be engaged, with full support from all of the affected roles and departments in the planning process. As the name implies, you need to get everyone together in a big room (or virtually via web conferencing, if necessary) and collaborate on a grander scale to make sure everyone agrees with and is committed to the plan.

Wolfe also stressed that leadership support is essential. He recommends finding someone to champion the concept—an executive to lead the Big Room Planning effort. In and of itself, this idea doesn’t seem all that revolutionary; common sense dictates that development will flow more smoothly and business will function more effectively when everyone is on the same page and working toward the same goals. The crucial part is to not take it for granted and to use a strategy such as Big Room Planning to ensure more effective cooperation and collaboration.

January 5, 2016