devopsy.kinsta.cloud

Category: DevOps Practice

  • How to Pivot and Join Time-Series Data in Flux

    How to Pivot and Join Time-Series Data in Flux

    Merging multiple time-series in InfluxDB using Flux may seem like a daunting task. This in-depth Flux tutorial will explain how to pivot, time shift and join time series in InfluxDB.

    In this InfluxDB tutorial, we will show you how to create an accurate overview of multiple time series with industrial data captured in consecutive production steps. Doing this requires a proper join mechanism for combining the time series into one after aligning them with respect to time.

    This tutorial is based on a real-world case we solved at Factry.io with InfluxDB in an industrial setting.

    Every second of the day, production sites all around the world are generating inconceivable amounts of process data. This data is directly acquired from production lines on-site and captured in the form of process parameters. Being able to reliably keep track of these process parameters throughout the production process is essential to continuously improving manufacturing operations.

    Industrial process parameters are represented as concurrent time series. They do not necessarily provide data points at a similar pace, nor are they always aligned over time. Obtaining a trustworthy overview of captured process parameters during consecutive production steps can be daunting.

    Time-Series Data Explained

    In brief, time-series data represents a number of different data points indexed by time. As time is a constituent of anything that is observable in our world, time-series data is everywhere. When narrowed down to a production environment, captured process parameters are distributed in time in order to become distinguishable from the process data preceding or following it.

    What is the difference between time-series and relational data? In contrast to time-series data, relational data holds records that are generally referenced by an artificial identifier. These identifiers can interrelate, which crafts the relational aspect.

    Pivot and Join Time-Series Data in Flux

    Before you get started

    On a production line, be aware that a product batch cannot be in two different production steps at the same point in time. As a result, time series measured in different production steps are naturally unaligned. Hence, before properly joining these time series into one, many of them will need to be shifted over time to get the overview right.

    Furthermore, retrieving data stored in a time-series database is commonly done by using the dedicated query language Flux.

    Two points about this:

    1. In Flux, a join can only make use of an inner join mechanism to melt multiple time series into one. As an inner join solely retains the data points for shared timestamps in multiple time series, data loss is unavoidable when dealing with unaligned time series or time series captured at different rates.
    2. In most use cases, only time series extracted from sensors at different production steps need to be joined. This is due to the ability to differentiate between multiple tables within one time series.

    Hence, reading in sensor data generated by multiple sensors at the same production step, table by table, allows us to bundle the sensor data. By ‘pivoting’ the tables into one, a time series is forged. Prior to being joined with the sensor data captured from another production step, the time series is shifted to align with respect to time. However, note that this requires all sensor data (potentially from different sensors) from one production step to be aggregated in the same way, omitting any differences in the rate at which it has been captured.

    Use Case: Welcome to the Cardboard Factory

    There are some best practices for pivoting and joining time series data in Flux from consecutive production steps. In this example, sensor data is captured from a replicated production site that is manufacturing cardboard panels out of recycled paperboard rolls.

    The production process of the cardboard factory consists of three different steps, which we’ll refer to as entries A, B and C.

    • In entry A, production starts with recycled paperboard rolls; the width and diameter of which is being measured by two separate sensors.
    • In entry B, the recycled paperboard rollsare cut into paperboard sheets and processed into cardboard using a certain temperature and pressure.
    • In entry C, the cardboard is cut into shape. For this entry, the height and the thickness of the panels are the key elements to be measured in terms of sensor data.

    With respect to time, it takes approximately one minute for the first sheet of recycled paperboard roll to reach entry B. Further, about two minutes elapse before the paperboard sheet is processed and shaped into a cardboard panel.

    Note that, in this production site, it is assumed that the speed of the production line is more or less fixed over time, making it possible to shift time series captured in different entries in order to get them aligned.

    Merging Multiple Time Series Into One Using Flux

    The goal in this use case is to create one trustworthy overview of all sensor data captured in each entry. Imagine today is May 4 at around 3:10 p.m. The overview should be reflecting all sensor data from the past ten minutes of the production site.

    Fortunately, we are aware that the sensor data between each entry is unaligned by a few minutes. Therefore, we should be fetching data between 3:00 and 3:10 with respect to the last entry (entry C); thereby trying to align these time series with the ones from both entry A and entry B.

    Think about it as being teleported to 3:10 on May 4. At that time, for a recycled paperboard roll in entry A or for a paperboard sheet in entry B, we do not yet have the sensor data for entry C. This is obviously because this sensor data will be captured only a few minutes later on the line.

    With this cleared up, a recycled paperboard roll delivered by one of the suppliers is referred to in the use case as a material lot. In contrast, a paperboard sheet derived from the recycled paperboard roll resulting in a cardboard panel is mentioned as a material sublot.

    At the end of the production line, a pile of cardboard panels is manufactured out of one material lot. To sum it up: In this use case, a material sublot can never be in two different entries at the same point in time, which means we need to align time series over consecutive entries.

    1. Reading in the Sensor Data

    First, we start by reading in the sensor data of both the width sensor and the diameter sensor for entry A. By reading in sensor data of both sensors in the same ‘filter’ (image: code snippet), we reduce this data to one time series for entry A.

    In this way, we hold different tables per sensor in one time series arranged by the group key consisting of ‘_field’ and ‘_measurement’. By reducing all sensor data from one entry to one time series holding multiple tables, the opportunity arises to pivot the tables into one unified table later on.

    Fetching the corresponding sensor data from entry A with respect to entry C is fulfilled by using a ‘range’ with sensor data that was captured between 2:57 and 3:07. This is applied by shifting the time range of entry A three minutes backward using ‘SHIFT_A’. Hence, for a cardboard panel at entry C, the corresponding sensor data at entry A was captured three minutes earlier.

    In essence, we are mapping the time range between 2:57 and 3:07 of entry A onto the time range between 3:00 and 3:10 of entry C and, thus, onto the overview.

    Note that for operations between a timestamp and a duration in ‘FROM_A’, you need to convert both the timestamp and the duration to an unsigned int to get Unix timestamps in ns prior to performing the calculation.

    Entry A sensor data – code snippet

    Entry A sensor data

    Regular and Irregular Time Series

    In most use cases, the sensor data will generate a regular time series (metrics), as the time period between captured data points will be fixed. However, when working with irregular time series (events) or time series generated at a different rate, these need to be transformed into a regular time series with an ‘aggregateWindow’ to eventually build up a nice overview.

    In this use case, there is only a regular time series. Therefore, retaining the ‘last’ data point for each time window of one minute should do the trick. In the case of an irregular time series, taking a ‘mean’ over a fixed time window would be the better choice.

    2. Pivoting the Time-Series Data

    After the sensor data from both sensors of entry A is collected into one time series with multiple tables, we now need to pivot the tables to become one. Therefore, the group key of each table needs to hold the same schema.

    In this use case, the group keys schema for all tables in the time series is as follows: ‘[‘_field’, ‘_measurement’]’. To make sure that the group keys schema is identical for all tables, we can use a ‘keep’ in front of the pivot to eliminate fields that are not present in all tables.

    Possible Deal-Breakers

    Bear in mind that the constraint for holding an identical schema for each table in the time series can be a deal-breaker for using the pivot. Another constraint lies in the lack of being able to pivot more than one value field, although when reading in sensor data from InfluxDB, there is typically only one value field.

    However, after adding some processing to the time series in Flux, chances are that an additional processed value column is present, eliminating the ability to use a pivot once more.

    When performing a pivot on multiple tables in the time series of entry A, for the table rows holding a shared timestamp (row key: ‘_time’), the data point in the value field (value column: ‘_value’) will be ingested into a column determined by the name of the measurement (column key: ‘_measurement’).

    In the use case, this means that the tables with the measurements ‘lot_ID’, ‘A_width’ and ‘A_diam’ will get their data points from the ‘_value’ field ingested into a separate column in the unified table, determined by their measurement names.

    In terms of performance, a pivot can suffer from having too many tables in the time series to be pivoted, which corresponds to the number of active sensors in one entry. Next to that, an extensive group key schema in the pivot will also take its toll. Having a considerable amount of rows per table in the time series of an entry with ns precision timestamps can easily blow up the pivot, as well.

    Finally, a small footnote on the functionality of the pivot in Flux. Keep in mind that the fields and tags that are not included in the pivot will not be ingested into the unified table. Additionally, the term pivot appears deceiving when being compared to its functionality in a relational database.

    Entry A pivot – code snippet

    Entry A shift

     

    3. Time Shifting the Time Series Data

    Entry A

    After pivoting the sensor data of entry A into one unified table, the entry A sensor data with time range between 2:57 and 3:07 gets aligned with the time range of the overview between 15:00 and 15:10 by performing a time shift of three minutes forward on the _time column.

    To preserve the original timestamps of entry A, we take a duplicate of the _time column in the unified table and refer to it as the time for which a sublot actually started at entry A, mentioned as entry_A.

    entry A shifted – code snippet

    Entry A Shift – Code Snippet

    Entry A Shift

    Entry B & C

    The same procedure for reading in the sensor data of entry A applies for entry B and entry C. While the sensor data for the former is shifted by two minutes to get its time range between 2:58 and 3:08 aligned with the time range of the overview, the sensor data of the latter is already in sync for the overview.

    Entry B – code snippet

     

     

    Entry B

    Entry C – code snippet

     

    Entry C

     

    4a. Joining the Time Series

    After pivoting the sensor data for each entry, three distinct time series are held, each containing one table with a separate schema and multiple value columns; for example, ‘C_height’ and ‘C_thickness’ for entry C. To create one overview, these three time series must be joined together, visualizing the data points of different sensors side-by-side with respect to time. Since the sensor data of the different entries has been formerly aligned by reading in the corresponding time ranges for each entry followed by a ‘timeshift’, the three time series can now be joined on the synced ‘_time’ column present in each of them.

    Possible Issues

    Since the schema differs between the three time series tables, you would encounter significant overhead when trying to pivot these three tables into one after appending them to a single new time series. Besides the superfluous complexity added, the tables of all time series are each holding more than one value column, which makes the pivot unusable in this case.

    A plausible solution would be to use a ‘join’ provided by Flux on the aligned ‘_time’ column. As discussed in the pivot and join section, the ‘join’ is represented as an inner join mechanism, solely retaining data points for shared timestamps between joined tables.

    In this use case, the inner join mechanism would not trigger any data loss, since only regular and aligned time series are present after the pivot. However in case of irregular time series (not using ‘aggregateWindow’ on events), time series captured at a different rate or regular yet unaligned time series, data points would get lost in the inner join mechanism.

    Next to that, the ‘join’ in Flux is only letting two time series be joined together at once, which constrains the possibilities. Moreover, seen from a performance perspective, the ‘join’ is considerably less optimized compared to dedicated functions in Flux, like the ‘aggregateWindow’.

    4b. Solution: Outer Join

    To tackle the issues subject to the ‘join’ provided by Flux, an outer join mechanism is proposed, retaining data points for both shared and unshared timestamps between tables of multiple joined time series with respect to time. After appending the three time series to a single new time series holding one table, the outer join mechanism groups all data points by time into ‘timetables’ prior to filling up rows with the appropriate data points. Furthermore, the single complete row per timetable is extracted, preceding the ungrouping of timetables to one overview table.

    Union

    Once the aligned sensor data is available into a dedicated time series per entry consisting of one table, the three time series are appended by a ‘union’ to a single new time series holding one table. The ‘union’ is ingesting the sensor data of each entry vertically (image: union), meaning the table rows per entry are placed underneath each other while the resulting table schema is the union of the separate table schemas of the three time series. In this manner, you can witness ‘data point blocks’ in the resulting table referring to sensor data of each entry in the production site. Note that the ‘union’ allows more than two time series to be appended to a new single time series at once.

    union – code snippet

    union

     

    Timetables

    The sensor data of all three entries in the single time series table is grouped by the ‘_time’ column to group the sensor data of different entries together in timetables. The ‘timetables’ refer to the tables that arise by using the group key ‘_time’ (image timetables). The forthcoming sort is fundamental to order the rows of a certain entry at the bottom of a timetable prior to starting to fill data points. In this use case the row originating from entry C is positioned at the bottom of each timetable’ by sorting for the ‘entry_C’ column in ascending order. In fact, regardless of sorting the ‘entry_C’ column in ascending or descending order, the empty data points in the ‘entry_C’ column of the rows originating from entry A or entry B will always be on top in Flux since they hold a null value.

    Timetables – code snippet

    Timetables 

    Fill

    Moving forward, the data points of the columns originating from entry A and entry B are copied into subsequent rows using the ‘fill’ with ‘usePrevious’ set to ‘true’. Since the data points are filled through the bottom, the mutual order of the rows originating from entry A and entry B is not important. Note that at the bottom of each timetable, the row originating from entry C is now a complete row holding all data points for all three entries (image: fill).

    fill – code snippet 

    fill

    Extract

    Finally, only the complete row at the bottom of each timetable is extracted by using the ‘tail’ with ‘n’ set to ‘1’. The timetables are ungrouped to one joined table by using the ‘group’ without any parameters to empty the group key. Check out the joined table (image: extract) and you’ll see that all sensor data originating from the three entries of the production site is visualized into one trustworthy overview with respect to the time behavior induced by a sublot passing through the production entries on site.

    Extract – code snippet 

    Extract

     

    4c. Outer Join Mechanism for Unaligned Sensor Data

    To illustrate the outer join mechanism, one can join unaligned sensor data for the corresponding time ranges from the 3 entries in the production site into one overview. This can be arranged by removing both the ‘timeShift’ of three minutes for entry A and the ‘timeShift’ of two minutes for entry B when compared to entry C.

    In that case, the sensor data of entry A will retain its time range between 2:57 and 3:07, while entry B will have a time range between 2:58 and 3:08, followed by a time range between 3:00 and 3:10 for entry C. The result is a broader overall time range for the overview than requested (image: outer join on unshifted sensor data). Besides, only unaligned sensor data is present for all three entries in the time range between 3:00 and 3:07.

    Note that for the top and the bottom rows, the unavailable sensor data for a certain entry is replaced by the ingestion of null values as data points in the columns are originating from the corresponding entry. As mentioned earlier, the same principles apply for an outer join when dealing with irregular time series or time series captured at a different rate. Ideally, such an outer join mechanism should be the default behavior in Flux when performing a ‘join’.

    Outer join on unshifted sensor data 

    Conclusion

    When creating a trustworthy overview of captured sensor data at a production site in consecutive production steps, take these four elements into account:

    1. The step in the production process where the sensor data is captured is key to understanding the time behavior of a product batch going through the different production steps on site.
    2. How sensor data is captured in production indicates the urgency of using a specific time window and which selection/aggregation of data points you should obtain. Hence, sensor data can generate a regular or irregular time series or can be captured at a different rate.
    3. Sensor data per production step is pivoted to a time series holding one unified table of sensor data for that production step. The pivot stands out in unifying the sensor data per production step, since the different tables each referencing a sensor in that production step typically hold an identical schema when reading them in.
      • Next to that, generally speaking, there will only be one data stream per sensor and thus only one value field per table in the time series derived from a production step.
      • Performing an outer join on the multiple time series originating from multiple production steps, gives you the ability to create a trustworthy overview in the end. To make the most out of it, make sure to align the time series first before joining them.
  • 4 Container Orchestration Security Concerns

    4 Container Orchestration Security Concerns

    Container orchestration is a must-have technical skill, according to 51% of respondents to the 2021 Upskilling survey from DevOps Institute. So, it’s no surprise, then, that container technology is becoming increasingly popular. However, container environments are far more complex and dynamic than other infrastructure, making container security essential. Security ensures containers are running as intended, protecting the infrastructure, software supply chain, runtime and everything in between. 

    If you’re looking to use container-based applications within your software infrastructure, you may want to keep in mind the security risks that come with this technology. To help, I asked DevOps Institute Ambassadors to weigh in on the security concerns they have with container orchestration. Here’s what they shared: 

    Anshul Lalit, head of technology and transformation, Kongsberg Digital

    “One of the most common security concerns with container orchestration is that it shares the host operating system and the host kernel. Thus, it raises the concern that a host OS or kernel vulnerability could potentially be taken advantage of and exploited in a container. An example of this is the recent attacks on Linux’s mmap syscall. Though you can avoid this by using defined best practices, it’s worth noting that any compromise in the host OS could compromise containers in that host. This scenario is in contrast to a virtual machine (VM), in which the guest OS and the host OS and kernel share no common binaries and are thus inherently separated.

    Similarly, a rogue process, a vulnerable container image that is running, and a security vulnerability in the orchestrator can all provide an attacker a means to get their foot in the door. For example, if a container is designed to run with root privileges, an attacker running a vulnerability scanner on the orchestration host will quickly find the vulnerable container.

    Users should follow best practices for securing the containers running in a container orchestration platform by providing a Linux security module, for example, that can enforce a policy that will protect against a rogue container or user from extracting sensitive data from the host system.”

    Supratip Banerjee, solutions architect, Principal Global Services

    “Containers are creating new challenges for cloud security. While these structures are dynamic and automated, new threat vectors are also present for precisely the same reasons containers are useful. DevOps and SecOps teams must be aware and prepared for these possible issues:

    • Containers, unlike virtual machines, share the host operating system on which they execute. If settings are not correctly maintained, the host and containers may be exposed to security concerns.
    • Orchestration automation adds a degree of complexity which can lead to misconfigurations that overprovision access and, as a result, an expanded attack surface.
    • Containers are constructed using pictures kept in public or private repositories; they are frequently reliant on other images, and a single flaw in any of them might spread across thousands of containers.
    • Because containers are ephemeral, IP address-based security measures are ineffective and complicate forensic investigations.”

    Parveen Arora, co-founder and director, VVnT SeQuor

    “If your project(s) involves the use of containers, you will have decomposed your application into its various constituent services, which, while beneficial, isn’t necessarily as secure as if you are using VMs. Since container platforms like Docker borrow code libraries from the host, they’re fundamentally not as secure as VMs. Vulnerabilities in a container environment can therefore allow malicious software to break out of a container and infect the host environment.”

    Vishnu Vasudevan, head of product engineering and development, Opsera

    “The security challenge, especially when it comes to container orchestration, is people think they can spin up their own image or they can download the image from the cloud and start using it. It’s not as easy as everybody thinks. There are vulnerabilities being introduced every minute. You need to have a security tool and processes that can automatically scan for every container that is being splintered. Furthermore, a team needs to create shared practices around blue-green deployments or rollbacks of the container changes that are being pushed into production. 

    From the pandemic alone, we have seen significant cloud movement, but on average, 50 vulnerabilities are introduced every day. In fact, there are people just waiting for loopholes where they can execute malicious scripts. And some people, especially people who are in mining, try to gain visibility into vulnerabilities within cloud accounts and penetrate them to run huge containers for their own cloud mining purposes. On average, it can take 280 days to even detect any vulnerabilities or a hack into the system. 

    Companies using containers must consider threat vulnerability management as part of their orchestration strategy. Do you think the team is experienced enough to scan 50 vulnerabilities a day and fix them? Probably not, since release velocity is shortening a release cycle to one or two weeks for modern application initiatives. This creates a lot of technical debt that can accumulate rapidly, which is why threat vulnerability tactics and processes must be considered when moving to the cloud.”

    Learn more about container orchestration and similar topics by registering for an upcoming SKILup Day. 

  • But What About When You Fail?

    But What About When You Fail?

    One thing Agile and DevOps definitely brought IT was a more accepting view of the whole “mistakes happen” mantra. Crashing systems is no longer a guarantee of a free ticket out the door. Indeed, off the top of my head, I can think of at least two cases where a CIO themselves checked in sloppy code and it trashed the system—CIOs at orgs big enough that they probably should not have been coding in the first place.

    And that brings us to today’s blog. The standard response to “But what if there is a bad update?” is “We’ll just roll out a new version!” That works in some instances. In a lot of instances—most that involve GitOps as part of DevOps—it doesn’t. Implosions can be huge and take out chunks of infrastructure. So you need a better plan than just assuming you can fix it with a forward update. Do you have rollback capability? Are you making use of it, assuring all is set to roll back if needed? Testing to make sure rollback works with the other changes made across the system in this update?

    That is part of the problem. Fixing with a new update is the best option, simply because in the age of massively distributed, microservices-based solutions, rolling back comes with a ton of baggage—enough that it may not be viable for you. Okay. So you can’t quickly roll forward. You can’t quickly roll back. Quick! What do you do?!

    And I don’t have the answer to that question, because I’m not in your organization working on your systems. At some point, all IT is personal. And this is one of those points. You need to know what your best options are if an update spirals everything, and you need to have a plan to implement it. But what your best options are is not going to be the same as the next org. This is that point.

    One option is to keep the ability to build the entire system. That’s a large setup, but every minute systems are down is hurting the company. And that’s what you need to plan for. Think of it as disaster planning for DevOps. A man-made disaster that destroys systems and infrastructure.

    So, like any other disaster planning scenario, walk through the chain that makes the system work, identify weaknesses and list ways to address them. Then test those to make certain they do what you hope they will. Then, set up an automated system to keep this whole plan up-to-date.

    In short, we’re in a new automated landscape, and we need a new automated tool to pull our rears out of the fire when the inevitable happens. Be it from dev error or malicious attacker, it is a safe bet that, sooner or later, you will have a massive systems outage that your DevOps toolchain can’t adequately address. Know what you are going to do. Or at least have thought about it, so you’re not just starting to think it through while coworkers and customers can’t access systems.

    And keep rocking it. This is just another layer of protection for all the hard work you’re doing. Take the extra step. Like insurance, if you ever need it, you will absolutely be glad you did.

  • Global Next-Generation Software Engineering Conference

    Global Next-Generation Software Engineering Conference

    GNGSEC sounds the clarion call for software engineering change!

    The pandemic made 2020 the year where we see some of this decade’s most exciting digital technologies become commercially viable, and others finally go mainstream. Software is the core of digital transformation. In the digital era, the methods by which we produce software also need to be digitized. In the software engineering field, the adoption of AI, big data, cloud computing and cloud-native are accelerating the digitization for businesses.

    Global Next Generation Software Engineering Conference (GNSEC) focuses on a full range of software engineering technologies and is committed to defining the standard of the next generation of software engineering. The event will gather 1000+ Attendees and it will be live with virtual experience where senior software engineers, architects, and team leads connect, learn, and grow. Experts and developers from various backgrounds will be sharing their cutting-edge technical practical experience and cases.

    The majority of the attendees of this event will be from the Southeast Asia region (Singapore, Thailand, Indonesia, Philippines, Malaysia, Vietnam), India and China. Due to the current travel restriction, GNSEC 2021 will be virtually presented via Airmeet. It will offer attendees with virtual exhibitions, virtual conference, peer networking session, business matching session, virtual discussion group.

    Global Next Generation Software Engineering Conference (GNSEC) is co-presented by DAOPS Foundation and Escom Events. The event is part of digital innovation series events that features AI, DevOps, Cybersecurity and digital transformation. Previously the event was presented in Shanghai, Shenzhen, Beijing, California and Hong Kong.

    There will be more than 30 speakers from DBS Bank, ING, IBM, Daimler, Tencent, JD.COM, DevOps Institute, Gitlab, New Relic, SUSE and more to share ideas, experiences and case studies in software engineering.

    Register here to join​​ the event with us to rebuild software engineering.

  • The Role of APIs in Mainframe Modernization

    The Role of APIs in Mainframe Modernization

    Mainframes are historically clunky systems. They are incredibly secure and can process impressive calculations, yet they are complex to manage and challenging to extract data from. This reality is at odds with modern cloud-native infrastructure, which emphasizes distributed computing and portable microservices. So, how can existing mainframes adapt to this connected, data-driven economy?

    I recently met with Dr. Alex Heublein, president at Adaptigent, to talk about the state of legacy modernization. According to Heublein, many mainframes are here to stay for the foreseeable future. However, enterprises are continuing to get value from these systems by adding an integration layer on top of them. By using a REST API layer on top of a mainframe, enterprises could finally open up these systems. This could facilitate bidirectional communication with newer cloud-native services, enable phase migration and edge closer to real-time processing.

    State of Mainframe Modernization

    The software industry changes every day. “Nothing in the tech field is permanent,” said Heublein, yet some legacy technologies will persist into the foreseeable future. In terms of mainframe adoption, Heublein sees a couple of categories of users. First are the large financial services organizations, such as large banks and insurance companies, which are considerably invested in their mainframe architectures and which will likely continue to rely on mainframe environments for their policy management and core infrastructure. “I think there’s a lot of good reasons for that,” explained Heublein. “If you want to process billions of transactions a day, [mainframe] is still a good option.”

    That said, regulations and market pressures are influencing a global movement toward open banking. For example, PSD2 mandates open banking in Europe and industry consortiums like Financial Data Exchange (FDX) are directing it in the U.S. market. These initiatives are encouraging financial services to transform their legacy environments and spurring cloud migration, especially for applications that aren’t mission-critical.

    Heublein pointed out a few different patterns in enterprise mainframe modernization. One approach is that enterprises are rewriting or re-architecting old COBOL applications and keeping them on-premises. Or they are porting and re-platforming the application; moving a COBOL app to a Windows or Linux system, for example. Finally, some are shifting straight to the cloud. For solutions that don’t need to be on the mainframe, enterprises may shift toward cloud-native to free up capacity.

    APIs Enable Legacy Modernization

    Distributed cloud technology is helping businesses scale to the right level. Yet, it’s difficult to jump off the mainframe entirely, noted Heublein. “Very rarely can you take all your stuff and move it all at once,” he said. “It’s a tough pill to swallow.” It’s especially difficult to move core banking infrastructure to a distributed environment—an instant shift could introduce security risks. For this reason, he recommended a ‘phase migration’ approach in which individual components are slowly migrated. For example, an insurance provider may shift their claims engine, policy engine and ratings engine to the cloud one at a time.

    As a result of phase migration, however, engineers must simultaneously support both on-premises and cloud-based applications. They must also ensure bidirectional communication so that these systems can exchange data. This is where a REST API transactional layer is necessary, Heublein explained. Such a layer could enable bidirectional communication between the mainframe and cloud-native applications, abstracting complexity when calling out to the modern world, or intercepting cloud data and filtering it into a format mainframe technologies can process.

    To increase developer usability, abstracting complexity from the mainframe integration process will be the key to building out connectivity. This could involve a codeless development approach with a COBOL sub-routine to call external systems, Heublein said. Automatic restructuring of data will be necessary to enable fluent bidirectional communication, too.

    Four Business Drivers of Mainframe Integration

    So what is the end result of putting an integration layer on top of mainframe systems? According to Heublein, there are four key business benefits of using such an abstraction layer:

    1. Orchestration: Software ecosystems often need to make multiple calls to different applications to collect and transform data. Yet, mainframe environments are very complex and challenging to integrate with. More usable integration fabric for mainframe systems could enable better orchestration of workflows in a developer-friendly manner. By using a REST interface, all apps could be calling the same interface, thus greatly simplifying orchestration between disparate services.
    2. Sustained Adaptability: The world is rapidly accelerating, and enterprises need to maintain and modify these APIs very quickly. Organizations require “sustained adaptability” to adapt and survive in the digital world, said Heublein. Mainframe systems, however, were not designed for the 21st century. Thus, low-code integration layers are critical to empowering rapid development.
    3. Data Transformation: Data is powerful for the business. Data is not only a commodity in and of itself but also is required for AI and machine learning. However, mainframes have a completely different way of encoding data. Thus, there is a need to transform this data into a modern format that businesses can consume. An integration layer between these systems can deliver an improved data consumption model.
    4. Real-Time Processing: You may think online payments happen in real-time; however, payment apps like Venmo are really just an abstraction layer over bank transactions, which may clear as little as twice a day. In short, mainframes are not great at real-time processing. However, more usable integrations to this infrastructure could enable user-facing applications to interact more closely to real-time.

    Another concern is security. Opening up mainframe environments that house core financial infrastructure must be done with extreme care—especially as API attacks remain the number-one rising threat in the software landscape. All organizations building out new integrations will require a secure layer from an outbound standpoint. Heublein said this often involves placing an API management layer or gateway in front of the integration layer. This system enables authorization and functions like a firewall between the internal and external world. It can also enable logging and prevent denial of service attacks, he explained.

    Improving Agility for Legacy Systems

    Legacy systems are entrenched in older technology and carry lots of technical debt, forcing operations to move at a slower pace. Maintainers are also culturally a bit different than the newer breed of cloud-native engineers. “Mainframe people are not … fast,” said Heublein.

    DevOps folks want to build, test and promote projects to production very quickly and, arguably, this agility is necessary to satisfy rising digital innovation demands. Improving integration capabilities for on-premises systems could enable businesses to better utilize preexisting investments while also improving agility.

    Low-code/no-code capabilities could also increase collaboration potential, Heublein added, which could bridge the gap between on-premises engineers and fast-moving business folks. When adopting low-code/no-code tools for mainframe integration, he cautioned against general-purpose citizen developer frameworks. Whereas universal integration layers can work well for many situations, he stressed that a very specific boundary box is best for building APIs for mission-critical applications.

    How is your organization approaching modernization for legacy systems? Let us know in the comments below!

  • A New Era of Observability, Monitoring and Analytics for DevOps

    A New Era of Observability, Monitoring and Analytics for DevOps

    [pdf-embedder url=”https://devopsy.kinsta.cloud/wp-content/uploads/2022/02/a-new-era-of-observability-monitoring-and-analytics-for-devops.pdf”]

  • 4 Tips To Plug The Cloud IT Skills Gap

    4 Tips To Plug The Cloud IT Skills Gap

    Technical talent is few and far between. The programming world faces a developer shortage, and with this shortage comes a lack of cloud know-how. By now, most organizations have become multi-cloud, yet 86% of IT professionals believe a lack of these skills will slow down cloud projects.

    As more teams transition from older architectures to cloud environments, the IT skills gap could pose a severe bottleneck to software development and deployment. Cloud servers require ongoing maintenance, and managing security and compliance is challenging, even for seasoned developers. Furthermore, it’s easy to overspend in the cloud—without keen insight into vendor nuances, it becomes challenging to select the most cost-effective package for the application at hand.

    I recently talked with Joe Spurrier, chief technology officer and co-founder, cloudtamer.io, to see how teams can improve their cloud strategies while navigating a short supply of IT cloud know-how. According to Spurrier, increasing automation, sharing reusable templates, centralizing best practices and maintaining agnosticism are strategies to amplify cloud results. Below, we’ll expand on these areas to see how they maximize cloud benefits while retaining a lean engineering footprint.

    Automation, Automation, Automation

    “Automation is key to furthering your adoption and use of the cloud,” said Spurrier. The drive to automate comes as no surprise to many DevOps and SRE professionals whose goal is to automate their job away. When automating the cloud, infrastructure-as-code (IaC) is a fundamental necessity. “If you’re getting into cloud, you should probably have some scripting skills,” said Spurrier.

    Automation can significantly aid scalability when setting up additional accounts since engineers must replicate similar operations, such as server provisioning, deployment patterns, monitoring and so on. Automation is also essential for shift-left testing and to increase observability into production environments. “It’s one thing to deploy an application via automation, but another to have tests in automation—that’s the next level,” said Spurrier.

    Now that so many tools are API-driven, orchestration becomes programmable. A cloud service provider’s CLIs or SDKs could also be leveraged to hook into these integrations to construct such automation.

    On the flip side, too much automation for high-level privileges could backfire. “Someone could break into a data center, steal a server, disconnect it, then run off,” Spurrier explained. Thus, automation will require a higher sense of security to ensure that every change is appropriately authorized and logged.

    Reusable Templating

    Along with automation, documenting operational processes to be shared amongst engineering teams is vital for maximizing efficiency in a lean talent landscape. The key here is to make the process self-documenting and reusable, stresses Spurrier. This will most likely involve someone on the team creating shared libraries and repositories for other team members to leverage.

    One such example is Amazon’s golden AMI pipeline. Golden AMIs are virtual machines that teams can “standardize through configuration, consistent security patching and hardening.” Golden AMIs can also enable a repeatable process for performing monitoring, logging and even distribution to business units.

    Reusable patterns are important for building CI/CD systems so that when you commit code, it is automatically deployed to preferred cloud providers. In general, Spurrier recommends building shared repositories that are easy to clone, with step-by-step instructions for usage. Sample applications that demonstrate components in different languages can also aid developer usability and save time.

    Adopt Centers of Cloud Excellence

    Another way to encourage better adoption is by creating a center of cloud excellence (CoCE). For an enterprise, having a cloud-centric project management group can be very helpful to organize knowledge and offer support. A CoCE could centralize operations, supply engineers with pre-configured containers and help distribute access to cloud accounts.

    A CoCE is a good way to start to unite disparate teams around shared best practices, said Spurrier. Depending on your organization’s needs and workloads, a CoCE will focus on varying skills, he added. In the beginning, a company may have just one center for cloud knowledge. However, as adoption grows, dedicated centers may likely emerge around specific cloud providers, too.

    Retain Agnosticism

    Contrary to popular belief, multi-cloud steups aren’t always about lift and shift. Business units often adopt a particular cloud to suit specific workload requirements. For example, researchers may choose GCP for its data science functions, or Microsoft shops may prefer Azure.

    In a multi-cloud world, management layers could help retain a modicum of vendor neutrality. For example, take Cloud Custodian, a tool developed at Capitol One and recently contributed to the CNCF. Using Cloud Custodian, engineers can run security checks across AWS, Azure and GCP and generate metrics for reporting. Such control and insights could empower teams to automate some of the multi-cloud management burdens.

    When working with multi-cloud, it’s also good to consider agnostic tooling that can work across different providers, like Terraform or Kubernetes, Spurrier said.

    Templating

    Cloud transformation is ongoing at most organizations. With this rise, more cloud-native processes and tools are emerging. Yet, with talent in short supply, DevOps isn’t always fully prepared. To review, we covered four ways businesses can circumnavigate this issue to get an edge:

    • Increased cloud automation is a necessary response to a slimmer labor pool. Though building automation requires upfront technical skill, it pays dividends in increased efficiencies.
    • Reusable libraries and templates can rapidly configure new projects and do so consistently across business units.
    • Consider maintaining a center of cloud excellence to centralize policies and offer support.
    • Opt for agnostic tools. This helps to retain vendor neutrality and apply consistent controls across clouds.

    Of these above strategies, Spurrier emphasized the importance of creating shared templates. When used right, a templating approach can be very effective, he says. “If you have a templating model, it should be easy for DevOps teams to migrate to new features. If you’re leveraging tools that set up templates, it’s easier to figure out how much infrastructure is going to cost.”

  • Red Hat Extends Scope of Ansible Automation Ambitions

    Red Hat Extends Scope of Ansible Automation Ambitions

    During the online Ansiblefest, Red Hat announced the availability of a preview edition of Red Hat Ansible Automation Platform 2 that is intended to make it easier to automate IT processes at scale.

    Richard Henshall, senior manager for Ansible product management at Red Hat, said previously most IT teams have been automating individual processes, but as organizations become more adept at creating playbooks to automate processes, they now need to find ways to manage multiple automations that are often dependent on one another.

    Ansible Automation Platform 2 also enables IT teams to automate processes that span both monolithic as well as cloud-native applications based on containers using an automation controller, formerly known as Ansible Tower, that is more tightly integrated with Red Hat OpenShift, an application development and deployment platform based on Kubernetes, using Red Hat Operator tools. There is also an automation mesh capability that connects disparate automation components together and provides status checks.

    Ansible Automation Platform 2 also introduces automation execution environments within Ansible Engine that make it possible to create self-contained automation spaces that can be easily replicated and repeated across an organization.

    Red Hat also includes an automation content navigator to helps teams validate that automation content is working as it should across even the largest environments, along with updates to the Red Hat Insights analytics service that enable issues that might occur with automation controllers to be surfaced and eliminated by automatically applying policies.

    Henshall said going forward, IT teams should expect to see Red Hat focus more on enabling events to be the catalyst for triggering automation. Today, most automated processes begin with an IT administrator triggering the process. However, as digital business transformation initiatives mature, it’s become apparent there will be a need to trigger automation when specific events occur.

    Henshall said while there are still many organizations that are just starting to automate IT processes, there are also many that are entering the next phase of maturity. As such, they now require an automation platform that makes it simpler to manage automation at scale.

    In general, Ansible is designed to make it possible for developers and IT operations teams alike to declaratively create automation without much programming expertise required. However, creating a playbook that works reliably still requires a deep understanding of the processes that are being automated. Most IT teams would be well advised to closely monitor processes when they are first automated to make sure they are performing tasks as anticipated.

    Ultimately, it’s now a question of when rather than if most IT processes will be automated. DevOps teams, of course, have always been at the forefront of automation, but now it’s clear that as IT environments become more complex—with the rise of both the cloud and edge computing platforms—enterprise IT teams need to embrace automation more aggressively. The biggest barrier to automation is simply finding someone who has the passion to automate a process to lead the charge, said Henshall.

    Naturally, most of those efforts will initially focus on repetitive, tedious and manual IT tasks, but the potential to automate IT processes at levels of unprecedented scale has never been greater.

  • The New(ish) Complexity

    The API economy and microservices architectures, led by Agile and DevOps, have introduced a massive amount of hidden complexity. I call it hidden complexity because we have a tendency in IT to focus on the problems in front of us, resolve them, then focus on the new problem in front of us. This leaves technical debt in little piles behind us, like the bits that fall off the ends of a snowplow blade (or the asphalt that squishes out the end of the steamroller for our southern friends).

    I was thinking about this when I was looking into how a majority of you handle scalability. The layers of complexity for scalability are surprising. In the past, we invested upfront to be prepared to scale to the levels we need. Today we allow expenses to grow as users grow. That is a great model if users are tied to revenue. If there is any fungibility at all in the correlation of users to revenue (and there almost always is), this is a potential disaster waiting to happen. “Our traffic spiked 200%, but revenue only grew 20%,” is pretty much the last thing your business leaders want to hear. Because in the “Pay as you grow” world, a 200% spike in users means a significant spike in overhead. How much overhead varies pretty wildly, of course, so I’ll just say “Guaranteed and significant” and leave it at that.

    This is a known bit of complexity, but those of you I’ve spoken with have done little to hedge against it. That’s … interesting. How many other hidden land mines are out there, being ignored by us?

    What can you do? That greatly depends upon your business. with internal systems, you have a lot of say in usage. External systems are where this is potentially an issue. You can pay your provider to block DDoS, so at least you’re not paying for that bandwidth if that happens—and most of the people I spoke with really do have some form of DDoS protection that stops traffic before it becomes a fee-generating hit. But for actual growth, the best you can do is get with business leaders and determine what influences the correlation between users and revenue. What makes those people upping your traffic actually spend money while they’re upping your traffic? What can you do to convince more of them to spend money? What services are you not offering that might be useful to those customers and not significantly increase overhead. Basically, this is the modern equivalent of the conversion problem. They’re on your site, how do you convert them to paying users of your site.

    We really can’t get more detailed than that—you know your business, and a media site is a different model than a retail site, for example—so you’ll have to develop beyond that. But have a plan that does what you can to make sure that when expenses go up, revenue goes with it.

    And keep rocking it. We’re outside the historical IT realm here—because you’re all rocking it so well that we have new things to worry about. Give those customers rockstar apps to work with and figure out how to make them paying customers. Develop those bonds with business leaders; you do indeed need each other.

  • THE UPSKILLING IT 2022 SURVEY IS NOW OPEN

    THE UPSKILLING IT 2022 SURVEY IS NOW OPEN

    IT SKILLS IN FOCUS

    By Eveline Oehrlich, DevOps Institute

    According to McKinsey, nine out of 10 global executives and leaders are saying that their organizations are facing a skill gap or expect to develop additional skill gaps in the next three to five years. During the past few years, IT organizations, in particular, have already been facing skill gaps. The desire to accelerate technical innovations and digital transformation only makes it harder to identify and narrow current and future skill gaps. IDC predicts 75% of organizations will have a comprehensive digital experience implementation roadmap by 2023, up from 27% today. While 2023 is still a few years away, it is essential that IT organizations prepare for what skills are necessary, how to manage their talent supply and look at the underlying skills of people that must support this digital journey.

    DevOps Institute research team and sponsors, Palo Alto Network, Cycloid, and The Linux Foundation, are excited to announce the launch of the Upskilling IT 2022 Survey. The survey takes approximately 25 minutes to complete, and we’d love to hear from you. Your answers will allow us to better understand which IT skills are essential for the future, ensuring that organizations are able to accelerate their digital journey.

    The Upskilling IT 2022 Survey, by DevOps Institute with its yearly sponsors, is the only and longest running research of its kind. The results provide an unbiased view into what skills are essential across a variety of IT and other skill domains.

    Like the past three research reports, our goal following this year’s survey is to share a detailed analysis to help various leaders and roles, in order to help them determine what skills they should hire, retrain or upskill for, following the publication of the Upskilling IT 2022 Report in the first quarter of 2022.

    Don’t let IT Skill gaps be a competitive liability for today or in the future

    The survey is open for everyone, regardless of the size of your organization, or your organization’s industry. This year we have expanded the focus beyond DevOps and are homing in on cognitive and human skills.

    Some of the key topics we look to deep dive into this year include:

    1. Exploring where the biggest skill gaps currently exist and within which roles, such as leaders, developers or operations.
    2. How skill gaps are identified across the IT industry.
    3. How upskilling is done, funded and executed.

    While each person has their own unique experience, expertise, examples of execution or outcomes and various tolerances for exploration, for the fourth year, we are setting out to explore what the must-have skills are within different skill domains. This year we are focusing on the skills needed within modern IT organizations today and in the future.

    There are a variety of roles that are involved in recruiting, hiring or working within IT organizations. We are setting out to highlight and understand which must-have, nice-to-have and optional skills are essential within key skill domains. The skill domains include Automation, Human skills, Technical skills, Functional skills and knowledge, Process skills and framework knowledge, Business skills and leadership skills. We are targeting the community of IT leaders, developers, practitioners, team leaders, consultants, human resources and other individuals who are familiar with modern IT best practices, such as DevOps. We are targeting all industry verticals and all company sizes. Our survey questions are developed with input from a variety of team members and experts, and our goal is to achieve a sample size of 4,000 individual responses across North America, Europe, Asia Pacific and Latin America.

    Help us shape the future of IT and DevOps and make your voice heard by completing the survey now.

    The survey will remain open until January 31, 2022. We look forward to hearing from you.

    Take the survey now:  https://www.surveymonkey.com/r/Upskilling2022Blog