CI/CD Misuse

Over the years, CI/CD systems have been released, deprecated, and morphed in use cases and complexity. The biggest promise of them has always been a transparent, easy-to-manage build, sometimes test, and deployment process that any technical person on your team can understand, and hopefully manage. In a sense, we’ve come a long way since the old programmer “jokes” about code compilation. Now in most well-oiled software shops (and what business today isn’t a software shop at its heart?), all it takes is a git push to start a build, and in some cases get as far as production. 

CI/CD systems are fantastic, and using them is at the core of any modern tech business. 

In our years of consulting, however, I personally, as well as the staff of BTS, have seen gross misuse of these systems for other purposes. Offending parties’ names have been removed for obvious reasons, and environments have been obfuscated. But here are the five top misuses we’ve seen for any given CI/CD system over the years. This list is non-exhaustive and presented in a humorous format. 

The Cron Orchestrator

You have a bunch of small scripts that need to be run at any given time. They’re small, unobtrusive, and if this was 2001, they’d be sitting in a cron job somewhere. Yes, that venerable UNIX crontab that nobody really remembers how to get the exact dates going, and seems to be a weird mix of asterisks and slashes and dayname abbreviations. Maybe the developers didn’t have access to a machine to put it there, or maybe managing it was too much of a pain, but now your Jenkins instance is serving double duty as a glorified crontab. 

Pros: 

• You no longer have to manually edit crontabs, and maybe even the interface in your CI/CD tool nicely obfuscates the cron syntax. 
• Theoretically, this is a centralized place for all of your automation jobs!
• Need to update a job? Just git push!

Cons: 

• In reality, these are never maintained
• All of the scripts are written inline with various assumptions at various times
• Welcome to maintenance hell
• Tracking down what fired when is a nightmare, because all of those logs are mixed with your build logs
• Congrats, you’ve re-implemented an existing feature of the OS that you’re already running the CI/CD platform on… with extra steps... 

The Security Scanner

Congratulations, you adopted the latest buzzword from 5 years ago, DevSecOps is your thing. And you take it very seriously. Your pipelines have security scanning steps. Your security steps have security steps. You have an entire subset of pipelines scanning your infrastructure for vulnerabilities, and your CSO is very happy. 

Pros: 

• We already have this tool, let’s leverage it to its FULL POTENTIAL is music to an exec’s ears. 
• Your infrastructure is theoretically secure. 
• It’s cheaper than paying for a security tool, right? right?

Cons: 

• You are now responsible for keeping abreast of every single CVE and implementing remediation and alerting policies for all of them. 
• Congrats, you’ve built a shoddy security scanning system from scratch, and now you’re the only one maintaining it. Potential job security, yes. Potential headache? Definitely. 

The Infrastructure Monitor

Zabbix? Pshaw. Datadog? Whatever. We have a fantastic tool that already has its claws deep within our infrastructure. We can monitor everything through here! Automation! Slack Triggers! Excitement? Now your CI/CD system is serving as an ersatz monitoring solution. Who needs an actual monitoring solution? Between cron triggers and events firing, this is an all-singing, all-dancing solution. Maybe we can even make this into a logging aggregator? 

Pros: 

• In many cases, true, your CI/CD system can hit your environment.
• No need to implement another tool
• A nice ITTT (If this then that) workflow
• You can potentially patch up broken applications by just having an infrastructure component reboot/redeploy/resync whatever

Cons: 

• You’ve now successfully re-implemented nagios, except worse. 

The Data Processor

A CI/CD system is not too far off removed from a data pipeline. AWS Glue? Pshaw, we got Jenkins. And has a built-in fully-featured language. Or we can just wrap whatever Python we need and make our ETLs work here. Look at all these features!

Pros: 

• Oh boy
• Well, it is an ETL system of a sort if you squint really hard,

Cons: 

• The gods of Big Data are now unhappy
• What’s performance?

The notification spamming service

This is the end stage of any misused CI/CD system. Whatever you happen to be using it for, your notifications of “builds” failing or succeeding may be making the FTC under the CAN-SPAM rule look at your organization a bit closer. Ok, that’s somewhat hyperbolic, but your coworker’s inbox is now at 140,000 emails and none of them are actionable. Your chat system of choice has a dedicated channel to notifications for “visibility”, yet when asked privately, your engineers will never actually admit to doing anything useful with those notifications. 

Pros: 

• Useful for testing your spam filtering and email rule creation
• Forces your engineers to use creative workarounds and modern communication tools like Slack!
• Drives return to the office if only to be able to talk to somebody to get some work done. 

Cons: 

• Can make the entire email system unusable for anything meaning you just missed that important meeting invite from a prospective client. 
• This is actually hell. 

CI/CD systems, like many general-purpose systems, are fantastic at being whatever you want to make of them. But this is like saying that a Makefile is turning complete, and therefore everything should be a makefile. But if we use turing-completeness as a measure of a tool, then Origami, is, Turing complete, but you’d never catch me folding paper to conduct a production deployment. Just because a tool can be something, doesn’t mean it should be. If your organization isn’t ready for full-blown continuous deployment, fine. Not many are. Continuous integration, on the other hand, is quite possible in many, even legacy and slow-moving organizations. Continuous builds (at minimum) should probably be part of any sane modern development workflow. But like everything in life, there are ways to go overboard even there, but that’s an article for another time. 

In the meantime, go forth, and Automate! All! The! Things! And if you have questions about the bizarre, give us a shout here at BTS. We’re good at helping folks untangle their messy infrastructure, misused CI systems inclusive.