High-profile incidents put spotlight on non-production system security

In 2018, the US Federal Trade Commission (FTC) entered a settlement with Uber over the company’s data privacy and protection policies. The FTC alleged that Uber software engineers would develop and test software that could connect to cloud data using inadequate cloud access controls for its test environments. In connection with this case, the Commission warned companies to secure their non-production systems because “an insecure software development environment can also create real problems.”

Since then, several high-profile cybersecurity incidents have involved non-production systems not designated for live production use, such as development, testing, stage, or retired production systems. Most recently, some major cybersecurity incidents involved non-production systems as vectors through which threat actors infiltrated or attempted to infiltrate significant organizations.

Experts say that all these and many other incidents underscore the importance of applying more stringent security practices to non-production systems, which often take a back seat to securing frontline production systems. As the SEC pointed out, “Insecure non-production environments leave a company open to corporate espionage, sabotage by competitors, and, yes, theft of private consumer data,” often just as much as insecure production systems do.

Recent incidents highlight insecure non-production environments

The concept of insecure non-production systems arose in several recent prominent cybersecurity incidents and investigations, including:

  • Microsoft: On January 19, Microsoft announced that the Russian state-sponsored actor Midnight Blizzard attacked its corporate systems using a password spray attack on a legacy non-production test tenant account to gain a foothold using the account’s permission to access the emails of the senior leadership team and cybersecurity, legal, and other employees.

SentinelOne’s Alex Stamos chastised Microsoft for dismissing this point of entry as less important than a live production system. “Calling this a ‘legacy’ tenant is a dodge; this system was clearly configured to allow for production access as of a couple of weeks ago, and Microsoft has an obligation to secure their legacy products and tenants just as well as ones provisioned today,” Stamos wrote.

Microsoft seemed to grasp his point. “We will act immediately to apply our current security standards to Microsoft-owned legacy systems and internal business processes, even when these changes might cause disruption to existing business processes,” the software giant said.

  • Cloudflare: On February 1, Cloudflare announced it had detected a threat actor on its self-hosted Atlassian server on November 23. Although the primary point of compromise in this incident came through account credentials that Cloudflare failed to rotate after an Okta compromise, the company said the threat actor attempted to gain access to a non-production console server in its São Paulo, Brazil, data center due to a non-enforced access control list. The threat actor was denied access and could not access Cloudflare’s global network.
  • First American Financial: On December 29, 2023, First American Financial reported to the US Securities and Exchange Commission (SEC) that it had identified unauthorized activity on certain information technology systems. While providing few details about this incident, First American said it “believes the perpetrator of the activity accessed certain company systems, exfiltrated data, and encrypted data on certain non-production systems.”
  • LastPass: On March 21, 2023, LastPass announced the results of its investigation into two major cybersecurity incidents, reporting that an unknown threat actor “exploited a vulnerability in third-party software, bypassed existing controls, and eventually accessed non-production development and backup storage environments.”

Real-world data can be found in non-production systems

One primary risk of insecure production systems is that threat actors can gain access to sensitive data such as encryption and access keys, passwords, knowledge of security controls, or intellectual property that could prove to be a goldmine for further exploitation.

“I think on the CISO and BISO [business information security officer] side of things, there are some fundamental truths that we can acknowledge about these environments that maybe not everyone is willing to admit, which is that oftentimes, development environments include a ton of materially significant intellectual property,” Andrew Krug, head of security advocacy at Datadog Security Labs, tells CSO. “You could have the best development practices and hygiene in the world. Some of your actual real data is going to make it in there at some point.”

Cost savings and complexity often kick in

However, many companies don’t necessarily have the best security practices regarding test environments and other non-production systems, often due to cost-saving measures. With the advent of cloud computing, “A lot of companies broke apart their infrastructure into at least development test production, and then they would have a security account,” Krug says. “Unfortunately, most of the cloud cost models they subscribed to for their vendor management or security platforms didn’t really scale with that segmentation. So, they just opted out of different resources and different things from monitoring” to save money.

“And I don’t just mean security monitoring; I mean all kinds of monitoring,” Krug says. “This is almost like a company culture question more than a legal or regulatory question: How high a value does that company hold for security best practices?”

Staff shortages make securing non-production systems a challenge

Even companies like Microsoft and Cloudflare, which aren’t likely to skimp on security spending, experience challenges in extending robust security measures to their non-production systems. “Cloud environments are getting more and more complex, and it just becomes more and more challenging to have the right governance to observe across all” of the components, Krug says. “We could probably say as we onboard more services and more complexity, it just gets harder and harder to know even what the right things are to observe.”

The lack of available cybersecurity talent only makes analyzing the complexity harder. “We could talk about the cyber skills shortage and that even if companies that are the size of Microsoft and CloudFlare and First American want to hire the right talent, they may not be available,” according to Krug.

Steps to enhance non-production system security

Although securing non-production environments might be an uphill battle, one thing organizations can do is “make sure they have a singular identity and access strategy that enforces a minimum burden for all authentication,” Krug says. “And I don’t just mean for people; I mean for machines and ensure that it’s a strong authentication guarantee.”

Another step to limit the impact of a breach of non-production systems is data masking to reduce threat actors’ access to sensitive information, such as access keys and passwords that threat actors could use to spread laterally within organization systems. For example, data masking on test systems changes data values while using the same format, providing a functional alternative to actual data in the development process.

Separating the development process from business functions can also make pivoting from a test environment to the broader business environment more difficult for threat actors to cause widespread theft and damage.

Perhaps the most significant step organizations can take is to raise the priority of non-production systems in the security process. Krug says, “I think that security teams are so lean and have so many priorities that these are far down the list.”

Advanced Persistent Threats, Cyberattacks

Go to Source