By Pushkar Tiwari
As forecasted by IDC, 80 percent of the workload is shifting to Containers/Microservices by 2023, which would curtail the need for per-app infrastructure by 60 percent, and accelerate the digital service resiliency by 70 percent. Containers have become the new norm for the development process, and Kubernetes has risen as a standard for container orchestration platforms. Through Kubernetes, you can manage the containers with clusters in the public cloud, hybrid cloud, as well as in the multi-cloud environment. More and more cloud workloads are shifting to containers as these make an ideal choice for cloud environments – thanks to their lightweight nature and efficiency. However, consequently, containers, containing the workloads running in the cloud, are increasingly becoming a target of cyber attackers. The article discusses the challenge of container security and highlights how container forensic can play its role to avoid damage if the container gets compromised.
The Compromised Containers Cases
In February 2018 – Tesla cloud resources were hacked to run cryptocurrencies. According to the RedLock researchers, the hackers had intruded Tesla’s Kubernetes console which was not password protected. Similarly, in 2019, Docker Hub usernames, hashed passwords, GitHub and Bitbucket access tokens were exposed in the hack. A user who fails to change his account password and may have their accounts auto builds modified to include malware.
It was further found that the Docker hub had public images with embedded crypto-mining malware. Docker containers have been gaining popularity over the past few years as an effective way of packaging software applications. This is also attracting the attention of malicious actors intending to make money by crypto-jacking within Docker containers and using Docker Hub to distribute these images. Based on malicious Docker Hub account analysis, it was revealed that it was hosting six malicious images intended to mine the cryptocurrency, Monero. The coin mining code within the image intends to evade network detection by using network anonymizing tools such as Proxy Chains and Tor. The images hosted on this account have been collectively pulled more than two million times.
The Factors Behind Container Attacks
The recent report from Skybox Security outlines the vulnerabilities and exploits in play over the first half of 2019 in a measured presentation that avoids overstating threats. The report reveals several trends that enterprises need to pay attention to, not least of which is the rapid growth of vulnerabilities in cloud containers.
Cloud containers are lightweight and more portable than virtual machines (VMs), they can replace traditional VMs in many cloud computing deployments because of their speed and simplicity. The problem is that ease of deployment can lead to security headaches.
Containerized applications rely on many supporting services that store containers (registries), orchestrate container lifecycles, monitor their execution, and direct traffic in and out of the containers. Furthermore, containerized applications and microservices go together, which increases the number of components and interactions for an application. All of which means several separate parts need to be secured and it also means there’s a larger number of potential attack targets.
The above-mentioned conditions might lead to the following vulnerable factors:
- Exploiting vulnerabilities in container images
- Security misconfiguration, stolen credentials
- Using containers from untrusted repositories
Besides, attackers have various incentives to break containers such as:
- Leverage the compute for cryptocurrencies mining
- Botnet attack
- DDoS attack
- Bring down your cloud service
- Bad reputation
How to operate containers at an enterprise-scale?
Step 1: Is your Incident Response Process Ready?
- Identification
Identification of incidents by monitoring security events, leveraging security monitoring tools from security vendors.
- Coordination
Notify the on-call responder, reviews and evaluates the nature of the incident report to determine if it represents a potential data incident, and initiates an incident response process.
- Resolution
Investigating the root cause, limiting the impact of the incident, resolving any immediate security risks, implementing necessary fixes as part of remediation, and recovering affected systems, data, and services.
- Continuous Improvements
New insights that can help you enhance your tools, training, and processes.
Step 2: Follow container security best practices
- Build the smallest image possible
- Remove unnecessary tools
- Package a single application per container
- Avoid running processes as root
- Use images from trusted sources
- Sync all your logs to a centralized location
- Invest in container-specific tools either from security vendors or open-source
- Keep run book ready to execute common mitigation options
Step 3: Follow key mitigation practices
- Send an alert – alert
- Restrict from other workloads – isolate
- Stop running processes – pause
- Kill and restart running processes – restart
- Kill running processes but not restart – Kill
- Keep run book ready to execute common mitigation options
The Challenges for the Container Forensic
Root cause analysis is the key for any attacks to understand the impact of damage and identify the correct resolution and prevention in the future. Container forensic is astronomically complex and challenging compared to forensic analysis of legacy cyber-attacks.
Container observability and forensic is challenging due to the following:
- Highly dynamic nature
- Not so great for visibility
- The complex nature of digital forensics (as container environments yield enormous amounts of data at high velocity necessitating the right tool and expertise)
- The short life span of containers
- Ephermal file systems
- No true snapshot capability
- When performing forensics on a container, it often feels like we are running blind with scissors
Container Forensic – Data sources
The data sources are critical in container forensic as they provide a different impact on forensics investigations depending on the incident being analyzed. A thorough overview of each of these data sources is required to forecast the impact they would have on investigation involving network intrusion, malware detection, and insider file deletion.
Following are the data sources for container forensics that must be taken into consideration:
Logs
- Cloud Infrastructure logs / Kubernetes logs
- Audit logs
- Application logs
- Operating system logs
- Network connections
- User logins
- SSH sessions
- Processes
- Execution
Snapshot of the node
The next step might be to snapshot the disk of the node that was running the container. You might then move other workloads off and quarantine the node to run the additional analysis.
- Identify affected nodes and all attached disks
- Create a duplicate of the disks while online
- Send the duplicated disk image for analysis
- Leverage docker explorer tool
- Compare the difference in binaries on disk snapshot
Container Visibility Tools
It is suggested that the devops, security analysts first leverage the variety of tools available to them when working with Docker and Kubernetes. These include utilizing the Docker statistics API to help obtain system metrics, which can be highly useful for those only needing to understand how one’s system is impacted by its container load when operating at scale.
Container visibility tools help in finding what is happening in the system and help answer these critical questions.
- Are there unexpected files?
- How do you get real-time info without logging in?
- How do you gather information remotely from multiple systems?
- Tools like Google Rapid Response (GRR) can be leveraged
- Download browser history
- Get details about a file
- Dump memory for a process
- Searching for a bad Jar/malware signature.
Conclusion
When it comes to containers, the recent increase in vulnerabilities is directly tied to a lack of security hygiene. The ease with which developers can deploy identical containers across environments means that container adoption will continue to grow and, as a result, your attack surface will grow if vulnerabilities aren’t aggressively managed. A container management process that includes frequent scanning — both pre-and post-image build and launch — orchestration engine patching and base/parent image standards will go a long way towards ensuring that only safe containers are being deployed.
About the Author
Pushkar is Director Development in Symantec Enterprise Division of Broadcom Inc. He has been leading Data Loss Prevention (DLP) and Cloud Access Security Broker (CASB) solutions. He has more than 15 years of professional experience in Cyber security and enterprise software.
