This post will be updated regularly.
Last Update: 10/28/2020

Background

Threat Modeling

  • Docker containers are not a security boundary
  • RCE in application running in a container (We want to limit the damage)
  • Restrict isolated attacker on the network as much as possible (Principle of least privilege)

Concerns

  • Want to avoid limiting developer productivity as much as possible

Controls

Secure Architecture

Problem/Need Description Solutions
Data in transit encryption
  • Network traffic should never be sent unencrypted, even if it's only within the local VPC.
  • istio/Nebula can be added as a sidecar to services with a simple annotation.
Secret Management
  • Just like in non-Dockerized services, credentials, keys and other secrets should not be stored in source code.
  • Use Vault rather than Kubernetes secrets or other solutions
  • Integrates with the reset of infrastructure
  • Restricts the ability for containers to read secrets intended for other containers in the cluster
  • Vault can be added as a sidecar to services with a simple annotation and should be used for all secret management in a service.
Protected registries
  • Only CI can push to prod registry
  • Prod can't pull from dev registry
  • ...
Segment sensitive workloads by cluster
  • Group clusters with similar workloads, e.g.: Internal tools, File processing
  • Strong workload isolation
  • Mitigates the consequences of a container escape
  • Cons: more expensive and additional complexity
  • Apply workload segregration during architecture design

Development Security

Problem/Need Description Solutions
Malicious base image Backdoored images downloaded 5 million times
  • Regular Image Scan
Limit base images
  • How can we decide what to trust whithout a huge amount of manual effort?
  • Only allow "offical" and internal images
Keep dependencies up to date
  • How do you patch the next Shellshock in hundreds of containers?
  • Build images regularly in CI
  • Good tests required to ensure functionality does not break
Secure Development Lifecycle (SDL)
  • Ensures all code and serivces are reviewed by Product Security Team before shipping
  • Helps catch anything our automated techniques may miss
  • Implement Tollgate
Linting
  • Lint Dockerfiles and deployment.yaml
  • Kube Linter
  • Run before merging to master to get maximum attention
Hardening scripts
  • A typical container will by its nature have a large number of binaries in it.
  • Many of these will not be necessary for container to operate its service and as such can be stripped out and removed as part off the build process in order to reduce the attack surface.
  • Develop and Run Hardening script before image build

Infrastructure Security

Problem/Need Description Solutions
Exposed cluster management Hackers Enlisted Tesla's Public Cloud to Mine Cryptocurrency
  • Restrict and monitoring K8s dashboards/API access
  • Restrict ccess to kubectl
  • Only available to small, trusted group of operators in production
  • An attacker with develop-level privilege cannot deploy production containers
Controlling access to the K8s API
  • Admission Controller able to enforce a lot of these restrictions
  • Written in golang
  • Implement Admission Controller

Run Time Security

Insecure container configuration
  • Be careful with volume mounts (Everything is a file: /proc)
  • Running at root
  • Privileged
  • Monitoring K8s Metadata
Don't run as the root user
  • By default, Docker containers run as the root user and group.
  • This can lead to issues where the root user can be used as part of a container breakout to the host.
  • K8s Security Scan
  • Use a SecurityContext
  • Monitor K8s metadata
Don't run as privileged
  • Docker containers can optionally be run with the --privileged flag to give them additional access to the file system and the host machine. This is very dangerous and should be avoided in almost all circumstances.
  • K8s Security Scan
  • Monitor K8s metadata
Drop capabilities where possible
  • In additional to sudo, Linux has slightly more fine grained privileges that can be applied call 'capabilities'.
  • If they are not required, they can be dropped from a service, further reducing the scope of a service's abilities.
  • This doesn't work with ambient capabilities
  • Secure computing mode (seccomp) is a Linux kernel feature. You can use it to restrict the actions available within the container.
  • It can add restrictions on a syscall by sycall basis.
  • Unfortunately, the process of identifying the syscalls that a service needs is very manual and time consuming, and prone to errors and changes as a service is updated.
  • AppArmor
Read only filesystem where possible
  • A lot of services do not need to write to the filesystem in order to operate, for example if they simply process network requests and write their results to a database.
  • In the event of a container compromise, making the filesystem read only would prevent an attacker from being able to download any tools or exfiltrating data a temporary location in the container before extraction.
  • Use a SecurityContext
  • Monitor K8s metadata
Resource limits
  • Due to the risk of resource exhaustion, either from a malfunctioning service or from deliberate misuse, containers should be given resource limits so that they don't overwhelm the host system's resources
  • Apply reousrce limits
  • Note: due to performance issues, it is not recommended to set a CPU limit, only a memory limit.
Use a SecurityContext
  • SecurityContexts can be used "to define privilege and access control settings for a pod or container"
  • runAsUser, runAsGroup, runAsNonRoot, allowPrivilegeEscalation, capabilities, readOnlyRootFileSystem
Audit instrumentation
  • Go Audit / auditd / auditbeat