Setup Policy Controller Constraints

OpenShift clusters contain some default SCCs used to define policies for Pods, as the actions that a Pod can perform and what resources it can access on the nodes.

Anthos provides, as part of Anthos Config Management (ACM), ACM Policy Controller, a Kubernetes dynamic admission controller, based on the Open Policy Agent (OPA) Gatekeeper open source project, that checks, audits, and enforces your clusters' compliance with centrally defined policies implemented as constraints.

In this section we will see how to use ACM Policy Controller constraints to implement, on your target Anthos cluster, a set of security policies that mirrors the ones defined on the source OpenShift cluster by SCCs.

The ocp-scc_2_opa-constraints.yaml file provides a list of constraints that allow you to map, on your target Anthos cluster, the following OpenShift default SCCs:

• privileged
• anyuid
• nonroot
• restricted

Pods will get different policy controller constraints, mapping the policies defined in a source OpenShift environment by the above SCCs, depending on the namespace they run in, as explained below:

• Workloads that need the highest privileged access, such as the ability to run in privileged mode or access any host resource, should run in one of the exempt namespaces that are defined in Policy Controller configuration, to which no constraints are applied. These workloads are typically system components or any workload to which the privileged SCC was applied in the source OpenShift environment.

• All pods created in any namespace that is not in the list of exempt namespaces will get the most restrictive constraints, as denied access to all host features and requirement to run with a UID that is part of a specific range. This configuration matches the policies applied by OpenShift restricted SCC. Exception to this are:

  • Pods created in a namespace that have the security=anyuid label will have the same above restrictive constraints but will be allowed to run with any UID and any GID, matching what the anyuid SCC does on OpenShift.
  • Pods created in a namespace that have the security=nonroot label will have the same restrictive above constraints but will be allowed to run with any non-root UID, matching what the nonroot SCC does on OpenShift.

Note: You may want to review and change the constraints or their scope in this file based on your organization's needs.

• Copy policies to your ACM configurations folder

mkdir -p clusterconfigs/cluster/policies/restricted
cp ./policies/restricted/ocp-scc_2_opa-constraints.yaml clusterconfigs/cluster/policies/restricted 

• Commit these changes to the git repo. ACM Policy Controller will apply these constraints to the target Anthos cluster.

• Verify that the constraints are installed by running kubectl get constraints. You should see a list of constraints applied.

NAME                                                                       AGE
k8spspallowedusers.constraints.gatekeeper.sh/nonroot-users-and-groups      19s
k8spspallowedusers.constraints.gatekeeper.sh/restricted-users-and-groups   19s

NAME                                                                           AGE
k8spspprivilegedcontainer.constraints.gatekeeper.sh/psp-privileged-container   20s

NAME                                                                   AGE
k8spspcapabilities.constraints.gatekeeper.sh/anyuid-capabilities       20s
k8spspcapabilities.constraints.gatekeeper.sh/restricted-capabilities   20s

NAME                                                               AGE
k8spsphostnamespace.constraints.gatekeeper.sh/psp-host-namespace   20s

NAME                                                           AGE
k8spspvolumetypes.constraints.gatekeeper.sh/psp-volume-types   20s

NAME                                                                         AGE
k8spsphostnetworkingports.constraints.gatekeeper.sh/psp-host-network-ports   20s

• Test by creating a workload that requires to run as privileged so allowed only in the exempt namespaces

kubectl create ns test-policy
kubectl create -n test-policy -f policies/restricted/example_allowed-by-privileged.yaml

And notice the rejection.

Error from server ([denied by psp-privileged-container] Privileged container is not allowed: sec-ctx-demo, securityContext: {"privileged": true}
[denied by capabilities-restricted] container <sec-ctx-demo> is not dropping all required capabilities. Container must drop all of ["KILL", "MKNOD", "SYS_CHROOT"]
[denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run as disallowed user 0. Allowed runAsUser: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}
[denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run without a required securityContext/fsGroup. Allowed fsGroup: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}): error when creating "example_allowed-by-privileged.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by psp-privileged-container] Privileged container is not allowed: sec-ctx-demo, securityContext: {"privileged": true}
[denied by capabilities-restricted] container <sec-ctx-demo> is not dropping all required capabilities. Container must drop all of ["KILL", "MKNOD", "SYS_CHROOT"]
[denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run as disallowed user 0. Allowed runAsUser: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}
[denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run without a required securityContext/fsGroup. Allowed fsGroup: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}

• Now test the same workload in one of the exempt namespaces, for example gke-connect (if you want to run this test in a production environment it is advisable to not use the gke-connect namespace as it contains system components. Create instead the pod in a specific namespace dedicated to privileged containers and exempted from constraints as described in the ACM Policy Controller set up instructions)

kubectl create -n gke-connect -f policies/restricted/example_allowed-by-privileged.yaml

and notice it gets deployed successfully

pod/privileged-compliant-pod created

• Test by creating a workload that requires to run as root, so it will be allowed only in namespaces with the specific security=anyuid label

kubectl create -n test-policy -f policies/restricted/example_allowed-by-anyuid.yaml

And notice the rejection.

Error from server ([denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run without a required securityContext/runAsUser. Allowed runAsUser: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}
[denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run without a required securityContext/fsGroup. Allowed fsGroup: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}
[denied by capabilities-restricted] container <sec-ctx-demo> is not dropping all required capabilities. Container must drop all of ["KILL", "MKNOD", "SYS_CHROOT"]): error when creating "example_allowed-by-anyuid.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run without a required securityContext/runAsUser. Allowed runAsUser: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}
[denied by restricted-users-and-groups] Container sec-ctx-demo is attempting to run without a required securityContext/fsGroup. Allowed fsGroup: {"ranges": [{"max": 2000, "min": 1000}], "rule": "MustRunAs"}
[denied by capabilities-restricted] container <sec-ctx-demo> is not dropping all required capabilities. Container must drop all of ["KILL", "MKNOD", "SYS_CHROOT"]

• Now test the same workload in a namespace that has been labeled to allow any UID, let's do this simply applying the security=anyuid label to the same namespace

kubectl label ns test-policy security=anyuid
kubectl create -n test-policy -f policies/restricted/example_allowed-by-anyuid.yaml

and notice it gets deployed successfully

pod/anyuid-compliant-pod created

• Cleanup

kubectl delete ns test-policy
kubectl -n gke-connect delete pod privileged-compliant-pod