Nestybox Blog System Containers as Docker Sandboxes

Introduction For software developers that use Docker, it’s often useful to have an isolated environment on which to experiment with Docker. This allows playing with Docker configurations and images without affecting the underlying host. While this can be done by deploying a VM, it’s a heavy-weight solution. An alternative is to use Docker’s official Docker-in-Docker (DinD) image to run Docker inside a Docker container. But it’s not the best solution because it’s very unsecure and too restrictive as described a bit later in this article. The Nestybox container runtime, Sysbox, provides a better solution. It integrates with Docker and enables it to deploy system containers. These containers support running Docker inside, but securely and with fewer restrictions. This article shows how to quickly and easily deploy Docker sandbox environments using Nestybox system containers. See it work! Contents Why use Docker Sandboxes? Problems with Docker’s “Docker-In-Docker� Solution Nestybox’s Solution Quick Word on System Containers running Multiple Services Using a System Container as a Docker Sandbox Docker Sandbox with Systemd, Docker, and sshd Creating A Snapshot Of A Docker Sandbox Pre-loading Inner Container Images Persistence of Inner Container Images Conclusion Try it for Free! Why use Docker Sandboxes? If you are a software developer or system administrator, using a Docker container as “Docker sandbox� allows you to: Play and experiment with Docker images and configurations without affecting the underlying host. Deploy multiple Docker sandboxed environments within the same host machine, isolated from each other. Avoid the need for resource hungry VMs for this same purpose. Split Docker image storage across storage devices. Problems with Docker’s “Docker-In-Docker� Solution Docker offers an official “docker-in-docker� (DinD) image for deploying Docker inside a Docker container. But it has the following drawbacks & limitations: It requires deployment of a privileged Docker container, which is very unsecure and provides no isolation between the container and the host. An error inside the container can mess up your host’s configuration (e.g., root inside the container is root on the host, will all capabilities enabled and with access to all host devices!). Runs a single service (Docker) inside the Docker container. This may be too restrictive for some users. Leads to Docker volume sprawl on the host. Each time you start a DinD image, Docker creates a volume on the host to store the inner container images. When you stop and remove the container, the volume remains, wasting storage on your machine. In addition: It does not support using a Dockerfile to build a Docker container sandbox that comes pre-loaded with inner images. And it does not support snapshots of the outer Docker container that include inner container images. Nestybox’s Solution The Nestybox container runtime, Sysbox, solves all of the limitations listed in the prior section. It integrates with Docker to enable it to deploy system containers. A system container can be used as secure Docker sandbox, with the following benefits: Easily deploy multiple light-weight, totally-isolated Docker sandbox environments within the same host quickly (using Docker itself). Run one or more services inside the system container Docker sandbox, according to your needs. Avoid using unsecure privileged containers. An error within the system container won’t affect the underlying host. Give unprivileged users on the host access to their own instance of a Docker engine, without granting them root or Docker group access on the host. Add an extra layer of security between your application containers and the underlying host. Use a Dockerfile to build system containers that come pre-loaded with inner container images (saving you the need to pull inner images from the network every time). Use docker commit to snapshot the system container contents, including inner images. The following sections show examples on how to use system containers to realize the benefits listed above. If you want to try the examples that follow, you must first install the Nestybox system container runtime Sysbox in your machine. You can get Sysbox for free here. Once you install it, you simply deploy system containers with Docker as shown in the examples below. Quick Word on System Containers running Multiple Services Normally Docker containers run a single application or micro-service inside of them. In contrast, Nestybox system containers are typically configured with multiple services as they are often used as virtual host environments. Of course, you can always run a single system level service on them. You decide what’s best. Using a System Container as a Docker Sandbox To use a Nestybox system container as a Docker sandbox, simply deploy a system container image that includes Docker inside of it. Nestybox has examples of such images in its DockerHub repository. The corresponding Dockerfiles are here. For example, we have a light-weight Alpine-based system container image that has Docker in it. You deploy the system container with Docker itself, by pointing it to the Nestybox system container runtime Sysbox: $ docker run --runtime=sysbox-runc -it --hostname=syscont nestybox/alpine-docker:latest / # Within the system container, you can then start Dockerd as follows: / # which docker /usr/bin/docker / # dockerd > /var/log/dockerd.log 2>&1 & / # tail /var/log/dockerd.log time="2019-10-23T20:48:51.960846074Z" level=warning msg="Your kernel does not support cgroup rt runtime" time="2019-10-23T20:48:51.960860148Z" level=warning msg="Your kernel does not support cgroup blkio weight" time="2019-10-23T20:48:51.960872060Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" time="2019-10-23T20:48:52.146157113Z" level=info msg="Loading containers: start." time="2019-10-23T20:48:52.235036055Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address" time="2019-10-23T20:48:52.324207525Z" level=info msg="Loading containers: done." time="2019-10-23T20:48:52.476235437Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: failed to set opaque flag on middle layer: operation not permitted" storage-driver=overlay2 time="2019-10-23T20:48:52.476418516Z" level=info msg="Docker daemon" commit=0dd43dd87fd530113bf44c9bba9ad8b20ce4637f graphdriver(s)=overlay2 version=18.09.8-ce time="2019-10-23T20:48:52.476533826Z" level=info msg="Daemon has completed initialization" time="2019-10-23T20:48:52.489489309Z" level=info msg="API listen on /var/run/docker.sock" / # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Once Dockerd is started, you can start using it to deploy inner containers: / # docker run -it busybox Unable to find image 'busybox:latest' locally latest: Pulling from library/busybox 7c9d20b9b6cd: Pull complete Digest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51e Status: Downloaded newer image for busybox:latest / # The inner container (busybox in this example) runs within the system container. The Docker engine on the host is completely unaware of its existence. You can deploy multiple system containers this way, and each will act as a Docker sandbox that is securely isolated from the underlying host and from all other system containers. Within each sandbox you can configure Docker as you wish, without affecting the rest of the system. Docker Sandbox with Systemd, Docker, and sshd In the prior example, we used a system container image was based on Alpine and did not contain a process manager within it. Let’s improve on this by including a process manager. We’ve chosen Systemd for this example as it gives you a system-container that resembles a physical host or VM. Alternatively you can choose a lighter-weight process manager such as Supervisord. The Nestybox DockerHub repository has system container images for this. Let’s deploy the system container image that includes Systemd, Dockerd, and sshd. The image is called nestybox/ubuntu-bionic-systemd-docker:latest and it’s in the Nestybox DockerHub repository. $ docker run --runtime=sysbox-runc -it --rm -P --hostname=syscont nestybox/ubuntu-bionic-systemd-docker:latest systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid) Detected virtualization container-other. Detected architecture x86-64. Welcome to Ubuntu 18.04.3 LTS! Set hostname to <syscont>. ... [ OK ] Started Docker Application Container Engine. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Ubuntu 18.04.3 LTS syscont console syscont login: The -P option tells Docker to publish all system container ports exposed by the Docker image, which in our case is port 22 (the ssh port). In the system container image we are using, we’ve configured the default console login and password to be admin/admin. You can always change this in the image’s Dockerfile. syscont login: admin Password: Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 5.0.0-31-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage T