Deploying Secure Containers for Training and Development

Deploying Secure Containers for Training and Development

Deploying Secure Containers for Training and Development
Jon Schipp
Contributing Editor
Henry Dalziel
AMSTERDAM • BOSTON • HEIDELBERG • LONDON
NEW YORK • OXFORD • PARIS • SAN DIEGO
SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO
Syngress is an imprint of Elsevier

Syngress is an imprint of Elsevier
225 Wyman Street, Waltham, MA 02451, USA
Copyright r 2016 Elsevier Inc. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher
’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency,
can be found at our website:
www.elsevier.com/permissions
This book and the individual contributions contained in it are protected under copyright by the
Publisher (other than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods or professional practices,
may become necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information or methods described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors,
assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products,
instructions, or ideas contained in the material herein.
ISBN: 978-0-12-804717-0
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data
A catalog record for this book is available from the Library of Congress
For Information on all Syngress publications visit our website at http://store.elsevier.com/Syngress

ABOUT THE AUTHORS
Henry Dalziel is a serial education entrepreneur, founder of Concise Ac
Ltd, online cybersecurity blogger, and e-book author. He writes for the
Concise-Courses.com blog and has developed numerous cybersecurity continuing education courses and books. Concise Ac Ltd develops and distributes continuing education content [books and courses] for cyber- security professionals seeking skill enhancement and career advance- ment. The company was recently accepted onto the UK Trade &
Investment
’s (UKTI) Global Entrepreneur Programme (GEP).
Jon Schipp is a Security Engineer at the National Center for
Supercomputing Applications, Director of Security at Draconyx, as well as the founder and chair of OpenNSM at the University of Illinois at
Urbana-Champaign. He is the author of ISLET, among other tools, as well as a contributor to many Free and Open Source Projects including
The Netsniff-NG Toolkit, SecurityOnion, and the Bro Project. With a few publications and many talks, he has been fortunate enough to have audi- ences at various conferences including DerbyCon, AIDE, Hack3rCon,
BroCon, XSEDE, MOSSCon, and more. In his free time, he enjoys reading about history and philosophy, listening to death metal and playing recreational volleyball.

INTRODUCTION
I first started looking into containers as a way to scale training environ- ments for BroCon 2014. We were trying to find a more sufficient solution to teach the Bro programming language that involved less setup time. We found that we can have hundreds of training environments on a single host, rather than distributing large virtual machine files to all conference users. Upon some research and experimentation, I found that Linux con- tainers were a good solution to this problem. I have since explored ways to further apply this technology to other remote areas, which include deploying secure containers for training and development. This short book is intended to present concepts and to spark ideas rather than be a walkthrough or tutorial of software. My name is Jon Schipp; I
’m a
Security
Engineer at the
National
Center for
Supercomputing
Applications (NCSA) at the University of Illinois at Urbana-Champaign.
The NCSA takes advantage of high-performance computing to facilitate scientific research. I am also the Director of Security at Draconyx, a full- service IT consulting business that specializes in information security. In addition, I have contributed to a number of free and open source projects,
including the Netsniff-NG Toolkit, Security Onion, and the Bro Project. I
am the founder and chair of OpenNSM, which is a weekly network secu- rity monitoring group with international participation.
Project Contributions
• Bro Team
• The Netsniff-NG Toolkit
• SecurityOnion
• OpenNSM

CHAPTER
1 1
Containers
Our Focus:
•
Event training
•
Internal software training
•
Development environments
•
Playing around
Excels at teaching command-line software on Linux
Containers are typically used to deploy web applications and soft- ware stacks. An example of this is deploying software such as
WordPress, MySQL, or just hosting your website in a container run- ning Apache or NGiNX. Reproducibility and scalability are two bene- fits of deployment with containers. You can think of container as a package of software including libraries, services, and the application itself. Having your application and all its dependencies wrapped up in a container allows you to deploy it to any system that uses the same container runtime engine or container specification. Reproducibility is achieved in that way, in addition it
’s easily scalable across multiple sys- tems when network traffic is high or provides redundancy when a sys- tem needs an upgrade. All one has to do is configure the container to run on multiple systems and tell a load balancer where to redirect traf- fic. Tools like Kubernetes, Docker Swarm, and others make it easy to scale containers across hosts. While these are typical uses, we will look into the use of containers for software development and training. For example, we can utilize Linux containers to provide a package of a filesystem containing all the common tools used for training on the
GNU/Linux operating system. Another example is to compile and run applications for testing and deployment in containers without installing them on your host filesystem. The result of which ends up providing a tidier filesystem and eliminates inconveniences of managing multiple versions of shared libraries or programs.
Deploying Secure Containers for Training and Development. DOI:
http://dx.doi.org/10.1016/B978-0-12-804717-0.00001-6
© 2016 Elsevier Inc. All rights reserved.

Isolation
Scalability via kernel namespaces and cgroups
The user can’t tell the difference.
Lightweight
100ms startup time, near bare metal performance, JeOS
Density
Higher density than virtual machines
Horizontally and vertically with faster hardware meaning more users or work can be performed
Let
’s move on to describe what containers are and what they can pro- vide. Containers enable a layer of isolation and this is done in the Linux
Kernel via two kernel components:
namespaces and cgroups.
Namespaces divide the operating system into virtual segments. For exam- ple, assigning a process to namespace A makes it invisible and unreach- able to a process in namespace B, even if root uses the
“ps” command in a different namespace. Cgroups are short for control groups, which can apply constraints, or limitations, to system resources such as memory, by utilization; and CPU, by a percentage of its use.
From the perspective of the user, interacting with software such as a web application in a container, a virtual machine, or on bare metal, acts and looks the same. The average user is not able to tell what kind of sys- tem they are in, or what kind of level of virtualization that they are using.
Scalability with containers seems to be very easy to do and that
’s one of its promising points. You can scale both horizontally and verti- cally. With vertical scaling we scale by applying more resources to the host: more CPU, more memory, faster disks, etc. You can do this with a virtual machine as well, any software really
. . . You can add more resources to the box and thus can usually perform more work. More interesting though is the number of tools that exist to scale horizontally by clustering nodes to run containers. The Apache Mesos project is one such example where running your application in a container can be scaled over pools of hundreds or thousands of different machines.
Newer tools include Google
’s Kubernetes and Docker’s very own
Swarm.
2
Deploying Secure Containers for Training and Development

Containers tend to be lightweight in size which makes them suitable for scaling. The acronym JeOS (Just Enough Operating
System) indicates this, in other words it
’s alright to say that many images are bare bones. A testament of this is that while using a Debian or Ubuntu Docker container image and issuing the
“man” command to view the manual pages will result in a
“command not found” error from the shell. This is because even the man-doc package isn
’t installed!
They tend to be measured in hundreds of megabytes instead of the full distribution one installs which is usually much larger.
Containers share the kernel that is running on the host. This means there is one running kernel, the Linux kernel, and it
’s asked for resources directly. The application can ask it to write a buffer to disk, to send a packet across the network, to read from an open file, etc. without having to pass instructions through another layer such as a Hypervisor as is the case in Virtual Machines. This tends to give us a performance increase because the system has to perform less work for kernel requests i.e. system calls. Less instructions and translations means less work and the result is higher level of density for containers than you would have with virtual machines. That means you can run many more containers on a host than you would have been able to with virtual machines. It
’s not unheard of to be able to run 1000 containers on commodity hardware.
In sum, applications in containers tend to run faster than their counter- parts in a virtual machine. A user can get near bare metal performance in the right conditions and the startup time for a container is often less than a second where multiple seconds is the norm when booting virtual machines.
Security
Portability
Less secure than virtual machines, containers isolate the user land (e.g filesystem, processes) not the kernel
Concurrency
Some implementations are designed to run a single application or process
Kernel Versions
Containers must use the same kernel as the host
Less portable across operating systems. Tends to be portable within an operating system.
3
Containers

Let
’s delve into a few other differences on security, portability, con- currency, and the kernel. Containers happen to be less secure than vir- tual machines. That
’s not to say that they’re not secure but rather they
’re less secure than virtual machines. This is mainly because con- tainers isolate their user land e.g. filesystem, processes etc. and not the kernel. Virtual machines will isolate the entire operating system: the kernel, user land, and all.
On the topic of portability containers tend to be less portable across operating systems. Note that containers are a subset of a virtualization type called light weight operating system virtualization, also known as lightweight process virtualization. These technical terms define an entire domain of technology that allows the operating system to run multiple instances of itself on the same host. Different operating sys- tems have different ways of implementing this and the word containers often refer to the Linux kernel implementation. Circling back around,
this means that if you run a container on GNU/Linux it
’s not going to be able to run on FreeBSD or SmartOS because each have their own operating system virtualization implementation. However, within a
GNU/Linux system you can run containers across different distribu- tions as long as the run time implementation is supported. Linux con- tainer runtime implementations such as Docker will run across Debian and Fedora families among others provided the kernel has the neces- sary features included for running containers. A modern kernel will have everything you need to run containers.
A note on concurrency is that some container implementations such as Docker are designed to run a single application or process inside the container. Docker is designed in a way that advocates running one pro- cess per container. An example use case of this is where you have 10 web- sites that need to be isolated and served to the world. Each website is placed in a container where it
’s running an Apache process to serve the web content. While running 10 instances of Apache may seem inefficient,
one gains from the extra layer of security that is provided. If an attacker compromises an Apache daemon in a container and can access the sys- tem and issue commands
—they’re not able to see or interact with the other websites running in the other 9 containers. The design of a single process per container is not a hard limit; Docker doesn
’t disallow it as there are plenty of use cases. You can run multiple processes by making the container
’s first process (PID 1) an init system that can spawn and
4
Deploying Secure Containers for Training and Development

manage other processes in the container. Docker doesn
’t provide an init system which is why multiple process cannot be managed but installing one such as supervisord is simple enough.
Regarding kernel versions, a container as you may remember uses the same running kernel as the host. Because of this we are not able to experiment with different kernel versions. You can
’t have a container that
’s using Linux kernel 4.1 and another container running 4.2 and your host running 4.3. Virtual machines don
’t have this problem because they virtualize an entire operating system. The benefit of shar- ing the kernel is what allows us to get that performance benefit that we talked about in the previous slide.
Containers
Important:
"Linux Based Containers"
There is no internal container specification.
As of recent, there is a container runtime specification called OCI
There are different container (and like) technologies
Linux:
LXC, OpenVZ, Google containers, etc.
Non-Linux:
BSD Jails, Solaris Zones, AIX WPAR, etc.
The technology isn’t new, but it’s gaining great momentum now.
What do containers do?
Light-weight process virtualization, A.K.A. operating system virtualization
What do virtual machines do?
Hardware virtualization
×
×
×
×
×
A note on terminology, when referring to Linux containers as a general technology it
’s better to use the more encompassing term
Linux-based containers. There are many runtime implementations of
Linux-based containers such as LXC which is short for Linux contain- ers and thus causes confusion. In addition there
’s also no kernel con- tainer specification, there are a few building blocks in the kernel that runtime implementations take advantage of to create containers.
Though, as of recent there is an effort to standardize a container runtime specification called OCI (Open Container Initiative); before
OCI the CoreOS group worked on a specification called Appsec which was subsumed into OCI. OCI is a Linux Foundation project that is
5
Containers

dedicated to providing a specification: a set of guidelines and rules for the packaging, deployment, and runtime of containers. Some tools fol- low this and some do not. The specification will be adopted by more runtime implementations as it becomes more mature.
On GNU/Linux I mentioned that there were multiple implementa- tions of containers. Popular ones like Docker, LXC, and Rocket are gaining traction. Many other exists including lesser known ones like
OpenVZ which uses a modified Linux kernel that the project maintains to provide resource controls and scheduling.
On non-Linux systems, but UNIX derivatives, the BSDs have jails,
Solaris has Zones, and AIX has Work Partitions, etc. Operating sys- tem virtualization technology isn
’t new, it’s been around for a long time. FreeBSD jails, one of the earliest implementations, came out in
2000 which is 15 years ago at the time of publication. The technology is gaining a lot of momentum as of late because of the high level of adoption rate for the GNU/Linux operating system as well as the advent of better runtime implementations like Docker that made crea- tion and deployment convenient and workable for system administra- tors and developers.
Contrast to operating system virtualization with virtual machines which provide hardware virtualization, virtual machines have to trans- late hardware instructions; they have to worry about hardware such as busses, and software such as schedulers for the guests. The hypervisor is additional layer of software required to create and manage virtual machines. Although hardware virtualization is getting better and faster,
containers simply don
’t have to do as much work. A simplified illustra- tion is an application in a container which makes a system call to the kernel to ask for a resource such as opening a file for reading and then the kernel (shared with the host) responds back with the result of the system call and the container
’s application moves on. Whereas in the case of an application in a virtual machine it makes a system call,
the kernel in the guest services the call but needs to talk to the hypervi- sor which is the real manager of the hardware (the file resides on disk).
The hypervisor translating for the hardware gives the kernel what it needs, then the kernel in the guest returns back to the application which can proceed. In general you can see that this is a longer code path,
though things are constantly improving over time.
6
Deploying Secure Containers for Training and Development

Linux Kernel Stuff
Support:
3.8 introduce the final building block for containers
Namespaces:
Process isolation
Currently available: pid, net, ipc, uts, mnt, and user
Control Groups:
Resource management e.g. cpu, cpuset, blkio, memory, etc.
It’s not magic, you can create namespaces and cgroups directly from your shell by modifying procfs and sysfs.
That’s how they were deployed before userland tools like
LXC and Docker existed
×
×
The Linux 3.8 kernel introduced what could be said to be the most complete set of building blocks for containers yet. Earlier versions from
2.4.19 had namespaces and more were added over time but not enough to give us the capabilities we have now. The two building blocks we spoke briefly about earlier are namespaces and control groups.
Namespaces provide a way to isolate resources. There are currently 6
namespaces provided by the kernel and they include: PID, Network,
Mount, IPC, UTS, and User. The user namespace which adds addi- tional security by mapping users such as root inside a container to an unprivileged user outside the container is the newest of them and not implemented in all runtimes at the time of this writing. An example of the PID namespace follows. Process Z is assigned to namespace A and process Y is assigned to namespace B. A user enters namespace A and runs top but only sees Process Z, correspondingly a user enters name- space B and runs top but only sees Process Y. The same with root entering namespace A or B and running top. Though, if on the host with the global namespace the root user can see that Process Z is run- ning in namespace A and that Process Y is running in namespace B.
You can see that the PID namespace effectively isolates the view of the operating system for processes. Note that namespaces don
’t have names, rather they have inode numbers, letters were assigned to sim- plify examples. Another example includes the Network namespace which allows each container to have its own network interface and IP
address to communicate. You can make a web request to the URL of the Apache process running in namespace A and get an entirely differ- ent website from the Apache process running in namespace B with a different URL. Apache in namespace A and Apache in namespace B
are not able to communicate cross containers unless configured.
7
Containers

Control groups provide a level of resource management or control on containers. Resources like memory utilization or amount of mem- ory allocated to a container, as well as limiting the disk read and write throughput that a container can service. CPU limitations such as shar- ing and quota values per container allow prioritization of processes in containers or prioritization of entire containers. In sum, Namespaces and cgroups when used together form what we know as containers.
One can create their own namespaces using the clone(2) system call.
Cgroups can be created directly from the shell by mounting the pseudo cgroup filesystem and then interacting with its files in sysfs.
Linux Containers?
Docker