getDevices() has been updated to skip `/dev/.lxc` and `/dev/.lxd-mounts`, which was breaking privileged Docker containers running on runC, inside of LXD managed Linux Containers
Signed-off-by: Carlton-Semple <carlton.semple@ibm.com>
CRIU gets pre-dump to complete iterative migration.
pre-dump saves process memory info only. And it need parent-path
to specify the former memory files.
This patch add pre-dump and parent-path arguments to runc checkpoint
Signed-off-by: Deng Guangxing <dengguangxing@huawei.com>
Signed-off-by: Adrian Reber <areber@redhat.com>
As the runtime-spec allows it, we want to be able to specify overlayfs
mounts with:
{
"destination": "/etc/pki",
"type": "overlay",
"source": "overlay",
"options": [
"lowerdir=/etc/pki:/home/amurdaca/go/src/github.com/opencontainers/runc/rootfs_fedora/etc/pki"
]
},
This patch takes care of allowing overlayfs mounts. Both RO and RW
should be supported.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
`label.InitLabels` takes options as a string slice in the form of:
user:system_u
role:system_r
type:container_t
level:s0:c4,c5
However, `DupSecOpt` and `DisableSecOpt` were still adding a docker
specifc `label=` in front of every option. That leads to `InitLabels`
not being able to correctly init selinux labels in this scenario for
instance:
label.InitLabels(DupSecOpt([%OPTIONS%]))
if `%OPTIONS` has options prefixed with `label=`, that's going to fail.
Fix this by removing that docker specific `label=` prefix.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
Correct container.Destroy() docs to clarify that destroy can only operate on containers in specific states.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
If a relative pathed exe is used for InitArgs init will fail to run if Cwd is not set the original path.
Prevent failure of init to run by ensuring that exe in InitArgs is an absolute path.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
If we pass a file descriptor to the host filesystem while joining a
container, there is a race condition where a process inside the
container can ptrace(2) the joining process and stop it from closing its
file descriptor to the stateDirFd. Then the process can access the
*host* filesystem from that file descriptor. This was fixed in part by
5d93fed3d2 ("Set init processes as non-dumpable"), but that fix is
more of a hail-mary than an actual fix for the underlying issue.
To fix this, don't open or pass the stateDirFd to the init process
unless we're creating a new container. A proper fix for this would be to
remove the need for even passing around directory file descriptors
(which are quite dangerous in the context of mount namespaces).
There is still an issue with containers that have CAP_SYS_PTRACE and are
using the setns(2)-style of joining a container namespace. Currently I'm
not really sure how to fix it without rampant layer violation.
Fixes: CVE-2016-9962
Fixes: 5d93fed3d2 ("Set init processes as non-dumpable")
Signed-off-by: Aleksa Sarai <asarai@suse.de>
When signaling children and the signal is SIGKILL wait for children
otherwise conditionally wait for children which are ready to report.
This reaps all children which exited due to the signal sent without
blocking indefinitely.
Also:
* Ignore ignore ECHILD, which means the child has already gone.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
Ensure that the pipe is always closed during the error processing of StartInitialization.
Also:
* Fix a comment typo.
* Use newContainerInit directly as there's no need for i to be an initer.
* Move the comment about the behaviour of Init() directly above it, clarifying what happens for all defers.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
Add the import of nsenter to the example in libcontainer's README.md, as without it none of the example code works.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
POSIX mandates that `cmsg_len` in `struct cmsghdr` is a `socklen_t`,
which is an `unsigned int`. Musl libc as used in Alpine implements
this; Glibc ignores the spec and makes it a `size_t` ie `unsigned long`.
To avoid the `-Wformat=` warning from the `%lu` on Alpine, cast this
to an `unsigned long` always.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Add godoc links to README.md files for runc and libcontainer so its easy to access the golang documentation.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
signalAllProcesses was making the assumption that the requested signal was SIGKILL, possibly due to the signal parameter being added at a later date, and hence it was safe to wait for all processes which is not the case.
BaseContainer.Signal(s os.Signal, all bool) exposes this functionality to consumers, so an arbitrary signal could be used which is not guaranteed to make the processes exit.
Correct the documentation for signalAllProcesses around the signal delivered and update it so that the wait is only performed on SIGKILL hence making it safe to process other signals without risk of blocking forever, while still maintaining compatibility to SIGKILL callers.
Signed-off-by: Steven Hartland <steven.hartland@multiplay.co.uk>
The parameters passed to `GetExecUser` is not correct.
Consider the following code:
```
package main
import (
"fmt"
"io"
"os"
)
func main() {
passwd, err := os.Open("/etc/passwd1")
if err != nil {
passwd = nil
} else {
defer passwd.Close()
}
err = GetUserPasswd(passwd)
if err != nil {
fmt.Printf("%#v\n", err)
}
}
func GetUserPasswd(r io.Reader) error {
if r == nil {
return fmt.Errorf("nil source for passwd-formatted
data")
} else {
fmt.Printf("r = %#v\n", r)
}
return nil
}
```
If the file `/etc/passwd1` is not exist, we expect to return
`nil source for passwd-formatted data` error, and in fact, the func
`GetUserPasswd` return nil.
The same logic exists in runc code. this patch fix it.
Signed-off-by: Wang Long <long.wanglong@huawei.com>
The `bufio.Scanner.Scan` method returns false either by reaching the
end of the input or an error. After Scan returns false, the Err method
will return any error that occurred during scanning, except that if it
was io.EOF, Err will return nil.
We should check the error when Scan return false(out of the for loop).
Signed-off-by: Wang Long <long.wanglong@huawei.com>
This sets the init processes that join and setup the container's
namespaces as non-dumpable before they setns to the container's pid (or
any other ) namespace.
This settings is automatically reset to the default after the Exec in
the container so that it does not change functionality for the
applications that are running inside, just our init processes.
This prevents parent processes, the pid 1 of the container, to ptrace
the init process before it drops caps and other sets LSMs.
This patch also ensures that the stateDirFD being used is still closed
prior to exec, even though it is set as O_CLOEXEC, because of the order
in the kernel.
https://github.com/torvalds/linux/blob/v4.9/fs/exec.c#L1290-L1318
The order during the exec syscall is that the process is set back to
dumpable before O_CLOEXEC are processed.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
On some systems, when we mount some cgroup subsystems into
a same mountpoint, the name sequence of mount options and
cgroup directory name can not be the same.
For example, the mount option is cpuacct,cpu, but
mountpoint name is /sys/fs/cgroup/cpu,cpuacct. In current
runc, we set mount destination name from combining
subsystems, which comes from mount option from
/proc/self/mountinfo, so in my case the name would be
/sys/fs/cgroup/cpuacct,cpu, which is differernt from
host, and will break some applications.
Fix it by using directory name from host mountpoint.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
`HookState` struct should follow definition of `State` in runtime-spec:
* modify json name of `version` to `ociVersion`.
* Remove redundant `Rootfs` field as rootfs can be retrived from
`bundlePath/config.json`.
Signed-off-by: Zhang Wei <zhangwei555@huawei.com>
A remount of a mount point must include all the current flags or
these will be cleared:
```
The mountflags and data arguments should match the values used in the
original mount() call, except for those parameters that are being
deliberately changed.
```
The current code does not do this; the bug manifests in the specified
flags for `/dev` being lost on remount read only at present. As we
need to specify flags, split the code path for this from remounting
paths which are not mount points, as these can only inherit the
existing flags of the path, and these cannot be changed.
In the bind case, remove extra flags from the bind remount. A bind
mount can only be remounted read only, no other flags can be set,
all other flags are inherited from the parent. From the man page:
```
Since Linux 2.6.26, this flag can also be used to make an existing
bind mount read-only by specifying mountflags as:
MS_REMOUNT | MS_BIND | MS_RDONLY
Note that only the MS_RDONLY setting of the bind mount can be changed
in this manner.
```
MS_REC can only be set on the original bind, so move this. See note
in man page on bind mounts:
```
The remaining bits in the mountflags argument are also ignored, with
the exception of MS_REC.
```
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
When checking if the provided networking namespace is the host
one or not, we should first check if it's a symbolic link or not
as in some cases we can use persistent networking namespace under
e.g. /var/run/netns/.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Both suffered from different race conditions.
SelinuxEnabled assigned selinuxEnabledChecked before selinuxEnabled.
Thus racing callers could see the wrong selinuxEnabled.
getSelinuxMountPoint assigned selinuxfs to "" before it know the right
value. Thus racing could see "" improperly.
The gate selinuxfs, enabled, and mclist all on the same lock
This fixes all of the tests that were broken as part of the console
rewrite. This includes fixing the integration tests that used TTY
handling inside libcontainer, as well as the bats integration tests that
needed to be rewritten to use recvtty (as they rely on detached
containers that are running).
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Switch to the actual source of the official Docker library of images, so
that we have a proper source for the test filesystem. In addition,
update to the latest released version (1.25.0 [2016-06-23]) so that we
can use more up-to-date applets in our tests (such as stat(3)).
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This allows for higher-level orchestrators to be able to have access to
the master pty file descriptor without keeping the runC process running.
This is key to having (detach && createTTY) with a _real_ pty created
inside the container, which is then sent to a higher level orchestrator
over an AF_UNIX socket.
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Since the gid=X and mode=Y flags can be set inside config.json as mount
options, don't override them with our own defaults. This avoids
/dev/pts/* not being owned by tty in a regular container, as well as all
of the issues with us implementing grantpt(3) manually. This is the
least opinionated approach to take.
This patch is part of the console rewrite patchset.
Reported-by: Mrunal Patel <mrunalp@gmail.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This implements {createTTY, detach} and all of the combinations and
negations of the two that were previously implemented. There are some
valid questions about out-of-OCI-scope topics like !createTTY and how
things should be handled (why do we dup the current stdio to the
process, and how is that not a security issue). However, these will be
dealt with in a separate patchset.
In order to allow for late console setup, split setupRootfs into the
"preparation" section where all of the mounts are created and the
"finalize" section where we pivot_root and set things as ro. In between
the two we can set up all of the console mountpoints and symlinks we
need.
We use two-stage synchronisation to ensures that when the syscalls are
reordered in a suboptimal way, an out-of-place read() on the parentPipe
will not gobble the ancilliary information.
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
To make the code cleaner, and more clear, refactor the syncT handling
used when creating the `runc init` process. In addition, document the
state changes so that people actually understand what is going on.
Rather than only using syncT for the standard initProcess, use it for
both initProcess and setnsProcess. This removes some special cases, as
well as allowing for the use of syncT with setnsProcess.
Also remove a bunch of the boilerplate around syncT handling.
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This adds C wrappers for sendmsg and recvmsg, specifically used for
passing around file descriptors in Go. The wrappers (sendfd, recvfd)
expect to be called in a context where it makes sense (where the other
side is carrying out the corresponding action).
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
In the cases that we got failure on a subsystem's Apply,
we'll get some subsystems' cgroup directories leftover.
On Docker's point of view, start a container failed, use
`docker rm` to remove the container, but some cgroup files
are leftover.
Sometimes we don't want to clean everyting up when something
went wrong, because we need these inter situation
information to debug what's going on, but cgroup directories
are not useful information we want to keep.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
This PR fix issue in this scenario:
```
in terminal 1:
~# cd /sys/fs/cgroup/cpuset
~# mkdir test
~# cd test
~# cat cpuset.cpus
0-3
~# echo 1 > cpuset.cpu_exclusive (make sure you don't have other cgroups under root)
in terminal 2:
~# echo $$ > /sys/fs/cgroup/cpuset/test/tasks
// set resources.cpu.cpus="0-2" in config.json
~# runc run test1
back to terminal 1:
~# cd test1
~# cat cpuset.cpus
0-2
~# echo 1 > cpuset.cpu_exclusive
in terminal 3:
~# echo $$ > /sys/fs/cgroup/test/tasks
// set resources.cpu.cpus="3" in config.json
~# runc run test2
container_linux.go:247: starting container process caused "process_linux.go:258:
applying cgroup configuration for process caused \"failed to write 0-3\\n to
cpuset.cpus: write /sys/fs/cgroup/cpuset/test2/cpuset.cpus: invalid argument\""
```
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
This allows a user to send a signal to all the processes in the
container within a single atomic action to avoid new processes being
forked off before the signal can be sent.
This is basically taking functionality that we already use being
`delete` and exposing it ok the `kill` command by adding a flag.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This moves the ambient capability support behind an `ambient` build tag
so that it is only compiled upon request.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The default terminal setting for a new pty on Linux (unix98) has +ONLCR,
resulting in '\n' writes by a container process to be converted to
'\r\n' reads by the managing process. This is quite unexpected, and
causes multiple issues with things like bats testing. To fix it, make
the terminal sane after opening it by setting -ONLCR.
This patch might need to be rewritten after the console rewrite patchset
is merged.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
I use the same tool (https://github.com/client9/misspell)
as Daniel used a few days ago, don't why he missed these
typos at that time.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
config.cloneflag is not mandatory, when using `runc exec`,
config.cloneflag can be empty, and even then it won't be
`-1` but `0`.
So this validation is totally wrong and unneeded.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
In user namespaces devices are bind-mounted from the host, so
we need to add them as external mounts for CRIU.
Reported-by: Ross Boucher <boucher@gmail.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
When spec file contains duplicated namespaces, e.g.
specs: specs.Spec{
Linux: &specs.Linux{
Namespaces: []specs.Namespace{
{
Type: "pid",
},
{
Type: "pid",
Path: "/proc/1/ns/pid",
},
},
},
}
runc should report malformed spec instead of using latest one by
default, because this spec could be quite confusing.
Signed-off-by: Zhang Wei <zhangwei555@huawei.com>
Previously we only tested failures, which causes us to miss issues where
setting sysctls would *always* fail.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
When changing this validation, the code actually allowing the validation
to pass was removed. This meant that any net.* sysctl would always fail
to validate.
Fixes: bc84f83344 ("fix docker/docker#27484")
Reported-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This reverts part of the commit eb0a144b5e
That commit introduced two issues.
- We need to make parent mount of rootfs private before bind mounting
rootfs. Otherwise bind mounting root can propagate in other mount
namespaces. (If parent mount is shared).
- It broke test TestRootfsPropagationSharedMount() on Fedora.
On fedora /tmp is a mount point with "shared" propagation. I think
you should be able to reproduce it on other distributions as well
as long as you mount tmpfs on /tmp and make it "shared" propagation.
Reason for failure is that pivot_root() fails. And it fails because
kernel does following check.
IS_MNT_SHARED(new_mnt->mnt_parent)
Say /tmp/foo is new rootfs, we have bind mounted rootfs, so new_mnt
is /tmp/foo, and new_mnt->mnt_parent is /tmp which is "shared" on
fedora and above check fails.
So this change broke few things, it is a good idea to revert part of it.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Namely, use an undocumented feature of pivot_root(2) where
pivot_root(".", ".") is actually a feature and allows you to make the
old_root be tied to your /proc/self/cwd in a way that makes unmounting
easy. Thanks a lot to the LXC developers which came up with this idea
first.
This is the first step of many to allowing runC to work with a
completely read-only rootfs.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Without this patch applied, RHEL's SELinux policies cause container
creation to not really work. Unfortunately this might be an issue for
rootless containers (opencontainers/runc#774) but we'll cross that
bridge when we come to it.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Print the error message to stderr if we are unable to return it back via
the pipe to the parent process. Also, don't panic here as it is most
likely a system or user error and not a programmer error.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
We need support for read/only mounts in SELinux to allow a bunch of
containers to share the same read/only image. In order to do this
we need a new label which allows container processes to read/execute
all files but not write them.
Existing mount label is either shared write or private write. This
label is shared read/execute.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
At some point InitLabels was changed to look for SecuritOptions
separated by a ":" rather then an "=", but DupSecOpt was never
changed to match this default.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
If copyup is specified for a tmpfs mount, then the contents of the
underlying directory are copied into the tmpfs mounted over it.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Depending on your SELinux setup, the order in which you join namespaces
can be important. In general, user namespaces should *always* be joined
and unshared first because then the other namespaces are correctly
pinned and you have the right priviliges within them. This also is very
useful for rootless containers, as well as older kernels that had
essentially broken unshare(2) and clone(2) implementations.
This also includes huge refactorings in how we spawn processes for
complicated reasons that I don't want to get into because it will make
me spiral into a cloud of rage. The reasoning is in the giant comment in
clone_parent. Have fun.
In addition, because we now create multiple children with CLONE_PARENT,
we cannot wait for them to SIGCHLD us in the case of a death. Thus, we
have to resort to having a child kindly send us their exit code before
they die. Hopefully this all works okay, but at this point there's not
much more than we can do.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This avoids us from running into cases where libcontainer thinks that a
particular namespace file is a different type, and makes it a fatal
error rather than causing broken functionality.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
In order to mount root filesystems inside the container's mount
namespace as part of the spec we need to have the ability to do a bind
mount to / as the destination.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Since Linux 4.3 ambient capabilities are available. If set these allow unprivileged child
processes to inherit capabilities, while at present there is no means to set capabilities
on non root processes, other than via filesystem capabilities which are not usually
supported in image formats.
With ambient capabilities non root processes can be given capabilities as well, and so
the main reason to use root in containers goes away, and capabilities work as expected.
The code falls back to the existing behaviour if ambient capabilities are not supported.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
grep -r "range map" showw 3 parts use map to
range enum types, use slice instead can get
better performance and less memory usage.
Signed-off-by: Peng Gao <peng.gao.dut@gmail.com>
For example, the /sys/firmware directory should be masked because it can contain some sensitive files:
- /sys/firmware/acpi/tables/{SLIC,MSDM}: Windows license information:
- /sys/firmware/ibft/target0/chap-secret: iSCSI CHAP secret
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
cgroupData.join method using `WriteCgroupProc` to place the pid into
the proc file, it can avoid attach any pid to the cgroup if -1 is
specified as a pid.
so, replace `writeFile` with `WriteCgroupProc` like `cpuset.go`'s
ApplyDir method.
Signed-off-by: Wang Long <long.wanglong@huawei.com>
if a container state is running or created, the container.Pause()
method can set the state to pausing, and then paused.
this patch update the comment, so it can be consistent with the code.
Signed-off-by: Wang Long <long.wanglong@huawei.com>
Currently if a user does a command like
docker: Error response from daemon: operation not supported.
With this fix they should see a much more informative error message.
docker run -ti -v /proc:/proc:Z fedora sh
docker: Error response from daemon: SELinux Relabeling of /proc is not allowed: operation not supported.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
Error sent from child process is already genericError, if
we don't allow recrusive generic error, we won't get any
cause infomation from parent process.
Before, we got:
WARN[0000] exit status 1
ERRO[0000] operation not permitted
After, we got:
WARN[0000] exit status 1
ERRO[0000] container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"operation not permitted\""
it's not pretty but useful for detecting root causes.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
This allows older state files to be loaded without the unmarshal error
of the string to int conversion.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
After #1009, we don't always set `cgroup.Paths`, so
`getCgroupPath()` will return wrong cgroup path because
it'll take current process's cgroup as the parent, which
would be wrong when we try to find the cgroup path in
`runc ps` and `runc kill`.
Fix it by using `m.GetPath()` to get the true cgroup
paths.
Reported-by: Yang Shukui <yangshukui@huawei.com>
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Alternative of #895 , part of #892
The intension of current behavior if to create cgroup in
parent cgroup of current process, but we did this in a
wrong way, we used devices cgroup path of current process
as the default parent path for all subsystems, this is
wrong because we don't always have the same cgroup path
for all subsystems.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Signed-off-by: rajasec <rajasec79@gmail.com>
Error handling when container not exists
Signed-off-by: rajasec <rajasec79@gmail.com>
Error handling when container not exists
Signed-off-by: rajasec <rajasec79@gmail.com>
Error handling when container not exists
Signed-off-by: rajasec <rajasec79@gmail.com>
This just moves everything to one function so we don't have to pass a
bunch of things to functions when there's no real benefit. It also makes
the API nicer.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
The original SETUID takes a 16 bit UID. Linux 2.4 introduced a new
syscall, SETUID32, with support for 32 bit UIDs. The setgid wrapper
already uses SETGID32.
Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>
TestCaptureTestFunc failed in localunittest:
# make localunittest
=== RUN TestCaptureTestFunc
--- FAIL: TestCaptureTestFunc (0.00s)
capture_test.go:26: expected package "github.com/opencontainers/runc/libcontainer/stacktrace" but received "_/root/runc/libcontainer/stacktrace"
#
Reason: the path for stacktrace is a fixed string which
only valid for container environment.
And we can switch to relative path to make both in-container
and out-of-container test works.
After patch:
# make localunittest
=== RUN TestCaptureTestFunc
--- PASS: TestCaptureTestFunc (0.00s)
#
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Removed a lot of clutter, improved the style of the code, removed
unnecessary complexity. In addition, made errors unique by making bail()
exit with a unique error code. Most of this code comes from the current
state of the rootless containers branch.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This device is not required by the OCI spec.
The rationale for this was linked to https://github.com/docker/docker/issues/2393
So a non functional /dev/fuse was created, and actual fuse use still is
required to add the device explicitly. However even old versions of the JVM
on Ubuntu 12.04 no longer require the fuse package, and this is all not
needed.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
It's possible that `cmd.Process` is still nil when we reach timeout.
Start creates `Process` field synchronously, and there is no way to such
race.
Signed-off-by: Alexander Morozov <lk4d4math@gmail.com>
This avoid the goimports tool from remove the libcontainer/keys import line due the package name is diferent from folder name
Signed-off-by: Guilherme Rezende <guilhermebr@gmail.com>
Revert: #935Fixes: #946
I can reproduce #946 on some machines, the problem is on
some machines, it could be very fast that modify time
of `memory.kmem.limit_in_bytes` could be the same as
before it's modified.
And now we'll call `SetKernelMemory` twice on container
creation which cause the second time failure.
Revert this before we find a better solution.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Previously we used the same JSON tag name for the regular and realtime
versions of the CpuRt* fields, which causes issues when you want to use
two different values for the fields.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Setting classid of net_cls cgroup failed:
ERRO[0000] process_linux.go:291: setting cgroup config for ready process caused "failed to write 𐀁 to net_cls.classid: write /sys/fs/cgroup/net_cls,net_prio/user.slice/abc/net_cls.classid: invalid argument"
process_linux.go:291: setting cgroup config for ready process caused "failed to write 𐀁 to net_cls.classid: write /sys/fs/cgroup/net_cls,net_prio/user.slice/abc/net_cls.classid: invalid argument"
The spec has classid as a *uint32, the libcontainer configs should match the type.
Signed-off-by: Hushan Jia <hushan.jia@gmail.com>
1. According to docs of Cmd.Path and Cmd.Args from package "os/exec":
Path is the path of the command to run. Args holds command line
arguments, including the command as Args[0]. We have mixed usage
of args. In InitPath(), InitArgs only take arguments, in InitArgs(),
InitArgs including the command as Args[0]. This is confusing.
2. InitArgs() already have the ability to configure a LinuxFactory
with the provided absolute path to the init binary and arguements as
InitPath() does.
3. exec.Command() will take care of serching executable path.
4. The default "/proc/self/exe" instead of os.Args[0] is passed to
InitArgs in order to allow relative path for the runC binary.
Signed-off-by: Yang Hongyang <imhy.yang@gmail.com>
Added a unit test to verify that 'cpu.rt_runtime_us' and 'cpu.rt_runtime_us'
cgroup values are set when the cgroup is applied to a process.
Signed-off-by: Ben Gray <ben.r.gray@gmail.com>
before trying to move the process into the cgroup.
This is required if runc itself is running in SCHED_RR mode, as it is not
possible to add a process in SCHED_RR mode to a cgroup which hasn't been
assigned any RT bandwidth. And RT bandwidth is not inherited, each new
cgroup starts with 0 b/w.
Signed-off-by: Ben Gray <ben.r.gray@gmail.com>
There's no point in changing directory here. Syscalls are resolved local
to the linkpath, not to the current directory that the process was in
when creating the symlink. Changing directories just confuses people who
are trying to debug things.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Comparisons with paths aren't really a good idea unless you're
guaranteed that the comparison will work will all paths that resolve to
the same lexical path as the compared path.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This removes the use of a signal handler and SIGCONT to signal the init
process to exec the users process.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Prior to this change a cgroup with a `:` character in it's path was not
parsed correctly (as occurs on some instances of systemd cgroups under
some versions of systemd, e.g. 225 with accounting).
This fixes that issue and adds a test.
Signed-off-by: Euan Kemp <euank@coreos.com>
This adds an `--no-new-keyring` flag to run and create so that a new
session keyring is not created for the container and the calling
processes keyring is inherited.
Fixes#818
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Delegate is only available in systemd >218, applying it for older systemd will
result in an error. Therefore we should check for it when testing systemd
properties.
Signed-off-by: Daniel, Dao Quang Minh <dqminh89@gmail.com>
Kernel memory cannot be set in these circumstances (before kernel 4.6):
1. kernel memory is not initialized, and there are tasks in cgroup
2. kernel memory is not initialized, and use_hierarchy is enabled,
and there are sub-cgroups
While we don't need to cover case 2 because when we set kernel
memory in runC, it's either:
- in Apply phase when we create the container, and in this case,
set kernel memory would definitely be valid;
- or in update operation, and in this case, there would be tasks
in cgroup, we only need to check if kernel memory is initialized
or not.
Even if we want to check use_hierarchy, we need to check sub-cgroups
as well, but for here, we can just leave it aside.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
In user namespaces, we need to make sure we don't chown() the console to
unmapped users. This means we need to get both the UID and GID of the
root user in the container when changing the owner.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Fix the following warnings when building runc with gcc 6+:
Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/nsenter/nsexec.c:
In function ‘nsexec’:
Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/nsenter/nsexec.c:322:6:
warning: ‘__s’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
pr_perror("Failed to open %s", ns);
Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/nsenter/nsexec.c:273:30:
note: ‘__s’ was declared here
static struct nsenter_config process_nl_attributes(int pipenum, char
*data, int data_size)
^~~~~~~~~~~~~~~~~~~~~
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
`libcontainer/cgroups/utils.go` uses an incorrect path to the
documentation for cgroups. This updates the comment to use the correct
URL. Fixes#794.
Signed-off-by: Jim Berlage <james.berlage@gmail.com>
See https://github.com/docker/docker/issues/22252
Previously we would apply seccomp rules before applying
capabilities, because it requires CAP_SYS_ADMIN. This
however means that a seccomp profile needs to allow
operations such as setcap() and setuid() which you
might reasonably want to disallow.
If prctl(PR_SET_NO_NEW_PRIVS) has been applied however
setting a seccomp filter is an unprivileged operation.
Therefore if this has been set, apply the seccomp
filter as late as possible, after capabilities have
been dropped and the uid set.
Note a small number of syscalls will take place
after the filter is applied, such as `futex`,
`stat` and `execve`, so these still need to be allowed
in addition to any the program itself needs.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Avoid parsing the whole lines of mountinfo after all mountpoints
of the target subsytems are found, or when the target subsystem
is not enabled.
Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>
Remove a wrongly added include which was added in commit 3c2e77ee (Add a
compatibility header for CentOS/RHEL 6, 2016-01-29) apparently to
fix this compile error on centos 6:
> In file included from
> Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/nsenter/nsexec.c:20:
> /usr/include/linux/netlink.h:35: error: expected specifier-qualifier-list before 'sa_family_t'
The glibc bits/sockaddr.h says that this header should never be included
directly[1]. Instead, sys/socket.h should be used.
The problem was correctly fixed later, in commit 394fb55 (Fix build
error on centos6, 2016-03-02) so the incorrect bits/sockaddr.h can
safely be removed.
This is needed to build musl libc.
Fixes#761
[1]: 20003c4988/bits/sockaddr.h (L20)
Signed-off-by: Natanael Copa <natanael.copa@docker.com>
This is the inital port of the libcontainer.Error to added a cause to
all the existing error messages. Going forward, when an error can be
wrapped because it is not being checked at the higher levels for
something like `os.IsNotExist` we can add more information to the error
message like cause and stack file/line information. This will help
higher level tools to know what cause a container start or operation to
fail.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
No substantial code change.
Note that some style errors reported by `golint` are not fixed due to possible compatibility issues.
Signed-off-by: Akihiro Suda <suda.kyoto@gmail.com>
When swap memory is unsupported, Docker will set
cgroup.Resources.MemorySwap as -1.
Fixes: https://github.com/docker/docker/pull/21937
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
setupDev was introduced in #96, but broken since #536 because spec 0.3.0 introduced default devices.
Fix#80 again
Fixdocker/docker#21808
Signed-off-by: Akihiro Suda <suda.kyoto@gmail.com>
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
One of our volume plugins needs to get the label of the target mount point
so that it can set the content inside of the volume to match.
We need label.GetFileLabel() to make this work.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
Some of the code was quite confusing inside libcontainer/user, so
refactor and comment it so future maintainers can understand what's
going and what edge cases we have to deal with.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Most shadow-related tools don't treat numeric ids as potential
usernames, so change our behaviour to match that. Previously, using an
explicit specification like 111:222 could result in the UID and GID not
being 111 and 222 respectively (which is confusing).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This adds a `--no-pivot` cli flag to runc so that a container's rootfs
can be located ontop of ramdisk/tmpfs and not fail because you cannot
pivot root.
This should be a cli flag and not part of the spec because this is a
detail of the host/runtime environment and not an attribute of a
container.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Currently, if we start a container with:
`docker run -ti --name foo --memory 300M --memory-swap 500M busybox sh`
Then we want to update it with:
`docker update --memory 600M --memory-swap 800M foo`
It'll get error because we can't set memory to 600M with
the 500M limit of swap memory.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Based on Golang document, %s is for "the uninterpreted bytes of the
string or slice", so %v is more appropriate.
Signed-off-by: Peng Gao <peng.gao.dut@gmail.com>
Users of libcontainer other than runc may also require parsing and
converting specification configuration files.
Since runc cannot be imported, move the relevant functions and
definitions to a separate package, libcontainer/specconv.
Signed-off-by: Ido Yariv <ido@wizery.com>
Fixes#680
This changes setupRlimit to use the Prlimit syscall (rather than
Setrlimit) and moves the call to the parent process. This is necessary
because Setrlimit would affect the libcontainer consumer if called in
the parent, and would fail if called from the child if the
child process is in a user namespace and the requested rlimit is higher
than that in the parent.
Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
Make sure we don't error out collecting statistics for cases where
pids.max == "max". In that case, we can use a limit of 0 which means
"unlimited".
In addition, change the name of the stats attribute (Max) to mirror the
name of the resources attribute in the spec (Limit) so that it's
consistent internally.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This is required because we manage some of the cgroups ourselves.
This recommendation came from talking with systemd devs about
some of the issues that we see when using the systemd cgroups driver.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
We need to make sure the container is destroyed before closing the stdio
for the container. This becomes a big issues when running in the host's
pid namespace because the other processes could have inherited the stdio
of the initial process. The call to close will just block as they still
have the io open.
Calling destroy before closing io, especially in the host pid namespace
will cause all additional processes to be killed in the container's
cgroup. This will allow the io to be closed successfuly.
This change makes sure the order for destroy and close is correct as
well as ensuring that if any errors encoutered during start or exec will
be handled by terminating the process and destroying the container. We
cannot use defers here because we need to enforce the correct ordering
on destroy.
This also sets the subreaper setting for runc so that when running in
pid host, runc can wait on the addiontal processes launched by the
container, useful on destroy, but also good for reaping the additional
processes that were launched.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This ensures that anything written to the logs are synced as they
happen.
This also changes the error message of the libcontainer error. The
original idea was to have this extra information in the message but it
makes it hard to parse and if the caller needed this information they
can just get it from the error type.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
In order to allow nice usage statistics (in terms of percentages and
other such data), add the value of pids.max to the PidsStats struct
returned from the pids cgroup controller.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Clears supplementary groups that have effect on the
mount permissions before joining the user specified
groups happens.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This updates runc and libcontainer to handle rlimits per process and set
them correctly for the container.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The rhel6 kernel returns EINVAL in this case
Known issue:
* CT with userns doesn't work
This is a copy of
d31e97fa28
to address https://github.com/opencontainers/runc/issues/613
Signed-off-by: Andrey Vagin <avagin@virtuozzo.com>
Signed-off-by: Andrew Fernandes <andrew@fernandes.org>
Now that all the user namespace code is moved into C, these routines are
no longer used.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
The re-work of namespace entering lost the setuid/setgid that was part
of the Go-routine based process exec in the prior code. A side issue was
found with setting oom_score_adj before execve() in a userns that is
also solved here.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
This commit adds support to libcontainer to allow caps, no new privs,
apparmor, and selinux process label to the process struct so that it can
be used together of override the base settings on the container config
per individual process.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This is needed to make 'runc delete' correctly run the post-stop hooks.
Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
Signed-off-by: Ed King <eking@pivotal.io>
currentState() always adds all possible namespaces to the state,
regardless of whether they are supported.
If orderNamespacePaths detects an unsupported namespace, an error is
returned that results in initialization failure.
Fix this by only adding paths of supported namespaces to the state.
Signed-off-by: Ido Yariv <ido@wizery.com>
The path in the stacktrace might not be:
"github.com/opencontainers/runc/libcontainer/stacktrace"
For example, for me its:
"_/go/src/github.com/opencontainers/runc/libcontainer/stacktrace"
so I changed the check to make sure the tail end of the path matches instead
of the entire thing
Signed-off-by: Doug Davis <dug@us.ibm.com>
This simply move the call to the Prestart hooks to be made once we
receive the procReady message from the client.
This is necessary as we had to move the setns calls within nsexec in
order to be accomodate joining namespaces that only affect future
children (e.g. NEWPID).
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
An init process can join other namespaces (pidns, ipc etc.). This leverages
C code defined in nsenter package to spawn a process with correct namespaces
and clone if necessary.
This moves all setns and cloneflags related code to nsenter layer, which mean
that we dont use Go os/exec to create process with cloneflags and set
uid/gid_map or setgroups anymore. The necessary data is passed from Go to C
using a netlink binary-encoding format.
With this change, setns and init processes are almost the same, which brings
some opportunity for refactoring.
Signed-off-by: Daniel, Dao Quang Minh <dqminh89@gmail.com>
[mickael.laventure@docker.com: adapted to apply on master @ d97d5e]
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@docker.com>
This adds orderNamespacePaths to get correct order of namespaces for the
bootstrap program to join.
Signed-off-by: Daniel, Dao Quang Minh <dqminh89@gmail.com>
This adds `configs.IsNamespaceSupported(nsType)` to check if the host supports
a namespace type.
Signed-off-by: Daniel, Dao Quang Minh <dqminh89@gmail.com>
This allows the mount syscall to validate the addiontal types where we
do not have to perform extra validation and is up to the consumer to
verify the functionality of the type of device they are trying to
mount.
Fixes#572
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Exec erros from the exec() syscall in the container's init should be
treated as if the container ran but couldn't execute the process for the
user instead of returning a libcontainer error as if it was an issue in
the library.
Before specifying different commands like `/etc`, `asldfkjasdlfj`, or
`/alsdjfkasdlfj` would always return 1 on the command line with a
libcontainer specific error message. Now they return the correct
message and exit status defined for unix processes.
Example:
```bash
root@deathstar:/containers/redis# runc start test
exec: "/asdlfkjasldkfj": file does not exist
root@deathstar:/containers/redis# echo $?
127
root@deathstar:/containers/redis# runc start test
exec: "asdlfkjasldkfj": executable file not found in $PATH
root@deathstar:/containers/redis# echo $?
127
root@deathstar:/containers/redis# runc start test
exec: "/etc": permission denied
root@deathstar:/containers/redis# echo $?
126
```
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Signed-off-by: rajasec <rajasec79@gmail.com>
Changing to name values for defer as per review comments
Signed-off-by: rajasec <rajasec79@gmail.com>
Fixed review comments
Signed-off-by: rajasec <rajasec79@gmail.com>
This prior fix to set "-1" explicitly was lost, and it is simpler to use
the same pointer type from the OCI spec to handle nil pointer == -1 ==
unset case.
Also, as a nearly humorous aside, there was a test for MemorySwappiness
that was actually setting Memory, and it was passing because of this
bug (as it was always setting everyone's MemorySwappiness to zero!)
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
Create a unique session key name for every container. Use the pattern
_ses.<postfix> with postfix being the container's Id.
This patch does not prevent containers from joining each other's session
keyring.
Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
Because we more than likely control dev and populate devices and files
inside of it we need to make sure that we fulfil the user's request to
make it ro only after it has been populated. This removes the need to
expose something like ReadonlyPaths in the config but still have the
same outcome but more seemless for the user.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Today mounts in pre-start hooks get overriden by the default mounts.
Moving the pre-start hooks to after the container mounts and before
the pivot/move root gives better flexiblity in the hooks.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
If you don't move the process out of the named cgroup for systemd then
systemd will try to delete all the cgroups that the process is currently
in.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Docker uses Prestart hooks to call a libnetwork hook to create
network devices and set addesses and routes.
Signed-off-by: Andrew Vagin <avagin@virtuozzo.com>
This options is set a namespace mask which will not be dumped and restored.
For example, we are going to use this option to restore network
for docker containers. CRIU will create a network namespace and
call a libnetwork hook to restore network devices, addresses and routes.
Signed-off-by: Andrew Vagin <avagin@virtuozzo.com>
When in a non-initial user namespace you cannot update the devices
cgroup whitelist (or blacklist). The kernel won't allow it. So
detect that case and don't try.
This is a step to being able to run docker/runc containers inside a user
namespaced container.
Signed-off-by: Serge Hallyn <serge.hallyn@ubuntu.com>
It's handled in `destroy()`, no need to do this in
`Apply()`. I found this because systemd cgroup didn't
do this removal and it works well.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
In order to avoid problems with security regressions going unnoticed,
add some unit tests that should make sure security regressions in cgroup
path safety cause tests to fail in runC.
Signed-off-by: Aleksa Sarai <asarai@suse.com>
Ensure that path safety is maintained, this essentially reapplies
c0cad6aa5e ("cgroups: fs: fix cgroup.Parent path sanitisation"), which
was accidentally removed in 256f3a8ebc ("Add support for CgroupsPath
field").
Signed-off-by: Aleksa Sarai <asarai@suse.com>
Because we are implemented in Go, the number of pids present in a
container is not very well-defined (other than it not being /much/
bigger than the limit you'd want to set). As a result, we need to make
the tests a bit less flaky in this regard.
Signed-off-by: Aleksa Sarai <asarai@suse.com>
Create a new session key ring '_ses' for every container. This avoids sharing
the key structure with the process that created the container and the
container inherits from.
This patch fixes it init and exec.
Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
GetMounts is very cpu-expensive. I'll change other funcs in this package
to reuse code from GetCgroupMounts later.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
It's never used and not needed. Our pipe is created with
syscall.SOCK_CLOEXEC, so pipe will be closed once container
process executed successfully, parent process will read EOF
and continue. If container process got error before executed,
we'll write procError to sync with parent.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
By adding detach to runc the container process is the only thing running
on the system is the containers process.
This allows better usage of memeory and no runc process being long
lived. With this addition you also need a delete command because the
detached container will not be able to remove state and the left over
cgroups directories.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
We don't need a CreatedTime method on the container because it's not
part of the interface and can be received via the state. We also do not
need to call it CreateTime because the type of this field is time.Time
so we know its time.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add some further (not critical, since Docker does this already)
validation to systemd slice names, to make sure users don't get cryptic
errors.
Signed-off-by: Aleksa Sarai <asarai@suse.com>
Marshall the raw objects for the sync pipes so that no new line chars
are left behind in the pipe causing errors.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Rather than using '/' to denote hierarchy in slice names, systemd uses
'-' in an odd way. This results in runC incorrectly assuming that
certain kernel features are missing (and using inconsistent paths for
the cgroups not supported by systemd), because the "subsystem path" used
is not the one that systemd has created. Fix all of this by properly
expanding slice names.
Signed-off-by: Aleksa Sarai <asarai@suse.com>
There were issues where a process could die before pausing completed
leaving the container in an inconsistent state and unable to be
destoryed. This makes sure that if the container is paused and the
process is dead it will unfreeze the cgroup before removing them.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Modify the memory cgroup code such that kmem is not managed by Set(), in
order to allow updating of memory constraints for containers by Docker.
This also removes the need to make memory a special case cgroup.
Signed-off-by: Aleksa Sarai <asarai@suse.com>
For existing consumers of libconatiner to not require cwd inside the
libcontainer code. This can be done at the runc level and is already
evaluated there.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Set `-1` doesn't mean disable swap, disable swap means you
can't use swap memory, set `-1` really means you can use
unlimited swap memory.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
Mohammad Arab
Carlton Semple
Deng Guangxing
Ma Shimiao
Qiang Huang
Antonio Murdaca
Steven Hartland
Michael Crosby
Mrunal Patel
Aleksa Sarai
Daniel, Dao Quang Minh
Justin Cormack
rainrambler
Wang Long
Mrunal Patel
Lei Jitang
Zhang Wei
Samuel Ortiz
Eric Paris
yupeng
Yuanhong Peng
Xianlu Bird
allencloud
rajasec
Crazykev
Andrei Vagin
Vivek Goyal
Ce Gao
Alexander Morozov
Yong Tang
Derek Carr
Dan Walsh
Shukui Yang
Xianglin Gao
William Martin
Peng Gao
Akihiro Suda
Adam Thomason
xiekeyang
Phil Estes
Carl Henrik Lunde
Zhao Lei
Serge Hallyn
Aleksa Sarai
Haiyan Meng
Buddha Prakash
Seth Jennings
Guilherme Rezende
Haiyan Meng
Yen-Lin Chen
Hushan Jia
Yang Hongyang
Vishnu kannan
Ben
ben
Petar Petrov
Euan Kemp
Christian Brauner
Joe Farrell
Michael Holzheu
Bhanu Valasa
Kenfe-Mickael Laventure
Jim Berlage
Bryan Boreham
Tatsushi Inagaki
Natanael Copa
Amit Krishnan
Alberto Leal
Akihiro Suda
George Lestaris
Marcos Lilljedahl
Matt Hartzler
Ido Yariv
Julian Friedman
Thomas Tanaka
Tonis Tiigi
pankit thapar
Jessica Frazelle
Thomas LE ROUX
Ed King
Ye Yin
Alexander Morozov
Doug Davis
Stefan Berger
Serge Hallyn
Chun Chen
Aleksa Sarai
Mike Brown
Stefan Berger
Andrew Fernandes