With the help of userfaultfd CRIU supports lazy migration. Lazy
migration means that memory pages are only transferred from the
migration source to the migration destination on page fault.
This enables to reduce the downtime during process or container
migration to a minimum as the memory does not need to be transferred
during migration.
Lazy migration currently depends on userfaultfd being available on the
current Linux kernel and if the used CRIU version supports lazy
migration. Both dependencies can be checked by querying CRIU via RPC if
the lazy migration feature is available. Using feature checking instead
of version comparison enables runC to use CRIU features from the
criu-dev branch. This way the user can decide if lazy migration should
be available by choosing the right kernel and CRIU branch.
To use lazy migration the CRIU process during dump needs to dump
everything besides the memory pages and then it opens a network port
waiting for remote page fault requests:
# runc checkpoint httpd --lazy-pages --page-server 0.0.0.0:27 \
--status-fd /tmp/postcopy-pipe
In this example CRIU will hang/wait once it has opened the network port
and wait for network connection. As runC waits for CRIU to finish it
will also hang until the lazy migration has finished. To know when the
restore on the destination side can start the '--status-fd' parameter is
used:
#️ runc checkpoint --help | grep status
--status-fd value criu writes \0 to this FD once lazy-pages is ready
The parameter '--status-fd' is directly from CRIU and this way the
process outside of runC which controls the migration knows exactly when
to transfer the checkpoint (without memory pages) to the destination and
that the restore can be started.
On the destination side it is necessary to start CRIU in 'lazy-pages'
mode like this:
# criu lazy-pages --page-server --address 192.168.122.3 --port 27 \
-D checkpoint
and tell runC to do a lazy restore:
# runc restore -d --image-path checkpoint --work-path checkpoint \
--lazy-pages httpd
If both processes on the restore side have the same working directory
'criu lazy-pages' creates a unix domain socket where it waits for
requests from the actual restore. runC starts CRIU restore in lazy
restore mode and talks to 'criu lazy-pages' that it wants to restore
memory pages on demand. CRIU continues to restore the process and once
the process is running and accesses the first non-existing memory page
the 'criu lazy-pages' server will request the page from the source
system. Thus all pages from the source system will be transferred to the
destination system. Once all pages have been transferred runC on the
source system will end and the container will have finished migration.
This can also be combined with CRIU's pre-copy support. The combination
of pre-copy and post-copy (lazy migration) provides the possibility to
migrate containers with minimal downtimes.
Some additional background about post-copy migration can be found in
these articles:
https://lisas.de/~adrian/?p=1253https://lisas.de/~adrian/?p=1183
Signed-off-by: Adrian Reber <areber@redhat.com>
When doing incremental dumps is useful to use auto deduplication of
memory images to save space.
Signed-off-by: Nikolas Sepos <nikolas.sepos@gmail.com>
Currently startContainer() is used to create and to run a container.
In the next patch it will be used to restore a container.
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
This enables the support for the rootless container mode. There are many
restrictions on what rootless containers can do, so many different runC
commands have been disabled:
* runc checkpoint
* runc events
* runc pause
* runc ps
* runc restore
* runc resume
* runc update
The following commands work:
* runc create
* runc delete
* runc exec
* runc kill
* runc list
* runc run
* runc spec
* runc state
In addition, any specification options that imply joining cgroups have
also been disabled. This is due to support for unprivileged subtree
management not being available from Linux upstream.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
The current support of systemd-notify has a race condition as the
message send to the systemd notify socket might be dropped if the sender
process is not running by the time systemd checks for the sender of the
datagram. A proper fix of this in systemd would require changes to the
kernel to maintain the cgroup of the sender process when it is dead (but
it is not probably going to happen...)
Generally, the solution to this issue is to specify the PID in the
message itself so that systemd has not to guess the sender, but this
wouldn't work when running in a PID namespace as the container will pass
the PID known in its namespace (something like PID=1,2,3..) and systemd
running on the host is not able to map it to the runc service.
The proposed solution is to have a proxy in runc that forwards the
messages to the host systemd.
Example of this issue:
https://github.com/projectatomic/atomic-system-containers/pull/24
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
CRIU gets pre-dump to complete iterative migration.
pre-dump saves process memory info only. And it need parent-path
to specify the former memory files.
This patch add pre-dump and parent-path arguments to runc checkpoint
Signed-off-by: Deng Guangxing <dengguangxing@huawei.com>
Signed-off-by: Adrian Reber <areber@redhat.com>
This implements {createTTY, detach} and all of the combinations and
negations of the two that were previously implemented. There are some
valid questions about out-of-OCI-scope topics like !createTTY and how
things should be handled (why do we dup the current stdio to the
process, and how is that not a security issue). However, these will be
dealt with in a separate patchset.
In order to allow for late console setup, split setupRootfs into the
"preparation" section where all of the mounts are created and the
"finalize" section where we pivot_root and set things as ro. In between
the two we can set up all of the console mountpoints and symlinks we
need.
We use two-stage synchronisation to ensures that when the syscalls are
reordered in a suboptimal way, an out-of-place read() on the parentPipe
will not gobble the ancilliary information.
This patch is part of the console rewrite patchset.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
For example:
./runc restore --empty-ns network CTID
In this case criu creates a network namespace, but doesn't restore it.
We are going to use this option to restore docker containers and
Docker sets a hook to restore a network namespace.
https://github.com/xemul/criu/issues/165
Signed-off-by: Andrew Vagin <avagin@virtuozzo.com>
The major change is the description of options, change
it as the latest cli help message shows, which specify
a "value" after an option if it takes value, and add
(default: xxx) if the option has a default value.
This also includes some other minor consistency fixes.
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
In user namespaces, we need to make sure we don't chown() the console to
unmapped users. This means we need to get both the UID and GID of the
root user in the container when changing the owner.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This adds a `--no-pivot` cli flag to runc so that a container's rootfs
can be located ontop of ramdisk/tmpfs and not fail because you cannot
pivot root.
This should be a cli flag and not part of the spec because this is a
detail of the host/runtime environment and not an attribute of a
container.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Users of libcontainer other than runc may also require parsing and
converting specification configuration files.
Since runc cannot be imported, move the relevant functions and
definitions to a separate package, libcontainer/specconv.
Signed-off-by: Ido Yariv <ido@wizery.com>
We need to make sure the container is destroyed before closing the stdio
for the container. This becomes a big issues when running in the host's
pid namespace because the other processes could have inherited the stdio
of the initial process. The call to close will just block as they still
have the io open.
Calling destroy before closing io, especially in the host pid namespace
will cause all additional processes to be killed in the container's
cgroup. This will allow the io to be closed successfuly.
This change makes sure the order for destroy and close is correct as
well as ensuring that if any errors encoutered during start or exec will
be handled by terminating the process and destroying the container. We
cannot use defers here because we need to enforce the correct ordering
on destroy.
This also sets the subreaper setting for runc so that when running in
pid host, runc can wait on the addiontal processes launched by the
container, useful on destroy, but also good for reaping the additional
processes that were launched.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The error handling on the runc cli is currenly pretty messy because
messages to the user are split between regular stderr format and logrus
message format. This changes all the error reporting to the cli to only
output on stderr and exit(1) for consumers of the api.
By default logrus logs to /dev/null so that it is not seen by the user.
If the user wants extra and/or structured loggging/errors from runc they
can use the `--log` flag to provide a path to the file where they want
this information. This allows a consistent behavior on the cli but
extra power and information when debugging with logs.
This also includes a change to enable the same logging information
inside the container's init by adding an init cli command that can share
the existing flags for all other runc commands.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add a waitgroup to wait for the io.Copy of stdout/err to finish before
existing runc. The problem happens more in exec because it is really
fast and the pipe has data buffered but not yet read after the process
has already exited.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
newSignalHandler needs to be called before the process is started, otherwise when
the process exits quickly the SIGCHLD is recieved (and ignored) before the
handler is set up. When this happens the reaper never runs, the
process becomes a zombie, and the exit code isn't returned to the user.
Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
Closes#532
This requires the container id to always be passed to all runc commands
as arg one on the cli. This was the result of the last OCI meeting and
how operations work with the spec.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This flag allows systems that are running runc to allocate tty's that
they own and provide to the container.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add state status() method
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Allow multiple checkpoint on restore
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Handle leave-running state
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Fix state transitions for inprocess
Because the tests use libcontainer in process between the various states
we need to ensure that that usecase works as well as the out of process
one.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Remove isDestroyed method
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Handling Pausing from freezer state
Signed-off-by: Rajasekaran <rajasec79@gmail.com>
freezer status
Signed-off-by: Rajasekaran <rajasec79@gmail.com>
Fixing review comments
Signed-off-by: Rajasekaran <rajasec79@gmail.com>
Added comment when freezer not available
Signed-off-by: Rajasekaran <rajasec79@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Conflicts:
libcontainer/container_linux.go
Change checkFreezer logic to isPaused()
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Remove state base and factor out destroy func
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add unit test for state transitions
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
I deleted possibility to specify config file from commands for now.
Until we decide how it'll be done. Also I changed runc spec interface to
write config files instead of output them.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Without this, multiple runc containers can accidentally share the same cgroup(s)
(and change each other's limits), when runc is invoked from the same directory
(i.e.: same cwd on multiple runc executions).
After these changes, each runc container will run on its own cgroup(s). Before,
the only workaround was to invoke runc from an unique (temporary?) cwd for each
container.
Common cgroup configuration (and hierarchical limits) can be set by having
multiple runc containers share the same cgroup parent, which is the cgroup of
the process executing runc.
Signed-off-by: Fabio Kung <fabio.kung@gmail.com>
There was previously a memory issue in the signal handler that showed up
when using restore. This has been fixed, therefore, restore can use the
signal handler.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Adrian Reber
Nikolas Sepos
Andrei Vagin
Tim Potter
Aleksa Sarai
Michael Crosby
Mrunal Patel
Giuseppe Scrivano
Deng Guangxing
Mrunal Patel
Zhang Wei
xiekeyang
Qiang Huang
Ido Yariv
Mike Brown
Julian Friedman
rajasec
Hui Kang
Alexander Morozov
Alexander Morozov
Fabio Kung