Description of problem:

Using -v to bind mount a file into a container does not appear to modify the behavior of lstat or readlink if the target path on the container already exists and is a symbolic link.

This causes programs that use these system calls to continue seeing the file present in the container instead of the file on the host system.

In our case this caused Java processes to incorrectly resolve the timezone. An example of this behavior is provided below.

I have not tested whether this behavior also occurs for directory bind mounts.

docker version:

Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64

docker info:

Containers: 15
Images: 14
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 44
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-41-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 15.56 GiB
WARNING: No swap limit support

uname -a:

Linux redacted 3.16.0-41-generic #55~14.04.1-Ubuntu SMP Sun Jun 14 18:43:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

HP laptop running Ubuntu 14.04 natively.

How reproducible:

We noticed this on our Java app server containers, which derive from the official CentOS7 base image. Recently that image was updated so that /etc/localtime sym links to ../usr/share/zoneinfo/UTC

Our docker run commands use -v /etc/localtime:/etc/localtime to ensure that containers inherit the timezone of the VM they run on. We noticed this failed to work with the latest CentOS 7 image. After digging a bit, it appears that the JVM uses lstat and readlink to resolve /etc/localtime and this leads to unexpected results when combined with -v

Steps to Reproduce:

I have uploaded an image with Java 8 and a small Java program which demonstrates the issue. This issue shouldn't be specific to Java, but it was the simplest way for me to demonstrate the precise symptom.

1. docker pull coopernurse/centos7-java
2. set timezone on host to something other than UTC
3. docker run -v /etc/localtime:/etc/localtime coopernurse/centos7-java /bin/bash -c '/bin/date && java Hello'

Actual Results:

Fri Jun 26 16:05:22 PDT 2015
Fri Jun 26 23:05:22 UTC 2015

Expected Results:

Fri Jun 26 16:05:22 PDT 2015
Fri Jun 26 16:05:22 PDT 2015

Additional info:

strace shows that /bin/date is simply calling open('/etc/localtime') whereas the JVM is calling lstat and then readlink

readlink still reports the original sym link, despite the -v bind mount:

$ docker run -v /etc/localtime:/etc/localtime coopernurse/centos7-java readlink /etc/localtime
../usr/share/zoneinfo/UTC

More strace output:

docker run -it --privileged coopernurse/centos7-java

[root@7befc66fc860 /]# strace -f java Hello 2>&1 | grep time
[pid    78] futex(0x7f1918009354, FUTEX_WAIT_BITSET_PRIVATE, 1, {5920, 570927155}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid    78] futex(0x7f1918009354, FUTEX_WAIT_BITSET_PRIVATE, 1, {5920, 572068364}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid    78] futex(0x7f1918009354, FUTEX_WAIT_BITSET_PRIVATE, 1, {5920, 573228882}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid    78] read(3, "\312\376\272\276\0\0\0004\0\v\1\0\3()V\1\0\31RuntimeVisibl"..., 201) = 201
[pid    78] open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
[pid    78] open("/etc/timezone", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid    78] lstat("/etc/localtime", {st_mode=S_IFLNK|0777, st_size=25, ...}) = 0
[pid    78] readlink("/etc/localtime", "../usr/share/zoneinfo/UTC", 4096) = 25
[pid    87] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)

vs

[root@7befc66fc860 /]# strace -f /bin/date 2>&1 | grep time 
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3

Hello.java source:

[root@e868f7d70484 /]# cat Hello.java
import java.util.Date;

public class Hello {

    public static void main(String argv[]) {
        System.out.println(new Date());
    }

}