"docker exec" hangs for 1~3 minutes and wrongly repeats execution #34978
Comments
|
This happens on many containers on that host, not only to Weave Scope's containers. |
Dieken
changed the title
|
I'm not able to reproduce this on a current version of docker, but do not have the same load on that system. Are you only seeing this on that particular host, or also on other hosts? Anything useful in the daemon or system-logs? Given that docker 1.12.6 reached end of life, are you able to reproduce on a current version of docker as well? |
|
@thaJeztah I can't reliably reproduce the issue, I just restarted the docker service, the issue has gone. I have four kubernetes worker node, this happens on one of heavily load node. Because Kubernetes doesn't officially recommend Docker > 1.12, I don't plan to upgrade Docker very soon. During the debugging, I found docker-containerd frequently quit due to SIGPIPE. Pid 20178 is docker-containerd. |
|
@mlaventure any suggestions? |
|
I also straced the docker client, it did receive multiple lines in a single read(), so it's not docker client's fault. Maybe dockerd resent the command upon crashes of containerd. I have no idea why SIGPIPE was triggered. During the debugging, dockerd didn't crash. |
|
This was fixed later in containerd: containerd/containerd#930 hence why it's not showing up on new version of Docker. |
Could you share the progress about the "proper fix"? |
|
@Dieken sorry, I lack context and couldn't find that particular comment. But, if my memory serves me right, this was a comment made by someone not working for Docker, so you would have to ask them what they meants. |
|
@mlaventure Sorry, it's my fault, the comment comes from containerd/containerd#930, it was made by an employee from Redhat. @runcom, could you share the progress about the proper fix mentioned in the pull request? |
|
closing as this one went stale |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Steps to reproduce the issue:
The load of host OS is relatively high, I can easily reproduce it with the command in the snapshot.
Describe the results you received:
"docker exec xxx" hangs a while, then repeats "uptime" several times.
Describe the results you expected:
I expect "docker exec" quickly executes the command and only once.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
The text was updated successfully, but these errors were encountered: