New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Amazon SSM agent is using all of my i-nodes #94
Comments
We are investigating the issue. |
Hi @andrewme2, I am also experiencing this issue. As a workaround, I am using this cron job in root's crontab:
This will be executed every day at midnight and delete all orchestration documents older than 3 days. |
Thanks @vjt ! That's a good workaround until this bug is fixed. |
has this been resolved in a release? |
@wael-amz Is there a workaround or updates on this issue? |
The issue has been addressed in the latest release 2.2.800.0 |
Hi @wael-amz , I'm trying to verify that this bug is fixed. I upgraded to 2.2.800.0. How long do I need to wait before the SSM agent will clean up old files? I do not know how often the agent runs its clean-up routine (every hour, once-per-day, etc?). After running for ~2 hours, I still have directories in the orchestration directory from May: root@ip-X-X-X-X:~# ls -ltr /var/lib/amazon/ssm//document/orchestration | head |
The cleanup logic will be triggered every time you run a command and it is not time based. |
The SSM agent (v2.2.93 and v2.2.120) does not seem to be cleaning up files in the orchestration directory. The issue is reproducible on many EC2 instances that I have.
I have several EC2 instances where i-node usage is 90%+ and one where it is 100%. Once 100% is reached, the system effectively locks up and the ssm log files show many "no space left on device" errors.
I tracked the issue down to the SSM agent not cleaning up files in this directory:
For example, one of my EC2 instances has files in the orchestration directory going back to Dec. 5.
A similar issue was fixed in 2.2.24.0:
The text was updated successfully, but these errors were encountered: