Depleting fds security issue
This was reported on ntirpc at Github. After DanG's fix for transport reference counting, Gaurav Gangalwar wrote that s/he can deplete the available fds with: for i in {1..1000}; do echo $i; mount -t nfs -o vers=3,tcp :/export /mnt; umount /mnt; done There is a configurable parameter limiting the number of concurrent fds. They are cleaned up after a fairly long inactivity timeout. So an adversary (or runaway client) can quickly deplete the fds, preventing valid clients from mounting. The most obvious patch would be for umount to add the same SVC_DESTROY line that DanG (and previously me) had added for the TCP well known ports in dispatch. But that would allow runaway clients to flood the server. Preventing that seems to be the (undocumented) purpose of the configurable parameter. The parameter and cleanup code are relatively old, and added to ntirpc circa 2013. Was it a bad idea? Which security issue is more important?
Thanks for raising it on Ganesha list. Just want to add, once fds are depleted, they are not getting cleaned up even after hours. I needed to restart Ganesha process to recover from this state. I am not sure if idle cleanup code is able to get rid of extra ref and release xprt. I tried with this fix in cleanup code its working for me. https://paste.fedoraproject.org/paste/lapz1NrOlBxS342S4Q79aw Regards, Gaurav On Sun, Dec 2, 2018 at 4:49 AM William Allen Simpson < william.allen.simpson@gmail.com> wrote:
This was reported on ntirpc at Github. After DanG's fix for transport reference counting, Gaurav Gangalwar wrote that s/he can deplete the available fds with:
for i in {1..1000}; do echo $i; mount -t nfs -o vers=3,tcp :/export /mnt; umount /mnt; done
There is a configurable parameter limiting the number of concurrent fds. They are cleaned up after a fairly long inactivity timeout. So an adversary (or runaway client) can quickly deplete the fds, preventing valid clients from mounting.
The most obvious patch would be for umount to add the same SVC_DESTROY line that DanG (and previously me) had added for the TCP well known ports in dispatch.
But that would allow runaway clients to flood the server. Preventing that seems to be the (undocumented) purpose of the configurable parameter.
The parameter and cleanup code are relatively old, and added to ntirpc circa 2013. Was it a bad idea?
Which security issue is more important? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
On 12/4/18 12:27 AM, gaurav gangalwar wrote:
Thanks for raising it on Ganesha list. Just want to add, once fds are depleted, they are not getting cleaned up even after hours. I needed to restart Ganesha process to recover from this state.
How many hours did you wait? Currently, the cleanup is triggered after 1023 epoll wakeups. If you stop doing anything and simply wait, that will be 8.24 hours (1023 * 29 seconds). Before 2.6, that was 120 seconds for 34 hours. After all, there's no good reason to cleanup with no activity. Also, cleanup means fd has no recv activity in __svc_params->idle_timeout. Ganesha default is nfs_param.core_param.rpc.idle_timeout_s = 300 seconds. If you're using a standard client doing idles, it will never cleanup.
I am not sure if idle cleanup code is able to get rid of extra ref and release xprt.
There is no extra ref. We had one too few refs. That's what DanG fixed.
I tried with this fix in cleanup code its working for me. https://paste.fedoraproject.org/paste/lapz1NrOlBxS342S4Q79aw
As I tried to explain where s/he reported it, that isn't a fix. It releases some other reference still remaining, which in turn will probably lead to referencing freed memory. All this does is force a close(fd), which allows more connections reusing the old fd. Masking the symptoms. The real underlying security issue is that a malicious adversary can run a resource exhaustion attack. That's what needs to be fixed.
I waited for more that 14 hours. This is the script I ran. for i in {1..300}; do echo $i; mount -t nfs -o vers=3,tcp 10.15.196.92:/export /mnt1; umount /mnt1; done It went out of ads at 169th iteration. 169 mount.nfs: Connection timed out umount: /mnt1: not mounted After this I was not able to mount or do any operation. There was no active mount on my client. So there was no activity from client side, so does it mean it will never cleanup? Regards, Gaurav On Tue, Dec 4, 2018 at 7:44 PM William Allen Simpson < william.allen.simpson@gmail.com> wrote:
On 12/4/18 12:27 AM, gaurav gangalwar wrote:
Thanks for raising it on Ganesha list. Just want to add, once fds are depleted, they are not getting cleaned up even after hours. I needed to restart Ganesha process to recover from this state.
How many hours did you wait?
Currently, the cleanup is triggered after 1023 epoll wakeups.
If you stop doing anything and simply wait, that will be 8.24 hours (1023 * 29 seconds).
Before 2.6, that was 120 seconds for 34 hours.
After all, there's no good reason to cleanup with no activity.
Also, cleanup means fd has no recv activity in __svc_params->idle_timeout. Ganesha default is nfs_param.core_param.rpc.idle_timeout_s = 300 seconds.
If you're using a standard client doing idles, it will never cleanup.
I am not sure if idle cleanup code is able to get rid of extra ref and release xprt.
There is no extra ref. We had one too few refs. That's what DanG fixed.
I tried with this fix in cleanup code its working for me. https://paste.fedoraproject.org/paste/lapz1NrOlBxS342S4Q79aw
As I tried to explain where s/he reported it, that isn't a fix. It releases some other reference still remaining, which in turn will probably lead to referencing freed memory.
All this does is force a close(fd), which allows more connections reusing the old fd. Masking the symptoms.
The real underlying security issue is that a malicious adversary can run a resource exhaustion attack. That's what needs to be fixed.
Wow, I really hate top posts. Very hard to comprehend your responses. On 12/5/18 12:28 AM, gaurav gangalwar wrote:
I waited for more that 14 hours. This is the script I ran. for i in {1..300}; do echo $i; mount -t nfs -o vers=3,tcp 10.15.196.92:/export /mnt1; umount /mnt1; done It went out of ads at 169th iteration. 169 mount.nfs: Connection timed out umount: /mnt1: not mounted
I really don't understand. There are no constants in the current V2.8-dev code base that add up to more than 14 hours, and no constant that would yield 169. In any case, this particular denial of service attack should be easy to fix by adding SVC_DESTROY to the Ganesha umount code path(s). However, because it seems rather unlikely this could happen in production, it's not currently a high priority. Do you have a production environment or customer report where this occurred? Would you like help writing the Ganesha patch? Are you with Nutanix? Or IBM?
After this I was not able to mount or do any operation. There was no active mount on my client. So there was no activity from client side, so does it mean it will never cleanup?
No, as I wrote, # If you stop doing anything and simply wait, that will be 8.24 hours # (1023 * 29 seconds). I've discussed this with DanG, and that ntirpc design logic from before my time seems faulty. It does cleanup more frequently on a busy system, but is very slow on an idle system (the most likely time to have an fd become idle). DanG suggested that a better approach would be to trigger cleanup based upon a high-water-mark. The best approach would be to order the list of fds by latest recv, and clean out the idle ones as soon as they expire -- instead of running through the list of (tens of) thousands of fds. That shouldn't be too hard, but we must be careful that we don't add new locking in the fast path. But this is an own-time project for me, and my primary interest is RDMA and performance. So it may be some time before I've a chance to look at it. Maybe over the holiday....
On Thu, Dec 6, 2018 at 1:41 AM William Allen Simpson < william.allen.simpson@gmail.com> wrote:
In any case, this particular denial of service attack should be easy to fix by adding SVC_DESTROY to the Ganesha umount code path(s).
Should I do SVC_DESTROY or SVC_RELEASE in unmount? Also I am not sure if we will hit this issue if client mount and reboot without unmount. I will test that, will unmount fix work in that case?
However, because it seems rather unlikely this could happen in production, it's not currently a high priority.
Do you have a production environment or customer report where this occurred?
No its not customer issue, I faced it after taking extra ref fix https://github.com/nfs-ganesha/ntirpc/pull/155. But as I mentioned earlier if it happens in client reboot case also then its likely to happen in production if there is client which is crashing and coming up. I will check it.
Would you like help writing the Ganesha patch?
I would surely like to help with the fix.
Are you with Nutanix? Or IBM?
I am with Nutanix. -Gaurav
On 12/6/18 4:42 AM, gaurav gangalwar wrote:
On Thu, Dec 6, 2018 at 1:41 AM William Allen Simpson <william.allen.simpson@gmail.com <mailto:william.allen.simpson@gmail.com>> wrote:
In any case, this particular denial of service attack should be easy to fix by adding SVC_DESTROY to the Ganesha umount code path(s).
Should I do SVC_DESTROY or SVC_RELEASE in unmount?
SVC_DESTROY, as it sets the atomic bit telling other parts of the code to stop making more references. It then calls SVC_RELEASE code path. SVC_RELEASE alone sets a different bit after the count goes to zero. In a perfect world, this bit wouldn't have been needed, but we've had years of problems with the refs bouncing up again. Maybe we've finally fixed them all, but that bit doesn't hurt anything.
Also I am not sure if we will hit this issue if client mount and reboot without unmount. I will test that, will unmount fix work in that case?
Possibly not, but that should be a different patch. Keep the patch focused, fixing one problem that we all can then test has been fixed.
Do you have a production environment or customer report where this occurred?
No its not customer issue,
[...]
I am with Nutanix. -Gaurav
Knowing that this will be needed in a distribution will help the argument for backporting to earlier releases. Thanks.
As the function naming (which partly predates ntirpc fork) and comments clearly imply, the semantics of svc_clean_idle is to do periodic processing of idle, not closed, connections. At some point in the not too distant past, regular processing of closed connections reliably triggered the conn_set_dead condition, hence caused svc_destroy() to be called on the xprt handles. Matt On Wed, Dec 5, 2018 at 3:11 PM William Allen Simpson <william.allen.simpson@gmail.com> wrote:
Wow, I really hate top posts. Very hard to comprehend your responses.
On 12/5/18 12:28 AM, gaurav gangalwar wrote:
I waited for more that 14 hours. This is the script I ran. for i in {1..300}; do echo $i; mount -t nfs -o vers=3,tcp 10.15.196.92:/export /mnt1; umount /mnt1; done It went out of ads at 169th iteration. 169 mount.nfs: Connection timed out umount: /mnt1: not mounted
I really don't understand. There are no constants in the current V2.8-dev code base that add up to more than 14 hours, and no constant that would yield 169.
In any case, this particular denial of service attack should be easy to fix by adding SVC_DESTROY to the Ganesha umount code path(s). However, because it seems rather unlikely this could happen in production, it's not currently a high priority.
Do you have a production environment or customer report where this occurred?
Would you like help writing the Ganesha patch?
Are you with Nutanix? Or IBM?
After this I was not able to mount or do any operation. There was no active mount on my client. So there was no activity from client side, so does it mean it will never cleanup?
No, as I wrote, # If you stop doing anything and simply wait, that will be 8.24 hours # (1023 * 29 seconds).
I've discussed this with DanG, and that ntirpc design logic from before my time seems faulty. It does cleanup more frequently on a busy system, but is very slow on an idle system (the most likely time to have an fd become idle).
DanG suggested that a better approach would be to trigger cleanup based upon a high-water-mark. The best approach would be to order the list of fds by latest recv, and clean out the idle ones as soon as they expire -- instead of running through the list of (tens of) thousands of fds. That shouldn't be too hard, but we must be careful that we don't add new locking in the fast path.
But this is an own-time project for me, and my primary interest is RDMA and performance. So it may be some time before I've a chance to look at it. Maybe over the holiday.... _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (3)
-
gaurav gangalwar -
Matt Benjamin -
William Allen Simpson