This was reported on ntirpc at Github. After DanG's fix for transport reference counting, Gaurav Gangalwar wrote that s/he can deplete the available fds with: for i in {1..1000}; do echo $i; mount -t nfs -o vers=3,tcp :/export /mnt; umount /mnt; done There is a configurable parameter limiting the number of concurrent fds. They are cleaned up after a fairly long inactivity timeout. So an adversary (or runaway client) can quickly deplete the fds, preventing valid clients from mounting. The most obvious patch would be for umount to add the same SVC_DESTROY line that DanG (and previously me) had added for the TCP well known ports in dispatch. But that would allow runaway clients to flood the server. Preventing that seems to be the (undocumented) purpose of the configurable parameter. The parameter and cleanup code are relatively old, and added to ntirpc circa 2013. Was it a bad idea? Which security issue is more important? _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org

On 12/4/18 12:27 AM, gaurav gangalwar wrote: > Thanks for raising it on Ganesha list. > Just want to add, once fds are depleted, they are not getting cleaned up even after hours. > I needed to restart Ganesha process to recover from this state. How many hours did you wait? Currently, the cleanup is triggered after 1023 epoll wakeups. If you stop doing anything and simply wait, that will be 8.24 hours (1023 * 29 seconds). Before 2.6, that was 120 seconds for 34 hours. After all, there's no good reason to cleanup with no activity. Also, cleanup means fd has no recv activity in __svc_params->idle_timeout. Ganesha default is nfs_param.core_param.rpc.idle_timeout_s = 300 seconds. If you're using a standard client doing idles, it will never cleanup. > I am not sure if idle cleanup code is able to get rid of extra ref and release xprt. There is no extra ref. We had one too few refs. That's what DanG fixed. > I tried with this fix in cleanup code its working for me. > https://paste.fedoraproject.org/paste/lapz1NrOlBxS342S4Q79aw > As I tried to explain where s/he reported it, that isn't a fix. It releases some other reference still remaining, which in turn will probably lead to referencing freed memory. All this does is force a close(fd), which allows more connections reusing the old fd. Masking the symptoms. The real underlying security issue is that a malicious adversary can run a resource exhaustion attack. That's what needs to be fixed.

In any case, this particular denial of service attack should be easy to fix by adding SVC_DESTROY to the Ganesha umount code path(s).

However, because it seems rather unlikely this could happen in production, it's not currently a high priority. Do you have a production environment or customer report where this occurred?

Would you like help writing the Ganesha patch?

Are you with Nutanix? Or IBM?

Wow, I really hate top posts. Very hard to comprehend your responses. On 12/5/18 12:28 AM, gaurav gangalwar wrote: > I waited for more that 14 hours. This is the script I ran. > for i in {1..300}; do echo $i; mount -t nfs -o vers=3,tcp 10.15.196.92:/export /mnt1; umount /mnt1; done > It went out of ads at 169th iteration. > 169 > mount.nfs: Connection timed out > umount: /mnt1: not mounted > I really don't understand. There are no constants in the current V2.8-dev code base that add up to more than 14 hours, and no constant that would yield 169. In any case, this particular denial of service attack should be easy to fix by adding SVC_DESTROY to the Ganesha umount code path(s). However, because it seems rather unlikely this could happen in production, it's not currently a high priority. Do you have a production environment or customer report where this occurred? Would you like help writing the Ganesha patch? Are you with Nutanix? Or IBM? > After this I was not able to mount or do any operation. There was no active mount on my client. So there was no activity from client side, so does it mean it will never cleanup? > No, as I wrote, # If you stop doing anything and simply wait, that will be 8.24 hours # (1023 * 29 seconds). I've discussed this with DanG, and that ntirpc design logic from before my time seems faulty. It does cleanup more frequently on a busy system, but is very slow on an idle system (the most likely time to have an fd become idle). DanG suggested that a better approach would be to trigger cleanup based upon a high-water-mark. The best approach would be to order the list of fds by latest recv, and clean out the idle ones as soon as they expire -- instead of running through the list of (tens of) thousands of fds. That shouldn't be too hard, but we must be careful that we don't add new locking in the fast path. But this is an own-time project for me, and my primary interest is RDMA and performance. So it may be some time before I've a chance to look at it. Maybe over the holiday.... _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org