proxyv3_getfd_blocking :FSAL :CRIT :Failed to ever acquire a new fd, dying
I've set: LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity in /usr/lib/systemd/system/nfs-ganesha.service and reloaded it then restarted the service but I keep hitting that error when writing large files. I'm using last night's next branch and built rpms from that.
The fds for the proxy_v3 code *must* be from the privileged port range (0 to 1024) because ~all NFSv3 servers require "privileged" clients. But it should also only be opening a few. Num sockets defaults to 32: https://github.com/nfs-ganesha/nfs-ganesha/blob/ef07e96e25e2b8521311d594b6bd... what does your proxy config look like? The proxy has okay logging as well, if you want to add a: ``` LOG { COMPONENTS { FSAL = FULL_DEBUG; NFSPROTO = DEBUG; } } ``` stanza to your config. On Wed, Nov 3, 2021 at 10:38 AM Matthew Hess <mxhess@gmail.com> wrote:
I've set:
LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity
in /usr/lib/systemd/system/nfs-ganesha.service and reloaded it then restarted the service but I keep hitting that error when writing large files.
I'm using last night's next branch and built rpms from that.
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
I pasted my current config below. I just turned on MDCACHE and set up PROXY_V3 to align r and w sizes with the clients and other settings we use just in case I was causing an issue. I also set num_sockets to something crazy large. I've gotten much further in my testing now since these changes. fstab nfs mount opts atm: nofail,vers=4,proto=tcp,nolock,sync,noatime,nodiratime,noac,lookupcache=none I also went crazy and set: LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity LimitNPROC=infinity LimitMEMLOCK=infinity LimitLOCKS=infinity LimitSIGPENDING=infinity in the unit config. Will try to make this more rational as I fine-tune. ## These are core parameters that affect Ganesha as a whole. NFS_CORE_PARAM { ## Allow NFSv3 to mount paths with the Pseudo path, the same as NFSv4, ## instead of using the physical paths. mount_path_pseudo = true; ## Configure the protocols that Ganesha will listen for. This is a hard ## limit, as this list determines which sockets are opened. This list ## can be restricted per export, but cannot be expanded. Protocols = 3,4; } ## These are defaults for exports. They can be overridden per-export. EXPORT_DEFAULTS { ## Access type for clients. Default is None, so some access must be ## given either here or in the export itself. Access_Type = RW; } ## Configure settings for the object handle cache MDCACHE { ## The point at which object cache entries will start being reused. Entries_HWMark = 100000; } ## Configure logging. Default is to log to Syslog. Basic logging can also be ## configured from the command line LOG { ## Default log level for all components Default_Log_Level = WARN; ## Configure per-component log levels. Components { FSAL = INFO; NFS4 = EVENT; } ## Where to log Facility { name = FILE; destination = "/var/log/ganesha/ganesha.log"; enable = active; } } EXPORT { Export_Id = 2345; Path = "/prod"; Pseudo = "/prod"; Access_Type = RW; Squash = No_Root_Squash; SecType = "sys"; Protocols = 3,4; Transports = "TCP"; MaxRead = 65536; MaxWrite = 65536; FSAL { Name = proxy_v3; Srv_Addr = <netapp ip>; } } PROXY_V3 { # How many sockets to keep open in the connection pool. num_sockets = 512; maxread = 65536; maxwrite = 65536; } On Wed, Nov 3, 2021 at 12:01 PM Solomon Boulos <boulos@waymo.com> wrote:
The fds for the proxy_v3 code *must* be from the privileged port range (0 to 1024) because ~all NFSv3 servers require "privileged" clients. But it should also only be opening a few. Num sockets defaults to 32:
https://github.com/nfs-ganesha/nfs-ganesha/blob/ef07e96e25e2b8521311d594b6bd...
what does your proxy config look like?
The proxy has okay logging as well, if you want to add a:
``` LOG { COMPONENTS { FSAL = FULL_DEBUG; NFSPROTO = DEBUG; } } ```
stanza to your config.
On Wed, Nov 3, 2021 at 10:38 AM Matthew Hess <mxhess@gmail.com> wrote:
I've set:
LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity
in /usr/lib/systemd/system/nfs-ganesha.service and reloaded it then restarted the service but I keep hitting that error when writing large files.
I'm using last night's next branch and built rpms from that.
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Glad to hear it. The v3 proxy will automatically respect the maxread / maxwrite settings reported by the backend, so you don't *have* to set them: https://github.com/nfs-ganesha/nfs-ganesha/blob/ef07e96e25e2b8521311d594b6bd... (But I made that a WARN, so maybe it's a bit noisy). On Wed, Nov 3, 2021 at 11:15 AM Matthew Hess <mxhess@gmail.com> wrote:
I pasted my current config below. I just turned on MDCACHE and set up PROXY_V3 to align r and w sizes with the clients and other settings we use just in case I was causing an issue. I also set num_sockets to something crazy large. I've gotten much further in my testing now since these changes.
fstab nfs mount opts atm: nofail,vers=4,proto=tcp,nolock,sync,noatime,nodiratime,noac,lookupcache=none
I also went crazy and set: LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity LimitNPROC=infinity LimitMEMLOCK=infinity LimitLOCKS=infinity LimitSIGPENDING=infinity
in the unit config. Will try to make this more rational as I fine-tune.
## These are core parameters that affect Ganesha as a whole. NFS_CORE_PARAM { ## Allow NFSv3 to mount paths with the Pseudo path, the same as NFSv4, ## instead of using the physical paths. mount_path_pseudo = true;
## Configure the protocols that Ganesha will listen for. This is a hard ## limit, as this list determines which sockets are opened. This list ## can be restricted per export, but cannot be expanded. Protocols = 3,4; }
## These are defaults for exports. They can be overridden per-export. EXPORT_DEFAULTS { ## Access type for clients. Default is None, so some access must be ## given either here or in the export itself. Access_Type = RW; }
## Configure settings for the object handle cache MDCACHE { ## The point at which object cache entries will start being reused. Entries_HWMark = 100000; }
## Configure logging. Default is to log to Syslog. Basic logging can also be ## configured from the command line LOG { ## Default log level for all components Default_Log_Level = WARN;
## Configure per-component log levels. Components { FSAL = INFO; NFS4 = EVENT; }
## Where to log Facility { name = FILE; destination = "/var/log/ganesha/ganesha.log"; enable = active; } }
EXPORT { Export_Id = 2345; Path = "/prod"; Pseudo = "/prod";
Access_Type = RW; Squash = No_Root_Squash; SecType = "sys"; Protocols = 3,4; Transports = "TCP"; MaxRead = 65536; MaxWrite = 65536;
FSAL { Name = proxy_v3; Srv_Addr = <netapp ip>; }
}
PROXY_V3 { # How many sockets to keep open in the connection pool. num_sockets = 512; maxread = 65536; maxwrite = 65536; }
On Wed, Nov 3, 2021 at 12:01 PM Solomon Boulos <boulos@waymo.com> wrote:
The fds for the proxy_v3 code *must* be from the privileged port range (0 to 1024) because ~all NFSv3 servers require "privileged" clients. But it should also only be opening a few. Num sockets defaults to 32:
https://github.com/nfs-ganesha/nfs-ganesha/blob/ef07e96e25e2b8521311d594b6bd...
what does your proxy config look like?
The proxy has okay logging as well, if you want to add a:
``` LOG { COMPONENTS { FSAL = FULL_DEBUG; NFSPROTO = DEBUG; } } ```
stanza to your config.
On Wed, Nov 3, 2021 at 10:38 AM Matthew Hess <mxhess@gmail.com> wrote:
I've set:
LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity
in /usr/lib/systemd/system/nfs-ganesha.service and reloaded it then restarted the service but I keep hitting that error when writing large files.
I'm using last night's next branch and built rpms from that.
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
The good news is that I was able to complete a 500GB write with only a warning now: nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 7439317, old: 7439253; deq new: 7439253, old: 7439253 any potential risk there? On Wed, Nov 3, 2021 at 12:22 PM Solomon Boulos <boulos@waymo.com> wrote:
Glad to hear it. The v3 proxy will automatically respect the maxread / maxwrite settings reported by the backend, so you don't *have* to set them:
https://github.com/nfs-ganesha/nfs-ganesha/blob/ef07e96e25e2b8521311d594b6bd...
(But I made that a WARN, so maybe it's a bit noisy).
On Wed, Nov 3, 2021 at 11:15 AM Matthew Hess <mxhess@gmail.com> wrote:
I pasted my current config below. I just turned on MDCACHE and set up PROXY_V3 to align r and w sizes with the clients and other settings we use just in case I was causing an issue. I also set num_sockets to something crazy large. I've gotten much further in my testing now since these changes.
fstab nfs mount opts atm: nofail,vers=4,proto=tcp,nolock,sync,noatime,nodiratime,noac,lookupcache=none
I also went crazy and set: LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity LimitNPROC=infinity LimitMEMLOCK=infinity LimitLOCKS=infinity LimitSIGPENDING=infinity
in the unit config. Will try to make this more rational as I fine-tune.
## These are core parameters that affect Ganesha as a whole. NFS_CORE_PARAM { ## Allow NFSv3 to mount paths with the Pseudo path, the same as NFSv4, ## instead of using the physical paths. mount_path_pseudo = true;
## Configure the protocols that Ganesha will listen for. This is a hard ## limit, as this list determines which sockets are opened. This list ## can be restricted per export, but cannot be expanded. Protocols = 3,4; }
## These are defaults for exports. They can be overridden per-export. EXPORT_DEFAULTS { ## Access type for clients. Default is None, so some access must be ## given either here or in the export itself. Access_Type = RW; }
## Configure settings for the object handle cache MDCACHE { ## The point at which object cache entries will start being reused. Entries_HWMark = 100000; }
## Configure logging. Default is to log to Syslog. Basic logging can also be ## configured from the command line LOG { ## Default log level for all components Default_Log_Level = WARN;
## Configure per-component log levels. Components { FSAL = INFO; NFS4 = EVENT; }
## Where to log Facility { name = FILE; destination = "/var/log/ganesha/ganesha.log"; enable = active; } }
EXPORT { Export_Id = 2345; Path = "/prod"; Pseudo = "/prod";
Access_Type = RW; Squash = No_Root_Squash; SecType = "sys"; Protocols = 3,4; Transports = "TCP"; MaxRead = 65536; MaxWrite = 65536;
FSAL { Name = proxy_v3; Srv_Addr = <netapp ip>; }
}
PROXY_V3 { # How many sockets to keep open in the connection pool. num_sockets = 512; maxread = 65536; maxwrite = 65536; }
On Wed, Nov 3, 2021 at 12:01 PM Solomon Boulos <boulos@waymo.com> wrote:
The fds for the proxy_v3 code *must* be from the privileged port range (0 to 1024) because ~all NFSv3 servers require "privileged" clients. But it should also only be opening a few. Num sockets defaults to 32:
https://github.com/nfs-ganesha/nfs-ganesha/blob/ef07e96e25e2b8521311d594b6bd...
what does your proxy config look like?
The proxy has okay logging as well, if you want to add a:
``` LOG { COMPONENTS { FSAL = FULL_DEBUG; NFSPROTO = DEBUG; } } ```
stanza to your config.
On Wed, Nov 3, 2021 at 10:38 AM Matthew Hess <mxhess@gmail.com> wrote:
I've set:
LimitAS=infinity LimitRSS=infinity LimitCORE=infinity LimitNOFILE=infinity
in /usr/lib/systemd/system/nfs-ganesha.service and reloaded it then restarted the service but I keep hitting that error when writing large files.
I'm using last night's next branch and built rpms from that.
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
participants (2)
-
Matthew Hess -
Solomon Boulos