The ganesha_mgr python script must be timing out since the DBUS service thread is blocked in your FSAL.

 

I’m not sure what kind of library call you would want, and any interface like that would be added to Ganesha V5 (we just decided this morning to start the release candidate cycle for V4 so no more features will be accepted into V4). V4 will have a robust dynamic export config with updates triggered by SIGHUP though you wouldn’t know when the update process is complete. The existing interface that DBUS calls or even the interface SIGHUP calls could be exported so you could hook up your own trigger either by incorporating Ganesha as library into your own front end, or driving the interface from code in your FSAL .so, either could then expose some kind of control interface that suits your needs better. If it’s just a matter of making sure the existing interfaces are exported of course that IS something that could drop during the release candidate cycle.

 

You may want to explore why the lookup_path is taking so long, no matter how you drive the update export interfaces, a long delay in that will result in a significant exposure where your exports are in flux.

 

Note that the V4 dynamic export management via SIGHUP is significantly improved, allowing exports to be removed (if an export is no longer in the config, it will be removed) and additional parameters including Pseudo capable of being modified.

 

Frank

 

From: Sandeep Nashikkar via Devel [mailto:devel@lists.nfs-ganesha.org]
Sent: Tuesday, October 12, 2021 8:22 AM
To: devel@lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Devel] ganesha_mgr add_export command returning premature success when underlying storage is slow

 

Hello

 

We are seeing issue as following:

 

  1. We are using our own FSAL with ganesha version 2.8.2 which eventually performs operation on underlying storage through a wrapper layer.
  2. We perform add_export programmatically when required. When add_export command tries to lstat, sometimes the underlying storage takes time to return the attributes. Here is the stack where it gets stuck for some time:
    #5  0x00007fdc9d24ea0f in lookup_path () from /usr/lib64/ganesha/libfsaltdfs.so

#6  0x000000000053f719 in mdcache_lookup_path (exp_hdl=0x7fdbfa43d900, path=0x7fdbfa40f200 "/Chk1234", handle=0x7fdbfc0daac8, attrs_out=0x0) at /usr/src/debug/nfs-gan

esha-2.6.3/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1697

#7  0x0000000000504454 in init_export_root (export=0x7fdbfa435448) at /usr/src/debug/nfs-ganesha-2.6.3/support/exports.c:2253

#8  0x00000000005029c2 in export_commit_common (node=0x7fdbfa4331a0, link_mem=0x0, self_struct=0x7fdbfa435448, err_type=0x7fdbfc0db160, commit_type=add_export) at /us

r/src/debug/nfs-ganesha-2.6.3/support/exports.c:1235

#9  0x0000000000502e27 in add_export_commit (node=0x7fdbfa4331a0, link_mem=0x0, self_struct=0x7fdbfa435448, err_type=0x7fdbfc0db160) at /usr/src/debug/nfs-ganesha-2.6

.3/support/exports.c:1351

#10 0x000000000055832b in proc_block (node=0x7fdbfa4331a0, item=0x7e51a8 <add_export_param+8>, link_mem=0x0, err_type=0x7fdbfc0db160) at /usr/src/debug/nfs-ganesha-2.

6.3/config_parsing/config_parsing.c:1362

#11 0x0000000000559116 in load_config_from_node (tree_node=0x7fdbfa4331a0, conf_blk=0x7e51a0 <add_export_param>, param=0x0, unique=false, err_type=0x7fdbfc0db160) at

/usr/src/debug/nfs-ganesha-2.6.3/config_parsing/config_parsing.c:1862

#12 0x0000000000518d42 in gsh_export_addexport (args=0x7fdbfc0db220, reply=0x7fdc97bf5180, error=0x7fdbfc0db270) at /usr/src/debug/nfs-ganesha-2.6.3/support/export_mg

r.c:1038

#13 0x000000000055330b in dbus_message_entrypoint (conn=0x7fdc97aaa0c0, msg=0x7fdbfa42a440, user_data=0x7e5ff0 <export_interfaces>) at /usr/src/debug/nfs-ganesha-2.6.

3/dbus/dbus_server.c:564

#14 0x00007fdc9ca50276 in _dbus_object_tree_dispatch_and_unlock () from /lib64/libdbus-1.so.3

#15 0x00007fdc9ca41b29 in dbus_connection_dispatch () from /lib64/libdbus-1.so.3

#16 0x00007fdc9ca41e42 in _dbus_connection_read_write_dispatch () from /lib64/libdbus-1.so.3

#17 0x0000000000553ed9 in gsh_dbus_thread (arg=0x0) at /usr/src/debug/nfs-ganesha-2.6.3/dbus/dbus_server.c:796

#18 0x00007fdc9b429e25 in start_thread () from /lib64/libpthread.so.0

#19 0x00007fdc9aae6bad in clone () from /lib64/libc.so.6

  1. Despite the lstat getting stuck, add_export command returns prematurely in 30-40 seconds or so and return code is success. This is not expected.
  2. This causes subsequent mount to fail since the export is not actually ready for the mount (init_export_root not being complete)

 

Can someone help here to understand why dbus interface returns success? Is this known issue? How can we fix this behavior?

 

Can we rather have a library interface to create/delete/update exports rather than relying on ganesha_mgr python script and dbus interface?

 

Thanks

Sandeep