[Nfs-ganesha-devel] Re: Pnfs flex layout: synthetic uid/gid

On Fri, 2018-06-29 at 11:29 +0200, Supriti Singh wrote: (resending since my earlier message apparently needs approval) One idea: stick the synthetic uid/gid into xattrs attached to the backing objects for each inode. The DS READ/WRITE calls would go to a NFS server running on each OSD. That then gets converted into a RADOS op. The one for a WRITE would do something like: rados_write_op_create(); /* Compare the uid vs the uid xattr */ rados_write_opcmpxattr("rpc uid"); /* Do the write */ rados_write_op_write(...); The READ -> read op conversion would look similar. Before handing out a layout stateid, we'd have to go through and check and/or set the xattrs on all of the objects in the layout. That could conceivably be done by ganesha without bothering the Ceph MDS. That said, we may need to consider having the Ceph MDS be final arbiter of these values, which may help prevent races in clustered MDS configurations? I haven't given this enough thought yet. Ganesha shouldn't need to send a NFS SETATTR to a ceph cluster. This can be done with the OSD protocol on the backend. Basically: 1) client does LAYOUTGET 2) Determine the location of the different layout segments 3) issue RADOS write ops to all of the objects to change their xattrs. (basically do a cmpxattr and then a setxattr in the same operation to set it) 4) build LAYOUTGET reply and send it back to client To fence, we'd just need to go through and reset all of the xattrs first (incrementing them is the usual thing to do) before allowing a conflicting operation to proceed. -- Jeff Layton <jlayton(a)poochiereds.net&gt;