Problems about cache virtual glusterfs ACLs for ganesha in md-cache
Cc nfs-ganesha, Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now. But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs. After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors. https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305 There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests. 2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd. Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not. So, I'd prefer #2. Any comments are welcome. thanks, Kinglong Mee
On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee <kinglongmee@gmail.com> wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
Not sure how virtual xattrs are consumed or who generates them. +Raghavendra Talur <rtalur@redhat.com> +Thottan, Jiffin <jthottan@redhat.com> - acl maintainers.
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2. Any comments are welcome.
thanks, Kinglong Mee
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
On Thursday 11 October 2018 08:10 AM, Raghavendra Gowdappa wrote:
On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee <kinglongmee@gmail.com <mailto:kinglongmee@gmail.com>> wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
Not sure how virtual xattrs are consumed or who generates them. +Raghavendra Talur <mailto:rtalur@redhat.com> +Thottan, Jiffin <mailto:jthottan@redhat.com> - acl maintainers.
The currently only consumers of this virtual xattr is nfs-ganesha. Nfsv4 acls were sent from client and ganesha converts to posix acl and sent as virtual xattr to glusterfs bricks using pub_glfs_h_acl_set/get api's. AFAIR in samba vfs module they convert windows acl to posix acl and sent as actual getxattr/setxattr calls on "system.posixl-acl" -- Jiffin
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2. Any comments are welcome.
thanks, Kinglong Mee
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org> https://lists.gluster.org/mailman/listinfo/gluster-devel
On 2018/10/11 12:10, Jiffin Tony Thottan wrote:> On Thursday 11 October 2018 08:10 AM, Raghavendra Gowdappa wrote:
On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee <kinglongmee@gmail.com <mailto:kinglongmee@gmail.com>> wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
Not sure how virtual xattrs are consumed or who generates them. +Raghavendra Talur <mailto:rtalur@redhat.com> +Thottan, Jiffin <mailto:jthottan@redhat.com> - acl maintainers.
The currently only consumers of this virtual xattr is nfs-ganesha. Nfsv4 acls were sent from client and ganesha converts to posix acl
and sent as virtual xattr to glusterfs bricks using pub_glfs_h_acl_set/get api's. AFAIR in samba vfs module they convert windows acl to
posix acl and sent as actual getxattr/setxattr calls on "system.posixl-acl"
--
Jiffin
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
Hi Jiffin, Thanks for your explain. I'm testing the cache of virtual glusterfs ACLs in md-cache for ganesha's nfsv4. I wanna to know the plan of ACLs in the future. Are both posix ACLs (used by samba) and virtual glusterfs ACLs (used by ganesha's nfsv4) reserved? Or Is only virtual glusterfs ACLs reserved? If both are reserved, there are two chooses for md-cache caches virtual glusterfs ACLs.
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2.
If only virtual glusterfs ACLs is reserved, I'd like chose the #1 implement for md-cache. Md-cache reserves caching posix acl until samba moving to the virtual glusterfs ACLs. thanks, Kinglong Mee
NFS-Ganesha's md-cache layer already does extensive caching of attributes and ACLs of each file looked upon. Do you see any additional benefit with turning on gluster md-cache as well? More replies inline.. On 10/11/18 7:47 AM, Kinglong Mee wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
I am not sure if posix layer can handle it. Virtual xattrs are in-memory and not stored on disk. They are converted to/from posix-acl in posix-acl xlator. So FWIU, posix-acl xlator should handle setting these attributes as part of LOOKUP response if needed. Same shall apply for any virtual xattr cached in md-cache. Request Poornima to comment. At a time, any gfapi consumer would use either posix-acl or virtual glusterfs ACLs. So having two options to selectively choose which one of them to cache sounds better to me instead of unnecessarily storing two different representations of the same ACL. Thanks, Soumya
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2. Any comments are welcome.
thanks, Kinglong Mee
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
On 2018/10/11 19:09, Soumya Koduri wrote:
NFS-Ganesha's md-cache layer already does extensive caching of attributes and ACLs of each file looked upon. Do you see any additional benefit with turning on gluster md-cache as well? More replies inline..
Yes, I think. The logical is different between md-cache and ganesha's cache, Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from back-end glusterfs; Md-cache caches depneds on timeout too, but md-cache can delay the timeout for some cases.
On 10/11/18 7:47 AM, Kinglong Mee wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
I am not sure if posix layer can handle it. Virtual xattrs are in-memory and not stored on disk. They are converted to/from posix-acl in posix-acl xlator. So FWIU, posix-acl xlator should handle setting these attributes as part of LOOKUP response if needed. Same shall apply for any virtual xattr cached in md-cache. Request Poornima to comment.
Posix-acl can hand it correctly now.
At a time, any gfapi consumer would use either posix-acl or virtual glusterfs ACLs. So having two options to selectively choose which one of them to cache sounds better to me instead of unnecessarily storing two different representations of the same ACL.
Make sense. I will add another option for virtual glusterfs ACLs in md-cache. thanks, Kinglong Mee
Thanks, Soumya
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2. Any comments are welcome.
thanks, Kinglong Mee
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
On 10/12/18 7:22 AM, Kinglong Mee wrote:
On 2018/10/11 19:09, Soumya Koduri wrote:
NFS-Ganesha's md-cache layer already does extensive caching of attributes and ACLs of each file looked upon. Do you see any additional benefit with turning on gluster md-cache as well? More replies inline..
Yes, I think.
The logical is different between md-cache and ganesha's cache, Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from back-end glusterfs; Md-cache caches depneds on timeout too, but md-cache can delay the timeout for some cases.
Could you please list out which xattrs fall into that category. AFAIK, like iatt, even all xattrs are invalidated post timeout in md-cache xlator. By turning on both these caches, the process shall consume more memory and CPUs to store and invalidate same set of attributes at two different layers right. Do you see much better performance when compared to that cost? Thanks, Soumya
On 10/11/18 7:47 AM, Kinglong Mee wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
I am not sure if posix layer can handle it. Virtual xattrs are in-memory and not stored on disk. They are converted to/from posix-acl in posix-acl xlator. So FWIU, posix-acl xlator should handle setting these attributes as part of LOOKUP response if needed. Same shall apply for any virtual xattr cached in md-cache. Request Poornima to comment.
Posix-acl can hand it correctly now.
At a time, any gfapi consumer would use either posix-acl or virtual glusterfs ACLs. So having two options to selectively choose which one of them to cache sounds better to me instead of unnecessarily storing two different representations of the same ACL.
Make sense. I will add another option for virtual glusterfs ACLs in md-cache.
thanks, Kinglong Mee
Thanks, Soumya
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2. Any comments are welcome.
thanks, Kinglong Mee
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
On 10/12/18 12:04 PM, Soumya Koduri wrote:
1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
I am not sure if posix layer can handle it. Virtual xattrs are in-memory and not stored on disk. They are converted to/from posix-acl in posix-acl xlator. So FWIU, posix-acl xlator should handle setting these attributes as part of LOOKUP response if needed. Same shall apply for any virtual xattr cached in md-cache. Request Poornima to comment.
Posix-acl can hand it correctly now.
Okay.
At a time, any gfapi consumer would use either posix-acl or virtual glusterfs ACLs. So having two options to selectively choose which one of them to cache sounds better to me instead of unnecessarily storing two different representations of the same ACL.
Make sense. I will add another option for virtual glusterfs ACLs in md-cache.
Cool. thanks! -Soumya
thanks, Kinglong Mee
On 2018/10/12 14:34, Soumya Koduri wrote:> On 10/12/18 7:22 AM, Kinglong Mee wrote:
On 2018/10/11 19:09, Soumya Koduri wrote:
NFS-Ganesha's md-cache layer already does extensive caching of attributes and ACLs of each file looked upon. Do you see any additional benefit with turning on gluster md-cache as well? More replies inline..
Yes, I think.
The logical is different between md-cache and ganesha's cache, Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from back-end glusterfs; Md-cache caches depneds on timeout too, but md-cache can delay the timeout for some cases.
Could you please list out which xattrs fall into that category. AFAIK, like iatt, even all xattrs are invalidated post timeout in md-cache xlator.
The iatt's expire time is ->ia_time + timeout, xatt's expire time is ->xa_time + timeout. Most FOPs reply (read/write/truncate...) contains the postattr incidentally, and update the ->ia_time. But, ->xa_time cannot be updated as ->ia_time now, it only be updated at mdc_lookup_cbk now. I will add a case of update ->xa_time if file's mtiem/ctime does not change when updating the ->ia_time. And, let stat/fstat/setattr/fsetattr/getxattr/fgetxattr's reply contains the xattr incidentally, and update the ->xa_time too.
By turning on both these caches, the process shall consume more memory and CPUs to store and invalidate same set of attributes at two different layers right. Do you see much better performance when compared to that cost?
I think md-cache supporting cache virtual ACLs is a function update, ganesha can use it by the new added option, or not. If setting a smaller timeout for ganesha's cache, it timeout frequently. But, md-cache can return the cached xattr. I do not have a comparing data between them. After testing, we can chose the better combination, 1. md-cache on and ganesha's cache on; 2. md-cache on but ganesha's cache off; 3. md-cache off but ganesha's cache on. thanks, Kinglong Mee
Thanks, Soumya
On 10/11/18 7:47 AM, Kinglong Mee wrote:
Cc nfs-ganesha,
Md-cache has option "cache-posix-acl" that controls caching of posix ACLs ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup requests. So, md-cache caches bad virtual glusterfs ACLs.
After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client gets many EIO errors.
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
There are two chooses for cache virtual glusterfs ACLs in md-cache, 1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" is added); And make sure _posix_xattr_get_set fills them when lookup requests.
I am not sure if posix layer can handle it. Virtual xattrs are in-memory and not stored on disk. They are converted to/from posix-acl in posix-acl xlator. So FWIU, posix-acl xlator should handle setting these attributes as part of LOOKUP response if needed. Same shall apply for any virtual xattr cached in md-cache. Request Poornima to comment.
Posix-acl can hand it correctly now.
At a time, any gfapi consumer would use either posix-acl or virtual glusterfs ACLs. So having two options to selectively choose which one of them to cache sounds better to me instead of unnecessarily storing two different representations of the same ACL.
Make sense. I will add another option for virtual glusterfs ACLs in md-cache.
thanks, Kinglong Mee
Thanks, Soumya
2. Does not cache it, only cache posix ACLs; If gfapi request it, md-cache lookup according posix ACL at cache, if exist, make the virtual glusterfs ACL locally and return to gfapi; otherwise, send the request to glusterfsd.
Virtual glusterfs ACLs are another format of posix ACLs, there are larger than posix ACLs, and always exist no matter the really posix ACL exist or not.
So, I'd prefer #2. Any comments are welcome.
thanks, Kinglong Mee
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
On 10/12/18 5:55 PM, Kinglong Mee wrote:
On 2018/10/12 14:34, Soumya Koduri wrote:> On 10/12/18 7:22 AM, Kinglong Mee wrote:
On 2018/10/11 19:09, Soumya Koduri wrote:
NFS-Ganesha's md-cache layer already does extensive caching of attributes and ACLs of each file looked upon. Do you see any additional benefit with turning on gluster md-cache as well? More replies inline..
Yes, I think.
The logical is different between md-cache and ganesha's cache, Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from back-end glusterfs; Md-cache caches depneds on timeout too, but md-cache can delay the timeout for some cases.
Could you please list out which xattrs fall into that category. AFAIK, like iatt, even all xattrs are invalidated post timeout in md-cache xlator.
The iatt's expire time is ->ia_time + timeout, xatt's expire time is ->xa_time + timeout. Most FOPs reply (read/write/truncate...) contains the postattr incidentally, and update the ->ia_time. But, ->xa_time cannot be updated as ->ia_time now, it only be updated at mdc_lookup_cbk now.
I will add a case of update ->xa_time if file's mtiem/ctime does not change when updating the ->ia_time. And, let stat/fstat/setattr/fsetattr/getxattr/fgetxattr's reply contains the xattr incidentally, and update the ->xa_time too.
+1
By turning on both these caches, the process shall consume more memory and CPUs to store and invalidate same set of attributes at two different layers right. Do you see much better performance when compared to that cost?
I think md-cache supporting cache virtual ACLs is a function update, ganesha can use it by the new added option, or not.
If setting a smaller timeout for ganesha's cache, it timeout frequently. But, md-cache can return the cached xattr.
I do not have a comparing data between them. After testing, we can chose the better combination, 1. md-cache on and ganesha's cache on; 2. md-cache on but ganesha's cache off; 3. md-cache off but ganesha's cache on.
yes..Please share your findings if you get to test these combinations. Thanks, Soumya
participants (4)
-
Jiffin Tony Thottan -
Kinglong Mee -
Raghavendra Gowdappa -
Soumya Koduri