On Feb 21, 2019, at 8:43 AM, Daniel Gryniewicz <dang(a)redhat.com&gt; wrote: > On 2/21/19 8:07 AM, TomK wrote: > Hey All, > Wondering if there would be a cross domain issue if KRB5 is enabled on NFS Ganesha and thereby causing an issue if the NFS Ganesha servers are on say, domain aaa.123.xyz while the clients are on domain bbb.123.xyz? > Users get a permission denied on clients when accessing the remote mount and while checking I see it's owned by nobody / nobody rather then their user and group. > The client is on a different DNS domain and ownership comes through with nobody/nobody. Hence clients get a permission denied accessing the folder I'm thinking. However the ONLY thing different that stands out ATM is the domains being different. > Wondering if anyone ran into a similar experience and could share their solution? Do I need to add the realm to the /etc/krb5.conf config on the NFS Ganesha servers perhaps? So, I believe that the clients and servers need to be in the same KRB5 domain. This would likely be 123.xyz in your case, rather than using the sub-domains.

Daniel _______________________________________________ Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org

> > > Sent from my iPhone > >> On Feb 21, 2019, at 8:43 AM, Daniel Gryniewicz <dang(a)redhat.com&gt; wrote: >> >>> On 2/21/19 8:07 AM, TomK wrote: >>> Hey All, >>> Wondering if there would be a cross domain issue if KRB5 is enabled on NFS Ganesha and thereby causing an issue if the NFS Ganesha servers are on say, domain aaa.123.xyz while the clients are on domain bbb.123.xyz? >>> Users get a permission denied on clients when accessing the remote mount and while checking I see it's owned by nobody / nobody rather then their user and group. >>> The client is on a different DNS domain and ownership comes through with nobody/nobody. Hence clients get a permission denied accessing the folder I'm thinking. However the ONLY thing different that stands out ATM is the domains being different. >>> Wondering if anyone ran into a similar experience and could share their solution? Do I need to add the realm to the /etc/krb5.conf config on the NFS Ganesha servers perhaps? >> >> So, I believe that the clients and servers need to be in the same KRB5 domain. This would likely be 123.xyz in your case, rather than using the sub-domains. > > K. > > Will have to add the realm section etc then. Need the one nfs server shared between two kdc ‘s . I have a separate kdc per subdomain. ABC.xyz is the ad dc in this case. Apologies, I misread I think. Thinking you mean there's only one domain configuration option in the NFS Ganesha config hence why NFS Ganesha only supports a single domain? /etc/ganesha/ganesha.conf NFSv4 { Lease_Lifetime = 20 ; IdmapConf = "/etc/idmapd.conf" ; DomainName = "b.abc.123" ; } NFS_KRB5 { PrincipalName = &quot;nfs/nfs01.b.abc.123(a)B.ABC.123&quot; ; KeytabPath = /etc/krb5.keytab ; Active_krb5 = YES ; } # cat /etc/idmapd.conf [General] Domain = b.abc.123 [Mapping] [Translation] [Static] [UMICH_SCHEMA] LDAP_server = ldap-server.local.domain.edu LDAP_base = dc=local,dc=domain,dc=edu You have new mail in /var/spool/mail/root The user is the same account on the second subdomain but the trust with the AD DC through which it's served is formed with a separate Linux KDC on that domain. Perhaps this diagram will come through correctly sam(a)abc.123 <-- IPA KDC 1 (a.abc.123) -> AD DC (abc.123) sam(a)abc.123 <-- IPA KDC 2 (b.abc.123) -----^ The NFS cluster is shared between IPA KDC 1 and 2 . So in this case, if I wanted this scenario to work, guessing I may have to create a separate NFS cluster and extend the Gluster FS over 3 new additional nodes on domain a.abc.123 to ensure the same user's files and storage is always in sync across domain / kerberos realm. Reason for all this is that we're testing options for isolating certain software / projects from other domains (keeping them contained within a specific realm). The software can modify the KDC extensively and hence why we prefer it has it's own KDC.

On 2/22/19 10:42 AM, Frank Filz wrote: > On 2/21/2019 11:34 AM, Tom wrote: >>> >>> >>> Sent from my iPhone >>> >>>> On Feb 21, 2019, at 8:43 AM, Daniel Gryniewicz <dang(a)redhat.com&gt; >>>> wrote: >>>> >>>>> On 2/21/19 8:07 AM, TomK wrote: >>>>> Hey All, >>>>> Wondering if there would be a cross domain issue if KRB5 is >>>>> enabled on NFS >> Ganesha and thereby causing an issue if the NFS Ganesha servers are >> on say, >> domain aaa.123.xyz while the clients are on domain bbb.123.xyz? >>>>> Users get a permission denied on clients when accessing the remote >>>>> mount >> and while checking I see it's owned by nobody / nobody rather then >> their user >> and group. >>>>> The client is on a different DNS domain and ownership comes >>>>> through with >> nobody/nobody.  Hence clients get a permission denied accessing the >> folder I'm >> thinking. However the ONLY thing different that stands out ATM is the >> domains >> being different. >>>>> Wondering if anyone ran into a similar experience and could share >>>>> their >> solution?  Do I need to add the realm to the /etc/krb5.conf config on >> the NFS >> Ganesha servers perhaps? >>>> >>>> So, I believe that the clients and servers need to be in the same >>>> KRB5 domain. >> This would likely be 123.xyz in your case, rather than using the >> sub-domains. >>> >>> K. >>> >>> Will have to add the realm section etc then.  Need the one nfs >>> server shared >> between two kdc ‘s .  I have a separate kdc per subdomain.  ABC.xyz >> is the ad dc >> in this case. >> >> Apologies, I misread I think.  Thinking you mean there's only one domain >> configuration option in the NFS Ganesha config hence why NFS Ganesha >> only >> supports a single domain? >> >> /etc/ganesha/ganesha.conf >> NFSv4 { >>       Lease_Lifetime = 20 ; >>       IdmapConf = "/etc/idmapd.conf" ; >>       DomainName = "b.abc.123" ; >> } >> NFS_KRB5 { >>           PrincipalName = &quot;nfs/nfs01.b.abc.123(a)B.ABC.123&quot; ; >>           KeytabPath = /etc/krb5.keytab ; >>           Active_krb5 = YES ; >> } >> >> >> # cat /etc/idmapd.conf >> [General] >> Domain = b.abc.123 >> [Mapping] >> [Translation] >> [Static] >> [UMICH_SCHEMA] >> LDAP_server = ldap-server.local.domain.edu LDAP_base = >> dc=local,dc=domain,dc=edu You have new mail in /var/spool/mail/root >> >> The user is the same account on the second subdomain but the trust >> with the AD >> DC through which it's served is formed with a separate Linux KDC on that >> domain. >> >> Perhaps this diagram will come through correctly >> >> sam(a)abc.123 <-- IPA KDC 1 (a.abc.123) -> AD DC (abc.123) >> sam(a)abc.123 <-- IPA KDC 2 (b.abc.123) -----^ >> >> The NFS cluster is shared between IPA KDC 1 and 2 . >> >> So in this case, if I wanted this scenario to work, guessing I may >> have to create a >> separate NFS cluster and extend the Gluster FS over 3 new additional >> nodes on >> domain a.abc.123 to ensure the same user's files and storage is >> always in sync >> across domain / kerberos realm. >> >> Reason for all this is that we're testing options for isolating >> certain software / >> projects from other domains (keeping them contained within a specific >> realm). >> The software can modify the KDC extensively and hence why we prefer >> it has it's >> own KDC. > > Yes, I think you pretty much need two separate NFS server clusters. > > If you only use NFS v4.x, you might be able to have them run on the > same nodes by assigning one to a different port, but then the clients > need to specify the port to connect to so they get the right KDC. > > That Domain parameter in the NFSv4 config block I think only gets used > if you don't use idmapd. > > I'm not sure how or if idmapd works with multiple domains (Ganesha > just passes the domain through, everything is configured using > idmapd.conf). But I don't think you need multiple idmap domains. > > So I think the issue is just going to come down to the server only has > a single Kerberos service principal specified by PrincipleName in the > NFS_KRB5 config block. That principal of course belongs to a single > Kerberos realm. > > Frank > Note, you don't need separate Gluster clusters.  There's no requirement that a Ganesha instance run on the same server as one of the bricks, so you can just add a second Ganesha instance pointed at the same Gluster cluster. Daniel _______________________________________________ Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org

> -----Original Message----- > From: TomK [mailto:tomkcpr@mdevsys.com] > Sent: Monday, February 25, 2019 5:48 PM > To: dang(a)redhat.com; Frank Filz <ffilzlnx(a)mindspring.com>; 'Tom' > <tk(a)mdevsys.com&gt; > Cc: support(a)lists.nfs-ganesha.org > Subject: [NFS-Ganesha-Support] Re: KRB5, nfs-ganesha and directories > appearing with ownership as nobody/nobody on client side. > > On 2/22/2019 10:45 AM, Daniel Gryniewicz wrote: >> On 2/22/19 10:42 AM, Frank Filz wrote: >>> On 2/21/2019 11:34 AM, Tom wrote: >>>>> >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Feb 21, 2019, at 8:43 AM, Daniel Gryniewicz <dang(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> On 2/21/19 8:07 AM, TomK wrote: >>>>>>> Hey All, >>>>>>> Wondering if there would be a cross domain issue if KRB5 is >>>>>>> enabled on NFS >>>> Ganesha and thereby causing an issue if the NFS Ganesha servers are >>>> on say, domain aaa.123.xyz while the clients are on domain >>>> bbb.123.xyz? >>>>>>> Users get a permission denied on clients when accessing the >>>>>>> remote mount >>>> and while checking I see it's owned by nobody / nobody rather then >>>> their user and group. >>>>>>> The client is on a different DNS domain and ownership comes >>>>>>> through with >>>> nobody/nobody. Hence clients get a permission denied accessing the >>>> folder I'm thinking. However the ONLY thing different that stands >>>> out ATM is the domains being different. >>>>>>> Wondering if anyone ran into a similar experience and could share >>>>>>> their >>>> solution? Do I need to add the realm to the /etc/krb5.conf config >>>> on the NFS Ganesha servers perhaps? >>>>>> >>>>>> So, I believe that the clients and servers need to be in the same >>>>>> KRB5 domain. >>>> This would likely be 123.xyz in your case, rather than using the >>>> sub-domains. >>>>> >>>>> K. >>>>> >>>>> Will have to add the realm section etc then. Need the one nfs >>>>> server shared >>>> between two kdc ‘s . I have a separate kdc per subdomain. ABC.xyz >>>> is the ad dc in this case. >>>> >>>> Apologies, I misread I think. Thinking you mean there's only one >>>> domain configuration option in the NFS Ganesha config hence why NFS >>>> Ganesha only supports a single domain? >>>> >>>> /etc/ganesha/ganesha.conf >>>> NFSv4 { >>>> Lease_Lifetime = 20 ; >>>> IdmapConf = "/etc/idmapd.conf" ; >>>> DomainName = "b.abc.123" ; >>>> } >>>> NFS_KRB5 { >>>> PrincipalName = &quot;nfs/nfs01.b.abc.123(a)B.ABC.123&quot; ; >>>> KeytabPath = /etc/krb5.keytab ; >>>> Active_krb5 = YES ; >>>> } >>>> >>>> >>>> # cat /etc/idmapd.conf >>>> [General] >>>> Domain = b.abc.123 >>>> [Mapping] >>>> [Translation] >>>> [Static] >>>> [UMICH_SCHEMA] >>>> LDAP_server = ldap-server.local.domain.edu LDAP_base = >>>> dc=local,dc=domain,dc=edu You have new mail in /var/spool/mail/root >>>> >>>> The user is the same account on the second subdomain but the trust >>>> with the AD DC through which it's served is formed with a separate >>>> Linux KDC on that domain. >>>> >>>> Perhaps this diagram will come through correctly >>>> >>>> sam(a)abc.123 <-- IPA KDC 1 (a.abc.123) -> AD DC (abc.123) >>>> sam(a)abc.123 <-- IPA KDC 2 (b.abc.123) -----^ >>>> >>>> The NFS cluster is shared between IPA KDC 1 and 2 . >>>> >>>> So in this case, if I wanted this scenario to work, guessing I may >>>> have to create a separate NFS cluster and extend the Gluster FS over >>>> 3 new additional nodes on domain a.abc.123 to ensure the same user's >>>> files and storage is always in sync across domain / kerberos realm. >>>> >>>> Reason for all this is that we're testing options for isolating >>>> certain software / projects from other domains (keeping them >>>> contained within a specific realm). >>>> The software can modify the KDC extensively and hence why we prefer >>>> it has it's own KDC. >>> >>> Yes, I think you pretty much need two separate NFS server clusters. >>> >>> If you only use NFS v4.x, you might be able to have them run on the >>> same nodes by assigning one to a different port, but then the clients >>> need to specify the port to connect to so they get the right KDC. >>> >>> That Domain parameter in the NFSv4 config block I think only gets >>> used if you don't use idmapd. >>> >>> I'm not sure how or if idmapd works with multiple domains (Ganesha >>> just passes the domain through, everything is configured using >>> idmapd.conf). But I don't think you need multiple idmap domains. >>> >>> So I think the issue is just going to come down to the server only >>> has a single Kerberos service principal specified by PrincipleName in >>> the >>> NFS_KRB5 config block. That principal of course belongs to a single >>> Kerberos realm. >>> >>> Frank >>> >> >> Note, you don't need separate Gluster clusters. There's no >> requirement that a Ganesha instance run on the same server as one of >> the bricks, so you can just add a second Ganesha instance pointed at >> the same Gluster cluster. >> >> Daniel >> _______________________________________________ >> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe >> send an email to support-leave(a)lists.nfs-ganesha.org > > Manged to get this working with a single implementation of NFS Ganesha and . > > Linux KDC on domain a.abc.xyz and Linux KDC on domain b.abc.xyz have a trust > to the _same_ upstream AD DC. This means that sam(a)abc.xyz has the same SID > on both because it's the same user from the same AD DC presented via a > separate trust to each of the Linux KDC's. > > Because the SID is the same, SSSD generates the same UID / GID. > > Because the same UID / GID is presented to hosts under the two different > domains, the same NFS folder is accessible on NFS Ganesha mounted shares > despite the configured domain being that of a.abc.xyz on the NFS Ganesha > Servers. And so this is correct because that means the same user, and only that > user, has access to their folder. > > For users defined individually on the two Linux KDC's (ie not in AD through which > these two KDC's have a trust with) then separate folders will exist on NFS > Ganesha and neither user would be able to see the other's files since each would > exist on a different domain ( bob(a)a.abc.xyz UID / GID != bob(a)b.abc.xyz UID / > GID ) and be given different UID / GID numbers. > > This effectively resolved the issue here. Great. If you want to write something up that would help others, I would be happy to post it to the wiki so others might benefit. Frank

_______________________________________________ Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org