Sharing flamegraphs captured on our systems, with both pthread_rwlock and after converting to pthread_mutex.


Regards,
Gaurav Gangalwar

On Fri, Apr 5, 2024 at 2:40 PM gaurav gangalwar <gaurav.gangalwar@gmail.com> wrote:
Hello,
Need some thoughts on the use of pthread_rwlock in Ganesha with zfs as backend filesystem on el8
We see regression with specfs swbuild workload on el8 with same Ganesha code that was performing better on el7.
To triage the issue we captured perf stats and using flamegraph found that Ganesha is spending a lot of cpu cycles in pthread_rwlock on el8 as compared to el7.
If we convert the pthread_rwlock that shows up in perf stats to pthread_mutex, we no longer see this issue.
Locks that we converted to pthread_mutex are fsm_lock, attr_lock, obj_lock, cih_lock and mdc_exp_lock

This issue gets more severe as the number of threads grows in Ganesha, more threads, higher CPU utilization and impact on IOPs.

Has anyone faced this issue on el8?

This is patch for reference that is working for us




Regards,
Gaurav Gangalwar