fix deadlock in async thread self-cancellation

with async cancellation enabled, pthread_cancel(pthread_self())
deadlocked due to pthread_kill holding killlock which is needed by
pthread_exit.

this could be solved by making pthread_kill block signals around the
critical section, at least when the target thread is itself, but the
issue only arises for cancellation, and otherwise would just be
imposing unnecessary cost.

instead just have pthread_cancel explicitly check for async
self-cancellation and call pthread_exit(PTHREAD_CANCELED) directly
rather than going through the signal machinery.

diff --git a/src/thread/pthread_cancel.c b/src/thread/pthread_cancel.c
index 3d229223ed1b00f622c3319b7cfa327086f530d7..bf07dbebd691778f25e16d824ecb92d7850518f6 100644 (file)
--- a/src/thread/pthread_cancel.c
+++ b/src/thread/pthread_cancel.c
@@ -92,6 +92,10 @@ int pthread_cancel(pthread_t t)
                init = 1;
        }
        a_store(&t->cancel, 1);
-       if (t == pthread_self() && !t->cancelasync) return 0;
+       if (t == pthread_self()) {
+               if (t->canceldisable == PTHREAD_CANCEL_ENABLE && t->cancelasync)
+                       pthread_exit(PTHREAD_CANCELED);
+               return 0;
+       }
        return pthread_kill(t, SIGCANCEL);
 }