Clarify finalize resource cleanup and sync

Signed-off-by: James Dinan <[email protected]>

content/execution_model.tex

@@ -8,15 +8,15 @@
 
 \ac{PE} execution is loosely coupled, relying on \openshmem operations to
 communicate and synchronize among executing \acp{PE}.  The \openshmem phase in
-a program begins with a call to the initialization routine \FUNC{shmem\_init}
+a program begins with the first call to the initialization routine \FUNC{shmem\_init}
 or \FUNC{shmem\_init\_thread}, which must be performed before using any of the
 other \openshmem library routines. 
-An \openshmem program concludes its use of the \openshmem library when all \acp{PE} call
+An \openshmem program concludes its use of the \openshmem library when all \acp{PE}
+make their final call call
 \FUNC{shmem\_finalize} or any \ac{PE} calls \FUNC{shmem\_global\_exit}.
-During a call to \FUNC{shmem\_finalize}, the \openshmem library synchronizes
-all \acp{PE} and completes all pending communication. During the last call to
-\FUNC{shmem\_finalize}, it releases all the resources associated to the
-library.
+During the last call to \FUNC{shmem\_finalize}, the \openshmem library synchronizes
+all \acp{PE}, completes all pending communication and releases all the resources
+associated to the library.
 
 The \acp{PE} of the \openshmem program are identified by unique integers.  The
 identifiers are integers assigned in a monotonically increasing manner from zero

content/shmem_finalize.tex

@@ -18,10 +18,7 @@
     \FUNC{shmem\_finalize} ends the \openshmem
     portion of a program previously initialized by \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread}.
    This is a collective
-    operation that requires all \acp{PE} to participate in the call. There is an
-    implicit global barrier in \FUNC{shmem\_finalize} to ensure that pending
-    communications are completed and that no resources are released until all
-    \acp{PE} have entered \FUNC{shmem\_finalize}.
+    operation that requires all \acp{PE} to participate in the call.
 
     An \openshmem program may perform a series of matching
     initialization and finalization calls.
@@ -34,6 +31,13 @@
     \CONST{SHMEM\_CTX\_PRIVATE} option enabled prior to this call;
     otherwise, the behavior is undefined.
 
+    The last call to \FUNC{shmem\_finalize} performs an implicit global barrier
+    to ensure that pending communications are completed and that no resources
+    are released until all \acp{PE} have entered \FUNC{shmem\_finalize}. All
+    other calls to \FUNC{shmem\_finalize} perform an operation semantically
+    equivalent to \FUNC{shmem\_barrier\_all} and return without freeing any
+    \openshmem resources.
+
     The last call to \FUNC{shmem\_finalize} causes the \openshmem library
     to enter an uninitialized state. No further \openshmem calls may be
     made until an \openshmem initialization routine is called.
@@ -48,12 +52,19 @@
 }
 
 \apinotes{
-    \FUNC{shmem\_finalize} releases all resources used by the \openshmem library
+    The last call to \FUNC{shmem\_finalize} releases all resources used by the \openshmem library
     including the symmetric memory heap and pointers initiated by
     \FUNC{shmem\_ptr}. This collective operation requires all \acp{PE} to
     participate in the call, not just a subset of the \acp{PE}. The
     non-\openshmem portion of a program may continue after a call to
     \FUNC{shmem\_finalize} by all \acp{PE}.
+
+    Calls to \FUNC{shmem\_finalize} that are not the last in a series of
+    initialization and finalization calls do not free any \openshmem resources.
+    Thus, teams, contexts, or symmetric memory allocations may be leaked until
+    the final call to \FUNC{shmem\_finalize}. Applications that perform
+    multiple initialization and finalization calls should free resources prior
+    to calling \FUNC{shmem\_finalize} to avoid such leaks.
 }
 
 \begin{apiexamples}