Add Multi-init/finalize Support

content/backmatter.tex

@@ -143,12 +143,6 @@ \chapter{Undefined Behavior in OpenSHMEM}\label{sec:undefined}
 immediately upon an \openshmem call into the uninitialized library.
 \tabularnewline
 \hline
-Multiple calls to initialization routines & In an \openshmem program where
-the initialization routines \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread}
-have already been called, any subsequent calls to these initialization routines
-result in undefined behavior.
-\tabularnewline
-\hline
 Specifying invalid \ac{PE} numbers & For \openshmem routines that accept a
 \ac{PE} number as an argument, if the \ac{PE} number is invalid for the
 team associated with the operation (either implicitly or explicitly), the
@@ -661,6 +655,11 @@ \section{Version 1.6}
 The following list describes the specific changes in \openshmem[1.6]:
 \begin{itemize}
 %
+\item Added support for initialization and finalization routines to be called
+    multiple times, and added an initialization status query API
+    \FUNC{shmem\_query\_initialized}.
+\ChangelogRef{subsec:shmem_init, subsec:shmem_finalize, subsec:shmem_query_initialized}%
+%
 \item Added interleaved block transfer APIs \FUNC{shmem\_ibget} and
     \FUNC{shmem\_ibput}.
 \ChangelogRef{subsec:shmem_ibget, subsec:shmem_ibput}%

content/execution_model.tex

@@ -8,17 +8,15 @@
 
 \ac{PE} execution is loosely coupled, relying on \openshmem operations to
 communicate and synchronize among executing \acp{PE}.  The \openshmem phase in
-a program begins with a call to the initialization routine \FUNC{shmem\_init}
+a program begins with the first call to the initialization routine \FUNC{shmem\_init}
 or \FUNC{shmem\_init\_thread}, which must be performed before using any of the
 other \openshmem library routines. 
-An \openshmem program concludes its use of the \openshmem library when all \acp{PE} call
+An \openshmem program concludes its use of the \openshmem library when all \acp{PE}
+make their final call to
 \FUNC{shmem\_finalize} or any \ac{PE} calls \FUNC{shmem\_global\_exit}.
-During a call to \FUNC{shmem\_finalize}, the \openshmem library must
-complete all pending communication and release all the resources associated to
-the library using an implicit collective synchronization across \acp{PE}.
-Calling any \openshmem routine before initialization or after
-\FUNC{shmem\_finalize} leads to undefined behavior. After finalization, a
-subsequent initialization call also leads to undefined behavior.
+During the last call to \FUNC{shmem\_finalize}, the \openshmem library synchronizes
+all \acp{PE}, completes all pending communication and releases all the resources
+associated to the library.
 
 The \acp{PE} of the \openshmem program are identified by unique integers.  The
 identifiers are integers assigned in a monotonically increasing manner from zero

content/shmem_finalize.tex

@@ -15,23 +15,33 @@
 \end{apiarguments}
 
 \apidescription{
-    \FUNC{shmem\_finalize} is a collective operation that ends the \openshmem
-    portion of a program previously initialized by \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread} and
-    releases all resources used by the \openshmem library. This collective
-    operation requires all \acp{PE} to participate in the call. There is an
-    implicit global barrier in \FUNC{shmem\_finalize} to ensure that pending
-    communications are completed and that no resources are released until all
-    \acp{PE} have entered \FUNC{shmem\_finalize}.
-    This routine destroys all teams created by the \openshmem program.
+    \FUNC{shmem\_finalize} ends the \openshmem
+    portion of a program previously initialized by \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread}.
+   This is a collective
+    operation that requires all \acp{PE} to participate in the call.
+
+    An \openshmem program may perform a series of matching
+    initialization and finalization calls.
+    The last call to \FUNC{shmem\_finalize} in this series
+    releases all resources used by the \openshmem library. 
+    This call destroys all teams created by the \openshmem program.
     As a result, all shareable contexts are destroyed.
     The user is
     responsible for destroying all contexts with the
-    \CONST{SHMEM\_CTX\_PRIVATE} option enabled prior to calling this routine;
+    \CONST{SHMEM\_CTX\_PRIVATE} option enabled prior to this call;
     otherwise, the behavior is undefined.
-    \FUNC{shmem\_finalize} must be
-    the last \openshmem library call encountered in the \openshmem portion of a
-    program. A call to \FUNC{shmem\_finalize} will release all resources
-    initialized by a corresponding call to \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread}. All processes
+
+    The last call to \FUNC{shmem\_finalize} performs an implicit global barrier
+    to ensure that pending communications are completed and that no resources
+    are released until all \acp{PE} have entered \FUNC{shmem\_finalize}. All
+    other calls to \FUNC{shmem\_finalize} perform an operation semantically
+    equivalent to \FUNC{shmem\_barrier\_all} and return without freeing any
+    \openshmem resources.
+
+    The last call to \FUNC{shmem\_finalize} causes the \openshmem library
+    to enter an uninitialized state. No further \openshmem calls may be
+    made until an \openshmem initialization routine is called.
+    All processes
     that represent the \acp{PE} will still exist after the
     call to \FUNC{shmem\_finalize} returns, but they will no longer have access
     to resources that have been released.
@@ -42,12 +52,19 @@
 }
 
 \apinotes{
-    \FUNC{shmem\_finalize} releases all resources used by the \openshmem library
+    The last call to \FUNC{shmem\_finalize} releases all resources used by the \openshmem library
     including the symmetric memory heap and pointers initiated by
     \FUNC{shmem\_ptr}. This collective operation requires all \acp{PE} to
     participate in the call, not just a subset of the \acp{PE}. The
     non-\openshmem portion of a program may continue after a call to
     \FUNC{shmem\_finalize} by all \acp{PE}.
+
+    Calls to \FUNC{shmem\_finalize} that are not the last in a series of
+    initialization and finalization calls do not free any \openshmem resources.
+    Thus, teams, contexts, or symmetric memory allocations may be leaked until
+    the final call to \FUNC{shmem\_finalize}. Applications that perform
+    multiple initialization and finalization calls should free resources prior
+    to calling \FUNC{shmem\_finalize} to avoid such leaks.
 }
 
 \begin{apiexamples}

content/shmem_init.tex

@@ -18,9 +18,15 @@
     library. It is a collective operation that all \acp{PE} must call before any
     other \openshmem routine may be called. At the end of the \openshmem program
     which it initialized, the call to \FUNC{shmem\_init} must be matched with a
-    call to \FUNC{shmem\_finalize}. After the first call to \FUNC{shmem\_init}, a
-    subsequent call to \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread} in the
-    same program results in undefined behavior.
+    call to \FUNC{shmem\_finalize}.
+
+    The \FUNC{shmem\_init} and \FUNC{shmem\_init\_thread} initialization
+    routines may be called multiple times within an \openshmem program. A
+    corresponding call to \FUNC{shmem\_finalize} must be made for each call to
+    an \openshmem initialization routine. The \openshmem library must not be
+    finalized until after the last call to \FUNC{shmem\_finalize} and may be
+    re-initialized with a subsequent call to an initialization routine.
+
 }
 
 \apireturnvalues{

content/shmem_init_thread.tex

@@ -24,13 +24,19 @@
 \VAR{requested} are \CONST{SHMEM\_THREAD\_SINGLE}, \CONST{SHMEM\_THREAD\_FUNNELED},
 \CONST{SHMEM\_THREAD\_SERIALIZED}, and \CONST{SHMEM\_THREAD\_MULTIPLE}.
 
-An \openshmem program is initialized either by \FUNC{shmem\_init} or \FUNC{shmem\_init\_thread}.
-Once an \openshmem library initialization call has been performed, a subsequent
-initialization call in the same program results in undefined behavior.
+The \FUNC{shmem\_init} and \FUNC{shmem\_init\_thread} initialization
+routines may be called multiple times within an \openshmem program. A
+corresponding call to \FUNC{shmem\_finalize} must be made for each call to
+an \openshmem initialization routine. The \openshmem library must not be
+finalized until after the last call to \FUNC{shmem\_finalize} and may be
+re-initialized with a subsequent call to an initialization routine.
+
 If the call to \FUNC{shmem\_init\_thread}
 is unsuccessful in allocating and initializing resources for the 
 \openshmem library, then the behavior of any subsequent call 
 to the \openshmem library is undefined.
+
+
 }
 
 \apireturnvalues{
@@ -43,6 +49,9 @@
 or \FUNC{shmem\_init\_thread}. If the \openshmem library is initialized 
 by \FUNC{shmem\_init}, the library implementation can choose to 
 support any one of the defined thread levels.
+
+The \openshmem library may not be able to change the level of threading support
+provided after the first initialization call has been made.
 }
 
 \end{apidefinition}

content/shmem_query_initialized.tex

@@ -0,0 +1,30 @@
+\apisummary{
+Returns the initialized status of the \openshmem library.
+}
+
+\begin{apidefinition}
+
+\begin{Csynopsis}
+void @\FuncDecl{shmem\_query\_initialized}@(int *initialized);
+\end{Csynopsis}
+
+\begin{apiarguments}
+\apiargument{OUT}{initialized}{Nonzero if the \openshmem library is in the initialized state. Zero otherwise.}
+\end{apiarguments}
+
+\apidescription{
+    The \FUNC{shmem\_query\_initialized} call returns the initialization status
+    of the \openshmem library. If the application has called an \openshmem
+    initialization routine and has not yet made the corresponding call to
+    \FUNC{shmem\_finalize}, this routine returns nonzero. Otherwise, it returns
+    zero.
+
+    This function may be called at any time, regardless of the thread safety
+    level of the \openshmem library.
+}
+
+\apireturnvalues{
+None.
+}
+
+\end{apidefinition}

content/shmem_query_thread.tex

@@ -19,6 +19,9 @@
 initialized by \FUNC{shmem\_init\_thread}. If the library was initialized by
 \FUNC{shmem\_init}, the implementation can choose to provide any one of the defined
 thread levels, and \FUNC{shmem\_query\_thread} returns this thread level.
+
+This function may be called at any time, regardless of the thread safety
+level of the \openshmem library.
 }
 
 \apireturnvalues{

main_spec.tex

@@ -92,6 +92,10 @@ \subsubsection{\textbf{SHMEM\_QUERY\_THREAD}}
 \label{subsec:shmem_query_thread}
 \input{content/shmem_query_thread}
 
+\subsubsection{\textbf{SHMEM\_QUERY\_INITIALIZED}}
+\label{subsec:shmem_query_initialized}
+\input{content/shmem_query_initialized}
+
 
 \subsection{Memory Management Routines}
 \label{sec:memory_management}