o ?e"[@sdZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddl mZdd lmZd d Zd d ZedgdGdddZddZGdddZdS)z)hasattrr type)objrk/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/training/session_manager.py _maybe_names    rcCs(td}|r |||||dS)z@Restores checkpoint values and SavedModel initializers if found.Zsaved_model_initializersN)rZget_collectionrunrestore)sesssaverpathZsaved_model_init_opsrrr:_restore_checkpoint_and_maybe_run_saved_model_initializers-s  rztrain.SessionManager)v1c@seZdZdZ       dddZ      ddd Z         dd d Z      dd d ZdedfddZ ddZ ddZ ddZ ddZ dS)SessionManageraTraining helper that restores from checkpoint and creates session. This class is a small wrapper that takes care of session creation and checkpoint recovery. It also provides functions that to facilitate coordination among multiple training threads or processes. * Checkpointing trained variables as the training progresses. * Initializing variables on startup, restoring them from the most recent checkpoint after a crash, or wait for checkpoints to become available. ### Usage: ```python with tf.Graph().as_default(): ...add operations to the graph... # Create a SessionManager that will checkpoint the model in '/tmp/mydir'. sm = SessionManager() sess = sm.prepare_session(master, init_op, saver, checkpoint_dir) # Use the session to train the graph. while True: sess.run() ``` `prepare_session()` initializes or restores a model. It requires `init_op` and `saver` as an argument. A second process could wait for the model to be ready by doing the following: ```python with tf.Graph().as_default(): ...add operations to the graph... # Create a SessionManager that will wait for the model to become ready. sm = SessionManager() sess = sm.wait_for_session(master) # Use the session to train the graph. while True: sess.run() ``` `wait_for_session()` waits for a model to be initialized by other processes. NcCsd|durt}||_||_||_||_||_d|_||_||_ |dur.|dur0t d|dSdS)a_Creates a SessionManager. The `local_init_op` is an `Operation` that is run always after a new session was created. If `None`, this step is skipped. The `ready_op` is an `Operation` used to check if the model is ready. The model is considered ready if that operation returns an empty 1D string tensor. If the operation returns a non empty 1D string tensor, the elements are concatenated and used to indicate to the user why the model is not ready. The `ready_for_local_init_op` is an `Operation` used to check if the model is ready to run local_init_op. The model is considered ready if that operation returns an empty 1D string tensor. If the operation returns a non empty 1D string tensor, the elements are concatenated and used to indicate to the user why the model is not ready. If `ready_op` is `None`, the model is not checked for readiness. `recovery_wait_secs` is the number of seconds between checks that the model is ready. It is used by processes to wait for a model to be initialized or restored. Defaults to 30 seconds. Args: local_init_op: An `Operation` run immediately after session creation. Usually used to initialize tables and local variables. ready_op: An `Operation` to check if the model is initialized. ready_for_local_init_op: An `Operation` to check if the model is ready to run local_init_op. graph: The `Graph` that the model will use. recovery_wait_secs: Seconds between checks for the model to be ready. local_init_run_options: RunOptions to be passed to session.run when executing the local_init_op. local_init_feed_dict: Optional session feed dictionary to use when running the local_init_op. Raises: ValueError: If ready_for_local_init_op is not None but local_init_op is None NzgIf you pass a ready_for_local_init_op you must also pass a local_init_op , ready_for_local_init_op [%s]) rZget_default_graph_local_init_op _ready_op_ready_for_local_init_op_graph_recovery_wait_secs_target_local_init_run_options_local_init_feed_dict ValueError)selfZ local_init_opZready_opZready_for_local_init_opgraphZrecovery_wait_secsZlocal_init_run_optionsZlocal_init_feed_dictrrr__init__rs1zSessionManager.__init__F c Cs||_t}|rt|jdr|jtj|j|j|d} |r&|r&t d|r,|s0|s0| dfS|r             z"SessionManager._restore_checkpointc  Cs|j|||||||d\} } | s0|dur| s|jdurtd|dur*| j|| d| r0| | || \} }| sDtdt|| |f|| \}}|sZtdt|| |j|f| S)a> Creates a `Session`. Makes sure the model is ready to be used. Creates a `Session` on 'master'. If a `saver` object is passed in, and `checkpoint_dir` points to a directory containing valid checkpoint files, then it will try to recover the model from checkpoint. If no checkpoint files are available, and `wait_for_checkpoint` is `True`, then the process would check every `recovery_wait_secs`, up to `max_wait_secs`, for recovery to succeed. If the model cannot be recovered successfully then it is initialized by running the `init_op` and calling `init_fn` if they are provided. The `local_init_op` is also run after init_op and init_fn, regardless of whether the model was recovered successfully, but only if `ready_for_local_init_op` passes. If the model is recovered from a checkpoint it is assumed that all global variables have been initialized, in particular neither `init_op` nor `init_fn` will be executed. It is an error if the model cannot be recovered and no `init_op` or `init_fn` or `local_init_op` are passed. Args: master: `String` representation of the TensorFlow master to use. init_op: Optional `Operation` used to initialize the model. saver: A `Saver` object used to restore a model. checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the dir will be used to restore. checkpoint_filename_with_path: Full file name path to the checkpoint file. wait_for_checkpoint: Whether to wait for checkpoint to become available. max_wait_secs: Maximum time to wait for checkpoints to become available. config: Optional `ConfigProto` proto used to configure the session. init_feed_dict: Optional dictionary that maps `Tensor` objects to feed values. This feed dictionary is passed to the session `run()` call when running the init op. init_fn: Optional callable used to initialize the model. Called after the optional `init_op` is called. The callable must accept one argument, the session being initialized. Returns: A `Session` object that can be used to drive the model. Raises: RuntimeError: If the model cannot be initialized or recovered. ValueError: If both checkpoint_dir and checkpoint_filename_with_path are set. r1r2r3r4r)NzMModel is not initialized and no init_op or init_fn or local_init_op was given) feed_dictz]Init operations did not make model ready for local_init. Init op: %s, init fn: %s, error: %szaInit operations did not make model ready. Init op: %s, init fn: %s, local_init_op: %s, error: %s)r5r RuntimeErrorr_try_run_local_init_opr _model_ready)r#r0Zinit_oprr1r2r3r4r)Zinit_feed_dictZinit_fnris_loaded_from_checkpointlocal_init_successmsgis_readyrrrprepare_sessionsD; zSessionManager.prepare_sessionc Cs|j|||||||d\}} ||\} } | s|dfS|p|} | s,td| | |dfS||\} } | s@td| | |dfStd| || fS)aCreates a `Session`, recovering if possible. Creates a new session on 'master'. If the session is not initialized and can be recovered from a checkpoint, recover it. Args: master: `String` representation of the TensorFlow master to use. saver: A `Saver` object used to restore a model. checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the dir will be used to restore. checkpoint_filename_with_path: Full file name path to the checkpoint file. wait_for_checkpoint: Whether to wait for checkpoint to become available. max_wait_secs: Maximum time to wait for checkpoints to become available. config: Optional `ConfigProto` proto used to configure the session. Returns: A pair (sess, initialized) where 'initialized' is `True` if the session could be recovered and initialized, `False` otherwise. Raises: ValueError: If both checkpoint_dir and checkpoint_filename_with_path are set. r6FzCRestoring model from %s did not make model ready for local init: %sz4Restoring model from %s did not make model ready: %szRestored model from %s)r5r9r,r-r:)r#r0rr1r2r3r4r)rr;r<r=Zrestoring_filer>rrrrecover_sessionVs6    zSessionManager.recover_sessionInfc Cs||_|dur td}t|} tj|j|j|d}d}d}||\}}|r2||\} }| r2|S||| |j } | dkrLt ddd|ft d||t|j q)aOCreates a new `Session` and waits for model to be ready. Creates a new `Session` on 'master'. Waits for the model to be initialized or recovered from a checkpoint. It's expected that another thread or process will make the model ready, and that this is intended to be used by threads/processes that participate in a distributed training configuration where a different thread/process is responsible for initializing or recovering the model being trained. NB: The amount of time this method waits for the session is bounded by max_wait_secs. By default, this function will wait indefinitely. Args: master: `String` representation of the TensorFlow master to use. config: Optional ConfigProto proto used to configure the session. max_wait_secs: Maximum time to wait for the session to become available. Returns: A `Session`. May be None if the operation exceeds the timeout specified by config.operation_timeout_in_ms. Raises: tf.DeadlineExceededError: if the session is not available after max_wait_secs. NrATr(rz,Session was not ready after waiting %d secs.zGWaiting for model to be ready. Ready_for_local_init_op: %s, ready: %s)rfloat_CountDownTimerrr+rr9r: _safe_closesecs_remainingrrZDeadlineExceededErrorr,r-r.r/) r#r0r)r4timerrZ not_ready_msgZnot_ready_local_msgr<r>Zremaining_ms_after_waitrrrwait_for_sessions8   zSessionManager.wait_for_sessioncCs$z|WdStyYdSw)zCloses a session without raising an exception. Just like sess.close() but ignores exceptions. Args: sess: A `Session`. N)close Exceptionr#rrrrrDs   zSessionManager._safe_closecCt|j|dS)aChecks if the model is ready or not. Args: sess: A `Session`. Returns: A tuple (is_ready, msg), where is_ready is True if ready and False otherwise, and msg is `None` if the model is ready, a `String` with the reason why it is not ready otherwise. zModel not ready)_readyrrJrrrr:s zSessionManager._model_readycCrK)a\Checks if the model is ready to run local_init_op. Args: sess: A `Session`. Returns: A tuple (is_ready, msg), where is_ready is True if ready to run local_init_op and False otherwise, and msg is `None` if the model is ready to run local_init_op, a `String` with the reason why it is not ready otherwise. zModel not ready for local init)rLrrJrrr_model_ready_for_local_inits z*SessionManager._model_ready_for_local_initcCsV|jdur)||\}}|r%td|j|j|j|jdtddSd|fSdS)anTries to run _local_init_op, if not None, and is ready for local init. Args: sess: A `Session`. Returns: A tuple (is_successful, msg), where is_successful is True if _local_init_op is None, or we ran _local_init_op, and False otherwise; and msg is a `String` with the reason why the model was not ready to run local init. NzRunning local_init_op.)r7optionszDone running local_init_op.TNF)rrMr,r-rr!r )r#rZis_ready_for_local_initr=rrrr9s    z%SessionManager._try_run_local_init_op)NNNNrNN)NNNFr&N) NNNNFr&NNN)__name__ __module__ __qualname____doc__r%r5r?r@rBrGrDr:rMr9rrrrrEsN, C I ^ @; rc Cs|durdSz(||}|dus|jtjks|jdkrWdSddd|D}dd|fWStjyX}zd t|vrHt d |t||dt|fWYd}~Sd}~ww) aChecks if the model is ready or not, as determined by op. Args: op: An op, either _ready_op or _ready_for_local_init_op, which defines the readiness of the model. sess: A `Session`. msg: A message to log to warning if not ready Returns: A tuple (is_ready, msg), where is_ready is True if ready and False otherwise, and msg is `None` if the model is ready, a `String` with the reason why it is not ready otherwise. NrOrz, cSsg|]}|dqS)zutf-8)decode).0irrr 6sz_ready..FzVariables not initialized: Z uninitializedz%s : error [%s]) rZdtypenpZint32sizejoinrZFailedPreconditionErrorstrr,warning)oprr=Z ready_valueZnon_initialized_varnameserrrrLs&   rLc@s(eZdZdZddgZddZddZdS) rCz.A timer that tracks a duration since creation._start_time_secs_duration_secscCst|_||_dS)N)r.r_r`)r#Z duration_secsrrrr%Ds  z_CountDownTimer.__init__cCs|jt|j}td|S)Nr)r`r.r_max)r#diffrrrrEHs z_CountDownTimer.secs_remainingN)rPrQrRrS __slots__r%rErrrrrC?s  rC)rSr.numpyrXZtensorflow.python.checkpointrZtensorflow.python.clientrZtensorflow.python.distributerZtensorflow.python.frameworkrrZtensorflow.python.platformrr,Z tensorflow.python.util.tf_exportrrrrrLrCrrrrs&        U'