o ®Ÿ?e‘ ã@spdZddlZddlZddlZddlZddlmZddlm Zddl m Z Gdd„de j ƒZ Gdd „d ejƒZdS) z@Implementation of the SessionRunHook for preemptible Cloud TPUs.éN)Útpu_cluster_resolver)Ú tf_logging)Úsession_run_hookc@s(eZdZdZdd„Zdd„Zdd„ZdS) ÚCloudTPUPreemptedHookaThe SessionRunHook for preemptible Cloud TPUs. This is an implementation of SessionRunHook for the pre-emptible Google Cloud TPU service. It attempts to close the session if the TPU is preempted, and exits the coordinator process if the session cannot be closed. cCs ||_dS©N)Ú_cluster)ÚselfÚcluster©r úe/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/tpu/preempted_hook.pyÚ__init__#s zCloudTPUPreemptedHook.__init__cCs(t ¡rt|j|ƒ|_|j ¡dSdSr)rÚis_running_in_gceÚ_TPUPollingThreadrÚ _tpu_pollerÚstart)rÚsessionZcoordr r r Úafter_create_session&sþz*CloudTPUPreemptedHook.after_create_sessioncCs|j ¡dSr)rÚstop)rrr r r Úend+szCloudTPUPreemptedHook.endN)Ú__name__Ú __module__Ú __qualname__Ú__doc__r rrr r r r rs  rcs0eZdZdZ‡fdd„Zdd„Zdd„Z‡ZS)ra%A thread that polls the state of a TPU node. When the node transitions into a TERMINAL state (PREEMPTED, TERMINATED) that's considered as not recoverable by the underlying infrastructure, it attempts to close the session, and exits the entire process if the session.close() stucks. csRtt|ƒ ¡d|_d|_d|_||_||_d|_dD] }t   |¡  t j ¡qdS)NTFé)zgoogleapiclient.discoveryzoauth2client.client) Úsuperrr ÚdaemonÚ_runningÚ_session_closedrÚ_sessionÚ _intervalÚ_loggingÚ getLoggerÚsetLevelÚWARNING)rr rÚname©Ú __class__r r r 8sÿz_TPUPollingThread.__init__cCsd|_d|_| ¡dS)NFT)rrÚjoin)rr r r rFs z_TPUPollingThread.stopcCspt ¡st d¡d|_dS|jr6|jj ¡}|s+t d|jj|jj  ¡¡t   d¡t   |j¡|jsdSdS)Nz@TPUPollingThread is running in a non-GCE environment, exiting...Fz)TPUPollingThread found TPU %s in state %sé)rr ÚloggingÚwarningrrZ_cloud_tpu_clientÚ recoverableZ_tpuÚstateÚosÚ_exitÚtimeÚsleepr)rr+r r r ÚrunKs ÿ þ  ùz_TPUPollingThread.run)rrrrr rr1Ú __classcell__r r r%r r/s  r)rr)r r-Ú threadingr/Z-tensorflow.python.distribute.cluster_resolverrZtensorflow.python.platformrZtensorflow.python.trainingrZSessionRunHookrÚThreadrr r r r Ús