o ?eB" @sdZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl m Zdd l mZdd l mZdd lmZdd lmZdd lmZdZdZdZdZdZdZedGdddedgdZ  d ddZ ddZ!ddZ"ddZ#dS)!z+TPU system metadata and associated tooling.N) config_pb2)session) device_util)contextconfig)device)errors)ops) tf_logging)tpu) tf_exportii Z tpu_workerZ coordinator)localz"tpu.experimental.TPUSystemMetadatacs eZdZdZfddZZS)TPUSystemMetadataaDescribes some metadata about the TPU system. Attributes: num_cores: interger. Total number of TPU cores in the TPU system. num_hosts: interger. Total number of hosts (TPU workers) in the TPU system. num_of_cores_per_host: interger. Number of TPU cores per host (TPU worker). topology: an instance of `tf.tpu.experimental.Topology`, which describes the physical topology of TPU system. devices: a tuple of strings, which describes all the TPU devices in the system. cstt|||||||SN)superr__new__)cls num_cores num_hostsnum_of_cores_per_hosttopologydevices __class__j/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/tpu/tpu_system_metadata.pyr=s zTPUSystemMetadata.__new__)__name__ __module__ __qualname____doc__r __classcell__rrrrr(s  rrrrrrFc CsLd}g}tt}trt}dd|D}nqd} td|z>t .t j |tt|d}|} WdWdWq1sKwYWdn1sZwYWn(tjyd |} |tkrtd | td |t|d7}nt| Ynwq|D]} tj| j} | jd kr|| j| j|d7}qd} |rtd d|D} t | dkrt!d"|| #} d}|r|st!d"||t$||}dd}t%t&||d}t'|t || ||d}|rtdtd|j(td|j)td|j*|j+D] } td| q|Std||S)zOs z._query_tpu_system_metadata..Tz8Querying Tensorflow master (%s) for TPU system metadata.rNzFailed to connect to the Tensorflow master. The TPU worker may not be ready (still scheduling) or the Tensorflow master address is incorrect: got (%s).z%szRetrying (%d/%d).ZTPUcSsg|]}t|qSr)len)r'Zcore_idsrrrr)yszHTPU cores on each host is not same. This should not happen!. devices: {}zCannot find any TPU cores in the system (master address {}). This usually means the master address is incorrect or the TPU worker has some problems. Available devices: {}cSs&tj|j}|j|j|j|j|jfSr) tf_device DeviceSpec from_stringr%jobZreplicataskr& device_index)rspecrrr _sort_keysz-_query_tpu_system_metadata.._sort_key)keyr#zFound TPU system:z*** Num TPU Cores: %dz*** Num TPU Workers: %dz *** Num TPU Cores Per Worker: %dz*** Available Device: %szFailed to find TPU: %s), collections defaultdictlistrZexecuting_eagerlyrZlist_logical_deviceslogginginfor Graph as_defaultr$Sessionget_session_config_with_timeout_PINGING_MASTER_TIMEOUT_IN_MSZ list_devicesr DeadlineExceededError _RETRY_TIMESwarning ValueErrorr,r-r.r%r&r0appendr1setvaluesr+ RuntimeErrorformatpop_obtain_topologytuplesortedrrrrr)master_address cluster_defZquery_topologyZtpu_core_countrZ device_dictZlogical_devicesZ retry_countsessmsgrr2rZnum_cores_per_host_setrr3metadatarrr_query_tpu_system_metadataDs          rQc CszNtd|t7tt|}tj||d}| t }|WdWdWS1s6wYWdWdS1sGwYWdSt j y\td|w)zObtains TPU fabric topology.zfInitializing TPU system (master: %s) to fetch topology for model parallelism. This might take a while.rNzaFail to initialize TPU system with master (%s). Please double check the TPU system is functional.)r8r9r r:r;r=!_INITIAL_TPU_SYSTEM_TIMEOUT_IN_MSr$r<runr Zinitialize_systemr r?rB)rLrMZsession_configrNrrrrrIs2&rIcCstj||d}|S)z>Returns a session given a timeout and a cluster configuration.)Zoperation_timeout_in_msrM)rZ ConfigProto)Ztimeout_in_secsrMZ config_protorrrr=sr=cCs|tvrdS|r |js tStdd|jD}t|vrtdt|dkr+|jdjSt|dkr>t|vr>|t| Std) aReturns the canonical job name to use to place TPU computations on. Args: master: A `string` representing the TensorFlow master to use. cluster_def: A ClusterDef object describing the TPU cluster. Returns: A string containing the job name, or None if no job should be specified. Raises: ValueError: If the user needs to specify a tpu_job_name, because we are unable to infer the job name automatically, or if the user-specified job names are inappropriate. Ncss|]}|jVqdSr)r%)r'r/rrr szmaster_job..z1Currently, tpu_worker is not an allowed job name.r*rzCould not infer TPU job name.) _LOCAL_MASTERSr/_DEFAULT_JOB_NAMErDrBr+r%_DEFAULT_COORDINATOR_JOB_NAMEremoverH)ZmasterrMZ job_namesrrr master_jobs     rZ)NF)$r!r5Ztensorflow.core.protobufrZtensorflow.python.clientrr$Ztensorflow.python.distributerZtensorflow.python.eagerrZtensorflow.python.frameworkrrr,r r Ztensorflow.python.platformr r8Ztensorflow.python.tpur Z tensorflow.python.util.tf_exportr r>r@rRrWrXrV namedtuplerrQrIr=rZrrrrs:             c