o ?ew>@sdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ejjZejjZejjZejjZdd Zd d Zd d ZddZddZddZddZddZddZdS)z?Training utilities for Estimator to use Distribute Coordinator.N)distribute_coordinator)distribute_coordinator_context)multi_worker_util) tf_logging) server_libcCs |stdt|tgS)z7Counts the number of parameter servers in cluster_spec.z?Internal error: `_count_ps` does not expect empty cluster_spec.) RuntimeErrorlenas_dictgetPS) cluster_specr p/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/distribute/estimator_training.py _count_ps$s rcCs4|stdt|tgt||gS)z?Counts the number of workers (including chief) in cluster_spec.zCInternal error: `_count_worker` does not expect empty cluster_spec.)rrr r WORKER)r chief_task_typer r r _count_worker-srcs|sdSg}|jvrg}|fddt|jDt|jvr'|td}|D]}||kr7||S|t||7}q+td||)z:Returns the global id of the given task type in a cluster.rcs g|] }|kr|tkr|qSr )r ).0trr r Bsz"_get_global_id..z=Internal Error: `task_type` ({}) is not in cluster_spec ({}).) jobsextendsortedr appendrZ job_tasksrformat)r task_typetask_idrZtask_type_ordered_listZnext_global_idrr rr_get_global_id7s$    rcCsd|_|j|_|j|_|j|_|j|_|j|_ |j |_ |jrP|jt kr?t |j|_t|jtd|_t|j|j|jtd|_dSti|_d|_d|_d|_dSd|_d|_d|_dS)zDInitializes run config from distribute coordinator's worker context.Nrr)Z_servicer _cluster_specrZ _task_typerZ_task_idZ master_targetZ_evaluation_masterZ_masteris_chiefZ _is_chief EVALUATORrZ_num_ps_replicasrCHIEFZ_num_worker_replicasrZ_global_id_in_clusterr ClusterSpec)configZworker_contextr r r$_init_run_config_from_worker_contextVs4     r&cCs |jr|jjr|jrtd|jj|_|jr&|jjr&|jr!td|jj|_t|di}| i|jrU|jrU|jj rU|rDtdt j j |_|jj |_tddS|r_d|jvs_|jsrd|_| ||tddS|svJ||_t j j|_td dS) z2Initializes RunConfig for distribution strategies.zREither `train_distribute` or`experimental_distribute.train_distribute` can be set.zPEither `eval_distribute` or`experimental_distribute.eval_distribute` can be set.ZclusterzXCannot set both "cluster_spec" of TF_CONFIG and `experimental_distribute.remote_cluster`zLRunConfig initialized for Distribute Coordinator with STANDALONE_CLIENT modeNZmaster!Not using Distribute Coordinator.zMRunConfig initialized for Distribute Coordinator with INDEPENDENT_WORKER mode)Z_experimental_distributetrain_distribute_train_distribute ValueErroreval_distribute_eval_distributerr$r Z._init_distributed_setting_from_environment_varZremote_clusterdcCoordinatorModeSTANDALONE_CLIENT_distribute_coordinator_moder logginginforZ8_maybe_overwrite_session_config_for_distributed_trainingINDEPENDENT_WORKER)r%Z tf_configr r r rinit_run_config{sF          r4cCspt|dr |jdurtddSt|jtjr#|jtjj tjj fvr,t d|jdS|j s6t ddSdS)z?Checks the config to see whether to run distribute coordinator.r0Nr'Fz*Unexpected distribute_coordinator_mode: %rzZRunning `train_and_evaluate` locally, ignoring `experimental_distribute_coordinator_mode`.T) hasattrr0r1r2 isinstancesix string_typesr-r.r/r3warningr )r%r r r!should_run_distribute_coordinators&   r:c s|jjs tdfdd}fdd}jtjjkr*j}|s)Jnd}tj|j|j j|j ddS)aRun distribute coordinator for Estimator's `train_and_evaluate`. Args: estimator: An `Estimator` instance to train and evaluate. train_spec: A `TrainSpec` instance to specify the training specification. eval_spec: A `EvalSpec` instance to specify the evaluation and export specification. executor_cls: the evaluation executor class of Estimator. Raises: ValueError: if `distribute_coordinator_mode` is None in RunConfig. z)strategylocal_estimatorcontextr?) estimator run_config train_specr r _worker_fns&     z&train_and_evaluate.._worker_fncs^t}||j_t|jttdt t |j||_ d|j_ |}| dS)zFunction for evaluator task.r<N)r@rArBr,r&rCrDr1r2rErF_eval_distributionr0Z_start_continuous_evaluation)rJrKexecutor)rM eval_spec executor_clsrOr r_eval_fns    z$train_and_evaluate.._eval_fnNmoder session_config) r%r0r*r-r.r/r run_distribute_coordinatorr(r+rX)rMrOrSrTrPrUr r )rMrSrTrNrOrtrain_and_evaluates, !  rZcjjsJj}jjsJtjj}jjsJd|jvr%tdjjtj j kr1tdjjj j rAtdjjj jfdd}tj|jj|j||jdS)z:Run distribute coordinator for Estimator's `train` method. evaluatorF'evaluator' job is not supported if you don't use `train_and_evaluate`zJOnly `STANDALONE_CLIENT` mode is supported when you call `estimator.train`zL`Estimator.train` API is not supported for %s with `STANDALONE_CLIENT` mode.csbt}||j_t}t|j|tdt t |j||_ |j r'}ng}||||S)r;r<) r@rArBr)rCrDr&r1r2rErFrGr!rJrKrLZ chief_hooksrMr?train_distributed_fnr rrP<s   z#estimator_train.._worker_fnrV)rBr0r rnormalize_cluster_specr)rr*r-r.r/extendedexperimental_between_graph __class____name__rYr(rX)rMr`r?rNr rPr r_restimator_train#s2      rfcr[)z=Run distribute coordinator for Estimator's `evaluate` method.r\r]zMOnly `STANDALONE_CLIENT` mode is supported when you call `Estimator.evaluate`zO`Estimator.evaluate` API is not supported for %s with `STANDALONE_CLIENT` mode.cs^t}||j_t}t|j|tdt t |j||_ |j r'}ng}|||S)zFunction for evaluation.r<) r@rArBr,rCrDr&r1r2rErFrQr!r^rMevaluate_distributed_fnr?r rrPms   z&estimator_evaluate.._worker_fnrV)rBr0r rrar,rr*r-r.r/rbrcrdrerYr+rX)rMrhr?rNr rPr rgrestimator_evaluateTs2      ri)__doc__r@r7Ztensorflow.python.distributerr-rrCrZtensorflow.python.platformrr1Ztensorflow.python.trainingrZ _TaskTyper#r"r rrrrr&r4r:rZrfrir r r rs*       %6\ 1