o ?e@sdZddlZddlmZddlmZddlmZddlmZddlmZddl m Z dd l m Z d d Zd d ZddZddZdS)z&Optional helper for gradient handling.N)dtypes)ops) array_ops)variable_scope) variables) tf_logging)tpu_opscCs:|}|||}dd|D}tt||}|S)aCompute gradients to send to TPU embedding. Args: optimizer: a subclass of optimizer.Optimizer, usually CrossShardOptimizer. Used to call compute_gradients(). loss: a Tensor to call optimizer.compute_gradients() on. activations: an OrderedDict mapping feature_name to Tensors of activations. Returns: An OrderedDict mapping from feature name Strings to Tensors of gradients of the loss wrt the activations of the features. cSsg|]\}}|qSr ).0Zgrad_r r m/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/tpu/tpu_embedding_gradient.py +sz;get_gradients_through_compute_gradients..)valuesZcompute_gradients collections OrderedDictzipkeys)Z optimizerZloss activationsZactivation_listZgrads_and_varsZgradsfeature_to_gradient_dictr r r 'get_gradients_through_compute_gradientss  rc Cst}t|jD]?\}}tjd|tjdgdddgd||<t }| d|}|r5t d|t |j|}|dd t|Dq |tj|d d fS) aCreate dummy embedding table variables. The sole purpose of these dummy variables are to trigger gradient calculation wrt them so that the gradients wrt activation can be captured and later sent to TPU embedding. Args: tpu_embedding: TPUEmbedding, dummy table variables will be created for use with tpu_embedding. Returns: A tuple of dummy variables and their initializer. Raises: RuntimeError: if collection to store gradients already exists and is not empty. z%tpu_embedding_dummy_table_variable_{}TZ#tpu_embedding_dummy_table_variables)ZdtypeshapeZ use_resourceZ trainabler tpu_embedding_gradients_table_{}z.tpu_embedding_gradients_table_{} is not empty.cSsg|]}dqSNr )r r r r r r [sz0create_dummy_table_variables..Z(tpu_embedding_dummy_table_variables_init)name)rr enumeratetable_to_features_dictrZ get_variableformatrZfloat32rget_default_graphZget_collection_ref RuntimeErrorlenextendrangerZvariables_initializerr) tpu_embeddingdummy_table_variablestable_idtablegtable_gradientsZ num_featuresr r r create_dummy_table_variables1s6r)cCsXt}|D]#}|j|j}tj||||t|j||j ||d||<q|S)aHave activations depend on dummy table variables for gradient intercept. Args: tpu_embedding: TPUEmbedding, activations and dummy_table_variables are from tpu_embedding. activations: An OrderedDict of feature name String to activation tensors. dummy_table_variables: An OrderedDict of table name String to dummy table variables. Returns: An OrderedDict of feature name String to activation tensors, which can be used just as the activations input. )r%Z lookup_id) rrfeature_to_config_dictr%rZtpu_embedding_activationslisttable_to_config_dictindexr)r#rr$Znew_activationsfeaturer&r r r )hook_dummy_table_variables_to_activationscs  r/c Cst}d}t|jD])\}}|d|}tdd|Dr)td|||p3tdd|D}q |ssz>get_gradients_through_dummy_table_variables..aTable {} with id {} has undefined gradients: this is probably because the model asked TPUEmbedding to compute activations that were not used, or tf.stop_gradient() is applied. Gradients of zeros are sent back to TPUEmbedding instead. Gradients of zeros and no gradients are equivalent for SGD, AdaGrad, FTRL, etc, but might differ for other optimizers due to implementation of TPU embedding optimizers.css|]}|duVqdSrr r0r r r r2s zAll tables have undefined gradients: this is probably because the model asked TPUEmbedding to compute activations that were not used. If all TPUEmbedding features have stop_gradients, consider using the INFERENCE mode instead.N)rrrr,Zget_collectionranyloggingwarnrrrr dimensionZbatch_size_per_corer*max_sequence_lengthrZzeros) r#r'Zgradients_foundr%r&r(rr.r1r6Z batch_sizer7r r r +get_gradients_through_dummy_table_variables}sR          r8)__doc__rZtensorflow.python.frameworkrrZtensorflow.python.opsrrrZtensorflow.python.platformrr4Ztensorflow.python.tpu.opsrrr)r/r8r r r r s       2