o ®Ÿ?eÐ¥ã@sÂdZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl m Z dd l m Z dd l mZe d ¡d d „ƒZe d¡dd„ƒZe d¡dd„ƒZe d¡dd„ƒZe d¡dd„ƒZe d¡dd„ƒZe d¡dd„ƒZe d ¡d!d"„ƒZe d#¡d$d%„ƒZe d&¡d'd(„ƒZe d)¡d*d+„ƒZe d,¡d-d.„ƒZe d/¡d0d1„ƒZe d2¡d3d4„ƒZe d5¡d6d7„ƒZe d8¡d9d:„ƒZe d;¡d¡d?d@„ƒZ!e dA¡dBdC„ƒZ"e dD¡dEdF„ƒZ#e dG¡dHdI„ƒZ$e dJ¡dKdL„ƒZ%e dM¡dNdO„ƒZ&e dP¡dQdR„ƒZ'e dS¡dTdU„ƒZ(e dV¡dWdX„ƒZ)e dY¡dZd[„ƒZ*e d\¡d]d^„ƒZ+e d_¡d`da„ƒZ,e db¡dcdd„ƒZ-dedf„Z.e dg¡dhdi„ƒZ/e dj¡dkdl„ƒZ0e dm¡dndo„ƒZ1e dp¡dqdr„ƒZ2e ds¡dtdu„ƒZ3e dv¡dwdx„ƒZ4e dy¡dzd{„ƒZ5e d|¡d}d~„ƒZ6e d¡d€d„ƒZ7e d‚¡dƒd„„ƒZ8e d…¡d†d‡„ƒZ9e dˆ¡d‰dŠ„ƒZ:e d‹¡dŒd„ƒZ;e dŽ¡dd„ƒZe d—¡d˜d™„ƒZ?dšd›„Z@e dœ¡ddž„ƒZAe dŸ¡d d¡„ƒZBe d¢¡d£d¤„ƒZC ¥dÃd¦d§„ZDe d¨¡d©dª„ƒZEe d«¡d¬d­„ƒZFe d®¡d¯d°„ƒZGe d±¡d²d³„ƒZHe d´¡e dµ¡d¶d·„ƒƒZIe d¸¡d¹dº„ƒZJe d»¡d¼d½„ƒZKd¾d¿„ZLe dÀ¡dÁd„ƒZMdS)Äz-Gradients for operators defined in nn_ops.py.éN)Úbackprop)Údtypes)Úops)Ú array_ops)Úarray_ops_stack)Ú gen_nn_ops)Úmath_ops)Únn_opsZConv2DBackpropInputc Cs dtj|t |jd¡|jd| d¡| d¡| d¡| d¡| d¡| d ¡ ¡d tj||jd| d¡| d¡| d¡| d¡| d¡| d ¡ ¡d gS) úËThe derivatives for deconvolution. Args: op: the Deconvolution op. grad: the tensor representing the gradient w.r.t. the output Returns: the gradients w.r.t. the input and the filter NééÚ dilationsÚstridesÚpaddingÚexplicit_paddingsÚuse_cudnn_on_gpuÚ data_format©r rrrrr)rÚconv2d_backprop_filterrÚshapeÚinputsÚget_attrÚdecodeÚconv2d©ÚopÚgrad©rú^/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/ops/nn_grad.pyÚ_Conv2DBackpropInputGrads. ÷  øôrZConv2DBackpropFilterc Cs tjt |jd¡||jd| d¡| d¡| d¡| d¡| d¡| d¡ ¡d dtj|jd|| d¡| d¡| d¡| d¡| d¡| d¡ ¡d gS) Nrr r rrrrrr)rÚconv2d_backprop_inputrrrrrrrrrrÚ_Conv2DBackpropFilterGradDs. ÷  øõr!Z"DepthwiseConv2dNativeBackpropInputc Csˆdtj|t |jd¡|jd| d¡| d¡| d¡| d¡| d¡d tj||jd| d¡| d¡| d¡| d¡| d¡d gS) r Nr r r rrrr©r rrrr)rÚ'depthwise_conv2d_native_backprop_filterrrrrÚdepthwise_conv2d_nativerrrrÚ'_DepthwiseConv2dNativeBackpropInputGrad_s* ø ùõr%Z#DepthwiseConv2dNativeBackpropFilterc Csˆtjt |jd¡||jd| d¡| d¡| d¡| d¡| d¡ddtj|jd|| d¡| d¡| d¡| d¡| d¡dgS) Nrr r rrrrr")rÚ&depthwise_conv2d_native_backprop_inputrrrrr$rrrrÚ(_DepthwiseConv2dNativeBackpropFilterGrad€s*øùör'ZConv3Dc CsŒ| d¡ ¡}t |jd|jdg¡\}}tj||jd|| d¡| d¡| d¡|dtj|jd||| d¡| d¡| d¡|dgS)Nrrr r rr©r rrr)rrrÚshape_nrr Úconv3d_backprop_input_v2Úconv3d_backprop_filter_v2)rrrÚshape_0Úshape_1rrrÚ _Conv3DGrad—s*ùù÷r.ZConv3DBackpropInputV2c Csz| d¡ ¡}dtj|t |jd¡|jd| d¡| d¡| d¡|dtj||jd| d¡| d¡| d¡|dgS)Nrr r r rrr()rrr r+rrrÚconv3d©rrrrrrÚ_Conv3DBackpropInputGrad¯s(ùúör1ZConv3DBackpropFilterV2c Csz| d¡ ¡}tjt |jd¡||jd| d¡| d¡| d¡|ddtj|jd|| d¡| d¡| d¡|dgS)Nrrr r rrr()rrr r*rrrr/r0rrrÚ_Conv3DBackpropFilterGradÆs(ùú÷r2Z AvgPool3Dc Cs>tjt |jd¡|| d¡| d¡| d¡| d¡ ¡dS©NrÚksizerrr)r4rrr)rZavg_pool3d_gradrrrrrrrrrÚ_AvgPool3DGradÜs úr5Z AvgPool3DGradc Cs@t |jd¡tj|| d¡| d¡| d¡| d¡ ¡dfS©Nrr4rrr)r)rÚ stop_gradientrrZ avg_pool3drrrrrrÚ_AvgPool3DGradGradçs ûÿr8Z MaxPool3Dc Cs@tj|jd|jd|| d¡| d¡| d¡| d¡ ¡dSr3)rÚmax_pool3d_gradrÚoutputsrrrrrrÚ_MaxPool3DGradòs ùr;Z MaxPool3DGradc Có^t |jd¡t |jd¡tj|jd|jd|| d¡| d¡| d¡| d¡ ¡dfS©Nrr r4rrr©rr)rÚ zeros_likerrZmax_pool3d_grad_gradrrrrrrÚ_MaxPool3DGradGradþó ùþr@ZMaxPool3DGradGradc Cr<r=)rr?rrr9rrrrrrÚ_MaxPool3DGradGradGrad rArBZSoftmaxcCs*|jd}tj||ddd}|||S)a The derivative of the softmax nonlinearity. We assume that probs is of shape [batch_size * dim] The formula for dsoftmax / dx = (diag(softmax) - softmax * softmax'). This matrix is diagonal minus a rank one matrix, so it is easy to implement as follows: grad_x = grad_softmax * softmax - sum(grad_softmax * softmax) * softmax Args: op: the Softmax op. grad_softmax: the tensor representing the gradient w.r.t. the softmax output. Returns: gradient w.r.t the input to the softmax réÿÿÿÿT©Úkeepdims)r:rÚ reduce_sum)rZ grad_softmaxÚsoftmaxZ sum_channelsrrrÚ _SoftmaxGrads  rHZ LogSoftmaxcCs(t |jd¡}|tj|ddd|S)aThe gradient for log_softmax. log_softmax = input - log(sum(exp(input)) dlog_softmax/dinput = diag - softmax(input) Args: op: The log softmax op. grad: The tensor representing the gradient w.r.t. the output. Returns: The gradients w.r.t. the input. rrCTrD)rÚexpr:rF)rrrGrrrÚ_LogSoftmaxGrad3srJZBiasAddcCs8z| d¡}Wn tyd}Ynw|tj||dfS)a§Return the gradients for the 2 inputs of bias_op. The first input of unused_bias_op is the tensor t, and its gradient is just the gradient the unused_bias_op received. The second input of unused_bias_op is the bias vector which has one fewer dimension than "received_grad" (the batch dimension.) Its gradient is the received gradient Summed on the batch dimension, which is the first dimension. Args: op: The BiasOp for which we need to generate gradients. received_grad: Tensor. The gradients passed to the BiasOp. Returns: Two tensors, the first one for the "tensor" input of the BiasOp, the second one for the "bias" input of the BiasOp. rN)Z out_backpropr)rÚ ValueErrorrZ bias_add_grad)rÚ received_gradrrrrÚ _BiasAddGradEs ÿÿÿrMZ BiasAddGradc Csìz| d¡}Wn tyd}Ynwt |jd¡}t |¡}|dkrNt t |dd…¡|t |dd…¡gd¡}t |dd…dg|dd…gd¡}nt t |dd…¡|gd¡}t |dd…dggd¡}t ||¡}t ||¡S)a(Gradient for the BiasAddGrad op. Args: op: BiasAddGrad op for which we are calculating gradients. received_grad: The gradients passed to the BiasAddGrad op. Returns: A single gradient Tensor for the input to BiasAddGrad (which is the gradient of the bias term in BiasAdd) rNróNCHWr r rC) rrKrrrÚconcatZ ones_likeÚreshapeZtile)rrLrrZ bias_shapeZexpanded_shapeZ tile_multsZ expanded_gradrrrÚ_BiasAddGradGradas*  ÿ þý&ÿ  rQZ BiasAddV1cCs$t t |¡d¡}|t ||¡fS)a³Return the gradients for the 2 inputs of bias_op. The first input of unused_bias_op is the tensor t, and its gradient is just the gradient the unused_bias_op received. The second input of unused_bias_op is the bias vector which has one fewer dimension than "received_grad" (the batch dimension.) Its gradient is the received gradient Summed on the batch dimension, which is the first dimension. Args: unused_bias_op: The BiasOp for which we need to generate gradients. received_grad: Tensor. The gradients passed to the BiasOp. Returns: Two tensors, the first one for the "tensor" input of the BiasOp, the second one for the "bias" input of the BiasOp. r )rÚrangerÚrankrF)Zunused_bias_oprLZreduction_dim_tensorrrrÚ_BiasAddGradV1…sÿrTZRelucCót ||jd¡S©Nr)rÚ relu_gradr:rrrrÚ _ReluGradórXZEluGradcCs8|jd}t ||¡t |dk||jdt |¡¡fS)Nr r)rrÚelu_gradrÚwherer?)rrZelu_xrrrÚ _EluGradGrad¢ó  ÿÿr\ZSeluGradcCs8|jd}t ||¡t |dk||jdt |¡¡fS)Nr gr)rrÚ selu_gradrr[r?)rrZselu_xrrrÚ _SeluGradGradªr]r_ZRelu6cCrUrV)rÚ relu6_gradr:rrrrÚ _Relu6Grad²rYraZ Relu6GradcCó |jd}t ||¡t |¡fS©Nr )rrr`rr?©rrÚxrrrÚ_Relu6GradGrad·ó rfZ LeakyRelucCs$|jd}| d¡}tj|||dS)NrÚalpha©rh)rrrÚleaky_relu_grad©rrrerhrrrÚ_LeakyReluGrad½s  rlZ LeakyReluGradcCs.|jd}| d¡}tj|||dt |¡fS)Nr rhri)rrrrjrr?rkrrrÚ_LeakyReluGradGradÄs  ÿÿrmZElucCrUrV)rrZr:rrrrÚ_EluGradÌrYrnZSelucCrUrV)rr^r:rrrrÚ _SeluGradÑrYroZSoftpluscCs|t |jd¡SrV)rZsigmoidrrrrrÚ _SoftplusGradÖsrpZ SoftplusGradcCsp|j\}}t |g¡#t ||¡}||t | ¡dt |¡}||fWdƒS1s1wYdS)Ng@)rrZcontrol_dependenciesrZ softplus_gradrrI)rrZdyreZddyZd2xrrrÚ_SoftplusGradGradÛs  "$ýrqZSoftsigncCrUrV)rZ softsign_gradrrrrrÚ _SoftsignGradèrYrrZReluGradcCrbrc)rrrWrr?rdrrrÚ _ReluGradGradírgrscCst |d¡}||S)zãMultiply after broadcasting vec to match dimensions of mat. Args: vec: A 1-D tensor of dimension [D0] mat: A 2-D tensor of dimension [D0, D1] Returns: A tensor of dimension [D0, D1], the result of vec * mat rC)rÚ expand_dims)ZvecÚmatrrrÚ _BroadcastMulós rvZSoftmaxCrossEntropyWithLogitsc Cs„|jd}t||ƒ}|jd}|dur7t|ddƒs7t |¡}||tjt  t  |d¡t  |d¡¡dd|7}|t|t  |¡ ƒfS)z4Gradient function for SoftmaxCrossEntropyWithLogits.r rNÚ_is_zeros_tensorFr ©Úaxis) r:rvrÚgetattrr rGrÚsqueezerÚmatmulrtZ log_softmax©rZ grad_lossZ grad_gradZ softmax_gradrZlogitsrGrrrÚ"_SoftmaxCrossEntropyWithLogitsGrads"    ÿ   þüür~Z#SparseSoftmaxCrossEntropyWithLogitsc Csv|jd}t||ƒ}|jd}|dur7t|ddƒs7t |¡}||tjt  t  |d¡t  |d¡¡dd|7}|dfS)z:Gradient function for SparseSoftmaxCrossEntropyWithLogits.r rNrwFr rx) r:rvrrzr rGrr{rr|rtr}rrrÚ(_SparseSoftmaxCrossEntropyWithLogitsGrads"   ÿ   þüürZConv2Dc Csž| d¡}| d¡}| d¡}| d¡}| d¡}| d¡}t |jd|jdg¡\}} tj||jd|||||||d tj|jd| |||||||d gS) zGradient function for Conv2D.r rrrrrrr r)rrr)rrr r) rrr rrrrrr,r-rrrÚ _Conv2DGrad5s<       ÷ ÷õr€ZDepthwiseConv2dNativec Cs”tjt |jd¡|jd|| d¡| d¡| d¡| d¡| d¡dtj|jdt |jd¡|| d¡| d¡| d¡| d¡| d¡dgS) Nrr r rrrrr")rr&rrrrr#rrrrÚ_DepthwiseConv2dNativeGrad^s*ø øörZ Dilation2Dc Csdt |jd|jd|| d¡| d¡| d¡¡t |jd|jd|| d¡| d¡| d¡¡gS)Nrr rZratesr)r Zdilation2d_backprop_inputrrZdilation2d_backprop_filterrrrrÚ_Dilation2DGradvsýýûr‚ZLRNc CsL| d¡}| d¡}| d¡}| d¡}t ||jd|jd||||¡gS)NÚ depth_radiusÚbiasrhÚbetar)rrZlrn_gradrr:)rrrƒr„rhr…rrrÚ_LRNGrad„s    ÿÿr†ZAvgPoolc Cs:tjt |jd¡|| d¡| d¡| d¡| d¡dSr6)rZ avg_pool_gradrrrrrrrrÚ _AvgPoolGradsúr‡Z AvgPoolGradc Cs<t |jd¡tj|| d¡| d¡| d¡| d¡dfSr6)rr7rrZavg_poolrrrrrÚ_AvgPoolGradGrad›sûÿrˆZMaxPoolc CsDtj|jd|jd|| d¡| d¡| d¡| d¡| d¡dS)Nrr4rrrr)rrr)rÚ max_pool_gradrr:rrrrrÚ _MaxPoolGrad¦sørŠZ MaxPoolV2c CsJ|jd}|jd}tj|jd|jd|||| d¡| d¡dddfS)Nr r rrrr>)rrZmax_pool_grad_v2r:r©rrr4rrrrÚ_MaxPoolGradV2³s  ùùrŒZMaxPoolWithArgmaxc Cs>~tj|jd||jd| d¡| d¡| d¡| d¡dS)Nrr r4rrÚinclude_batch_in_index)rr)rZmax_pool_grad_with_argmaxrr:r)rrZunused_argmax_gradrrrÚ_MaxPoolGradWithArgmaxÁsùrŽZ MaxPoolGradc CóZt |jd¡t |jd¡tj|jd|jd|| d¡| d¡| d¡| d¡dfSr=)rr?rrZmax_pool_grad_gradrrrrrÚ_MaxPoolGradGradÎóùþrZ MaxPoolGradV2c Csf|jd}|jd}t |jd¡t |jd¡tj|jd|jd|||| d¡| d¡dddfS)Néérr rrr>)rrr?rZmax_pool_grad_grad_v2rr‹rrrÚ_MaxPoolGradGradV2Üs  ù÷r”ZMaxPoolGradGradc Crr=)rr?rrr‰rrrrrÚ_MaxPoolGradGradGradìr‘r•ZFractionalMaxPoolc Cs2t |jd|jd||jd|jd| d¡¡S)aReturns gradient for FractionalMaxPool. Since FractionalMaxPool has three outputs, there are three gradients passed in for each of the outputs. Only the first one is useful, the other two gradients are empty. Args: op: The FractionalMaxPoolOp. grad_0: Gradient with respect to op.outputs[0] unused_grad_1: Gradient with respect to op.outputs[1]/row_seq. It is empty. unused_grad_2: Gradient with respect to op.outputs[2]/col_seq. It is empty. Returns: Input backprop for FractionalMaxPool op. rr r Ú overlapping)rZfractional_max_pool_gradrr:r©rZgrad_0Z unused_grad_1Z unused_grad_2rrrÚ_FractionalMaxPoolGradús"þr˜ZFractionalAvgPoolc Cs.t |jd ¡||jd|jd| d¡¡S)aReturns gradient for FractionalAvgPool. Since FractionalAvgPool has three outputs, there are three gradients passed in for each of the outputs. Only the first one is useful, the other two gradients are empty. Args: op: The FractionalAvgPoolOp. grad_0: Gradient with respect to op.outputs[0] unused_grad_1: Gradient with respect to op.outputs[1]/row_seq. It is empty. unused_grad_2: Gradient with respect to op.outputs[2]/col_seq. It is empty. Returns: Input backprop for FractionalAvgPool op. rr r r–)rZfractional_avg_pool_gradrÚ get_shaper:rr—rrrÚ_FractionalAvgPoolGradsþršZ BatchNormWithGlobalNormalizationc CsRt |jd|jd|jd|jd|| d¡| d¡¡\}}}}}|||||fS)a$Return the gradients for the 5 inputs of BatchNormWithGlobalNormalization. We do not backprop anything for the mean and var intentionally as they are not being trained with backprop in the operation. Args: op: The BatchNormOp for which we need to generate gradients. grad: Tensor. The gradients passed to the BatchNormOp. Returns: dx: Backprop for input, which is (grad * (g * rsqrt(v + epsilon))) dm: Backprop for mean, which is sum_over_rest(grad * g) * (-1 / rsqrt(v + epsilon)) dv: Backprop for variance, which is sum_over_rest(grad * g * (x - m)) * (-1/2) * (v + epsilon) ^ (-3/2) db: Backprop for beta, which is grad reduced in all except the last dimension. dg: Backprop for gamma, which is (grad * ((x - m) * rsqrt(v + epsilon))) rr r r“Zvariance_epsilonZscale_after_normalization)rZ)batch_norm_with_global_normalization_gradrr)rrÚdxÚdmZdvÚdbÚdgrrrÚ%_BatchNormWithGlobalNormalizationGrad&s "þrŸc Gs°|jd}|d}|jd}| d¡}| d¡}| d¡}|dkr%tj} n |dkr-tj} ntj} |r[||||jd|jd|||d œ} |dkrN|jd | d <| di| ¤Ž\} } } }}nv|jd}|jd}|d krzt |gd ¢¡}t |gd ¢¡}n|dkrŽt |gd¢¡}t |gd¢¡}|dvr”dnd}||||||||d œ} |dkr¬|jd | d <| di| ¤Ž\} } } }}|d krÅt | gd¢¡} n |dkrÑt | gd¢¡} | | | ddfS)aãReturn the gradients for the 3 inputs of BatchNorm. Args: op: The BatchNormOp for which we need to compute gradients. version: Integer indicating which version to use of the fused batch norm gradient. *grad: An argument list for tensors of gradients wrt the outputs with grad[0] as grad_y. Returns: grad_x: gradient for x, which is scale * rsqrt(variance + epsilon) * [grad_y - mean(grad_y) - (x - mean(x)) * mean(grad_y * (x - mean(x))) / (variance + epsilon)] in training mode; grad_y * scale * rsqrt(pop_variance + epsilon) in freeze mode. grad_scale: gradient for scale, which is sum(grad_y * (x - mean(x)) * rsqrt(variance + epsilon)) in training mode; sum(grad_y * (x - pop_mean) * rsqrt(pop_variance + epsilon)) in freeze mode. grad_offset: gradient for offset, which is sum(grad_y) in training mode; sum(grad_y) in freeze mode. rr ÚepsilonrÚ is_trainingr r’r“)Z y_backpropreÚscaleZreserve_space_1Zreserve_space_2r rr¡éZreserve_space_3rN)rr r’r óNCDHW)rr r’r“r )rNóNHWCZNHWCZNDHWC)rr’r r )rr“r r r’Nr) rrrZfused_batch_norm_grad_v3Zfused_batch_norm_grad_v2Zfused_batch_norm_gradr:rZ transpose)rÚversionrreÚgrad_yr¢r rr¡Zgrad_funÚargsr›ZdscaleZdoffsetÚ_Úpop_meanÚpop_varZtarget_data_formatrrrÚ_BaseFusedBatchNormGradAsh     ø    ÿø r¬ZFusedBatchNormcGót|dg|¢RŽSrV©r¬rrrrÚ_FusedBatchNormGrad“rYr¯ZFusedBatchNormV2cGr­rcr®rrrrÚ_FusedBatchNormV2Grad˜rYr°ZFusedBatchNormV3cGr­)Nr r®rrrrÚ_FusedBatchNormV3GradrYr±TcCs´|jj}|tjks|tjkrt |tj¡}t |tj¡}|rÙ|dkr)d} gd¢} n;|dkr4d} gd¢} n0|dkrNd} gd¢} d t  |¡d d g} t  || ¡}nd} gd ¢} d t  |¡d d d g} t  || ¡}tj || | d } tj || | d } tj t  |t  | ¡¡| | d }|| }|| }tj ||| | d }|t ||¡|t ||¡||}t ||¡tj||| | d }|dksÄ|d krÉt |¡}tj|| d}t ||¡||fS|dkrâgd¢} nN|dkrëgd¢} nE|dkrgd¢} d t  |¡d d g} t  || ¡}t  || ¡}t  || ¡}n gd ¢} d t  |¡d d d g} t  || ¡}t  || ¡}t  || ¡}tj|| d}t ||¡}tj||||| d}|||}t ||¡||fS)aSReturns the gradients for the 3 inputs of BatchNorm. Args: grad_y: A `Tensor` of 4 or 5 dimensions for gradient for y. x: A `Tensor` of 4 or 5 dimensions for x. scale: A `Tensor` of 1 dimension for scaling. pop_mean: A `Tensor` of 1 dimension for the population mean. Only used when is_training=False. pop_var: A `Tensor` of 1 dimension for the population variance. Only used when is_training=False. epsilon: A small float number added to the variance of x. data_format: The data format for input. Either b"NHWC" or b"NCHW". is_training: A bool value to indicate the operation is for training (default) or inference. Returns: A tuple (grad_x, grad_scale, grad_offset), where grad_x is the gradient for x, grad_scale the gradient for scale, and grad_offset the gradient for offset. r¥F)rr r sNDHWC)rr r r’rNT)rr r’r )rr r’r“rD)ryrEr¤rx)ÚdtypeZ base_dtyperZfloat16Zbfloat16rÚcastZfloat32rÚsizerPZ reduce_meanZsquared_differencer7ZrsqrtZ reciprocalrFr{)r§rer¢rªr«r rr¡Zx_dtyperEZ reduce_axisrZ mean_grad_yZmean_xZvar_xZ grad_y_offsetZx_offsetÚmeanÚgrad_xÚ grad_scaleÚ grad_offsetZ var_rsqrtrrrÚ_BatchNormGrad¢s€   ý ÿÿ ÿ         ÿ r¹ZFusedBatchNormGradc Gs| d¡}| d¡}| d¡}|jd}|jd}|jd}|jd}|jd} |d} |d} |d} t ¡*} |  |¡|  |¡|  |¡t||||| |||ƒ\}}}| | | g}Wd ƒn1sewY|  |||g|||g|¡\}}}|||d d fS) aôReturns the gradients for the 3 inputs of FusedBatchNormGrad. Args: op: The FusedBatchNormGradOp for which we need to compute gradients. *grad: An argument list for tensors of gradients wrt the outputs with grad[0] as grad_grad_x, grad[1] as grad_grad_scale, grad[2] as grad_grad_offset. Returns: A tuple (grad_grad_y, grad_x, grad_scale, None, None), where grad_grad_y is the gradient for grad_y, grad_x the gradient for x, grad_scale the gradient for scale. rr r¡rr r r’r“N)rrrZ GradientTapeÚwatchr¹Zgradient)rrrr r¡r§rer¢rªr«Z grad_grad_xZgrad_grad_scaleZgrad_grad_offsetZtaper¶r·r¸Z grad_initialÚ grad_grad_yrrrÚ_FusedBatchNormGradGrads0             ÿ ú ÿr¼ZFusedBatchNormGradV2cGst|g|¢RŽS©N©r¼rrrrÚ_FusedBatchNormGradGradV2'sr¿ZFusedBatchNormGradV3cGs*t|g|¢RŽ\}}}}}|||dddfSr½r¾)rrr»r¶r·r©rrrÚ_FusedBatchNormGradGradV3,srÀZL2LosscCs|jd|S)zÊReturn the gradients for L2Loss. Args: op: The L2LossOp for which we need to generate gradients. grad: Tensor containing a single number. Returns: The gradient, which is (x * grad). r)rrrrrÚ _L2LossGrad2s rÁZTopKZTopKV2c Cst |jd¡}t |jd¡}t t |tj¡t  |¡d¡}t  |jdt   d|g¡¡}t t |tj¡t  |¡d¡}t |¡d}t  |t t  t dt |tj¡||¡d¡tj¡dg¡} t  t t  | d¡t  |dg¡t |¡g¡|¡tjgtjdgS)aEReturn the gradients for TopK. Args: op: The TopKOp for which we need to generate gradients. grad: Tensor. The gradients passed to the TopKOp. Returns: A list of two tensors, the first being the gradient w.r.t to the input and TopK, and the second being the gradient w.r.t. to the indices (all zero). rr rC)r²)rrrr:Úgatherrr³rZint64r´rPrÚstackrtrRZint32Ú scatter_ndZ reduce_prodZzeros) rrr©Zin_shapeZ ind_shapeZ ind_lastdimZind_2dZ in_lastdimZouterdimÚindrrrÚ _TopKGrad@sH   þÿ  þþýüû  þýûrÆZ ApproxTopKcs°ˆjdj‰ˆdg‰t tjˆ¡}ˆj‰ˆ d¡‰ˆdkr"ˆˆ‰‡‡‡‡‡fdd„‰tj t ‡fdd„t ˆƒDƒƒˆd}t  ||ˆg¡}t  ||g¡}t  ||ˆjdj¡S) zÕReturn the gradients for ApproxTopK. Args: op: The ApproxTopK for which we need to generate gradients. grad: The gradients for backprop. Returns: Scattered gradient based on the top-k indices. r Zreduction_dimensionrcs\|ˆkr t ˆjdˆ¡Sˆ|}tt dˆd¡ƒ}|||<t t |¡|¡}t |ˆ¡Src) rrPr:ÚlistÚ itertoolsÚrepeatrrRZ broadcast_to)ÚdZiota_lenZ iota_shapeÚiota)Ú idx_shapeÚlifted_idx_shaperrSÚ reduction_dimrrÚ GetLiftedIdxˆs z)_ApproxTopKGradient..GetLiftedIdxc3s|]}ˆ|ƒVqdSr½r)Ú.0rÊ)rÏrrÚ ’s€z&_ApproxTopKGradient..rx)r:rÚ functoolsÚreduceÚoperatorÚmulrSrrrOrÇrRrPrÄr)rrr©Zflat_shape_lenZ lifted_idxZflat_idxZ flat_gradr)rÏrÌrÍrrSrÎrÚ_ApproxTopKGradientos    ÿrÖZ NthElementcCsf|jd}|jd}t t t |d¡|¡|j¡}t |d¡}t t |d¡d¡}t  ||¡|dgS)a:Return the gradients for NthElement. Args: op: The NthElementOp for which we need to generate gradients. grad: Tensor. The gradients passed to the NthElementOp Returns: A list of two tensors, the first being the gradient w.r.t. the input, the second being the gradient w.r.t. the N (None). rrCN) rr:rr³Úequalrrtr²rFÚdivide)rrÚinputÚoutputZ indicatorsZ num_selectedrrrÚ_NthElementGrad˜s ÿ rÛc Csxg}tt ||jd¡t ||jd¡ƒD]\}}tj||t |¡dd}| t t  ||¡dg¡¡qt j |ddS)a}Replaces each segment with its mean along the last axis. Specifically, each value in the `inputs` tensor gets replaced by the mean value computed from the values that belong to the same segment. Args: inputs: A 2-tensor. Aggregation is done over dimension 1. segments: A 2-tensor, same shape as `input`. Returns: The result, same shape and type as `inputs`. rr )Z num_segmentsrCrx) ÚziprÚsplitrrZunsorted_segment_meanZ reduce_maxÚappendrPrÂrrÃ)rÚsegmentsÚresultZinputs_iZ segments_iZmeans_irrrÚ_MeanAggregator³s  þÿÿráZIsotonicRegressioncCs~|jd}t||ƒS)afGradient for the isotonic regression function. Args: op: The IsotonicRegression tensorflow op. grad_output: Tensor of incoming gradients with respect to the output. grad_segments: Tensor of incoming gradients with respect to the segments. Returns: A tensor, same size as `grad_output` with the gradient with respect to the input. r )r:rá)rZ grad_outputZ grad_segmentsrßrrrÚ_IsotonicRegressionGradÎs  râ)T)NÚ__doc__rÒrÈrÔZtensorflow.python.eagerrZtensorflow.python.frameworkrrZtensorflow.python.opsrrrrr ZRegisterGradientrr!r%r'r.r1r2r5r8r;r@rBrHrJrMrQrTrXr\r_rarfrlrmrnrorprqrrrsrvr~rr€rr‚r†r‡rˆrŠrŒrŽrr”r•r˜ršrŸr¬r¯r°r±r¹r¼r¿rÀrÁrÆrÖrÛrárârrrrÚs          $               #                 (              R    ù_ %     - (