o ?e@sdZddlZddlmZddlmZddlmZddlmZddlm Z e Z da dd Z ed d d Zd dZddZddZddZedddZddZedddZddZdd Zd!d"Zd%d#d$ZdS)&z@Ops for GPU collective operations implemented using NVIDIA nccl.N)context) def_functiondevice)ops) gen_nccl_opscC td|S)aReturns a list of tensors with the all-reduce sum across `tensors`. The computation is done with an all-reduce operation, so if only some of the returned tensors are evaluated then the computation will hang. Args: tensors: The input tensors across which to sum; must be assigned to GPU devices. Returns: List of tensors, each with the sum of the input tensors, where tensor i has the same device as `tensors[i]`. sum_apply_all_reducetensorsr_/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/ops/nccl_ops.pyall_sum rZ NcclAllReducecCs~|ddkr tdt||jd|d}|dd}t|jtj|d||d Wd S1s8wYd S) a)The gradients for `all_sum`. Args: op: The `all_sum` `Operation` that we are differentiating. grad: Gradient with respect to the output of the `all_sum` op. Returns: The gradient with respect to the output of `all_sum`. Raises: LookupError: If `reduction` is not `sum`. reductionsumANo gradient defined for NcclAllReduce except for reduction="sum".expected num_devices shared_names_gradr inputrrrN)get_attr LookupError _check_devicerrrnccl_all_reduce)opgradrrrrr _all_sum_grad.s $r!cCr)aReturns a list of tensors with the all-reduce product across `tensors`. The computation is done with an all-reduce operation, so if only some of the returned tensors are evaluated then the computation will hang. Args: tensors: The input tensors across which to multiply; must be assigned to GPU devices. Returns: List of tensors, each with the product of the input tensors, where tensor i has the same device as `tensors[i]`. prodr r rrrall_prodLrr#cCr)aReturns a list of tensors with the all-reduce min across `tensors`. The computation is done with an all-reduce operation, so if only some of the returned tensors are evaluated then the computation will hang. Args: tensors: The input tensors across which to reduce; must be assigned to GPU devices. Returns: List of tensors, each with the minimum of the input tensors, where tensor i has the same device as `tensors[i]`. minr r rrrall_min]rr%cCr)aReturns a list of tensors with the all-reduce max across `tensors`. The computation is done with an all-reduce operation, so if only some of the returned tensors are evaluated then the computation will hang. Args: tensors: The input tensors across which to reduce; must be assigned to GPU devices. Returns: List of tensors, each with the maximum of the input tensors, where tensor i has the same device as `tensors[i]`. maxr r rrrall_maxnrr'cCr)aReturns a tensor with the reduce sum across `tensors`. The computation is done with a reduce operation, so only one tensor is returned. Args: tensors: The input tensors across which to sum; must be assigned to GPU devices. Returns: A tensor containing the sum of the input tensors. Raises: LookupError: If context is not currently using a GPU device. r ) _apply_reducer rrr reduce_sums r)Z NcclReducecCsp|ddkr tdt||jdt|jtj||jd}Wdn1s+wY|gt|j S)a\The gradients for input `Operation` of `reduce_sum`. Args: op: The `sum send` `Operation` that we are differentiating. grad: Gradient with respect to the output of the `reduce_sum` op. Returns: The gradient with respect to the input of `reduce_sum` op. Raises: LookupError: If the reduction attribute of op is not `sum`. rrrrrshapeN) rrrrrrnccl_broadcastr+leninputs)rr resultrrr_reduce_sum_gradsr0cCsHt|t|jtj||jdWdS1swYdS)a Returns a tensor that can be efficiently transferred to other devices. Args: tensor: The tensor to send; must be assigned to a GPU device. Returns: A tensor with the value of `src_tensor`, which can be used as input to ops on other GPU devices. r*N)rrrrr,r+)tensorrrr broadcasts $r2Z NcclBroadcastcCsbdd|jjD}|D]}t|q t|jtj|ddWdS1s*wYdS)a-The gradients for input `Operation` of `broadcast`. Args: op: The `broadcast send` `Operation` that we are differentiating. accumulated_grad: Accumulated gradients with respect to the output of the `broadcast` op. Returns: Gradients with respect to the input of `broadcast`. cSsg|]}|qSrr.0trrr sz#_broadcast_grad..r rrN)rr.rrrr nccl_reduce)rZaccumulated_gradZgradsr5rrr_broadcast_grads    $r9cs<stdtfdd}trt|S|S)z$Helper function for all_* functions.z-Must pass >0 tensors to all reduce operationsc sbg}D]*}t|t|j|tj|tdWdn1s)wYq|S)zCall nccl allreduce.rN)rrrappendrrr-)resr5rrr rr _all_reducesz&_apply_all_reduce.._all_reduce) ValueError_get_shared_namerZexecuting_eagerlyrfunction)rr r=rr<rr s r cs`|std|D]}t|qtj||dztfdd|DWSty/tdw)z'Helper function for reduce_* functions.z)Must pass >0 tensors to reduce operationsr7c3s |] }|jjkr|VqdSNrr3r/rr sz _apply_reduce..z3One input tensor must be assigned to current device)r>rrr8next StopIteration)rr r5rrBrr(s  r(cCs>tt}td7aWdd|S1swYd|S)Nzc%s) _module_lock_shared_name_counter)valrrrr?s  r?cCsPt|jstd|d|r$||jkr&td|d|jd|ddSdS)NzDevice assignment for tensor=z! required for nccl collective opszExpected device z, got z for tensor=.)rcanonical_namer>)r1rrrrrs  rrA)__doc__ threadingZtensorflow.python.eagerrrZtensorflow.python.frameworkrrZtensorflow.python.opsrLockrGrHrZRegisterGradientr!r#r%r'r)r0r2r9r r(r?rrrrrs2