o ®Ÿ?eµƒã@sÜdZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z dd lm Z dd lm Z dd lm Z dd lmZdd lmZdd„ZGdd„dƒZGdd„dƒZdd„Zdd„Zdd„Zdd„ZdS)zHUtilities for managing state of v1 control flow for computing gradients.é)Ú constant_op)Údtypes)Úops)Ú tensor_util)Ú array_ops)Úcontrol_flow_ops)Úcontrol_flow_util)Úcontrol_flow_v2_func_graphs)Údefault_gradient)Úgen_data_flow_ops)Úgen_resource_variable_ops)Úresource_variable_opsc CsÐ|j}t ¡ ¡}|dur|jnd}t d¡}|d|fvrf|j}|dur-td||jfƒ‚|j ¡}t   ||¡r=||9}nt   |¡}|durTtd||j|j||jfƒ‚||9}t j |j|d}|d|fvs|S)aCalculate a max_size for use by stack ops inside an XLA while_loop. Args: value: The value inside the while_loop forward context. Used for printing error messages. while_ctxt: The forward context inside which value resides. This does not always match the value's immediate context, as `value` may be inside e.g. a cond context inside the while_loop. Returns: A tensor containing the `max_size` to feed to a Stack initializer. Raises: ValueError: If `value` is nested inside a `while_loop` that either lacks a `maximum_iterations` parameter, or the `maximum_iterations` parameter: - is inside a `while_loop` that is a parent of the calling context, and - cannot be evaluated at graph build time to a constant. NÚéz–Cannot create a gradient accumulator for tensor '%s' inside XLA while_loop because maximum_iterations was not passed to the tf.while_loop call ('%s').a&Cannot create a gradient accumulator for tensor '%s' inside XLA while_loop. maximum_iterations tensor '%s' for while_loop context '%s' must be statically known (e.g. a constant value or known shape dimension), or be defined at or outside the while loop context '%s' (currently defined in '%s').)Z stop_ctxt)ÚnamerÚget_default_graphÚ_get_control_flow_contextrÚconstantÚmaximum_iterationsÚ ValueErrorÚopÚutilZIsContainingContextrÚconstant_valueZGetContainingWhileContextÚ outer_context) ÚvalueZ while_ctxtZ value_nameÚ curr_ctxtZcurr_ctxt_nameÚmax_sizeZmax_iterZ max_iter_ctxtZconst_max_iter©rúi/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/ops/control_flow_state.pyÚ&_GetMaxSizeFromNestedMaximumIterations!s<   þÿ     ÿûÿÿ Þ%rc@sàeZdZdZdd„Zedd„ƒZedd„ƒZedd „ƒZed d „ƒZ ed d „ƒZ edd„ƒZ edd„ƒZ edd„ƒZ edd„ƒZedd„ƒZedd„ƒZedd„ƒZedd„ƒZejdd„ƒZd'd d!„Z d'd"d#„Zd$d%„Zd&S)(Ú_GradLoopStateaThe state used for constructing the gradient graph for a while loop. We create a _GradLoopState for each while loop in forward and its corresponding while loop in backprop. This gives us access to both the forward and the backprop WhileContexts. During the construction of gradient graph, any time when we detect a forward value that is needed for backprop, we create a history accumulator and add it to `history_map`. Any time when we backprop a loop switch op (in _SwitchGrad), we add the grad merge op in `switch_map`. c Cs¤d|_d|_d|_d|_d|_d|_d|_i|_i|_g|_ g|_ t |j ƒ|_ t|j ƒ|_||_|r6|j}n t|dƒs?tdƒ‚|j}|j ¡|rN| ¡| |¡\}}|r[| ¡Wdƒn1sewY||_||_|r©| |j¡| |¡}|j}| ¡tj|j|j |j!|j"|j|d|_| #||¡}|j $||¡|_| ¡dS|r¯| ¡tj|j|j |j!|j"|j|d|_|j $||¡|_|rÐ| ¡dSdS)Nrz[Failed to call gradients on a while loop withoutproperly serializing graph via MetaGraphDef)rÚparallel_iterationsÚ back_propÚ swap_memoryrÚ grad_state)%Ú_outer_grad_stateÚ_forward_contextÚ_forward_indexÚ _forward_syncÚ _grad_contextÚ _grad_indexÚ _grad_syncÚ _history_mapÚ _switch_mapÚ _unused_exitsÚ_deferred_exitsÚlistÚ loop_exitsÚ_forward_loop_exitsÚlenÚ_pending_exits_countÚforward_contextÚhasattrrrZ_graphÚ as_defaultÚEnterZAddForwardLoopCounterÚExitZAddNamerÚAddForwardAccumulatorÚ grad_contextrZ WhileContextrr!r"r#ÚAddBackpropAccumulatedValueZAddBackpropLoopCounter) ÚselfÚ forward_ctxtÚouter_grad_stateÚouter_forward_ctxtÚcntÚ forward_indexZ history_cntÚouter_grad_ctxtZreal_cntrrrÚ__init__vs~    €û  ú ÿ úÿ ÿz_GradLoopState.__init__cCó|jS)z#The grad loop state for outer loop.)r%©r=rrrr?Ìóz_GradLoopState.outer_grad_statecCrE)z#The while loop context for forward.)r&rFrrrr5ÑrGz_GradLoopState.forward_contextcCrE)zThe loop index of forward loop.)r'rFrrrrBÖrGz_GradLoopState.forward_indexcCsf|jdur0t d¡tjdd|_Wdƒn1swY|j |j¡|jj  |j¡|jS)zžA control trigger node for synchronization in the forward loop. One main use is to keep the push ops of a stack executed in the iteration order. NZf_sync©r) r(rÚcontrol_dependenciesrÚcontrol_triggerÚ_set_control_flow_contextr&r'rÚ_add_control_inputrFrrrÚ forward_syncÛs  ÿz_GradLoopState.forward_synccCrE)z,The corresponding WhileContext for gradient.)r)rFrrrr;érGz_GradLoopState.grad_contextcCrE)z The loop index of backprop loop.)r*rFrrrÚ grad_indexîrGz_GradLoopState.grad_indexcCs~|jdurt|tjƒsŽt d|ƒ‚|r©|j! ¡tj|||d} |j! ¡| j  "|¡n| ¡tj|||d} | ¡|j# | j ¡|jj j$dj } | j  | ¡|WdƒWdƒS1sáwYWdƒdS1sñwYdS)aÆAdd an accumulator for each forward tensor that is needed in backprop. This is added to the forward loop at the first time when a tensor in the forward loop is used by backprop gradient computation loop. We create an accumulator that accumulates the value of tensor at each iteration. Called in the control flow context where gradients() is called. The pseudocode is: ``` acc = stack(); while (_pivot) { acc = stack_push(acc, value); } ``` We make sure that the stack push op in one iteration is executed before next iteration. This is achieved by adding a control edge from `forward_index.op.inputs[0].op` to the push op, and another control edge from the push op to either `forward_index.op` or `forward_sync`. Args: value: The source tensor in forward that is to be accumulated. dead_branch: True iff the tensor is on a dead branch of a cond. Returns: The stack that contains the accumulated history of the tensor. Raises: TypeError: For internal errors involving the value condition context. ValueError: If `value` is inside a XLA scope and a valid max size for the stack can't be found. NéÿÿÿÿZf_acc)rZ elem_typer)r#z#value_ctxt is not a CondContext: %sr)%r'Úgraphr7rrrrIr8Ú colocate_withrZIsInXLAContextrrrrZint32rr5r Zstack_v2ÚdtypeÚ base_dtyper9ÚAddValuer#ZGetOutputContextZ stack_push_v2rBrLÚ isinstancerÚ CondContextÚ TypeErrorrrKrMÚinputs) r=rÚ dead_branchrrÚaccZ enter_accZ swap_enabledÚ value_ctxtÚpushZadd_oprrrr:&sb"    ÿ ÿø     ÿ    ÿ ÿ Ìþ"þz$_GradLoopState.AddForwardAccumulatorc Cs2|j ¡}d}|j ¡}|r$||kr$t|tjƒr|}q$|j}|r$||kst d¡W|j  ¡|re|}d}|durL|rL|j   |j j ¡}|j}|durL|s;|durS|j }|rZd|jn|j} t ||¡| }t ||jj¡} |  | ¡¡|j ¡Wdƒn1sƒwY|jj} | dkr—|j | j¡| S)aùAdd the getter for an accumulated value in the grad context. This is added to the backprop loop. Called in the grad context to get the value of an accumulated value. The stack pop op must be guarded by the pred of the controlling cond. Args: history_value: The history (a stack) of a value. value: The value that is pushed onto the stack. dead_branch: True iff the tensor is on a dead branch of a cond. Returns: The current value (the top of the stack). Nr)rrr]rr^rrrIr;r8rPÚgetÚpredrr?ÚbranchÚ_SwitchRefOrTensorr Z stack_pop_v2rZr[Ú set_shapeÚ get_shaper9r!rOrL) r=Ú history_valuerraZ history_ctxtZ cond_ctxtrcr$rfrgÚpopr!rrrr<€sH     ü    þÿÿÿ ïz*_GradLoopState.AddBackpropAccumulatedValuecCsâ|jjdvsJ‚|j |j¡}|duro|}|} t |¡}|r3|jd}|j}|dur2|j   |¡}qUn!t   |¡rDt j t |¡|jd}qU|j  ¡| |¡}|j  ¡qUq|duri| ||¡}||kri|j   |¡}||j|j<|S)a²Get the real value of `value`. If backprop "uses" a value produced by forward inference, an accumulator is added in the forward loop to accumulate its values. We use the accumulated value. This method must be called in the grad loop context. `value` must be in forward and needed for backprop. Args: value: A tensor to be captured. Returns: The same tensor obtained from the saved history. )ÚVariableZ VariableV2NTr©rZ)rÚtyper,rerrZGetLoopConstantEnterr`r?r)r\rÚ is_constantrrrrZr9r:r8r<)r=rÚ real_valueZ cur_valueZcur_grad_stateZenter_oprkrrrÚ GetRealValue±s@   ú  ÿ   æÿ  z_GradLoopState.GetRealValueN)F)Ú__name__Ú __module__Ú __qualname__Ú__doc__rDÚpropertyr?r5rBrMr;rNrOrPrQrRrSrTrVÚsetterr:r<rrrrrrr hsF V               [ ÿ 1r c@sXeZdZdZdd„Zdd„Zdd„Zdd „Zd d „Zd d „Z dd„Z dd„Z dd„Z dS)Ú_ControlFlowStatez9Maintain the mapping from the loops to their grad states.cCs i|_dS)N)Ú_maprFrrrrDís z_ControlFlowState.__init__cCsH|rt |¡r| ¡}|j}|r| ¡}nt |¡}|r"|j |¡SdS)zDReturn the grad state for this op if it's in a forward loop context.N)rÚ IsLoopExitrrÚGetWhileContextrzre)r=rÚbeforer>rrrÚ GetGradStateðs€  z_ControlFlowState.GetGradStatecCs’g}|j ¡D]?}|jD]&}||jdkr2|jd8_|j|vr'|j |¡|jdkr2| |j¡q |jj D]}||jdkrEd||j<q7q|S)aNProcess all the "unused" loop exits. The "unused" exits of the loops are added to `unused_exits`. An exit is unused if its pending_count is 0. If there is an exit with real gradient, all these deferred exits will enter the backprop loop with zero gradient. Otherwise, they will enter the backprop loop with None. As an example, people often write: ```python v1, _ = tf.while_loop(p, b, [x1, x2]) result = gradients(v1, x1) ``` The exit node for x2 is not included by the betweenness analysis. But we need to backprop x2 if x2 is involved in computing v1. Args: pending_count: The number of backprop inputs for every op. to_ops_set: The set of ops for ys in gradients(ys, xs) Returns: The set of unused loop exits that we know at this point we need to backprop. rr) rzÚvaluesrTrrVrRÚappendÚextendr5Z loop_enters)r=Ú pending_countZ to_ops_setr1r$ÚyrrrÚProcessUnusedLoopExitsýs      €  €þz(_ControlFlowState.ProcessUnusedLoopExitscCó"| ||¡}|r|j ¡dSdS)z0Enter the WhileContext for gradient computation.N)r~r;r8©r=rr}r$rrrÚEnterGradWhileContext%ó ÿz'_ControlFlowState.EnterGradWhileContextcCr…)z/Exit the WhileContext for gradient computation.N)r~r;r9r†rrrÚExitGradWhileContext+rˆz&_ControlFlowState.ExitGradWhileContextc CsŽt |¡}|j |¡}|durC|j}|r| ¡}d}|r"|j |¡}t||ƒ}||j|<|jD]}|j|vrB| |j¡|  |j¡q/dSdS)aAdd the grad state for the while loop that op belongs to. Note that op is an Exit, and this method must be called in the control flow context where gradients() is called. Note that this method modifies `between_op_list` and `between_ops`. N) rr|rzrerr rTrÚaddr€) r=rÚbetween_op_listÚ between_opsr>r$r@r?Z loop_exitrrrÚAddWhileContext1s$         €ñ z!_ControlFlowState.AddWhileContextc Csú| ¡}|j ¡}|j}|r| ¡}d}|r|j |¡}|rf| ¡r6|j  ¡t   |j |j ¡}|j ¡|S|j  ¡t j|dd}|j ¡| |¡}|j} |   ¡| ||¡} t   | |j ¡}|  ¡|S| ¡rtt   |j |j ¡}|St j|dd}|S)aCreate zeros_like gradient for a loop exit. If the result of a loop variable is not used but is involved in computing the result of some needed loop variable, we create a zero-valued tensor that is fed as gradient for the Exit node of that loop variable. Note that val.op is an Exit, and this method must be called in the control flow context where gradients() is called. Args: val: The output tensor of an Exit op. Returns: A zero tensor of the same shape of val. NF©Úoptimize)rjrrrr|rzreÚis_fully_definedr;r8rÚzerosÚdimsrZr9Úshape_internalr:r<Ú zeros_like) r=ÚvalZ val_shaper>r@r?ÚresultÚshapeZ history_shaperCZ real_shaperrrÚZerosLikeForExitLs>     ì  ÿ úÿz"_ControlFlowState.ZerosLikeForExitcCs¸t |¡rdS|jjrt |j|¡St |¡}t |¡}|j   |¡}|dur,t ||ƒS|  ¡}t j|j|dd}| ¡}| ¡ry|jtjkrVtjt |¡t |¡d} n tjd|j|jd} |rw|j  |jj¡} |j} t  !| | ¡d| } | S|r¬|j} |j} |j" #¡t  !|j$d| ¡d| }tj%|dd } |j" &¡|j' (|¡| j' (|¡n| #¡tj%|dd } | &¡|j) &¡|j*| |d } |j) #¡| +| | |¡}t ||j¡} | S) a\Create zeros_like for the specified output of an op. If op is in a while loop that is part of gradients(), this method must be called in its grad loop context. Args: op: A tensorflow operation. index: the index for a specific output of the op. Returns: A zero tensor of the same shape of op.outputs[index]. NZtensorrHrnr©r—rZrFrŽ)ra),rZ IsLoopSwitchrXZbuilding_functionrr”ÚoutputsÚIsSwitchr|rzreÚ ZerosLikerrZconvert_to_tensorrjrrZrÚresourcer‘r Úvariable_shaper Úget_zeros_dtyperrr’rPrfrrgrrhrr8r`r“r9rrKr;r:r<)r=rÚindexrar>r$Úop_ctxtr•r—r–rfrgÚ zeros_shapeZhistory_zeros_shaperrrÚZerosLikeV1WhileLoopƒsd      þæ  ÿÿ   ÿ ÿz&_ControlFlowState.ZerosLikeV1WhileLoopc Cs|j ¡D]ƒ\}}|j ¡D]y\}}|jjd|jjdkr‡|jjdj}|jjd ¡}| ¡rI|j  ¡t j d||d}t   |¡}|j ¡n7|jj}|rS|  ¡|jjdj} | jd} tj| dd} t | ¡}|rq| ¡|j  ¡t   |¡}|j ¡|j d|¡qqdS)acPerform postprocessing at the end of gradients(). We have created the gradient graph at this point. So this function can be used to perform any postprocessing on the gradient graph. We currently perform the following postprocessing: 1. Patch the gradient graph if the output of a loop variable doesn't depend on its input. rr)rZr—FrŽN)rzÚitemsrQrr`rZrjrr;r8rrrZ_NextIterationr9rrr“r‘Z _update_input) r=Ú_r$Zb_mergerZr—Zgrad_valZ next_grad_valrCZ enter_grad_opZ enter_gradZ grad_shaperrrÚPostProcessingÊs4         €ãÿz _ControlFlowState.PostProcessingN) rsrtrurvrDr~r„r‡r‰rr˜r£r¦rrrrryês (7 Gryc Csvd}|D]4}t |¡r8|durtƒ}|r1t |¡| |||¡Wdƒn1s+wYq| |||¡q|S)aCCreate the state for all the while loops involved in one gradients(). We create a _ControlFlowState when there are while loops involved in gradients(). In gradients(), control flow logic is only invoked when the _ControlFlowState is not None. Note that this method modifies `between_op_list` and `between_ops`. N)rr{ryrrYr)r‹rŒZcolocate_gradients_with_opsZ loop_staterrrrÚMaybeCreateControlFlowStateõs   ÿ€€r§c Csô|j|}| ¡}|rs|j}|j}t |jd|¡d|}t |¡}|j t j krLt   |g¡tjt |¡t |¡dWdƒS1sGwYtj|dd}t   |g¡tj||j dWdƒS1slwYdStj|ddS)zBranch of ZerosLike for TF1.rrrnNFrŽ)ršrrfrgrÚswitchr`rÚidentityrZrrrrIr‘r ržr rŸr“r”) rr r•r¡rfrgZ switch_valZpivotr¢rrrÚ _ZerosLikeV1 s&   þ ÿ$ÿrªcCs–|j|}|jtjkrtjt |¡t  |¡dSt |j j t jƒrD|jtjkrD|j ¡r6tjd|jj|jdStj|dd}t ||j¡Stj|ddS)zBranch of ZerosLike for TF2.rnrr™FrŽ)ršrZrrrr‘r ržr rŸr]rrXr ZWhileBodyFuncGraphÚvariantr—rrrr’r“r”)rr r•r¢rrrÚ _ZerosLikeV2&s  þ ÿ r¬cCst |¡s t||ƒSt||ƒS)z4Create zeros_like for the specified output of an op.)rr›r¬rª)rr rrrrœ>s   rœN)rvZtensorflow.python.frameworkrrrrZtensorflow.python.opsrrrrr r r r r rr ryr§rªr¬rœrrrrÚs2            G