o ?e@s2dZddlZddlmZddlZddlmZddlmZddlm Z ddlm Z ddlm Z dd lm Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZ ddlm!Z"ddl#m$Z$ddl%m&Z'ddl(m)Z)ddl*m+Z+ddl*m,Z-ddl*m.Z.dd Z/d!d"Z0Gd#d$d$e-j1Z1Gd%d&d&e1Z2Gd'd(d(e1ej3e-j4Z4dPd*d+Z5e6e4e5Gd,d-d-ej7Z8e$9e$:e8ej;j<Gd.d/d/e2e-j=Z=Gd0d1d1e>Z?Gd2d3d3e.j@ZAGd4d5d5e2e"jBe+jCZDGd6d7d7e)jEZFGd8d9d9e)jEZGGd:d;d;eDe=ZHGdd?d?eDZJ   )dPd@dAZKe6eDeKdPdBdCZLe6eHeLdPdDdEZMe6e=eMdPdFdGZNe6eJeNGdHdIdIe>ZOGdJdKdKeOZPGdLdMdMeOZQGdNdOdOZRdS)Qz0Various classes representing distributed values.N)Optional) struct_pb2) device_util)distribute_lib)packed_distributed_variable) reduce_util) values_util)context)record)composite_tensor)dtypes)ops)tensor)tensor_conversion_registry) tensor_util) type_spec) array_ops)control_flow_ops)math_ops)resource_variable_ops)variable_scope) variables)nested_structure_coder)base)saveable_object)core) distribute)tracec sjtjjkr|fi|StjsEjtjj kr-j j s-t |r-tdt|j}ttj|f|ddSfdd}tj||f|dS)zCUpdates variables with ON_WRITE synchronization in replica context.Cannot update non-float variables with tf.VariableAggregation.MEAN aggregation in replica context. Either change the variable dtype to float or update it in cross-replica context.Targskwargsgroupcs\jtjjkrjjst|trtd|j ksJt ||j}j |fi|S)zCAggregate values and update all variables in cross replica context.r) aggregationvsVariableAggregationMEANdtype is_floating isinstance PerReplica ValueErrordistribute_strategyrZapply_aggregation_update_cross_replica)strategyvaluer!v update_fnvard/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/distribute/values.pymerge_fnPs z*_on_write_update_replica..merge_fn)r r!)r#r$r%NONE_get_on_device_or_primaryr get_strategyextendedZ_use_merge_callr&r'r(r is_tf_typer+!apply_aggregation_replica_contextrmark_as_unsaveableget_replica_context_update merge_call)r3r2r/r!aggregated_valuer6r4r1r5_on_write_update_replica2s4 rBcsnt|tr tdt|s|S|tjjkr&fdd}t j ||fdSt j |}tj||}|S)zBAggregate `value` to `destinations` as specified by `aggregation`.zDCannot use DistributedValues to update variables in replica context.cs|jj||ddS)Nr destinations)r:Z broadcast_toZexperimental_local_results)r.r/rCr4r5r6ts z3apply_aggregation_replica_context..merge_fn)r )r)DistributedValues TypeErrorrr;r$r%ONLY_FIRST_REPLICArr>r@rReduceOpfrom_variable_aggregationr9r:_replica_ctx_all_reduce)r/r#rDr6Z reduce_oprAr4rCr5r<is      r<c@sXeZdZdZddZddZddZdd Zed d Z ed d Z ddZ ddZ dS)rEz/Base class for representing distributed values.cCst||_dS)z+Should only be called by subclass __init__.Ntuple_values)selfvaluesr4r4r5__init__zDistributedValues.__init__cCs"t}|dur |S|j|Sz@Returns the value for the current device or raises a ValueError.N)rget_current_replica_id_as_int_get_cross_replicarMrN replica_idr4r4r5_gets zDistributedValues._getcCtd)NzoDistributedValues._get_cross_replica should be implemented by sub-classes which support cross-replica accesses.NotImplementedErrorrNr4r4r5rTz$DistributedValues._get_cross_replicacCsRt}|dur$tt}|jD]}t|j|kr |Sq|jS|j|S)GReturns value in same replica or device if possible, else the _primary.N)rrSr canonicalizecurrentrMdevice_primary)rNrVcurrent_devicer/r4r4r5r8s  z+DistributedValues._get_on_device_or_primarycCs |jdS)z#Returns a representative component.rrMr[r4r4r5ras zDistributedValues._primarycCstdd|jDS)Ncs|]}|jVqdSN)r`.0r0r4r4r5 z-DistributedValues._devices..rKr[r4r4r5_devicesszDistributedValues._devicescC*dddt|jD}d|jj|fS)N, cs |] \}}d||fVqdS)z %d: %sNr4rgir0r4r4r5rh z,DistributedValues.__str__.. %s:{ %s }join enumeraterM __class____name__)rNZ debug_strr4r4r5__str__ zDistributedValues.__str__cCrk)Nrlcsrm)z %d: %rNr4rnr4r4r5rhrpz-DistributedValues.__repr__..rqrr)rNZ debug_reprr4r4r5__repr__rxzDistributedValues.__repr__N) rv __module__ __qualname____doc__rPrWrTr8propertyrarjrwryr4r4r4r5rEs   rEcs6eZdZdZfddZeddZddZdd Zd d Z d d Z ddZ ddZ ddZ ddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd4d5ZdId7d8Zd9d:Z d;d<Z!d=d>Z"d?d@Z#dAdBZ$dCdDZ%dEdFZ&dGdHZ'Z(S)JDistributedDelegatezAA map from device to values; acts as the same type as the values.cs>|ds |dvrtt||S|dkrtt||S)NZ_self_)Z_use_resource_variablesZ_attribute_sentinel_distributed_containerrM) startswithsuperr~ __getattr__AttributeErrorgetattrrWrNnamerur4r5rs  zDistributedDelegate.__getattr__cC|jSzReturns the per replica values.rcr[r4r4r5rOzDistributedDelegate.valuescC|S)aReturns the value for operations for the current device. Some implementations, e.g. `TPUMirroredVariable`, are not able to return the value type within a replica context. They can, however, return a value that can be used by the operations below. )rWr[r4r4r5_get_as_operandsz#DistributedDelegate._get_as_operandcCs ||SrerrNor4r4r5__add__ zDistributedDelegate.__add__cCs ||Srerrr4r4r5__radd__rzDistributedDelegate.__radd__cCs ||Srerrr4r4r5__sub__rzDistributedDelegate.__sub__cCs ||Srerrr4r4r5__rsub__rzDistributedDelegate.__rsub__cCs ||Srerrr4r4r5__mul__rzDistributedDelegate.__mul__cCs ||Srerrr4r4r5__rmul__rzDistributedDelegate.__rmul__cCs ||Srerrr4r4r5 __truediv__rzDistributedDelegate.__truediv__cCs ||Srerrr4r4r5 __rtruediv__rz DistributedDelegate.__rtruediv__cCs ||Srerrr4r4r5 __floordiv__rz DistributedDelegate.__floordiv__cCs ||Srerrr4r4r5 __rfloordiv__rz!DistributedDelegate.__rfloordiv__cCs ||Srerrr4r4r5__mod__ rzDistributedDelegate.__mod__cCs ||Srerrr4r4r5__rmod__rzDistributedDelegate.__rmod__cCs ||kSrerrr4r4r5__lt__rzDistributedDelegate.__lt__cCs ||kSrerrr4r4r5__le__rzDistributedDelegate.__le__cCs ||kSrerrr4r4r5__gt__rzDistributedDelegate.__gt__cCs ||kSrerrr4r4r5__ge__rzDistributedDelegate.__ge__cCs ||@Srerrr4r4r5__and__rzDistributedDelegate.__and__cCs ||@Srerrr4r4r5__rand__ rzDistributedDelegate.__rand__cCs ||BSrerrr4r4r5__or__#rzDistributedDelegate.__or__cCs ||BSrerrr4r4r5__ror__&rzDistributedDelegate.__ror__cCs ||ASrerrr4r4r5__xor__)rzDistributedDelegate.__xor__cCs ||ASrerrr4r4r5__rxor__,rzDistributedDelegate.__rxor__cCs ||Srerrr4r4r5 __getitem__/rzDistributedDelegate.__getitem__NcCst|||Srepowr)rNrZmodulor4r4r5__pow__2zDistributedDelegate.__pow__cCst||Srerrr4r4r5__rpow__5zDistributedDelegate.__rpow__cCs |Srerr[r4r4r5 __invert__8 zDistributedDelegate.__invert__cCs | Srerr[r4r4r5__neg__;rzDistributedDelegate.__neg__cCs t|Sre)absrr[r4r4r5__abs__>rzDistributedDelegate.__abs__cC(z||WStytYSwre)r__div__rNotImplementedrr4r4r5rA  zDistributedDelegate.__div__cCrre)r__rdiv__rrrr4r4r5rHrzDistributedDelegate.__rdiv__cCrre)r __matmul__rrrr4r4r5rOrzDistributedDelegate.__matmul__cCrre)r __rmatmul__rrrr4r4r5rVrzDistributedDelegate.__rmatmul__re))rvrzr{r|rr}rOrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr __classcell__r4r4rr5r~sL    r~c@s(eZdZdZeddZeddZdS)r*z2Holds a map from replica to unsynchronized values.cCstdd|jDS)Ncss|]}t|VqdSre)rZtype_spec_from_valuerfr4r4r5rhgsz(PerReplica._type_spec..)PerReplicaSpecrMr[r4r4r5 _type_specdszPerReplica._type_speccCrrrcr[r4r4r5rOirzPerReplica.valuesN)rvrzr{r|r}rrOr4r4r4r5r*`s  r*FcCsb~|dur||jstd|j|jj|rtdts$ts(tdt }|j |S)z&Converts a `PerReplica` to a `Tensor`.NzMIncompatible type conversion requested to type {!r} for variable of type {!r}z5PerReplica doesn't support being used as a reference.zIt looks like you are using a PerReplica object while not inside a replica context, which is not supported. Try running your op or function inside a replica context by using `strategy.run`) Zis_compatible_withr'r+formatrrZrin_cross_replica_contextZ has_strategyrrSrO)r3r'ras_refrVr4r4r5_per_replica_to_tensoros  rc@sNeZdZdZdgZeddZddZddZed d Z d d Z d dZ dS)rz&Type specification for a `PerReplica`. _value_specscCstSrer*r[r4r4r5szPerReplicaSpec.cGst||_dSre)rLr)rNZ value_specsr4r4r5rPrzPerReplicaSpec.__init__cCrrerr[r4r4r5 _serializezPerReplicaSpec._serializecCrrerr[r4r4r5_component_specszPerReplicaSpec._component_specscCs(t}|dur|jdkrtd|jS)NzJFlattening a PerReplica to components is not supported in replica context.)rr>Znum_replicas_in_syncr+rM)rNr/replica_contextr4r4r5_to_componentss zPerReplicaSpec._to_componentscCst|Srer)rNZ tensor_listr4r4r5_from_componentszPerReplicaSpec._from_componentsN) rvrzr{r| __slots__r}Z value_typerPrrrrr4r4r4r5rs   rc@(eZdZdZddZddZddZdS) Mirroredz:Holds a map from replica to values which are kept in sync.cCrrer8r[r4r4r5rTrzMirrored._get_cross_replicacCs*|}t|dd}|rt|r|S|S)N_as_graph_element)rWrcallable)rNobjZconv_fnr4r4r5rs   zMirrored._as_graph_elementcCdSNTr4r[r4r4r5 _is_mirroredzMirrored._is_mirroredN)rvrzr{r|rTrrr4r4r4r5rs  rc@r) DistributedVarOpz'A class that looks like `tf.Operation`.cCs||_||_||_||_dSre)rgraph tracebacktype)rNrrrtypr4r4r5rPs zDistributedVarOp.__init__cCs@t||jst|j|jko|j|jko|j|jko|j|jkSre)r)rurZrrrrrr4r4r5__eq__s   zDistributedVarOp.__eq__cCst|j|jt|j|jfSre)hashrrrLrrr[r4r4r5__hash__szDistributedVarOp.__hash__N)rvrzr{r|rPrrr4r4r4r5rs  rc@s^eZdZdZddZddZddZdd d Zd d Zd dZ de fddZ de fddZ dS)DistributedVariableTraceTypez)TraceType of DistributedVariable objects.cCs ||_t|j|jf|_dSre)distributed_variablerLshapeas_listr' components)rNrr4r4r5rPs  z%DistributedVariableTraceType.__init__cCs||kSrer4rNotherr4r4r5 is_subtype_ofrz*DistributedVariableTraceType.is_subtype_ofcstfdd|Dr SdS)Nc3s|]}|kVqdSrer4)rgrr[r4r5rhszNDistributedVariableTraceType.most_specific_common_supertype..)all)rNZothersr4r[r5most_specific_common_supertypesz;DistributedVariableTraceType.most_specific_common_supertypeNcCrre)r)rNZplaceholder_contextr4r4r5placeholder_valuerz.DistributedVariableTraceType.placeholder_valuecCsgSrer4)rNr/r4r4r5 _to_tensorsrz(DistributedVariableTraceType._to_tensorscCs|Srer4)rNr/_r4r4r5_castrz"DistributedVariableTraceType._castreturncCs t|jSre)rrr[r4r4r5rrz%DistributedVariableTraceType.__hash__cCst|tsdS|j|jkSNF)r)rrrr4r4r5rs  z#DistributedVariableTraceType.__eq__re)rvrzr{r|rPrrrrrintrboolrr4r4r4r5rs rcseZdZdZdfdd ZddZddZdd d Zed d Z d dZ ddZ eddZ eddZ eddZeddZeddZeddZeddZedd Zed!d"Zed#d$Zed%d&Zed'd(Zed)d*Zdd+d,Zed-d.Zd/d0Zd1d2Zed3d4Zed5d6Zed7d8Zd9d:Z dd;d<Z!ed=d>Z"ed?d@Z#dAdBZ$dCdDZ%dEdFZ&dGdHZ'dIdJZ(dKdLZ)ddOdPZ*ddQdRZ+ddSdTZ,ddUdVZ-ddWdXZ.ddYdZZ/dd[d\Z0dd]d^Z1dd_d`Z2ddadbZ3dcddZ4dedfZ5dgdhZ6didjZ7dkdlZ8dmdnZ9dodpZ:dqdrZ;ddsdtZ<  ddue=e>j?dve=e@dweAjBfdxdyZC   ddzd{ZDd|d}ZEed~dZFddZGZHS)DistributedVariablez&Holds a map from replica to variables.Nc s|tjjkr|djjstd||_||_tt | ||j j dd|_|D]}t|tjr?t|dr?t||j_q*t||_q*trt|ddrd|j}t|ddrwtjtd d |Dg|d }|D]}||_qmd|_qtj||d |_nd|_d|_d|_||_ dS) Nrzcreating distributed tf.Variable with aggregation=MEAN and a non-floating dtype is not supported, please use a different aggregation or dtype:handleZ%_enable_packed_variable_in_eager_modeFz %s/packed/_varscsrdre)r)rgr/r4r4r5rhriz/DistributedVariable.__init__..r)! variables_libr%r&r'r(r+_distribute_strategy _aggregationrrrPrarsplit _common_namer)r CompositeTensorhasattrweakrefrefrrr Z#executing_eagerly_outside_functionsrpackedZPackedDistributedVariablesum _packed_varZ_keras_initialized_initializer_op_policy) rNr.rOr# var_policyr0rZ packed_varr/rr4r5rPs@     zDistributedVariable.__init__c Cst|j/g}|jD]!}t|j|t||Wdn1s(wYq Wdn1s8wYt ||j||j t|j |d}||t |<|S)abPerform a deepcopy of the `DistributedVariable`. Unlike the deepcopy of a regular tf.Variable, this keeps the original strategy and devices of the `DistributedVariable`. To avoid confusion with the behavior of deepcopy on a regular `Variable` (which does copy into new devices), we only allow a deepcopy of a `DistributedVariable` within its originating strategy scope. Args: memo: The memoization object for `deepcopy`. Returns: A deep copy of the current `DistributedVariable`. Raises: RuntimeError: If trying to deepcopy into a different strategy. N)r.rOr#r) renter_or_assert_strategyrrMr r`appendcopydeepcopyrrrid)rNmemo new_valuesr/Zcopied_variabler4r4r5 __deepcopy__5s"   z DistributedVariable.__deepcopy__cCs|jduo t Sre)rris_saving_non_distributedr[r4r4r5_use_packed_variableXs z(DistributedVariable._use_packed_variablecCsptr |jS|r|jS|j}|jddD] }t||}qtj||jd|d}|S)zIdentifies if all the component variables are initialized. Args: name: Name of the final `logical_and` op. Returns: The op that evaluates to True or False depending on if all the component variables are initialized. rr) rrrais_initializedrrrMr logical_and)rNrresultr0r4r4r5r^s   z"DistributedVariable.is_initializedcCs>tr|jjS|jr|j}|Sttdd|jD}|S)Ncsrdre) initializerrfr4r4r5rhriz2DistributedVariable.initializer..) rrrarrrr"rLrM)rNZinit_opr4r4r5rwszDistributedVariable.initializercC |Sre)r8initialized_valuer[r4r4r5rrz%DistributedVariable.initialized_valuecCs|jduo |jSre)rrr[r4r4r5rz DistributedVariable._is_mirroredcC |jSre)r8 initial_valuer[r4r4r5r z!DistributedVariable.initial_valuecC|jjSre)ra constraintr[r4r4r5rzDistributedVariable.constraintcCrre)rarr[r4r4r5rrzDistributedVariable.graphcCrre)rr[r4r4r5 _shared_namerz DistributedVariable._shared_namecCrre)ra _unique_idr[r4r4r5r rzDistributedVariable._unique_idcCr)z7Lets Optimizers know which graph this variable is from.)ra _graph_keyr[r4r4r5r!szDistributedVariable._graph_keycCrre)rarr[r4r4r5rrzDistributedVariable.namecCrre)rar'r[r4r4r5r'rzDistributedVariable.dtypecCrre)rarr[r4r4r5rrzDistributedVariable.shapecCrre)rasynchronizationr[r4r4r5r"rz#DistributedVariable.synchronizationcCrrerr[r4r4r5r#rzDistributedVariable.aggregationcCs|r|jSdSre)rrr[r4r4r5_packed_variablesz$DistributedVariable._packed_variablecCsDtr|jjSt}|durtd|r|jjS|j|jS)NztDistributedVariable.handle is not available outside the replica context or a `tf.distribute.Strategy.update()` call.) rrrarrSr+rrrMrUr4r4r5rs zDistributedVariable.handlecCs||Sre)r8eval)rNsessionr4r4r5r%rzDistributedVariable.evalcCrre)ra_save_slice_infor[r4r4r5r'rz$DistributedVariable._save_slice_infocC |jSre)ra_get_save_slice_infor[r4r4r5r)rz(DistributedVariable._get_save_slice_infocCs|jD]}||qdSre)rM_set_save_slice_info)rNZsave_slice_infor0r4r4r5r*s  z(DistributedVariable._set_save_slice_infocCrre)r8r`r[r4r4r5r`rzDistributedVariable.devicecCrre)ra trainabler[r4r4r5r+rzDistributedVariable.trainablecCrre)rr[r4r4r5r,rz'DistributedVariable.distribute_strategycCr(re)ra get_shaper[r4r4r5r,rzDistributedVariable.get_shapecCs|jj|dS)N) export_scope)rato_proto)rNr-r4r4r5r.rzDistributedVariable.to_protocCsHtr|jjStrt|jjj|jjj|jjj |jjj S| jSre) rrraoprrrrrrrrWr[r4r4r5r/s zDistributedVariable.opcCrre)ra_in_graph_moder[r4r4r5r0rz"DistributedVariable._in_graph_modecCs$|j|}|r|j|jS|S)z8Returns the value on a device with the given replica_id.)rMrrZ on_devicer`)rNrVr/r4r4r5 _get_replicas z DistributedVariable._get_replicacCs0tr|jSt}|dur|S||SrR)rrrarSrTr1rUr4r4r5rWs  zDistributedVariable._getcCsrtr|jSt}|dur4tt}t|jD]\}}t|j |kr.| |Sq| dS| |S)r]Nr) rrrarSrr^r_rtrMr`r1)rNrVrbror/r4r4r5r8 s  z-DistributedVariable._get_on_device_or_primarycCsPtr |jSt|jt| WdS1s!wYdSre) rrra read_valuerrrridentityrWr[r4r4r5r2s   $zDistributedVariable.read_valuecCs0tr |jS|jr|j|S|Sre)rrrar/rr8r[r4r4r5r/"s    zDistributedVariable.valuecCstr |Std)NzNDistributedVariable.numpy() is only available when eager execution is enabled.)r Zexecuting_eagerlyr2numpyrZr[r4r4r5r4)s zDistributedVariable.numpyFTcCJtr |j||||S|jr|jj|||||dStj|||||dSN use_lockingrr2)rrra assign_subron_write_assign_subrNr/r8rr2r4r4r5r90 zDistributedVariable.assign_subcCr5r6)rrra assign_addron_write_assign_addr;r4r4r5r==r<zDistributedVariable.assign_addcCr5r6)rrraassignron_write_assignr;r4r4r5r?Jr<zDistributedVariable.assigncCDtr |j|||S|jr|jj||||dStj||||dSNr8r)rrra scatter_subrrN sparse_deltar8rr4r4r5rDWzDistributedVariable.scatter_subcCrArB)rrra scatter_addrrEr4r4r5rH`rGzDistributedVariable.scatter_addcCrArB)rrra scatter_mulrrEr4r4r5rIirGzDistributedVariable.scatter_mulcCrArB)rrra scatter_divrrEr4r4r5rJrrGzDistributedVariable.scatter_divcCrArB)rrra scatter_minrrEr4r4r5rK{rGzDistributedVariable.scatter_mincCrArB)rrra scatter_maxrrEr4r4r5rLrGzDistributedVariable.scatter_maxcCrArB)rrrascatter_updaterrEr4r4r5rMrGz"DistributedVariable.scatter_updatecCst|Sre)rrNrr4r4r5__tf_tracing_type__rz'DistributedVariable.__tf_tracing_type__cjffdd }tj|iS)zOverrides Trackable method. This allows both name-based and object-based save and restore of DistributedVariables. Returns: A dictionary mapping attribute names to `SaveableObject` factories. ctj|Sre)_DistributedVariableSaveablerarr[r4r5_saveable_factoryrzODistributedVariable._gather_saveables_for_checkpoint.._saveable_factoryr trackableZVARIABLE_VALUE_KEYrNrSr4r[r5 _gather_saveables_for_checkpoint z4DistributedVariable._gather_saveables_for_checkpointcCs,tr |jS|jr|j|Std)NzDistributedVariable._as_graph_element requires a valid VariablePolicy. Please set the policy via the `var_policy` argument in the constructor, or override this method in sub-classes which support cross-replica accesses.)rrrarrrZr[r4r4r5rs  z%DistributedVariable._as_graph_elementcCs(tr|jS|jr|j|Std)NzDistributedVariable._get_cross_replica requires a valid VariablePolicy. Please set the policy via the `var_policy` argument in the constructor, or override this method in sub-classes which support cross-replica accesses.)rrrarrTrZr[r4r4r5rTs z&DistributedVariable._get_cross_replicacKs"t|jjj|||f|ddS)agApplies updates across replicas. Args: update_fn: A callable to pass to `strategy.extended.update` to update the variable. It should has the same signature as `Variable.assign()`. value: value to be passed to `update_fn`. **kwargs: remaining arguments to `update_fn`. Returns: Updated variable or `tf.Operation`. Tr)rr=r,r:updaterNr2r/r!r4r4r5r-s  z)DistributedVariable._update_cross_replicacKs&|jr|jj|||fi|Std)a@Applies updates in one replica. Args: update_fn: A callable to update the variable. It should has the same signature as `Variable.assign()`. value: value to be passed to `update_fn`. **kwargs: remaining arguments to `update_fn`. Returns: Updated variable or `tf.Operation`. zDistributedVariable._update_replica requires a valid VariablePolicy. Please set the policy via the `var_policy` argument in the constructor, or override this method in sub-classes which support cross-replica accesses.)r_update_replicarZrZr4r4r5r[s  z#DistributedVariable._update_replicacKstr||j|fi|St|jJtrGt}|dur6||}|||fi|WdS|j ||fi|WdSt |j|j ||fi|WdS1sawYdS)aApplies updates depending on the context. The method calls `_update_replica` in replica context, `_update_cross_replica` in cross replica context, and `update_fn` in update context. If `read_value` is True, the method returns the updated Variable. If `read_value` is False, the method returns the update `tf.Operation`. Args: update_fn: A callable to pass to `strategy.extended.update` to update the variable. It should have the same signature as `Variable.assign()`. value: value to be passed to `update_fn`. **kwargs: keyword arguments to `update_fn`. Returns: Updated variable or `tf.Operation`. N) rrrarrr,rZget_update_replica_idr1r-Zassert_replica_contextr[)rNr2r/r!Zupdate_replica_idZ replica_valuer4r4r5r?s  $zDistributedVariable._updatecCr)z6Pass resource_variable_ops.is_resource_variable check.Nr4r[r4r4r5 _should_act_as_resource_variablez4DistributedVariable._should_act_as_resource_variablecCsbtrtj|j|||dSt|jtj||||dWdS1s*wYdS) Converts a variable to a tensor.r'rrN) rrr convert_to_tensorrarrrrWrNr'rrr4r4r5_dense_var_to_tensor s  $z(DistributedVariable._dense_var_to_tensorr'rrcCs |||Srerb)rNr'rr4r4r5 __tf_tensor__s z!DistributedVariable.__tf_tensor__c sjjd|||d|}fddjDD]-}|jr.||jd|||d|q|j||<|jj||j<||jq|j|<|jj|<|jduro|jj|jj <|jj |S)N) object_map tensor_mapoptionscsg|] }|jkr|qSr4)rarfr[r4r5 #szDDistributedVariable._export_to_saved_model_graph..r4) ra_export_to_saved_model_graphrMZexperimental_variable_policyZ_expand_distributed_variablesextendrrrZ packed_handle)rNrerfrgr!Z resource_listr0r4r[r5ris@   z0DistributedVariable._export_to_saved_model_graphcCs t|||t|||dS)aUpdate a SavedObject proto for the caller. If a DistributedVariable object supports this method, it will be called when saving with a pre-built `SavedObject` proto representing the object, plus an instance of `SaveOptions`. This method is then free to modify that proto instance. `DistributedVariable` with `AUTO` or `ON_WRITE` synchronization optionally write out information about their components to the `experimental_distributed_variable_components` field of a `SavedVariable` (depending on the `SaveOptions` variable policy). Args: proto: A pre-built `SavedObject` proto for this object. It is assumed this will be a `SavedVariable` instance. options: A `SaveOptions` instance. N)rZ(write_object_proto_for_resource_variablerZwrite_object_proto)rNprotorgr4r4r5_write_object_proto9sz'DistributedVariable._write_object_protocCrrr4r[r4r4r5is_distributed_variableQr]z+DistributedVariable.is_distributed_variablecCs6|j}|||tjd|g|gddddd|S)NZcaptured_valuecS|gSrer4xr4r4r5r\zIDistributedVariable.__tf_experimental_restore_capture__..cSrnrer4ror4r4r5r]rq)Zbackward_functionZforward_function)rZreplace_capturer Zrecord_operation)rNZconcrete_functionZinternal_capturerr4r4r5#__tf_experimental_restore_capture__Us  z7DistributedVariable.__tf_experimental_restore_capture__reFNTFNNNF)NN)NNN)Irvrzr{r|rPrrrr}rrrrrrrr r!rr'rr"r#r$rr%r'r)r*r`r+r,r,r.r/r0r1rWr8r2r/r4r9r=r?rDrHrIrJrKrLrMrOrWrrTr-r[r?r\rbrr ZDTypestr tensor_libTensorrdrirlrmrrrr4r4rr5rs;#                              !    " rc(eZdZdZfddZddZZS)rRz8Class for defining how to restore a DistributedVariable.csB||_|jjs td|j|||\}}tt||||dS)NzThe VariablePolicy of the argument `distributed_variable` must be set to create a _DistributedVariableSaveable. Please set it via the `var_policy` argument in the constructor of DistributedVariable.)_distributed_variablerr+ get_saveablerrRrP)rNrprimary_variablerrspecrr4r5rPksz%_DistributedVariableSaveable.__init__cCs|\}|jj|j|Sz*Restore the same value into all variables.)rzrget_restore_opsrNZrestored_tensorsZrestored_shapesrr4r4r5restorewsz$_DistributedVariableSaveable.restorervrzr{r|rPrrr4r4rr5rRhs  rRcry)_MirroredSaveablez5Class for defining how to restore a MirroredVariable.cs2||_t|j||\}}tt||||dSre)_mirrored_variablerget_on_write_saveablerrrP)rNZmirrored_variabler|rrr}rr4r5rPs z_MirroredSaveable.__init__cCs|\}t|j|Sr~)rget_on_write_restore_opsrrr4r4r5rsz_MirroredSaveable.restorerr4r4rr5r~s rcsjeZdZdZddZddZfddZfdd Zfd d Zd d Z ddZ ddZ dddZ Z S)MirroredVariablezDHolds a map from replica to variables whose values are kept in sync.cCs t|Sre)rrr[r4r4r5rrzMirroredVariable._is_mirroredcKst|||fi|Sre)rBrZr4r4r5r[rz MirroredVariable._update_replicacbtr |jj|i|S|jtjjkr&|jtjjkr&t tj j d|jdt t |j|i|S)NrKZop_namer#)rrrarKrr$r%rGr7rZscatter_error_msgrrrrNr r!rr4r5rK zMirroredVariable.scatter_mincr)NrLr)rrrarLrr$r%rGr7rZrrrrrrr4r5rLrzMirroredVariable.scatter_maxcr)NrMr)rrrarMrr$r%rGr7rZrrrrrrr4r5rMrzMirroredVariable.scatter_updatecCstt|Sre)rr3rrTr[r4r4r5rTsz#MirroredVariable._get_cross_replicacCrrer8rr[r4r4r5rrz"MirroredVariable._as_graph_elementcrP)zOverrides Trackable method. This allows both name-based and object-based save and restore of MirroredVariables. Returns: A dictionary mapping attribute names to `SaveableObject` factories. crQre)rrarr[r4r5rSrzLMirroredVariable._gather_saveables_for_checkpoint.._saveable_factoryrTrVr4r[r5rWrXz1MirroredVariable._gather_saveables_for_checkpointNFcCs"|rtdtj||||dS)r^zYou may be using variable created under distribute strategy in TF 1.x control flows. Try explicitly converting the variable to Tensor using variable.read_value(), or switch to TF 2.x.r_)r+r r`rWrar4r4r5rbs z%MirroredVariable._dense_var_to_tensorru)rvrzr{r|rr[rKrLrMrTrrWrbrr4r4rr5rs   rcry)_SyncOnReadSaveablez7Class for defining how to restore a SyncOnReadVariable.cs2||_t||j|\}}tt||||dSre)_sync_on_read_variablerget_on_read_saveablerarrrP)rNZsync_on_read_variablerrr}rr4r5rPs z_SyncOnReadSaveable.__init__cCs|\}t|j||jjSr~)rget_on_read_restore_opsrr#rr4r4r5rs z_SyncOnReadSaveable.restorerr4r4rr5rs rcseZdZdZddZfddZd+fd d Zd+fd d Zd+fd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZdd Zfd!d"Zd#d$Zd%d&Zd'd(Zd,d)d*ZZS)-SyncOnReadVariablezGHolds a map from replica to variables whose values are reduced on save.cKs|||fi|SrerrZr4r4r5r[z"SyncOnReadVariable._update_replicacs>t|jtt|WdS1swYdS)aReturns the value of SyncOnReadVariable based on surrounding context. If called under a non-default replica-context, returns the corresponding variable on that replica. If called under default replica-context or cross-replica context, returns the synced value. N)rrrrrrWr[rr4r5rWs $zSyncOnReadVariable._getFNTctr |j||||St|j.tr/ts/t tj |||dWdSt t |||||WdS1sDwYdSNr2) rrrar9rrrrin_replica_update_contextr= on_read_assign_sub_cross_replicarrr;rr4r5r9  $zSyncOnReadVariable.assign_subcrr) rrrar=rrrrrr= on_read_assign_add_cross_replicarrr;rr4r5r=rzSyncOnReadVariable.assign_addcrr) rrrar?rrrrrr=on_read_assign_cross_replicarrr;rr4r5r?s$zSyncOnReadVariable.assigncCtd|d)Nz:Variables with `synchronization=ON_READ` doesn't support ``rYrNmethodr4r4r5_scatter_not_implemented)s z+SyncOnReadVariable._scatter_not_implementedcO(tr |jj|i|S|ddSNrD)rrrarDrrr4r4r5rD-zSyncOnReadVariable.scatter_subcOrNrH)rrrarHrrr4r4r5rH2rzSyncOnReadVariable.scatter_addcOrNrI)rrrarIrrr4r4r5rI7rzSyncOnReadVariable.scatter_mulcOrNrJ)rrrarJrrr4r4r5rJ<rzSyncOnReadVariable.scatter_divcOrNrK)rrrarKrrr4r4r5rKArzSyncOnReadVariable.scatter_mincOrNrL)rrrarLrrr4r4r5rLFrzSyncOnReadVariable.scatter_maxcOrNrM)rrrarMrrr4r4r5rMKrz!SyncOnReadVariable.scatter_updatecCstrtdtr|jSt|j6t r@t s@|j t j jkr5|dWdS|WdS|WdS1sPwYdS)NzMcall `variable.value()` inside variable_sync_on_read_context is not supportedr)r in_variable_sync_on_read_contextrZrrrar/rrrrrr$r%rGr1rTr8r[r4r4r5r/Ps"   $zSyncOnReadVariable.valuecstrtdtS)NzRcall `variable.read_value()` inside variable_sync_on_read_context is not supported)rrrZrr2r[rr4r5r2as  zSyncOnReadVariable.read_valuecCsz|jtjjkr |dS|jtjjkrtt |j |j j t j |j|ddWdS1s6wYdSNr)Zaxis)rr$r%rGr1SUMrr=rrrreducerrHrIr[r4r4r5rThs  $z%SyncOnReadVariable._get_cross_replicacCsntr |jSt|jtr"t | WdSWdn1s,wY| Sre) rrrarrrrrr r`rTrWr[r4r4r5rus   z$SyncOnReadVariable._as_graph_elementcrP)zOverrides Trackable method. This allows both name-based and object-based save and restore of `SyncOnReadVariable`s. Returns: A dictionary mapping attribute names to `SaveableObject` factories. cs t|Sre)rrr[r4r5rSrzNSyncOnReadVariable._gather_saveables_for_checkpoint.._saveable_factoryrTrVr4r[r5rW~rXz3SyncOnReadVariable._gather_saveables_for_checkpointcCstrtj|j|||dSt|jet}|durgt rg|j t j j kr;tj|d|||dWdS|j t j jkrFt|jjtj|j |}tj||||dWdStj||||dWdS1s|wYdS)z*Converts a SyncOnReadVariable to a tensor.r_Nr)rrr r`rarrrr>rrr$r%rGr1rr=r.r:rJrrHrIrWr2)rNr'rrrZreducedr4r4r5rbs>   $z'SyncOnReadVariable._dense_var_to_tensorrsru)rvrzr{r|r[rWr9r=r?rrDrHrIrJrKrLrMr/r2rTrrWrbrr4r4rr5rs*        rcC|j|||dSNr_rcr3r'rrr4r4r5"_tensor_conversion_distributed_varsrcCrrrcrr4r4r5_tensor_conversion_mirroredrrcCstj||||dSr)r r`rW)r/r'rrr4r4r5_tensor_conversion_mirrored_vals rcCrrrcrr4r4r5_tensor_conversion_sync_on_readrrc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)VariablePolicyaPolicy defining synchronization and aggregation of a distributed variable. Given `synchronization` and `aggregation` parameters set on a `tf.Variable` during variable creation within `tf.distribute` scope, `tf.distribute` creates an appropriate policy object and assigns it to the distributed variable. All variable operations are delegated to the respective policy object. cCs ||_dSrer#)rNr#r4r4r5rPrzVariablePolicy.__init__cCrX)Nz8VariablePolicy.value should be overriden by sub-classes.rYr[r4r4r5r/r\zVariablePolicy.valuecCrX)Nz?VariablePolicy._is_mirrored should be overriden by sub-classes.rYr[r4r4r5rr\zVariablePolicy._is_mirroredcCrX)NzDVariablePolicy._as_graph_element should be overriden by sub-classes.rYrNr4r4r5rr\z VariablePolicy._as_graph_elementcCrX)NzEVariablePolicy._get_cross_replica should be overriden by sub-classes.rYrNr3r4r4r5rTr\z!VariablePolicy._get_cross_replicacKrX)NzBVariablePolicy._update_replica should be overriden by sub-classes.rYrNr3r2r/r!r4r4r5r[r\zVariablePolicy._update_replicaN) rvrzr{r|rPr/rrrTr[r4r4r4r5rs rc@seZdZdZddZddZddZdd Zd d Zd d Z   d)ddZ   d)ddZ d)ddZ ddZ ddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(ZdS)* OnReadPolicya{Policy defined for `tf.VariableSynchronization.ON_READ` synchronization. This policy is created when `synchronization` is set to `tf.VariableSynchronization.ON_READ` and `aggregation` is set to any of the values allowed by the `tf.VariableAggregation` enum such as `NONE`, `SUM`, `MEAN` or `ONLY_FIRST_REPLICA`when creating a `tf.Variable` in `tf.distribute` scope. cCrrr4r[r4r4r5rrzOnReadPolicy._is_mirroredcCst|j6tr/ts/|jtjj kr$| d WdS| WdS| WdS1s?wYdS)Nr)rrr,rrrrr$r%rGr1r/rTr8rr4r4r5r/s  $zOnReadPolicy.valuecCs\t|jtrt|WdSWdn1s#wY|Sre) rrr,rr r`rTrWrrr4r4r5r s  zOnReadPolicy._as_graph_elementcCsz|jtjjkr |dS|jtjjkrtt |j |j j t j |j|ddWdS1s6wYdSr)rr$r%rGr1rrr=rrr,rrrHrIrr4r4r5rTs  $zOnReadPolicy._get_cross_replicacKs|||fi|Srerrr4r4r5r[rzOnReadPolicy._update_replicacCr)Nz#ON_READ variables doesn't support `z` in cross replica contextrYrr4r4r5rrz%OnReadPolicy._scatter_not_implementedFNTcCzt|j-tr"ts"ttj|||dWdStj|||||dWdS1s6wYdS)z%Subtracts a value from this variable.rNr7) rrr,rrrr=rr:rNr3r/r8rr2r4r4r5r9""$zOnReadPolicy.assign_subcCr)zAdds a value to this variable.rNr7) rrr,rrrr=rr>rr4r4r5r=7rzOnReadPolicy.assign_addcCszt|j-tr"ts"ttj|||dWdStj|||||dWdS1s6wYdS)Nrr7) rrr,rrrr=rr@rr4r4r5r?Ls"$zOnReadPolicy.assigncO~~|ddSrrrr4r4r5rD[zOnReadPolicy.scatter_subcOrrrrr4r4r5rH_rzOnReadPolicy.scatter_addcOrrrrr4r4r5rIcrzOnReadPolicy.scatter_mulcOrrrrr4r4r5rJgrzOnReadPolicy.scatter_divcOrrrrr4r4r5rKkrzOnReadPolicy.scatter_mincOrrrrr4r4r5rLorzOnReadPolicy.scatter_maxcOrrrrr4r4r5rMsrzOnReadPolicy.scatter_updatecCt|||S)z0Create a saveable object for the given variable.)rrrNr3Z primary_varrr4r4r5r{wrQzOnReadPolicy.get_saveablecCst|||jSr~)rrrrNr3rr4r4r5r{szOnReadPolicy.get_restore_opsrs)rvrzr{r|rr/rrTr[rr9r=r?rDrHrIrJrKrLrMr{rr4r4r4r5rs4      rc@seZdZdZddZddZddZdd Zd d Zd'ddZ d'ddZ d'ddZ d(ddZ d(ddZ d(ddZd(ddZd(ddZd(dd Zd(d!d"Zd#d$Zd%d&Zd S)) OnWritePolicyaaPolicy defined for `tf.VariableSynchronization.ON_WRITE` synchronization. This policy is created when the following `synchronization` and `aggregation` parameters are specified when creating a `tf.Variable` in `tf.distribute` scope and `synchronization` is equal to `tf.VariableSynchronization.ON_WRITE` or `tf.VariableSynchronization.AUTO`. cCrrr4r[r4r4r5rrzOnWritePolicy._is_mirroredcC |Sre)r8r/rr4r4r5r/rzOnWritePolicy.valuecCrrerrr4r4r5rrzOnWritePolicy._as_graph_elementcCst|Sre)rr3r8rr4r4r5rTsz OnWritePolicy._get_cross_replicacKs8|jtjjkr|||fi|St|||fi|Sre)r#rr%r7r8rBrr4r4r5r[szOnWritePolicy._update_replicaFNTcCtj|||||dSr6)rr@rr4r4r5r?s zOnWritePolicy.assigncCrr6)rr>rr4r4r5r= zOnWritePolicy.assign_addcCrr6)rr:rr4r4r5r9rzOnWritePolicy.assign_subcCtj||||dSrB)rrDrNr3rFr8rr4r4r5rDzOnWritePolicy.scatter_subcCrrB)rrHrr4r4r5rHrzOnWritePolicy.scatter_addcCrrB)rrIrr4r4r5rIrzOnWritePolicy.scatter_mulcCrrB)rrJrr4r4r5rJrzOnWritePolicy.scatter_divcCD|jtjjkr|jtjjkrttjjd|jdtj ||||dS)NrKrrC) rr$r%rGr7rZrrrrKrr4r4r5rK zOnWritePolicy.scatter_mincCr)NrLrrC) rr$r%rGr7rZrrrrLrr4r4r5rLrzOnWritePolicy.scatter_maxcCr)NrMrrC) rr$r%rGr7rZrrrrMrr4r4r5rMrzOnWritePolicy.scatter_updatecCr)z Saveable ops for AUTO variables.)rrrr4r4r5r{rQzOnWritePolicy.get_saveablecCs t||Sre)rrrr4r4r5rrzOnWritePolicy.get_restore_opsrsrt)rvrzr{r|rr/rrTr[r?r=r9rDrHrIrJrKrLrMr{rr4r4r4r5rs2         rcs<eZdZdZddZfddZfddZdd ZZS) PerWorkerResourceaA per-worker CapturableResource class for non-ParameterServer strategy. Resources that populate `host_to_resources` should be instances of classes subclassing CapturableResource, although currently it's only used and tested for StaticHashTable with TPUStrategy. cCs$tjddd||_||_dS)NrZTPUDistributedLookupTabler)rZ'distribution_strategy_input_api_counterZget_cellZ increase_by _strategy_host_to_resources)rNr.Zhost_to_resourcesr4r4r5rPs  zPerWorkerResource.__init__cs&|dvr t||Stt||S)N)rP__getattribute__rrlocal_resource)rrrrrrrr4r5rsz"PerWorkerResource.__getattribute__cs*|dvr t|||Stt|||S)N)rr)setattrrrr __setattr__)rNrr/rr4r5rszPerWorkerResource.__setattr__cCs<tt}tt|}|j||jtt|jS)z)Returns the resource on the local worker.)rr^r_Zget_host_for_devicergetnextiter)rNrbZ host_devicer4r4r5rsz PerWorkerResource.local_resource) rvrzr{r|rPrrrrr4r4rr5rs   rru)Sr|r typingrrZtensorflow.core.protobufrZtensorflow.python.distributerrrrrrZtensorflow.python.eagerr r Ztensorflow.python.frameworkr r r rrwrrrZtensorflow.python.opsrrrrrr$rrZtensorflow.python.saved_modelrZtensorflow.python.trackablerrUZ!tensorflow.python.training.savingrZtensorflow.python.typesrrZds_typesrrBr<rEr~rr*rZ#register_tensor_conversion_functionZTypeSpecrZregister_codecZBuiltInTypeSpecCodecZ TypeSpecProtoZPER_REPLICA_SPECrobjectrZ TraceTyperVariablerxrZSaveableObjectrRrrrrrrrrrrrrr4r4r4r5s                            7>   !vP>    !e