o ?e-@sdZddlmZmZmZddlmZddlZddl m Z ddl m Z ddl m Z ddl m Z dd l mZdd lmZdd lmZdd lmZdd lmZddlmZdedededede jddf ddZ  d+deeedeedeeejeffddZedgddddde jfdeeeeefdedeeedeede de jf d d!Z!ed"gdddde jfdeeeefded#eeedeede de jf d$d%Z"iZ#ed&gd  d+de jd'eed(eefd)d*Z$dS),z%Utilities to help with mesh creation.)ListOptionalTuple)loggingN)accelerator_util)api)config)layout)tpu_util)context)device) array_ops)math_ops) tf_exportnum_global_devices num_clients client_id device_typemeshreturncCsTtd||td||td|td|td|dS)NzThis is client %d of %d clientszNumber of global %s devices: %dzGlobal device IDs: %szLocal device IDs: %szLocal devices: %s)rinfoupperglobal_device_idslocal_device_ids local_devicesrrrrrrd/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/dtensor/python/mesh_util.py_print_context!s rdevicescCs||dur|dur d}t|}||fSdd|D}|dur#|dj}||djkr:tdt|d|||fS)zHMakes device specs from local devices names or number of global devices.NCPUcSsg|]}tj|qSr) tf_device DeviceSpecZ from_string.0drrr 8sz&_make_device_specs..rzConflicting devices z and device_type )rrrr ValueErrorstr)rrrrr_make_device_specs-s   r)z experimental.dtensor.create_mesh)v1 mesh_dims mesh_name use_xla_spmdc st||\}}tjtdddfdd|D}|dur&dt|fg}nt|dkr@|dddkr@|ddt|f|d<d d|D}d d|D}t|t|krctd t|d |t t| |}t | } t j||| |||d } tt|dd|| d| S)aCreates a single-client mesh. If both `mesh_dims` and `devices` are specified, they must match each otehr. As a special case, when all arguments are missing, this creates a 1D CPU mesh with an empty name, assigning all available devices to that dimension. Args: mesh_dims: A list of (dim_name, dim_size) tuples. Defaults to a single batch-parallel dimension called 'x' using all devices. As a special case, a single-element mesh_dims whose dim_size is -1 also uses all devices. mesh_name: Name of the created mesh. Defaults to ''. devices: String representations of devices to use. This is the device part of tf.DeviceSpec, e.g. 'CPU:0'. Defaults to all available logical devices. device_type: If `devices` is missing, the type of devices to use. Defaults to 'CPU'. use_xla_spmd: Boolean when True, will use XLA SPMD instead of DTensor SPMD. Returns: A single-client mesh created from specified or default arguments. rZjobZreplicataskcg|]}|qSrZmake_merged_specr#Z local_specrrr&azcreate_mesh..NxcSg|]}|dqS)rrr#rrrr&icSr8)r6rr#rrrr&jr9zlength of devices (3) must be equal to total size of the mesh of shape  dim_namesrrrr-r.r)r)r!r"rjob_namelennpprodr'arangereshaperaveltolistr Meshr) r,r-rrr. device_specsr<shaperrrrr3r create_meshCs>rHz,experimental.dtensor.create_distributed_meshrc st|\}}tstd|r)|dkr)|dur)td|d|dt||\}}|dvrtjt dt d fd d |D}t |t }t ||krntd t |d t d|d|t ||} t | } t |t } | | | t |} tj|| | |||d} t|t t || | S|dkrtj||||d} tt|t t || | Std|d)aCreates a distributed mesh. This is similar to `create_mesh`, but with a different set of arguments to create a mesh that spans evenly across a multi-client DTensor cluster. For CPU and GPU meshes, users can choose to use fewer local devices than what is available `local_devices`. For TPU, only meshes that uses all TPU cores is supported by the DTensor runtime. Args: mesh_dims: A list of (dim_name, dim_size) tuples. mesh_name: Name of the created mesh. Defaults to ''. local_devices: String representations of devices to use. This is the device part of tf.DeviceSpec, e.g. 'CPU:0'. Defaults to all available local logical devices. device_type: Type of device to build the mesh for. Defaults to 'CPU'. Supported values are 'CPU', 'GPU', 'TPU'.6 use_xla_spmd: Boolean when True, will use XLA SPMD instead of DTensor SPMD. Returns: A mesh that spans evenly across all DTensor clients in the cluster. zYAccelerators are uninitialized, please run dtensor.initialize_accelerator_system() first.ZTPUNzDo not specify devices for z- meshes. Using a partial list of devices for z is not supported.)r ZGPUrr/cr1rr2r#r3rrr&r4z+create_distributed_mesh..zGlobal number of devices (z per client * z clients = r:r;)Zmesh_dim_namesZ mesh_shaper-r.z Device type z is not CPU, GPU or TPU)ziprZis_initializedr'rr)r!r"rr=rr>rr?r@rArBrCrDr rErr Zcreate_tpu_meshr)r,r-rrr.r<rGrFrrZ flattenedZ start_idxrrrr3rcreate_distributed_meshst     rJzexperimental.dtensor.barrier barrier_name timeout_in_mscCs|durd}td|ttddgt|}t |g| t |j |}t|}||jkr@td||jttjrj|durOd}t|d}|dt|<|d |}t||td |dS) aRuns a barrier on the mesh. Upon returning from the barrier, all operations run before the barrier would have completed across all clients. Currently we allocate a fully sharded tensor with mesh shape and run an all_reduce on it. Example: A barrier can be used before application exit to ensure completion of pending ops. ```python x = [1, 2, 3] x = dtensor.relayout(x, dtensor.Layout.batch_sharded(mesh, 'batch', 1)) dtensor.barrier(mesh) # At this point all devices on all clients in the mesh have completed # operations before the barrier. Therefore it is OK to tear down the clients. sys.exit() ``` Args: mesh: The mesh to run the barrier on. barrier_name: The name of the barrier. Mainly used for logging purpose. timeout_in_ms: The timeout of the barrier in ms. If omitted, blocks indefinitely till the barrier is reached from all clients. Nz (barrier)zentering barrier before op: %sg?r6zMGlobal barrier produced wrong mesh size : {0} while mesh has actualsize : {1}i\&r:z8finished running barrier across all clients after op: %s)rrr Z async_waitr rBr>rGrpackZnum_local_devicesr ZLayoutr<rZ reduce_sumsizer'formatZcoordination_service _BARRIER_DICT setdefaultZwait_at_barrier)rrKrL componentZonesZ mesh_sizeZ num_callsZ barrier_idrrrbarriers2         rT)NN)%__doc__typingrrrZabslrnumpyr?Ztensorflow.dtensor.pythonrrrr r Ztensorflow.python.eagerr Ztensorflow.python.frameworkr r!Ztensorflow.python.opsr rZ tensorflow.python.util.tf_exportrintr(rErr"r)Z USE_XLA_SPMDboolrHrJrQrTrrrrs                 >  b