o ?e?@sdZddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd l mZdd lmZdd lmZd d ZddZGddde jZGddde jZddZddZdS)z(Implementation of LoadDataset in Python.N)message) text_format) snapshot_pb2)_pywrap_snapshot_utils) dataset_ops)structured_function)gen_experimental_dataset_ops)gfile)nested_structure_codercsfdd}|durdd}|dur6ttjtjd }|}Wdn1s-wYt|}|}|rHt |||t ||St |||S)z$Loads dataset from tf.data snapshot.c sjz%ttd}t|tWdWS1swYWdStjt j t fy4YdSw)zReads the distributed snapshot metadata. Returns: DistributedSnapshotMetadata if the snapshot is a distributed snapshot. Returns None if it is a non-distributed snapshot. rN) r GFilerZ TF_DATA_SnapshotMetadataFilePathrZ ParseLinesrZDistributedSnapshotMetadataZ ParseErrorr DecodeErrorUnicodeDecodeError)fpathc/home/www/facesmatcher.com/pyenv/lib/python3.10/site-packages/tensorflow/python/data/ops/load_op.py"_get_distributed_snapshot_metadata"s (z1_load.._get_distributed_snapshot_metadataNcSs|jddttjdS)NcSs|SNr)xrrr4sz)_load....)Z cycle_lengthZnum_parallel_calls) interleavemultiprocessing cpu_countrZAUTOTUNE)Zdatasetsrrrr3s z_load..rb) r r osrjoinrZDATASET_SPEC_FILENAMEread_parse_element_spec_validate_snapshot_load_distributed_snapshot _LoadDataset)r element_spec compression reader_funcrrZ encoded_specZdistributed_snapshot_metadatarrr_loads(  r&csHt|fddtD}tj|}|fdd}||S)zLoads a distributed snapshot.csg|] }tj|qSr)rrr).0r) chunks_dirrr Ksz._load_distributed_snapshot..cst|tjjdS)N)r#r$)_SnapshotChunkDatasetrr#r$) chunk_file)metadatarrrOs z,_load_distributed_snapshot..)rZ TF_DATA_CommittedChunksDirectoryr Z ListDirectoryrZDatasetZfrom_tensor_slicesmap)rr,r%Z chunk_filesZdatasetr)r(r,rr!Gs    r!c,eZdZdZfddZeddZZS)r"z.A dataset that loads previously saved dataset.csh||_||_||_tj|dtt|jd|_tj |f|jj j ||jj d|j }t |dS)Nzload())Zinput_structure)Zreader_func_other_argsr$r%)_path _element_spec _compressionrZStructuredFunctionWrapperrZ DatasetSpecZ _reader_funcged_opsZ load_datasetfunctionZcaptured_inputs_flat_structuresuper__init__)selfrr#r$r%variant_tensor __class__rrr6Ys( z_LoadDataset.__init__cC|jSrr0r7rrrr#lz_LoadDataset.element_spec__name__ __module__ __qualname____doc__r6propertyr# __classcell__rrr9rr"Vs  r"cr.)r*zAA dataset for one chunk file from a tf.data distributed snapshot.cs4||_||_tj|fd|i|j}t|dS)Nr$)Z _chunk_filer0r2Zsnapshot_chunk_datasetr4r5r6)r7r+r#r$r8r9rrr6tsz_SnapshotChunkDataset.__init__cCr;rr<r=rrrr#}r>z"_SnapshotChunkDataset.element_specr?rrr9rr*qs   r*cCst|s td|dt|}t|r2t|d }td|d|1s-wYt|}t|sDtd|dt|j }|r]||kr]td|d|d|d |rs||j krutd|d |d |j d dSdS) a8Validates a tf.data distributed snapshot. Args: path: Root path of the distributed snapshot. metadata: The DistributedSnapshotMetadata of the snapshot. element_spec: Dataset element_spec. compression: Compression method used for saving. Raises: ValueError if the snapshot is invalid. z#Failed to load tf.data snapshot at z(: The snapshot directory does not exist.r z+. The save job failed to write it. Status: Nz5. The save job has not finished writing the snapshot.z. User specified element_spec z!, but the actual element_spec is .z. User specified compression z , but the actual compression is ) r ZExists ValueErrorrZTF_DATA_SnapshotErrorFilePathr rZTF_DATA_SnapshotDoneFilePathrr#r$)rr,r#r$Z error_filerZ done_fileZsnapshot_element_specrrrr sH         r cCstj}||t|Sr)r Z struct_pb2ZStructuredValueZParseFromStringZ decode_proto)Zencoded_element_specZ struct_pbrrrrs   r)rCrrZgoogle.protobufrrZtensorflow.core.protobufrZ+tensorflow.python.data.experimental.servicerZtensorflow.python.data.opsrrZtensorflow.python.opsrr2Ztensorflow.python.platformr Ztensorflow.python.saved_modelr r&r!Z DatasetSourcer"r*r rrrrrs$         ( -