\;jOddlZddlZddlZddlZddlZddlZddlZddlmZm Z m Z ddl m Z m Z ddlmZmZmZddlmZddlmZmZmZd Zdad ad add Zd ZdZGddZGddZ y)N)_current_expected_place_get_paddle_place_get_paddle_place_list)corein_dynamic_mode) BatchSamplerIterableDatasetSubset)_InfiniteIterableSampler)_DataLoaderIterMultiProcess_DataLoaderIterSingleProcess _DatasetKind<Fc |a|ayN) USE_AUTOTUNE TUNING_STEPS) use_autotune tuning_stepss YG:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/io/reader.pyset_autotune_configr0s LLc|t|dk(rtSt|dk(rt|dtsJ|day)Nrr )lenUSE_PINNED_MEMORY isinstancebool)argss ruse_pinned_memoryr"7s; 4yA~  4yA~*T!Wd";;; Grct|ttfs|g}g}|D]T}t|tjs'tj}|j ||}|j |V|Sr)rlisttuplerPlace set_placeappend)placesretptmps r_convert_placesr-@sb ftUm , C !TZZ(**,C MM! A 1  Jrc6eZdZdZdZdZdZdZdZdZ y) AuToTunecJ||_tjdz |_y)Nr)loadermultiprocessing cpu_countmax_num_worker)selfr1s r__init__zAuToTune.__init__Ps -779A=rctr|js|jjS|j }||jjSt j }t jdt jdt|jjzd}td}t jdt|jzd}||jkr||_|j|}|dz|kDr|}|}n&|j||||j}||k(rnG|}t jdt|zdzt|z|d z }||jkrt jd t|zt jd tt j |z zd z|S) Nz(========= DataLoader Auto Tune =========zUser config for DataLoader: rinfz"Tuning Range for num_workers: 0 ~ g?z num_workers: avg_cost: rz'auto_tune dataLoader best_num_workers: z AutoTuning Cost for DataLoader: z seconds)r need_autotuner1 num_workersget_autotune_loadertimeloggingdebugstrfloatr4evaluate_reader_costis_bestinfo)r5auto_tune_loaderauto_tune_startbest_num_workersmin_costr;avg_cost update_nums r__call__zAuToTune.__call__Tsd&8&8&:;;** * 335  #;;** *))+ @A *S1H1H-I I < 03t7J7J3K K  D///+6  (001ABH$)##. !\\$$''  !11'1$ MMk"# h-   1 K/D///0  5rc|t|tzt|}t|t t |}|S)N)indices)minrrr r$range)r5dataset batch_size num_samples sub_datasets rget_sub_datasetzAuToTune.get_sub_datasets3*|3S\B Wd53E.FG rctj|j}|jjj}t |jjt j jr|jjj}|j||}t j j|||jjj|jjj|jjj|jjj|_|St |jjt j jr|jjjj }|j||}t j j|||jjj|_|Sd}|S)N)rVrW num_replicasrankshuffle drop_last)rVrWr_)copyr1 batch_samplerrWrpaddleioDistributedBatchSamplerrVrZnranks local_rankr^r_r sampler data_source)r5r1rWrVrYs rr<zAuToTune.get_autotune_loaders~4;;'[[..99  KK % %vyy'H'H kk//77G..w CK#)99#D#D#%![[66==[[..99 1199++33== $E$F $  116993I3I Jkk//77CCG..w CK#)99#9#9#%++33==$:$F  F rcdg}d}tj}t|D]?\}}|jtj|z tj}At|dkDrt |ddt|ddz }|St |ddt|ddz }|S)Nrr)r= enumerater(rsum)r5readercostsrIstartidatas rrBzAuToTune.evaluate_reader_costs  (GAt LLu, -IIKE) u:>59~E!"I6H59~E!"I6Hrcd}|dz}d}||kr~|dkry||j_|j|}tjdt |zdzt |z|dz }||dz|zkr|S|dz }|dz}||kr|dkry|S)Nrr zfor back num_workers: r9gffffff?g?)r1r;rBr>r?r@) r5rl best_workers best_timenum_work_boundarystepr;boundaryr=s rrCzAuToTune.is_bests"Q& --$(&1DKK #,,V4D MM(k"# d)  AIDi$&11""q  H--$(rN) __name__ __module__ __qualname__r6rKr:rZr<rBrCrrr/r/Os&>8 t  8 rr/cHeZdZdZ ddZdZdZdZy) DataLoadera DataLoader prodives an iterator which iterates given dataset once by the batch_sampler. DataLoader supports single-process and multi-prcess data loading, multi-process workers will be used to load data asynchronously if :attr:`num_workers` is set as a positive number. DataLoader supports map-style dataset and iterable-style dataset. For map-style datast(can get a sample from dataset with a given index), please see :code:`paddle.io.Dataset`. For iterable-style datast(get samples from dataset iteratively, like a Python iterator), please see :code:`paddle.io.IterableDataset`. For :code:`batch_sampler` please see :code:`paddle.io.BatchSampler` Notes: GPU tensor operation is not supported in subprocess currently, please don't use GPU tensor operations in pipeline which will be performed in subprocess, such as dataset transforms, collte_fn, etc. Numpy array and CPU tensor operation is supported. **Disable automatic batching** In certain cases such as some NLP tasks, instead of automatic batching, handling batching manually in dataset is needed by users. For these cases, automatic batching is disabled if both :attr:`batch_size` and :attr:`batch_sampler` is set as None, each data got from :attr:`dataset` should be batched data and will be processed with function define by :attr:`collate_fn` or :attr:`default_collate_fn`. Notes: When automatic batching is disabled, :attr:`default_collate_fn` will do nothing to data from dataset. Args: dataset(Dataset): the dataset to load data from, should be an instance of subclass of :code:`paddle.io.Dataset` or :code:`paddle.io.IterableDataset`. feed_list (list(Tensor)|tuple(Tensor), optional): feed Tensor list. The Tensors should be created by :code:`paddle.static.data()`. :attr:`feed_list` must be set if :attr:`return_list` is False. Default None. places(list(Place)|tuple(Place)|list(str), optional): a list of Place, to put data onto, :attr:`places` can be None, if :attr:`places` is None, default place(CPUPlace or CUDAPlace(0)) will be used. Default None. If ``places`` is list of string, the string in the list can be ``cpu``, ``gpu:x`` and ``gpu_pinned``, where ``x`` is the index of the GPUs. return_list (bool, optional): whether the return value on each device is presented as a list. If :attr:`return_list=False`, the return value on each device would be a dict of str -> Tensor, where the key of the dict is the name of each fed Tensors. If :attr:`return_list=True`, the return value on each device would be a list(Tensor). :attr:`return_list` can only be True in dynamic graph mode. Default True. batch_sampler(BatchSampler, optional): an instance of `paddle.io.BatchSampler` to generate batch indices to draw samples from :attr:`dataset` and combine a batch. Default None. batch_size(int|None, optional): sample number in a mini-batch, a substitution parameter for :attr:`batch_sampler`, if :attr:`batch_sampler` is not set, a default `paddle.io.BatchSampler` will be used and initialize by :attr:`batch_size`, :attr:`shuffle` and :attr:`drop_last`. Default 1. shuffle(bool, optional): whther to shuffle indices order before genrate batch indices, a substitution parameter for :attr:`batch_sampler` see :attr:`batch_size`. Default False. drop_last(bool, optional): whether drop the last incomplete batch dataset size is not divisible by the batch size, a substitution parameter for :attr:`batch_sampler`, see :attr:`batch_size`. Default False collate_fn(callable, optional): function to generate mini-batch data by merging the sample list, None for only stack each fields of sample in axis 0(same as :attr::`np.stack(..., axis=0)`). Default None num_workers(int, optional): the number of subprocess to load data, 0 for no subprocess used and loading data in main process. Default 0 use_buffer_reader (bool, optional): whether to use bufferred reader. If use_buffer_reader=True, the DataLoader would prefetch batch data asynchronously, so it would speed up data feeding and occupies a little more CPU or GPU memory, i.e., the memory of one batch input data. Default True. prefetch_factor (int, optional): Number of batch data the DataLoader would prefetch if use_buffer_reader=True. Default 2. use_shared_memory (bool, optional): whether to use shared memory to speed up putting data into inter-process queue, set :attr:`use_shared_memory` as True only when the shared memory space on your machine(e.g. space of '/dev/shm' on Linux operating sysytem) is large enough. Shared memory will only be enabled in multi-process mode(num_workers > 0). Default True. timeout(int, optional): the timeout value for getting data form output queue of subprocesses. Default 0. worker_init_fn(callable, optional): init function which will be called with worker id on each subproces starting if not set as None. Default None. Returns: DataLoader: an iterable object for data iterating, each elemnet of the generated data is a Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> import paddle.nn as nn >>> import paddle.nn.functional as F >>> from paddle.io import Dataset, BatchSampler, DataLoader >>> BATCH_NUM = 20 >>> BATCH_SIZE = 16 >>> EPOCH_NUM = 4 >>> IMAGE_SIZE = 784 >>> CLASS_NUM = 10 >>> # define a random dataset >>> class RandomDataset(Dataset): ... def __init__(self, num_samples): ... self.num_samples = num_samples ... ... def __getitem__(self, idx): ... image = np.random.random([IMAGE_SIZE]).astype('float32') ... label = np.random.randint(0, CLASS_NUM - 1, (1, )).astype('int64') ... return image, label ... ... def __len__(self): ... return self.num_samples ... >>> dataset = RandomDataset(BATCH_NUM * BATCH_SIZE) >>> class SimpleNet(nn.Layer): ... def __init__(self): ... super().__init__() ... self.fc = nn.Linear(IMAGE_SIZE, CLASS_NUM) ... ... def forward(self, image, label=None): ... return self.fc(image) ... >>> simple_net = SimpleNet() >>> opt = paddle.optimizer.SGD(learning_rate=1e-3, ... parameters=simple_net.parameters()) ... >>> loader = DataLoader(dataset, ... batch_size=BATCH_SIZE, ... shuffle=True, ... drop_last=True, ... num_workers=2) ... >>> for e in range(EPOCH_NUM): ... for i, (image, label) in enumerate(loader()): ... out = simple_net(image) ... loss = F.cross_entropy(out, label) ... avg_loss = paddle.mean(loss) ... avg_loss.backward() ... opt.minimize(avg_loss) ... simple_net.clear_gradients() ... print("Epoch {} batch {}: loss = {}".format(e, i, np.mean(loss.numpy()))) Notes: For reading iterable dataset with multiprocess Dataloader, please see :code:`paddle.io.IterableDataset` Nc||_| |_| |_| |_||_||_|st s |Jd||_| t}t|ttfr t|}n t|}t||_| dk\sJd| dkDr=t j"dk(st j"dk(rt%j&dd} | |_| dkDsJd| |_| r | dk(rd|_|dk\sJd ||_t|t.r2t0j2|_|rt7d || t7d t0j8|_||d k(r|s|rJd ||_d|_nZ|d|_d|_nI|dkDsJd||_t|t.rt?|||_ntA|||||_||_!|j:du|_"d|_#t rtIdn tI|_#||_%d|_&tO|jQ|_y)Nz.feed_list should be set when return_list=Falserz*num_workers should be a non-negative valuerMrNzDataLoader with multi-process mode is not supported on MacOs and Windows currently. Please use signle-process mode with num_workers = 0 insteadz*prefetch_factor should be a positive valueFz&timeout should be a non-negative valuez5IterableDataset not support shuffle, but got shuffle=z0IterableDataset expect unspecified batch_samplerr zJbatch_size/shuffle/drop_last should not be set when batch_sampler is givenzMbatch_size should be None or a positive value when batch_sampler is not given)rVrWr^r_T)) return_list collate_fnuse_buffer_readerprefetch_factorworker_init_fnrVr feed_listrrr$r%rrr-r)rOrPwarningswarnr;use_shared_memorytimeoutr rITER dataset_kind ValueErrorMAPrarWr r r_auto_collate_batch pin_memoryr"_persistent_workers _iteratorr/rK)r5rVrr)rrarWr^r_rr;rrrrrpersistent_workerss rr6zDataLoader.__init__s&'$!2., ?#4% @? @%" >,.F ftUm ,+F3F&v.F%f- aM!MM ? LLH $ (? MMO K&"P$PP"!2 !1%*D "!|EEE| g / , 1 1D  KG9U( F!- 0 0D   $?79 ) D"/D "DO  !%D "DO> - >)DO'?3%=Z&"&2#)#' &"#"&"4"4D"@  )+39J9L O$6 #D>224rc|jtjk(r td|jrt |j St |jS)Nz'length of IterableDataset not supported)rrrrrrrarVrQs r__len__zDataLoader.__len__sK    1 1 1FG G&&4--..4<<((rc|jdk(r t|S|jrN|jt ||_|jS|jj |jSt |S)Nr)r;rrrr_resetrQs r__iter__zDataLoader.__iter__sm   q /5 5  % %~~%!> !%%'>> !.t4 4rc"|jSr)rrQs rrKzDataLoader.__call__s}}r)NNTNr FFNrTrTrNF)rxryrz__doc__r6rrrKr{rrr}r}sNeT #q5f) 5rr})r)!r`r>r2rOr=rrbbase.frameworkrrr frameworkrr dataloaderr r r dataloader.batch_samplerr dataloader.dataloader_iterrrrQUEUE_GET_TIMEOUTrrrrr"r-r/r}r{rrrs~    .==>   $ FFRqqr