\;jsdZddlZddlZddlZddlZddlZddlmZgZ dZ dZ dZ dZ edd d d d Zedd d d dZedd d d dZedd d d dZedd d d dZy)a IMDB dataset. This module downloads IMDB dataset from http://ai.stanford.edu/%7Eamaas/data/sentiment/. This dataset contains a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. Besides, this module also provides API for building dictionary. N) deprecatedz6https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gz 7c2ac02c03563afcf9b574c7e56c153ac#FKtjtjjj t dt5}|j}|t|j|jry|j|jjdjdt j"j%dj'j)|j}|dddy#1swYyxYww)zQ Read files that match the given pattern. Tokenize and yield each file. imdbNs zlatin-1)tarfileopenpaddledatasetcommondownloadURLMD5nextboolmatchname extractfilereadrstrip translatestring punctuationencodelowersplit)patterntarftfs \G:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/dataset/imdb.pytokenizer &s fnn++44S&#F G4 YY[nGMM"''*+&&r*//188AKK&,,33I>%'%%'"B n H G GsAD! CD D!DD!c tjt}t|D]}|D]}||xxdz cc<|j Dcgc] }|d|kDs |}}t |d}t t|\}}tt t|tt|} t|| d<| Scc}w)z Build a word dictionary from the corpus. Keys of the dictionary are words, and values are zero-based IDs of these words. c|d |dfS)Nr"r)xs rzbuild_dict..Gs1Q4%1)key) collections defaultdictintr itemssortedlistzipdictrangelen) rcutoff word_freqdocwordr% dictionarywords_word_idxs r build_dictr<:s '',I D dOq O! &OO-?-q1-I? '>?JC$%HE1DUE#e*$5678HE HW O @s  CCz2.0.0zpaddle.text.datasets.Imdbr"z>Please use new dataset API which supports paddle.io.DataLoader)since update_tolevelreasoncXdgfd}||d||dfd}|S)Nr)c t|D]3}|j|Dcgc]}j|c}|f5ycc}wN)r appendget)routlabelr6wUNKr;s rloadzreader_creator..loadXs@G$C JJs;s!a-s;UC D%;sA rr"c3&KEd{y7wrCr$)INSsrreaderzreader_creator..reader_ss r$) pos_pattern neg_patternr;rJrMrLrIs ` @@rreader_creatorrPNs@ 7 C CE c1c1 Mr'chttjdtjd|S)a IMDB training set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Training reader creator :rtype: callable zaclImdb/train/pos/.*\.txt$zaclImdb/train/neg/.*\.txt$rPrecompiler;s rtrainrVes.$  01 01 r'chttjdtjd|S)a IMDB test set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Test reader creator :rtype: callable zaclImdb/test/pos/.*\.txt$zaclImdb/test/neg/.*\.txt$rRrUs rtestrX~s.$  /0 /0 r'c@ttjddS)za Build a word dictionary from the corpus. :return: Word dictionary :rtype: dict z/aclImdb/((train)|(test))/((pos)|(neg))/.*\.txt$)r<rSrTr$r'r word_dictr[s   EF r'cjtjjjtdt y)Nr)r r r r r rr$r'rfetchr]s  NN""34r')__doc__r*rSrrpaddle.dataset.commonr paddle.utilsr__all__r rr r<rPrVrXr[r]r$r'rrbs # ?((( ) K    " ) K    & ) K    & ) K     ) K  5  5r'