\;j15.dZddlZddlZddlmZddlZddlmZgZdZ dZ dZ dZ d Z d Zd Zd Zdd ZdZdZeddddddZeddddddZeddddddZeddddddZedddddZy)aW ACL2016 Multimodal Machine Translation. Please see this website for more details: http://www.statmt.org/wmt16/multimodal-task.html#task1 If you use the dataset created for your task, please cite the following paper: Multi30K: Multilingual English-German Image Descriptions. @article{elliott-EtAl:2016:VL16, author = {{Elliott}, D. and {Frank}, S. and {Sima"an}, K. and {Specia}, L.}, title = {Multi30K: Multilingual English-German Image Descriptions}, booktitle = {Proceedings of the 6th Workshop on Vision and Language}, year = {2016}, pages = {70--74}, year = 2016 } N) defaultdict) deprecatedz2http://paddlemodels.bj.bcebos.com/wmt/wmt16.tar.gz 0c38be43600334966403524a40dcd81ei+iKzzzc tt}tj|d5}|j dD]q}|j }|j jd}t|dk7rA|dk(r|dn|d}|jD]} || xxdz cc<s dddt|d 5} | jtd td td jtt|j!d d D]B\} } | dz|k(rn5| j| dj| jdDdddy#1swYxYw#1swYyxYw)Nrmode wmt16/train enrwb c |dS)Nr)xs ]G:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/dataset/wmt16.pyz__build_dict..BsAaDT)keyreverse )rinttarfileopen extractfiledecodestripsplitlenwrite START_MARKEND_MARKUNK_MARKencode enumeratesorteditems) tar_file dict_size save_pathlang word_dictfline line_splitsenwfoutidxwords r __build_dictr83sAC I hS )QMM-0D;;=D++D1J:!##'4<*Q-Z]CYY[! ! ! 1 * i $ zl"XJb "=EEGH" 9??$.$ G ICQw)# JJtAw~~' ( JJu      * )  sBE+B E7+E47Fc4tjjtjj j d||fz}tjj|r&tt|dj|k7rt||||i}t|d5}t|D]J\}}|r"|jj||<*|||jj<L ddd|S#1swY|SxYw)Nwmt16/%s_%d.dictrb)ospathjoinpaddledatasetcommon DATA_HOMEexistsr"r readlinesr8r(r r) r+r,r.r dict_pathr/fdictr6r1s r __load_dictrGJs '');tY>O)OI 77>>) $ DD ! + + -.);Xy)T:I i %"5)IC!%!4!4!6 #36 $**,--/0 *    s )AD  Dcvt||dk(rtnt}t||dk(rtnt}||fS)Nr )minTOTAL_EN_WORDSTOTAL_DE_WORDS src_dict_size trg_dict_sizesrc_langs r__get_dict_sizerP]sA(d*:M(d*:M - ''rc"fd}|S)Nc 3Kt}tdk(rdnd}|t}|t}|t}dk(rdnd}d|z }t j d5}|j D]}|j}|jjd} t| dk7rA| |j} |g| D cgc]} |j| |c} z|gz} | |j} | D cgc]} |j| |}} ||gz}|g|z}| ||f dddycc} wcc} w#1swYyxYww) Nr derrrrr r ) rGr$r%r&rrrrr r!r"get)src_dicttrg_dictstart_idend_idunk_idsrc_coltrg_colr0r1r2 src_wordsr4src_ids trg_wordstrg_ids trg_ids_next file_namerMrOr+rNs rreaderzreader_creator..readerhsxxA mh$.>dD J'(#(#4'!Qg+ \\( - i0{{}!ZZ\//5 z?a'&w/557 J8AB 1x||Av. BCh 'w/557 Please use new dataset API which supports paddle.io.DataLoader)since update_tolevelreasonc|dvr tdt|||\}}ttjj j tdtdd|||S)a} WMT16 train set reader. This function returns the reader for train data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for training data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The train reader. r rSzIAn error language type. Only support: en (for English); de(for Germany).wmt16 wmt16.tar.gzr r+rarMrNrO ValueErrorrPrcr?r@rAdownloadDATA_URLDATA_MD5rLs rtrainrrsuP|# 1  $3}h$ M= &&// gx  ## rc|dvr tdt|||\}}ttjj j tdtdd|||S)a} WMT16 test set reader. This function returns the reader for test data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for test data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The test reader. riHAn error language type. Only support: en (for English); de(for Germany).rjrkz wmt16/testrlrmrLs rtestrusuN|# ?  $3}h$ M= &&// gx ## rc|dvr tdt|||\}}ttjj j tdtdd|||S)a WMT16 validation set reader. This function returns the reader for validation data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for validation data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The validation reader. rirtrjrkz wmt16/valrlrmrLs r validationrwsuL|# ?  $3}h$ M= &&// gx ## rc|dk(rt|t}nt|t}tjj t jjjd||fz}tjj|sJd tjj t jjjd}t||||S)a return the word dictionary for the specified language. Args: lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. dict_size(int): Size of the specified language dictionary. reverse(bool): If reverse is set to False, the returned python dictionary will use word as key and use index as value. If reverse is set to True, the returned python dictionary will use index as key and word as value. Returns: dict: The word dictionary for the specific language. r r:z Word dictionary does not exist. rk) rIrJrKr<r=r>r?r@rArBrCrG)r.r,rrEr+s rget_dictryBs0 t| >2  >2  '');tY>O)OI 77>>) $H&HH $Eww||FNN11;;^LH xD' ::rctjjjj t dt dy)zdownload the entire dataset.rjrkN)r?v4r@rArorprqrrrfetchr|is+ II%%'8^r)F)r )__doc__r<r collectionsrr? paddle.utilsr__all__rprqrJrKr$r%r&r8rGrPrcrrrurwryr|rrrrs" # #  ? -   .&(&R * K  3  3l * K  3  3l * K  1  1h * K  ;  ;B * K    r