:jÕdZddlmZmZddlZddlmZmZmZm Z m Z m Z m Z m Z ddlmZddlmZdZej&j)e d ZGd d ZGd d ZGddZy)zC Unit tests for nltk.tokenize. See also nltk/test/tokenize.doctest )ListTupleN)LegalitySyllableTokenizerStanfordSegmenterSyllableTokenizerTreebankWordTokenizerTweetTokenizerpunkt sent_tokenize word_tokenize) CharTokenizer)TreebankWordDetokenizerc| t}|jd|jdy#t$rYywxYw)NarzhTF)rdefault_config LookupError)segs q/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/nltk/test/unit/test_tokenize.pyload_stanford_segmenterrs@! 4  4  s ,/ ;;z/NLTK was unable to find stanford-segmenter.jar.)reasonceZdZdZej j ddgdgdffdgdgdffd gd gd ffd gd gd ffd gdgdffdgdgdffdgdgdffdgdgdffgdedee ee effdZ dZ dZ dZ edZed Zd!Zd"Zd#Zd$Zd%Zd&Zd'Zd(Zd)Zd*Zd+Zd,Zd-Zej j d.gd/dPd1Zd2Zej j d3d4gd5fd6gd7fd8d8gfd9d9gfd:d;dd?gfd@gdAfdBdCdr?r@rArB) r*r=r.z601-984-4813r>r?r@rArBz/My number is (393) 928 -3010, except it's not.) r*r=r.(393) 928 -3010r>r?r@rArB)r*r=r.(393)928-3010r>r?r@rArBz1The product identification number is 48103284512.)Theproductidentificationr=r. 48103284512rB)rKrLrMr=r. 48103284512rBz(My favourite substraction is 240 - 1353.)r* favourite substractionr.z 240 - 1353rB)r*rQrRr.240rI1353rB test_input expectedsc|tddg|D]+\}}tdd|}|j|}||k(r+Jy)a Test `match_phone_numbers` in TweetTokenizer. Note that TweetTokenizer is also passed the following for these tests: * strip_handles=True * reduce_len=True :param test_input: The input string to tokenize using TweetTokenizer. :type test_input: str :param expecteds: A 2-tuple of tokenized sentences. The first of the two tokenized is the expected output of tokenization with `match_phone_numbers=True`. The second of the two tokenized lists is the expected output of tokenization with `match_phone_numbers=False`. :type expecteds: Tuple[List[str], List[str]] TF)rrmatch_phone_numbersN)zipr r")r#rUrVrXr'r$ predicteds rtest_tweet_tokenizer_expandedz*TestTokenize.test_tweet_tokenizer_expanded?sXH.1$ -J ) ) &"$7I "**:6I( (( )r)cLt}|jd}|gdk(sJy)3 Test SyllableTokenizer tokenizer. justification)justificationNrr")r#r$r&s r+test_sonority_sequencing_syllable_tokenizerz8TestTokenize.test_sonority_sequencing_syllable_tokenizer s+&' ##O4::::r)cTt}ddz}|j|}||gk(sJy)r]9i'Nrd)r#r$r+r&s rtest_syllable_tokenizer_numbersz,TestTokenize.test_syllable_tokenizer_numberss5&' U{##D)$r)czddlm}d}t|j}|j|}|gdk(sJy)z; Test LegalitySyllableTokenizer tokenizer. r)r wonderful)wonderfulN) nltk.corpusr rr")r#r test_wordr$r&s r*test_legality_principle_syllable_tokenizerz7TestTokenize.test_legality_principle_syllable_tokenizers; & -ekkm< ##I.....r)ct}|jdd}|j|j}|jgdk(sJy)zN Test the Stanford Word Segmenter for Arabic (default config) runيبحث علم الحاسوب استخدام الحوسبة بجميع اشكالها لحل المشكلات) uيبحثuعلمuالحاسوبuاستخدامuالحوسبةuبuجميعu اشكالuهاuلuحلuالمشكلاتNrrsegmentsplitr#rsentsegmented_sents rtest_stanford_segmenter_arabicz+TestTokenize.test_stanford_segmenter_arabic(sS  ! 4 TZZ\2##% *   r)ct}|jdd}|j|j}|jgdk(sJy)zO Test the Stanford Word Segmenter for Chinese (default config) ru$这是斯坦福中文分词器测试)u这u是u 斯坦福u中文u 分词器u测试Nrrrus rtest_stanford_segmenter_chinesez,TestTokenize.test_stanford_segmenter_chinese@sS  ! 4 5TZZ\2##%*    r)ct}d}dg}|j|}||k(sJd}gd}|j|}||k(sJy)zT Test a string that resembles a phone number but contains a newline rDz(393) 928 -3010)rErFrGz 928 -3010Nr!)r#r$test1r'resulttest2s rtest_phone_tokenizerz!TestTokenize.test_phone_tokenizerRsa #$ "&'##E*!!!#1##E*!!!r)c|t}d}dg}|j|}||k(sJd}dg}|j|}||k(sJd}gd}|j|}||k(sJd}gd}|j|}||k(sJd}gd}|j|}||k(sJd } gd }|j| }||k(sJy ) zX Test a string that contains Emoji ZWJ Sequences and skin tone modifier u👨‍👩‍👧‍👧u👨🏿u🤔 🙈 me así, se😌 ds 💕👭👙 hello 👩🏾‍🎓 emoji hello 👨‍👩‍👦‍👦 how are 😊 you today🙅🏽🙅🏽)u🤔u🙈meuasír>seu😌dsu💕u👭u👙hellou👩🏾‍🎓emojiru👨‍👩‍👦‍👦howareu😊r;today🙅🏽ru🇦🇵🇵🇱🇪)u🇦🇵🇵🇱u🇪uHi 🇨🇦, 😍!!)Hi🇨🇦r>u😍r<r<u<3 🇨🇦 🤝 🇵🇱 <3)<3ru🤝rrNr!) r#r$r|r'r}r~test3test4test5test6s rtest_emoji_tokenizerz!TestTokenize.test_emoji_tokenizeres#$ ,/0##E*!!!<##E*!!!_ 2##E*!!!'3##E*!!!%<##E*!!!.?##E*!!!r)c0d}gd}t||k(sJy)zA Test padding of asterisk for word tokenization. z1This is a, *weird sentence with *asterisks in it.) r6r.ar>*weirdsentencewithr asterisksinitrBNr r#r+r's rtest_pad_asteriskzTestTokenize.test_pad_asterisks&C T"h...r)cd}gd}t||k(sJd}gd}t||k(sJd}gd}t||k(sJd}gd}t||k(sJtd d gk(sJy ) z@ Test for em-dashes and en-dashes (Issue #3459) u5The food—which was delicious—reminded me of home.) rKfood—whichwas deliciousrremindedrofhomerBuThe score was 3–1.)rKscorer3u–1rBu(He said ― wait for it ― a new thing.) Hesaid―waitr:rrrnewthingrBuThe number is 12‒34.)rKr=r.12u‒34rBzstate-of-the-artNr) r#s_em expected_ems_en expected_ens_bar expected_bars_fig expected_figs rtest_dash_tokenizationz#TestTokenize.test_dash_tokenizations G  T"k111&C T"k111;  U#|333)F U#|333/05G4HHHHr)c0d}gd}t||k(sJy)z@ Test padding of dotdot* for word tokenization. zPWhy did dotdot.. not get tokenized but dotdotdot... did? How about manydots.....)Whydiddotdotz..rAget tokenizedbut dotdotdotz...r?Howaboutmanydotsz.....Nrrs rtest_pad_dotdotzTestTokenize.test_pad_dotdots&b $T"h...r)ctd}d}gd}|j|}||k(sJd}gd}|j|}||k(sJd}gd}|j|}||k(sJd }gd }|j|}||k(sJd }gd }|j|}||k(sJd } gd}|j| }||k(sJd} gd}|j| }||k(sJy)zW Test remove_handle() from casual.py with specially crafted edge cases T)rz-@twitter hello @twi_tter_. hi @12345 @123news)rrBhiu]@n`@n~@n(@n)@n-@n=@n+@n\@n|@n[@n]@n{@n}@n;@n:@n'@n"@n/@n?@n.@n,@n<@n>@n @n @n ñ@n.ü@n.ç@n.)`~rErGrI=+\|[]{};r'"/rrBr><>ñrBürBçrBzKa@n j@n z@n A@n L@n Z@n 1@n 4@n 7@n 9@n 0@n _@n !@n @@n #@n $@n %@n &@n *@n)&r@njrzrArLrZrrr4r7rrgr0r_rr<r@r#r$r%r&rrrz@n!a @n#a @n$a @n%a @n&a @n*a) r<rrrrrrrrrrrzD@n!@n @n#@n @n$@n @n%@n @n&@n @n*@n @n@n @@n @n@@n @n_@n @n7@n @nj@n)r<rrrrrrrrrrrrrrrrrrz@n_rz@n7rz@njrz^@abcdefghijklmnopqrstuvwxyz @abcdefghijklmno1234 @abcdefghijklmno_ @abcdefghijklmnoendofhandle) pqrstuvwxyz1234r endofhandlez^@abcdefghijklmnop@abcde @abcdefghijklmno@abcde @abcdefghijklmno_@abcde @abcdefghijklmno5@abcde)p@abcdez@abcdefghijklmnorrr5rNr!) r#r$r|r'r}r~rrrrtest7s rtest_remove_handlezTestTokenize.test_remove_handle sK #6 @'##E*!!!s >##E*!!!^' P##E*!!!0O##E*!!!W 6##E*!!!q>##E*!!!q  ##E*!!!r)ct}d}gd}t|j|}||k(sJd}gd}t|j|}||k(sJd}gd}t|j|}||k(sJy)zC Test TreebankWordTokenizer.span_tokenize function zNGood muffins cost $3.88 in New (York). Please (buy) me two of them. (Thanks).))r) ) ))r))) )r$)r%)r&)(.)/0)r 3)r 4)57)8;)<>)?D)EF)rL)rM)rNzmThe DUP is similar to the "religious right" in the United States and takes a hardline stance on social issues)rr  rrrrrrr*r++,rr 2r 9:@ArrJKr)rU)V\)]_)`f)gmzqThe DUP is similar to the "religious right" in the United States and takes a ``hardline'' stance on social issues)rrrr!r#r%r(r)r*r,r.r0r2r4r7r9r;)rO)rFW)rGY)ZrB)ac)dj)kqN)rlist span_tokenize)r#r$r|r'r}r~rs rtest_treebank_span_tokenizerz)TestTokenize.test_treebank_span_tokenizers *+ d 2i--e45!!!@ 0i--e45!!!F 4i--e45!!!r)c\d}gd}t||k(sJd}gd}t||k(sJy)z- Test word_tokenize function z0The 'v', I've been fooled but I'll seek revenge.)rKrvrr>I'vebeenfooledrrU'llseekrevengerBz'v' 're')rrTrz'rerNr)r#rr's rtest_word_tokenizezTestTokenize.test_word_tokenizesC F "X&(222.X&(222r)cdddgfdgdfdgdfg}|D].\}}tj|Dcgc]}|}}||k(r.Jycc}w)NrrrO)rON123)r^rOr)rNr)r^r`)rr)rN)r _pair_iter)r# test_casesrUexpected_outputx actual_outputs rtest_punkt_pair_iterz!TestTokenize.test_punkt_pair_iter!sm J , - 9 : F G ,6 4 'J(-(8(8(DE1QEME O3 33 4Es AcZtg}tj|}t|yN)iterr rarP)r#rgens r5test_punkt_pair_iter_handles_stop_iteration_exceptionzBTestTokenize.test_punkt_pair_iter_handles_stop_iteration_exception-s" "Xr" S r)ctj}Gdd}||_t|j dy)NceZdZdZy)kTestTokenize.test_punkt_tokenize_words_handles_stop_iteration_exception..TestPunktTokenizeWordsMockctgSrh)ri)r#ss rr zyTestTokenize.test_punkt_tokenize_words_handles_stop_iteration_exception..TestPunktTokenizeWordsMock.word_tokenize9s Bxr)N)__name__ __module__ __qualname__r r)rTestPunktTokenizeWordsMockrn8s r)rur)r PunktBaseClass _lang_varsrP_tokenize_words)r#objrus r:test_punkt_tokenize_words_handles_stop_iteration_exceptionzGTestTokenize.test_punkt_tokenize_words_handles_stop_iteration_exception5s9""$  45 S  ()r)cGddtj}tj|}d}gd}|j||k(sJy)NceZdZdZy)NTestTokenize.test_punkt_tokenize_custom_lang_vars..BengaliLanguageVars)rBrr<u।Nrqrrrssent_end_charsrtr)rBengaliLanguageVarsr}Bs6Nr)r) lang_varscউপরাষ্ট্রপতি শ্রী এম ভেঙ্কাইয়া নাইডু সোমবার আই আই টি দিল্লির হীরক জয়ন্তী উদযাপনের উদ্বোধন করেছেন। অনলাইনের মাধ্যমে এই অনুষ্ঠানে কেন্দ্রীয় মানব সম্পদ উন্নয়নমন্ত্রী শ্রী রমেশ পোখরিয়াল ‘নিশাঙ্ক’ উপস্থিত ছিলেন। এই উপলক্ষ্যে উপরাষ্ট্রপতি হীরকজয়ন্তীর লোগো এবং ২০৩০-এর জন্য প্রতিষ্ঠানের লক্ষ্য ও পরিকল্পনার নথি প্রকাশ করেছেন।)uউপরাষ্ট্রপতি শ্রী এম ভেঙ্কাইয়া নাইডু সোমবার আই আই টি দিল্লির হীরক জয়ন্তী উদযাপনের উদ্বোধন করেছেন।u+অনলাইনের মাধ্যমে এই অনুষ্ঠানে কেন্দ্রীয় মানব সম্পদ উন্নয়নমন্ত্রী শ্রী রমেশ পোখরিয়াল ‘নিশাঙ্ক’ উপস্থিত ছিলেন।u/এই উপলক্ষ্যে উপরাষ্ট্রপতি হীরকজয়ন্তীর লোগো এবং ২০৩০-এর জন্য প্রতিষ্ঠানের লক্ষ্য ও পরিকল্পনার নথি প্রকাশ করেছেন।)r PunktLanguageVarsPunktSentenceTokenizerr")r#rry sentencesr's r$test_punkt_tokenize_custom_lang_varsz1TestTokenize.test_punkt_tokenize_custom_lang_vars@sR 7%"9"9 7**5H5JKz   ||I&(222r)cbtj}d}dg}|j||k(sJy)Nr)r rr")r#ryrr's r'test_punkt_tokenize_no_custom_lang_varsz4TestTokenize.test_punkt_tokenize_no_custom_lang_varsQs>**,z  r  ||I&(222r)z%input_text,n_sents,n_splits,lang_vars))z4Subject: Some subject. Attachments: Some attachments)z4Subject: Some subject! Attachments: Some attachmentsrr)z4This is just a normal sentence, just like any other.rrNctj}|||_t|j ||k(sJtt |j ||k(sJyrh)r rrwlenr"rPdebug_decisions)r# input_textn_sentsn_splitsrr$s rpunkt_debug_decisionsz"TestTokenize.punkt_debug_decisions\sa$002  #,I 9%%j12g===4 11*=>?8KKKr)cjGddtj}|jddd|y)NceZdZdZy)GTestTokenize.test_punkt_debug_decisions_custom_end..ExtLangVars)rBrr<^Nr~rtr)r ExtLangVarsrxs1Nr)rz4Subject: Some subject^ Attachments: Some attachmentsrr)rrr)r rr)r#rs r%test_punkt_debug_decisions_custom_endz2TestTokenize.test_punkt_debug_decisions_custom_endus7 2%11 2 "" B!m # r)zsentences, expectedzthis is a test. . new sentence.)zthis is a test.rBz new sentence.zThis. . . That)This.rBrBThatzThis..... Thatz This... Thatz This.. . ThatzThis.. .rz This. .. Thatrz.. Thatz This. ,. That)rz,.rz This!!! ThatzThis!!!z This! ThatzThis!z+1. This is R . 2. This is A . 3. That's all)z1.z This is R .z2.z This is A .z3.z That's allz+1. This is R . 2. This is A . 3. That's allz Hello. TherezHello.Thererr'c$t||k(sJyrh)r )r#rr's rtest_sent_tokenizezTestTokenize.test_sent_tokenizes6Y'8333r)cd}t}|j|t|k(sJt|j|gdk(sJy)Nz Hello there) )rr)rr)rr)rr)rr)r)rr)rr)r )rr )r r")r r"rPrQ)r#rr$s rtest_string_tokenizerz"TestTokenize.test_string_tokenizersR !O !!(+tH~===I++H56 ;   r)rhreturnN)"rqrrrsr(pytestmark parametrizestrtuplerPr[rerhrpcheck_stanford_segmenterrxrzrrrrrrrRr\rfrkrzrrrrrrrtr)rrr(s", [[6NJ 0EA O   L=   <>   >B  FD  6;PV Mn qd))*/S 490D*E)eqd)6;  /  .  ""&?"B/,3Ij/2W"r_"B3: 4 *3" 3 [[/ "L#"L   [[29 : ;  01 2 n- . z62 3 w 2 3 5 6 i0 1 GV, -?N  @N x1 2+ 44C4494544 r)rc$eZdZddZddZddZy)TestPunktTrainerNcNtj}|jdy)NzThis is a test.r PunktTrainertrainr#trainers rtest_punkt_trainz!TestPunktTrainer.test_punkt_trains$$& '(r)cNtj}|jdy)Nrrrs rtest_punkt_train_single_wordz-TestPunktTrainer.test_punkt_train_single_words$$& gr)cNtj}|jdy)NzThis is a testrrs rtest_punkt_train_no_puncz)TestPunktTrainer.test_punkt_train_no_puncs$$& &'r)r)rqrrrsrrrrtr)rrrs)(r)rceZdZeZdZdZdZdZdZ dZ dZ dZ d Z d Zd Zd Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ dZ!dZ"y) TestTreebankWordDetokenizercLgd}|jj|dk(sJy)N)Hellor>worldrB Hello, world.detok detokenizer#r&s rtest_simple_sentencez0TestTreebankWordDetokenizer.test_simple_sentences$-zz$$V,???r)cLgd}|jj|dk(sJy)N)rUz'msurerBz I'm sure.rrs rtest_contractionsz-TestTreebankWordDetokenizer.test_contractionss$)zz$$V, ;;;r)cLgd}|jj|dk(sJy)N)YourYseerBz You'll see.rrs rtest_contraction_llz/TestTreebankWordDetokenizer.test_contraction_ll$+zz$$V, ===r)cLgd}|jj|dk(sJy)N)rUdozn'tknowrBz I don't know.rrs rtest_contraction_notz0TestTreebankWordDetokenizer.test_contraction_nots$0zz$$V,???r)cLgd}|jj|dk(sJy)N``r''"Hello"rrs rtest_double_quotesz.TestTreebankWordDetokenizer.test_double_quotess$&zz$$V, 999r)cLgd}|jj|dk(sJy)N)rrrrrrBz He said "hi".rrs rtest_double_quotes_with_periodz:TestTreebankWordDetokenizer.test_double_quotes_with_periods$6zz$$V,???r)cLgd}|jj|dk(sJy)N)rYesr>rherrBz"Yes," he said.rrs r&test_comma_before_closing_double_quotezBTestTreebankWordDetokenizer.test_comma_before_closing_double_quotes%<zz$$V,0AAAAr)cLgd}|jj|dk(sJy)N)rKdog'sbonerBzThe dog's bone.rrs rtest_possessivez+TestTreebankWordDetokenizer.test_possessives%2zz$$V,0AAAAr)cTgd}|jj|d}|dk(sJy)z@PTB bracket symbols are converted when convert_parentheses=True.)z-LRB-rz-RRB-T)convert_parenthesesz(hello)Nrr#r&r}s rtest_parentheses_ptbz0TestTreebankWordDetokenizer.test_parentheses_ptbs.,&&v4&H"""r)cLgd}|jj|dk(sJy)N)fooz--barzfoo--barrrs rtest_double_dashesz.TestTreebankWordDetokenizer.test_double_dashess$%zz$$V, :::r)cLgd}|jj|dk(sJy)z(`` is converted to opening double quote.rrNrrs rtest_opening_backtick_doublez8TestTreebankWordDetokenizer.test_opening_backtick_doubles$&zz$$V, 999r)cXgd}|jj|}d|vsd|vsJyy)z=` is preserved as-is (no standard conversion in detokenizer).)rrrrrNrrs rtest_opening_backtick_singlez8TestTreebankWordDetokenizer.test_opening_backtick_singles2$&&v.f}v -- }r)cNgd}|jj|}d|vsJy)z('' is converted to closing double quote.)rrryesrrBz"yes"Nrrs r#test_closing_double_quote_backticksz?TestTreebankWordDetokenizer.test_closing_double_quote_backtickss)7&&v.&   r)cXgd}|jj|}d|vsd|vsJyy)z8Fix #3260: closing single then double quote after comma.)rShippersrsayingrthepartyroverr>rrrzMr.LaLonderBz,'z,'"Nrrs rtest_issue_3260_nested_quotesz9TestTreebankWordDetokenizer.test_issue_3260_nested_quotess8 $&&v.v~6!111!1~r)c\gd}|jj|}|dk(s|dk(sJyy)z@Minimal case: comma, single quote, double quote should collapse.)wordr>rrzword,'"Nrrs rtest_issue_3260_minimalz3TestTreebankWordDetokenizer.test_issue_3260_minimals6)&&v.#v';;;';#r)cd}t}|j|}|jj|}||k(sJy)z9Tokenize then detokenize should approximate the original.rN)rr"rr)r#originalr$r&r}s rtest_roundtrip_simplez1TestTreebankWordDetokenizer.test_roundtrip_simplesA")+ ##H-&&v.!!!r)cLgd}|jj|dk(sJy)N)rrbza; brrs rtest_semicolon_spacingz2TestTreebankWordDetokenizer.test_semicolon_spacing&$ zz$$V,666r)cLgd}|jj|dk(sJy)N)rrrza: brrs rtest_colon_spacingz.TestTreebankWordDetokenizer.test_colon_spacing*rr)cLgd}|jj|dk(sJy)N)werVseenrBz we've seen.rrs rtest_contraction_vez/TestTreebankWordDetokenizer.test_contraction_ve.rr)cLgd}|jj|dk(sJy)N)whoz'drrBz who'd know.rrs rtest_contraction_dz.TestTreebankWordDetokenizer.test_contraction_d2rr)cLgd}|jj|dk(sJy)N)rrBrzword."rrs r&test_closing_double_quote_after_periodzBTestTreebankWordDetokenizer.test_closing_double_quote_after_period6$$zz$$V,888r)cLgd}|jj|dk(sJy)N)rr>rzword,"rrs r%test_closing_double_quote_after_commazATestTreebankWordDetokenizer.test_closing_double_quote_after_comma:r r)cLgd}|jj|dk(sJy)N)rr<rzword!"rrs r+test_closing_double_quote_after_exclamationzGTestTreebankWordDetokenizer.test_closing_double_quote_after_exclamation>r r)cLgd}|jj|dk(sJy)N)rrrzword?"rrs r(test_closing_double_quote_after_questionzDTestTreebankWordDetokenizer.test_closing_double_quote_after_questionBr r)cLgd}|jj|dk(sJy)N)rrBCrz"A B C"rrs rtest_multiple_tokens_in_quotesz:TestTreebankWordDetokenizer.test_multiple_tokens_in_quotesFs$,zz$$V, 999r)cLgd}|jj|dk(sJy)N)rrr>rrJorBz"Hi," said Jo.rrs rtest_quote_comma_saidz1TestTreebankWordDetokenizer.test_quote_comma_saidJs%;zz$$V,0@@@@r)cJ|jjgd}d|vsJy)N)rr>rz ,rr#r}s r#test_no_spurious_space_before_commaz?TestTreebankWordDetokenizer.test_no_spurious_space_before_commaNs$&&76!!!r)cJ|jjddg}d|vsJy)NrrBz .rrs r$test_no_spurious_space_before_periodz@TestTreebankWordDetokenizer.test_no_spurious_space_before_periodRs(&&Sz26!!!r)cn|jjgd}d|vs|jdsJyy)N)rrrz "r)rr startswithrs r+test_no_spurious_space_before_closing_quotezGTestTreebankWordDetokenizer.test_no_spurious_space_before_closing_quoteVs8&&'896!V%6%6s%;;;%;!r)cNgd}|jj|}d|vsJy)zRound-trip a quoted sentence.)rrr>rrrrBrNr)r#original_tokensr}s rtest_roundtrip_quotesz1TestTreebankWordDetokenizer.test_roundtrip_quotesZs'G&&7f}}r)N)#rqrrrsrrrrrrrrrrrrrrrrrrrrrrr r rrrrrrrr"rtr)rrrs # %E@<>@:@BB# ;: . ! 20< "77>>9999:A""<r)r)__doc__typingrrr nltk.tokenizerrrrr r r r nltk.tokenize.simpler nltk.tokenize.treebankrrrskipifrrrrrtr)rr)sy     /:";;--!! <. I I X ( (\\r)