\ji!\dZddlmZmZmZdZGddZGddZGdd Zy ) z Unit tests for nltk.tokenize.texttiling. Marti A. Hearst (1997) "TextTiling: Segmenting Text into Multi-Paragraph Subtopic Passages", Computational Linguistics, 23(1), pp. 33-64. https://aclanthology.org/J97-1003.pdf )BLOCK_COMPARISONVOCABULARY_INTRODUCTIONTextTilingTokenizera The quick brown fox jumped over the lazy dog. The dog was sleeping in the sun. It was a beautiful day outside in the forest. The fox continued on its way through the dense woodland. Birds were singing in the trees and rabbits scurried through the underbrush. The animals were enjoying the warm afternoon sunshine. Meanwhile, in the city, the stock market was experiencing significant changes. Traders were anxious about the new economic policies. The financial sector was particularly affected by the recent changes in interest rates. Banks reported lower quarterly earnings. Investors were pulling their money out of risky assets. The central bank was considering raising rates again. In other news, scientists discovered a new species of deep-sea fish. The fish was found at a depth of three thousand meters below the surface. Marine biologists were excited about the discovery. The specimen was brought to the laboratory for further study. Researchers believe the species has been living in the deep ocean for millions of years. The weather forecast predicted rain for the weekend across the region. Farmers were hoping for the precipitation to help their growing crops. The drought had lasted several months with no relief in sight. Water reservoirs were at critically low levels. Irrigation systems were being strained beyond their capacity. The quick brown fox jumped over the lazy dog. The dog was sleeping in the sun. It was a beautiful day outside in the forest. The fox continued on its way through the dense woodland. Birds were singing in the trees and rabbits scurried through the underbrush. The animals were enjoying the warm afternoon sunshine. Meanwhile, in the city, the stock market was experiencing significant changes. Traders were anxious about the new economic policies. The financial sector was particularly affected by the recent changes in interest rates. Banks reported lower quarterly earnings. Investors were pulling their money out of risky assets. The central bank was considering raising rates again. In other news, scientists discovered a new species of deep-sea fish. The fish was found at a depth of three thousand meters below the surface. Marine biologists were excited about the discovery. The specimen was brought to the laboratory for further study. Researchers believe the species has been living in the deep ocean for millions of years. The weather forecast predicted rain for the weekend across the region. Farmers were hoping for the precipitation to help their growing crops. The drought had lasted several months with no relief in sight. Water reservoirs were at critically low levels. Irrigation systems were being strained beyond their capacity. The quick brown fox jumped over the lazy dog. The dog was sleeping in the sun. It was a beautiful day outside in the forest. The fox continued on its way through the dense woodland. Birds were singing in the trees and rabbits scurried through the underbrush. The animals were enjoying the warm afternoon sunshine. Meanwhile, in the city, the stock market was experiencing significant changes. Traders were anxious about the new economic policies. The financial sector was particularly affected by the recent changes in interest rates. Banks reported lower quarterly earnings. Investors were pulling their money out of risky assets. The central bank was considering raising rates again. In other news, scientists discovered a new species of deep-sea fish. The fish was found at a depth of three thousand meters below the surface. Marine biologists were excited about the discovery. The specimen was brought to the laboratory for further study. Researchers believe the species has been living in the deep ocean for millions of years. The weather forecast predicted rain for the weekend across the region. Farmers were hoping for the precipitation to help their growing crops. The drought had lasted several months with no relief in sight. Water reservoirs were at critically low levels. Irrigation systems were being strained beyond their capacity. c.eZdZdZdZdZdZdZdZy)TestTextTilingBlockComparisonzATests for the block comparison method (Hearst 1997, Section 3.2).ctddt}|jt}t |t sJt |dk\sJy)z7Block comparison should return a list of text segments. wksimilarity_methodN)rrtokenizeMULTI_TOPIC_TEXT isinstancelistlenselfttsegmentss H/root/env/lib/python3.12/site-packages/nltk/test/unit/test_texttiling.pytest_returns_segmentsz3TestTextTilingBlockComparison.test_returns_segments7sB 2>N O;;/0(D)))8}!!!ctddt}|jt}dj |tk(sJyz;Concatenated segments should reconstruct the original text.r r r N)rrrrjoinrs rtest_segments_cover_full_textz;TestTextTilingBlockComparison.test_segments_cover_full_text>s8 2>N O;;/0wwx $4444rctddtd}|jt}t |dk(sJ|\}}}}t |dkDsJt |dkDsJt |dkDsJt |dkDsJyzDIn demo mode, should return (gap_scores, smooth, depth, boundaries).r r Tr r r demo_moderN)rrrrrrrresult gap_scores smooth_scores depth_scores boundariess rtest_demo_mode_returns_scoresz;TestTextTilingBlockComparison.test_demo_mode_returns_scoresDs A)9T -.6{a>D; M<:"""=!A%%%< 1$$$:"""rctddtd}|jt\}}}}|D]}d|cxkrdkrnJd|dy ) zDBlock comparison gap scores should be in [0, 1] (cosine similarity).r r Tr#g?Score z out of [0, 1] rangeN)rrrrrrr(_scores r$test_gap_scores_between_zero_and_onezBTestTextTilingBlockComparison.test_gap_scores_between_zero_and_oneQsc A)9T !kk*:; Aq! ME%&3& L&7K(L L& Mrcd}dj|gdzdz}tddt}|j|}t |dksJy)z;Repeating the same paragraph should produce few boundaries.1The cat sat on the mat. The dog chased the ball.  r r r N)rrrrr)r paragraphrepeatedrrs r$test_homogeneous_text_few_boundarieszBTestTextTilingBlockComparison.test_homogeneous_text_few_boundariesZsPG KK b 01V; 2>N O;;x(8}!!!rN) __name__ __module__ __qualname____doc__rr r,r3r9rrrr4sK"5 #M"rrc.eZdZdZdZdZdZdZdZy)TestTextTilingVocabIntroductionzHTests for the vocabulary introduction method (Hearst 1997, Section 3.2).ctddt}|jt}t |t sJt |dk\sJy)z>Vocabulary introduction should return a list of text segments.r r r rN)rrrrrrrrs rrz5TestTextTilingVocabIntroduction.test_returns_segmentshsB 2>U V;;/0(D)))8}!!!rctddt}|jt}dj |tk(sJyr)rrrrrrs rr z=TestTextTilingVocabIntroduction.test_segments_cover_full_textos8 2>U V;;/0wwx $4444rctddtd}|jt}t |dk(sJ|\}}}}t |dkDsJt |dkDsJt |dkDsJt |dkDsJyr")rrrrrr&s rr,z=TestTextTilingVocabIntroduction.test_demo_mode_returns_scoresus A)@D -.6{a>D; M<:"""=!A%%%< 1$$$:"""rctddtd}|jt\}}}}|D]}|dk\r Jd|dy) z6Vocabulary introduction scores should be non-negative.r r Tr#r.r/z is negativeN)rrrrr0s rtest_gap_scores_non_negativezEC< =6% != =< >rcd}dj|gdzdz}tddtd}|j|\}}}}|t |dzd}|rt |t |z nd }|d ks Jd |d y) zCRepeating the same paragraph should produce low vocab intro scores.r5r6r r Tr#Nrg333333?zAverage later score z too high for repeated text)rrrrrsum)rr7r8rr(r1 later_scores avg_laters r test_homogeneous_text_low_scoresz@TestTextTilingVocabIntroduction.test_homogeneous_text_low_scoressG KK b 01V; A)@D !kk(3 Aq!"#j/Q"6"89 =IC %L(99q   I !),G H I rN) r:r;r<r=rr r,rErKr>rrr@r@esR"5 #>Irr@c(eZdZdZdZdZdZdZy)TestTextTilingCommonzTests common to both methods.cttfD]>}tdd|}|jt}t |dkDr5Jd|dy)zCBoth methods should find at least one boundary in multi-topic text.r r r rzMethod z( found no boundaries in multi-topic textN)rrrrrr)rmethodrrs r!test_both_methods_find_boundariesz6TestTextTilingCommon.test_both_methods_find_boundariess^')@A JF$rQ&IB{{#34HH ! J HI J!  Jrcrtd} |jtJd#t$rYywxYw)z2Invalid similarity method should raise ValueError.invalid)rExpected ValueErrorNrrr ValueErrorrrs rtest_invalid_similarity_methodz3TestTextTilingCommon.test_invalid_similarity_methods; 9 =  KK( ) // /5    * 66crtd} |jtJd#t$rYywxYw)z1Invalid smoothing method should raise ValueError.rR)smoothing_methodrSNrTrVs rtest_invalid_smoothing_methodz2TestTextTilingCommon.test_invalid_smoothing_methods; ) <  KK( ) // /5   rXcttfD]l}tdd|}|jt}tdd|d}|jt\}}}}t |}|t |dz k\rlJy)z:Number of boundaries should be number of segments minus 1.r r r Tr#rGN)rrrrrrHr)rrOrrtt_demor1r+num_boundariess r$test_boundary_count_matches_segmentsz9TestTextTilingCommon.test_boundary_count_matches_segmentss')@A 7F$rQ&IB{{#34H)VtG#*"2"23C"D Aq!Z _N!S]Q%66 66 7rN)r:r;r<r=rPrWr[r_r>rrrMrMs'J 7rrMN) r=nltk.tokenize.texttilingrrrrrr@rMr>rrrasA@."."b4I4In)7)7r