]j dZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZmZm Z m!Z!m"Z"m#Z#e$ddejJdZ&ejNjQde&ddddiddddigdddife&dddddigdd gd!ddddigdddife&dddddigdd gd!ddddd"igdd gd!gd#d$d%hife&dddddigdd gd!ddddd"igdd gd!gd#ddife$ddejJd&dd'dd(idd'dd(igdddifgejNjRd)Z*ejNjQd*e$ddejJd+d,-dd'dd(idd'dd(igd.d/igd0dd1d.d2igd3dd1gddgddd.d2igfe$ddejJd+d4-dd'dd(idd'dd(igd.d/igd3dd1d.d2igd0dd1gddgddd.d/igfe$ddejJdd+d45dd'dd(idd'dd(idd'd6d(igd.d/igd3dd1d.d2igd0dd1d.d7igd8dd1gd9d.d/igiddgddd.d7igfe$ddejJd+d4-dd'dd(idd'd:d(idd'dd(idd'd6d(igd.d/igd3dd1d.d/igd;dd1d.d2igd0dd1d.d7igd8dd1gd9d.d/igidddgddd.d7igfe$ddejJd+d4-dd'dd(idd'd:d(idd'ddd1d.d?igd;dd1d.d2igd0dd1d.d7igd8dd1d.d7igd8dd1gd9d.d/igidddd9d.d7igid9d.d7igigddd.d?igfe$ddejJd+d,-dd'dd(idd'dd(idd'd6d(igd.d/igd3dd1d.d2igd0dd1d.d7igd8dd1gd9d.d/igiddgddd.d7igfe$ddejJd+d4-dd'dd(idd'dd(idd'd6d(igd.d/igd3dd1d.d2igd0dd1d.d7igd8dd1gd9d.d/igiddgddd.d7igfe$ddejJd+d"d4@dd'dd(idd'dd(idd'd6d(idd'dAd(idd'dBd(igd.d/igdCdd1d.d2igd3dd1d.d7igd8dd1d.dDigdEdd1d.dFigd>dd1gd9d.d/igiddddgd"dd.d/igfggdGHejNjRdIZ+ejNjRdJZ,ejNj[e dKLejNjQdMej\ejJej^fejNjRdNZ0ejNj[e dKLejNjQdMej\ejJej^fejNjRdOZ1ejNjRdPZ2ejNjRdQZ3ejNj[e dKLejNjRdRZ4ejNj[e dKLejNjRdSZ5ejNj[e dKLejNjRdTZ6ejNj[e dKLejNjRdUZ7ejNjRdVZ8ejNj[e dKLejNjRdWZ9ejNjQdXdYejNjQdZdYejNjRd[Z:Gd\d]e Z;y)^zThis file and its contents are licensed under the Apache License 2.0. Please see the included NOTICE for copyright information and LICENSE for a copy of the license. N)mock)redis_healthcheck)apps)Q)TestCase)_try_breadth_first)Project)ProjectFactory) Annotation PredictionTask)AnnotationFactory TaskFactory) UserFactory)_client_is_annotatorinvite_client_to_projectmake_annotationmake_annotator make_project make_taskTestTa6 title is_publishedsampling label_configz?project_config, tasks, status_code, expected_response_value_setdataz meta info Aztext A) meta_infotextz meta info Bztext Biduncompleted_task_idsrF)result ground_truth)r annotationsdetailz Not found.a8 London)locationr c lt||j}t|r t|||j d|j dt j|d}|jdk(sJtjjj|jt}tjjD]H} | j j#dj%r.|j'| j J|j)d|j d }|j|k(sJt j*|j,} |jd k7r8|j/D]$\} } | d k(r|} | | | vrJd | d | d| dyy)N/api/projects/z /tasks/bulkapplication/jsonr content_type) completed_byF)r&/nextr)r#zFailed on response z: expecting value set "z " for key "")ruserrrpostr"jsondumps status_coder objectsallupdate annotatorsetr r'filterexistsaddgetloadscontentitems) business_client any_clientproject_configtasksr:expected_response_value_setprojectr$r#trdata response_keyexpected_value_sets K/root/env/lib/python3.12/site-packages/label_studio/tests/test_next_task.pytest_next_taskrR,sZ>?+?+?@GJ' W5   K0tzz%7HWi  A ==C  ##1E1E#F5 \\   +}}###7>>@ $ $QTT *+  |59:A ==K '' ' JJqyy !E}}0K0Q0Q0S r ,L,!%;;%9"l#'99 r$UG+BCUBVVabnaoopq r9  rz`project_config, tasks, predictions, annotations, num_annotators, status_code, prelabeling_result12345z )rrr model_versionrsomez prediction Ag?)r%scoreclusterz prediction B?z )rrrmaximum_annotationsrUrztext Cz prediction Cg?r%ztext A 2皙?ztext A 3ztext C 2皙?z prediction A1)rrrrUrZrztext Dztext Eg333333?z prediction D皙?z prediction E)z.MockAnnotatorCountcSN)selfnum_annotatorss rQcountzEtest_next_task_with_active_learning..MockAnnotatorCount.counts ! !rSN)__name__ __module__ __qualname__ri)rhsrQMockAnnotatorCountrcs "rSrm annotators) return_value)taskrLrUr)rpr3rLr.r4r! predictionsrr%rf)rr6rrpatchobjectr ziprr r;createrLrUr>r rCr"r:r8rDrE)mockerrGrHannotator2_clientrIrJrqr'rhr:prelabeling_resultrLrmrp prediction annotationr3r$rNs ` rQ#test_next_task_with_active_learningr{svp >?+?+?PUVGJ' W5-. !2G<"" LL rFr` text_classr,choicesclass_A from_nameto_nametypevaluer z score = 0.5rY)r%rW)rrqz score = 0.1r[z score = 0.3g333333?z score = 0.2r\z score = 0.4r]r. /tasks/bulk/r/r0r2c:jdjd}|jdk(sJtj|j }j d|dd|dtjd}|jd k(sJ|d d S) Nr.r4r! /api/tasks/r" /annotations/rpr%rr2rr )rCr"r:r8rDrEr7r9)r$rprGrLr%s rQ get_next_task_id_and_complete_itzXtest_active_learning_with_uploaded_predictions..get_next_task_id_and_complete_its   . E B C}}###zz!))$  $t*] 34:Y]YcYcdjYk:l ! }}###F|F##rS) dictr UNCERTAINTYrr6r7r"r8r9r:rU)rGconfigrJr$rrLr%s` @@rQ.test_active_learning_with_uploaded_predictionsrs $$ F6?#7#7NG(Z^gjsit]uv wF-(VVY:Z9[\-(VVY:Z9[\-(VVY:Z9[\-(VVY:Z9[\-(VVY:Z9[\  E   L1 58IXj  A ==C   $  B && & , - >> > + - >> > + - >> > + - >> > + - >> >rSz2Multi user locks only supported with redis enabled)reasonrctddd}t||j}||_|j t dddii|j }t dddii|j }td d i|d}td d i|d}|jd |j d } | jdk(sJtj| jd} |jtjtjfvr| ||fvsJt!||h| hz d}n| |k(sJ|jd |j d } | jdk(sJtj| jd|k(sJy)Ntest_label_racesT rrrrr aaabbbemailzann1@testlabelraces.comzann2@testlabelraces.comr.r4r!r"rrrr6rsaverr"rrCr:r8rDrEr UNIFORMrlist configured_projectrGrrrLid1id2ann1ann2r$ selected_ids rQrrs  F6?#7#78GG LLN Vfe_-w 7 : :C Vfe_-w 7 : :C 7$=> ND 7$=> ND >'**U34A ==C  **QYY'-KGOOW-@-@AAsCj(((C: -.q1c!!! >'**U34A ==C   ::aii  &# -- -rScptddd}t||j}||_|j t dddii|j }t dddii|j }td d i|d}td d i|d}|jd |j d } | jdk(sJtj| jd} |jtjtjfvr| ||fvsJt!||h| hz d}n| |k(sJ| }|jd |j d } | jdk(sJtj| jd|k(sJ|jd |j d } | jdk(sJtj| jd|k(sJ|jd |j d } | jdk(sJtj| jd|k(sJy)NrTrrrr rrrzann1@testlabelracesalltaken.comzann2@testlabelracesalltaken.comr.r4r!r"rrrs rQ test_label_races_after_all_takenrEs-  F6?#7#78GG LLN Vfe_-w 7 : :C Vfe_-w 7 : :C 7$EFQU VD 7$EFQU VD >'**U34A ==C  **QYY'-KGOOW-@-@AAsCj(((C: -.q1c!!! C >'**U34A ==C   ::aii  &# -- - >'**U34A ==C   ::aii  &# -- - >'**U34A ==C   ::aii  &# -- -rSctdddd}tjddddd gid g}t||j}t j |_|jtd dd ii|j}td dd ii|j}tddi|d}tddi|d}|jd|jd}|jdk(sJtj|jd|k(sJ|j!d|d||d}|jdk(sJ|jd|jd}|jdk(sJtj|jd|k(sJ|j!d|d||d}|jdk(sJt"j$j|} | j&sJt)j*dr| j,dk(sJ|jd|jd}|jdk(sJtj|jd|k(sJy)NrTr(" rrrZrr}r r~rrrrrrzann1@testbreadthfirst.comzann2@testbreadthfirst.comr.r4r!r"rrrrr2r" businesses?rr8r9rr6r SEQUENCErrrr"rrCr:rDrEr7r r; is_labeledr is_installedaccuracy) rGrannotation_resultrLrrrrr$completed_tasks rQtest_breadth_first_simplerxsl   F # U^aj`kTl mn6?#7#78G''G LLN Vfe_-w 7 : :C Vfe_-w 7 : :C 7$?@'4 PD 7$?@'4 PD >'**U34A ==C   ::aii  &# -- - KuM2#Qb9c dA ==C   >'**U34A ==C   ::aii  &# -- - KuM2#Qb9c dA ==C  \\%%%-N  $ $$ $ &&&#--- >'**U34A ==C   ::aii  &# -- -rSct tdddd}tjddddd gid g t||j t j _ j fd }td dd ii jtd ddii jtd ddii jtddi d}tddi d}tddi d}||}||}||k(sJ||||||}||k(sJ||}||}||k(sJy)NrTrrr}r r~rrc|jdjd}|jdk(sJtj|j d}|j d|d|d|S Nr.r4r!r"rrrrrCr"r:r8rDrEr7r>_rtask_idrrLs rQ complete_taskz3test_breadth_first_overlap_3..complete_taskk ]]^GJJ~~$$$**RZZ(.WI];7^oBpqrSrrrcccrz!ann1@testbreadthfirstoverlap3.comz!ann2@testbreadthfirstoverlap3.comz!ann3@testbreadthfirstoverlap3.com) rr8r9rr6r rrrrr"r) rGrrrrann3 task_id_ann1 task_id_ann2 task_id_ann3rrLs @@rQtest_breadth_first_overlap_3rsm   F # U^aj`kTl mn6?#7#78GG LLNv'144 v'144 v'144 7$GH'SW XD 7$GH'SW XD 7$GH'SW XD!&L &L < '' '$$ &L < '' ' &L &L < '' 'rSctdddd}tjddddd gid gt||jt j _jfd }td dd iitd ddiitddid}tddid}tddid}|||||||jdjd}|jdk(sJ|jdjd}|jdk(sJ|jdjd}|jdk(sJy)N(test_try_take_last_task_at_the_same_timeTr(rrr}r r~rrc|jdjd}|jdk(sJtj|j d}|j d|d|d|Srrrs rQrz?test_try_take_last_task_at_the_same_time..complete_taskrrSrrrrzann1@lasttask.comzann2@lasttask.comzann3@lasttask.comr.r4r!r))rr8r9rr6r rrrrrrCr"r:) rGrrrrrrrrLs @@rQrrs~8 F # U^aj`kTl mn6?#7#78G''G LLNv'1 v'1 7$78'4 HD 7$78'4 HD 7$78'4 HD$$$ N7::,e4 5B >>S  N7::,e4 5B >>S  N7::,e4 5B >>S  rSc2tdddd}tjddddd gid g}t||j}t j |_|jtd dd ii|j}td dd ii|j}tddi|d}tddi|d}|jd|jd} | jdk(sJtj| jd|k(sJ|j!d|d||d} | jdk(sJ|jd|jd} | jdk(sJtj| jd|k(sJ|jd|jd} | jdk(sJtj| jd|k(sJ|j!d|d||d} | jdk(sJt"j$j|} | j&sJt)j*dr| j,dk(sJ|jd|jd} | jdk(sJtj| jd|k(sJ|jd|jd} | jdk(sJtj| jd|k(sJy)NrTr(rrr}r r~rrrrrrzann1@testbreadthlabelraces.comzann2@testbreadthlabelraces.comr.r4r!r"rrrrr2rrrr) rrGrrrLrrrrr$rs rQ"test_breadth_first_with_label_racers  F # U^aj`kTl mn6?#7#78G''G LLN Vfe_-w 7 : :C Vfe_-w 7 : :C 7$DEwPT UD 7$DEwPT UD >'**U34A ==C   ::aii  &# -- - KuM2#Qb9c dA ==C   >'**U34A ==C   ::aii  &# -- - >'**U34A ==C   ::aii  &# -- - KuM2#Qb9c dA ==C  \\%%%-N  $ $$ $ &&&#--- >'**U34A ==C   ::aii  &# -- - >'**U34A ==C   ::aii  &# -- -rSc tddd}tjdddddgid g}t||j}t j |_|jtd d i|d}td d i|d}g}d }t|D]#} |jdddt| ii%|jd|jdtj|d} | jdk(sJ|j!d|jdtjd dddd} | jdk(sJt"j$j'|jj'd } | j)dk(sJt"j$j'|jj'd} | j)dk(sJ| j+j} | j+j}|j-d|jd} | jdk(sJtj.| j0d| k(sJ|j-d|jd} | jdk(sJtj.| j0d| k(sJt"j$j-| j3sJ|j-d|jd} | jdk(sJtj.| j0d| k(sJ|jd| d | |d!"} | jdk(sJ|j-d|jd} | jdk(sJtj.| j0d| k(sJ|jd| d | |d!"} | jdk(sJ|j-d|jd} | jdk(sJtj.| j0d|k(sJy#)$a" 2 annotators takes and finish annotations one by one depending on project settings overlap create project make annotation result make 2 annotators bulk create tasks change project settings check overlap next annotate tasks check code comments rTrrr}r r~rrrz!ann1@testlabelracewithoverlap.comz!ann2@testlabelracewithoverlap.comr(rthis is r.rr/r0r2/2rZoverlap_cohort_percentageshow_overlap_firstr!rLoverlaprr4r"rrrrrN)rr8r9rr6r rrrrrangeappendstrr7r"r:rrr r;r@rifirstrCrDrEhas_lock)rrGrrrLrrrJ num_tasksir$rMt1 overlap_idother_ids rQtest_label_race_with_overlaprWs "  F # U^aj`kTl mn6?#7#78G''G LLN 7$GH'SW XD 7$GH'SW XD EI 9 > fv#a&':;<=>  L1 58IXj  A ==C     A& ZZPRjno p'  A ==C   GJJ/66q6AA 779>>   WZZ  0 7 7 7 BB 88:??Jxxz}}H >'**U34A ==C   ::aii  &* 44 4 >'**U34A ==C   ::aii  &* 44 4 <<  z  * 3 3 55 5 >'**U34A ==C   ::aii  &* 44 4 K |=9_p@q rA ==C   >'**U34A ==C   ::aii  &* 44 4 K |=9_p@q rA ==C   >'**U34A ==C   ::aii  &( 22 2rSc  tddd}tjdddddgid g}t||j}t j |_|jtd d i|d}td d i|d}g}d }t|D]#} |jdddt| ii%|jd|jdtj|d} | jdk(sJ|j!d|jdtjd dddd} | jdk(sJt"j$j'|jj'd } | j)dk(sJt"j$j'|jj'd} | j)dk(sJ| j+j} | j+j}| dtjdddddgid gtjgd}|j-d|jd} | jdk(sJtj.| j0d| k(sJ|j-d|jd} | jdk(sJtj.| j0d| k(sJt"j$j-| j3sJ|j-d|jd} | jdk(sJtj.| j0d| k(sJ|jd!| d"|#} | jdk(sJ|j-d|jd} | jdk(sJtj.| j0d| k(sJ|jd!| d"| |d$#} | jdk(sJ|j-d|jd} | jdk(sJt5| |tj.| j0d|k(sJ|j-d|jd} | jdk(sJy%)&a- 2 annotators takes and leaves with draft annotations one by one depending on project settings overlap create project make annotation result make 2 annotators bulk create tasks change project settings check overlap next annotate tasks check code comments rTrrr}r r~rrrzann1@testlabelracewdrafts.comzann2@testlabelracewdrafts.comr(rrr.rr/r0r2rrrr!rrrgZd;@)rp lead_timedraftr%r4r"rrrrrN)rr8r9rr6r rrrrrrrr7r"r:rrr r;r@rirrCrDrErprint)rrGrrrLrrrJrrr$rMrrrannotation_draft_results rQ%test_label_w_drafts_race_with_overlaprs"  F # U^aj`kTl mn6?#7#78G''G LLN 7$CDgt TD 7$CDgt TD EI 9 > fv#a&':;<=>  L1 58IXj  A ==C     A& ZZPRjno p'  A ==C   GJJ/66q6AA 779>>   WZZ  0 7 7 7 BB 88:??Jxxz}}H'FIYbendoXp q r **R.  >'**U34A ==C   ::aii  &* 44 4 >'**U34A ==C   ::aii  &* 44 4 <<  z  * 3 3 55 5 >'**U34A ==C   ::aii  &* 44 4 K |=9@W XA ==C   >'**U34A ==C   ::aii  &* 44 4 K |=9_p@q rA ==C   >'**U34A ==C   *h ::aii  &( 22 2 >'**U34A ==C  rSc jtddd}tjdddddgid g}t||j}t j |_|jtd d i|d}td d i|d}g}d }t|D]#}|jdddt|ii%|jd|jdtj|d} | jdk(sJ|j!d|jdtjdd id} | jdk(sJt#d|j%d|jd} tj&| j(d} |jd| d| |dt#d|j%d|jd} tj&| j(d| k(sJt#d |j%d|jd} tj&| j(d} | | k7sJt#d!td"D]H}|j%d|jd} tj&| j(d| k(rHJy)#NrTrrr}r r~rrrzann1@testfetchfinal.comzann2@testfetchfinal.comr(rrr.rr/r0r2rrZr!z#ann1 takes any task and complete itr4r"rrrrzSann2 takes the same task (because of depth-first) but just lock it - don't completezann1 takes another taskz8ann1 should never take task_id since he has completed itr)rr8r9rr6r rrrrrrrr7r"r:rrrrCrDrE) rGrrrLrrrJrrr$ranother_task_ids rQtest_fetch_final_taken_taskr-s   F # U^aj`kTl mn6?#7#78G''G LLN 7$=> ND 7$=> ND EI 9 > fv#a&':;<=>  L1 58IXj  A ==C     A&TZZ9NPQ8R-Sbt  A ==C   /0 >'**U34Ajj#D)GII G9M2'Uf9gIh _` >'**U34A ::aii  &' 11 1 #$ >'**U34Ajj+D1O g %% % DE 1X> HH~gjj\7 8zz!))$T*o===>rScv tddtjdd}t||jd dd d d ggd gd dddddddd }ddddd dddgddgddgddgddgdd gd!d"gd#d$gd%d&gd'd(gd)d*gd+d,gd-d.gd/d0gd1d2gd3d4gd5d6gd7d8gd9d:ggd gd dddd; d<}g}t |D]$}|j td=d>|d?i d&tjd@5t dAD]W}tdBdCdD|dEii }t |D]3}t|gdAz gdAzz||jdF|j5Y dddtdBdCdGii }tdH|djdI jdJ}t!j"|j$dK |jk(sJdN fdL } |j'sJt)j(} | |dt)j(| z dksJdM|j'sJy#1swYxYw)ONtest_with_failed_matching_scoreTraW
)rrrrZrFr` Yv_lLEp_8I polygonlabelsgmX /P@glPlGR@t11)pointsrz$imageimgtagrii ) r"rrsourcerr parent_idimage_rotationoriginal_widthoriginal_heightz NsccF-AYMTg޺(FA@gxJ.@g`[=A@g6+@g} @@gǿzv?@g8j.@gt+>@g.Bz0@gN=@gZD2@giV=@g2Vё4@g+l7 <@gE$W,7@g78eg=@g6j9@gĠZSh>@gVL̨9@g6h?@gUv&F9@guU`@@g\X8@gH$A@g4ݵC!8@gVIA@g5=7@gV8B@gj6@g qB@gwsS5@goB@gf'93@gn'QB@g>21@g>~ B@g 3ڼ0@) r"rrrrrrrrrrannz @testwithbadannotationresult.comz>tasks.models.update_project_summary_annotations_and_is_labeled rimagez$https://data.s3.amazonaws.com/image/z.jpg)r%r3z3https://data.s3.amazonaws.com/image/uncompleted.jpgz7ann1 takes any task with bad annotation and complete itr.r4r"ctd|tddid}|jddtjgdy) NzAsync annotation submitrz'new_ann@testwithbadannotationresult.comTrrrr)rrr7r8r9)new_ann good_resultrLrs rQmake_async_annotation_submitzEtest_with_bad_annotation_result..make_async_annotation_submitsW '( ?$g/X%Y[bdhiG '- 0!TZZ -FG  rSzTime of annotation.submit() increases - that might be caused by redundant computations over the rest of the tasks - check that only a single task is affected by /api/tasks//annotationsre)rr rrr6rrrrrrrrr>r"rrCr8rDrErtime)rGr bad_resultrhannsrrpuncompleted_taskr$rrMrrLrs @@@rQtest_with_bad_annotation_resultris//!!  F&6?#7#7NG/1BCDX]W^_ J#$56"$67#%78#%67#%78#%67"$67#%78#%78#%67#%67#%78"$56"$67#%67#%67#%67"$56"$67'*$W- 0C"KHN D > "i NGs1#5U-V#WY`bfghi T Ur Afw2VWXVYY]0^&_`bijD>*  *|b0K=23EEW[\]W^WhWhikokrkr  !&74i*j!kmtu CD Q nWZZL67Ajj#D)G &)) )) )   $ $ && & A a) a  O OO   $ $ && &Ks #A&H//H8setup_before_upload)FTrc z|}tddd|dd}t||jtjddd d d gid gd }d}d}|rK|j d j dtjd|dd}|jdk(sJd}g} t|D]#} | jdddt| ii%|jd j dtj| d}|jdk(sJ|rI|j d j dtjd|dd}|jdk(sJt|dz |z} tjjt!j t!dzj#| k(sJfd} t%ddid} t%ddid}t| D]} | | | |ft'd tjjt!j t!dzD}t'd!tjjt!j t!d"zD}|r |sJ|sJy|rJ|rJy)#Ntest_overlap_firstTrzUniform samplingr)rrrZrrrr}r r~rrir.rr()rZrr/r0r!Frrrr2gY@) project_id) overlap__gtc|jdjd}|jdk(sJtj|j d}|j d|d|dyrrrs rQrz)test_overlap_first..complete_tasksf ]]^GJJ~~$$$**RZZ(.WI];7^oBpqrSrzann1@testoverlapfirst.comzann2@testoverlapfirst.comc34K|]}|jywrer.0rMs rQ z%test_overlap_first..*s- -sc36K|]}|j ywrerr s rQr z%test_overlap_first..-s4ALL4sr)rrr6r8r9rrr"r:rrrr7intr r;r@rrirr<)rGrrcrrrsetup_after_uploadr$rJrexpected_tasks_with_overlaprrr"all_tasks_with_overlap_are_labeled)all_tasks_without_overlap_are_not_labeledrrLs @@rQrrs A "-# F 6?#7#78G # U^aj`kTl mnI ! GGWZZL *ATmno+   }}###" E 9 > fv#a&':;<=>  L1 58IXj  A ==C   GGWZZL *ATmno+   }}###"%&?%&G)&S"T <<  qGJJ7!:JJ K Q Q SWr rr rr 7$?@'4 PD 7$?@'4 PD . /1d]4001*--"ll11!wzz2JQ[\M]2]^-*&144"&,,"5"5a7::6NQR[\Q]6]"^41-111888555<<<=$!/ !)-!  $2 !).$  rSct|j}t|j}t|j}tjd|dt|dt|dt|dt|dtj j |j}t||j|j}||k(sJy)zh Test that _try_breadth_first excludes GT tasks when annotator_evaluation_enabled=True. rr(Frpr&TN) rrr create_batchr r;r@rr6rgtask_1task_2 task_3_gtrJr%s rQ8test_excludes_ground_truth_tasks_when_evaluation_enabledzLTestTryBreadthFirst.test_excludes_ground_truth_tasks_when_evaluation_enabledKs T%;%;<T%;%;<(>(>?  &&qvEJ vE: yt<yu=yu= ##D,B,B#C$E499d6L6LMrSct|j}t|j}t|j}tjd|dt|dt|dt|dt|dtj j |j}t||j|j}||k(sJy)zi Test that _try_breadth_first includes GT tasks when annotator_evaluation_enabled=False. rr(FrTN) rrrrr r;r@rr6rs rQ9test_includes_ground_truth_tasks_when_evaluation_disabledzMTestTryBreadthFirst.test_includes_ground_truth_tasks_when_evaluation_disabledhs T%>%>?T%>%>?(A(AB  &&qvEJ vE: yt<yu=yu= ##D,E,E#F$E499d6O6OP"""rSct|j}t|dtjj |j}t ||j|j}|Jy)NrTr)rrrr r;r@rr6)rgtask_gtrJr%s rQGtest_returns_none_when_no_tasks_with_annotations_and_evaluation_enabledz[TestTryBreadthFirst.test_returns_none_when_no_tasks_with_annotations_and_evaluation_enableds_d&<&<=wT: ##D,B,B#C$E499d6L6LM~~rSN)rjrkrl classmethodrr#r%r(rfrSrQrr9s#   :#: rSr)<__doc__r8runittestrpytest core.redisr django.appsrdjango.db.modelsr django.testrprojects.functions.next_taskrprojects.modelsr projects.tests.factoriesr tasks.modelsr r r tasks.tests.factoriesrrusers.tests.factoriesrutilsrrrrrrrr*_project_for_text_choices_onto_A_B_classesmark parametrize django_dbrRr{rskipifrrrrrrrrrrrrrrrfrSrQr=s_   ( ;#355@-.2    .* E 7}hGH}hGH   ) *  7+8J14az5$Q#R}hGH    ) *  7+8J14az5$Q#R +8J14az5$Q#R    ~ & " 7+8J14az4$P#Q +8J14az5$Q#R   ) * ! ,, 8X> ?&W_iqJrAs t  ) *! kGJVrWJXr@f ! ,,% 8X> ?&W_iqJrAs t#^45PQR#^45PQR     n % &5 < ! ,,% 8X> ?&W_iqJrAs t#^45PQR#^45PQR     n % &5 < ! ,,$%%   hABhABhAB  $^45PQR#^45PQR#^45PQR  V^456   n % &C" J ! ,,% hABh CDhABhAB  $^45PQR#^45PQR#^45PQR#^45PQR  V^456    n % &G$ N ! ,,% hABh CDh CDhABhABh CD  $^45PQR#^45PQR#_56QRS#^45PQR#^45PQR#^45PQR  V^456V^456V^456    o & 'S* Z ! ,,% hABhABhAB  $^45PQR#^45PQR#^45PQR  V^456   n % &A! H ! ,,% hABhABhAB  $^45PTU#^45PTU#^45PTU  V^456   n % &A! H ! ,,%$%   hABhABhABhABhAB  $^45PTU#^45PTU#^45PTU#^45PTU#^45PTU  &.12 3T4t L n % &C" q[x }hR $GS hT $GN0?0?f)++4higoow7J7JGL\L\%]^"._j".J)++4higoow7J7JGL\L\%]^-._j-.`3.3.l.(.(b)++4hi0!j0!f)++4hi<.j<.~)++4hi^3j^3B)++4hin jn b8>8>v)++4hir'jr'j. >-}=S=>?S=lW(WrS