]j#dUdZddlZddlmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZej<eZ e Z!dZ"dZ#dZ$dZ%ddZ&dZ'e"e!jPe!jRgdddddddgZ*e+e e,d<y)zThis file and its contents are licensed under the Apache License 2.0. Please see the included NOTICE for copyright information and LICENSE for a copy of the license. N) defaultdict)-replace_task_data_undefined_with_config_field)AllPermissions)start_job_async_or_sync)DataManagerAction) delete_tasks)AzureBlobImportStorageLink)GCSImportStorageLink)LocalFilesImportStorageLink)RedisImportStorageLink)S3ImportStorageLink)ValidationError)Taskc Ftt|||jdddiS)zRemove duplicated tasks with the same data fields: Duplicated tasks will be deleted and all annotations will be moved to the first of the duplicated tasks. Storage links will be restored for the first task. high)organization_id queue_name response_code)rremove_duplicates_jobr)projectquerysetkwargss ]/root/env/lib/python3.12/site-packages/label_studio/data_manager/actions/remove_duplicates.pyremove_duplicatesrs. //  S !!c t||}t|t|t||||j |j j ddddy)zJob for start_job_async_or_syncTF)maximum_annotations_changed!overlap_cohort_percentage_changedtasks_number_changed from_scratchN)find_duplicated_tasks_by_data*restore_storage_links_for_duplicated_tasksmove_annotationsremove_duplicated_tasks&_update_tasks_counters_and_task_statestasksall)rrr duplicatess rrr(s[.wAJ.z:Z J: 22 $(*." 3rc g}|D]c}||}t|dk(rd}g}|D]$}|d|dzdkDrd}|j|&|D]}|sd}|j|de|j|d}|j|d} |j t|k7r)t d |j d t|d t ||tjd t|d | S)aRemove duplicated tasks from queryset with condition that they don't have annotations :param duplicates: dict with duplicated tasks :param project: Project instance :param queryset: queryset with input tasks :return: queryset with tasks which should be kept Ftotal_annotationscancelled_annotationsrTid)id__inannotations__isnullzDRemove duplicates failed, operation is not finished: queryset count z != removing zK. It means that some of duplicated tasks have been annotated twice or more.zRemoved  duplicated tasks) lenappendfilterexcludecountrrloggerinfo) r)rrremovingdatarootone_task_savednew_roottaskkepts rr%r%9s=H,$ t9>  &D'(40G+HH1L!%%  & ,D!!%T +  ,,.hDIH   8  FD~~3x=(&nn./}S]OLX X  (# KK(3x=/):;< Krc d}|D]}||}t|dk(rd|d}}t|D]\}}|}|d|dzdkDsn||dzdD]}|d|dzdkDstjj |dj j |d||d|dzz }tjd |dd |dd |dd|d<d|d<y) z8Move annotations to the first task from duplicated tasksrr+r,r-Nr.r.)task_idzMoved z annotations from task z to task ) r2 enumeraterobjectsget annotationsupdater7r8)r)total_moved_annotationsr:r;ifirstr>s rr$r$jsH2$ t9> d1g5  GAtE'(40G+HH1L  QM 2D'(40G+HH1L   DJ /;;BB5QU;BW'40C+DtLcGd+dd' T"5677NtTXzlZcdijndocpq-.()01,- 22rc ^ tttttd d}t |D]}||} fd}g}g}|D]-}||r|j ||j |/|sF||d\}} |jj| } |D]s}||d| j| j| j| j} | j|dz }tjd|dd |ddutjd |d y ) zDBuild storage links for duplicated tasks and save them to Task in DB)io_storages_s3importstoragelink io_storages_gcsimportstoragelink&io_storages_azureblobimportstoragelink'io_storages_localfilesimportstoragelink"io_storages_redisimportstoragelinkrcLD]}|j|x}s||fcSy)N)rE)r>linklink_idclassess r_get_storagelinkzDrestore_storage_links_for_duplicated_tasks.._get_storagelinks9 2"hhtn,7,"4='11 2rrAr.)rBkey row_index row_groupstorager+zRestored storage link for task z from source task z Restored z# storage links for duplicated tasksN)r r r r r listr3rDrErVrWrXrYsaver7r8) r)total_restored_linksr:r'rUtasks_without_storagelinkstasks_with_storagelinksr>storage_link_classstorage_link_id link_instancerRrTs @rr#r#s` ,?,@2L3N.D GZ &4  &(""$ 8D%'..t4*11$7  8 #2BCZ[\C]2^ / .66::o:NM2 ) J%))+55+55)11  $)$ 5d4j\ASTklmTnosTtSuv 3&P KK)011TUVrc g}ttD]}|jds||gz }tt}t |j ddddg|}t jdt|dt |D]C}t|d|tj|d|d<||dj|E|Dcic]}t||dkDs|||}}|Dcic]}|||Dcgc]}|d c}} }}t jd t|d t jd | |Scc}wcc}wcc}}w) z>Find duplicated tasks by `task.data` and return them as a dict io_storages_r:r.r,r-z Retrieved z tasks from querysetr+zFound r1zDuplicated tasks: ) dirr startswithrrZvaluesr7r8r2rjsondumpsr3) rrstoragesfieldgroupsr'r>dr)r8s rr"r"seHT    N +  H  F /BD[g^fg hE KK*SZL(<=>U *5d6lGLzz$v,/V tF|##D)* )/E1#fQi.12D!VAY,EJE?I J!Az!}5tT 5 5 JD J KK&Z)):;< KK$TF+,  F5 Js$E+E: E! EE!E!zRemove Duplicated Tasks_FaRConfirm that you want to remove duplicated tasks with the same data fields. Duplicated tasks will be deleted and all annotations will be moved to the first task from duplicated tasks. Also Source Storage Links will be restored if at least one duplicated task has a storage link. Warning: Task assignments (enterprise only) won't be saved.confirm)texttype) entry_point permissiontitleorder experimentaldialogactions)returnN)-__doc__logging collectionsrujsonrgcore.label_configrcore.permissionsr core.redisrdata_manager.actionsrdata_manager.actions.basicrio_storages.azure_blob.modelsr io_storages.gcs.modelsr io_storages.localfiles.modelsr io_storages.redis.modelsr io_storages.s3.modelsr rest_framework.exceptionsr tasks.modelsr getLogger__name__r7all_permissionsrrr%r$r#r"projects_change tasks_deleterwrZ__annotations__rrrs#K+.23D7E;55   8 $ " "".b286Wr:)&668T8TU*N  $ r