:jO2 <ddlZddlZddlZddlmZddlmZmZmZm Z ddl m Z ddl m ZddlmZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ejFe$Z%dddedededef dZ&ddde'ddfdZ(d)dZ)d)dZ*d)dZ+ddddde,dededdedef dZ-ddde,dededef dZ.dd ddde,ded!ee/deddf d"Z0ddde1e,e1ffd#Z2ddd$e1e,efddfd%Z3ddd$e1e,efddfd&Z4ddd$e1e,efddfd'Z5ddde,dededef d(Z6y)*N)deepcopy)AnyCallableOptionalUnion)Version)_DeviceDtypeModuleMixin) Checkpoint EarlyStopping) WandbLogger)_SubprocessScriptLauncher)_get_sigkill_signal) TrainerStatus)_TunerExitException) is_overridden)rank_zero_inforank_zero_warntrainer pl.Trainer trainer_fnargskwargsreturnc< |jj,|jjj|g|d|i|S||i|S#t$rNt ||j t j|j_ d|j_ Yyt$r}tdtjtjtjt!|||j |jj}t#|t$r|j't)t+j,dYd}~yd}~wt.$r3}t!|||j d|j_ d}~wwxYw)amError handling, intended to be used only for main trainer function entry points (fit, validate, test, predict) as all errors should funnel through them. Args: trainer_fn: one of (fit, validate, test, predict) *args: positional arguments to be passed to the `trainer_fn` **kwargs: keyword arguments to be passed to `trainer_fn` Nrz= Detected KeyboardInterrupt, attempting graceful shutdown ...)strategylauncherlaunchr_call_teardown_hook _teardownrFINISHEDstatestatusstageKeyboardInterruptrsignalSIGINTSIG_IGN _interrupt isinstancer killrsysexit BaseException)rrrr exceptionrs s/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/lightning/pytorch/trainer/call.py_call_and_handle_interruptr1$s>    $ $ 037##,,33JaawaZ`a a4*6** #G$,55 "   WX fmmV^^47I&##,, h 9 : MM-/ 0  7I&"  s2AA A AF"F*B-E F(.FFr/ctj|j_t |d||j t |d||jj||jD]}|jdy)N on_exceptionfailed) r INTERRUPTEDr"r#_call_callback_hooks datamodule_call_lightning_datamodule_hookrr3loggersfinalize)rr/loggers r0r)r)Lsk(44GMM.)<%'K !!),//"!"c>|jjJ|jj}|jjD].}t |t s|j j|_0t|jd}|D]}t|ds|j}|j jd|jt|d|t!|d|t#|d||j jdy)Nc$t|t S)N)r*r )r;s r0z"_call_setup_hook..bsZP[=\9\r<)key experiment pre_setupsetupr$ post_setup)r"fnlightning_modulemodulesr*r r root_device_devicesortedr9hasattrrAbarrierr7r8r6_call_lightning_module_hook)rrFmoduler9r;_s r0_call_setup_hookrQVs ==   '' '   B**224: f5 6$--99FN: W__*\]G" 6< (!!A" [)%'C'4; \*r<ctd|jr/|jj5t |ddddtd|jrv|jj 5|jj5|j j5t |ddddddddddyy#1swYxYw#1swY'xYw#1swY+xYw#1swYyxYw)Nconfigure_sharded_modelconfigure_model)rrGrmodel_sharded_contextrNtensor_init_contextprecision_pluginmodule_init_context)rs r0_call_configure_modelrYss.0H0HI    3 3 5 L '1J K L &(@(@A    0 0 2 D    2 2 4 D  $ $ 8 8 : D (1B C  D D D DB  L L  D D D D D DsG C6C7C+, C9C+C7CC( $C++C4 0C77Dc|jjJ|jj}|jt|d|t |d|t |d|d|j _d|j _|jD]}|jd|jjy)NteardownrDsuccess) r"rFr7r8r6rNrG_current_fx_name_metric_attributesr9r:profilerdescribe)rrFr;s r0rrs ==   '' '   B%'2F*B72>04G-26G///# "# r<) pl_module hook_namerazpl.LightningModulectj|jjd||xs |j}| t dt ||}t|sy|j}||_|jjd|jjd|5||i|}ddd||_S#1swYxYw)Nz!: calling lightning module hook: z3No `LightningModule` is available to call hooks on.z[LightningModule].) logdebug __class____name__rG TypeErrorgetattrcallabler]r_profile)rrbrarrrF prev_fx_nameoutputs r0rNrNsII""++,,Mi[YZ5W55IMNN I &B B<--L!*I    ! !$5i6I6I6R6R5SSTU^T_"` a%T$V$%".I M %%s , CCctj|jjd||j t dt |j|}t|rQ|jjd|jjjd|5||i|cdddSy#1swYyxYw)Nz%: calling lightning datamodule hook: z7No `LightningDataModule` is available to call hooks on.z[LightningDataModule]rd) rerfrgrhr7rirjrkr_rl)rrbrrrFs r0r8r8s II""++,,QR[Q\]^!QRR ##Y /B|    % %(=g>P>P>Z>Z>c>c=ddefoep&q r 't&v& ' '  ' s &B99C)monitoring_callbacksrpcxtj|jjd||j}|r|j }||_|j }|dur'|Dcgc]}t|ttfs|}}n*|dur&|Dcgc]}t|ttfr|}}|D]e} t| |} t| s|jjd| jd|5| ||jg|i|dddg|r|_yycc}wcc}w#1swYxYw)Nz: calling callback hook: TF [Callback]rd)rerfrgrhrGr] callbacksr*r r rjrkr_rl state_key) rrbrprrrarmrscbcallbackrFs r0r6r6sHII""++,,Ei[QR((I 11 %. "!!It#"+[Bz"}j>Y/ZR[ [  &"+_B:b=R\B]3^R_ _G Xy ) B<!!))Jx7I7I6J!I;*WX G7G44FtFvF G GG %1 "\_  G Gs$$D&D&D++D+9D00D9 cpi}|jD]$}|j}|s|||j<&|S)zzCalled when saving a model checkpoint, calls and returns every callback's `state_dict`, keyed by `Callback.state_key`.)rs state_dictrt)rcallback_state_dictsrvrxs r0_call_callbacks_state_dictrzsK%%B((* 7A !3!3 4B r< checkpointc2|j}|r|j}d|_|jD]Q}|jj d|j d5|j ||j|dddS|r|_yy#1swYhxYw)zXCalled when saving a model checkpoint, calls every callback's `on_save_checkpoint` hook.on_save_checkpointrrz.on_save_checkpointN)rGr]rsr_rlrtr})rr{rarmrvs r0"_call_callbacks_on_save_checkpointr~s((I 11 %9 "%%W    % % 83E3E2FFY&Z [ W  ' '1I1I: V W WW%1 " W Ws B  B cT|j}|r|j}d|_|jd}|yt|dtdk}|jDchc]}|r |j n |j }}|j|z }|rtdt|d|jD]Q} |jjd| j d 5| j||j|dddS|r|_yycc}w#1swYmxYw) zCalled when loading a model checkpoint. Calls every callback's `on_load_checkpoint` hook. We have a dedicated function for this rather than using `_call_callback_hooks` because we have special logic for getting callback_states. on_load_checkpointrsNzpytorch-lightning_versionz1.5.0devzBe aware that when using `ckpt_path`, callbacks used to create the checkpoint need to be provided during `Trainer` instantiation. Please add the following callbacks: rdrrz.on_load_checkpoint) rGr]getrrs_legacy_state_keyrtkeysrlistr_rlr) rr{rarmcallback_statesis_legacy_ckptrucurrent_callbacks_keys differencervs r0"_call_callbacks_on_load_checkpointrsG((I 11 %9 ">Hnn[>YOZ(CDEPZH[[Naharars[]nb22",,Vss %%'*@@J 4484D3EQ H %%W    % % 83E3E2FFY&Z [ W  ' '1I1I: V W WW%1 "t W Ws!D&DD' c|jd}|y|jD]V}|j|j|j|j}|s;t |}|j |Xy)zQCalled when loading a model checkpoint, calls every callback's `load_state_dict`.rsN)rrsrtrrload_state_dict)rr{rrvr"s r0_call_callbacks_load_state_dictr*sq>Hnn[>YO%%,##H$6$68K8KHLfLf8gh UOE  $ $U + ,r<ctj|jjd||j}|j }||_t |j|}t|sy|jjd|jjjd|5||i|}ddd||_S#1swYxYw)Nz: calling strategy hook: z [Strategy]rd) rerfrgrhrGr]rjrrkr_rl)rrbrrrarmrFrns r0_call_strategy_hookr8s II""++,,Ei[QR((I--L!*I !!9 -B B<    ! !Jw/?/?/I/I/R/R.SSTU^T_"` a%T$V$%".I M %%s / C  C)rrrN)7loggingr&r,copyrtypingrrrrpackaging.versionrlightning.pytorchpytorchpl-lightning.fabric.utilities.device_dtype_mixinr lightning.pytorch.callbacksr r lightning.pytorch.loggersr &lightning.pytorch.strategies.launchersr 5lightning.pytorch.trainer.connectors.signal_connectorr lightning.pytorch.trainer.statesr&lightning.pytorch.utilities.exceptionsr)lightning.pytorch.utilities.model_helpersr%lightning.pytorch.utilities.rank_zerorr getLoggerrhrer1r.r)rQrYrstrrNr8boolr6dictrzr~rrrr<r0rs 11%QA1LU:FCPg!% %(%SV%be%jm%P" ""4"+:D" 615   ,-     <     ,,0 2 22 2#4. 2  2  2@  c4i  2 2$sTWx. 2]a 2 !2 !2$sTWx.!2]a!2H ,\ ,tCQTH~ ,Z^ ,     r<