:j{m#NddlZddlmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl m ZddlmZddlmZdd lmZmZmZmZdd lmZdd lmZer erdd lmZmZesd gZedZ eGddeZ!GddZ"de de de#de$de$def dZ%ddde&e'e$ffdZ(de de$de$de$de f d Z)d!d"d#e&e'e fd$e*de#d%e&e'e$fde f d&Z+e jXd!d"d'ed$e*de#d%e&e'e$fd(e#de fd)Z-d*e e'ee'fd+e e'ee'fddd,e$de.e e e e ff d-Z/ d:d!d"d.ed/ed$e*de#d0e"d%e&e'e$fd(e#de fd1Z0 d;d*e e'ee'fd+e e'ee'fd2e e'ejbfd$e*d3e de#d4ee*d5ee*d6ee e'e jdfd,ee$de$d7e$d(e#d8e#de e e.e e fffd9Z3y)<N)Sequence)unique) TYPE_CHECKINGListOptionalUnion)Tensor) functional) DataLoader)Literal)TokenizedDataset_get_progress_bar_input_data_collator_load_tokenizer_and_model)EnumStr)_TRANSFORMERS_GREATER_EQUAL_4_4)PreTrainedModelPreTrainedTokenizerBaseinfolm) kl_divergencealpha_divergencebeta_divergence ab_divergencerenyi_divergence l1_distance l2_distancel_infinity_distancefisher_rao_distancecJeZdZdZedefdZdZdZdZ dZ dZ d Z d Z d Zd Zy )_IMEnumz8A helper Enum class for storing the information measure.returncy)NzInformation measurer#x/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/torchmetrics/functional/text/infolm.py_namez _IMEnum._name:s$r$rrrrrrrrrN)__name__ __module__ __qualname____doc__ staticmethodstrr& KL_DIVERGENCEALPHA_DIVERGENCEBETA_DIVERGENCE AB_DIVERGENCERENYI_DIVERGENCE L1_DISTANCE L2_DISTANCEL_INFINITY_DISTANCEFISHER_RAO_DISTANCEr#r$r%r r 6sQB%3%%$M)'O#M)KK//r$r c 4eZdZdZ ddedeedeeddfdZded edefd Z e ded edefd Z ded edefd Z ded edefd Z ded edefdZded edefdZe ded edefdZe ded edefdZe ded edefdZe ded edefdZy)_InformationMeasureuA wrapper class used for the calculation of different information measures. This metric can be used to measure the information between the discrete reference distributions of predicted and reference sentences. The class also handles input validation for `alpha` and `beta` parameters. Args: information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. Raises: ValueError: If information measure is one from alpha, AB or Rényi divergence and parameter `alpha` is `None`. ValueError: If information measure is one from beta or divergence and parameter `beta` is `None`. ValueError: If information measure is alpha divergence and parameter `alpha` equals 0 or 1. ValueError: If information measure is beta divergence and parameter `beta` equals 0 or -1. ValueError: If information measure is AB divergence and parameter `alpha`, `beta` or `alpha + beta` equal 0. ValueError: If information measure is Rényi divergence and parameter `alpha` equals 1. Ninformation_measurealphabetar!ctj||_tjtjtj f}|j|vrt |tstd|d|jtjtjfvrt |tstd|d|jtjk(r#t |tr|dvrtd|d|jtjk(r#t |tr|dvrtd|d|jtjk(r1| |td||fDs d ||||zfvrtd |d|jtj k(r$t |tr|d k(rtd |d|xsd |_ |xsd |_ y) Nz0Parameter `alpha` is expected to be defined for .z/Parameter `beta` is expected to be defined for )rzFParameter `alpha` is expected to be float differened from 0 and 1 for )rzFParameter `beta` is expected to be float differened from 0 and -1 for c3>K|]}t|t yw)N) isinstancefloat).0ps r% z/_InformationMeasure.__init__..sD 1e,,DsrzRParameters `alpha`, `beta` and their sum are expected to be differened from 0 for r=z@Parameter `alpha` is expected to be float differened from 1 for ) r from_strr8r.r0r1r@rA ValueErrorr/anyr9r:)selfr8r9r: _bad_measuress r%__init__z_InformationMeasure.__init__is $+#3#34G#H  1173H3H'JbJbc  # #} 4Zu=UOPcOddefg g  # #(?(?AVAV'W W`jkoqv`wNObNccdef f  # #w'?'? ?TY[`IaejntetXYlXmmno   # #w'>'> > SWY^H_cgkrcrXYlXmmno   # #w'<'< < M|DudmDDeUY[`cg[gMhHhd&'q*   # #w'?'? ?TY[`Iaejnoeo_`s_ttuvw wZa IA r$preds_distributiontarget_distributionct|d|jj}tj|||S)N _calculate_)getattrr8valuetorch nan_to_num)rHrKrLinformation_measure_functions r%__call__z_InformationMeasure.__call__s>'.t{4C[C[CaCaBb5c'd$ <=OQd effr$cbtj|tj||z zdS)aCalculate Kullback-Leibler divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Kullback-Leibler divergence between discrete distributions of predicted and reference sentences. r>dim)rQsumlogrKrLs r%_calculate_kl_divergencez,_InformationMeasure._calculate_kl_divergences,yy,uyy9KNa9a/bbhjkkr$c|j|jdz z}dtj||jz|d|jz zzdz |z S)aCalculate alpha divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Alpha divergence between discrete distributions of predicted and reference sentences. r=r>rV)r9rQrX)rHrKrL _alpha_denoms r%_calculate_alpha_divergencez/_InformationMeasure._calculate_alpha_divergences]zzTZZ!^4  -tzz9rV)rQrYrXr:r9)rHrKrLabcs r%_calculate_ab_divergencez,_InformationMeasure._calculate_ab_divergences IIeii 3 DJJ8N OUWX Y TYY$))djj0 11 IIeii 2tyy4::7M NTVW X TZZ499tzz1 22 IIeii 3TZZ ?BTVZV_V_B_ _egh i TZZ$)) ##1uqyr$c4d|_|j||S)aCalculate beta divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Beta divergence between discrete distributions of predicted and reference sentences. g?)r9rcrHrKrLs r%_calculate_beta_divergencez._InformationMeasure._calculate_beta_divergences  ,,-?ATUUr$ctjtj||jz|d|jz zzd|jdz z S)uCalculate Rényi divergence between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Rényi divergence between discrete distributions of predicted and reference sentences. r=r>rV)rQrYrXr9res r%_calculate_renyi_divergencez/_InformationMeasure._calculate_renyi_divergencesS IIeii 3TZZ ?BTYZ]a]g]gYgBh hnpq r ZZ!^ r$c8tj||z ddS)aCalculate L1 distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L1 distance between discrete distributions of predicted and reference sentences. r=r>rCrWrQnormrZs r%_calculate_l1_distancez*_InformationMeasure._calculate_l1_distancezz-0BBaRPPr$c8tj||z ddS)aCalculate L2 distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L2 distance between discrete distributions of predicted and reference sentences. r>rjrkrZs r%_calculate_l2_distancez*_InformationMeasure._calculate_l2_distancernr$cJtj||z tddS)aCalculate L-infinity distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: L-infinity distance between discrete distributions of predicted and reference sentences. infr>rj)rQrlrArZs r%_calculate_l_infinity_distancez2_InformationMeasure._calculate_l_infinity_distance s#zz-0BBeElXZ[[r$c dtjtjtj||zj dddzS)aCalculate Fisher-Rao distance between discrete distributions of predicted and reference sentences. Args: preds_distribution: Discrete reference distribution of predicted sentences over the vocabulary. target_distribution: Discrete reference distribution of reference sentences over the vocabulary. Return: Fisher-Rao distance between discrete distributions of predicted and reference sentences. rpr>rr=)rQacosclampsqrtrXrZs r%_calculate_fisher_rao_distancez2_InformationMeasure._calculate_fisher_rao_distancesC5::ekk%**5GJ]5]*^*b*bce*fhiklmnnnr$)NN)r'r(r)r*$_ALLOWED_INFORMATION_MEASURE_LITERALrrArJr rTr+r[r^rcrfrhrmrqrtryr#r$r%r7r7IsD"& $ "A""uo "  "Hg6ggSYg lV lRX l]c l lf[afl$6X^ci,VVVZ`VekV f[afl" Q6 QPV Q[a Q Q Q6 QPV Q[a Q Q \6 \X^ \ci \ \ o6 oX^ oci o or$r7 input_idsattention_maskidf batch_size num_workersr!c8t|||}t|||S)aHPrepare dataloader. Args: input_ids: Indices of input sequence tokens in the vocabulary. attention_mask: Mask to avoid performing attention on padding token indices. idf: A bool indicating whether normalization using inverse document frequencies should be used. batch_size: A batch size used for model processing. num_workers: A number of workers to use for a dataloader. Return: An instance of ``torch.utils.data.DataLoader`` used for iterating over examples. )r~r)r r )r{r|r}r~rdatasets r%_get_dataloaderr+s!*y.#>G g*+ NNr$ tokenizerrc`|j|j|j|jdS)aBuild a dictionary of model/tokenizer special tokens. Args: tokenizer: Initialized tokenizer from HuggingFace's `transformers package. Return: A dictionary containing: mask_token_id, pad_token_id, sep_token_id and cls_token_id.  mask_token_id pad_token_id sep_token_id cls_token_idr)rs r%_get_special_tokens_maprDs2#00!..!..!..  r$rrrcr|j||j|z|j|z}|S)a(Generate a token mask for differentiating all special tokens in the input batch. There are 0s for special tokens and 1s otherwise. Args: input_ids: Indices of input sequence tokens in the vocabulary. pad_token_id: An id of ```` tokens that are used to make arrays of tokens the same size for batching purpose cls_token_id: An id of ```` token that represents the class of the input. (It might be ```` token for some models.) sep_token_id: An id of ```` token that separates two different sentences in the same input. (It might be ```` token for some models.) Return: Tensor mask of 0s and 1s that masks all special tokens in the ``input_ids`` tensor. )eq)r{rrr token_masks r%_get_token_maskrWs7*l+ill<.HH9<rV input_ids_idfzbsv, bs -> bsv)shaperrangeclonelogitsFsoftmax unsqueezetodeviceappendcpurQcateinsumrX)rrrr}rseq_lenprob_distribution_batch_listrmask_idxr{logits_distributionprob_distributionprob_distribution_batchmasked_input_ids_idfs r%_get_batch_distributionrps6K &&q)G13   k>*>*>* J 'N >+&,,. !3O!D !X+#Iu5E/FGNN1!Xq.AII&9K&GRP  !78 !D!N!Nq!Q!T!TUfUmUm!n n $++,=,G,G,J,N,N,PQ *,= >$ii(D!L#ll+;=T=W=WXbXiXi=jlvw )E/,B,E,EjFWFW,XX&**q*14H4L4LQR4L4S4]4]^_4``` " & &1 & - 10E0O0OPQ0R RRr$ dataloaderverbosec |j}g}t||D],}t||}|jt |||||.t j |dS)aCalculate a discrete probability distribution according to the methodology described in `InfoLM`_. Args: model: Initialized model from HuggingFace's `transformers package. dataloader: An instance of `torch.utils.data.DataLoader` used for iterating over examples. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. idf: An indication of whether normalization using inverse document frequencies should be used. special_tokens_map: A dictionary mapping tokenizer special tokens into the corresponding integer values. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. Return: A discrete probability distribution. rrV)rrrrrrQr) rrrr}rrrrrs r%_get_data_distributionrsi>\\F&(":w7n$UF3  !8{TWYk!lmn 99&A ..r$predstarget max_lengthct|ttfs t|}t|ttfs t|}||d|dd}||d|dd}|j|j|j|jfS)a8Update the metric state by a tokenization of ``preds`` and ``target`` sentencens. Args: preds: An iterable of hypothesis corpus. target: An iterable of reference corpus. tokenizer: Initialized tokenizer from HuggingFace's `transformers package. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. Return: Tokenizerd ``preds`` and ``target`` sentences represented with ``input_ids`` and ``attention_mask`` tensors. rTpt)paddingr truncationreturn_tensors)r@r,listr{r|)rrrr preds_input target_inputs r%_infolm_updaters. ec4[ )U  fsDk *fET>TVbVqVq qqr$preds_dataloadertarget_dataloaderinformation_measure_clsct||||||}t||||||} ||jj}| |jj} ||| S)alCalculate selected information measure using the pre-trained language model. Args: model: Initialized model from HuggingFace's `transformers package. preds_dataloader: Loader iterating over tokenizer predicted sentences. target_dataloader: Loader iterating over tokenizer reference sentences. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. idf: An indication of whether normalization using inverse document frequencies should be used. information_measure_cls: Information measure class containing all parameters necessary for calculating information measure values using ``preds_distribution`` and ``target_distribution``. special_tokens_map: A dictionary mapping tokenizer special tokens into the corresponding integer values. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. Return: A corpus-level InfoLM score. )rrsorting_indices) rrrrr}rrrrKrLs r%_infolm_computersyF07GVY[movw0  +s4F,,<,D,D,T,TU-.?.G.G.W.WX "#57J KKr$model_name_or_pathr8r9r:r num_threadsreturn_sentence_level_scorec Rt||\}}t|||}| xs|jj} t |}t |||| \}}}}t |||| | }t |||| | }t|||||||| }| r|j|fS|jS)uo Calculate `InfoLM`_ [1]. InfoML corresponds to distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures: - `KL divergence`_ - `alpha divergence`_ - `beta divergence`_ - `AB divergence`_ - `Rényi divergence`_ - L1 distance - L2 distance - L-infinity distance - `Fisher-Rao distance`_ `InfoLM`_ is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks. If you want to use IDF scaling over the whole dataset, please use the class metric. The implementation of this metric is fully based HuggingFace `transformers`' package. Args: preds: An iterable of hypothesis corpus. target: An iterable of reference corpus. model_name_or_path: A name or a model path used to load `transformers` pretrained model. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] idf: An indication of whether normalization using inverse document frequencies should be used. alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. return_sentence_level_score: An indication whether a sentence-level InfoLM score to be returned. Returns: A corpus-level InfoLM score. (Optionally) A list of sentence-level InfoLM scores if `return_sentence_level_score=True`. Example: >>> from torchmetrics.functional.text.infolm import infolm >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> infolm(preds, target, model_name_or_path='google/bert_uncased_L-2_H-128_A-2', idf=False) tensor(-0.1784) References: [1] InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation by Pierre Colombo, Chloé Clavel and Pablo Piantanida `InfoLM`_ ) rr7configrrrrrmean)rrrrr8r}r9r:rrr~rrrrrrrpreds_input_idspreds_attention_masktarget_input_idstarget_attention_maskrr info_lm_scores r%rr"sn11CVLIu12EudS6u||66J0;Uc vy*VRO)+;=R'8LcS]_jk'(8:OQTV`bmn#   M#!!#]22    r$)T) zbert-base-uncasedg?rTNNNN@rTF)4oscollections.abcrenumrtypingrrrrrQr torch.nnr rtorch.utils.datar typing_extensionsr 4torchmetrics.functional.text.helper_embedding_metricr rrrtorchmetrics.utilities.enumsrtorchmetrics.utilities.importsr transformersrr__doctest_skip__rzr r7boolintrdictr,rrrArno_gradrtuplerrPathLikerrr#r$r%rs $77 $'% 1J4E& z(/ ($0g00$_o_oDOO'-O48OFIOX[OO2'@T#s(^&vS[^ci26S 6S V 6S6S  6S S#X 6S  6Sr%/ %/%/%/  %/ S#X %/  %/ %/%/Pr hsm# $r #x}$ %r)r r  6666 )* rT+L +L +L"+L +L  +L 1 +LS#X+L+L +Lb3F@O! 15 $(-p hsm# $p #x}$ %p c2;;./p  p > p  p  E?p  5/p  U3 ,- .p  p p p p "&p  65( ()p r$