:jn}ddlZddlmZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddlmcmZddl mZmZddlmZmZmZddlmZdd lmZmZdd lmZmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%gdZ&de'e(e(fde(de(de(de'e(e(ff dZ)de'e(e(fde(de*e'e(e(ffdZ+de(de(de j$fdZ,GddejZZ.GddejZZ/GddejZZ0Gd d!ejZZ1Gd"d#ejZZ2Gd$d%ejZZ3Gd&d'ejZZ4Gd(d)ejZZ5Gd*d+ejZZ6 d=d,e(d-e*e(d.e*e(d/e7d0e(d1e(d2e ed3e8d4ede6fd5Z9Gd6d7eZ:eed8e:jvf9dd:d;d2e e:d3e8d4ede6fd<ZN) OrderedDict)Sequence)partial)AnyCallableOptional)nnTensor)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface)Conv2dNormActivationSqueezeExcitation)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_once)MaxVitMaxVit_T_Weightsmaxvit_t input_size kernel_sizestridepaddingreturncR|d|z d|zz|zdz|d|z d|zz|zdzfS)Nr)rrrrs n/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/torchvision/models/maxvit.py_get_conv_output_shaper$sJ A $q7{ 2v=A A $q7{ 2v=A n_blockscg}t|ddd}t|D]!}t|ddd}|j|#|S)zQUtil function to check that the input size is correct for a MaxVit configuration.r r!)r$rangeappend)rr&shapesblock_input_shape_s r#_make_block_input_shapesr.!sQ F.z1aC 8_)23DaAN '() Mr%heightwidthctjtjtj|tj|gd}tj|d}|dddddf|dddddfz }|j dddj }|dddddfxx|dz z cc<|dddddfxx|dz z cc<|dddddfxxd|zdz zcc<|jdS)Nij)indexingr!r r)torchstackmeshgridarangeflattenpermute contiguoussum)r/r0coords coords_flatrelative_coordss r#_get_relative_position_indexr@+s [[f)=u||E?R(S^bc dF--*K!!Q*- AtQJ0GGO%--aA6AACOAq!G *Aq!G )Aq!GE A -   r ""r%ceZdZdZ ddededededededejfd edejfd ed d ffd Z de d e fdZ xZ S)MBConva=MBConv: Mobile Inverted Residual Bottleneck. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. p_stochastic_dropout (float): Probability of stochastic depth. in_channels out_channelsexpansion_ratio squeeze_ratioractivation_layer. norm_layerp_stochastic_dropoutrNc t|||dk7xs||k7} | rTtj||dddg} |dk(rtjd|dg| z} tj | |_ntj|_t||z} t||z} |rt|d|_ ntj|_ t} ||| d <t|| ddd ||d | d <t| | d|d||| d | d<t| | tj| d<tj| |dd| d<tj | |_y)Nr!T)rrbiasr r(rrrrowmodepre_normr)rrrrGrHinplaceconv_a)rrrrGrHgroupsrQconv_b) activationsqueeze_excitation)rCrDrrKconv_c)super__init__r Conv2d AvgPool2d SequentialprojIdentityintrstochastic_depthrrrSiLUlayers)selfrCrDrErFrrGrHrI should_projr] mid_channels sqz_channels_layers __class__s r#rYzMBConv.__init__Dst  k@[L%@ IIk2;-S"))^,;-$;- ;-z  F r%rBcdeZdZdZdedededdffd ZdejfdZd edefd Z xZ S) $RelativePositionalMultiHeadAttentionzRelative Positional Multi-Head Attention. Args: feat_dim (int): Number of input features. head_dim (int): Number of features per head. max_seq_len (int): Maximum sequence length. feat_dimhead_dim max_seq_lenrNcbt|||zdk7rtd|d|||z|_||_t t j||_||_ tj||j|jzdz|_ |dz|_ tj|j|jz||_tjj!t#j$d|jzdz d|jzdz z|jft"j&|_|j+d t-|j|jt"jj.j1|j(d y) Nrz feat_dim: z must be divisible by head_dim: r(gr r!)dtyperelative_position_index{Gz?std)rXrY ValueErrorn_headsrzr_mathsqrtsizer{r Linearto_qkv scale_factormerge parameter Parameterr5emptyfloat32relative_position_bias_tableregister_bufferr@init trunc_normal_)rcryrzr{rhs r#rYz-RelativePositionalMultiHeadAttention.__init__sU  h ! #z(3ST\S]^_ _8+    +./ &ii$,,*F*JK $dNYYt}}t||;XF ,.LL,B,B KK!dii-!+DII 0ABDLLQY^YfYf g- ) 68TUYU^U^`d`i`i8jk  ##D$E$E4#Pr%c|jjd}|j|j|j|jd}|j dddj }|j dS)Nr4r rr!)r~viewrr{r:r; unsqueeze)rc bias_index relative_biass r#get_relative_positional_biaszARelativePositionalMultiHeadAttention.get_relative_positional_biasss1166r: 99*EJJ4K[K[]a]m]moqr %--aA6AAC &&q))r%ric|j\}}}}|j|j}}|j|}t j |dd\} } } | j |||||jddddd} | j |||||jddddd} | j |||||jddddd} | |jz} t jd| | } |j} tj| | zd} t jd | | }|jdddddj ||||}|j|}|S) z Args: x (Tensor): Input tensor with expected layout of [B, G, P, D]. Returns: Tensor: Output tensor with expected layout of [B, G, P, D]. r(r4)dimrr!r z!B G H I D, B G H J D -> B G H I Jz!B G H I J, B G H J D -> B G H I D)shaperrzrr5chunkreshaper:reinsumrFsoftmaxr)rcriBGPDHDHqkvqkvdot_prodpos_biasouts r#rmz,RelativePositionalMultiHeadAttention.forwardsXWW 1a dmm2kk!n++c1"-1a IIaAq" % - -aAq! < IIaAq" % - -aAq! < IIaAq" % - -aAq! < !! !<< CQJ44699X0b9ll>!Lkk!Q1a(00Aq!<jjo r%) rorprqrrr_rYr5r rrmrurvs@r#rxrxs[QQQ Q  Q8*ell* Fr%rxcheZdZdZdededdffd ZdejdejfdZxZ S) SwapAxeszPermute the axes of a tensor.abrNc>t|||_||_yN)rXrYrr)rcrrrhs r#rYzSwapAxes.__init__s r%ric\tj||j|j}|Sr)r5swapaxesrrrks r#rmzSwapAxes.forwards!nnQ/ r%) rorprqrrr_rYr5r rmrurvs@r#rrs;'##$ %,,r%rc8eZdZdZdfd ZdededefdZxZS)WindowPartitionzB Partition the input tensor into non-overlapping windows. rc"t|yrrXrYrcrhs r#rYzWindowPartition.__init__ r%ripc|j\}}}}|}|j||||z|||z|}|jdddddd}|j|||z||zz||z|}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. p (int): Number of partitions. Returns: Tensor: Output tensor with expected layout of [B, H/P, W/P, P*P, C]. rr rr(r!rrr:)rcrirrCrWrs r#rmzWindowPartition.forwards|WW 1a  IIaAFAqAvq 1 IIaAq!Q ' IIa!q&Q!V,a!eQ 7r%rN rorprqrrrYr r_rmrurvs@r#rrs'CFr%rc @eZdZdZd fd Zdededededef dZxZS) WindowDepartitionzo Departition the input tensor of non-overlapping windows into a feature volume of layout [B, C, H, W]. rc"t|yrrrs r#rYzWindowDepartition.__init__rr%rir h_partitions w_partitionsc|j\}}}}|} ||} } |j|| | | | |}|jdddddd}|j||| | z| | z}|S)ar Args: x (Tensor): Input tensor with expected layout of [B, (H/P * W/P), P*P, C]. p (int): Number of partitions. h_partitions (int): Number of vertical partitions. w_partitions (int): Number of horizontal partitions. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrr!r(r rr) rcrirrrrrPPrrHPWPs r#rmzWindowDepartition.forwardstgg 1b! |B IIaRAq ) IIaAq!Q ' IIaBFBF +r%rrrvs@r#rrs6Cs#RXr%rceZdZdZdededededeeefdeded ejfd ed ejfd e d e d e ddffd Z de de fdZ xZS)PartitionAttentionLayera Layer for partitioning the input tensor into non-overlapping windows and applying attention to each window. Args: in_channels (int): Number of input channels. head_dim (int): Dimension of each attention head. partition_size (int): Size of the partitions. partition_type (str): Type of partitioning to use. Can be either "grid" or "window". grid_size (Tuple[int, int]): Size of the grid to partition the input tensor into. mlp_ratio (int): Ratio of the feature size expansion in the MLP layer. activation_layer (Callable[..., nn.Module]): Activation function to use. norm_layer (Callable[..., nn.Module]): Normalization function to use. attention_dropout (float): Dropout probability for the attention layer. mlp_dropout (float): Dropout probability for the MLP layer. p_stochastic_dropout (float): Probability of dropping out a partition. rCrzpartition_sizepartition_type grid_size mlp_ratiorG.rHattention_dropout mlp_dropoutrIrNc t |||z|_||_|d|z|_||_||_|dvr td|dk(r||jc|_|_ n|j|c|_|_ t|_ t|_ |dk(r tddntj |_|dk(r tddntj |_tj&||t)|||dztj*| |_tj&tj.|tj0|||z|tj0||z|tj*| |_t5| d |_y) Nr)gridwindowz0partition_type must be either 'grid' or 'window'rrr rMrN)rXrYrrz n_partitionsrrrrgr partition_oprdepartition_oprr r^partition_swapdepartition_swapr\rxDropout attn_layer LayerNormr mlp_layerrstochastic_dropout) rcrCrzrrrrrGrHrrrIrhs r#rYz PartitionAttentionLayer.__init__-s" "h.   %aLN:," !3 3OP P X %+T->->NDFDF!..NDFDF+-/12@F2Jhr2.PRP[P[P]4Bf4LR 0RTR]R]R_-- { # 1hXYHY Z JJ( )   LL % IIk;#: ;   IIkI-{ ; JJ{ #  #22FU"Sr%ric|jd|jz|jd|jz}}tj|jd|jzdk(xr|jd|jzdk(dj |j|j|j ||j}|j |}||j|j|z}||j|j|z}|j|}|j||j||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rr!z[Grid size must be divisible by partition size. Got grid size of {} and partition size of {}) rrr5_assertformatrrrrrrr)rcrighgws r#rmzPartitionAttentionLayer.forwardgs!"dff,dnnQ.?466.IB NN1  &! + Oq0ADFF0Ja0O i p p     a (    " ''(:; ; ''q(9: :  ! !! $   4662r 2r%)rorprqrrr_strtuplerr rtrsrYr rmrurvs@r#rrs"8T8T8T  8T  8Tc?8T8T#3 >28TS"))^,8T!8T8T$8T !8TtFr%rceZdZdZdededededededejfd edejfd ed ed ed ededede eefddffd Z de de fdZ xZ S) MaxVitLayera MaxVit layer consisting of a MBConv layer followed by a PartitionAttentionLayer with `window` and a PartitionAttentionLayer with `grid`. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. grid_size (Tuple[int, int]): Size of the input feature grid. rCrDrFrErrH.rGrzrrrrIrrrNc 2t|t}t|||||||| |d<t ||| d|| |t j | | |  |d<t ||| d|| |t j | | |  |d<t j||_y)N)rCrDrErFrrGrHrIMBconvr) rCrzrrrrrGrHrrrIwindow_attentionrgrid_attention) rXrYrrBrr rr\rb)rcrCrDrFrErrHrGrzrrrrIrrrbrhs r#rYzMaxVitLayer.__init__s* )m"#%+'-!!5  x&=$)#-||/#!5 & !"$;$)!-||/#!5 $  mmF+ r%ric(|j|}|Sz Args: x (Tensor): Input tensor of shape (B, C, H, W). Returns: Tensor: Output tensor of shape (B, C, H, W). rb)rcris r#rmzMaxVitLayer.forwards KKNr%)rorprqrrr_rsrr rtrrYr rmrurvs@r#rrs(?,?, ?,  ?,  ?,?,S"))^,?,#3 >2?,?,?,?,!?, $!?,$%?,&c?'?,( )?,BFr%rceZdZdZdedededededejfdedejfd ed ed ed ed ede eefdede eddffd Z de de fdZ xZS) MaxVitBlocka( A MaxVit block consisting of `n_layers` MaxVit layers. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. input_grid_size (Tuple[int, int]): Size of the input feature grid. n_layers (int): Number of layers in the block. p_stochastic (List[float]): List of probabilities for stochastic depth for each layer. rCrDrFrErH.rGrzrrrrinput_grid_sizen_layers p_stochasticrNcpt|t|| k(std| d|dt j |_t| ddd|_t|D]L\}}|dk(rdnd}|xj t|dk(r|n|||||||||| | | |j| gz c_Ny) Nz'p_stochastic must have length n_layers=z, got p_stochastic=.r(r r!rLr)rCrDrFrErrHrGrzrrrrrrI) rXrYlenrr ModuleListrbr$r enumerater)rcrCrDrFrErHrGrzrrrrrrridxrrrhs r#rYzMaxVitBlock.__init__s, < H,FxjPcdpcqqrst tmmo /QWXbcd - FC(QF KK/2ax \!-"/$3!)%5%' +&7#1"nn)* K r%ric8|jD] }||} |Srr)rcrilayers r#rmzMaxVitBlock.forward-s%[[ EaA r%)rorprqrrr_rsrr rtrlistrYr rmrurvs@r#rrs*11 1  1  1S"))^,1#3 >21111!1 !1"sCx#1&'1(5k)1* +1f  F r%rc!eZdZdZdej ddddddfdeeefded ed eed eed ed e de e dejfde dejfde de dede de deddf fd Z dedefdZdZxZS)ray Implements MaxVit Transformer from the `MaxViT: Multi-Axis Vision Transformer `_ paper. Args: input_size (Tuple[int, int]): Size of the input image. stem_channels (int): Number of channels in the stem. partition_size (int): Size of the partitions. block_channels (List[int]): Number of channels in each block. block_layers (List[int]): Number of layers in each block. stochastic_depth_prob (float): Probability of stochastic depth. Expands to a list of probabilities for each layer that scales linearly to the specified value. squeeze_ratio (float): Squeeze ratio in the SE Layer. Default: 0.25. expansion_ratio (float): Expansion ratio in the MBConv bottleneck. Default: 4. norm_layer (Callable[..., nn.Module]): Normalization function. Default: None (setting to None will produce a `BatchNorm2d(eps=1e-3, momentum=0.01)`). activation_layer (Callable[..., nn.Module]): Activation function Default: nn.GELU. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Expansion ratio of the MLP layer. Default: 4. mlp_dropout (float): Dropout probability for the MLP layer. Default: 0.0. attention_dropout (float): Dropout probability for the attention layer. Default: 0.0. num_classes (int): Number of classes. Default: 1000. Ng?rrnir stem_channelsrblock_channels block_layersrzstochastic_depth_probrH.rGrFrErrr num_classesrct|t|d}|ttj dd}t |t|}t|D]3\}}|d|zdk7s |d|zdk7std|d|d |d |d t jt||dd || d dt||ddddd|_ t|dd d}||_t j|_|g|ddz}|}t#j$d|t'|j)}d}t+|||D]\\}}}|j j-t/||| | || || | ||||||||z|j dj0}||z }^t jt j2dt j4t j6|dt j8|d|dt j:t j8|d|d |_|j?y)Nr(gMbP?g{Gz?)epsmomentumrr!z Input size z of block z$ is not divisible by partition size zx. Consider changing the partition size or the input size. Current configuration yields the following block input sizes: rr F)rrHrGrKrQT)rrHrGrKrLr4)rCrDrFrErHrGrzrrrrrrr)rK) rXrYrrr BatchNorm2dr.rrrr\rstemr$rrblocksnplinspacer<tolistzipr*rrAdaptiveAvgPool2dFlattenrrTanh classifier _init_weights)rcrrrrrrzrrHrGrFrErrrr input_channelsblock_input_sizesrblock_input_sizerCrDrp_idx in_channel out_channel num_layersrhs r#rYzMaxVit.__init__Ns: D!   TDIJ 5Z^ATU%./@%A  !C!"^3q8 IIaL[[ EaA  OOA r%c|jD]l}t|tjrbtjj |j d|jVtjj|jt|tjrUtjj|j dtjj|jdt|tjs tjj |j d|jDtjj|joy)Nrrr!r) modules isinstancer rZrnormal_weightrKzeros_r  constant_r)rcms r#rzMaxVit._init_weightss +A!RYY'd366%GGNN166*Ar~~.!!!((A.!!!&&!,Aryy)d366%GGNN166* +r%)rorprqrrr GELUrr_rrsrrrtrYr rmrrurvs@r#rr9s%J:>57WW#!" #&7t#s(Ot  ttS t3itt %t"Xc299n56#t$#3 >2%t()t*+t./t01t2!3t67t8 9tlF +r%rrrrrrrzweightsprogresskwargsc d|dt|dt|jd|jdd|jddk(sJt|d|jd|jdd} t d ||||||| d|} |"| j |j |d | S) Nr  categoriesmin_sizerr!rr2)rrrrrzrrT)r, check_hashr")rrmetapoprload_state_dictget_state_dict) rrrrrrzr+r,r-rmodels r#_maxvitr9s$fmSl9S5TU||J'*gll:.Fq.IIIIflGLL4LML*5J  #%!3%   E g44hSW4XY Lr%c jeZdZedeeddejedddddd d id d d dZ e Z y)rz9https://download.pytorch.org/models/maxvit_t-bc5ab103.pthr2) crop_size resize_size interpolationir1zLhttps://github.com/pytorch/vision/tree/main/references/classification#maxvitz ImageNet-1KgT@g|?5.X@)zacc@1zacc@5gZd;@gK7]@zThese weights reproduce closely the results of the paper using a similar training recipe. They were trained with a BatchNorm2D momentum of 0.99 instead of the more correct 0.01.)r/ num_paramsr0recipe_metrics_ops _file_size_docs)url transformsr4N) rorprqr rrrBICUBICr IMAGENET1K_V1DEFAULTr"r%r#rrsb G 3CO`OhOh /""d##  !g M.Gr%r pretrained)r+T)r+r,c \tj|}tddgdgdddd||d|S) a Constructs a maxvit_t architecture from `MaxViT: Multi-Axis Vision Transformer `_. Args: weights (:class:`~torchvision.models.MaxVit_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.MaxVit_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.maxvit.MaxVit`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.MaxVit_T_Weights :members: @)rKi)r r rr g?)rrrrzrrr+r,r")rverifyr9)r+r,r-s r#rrsH.%%g.G  *!!    r%)NF)=r collectionsrcollections.abcr functoolsrtypingrrrnumpyrr5torch.nn.functionalr functionalrr torchvision.models._apir r r torchvision.models._metartorchvision.models._utilsrrtorchvision.ops.miscrr torchvision.ops.stochastic_depthrtorchvision.transforms._presetsrrtorchvision.utilsr__all__rr_r$rr.r@rtrBrxrrrrrrrrsboolr9rrGrr"r%r#ras[ #$** HH9TH<R1 uS#XSRU`chmnqsvnvhwsCxCDQVWZ\_W_Q`La##S#U\\#TRYYTnF299FR ryy bii4