:jC+ddlZddlZddlmZmZmZmZddlmZddl m Z ddl m Z m Z dedefdZGd d ZGd d eZy) N)AnyDictListOptional) EinopsError)_product)ParsedExpression _ellipsisaxesreport_messagecTt|dkDrt|j|y)Nr)lenrformat)r r s j/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/einops/layers/_einmix.py _report_axesr s( 4y1}.//566c eZdZddededeedeffd ZdededeedefdZdeedee d eed ee fd Z d Z d Z xZ S) _EinmixMixinpattern weight_shape bias_shape axes_lengthsc t|||_||_||_||_|j ||||y)a EinMix - Einstein summation with automated tensor management and axis packing/unpacking. EinMix is a combination of einops and MLP, see tutorial: https://github.com/arogozhnikov/einops/blob/main/docs/3-einmix-layer.ipynb Imagine taking einsum with two arguments, one of each input, and one - tensor with weights >>> einsum('time batch channel_in, channel_in channel_out -> time batch channel_out', input, weight) This layer manages weights for you, syntax highlights a special role of weight matrix >>> EinMix('time batch channel_in -> time batch channel_out', weight_shape='channel_in channel_out') But otherwise it is the same einsum under the hood. Plus einops-rearrange. Simple linear layer with a bias term (you have one like that in your framework) >>> EinMix('t b cin -> t b cout', weight_shape='cin cout', bias_shape='cout', cin=10, cout=20) There is no restriction to mix the last axis. Let's mix along height >>> EinMix('h w c-> hout w c', weight_shape='h hout', bias_shape='hout', h=32, hout=32) Example of channel-wise multiplication (like one used in normalizations) >>> EinMix('t b c -> t b c', weight_shape='c', c=128) Multi-head linear layer (each head is own linear layer): >>> EinMix('t b (head cin) -> t b (head cout)', weight_shape='head cin cout', ...) ... and yes, you need to specify all dimensions of weight shape/bias shape in parameters. Use cases: - when channel dimension is not last, use EinMix, not transposition - patch/segment embeddings - when need only within-group connections to reduce number of weights and computations - next-gen MLPs (follow tutorial link above to learn more!) - in general, any time you want to combine linear layer and einops.rearrange Uniform He initialization is applied to weight tensor. This accounts for the number of elements mixed and produced. Parameters :param pattern: transformation pattern, left side - dimensions of input, right side - dimensions of output :param weight_shape: axes of weight. A tensor of this shape is created, stored, and optimized in a layer If bias_shape is not specified, bias is not created. :param bias_shape: axes of bias added to output. Weights of this shape are created and stored. If `None` (the default), no bias is added. :param axes_lengths: dimensions of weight tensor )rrrrN)super__init__rrrrinitialize_einmix)selfrrrr __class__s rrz_EinmixMixin.__init__sLT  ($( ,:\h  rc |jd\}}t|}t|}t|} ttj |j h|j | j d| j r td|j s |j r@|j r |j std||jrtd|td||| fDr tdd|vsd |vrtd |d} d} d} td |jDrrg} |jD]}| |z } | Dcgc]}|tk7r|nd } }d j| }|d|} |jDcic] \}}|| vs ||} }}td|jDs |jrJg} |jD]}| |z } | Dcgc]}|tk7r|nd } }d j| }|d|} |j| | | i| j D]}||vstd|dttj t|h|j | j dttj | j h|j |j dt| j dk(rt!j"dd| jDcgc] \}|| }}t%| jDcgc]\}||j vs||c}}|t'|t(s tdt|}ttj |j |j dttj |j t|dg}d}|jD]m}|tk(r|std|D]O}|tk(r|std||j vr|j+||d}?|j+dQond}d |z d!z}d|z d!z}|j-||||h|j |j | j }t|vr|j/tt1|}t3t4j6|Dcic]\}}|| c}}d t<d"tffd# }||d$|| d|||_ycc}wcc}}wcc}wcc}wcc}wcc}}w)%Nz->z7Unrecognized identifiers on the right side of EinMix {}zKEllipsis is not supported in weight, as its shape should be fully specifiedz,Ellipsis in EinMix should be on both sides, z3Ellipsis on left side can't be in parenthesis, got c34K|]}|jywN)has_non_unitary_anonymous_axes).0xs r z1_EinmixMixin.initialize_einmix..SsOAq//Osz2Anonymous axes (numbers) are not allowed in EinMix()z,Parenthesis is not allowed in weight shape: c38K|]}t|dk7ywNrr#groups rr%z1_EinmixMixin.initialize_einmix..[s=5s5zQ=... z-> c38K|]}t|dk7ywr)r+r,s rr%z1_EinmixMixin.initialize_einmix..ds>5s5zQ>r.z ->z Dimension z of weight should be specifiedzAxes {} are not used in patternzWeight axes {} are redundantrzCEinMix: weight has no dimensions (means multiplication by a number)) stacklevelzAbias shape should be string specifying which axes bias depends onz"Bias axes {} not present in outputz#Sizes not provided for bias axes {}Fz:all bias dimensions should go after ellipsis in the outputTr*g?r cg}|jD]Q}t|tr#|j|Dcgc]}| c}6|tk(sJ|j dSdj |Scc}w)Nr/) composition isinstancelistextendr appendjoin)r result composed_axisaxismapping2letterss rwrite_flat_remappedz;_EinmixMixin.initialize_einmix..write_flat_remappedssF!%!1!1 ) mT2MM]"ST?4#8"ST(I555MM%(  ) 776? " #Ts A5 ,)splitr rset difference identifiers has_ellipsisrhas_ellipsis_parenthesizedanyr7r r<items_create_rearrange_layersrwarningswarnrr8strr;_create_parametersremovesortedzipstringascii_lowercaseeinsum_pattern) rrrrr left_pattern right_patternleftrightweightpre_reshape_patternpre_reshape_lengthspost_reshape_patternnamesr-namer7lengthr? _weight_shape_fan_inbias _bias_shapeused_non_trivial_sizer weight_bound bias_boundmapped_identifiersletterkrAr@s @rrz_EinmixMixin.initialize_einmixCs&-mmD&9# m - /!,/ NN5,,.V0@0@.V6CUCU.V W E    kl l    2 2%%%*<*<!$PQXPY"Z[[..!$WX_W`"abb O$v9NO ORS S , #"5 L\N[\ \""# =D,<,<= =!E))  FKLdTY.TE9LEL((5/K%1N#k]"C DPDVDVDX"jLD&\`di\i4<"j "j >E,=,=> >%BbBbE**  FKLdTY.TE9LEL((5/K&1]#m_#E  %%&9;NPdfhi&& UD<'!Jtf4R"STT U  NN3|,.V0@0@.V6CUCU.V W -   NN6--/V1A1A/VEDUDU/V WYw  v!! "a ' MM_lm n;A;M;MNd+N Nf>P>Pr7DTX`e`q`qTqL.rs  !j#.!"eff#J/D t//1B1BC4  t//\1BC5  K$) !)) 29$,)*fgg!%29,4&12n&o o!T%5%55'..|D/AB481'..q12  2 KG + 'kc)   |[*UZt//Y%2C2CYfFXFXY * *  % %i 0#$6769&:P:PRd6ef1f9f%* " #&6 ##4()+>v+F*GrJ]^cJdIe f qM#k M&OrZgs0U  U*U6UU?U"U" U'r[r\r]post_reshape_lengthsctd)N.Should be defined in framework implementationsNotImplementedErrorrr[r\r]rks rrKz%_EinmixMixin._create_rearrange_layerss""RSSrctd)zShape and implementationsrmrnrrrfrrgs rrOz_EinmixMixin._create_parameterss!"RSSrc"t|j}|d|jdz }|j|d|jdz }|jj D]\}}|d|d|z }|j jd|dS)Nz, ''z, =r&r')reprrrrrrJr__name__)rparamsr?r`s r__repr__z_EinmixMixin.__repr__sdll#C))*!,, ?? & DOO,A. .F --335 +LD& 4&&* *F +..))*!F8155rr!)rw __module__ __qualname__rNrrrdictrrrKrOry __classcell__)rs@rrrs1 1 31 HSM1 jm1 fv v Cv XVY]v jnv pT%c]T&d^T'sm T 'tn TT6rrc FeZdZdZdeedeedeedeefdZdZy) _EinmixDebuggerzUsed only to test mixinr[r\r]rkc<||_||_||_||_yr!)r[r\r]rkrps rrKz(_EinmixDebugger._create_rearrange_layerss$$7 #6 $8!$8!rc ||_||_yr!)saved_weight_shapesaved_bias_shaperrs rrOz"_EinmixDebugger._create_parameterss". *rN) rwrzr{__doc__rrNrrKrOrrrrsE! 9%c] 9&d^ 9'sm 9 'tn 9+rr)rSrLtypingrrrreinopsr einops.einopsreinops.parsingr r rDrNrrrrrrrsF ,,"67s7C7 @6@6F+l+r