:j(ddlmZddlZddlmZddlmZddlmZddl Z ddl Z ddl m Z ddl m Zdd lmZmZdd lmZdd lmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-Gd deZ.eGddZ/GddZ0eGddZ1eGddZ2eGddZ3eGddZ4eGddZ5GddeZ6y)) annotationsN) dataclass)Enum)Any) TensorProto)onnx_pb) BaseQuantizerQuantizationParams) TensorData)DEQUANT_OP_NAMEONNX_TYPE_TO_NP_TYPE QUANT_OP_NAMEQuantizedValueQuantizedValueType __producer__ __version__add_dequant_output_suffixadd_dequant_suffixadd_quant_input_suffixadd_quant_output_suffixadd_quant_suffixcompute_data_quant_paramscompute_scale_zpcompute_scale_zp_blockedcompute_scale_zp_float8 find_by_nameget_qmin_qmax_for_qType ms_domainnormalize_axisquantize_onnx_initializersnap_zero_point_to_uint8tensor_proto_to_array)CreateQDQQuantizerceZdZdZdZdZy)QDQQuantTensorTyperr N)__name__ __module__ __qualname__ ACTIVATIONWEIGHTBIAS{/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/onnxruntime/quantization/qdq_quantizer.pyr&r&0sJ F Dr/r&c"eZdZUded<ded<y)QDQQuantParamProviderstr input_name node_nameNr(r)r*__annotations__r.r/r0r2r29s ONr/r2c0eZdZejdddfdZy)QDQTensorQuantInfoNcV||_||_||_|du|_|J||_yN) tensor_typequant_para_provideraxis is_shared data_type)selfr<r=r>r@s r0__init__zQDQTensorQuantInfo.__init__Bs8&#6  ,D8$$$"r/)r(r)r*r&r+rBr.r/r0r9r9As#5#@#@VZaequ#r/r9c6eZdZUded<ded<ded<ded<y)QDQBiasQuantInfor3r5r4 weight_namefloatbetaNr6r.r/r0rDrDLsNO Kr/rDc4eZdZUded<ded<ded<d dZy) QDQTensorQuantParamsr originalzQuantizationParams | None convertedset[str] | Noneconverted_recv_nodesc|j |jS|j |jS||jvr |jS|jSr;rKrJrMrAconsumer_node_names r0get_for_consumerz%QDQTensorQuantParams.get_for_consumer]Q >> !==  $ $ ,>> ! #58Q8Q"Qt~~eX\XeXeer/N)returnr r(r)r*r7rRr.r/r0rIrIWs  (()) fr/rIc"eZdZUded<ded<y)QDQScaleZpInitializersrscale zero_pointNr6r.r/r0rWrWks r/rWc,eZdZUded<ded<ded<y)QDQTensorScaleZpInitializersrWrJzQDQScaleZpInitializers | NonerKrLrMNr6r.r/r0r[r[ts$$,,))r/r[c4eZdZUded<ded<ded<d dZy) QDQTensorQuantizedValuerrJzQuantizedValue | NonerKrLrMc|j |jS|j |jS||jvr |jS|jSr;rOrPs r0rRz(QDQTensorQuantizedValue.get_for_consumerrSr/N)rTrrUr.r/r0r]r]~s$$)) fr/r]ceZdZ d$dZdZdZdejfdZd%dZ d&dZ d%dZ d Z d'd Z d(d Z d)d Zd ZdZdZdZdZ d* d+dZ d* d,dZ d* d-dZd.dZd$dZdZdZdZdZd%dZ d$ d/dZd0dZ d1dZ! d2 d3dZ"d4d Z#d5d!Z$d6d"Z%d7d#Z&y)8 QDQQuantizerNc  tj|||||||||| | i|_i|_g|_| j dg|_| j dd|_| j dd|_| j dd|_ i|_ i|_ | j di|_ | j ddrtnd|_| j d d|_| j d d|_| j d d |_|j$d krt&j(t&j*t&j,t&j.f t1 fd|j2D} |jsF|j4 vs|j6 vs| r(t9j:dtdt|_|j=|_i|_ i|_!y)N"OpTypesToExcludeOutputQuantizationAddQDQPairToWeightF QuantizeBiasTDedicatedQDQPair QDQOpTypePerChannelSupportToAxisUseQDQContribOpsQDQKeepRemovableActivations"QDQDisableWeightAdjustForInt32Bias BlockSizerc3:K|]}|jvywr;)r<).0t opset21_typess r0 z(QDQQuantizer.__init__..s/34 ./szONNX QuantizeLinear and DequantizeLinear operators do not support 16-bit/4-bit integer quantization types prior to opset 21. The domain of QuantizeLinear and DequantizeLinear operators will be set to 'z' to enable support.)"r rBtensors_to_quantizebias_to_quantizenodes_to_removeget'op_types_to_exclude_output_quantizationadd_qdq_pair_to_weight quantize_biasdedicated_qdq_pairtensor_to_its_receiving_nodestensor_to_producing_dq'qdq_op_type_per_channel_support_to_axisr qdq_op_domainqdq_keep_removable_activations(qdq_disable_weight_adjust_for_int32_bias block_size opset_versionrUINT16INT16UINT4INT4anytensor_quant_override_qtypesactivation_qType weight_qTypeloggingwarningcalc_graph_quant_paramsquantization_paramsinitializer_quant_paramsquantized_value_map) rAmodel per_channel reduce_rangerr tensors_rangenodes_to_quantizenodes_to_excludeop_types_to_quantize extra_optionsoverrides_have_opset21_typesros @r0rBzQDQQuantizer.__init__s            CE =?!8E7H7HImoq7r4 '4&7&78Le&T#+..~tD#0"3"34F"NNP*BD#8E7H7HIkmo7p4*7*;*; 8J +AWcl9((5D$<$<< 11+> 8JWbnw8x((5= 4r/cD|j|dtjS)z Adds a tensor to the list of tensors to quantize. Called by op quantizers that want to quantize a tensor (i.e., "mark" a tensor for quantization). Args: tensor_name: name of the tensor to quantize N)_QDQQuantizer__quantize_tensorr&r+rArs r0quantize_activation_tensorz'QDQQuantizer.quantize_activation_tensor7s %%k49K9V9VWWr/cX|j|t||tjS)a Adds a tensor to the list of tensors to quantize. Called by op quantizers that want to quantize an output tensor using the same quantization parameters as one of the node's inputs. Ex: A Tranpose node's output will typically use the same quantization parameter initializers used at the Transpose node's input. Args: output_name: name of the node output to quantize so that it uses the same quantization params as an input. input_name: name of the node input from which the output tensor will get its quantization params. node_name: name of the node that consumes `input_name`. )rr2r&r+)rA output_namer4r5s r0quantize_output_same_as_inputz*QDQQuantizer.quantize_output_same_as_inputAs-%% .z9EGYGdGd  r/cD|j|dtjS)z Adds a tensor to the list of weight tensors to quantize. Called by op quantizers that want to quantize a weight (i.e., "mark" a weight for quantization). Args: tensor_name: name of the weight to quantize N)rr&r,rs r0quantize_weight_tensorz#QDQQuantizer.quantize_weight_tensorRs %%k49K9R9RSSr/clt||jj}|ru|jtj j tj jfvr4ttj||j|j|<yytjd|dy)N)r<r>r@z9only support per-channel quantization on weight. Tensor: z is not quantized.)rrrr@rrrrr9r&r,rqrr)rArr>rs r0"quantize_weight_tensor_per_channelz/QDQQuantizer.quantize_weight_tensor_per_channel\sk4::+A+A+CD J$:$:$@$@*BXBXB`B`#aa8J 2 9 9PVP`P`9((5b OOWXcWddvw xr/c|jj|jdz}|j|}tj}|j |||_|jj ||S)zk Duplicates an existing initializer and adds it to the model. Returns the new initializer. r )r#get_largest_initializer_name_suffixnameonnxrCopyFromadd_initializer)rAr name_suffixnew_initializer_namenew_initializers r0_dup_initializerzQDQQuantizer._dup_initializerfsv ::II+JZJZ[^__ "-"2"2!3K=A**,  -3 ""?3r/c|jj|rVtjd|d|j |d\}}|r|j ||y|j |yt||jj}|tjd|dy|jtjjtjjfvrtjd|dy|} ||j vrW|j#|} | j$} |jj'|| |htjd |d | d t)|||||j | <y) a Adds a bias tensor to the list of bias tensors to quantize. Called by op quantizers that want to quantize a bias with bias_zero_point = 0 and bias_scale = input_scale * weight_scale * beta. TODO: Explain the reasoning for using this formula. Args: node_name: name of the node that consumes the bias, input, and weight tensors. bias_name: name of the bias tensor to quantize. input_name: name of the input tensor whose scale is used to compute the bias's scale. weight_name: name of the weight tensor whose scale is used to compute the bias's scale. beta: Multiplier used to compute the bias's scale. zQuantizing bias tensor 'z=' as a weight due to the presence of user-specified overridesr) default_axisNzExpected bias 'z' to be an initializerz%' to be an floating-point initializerzCreated a copy of bias input 'z ' called '')tensor_quant_overridesrtrinfois_tensor_per_channelrrrrrrr@rrrrrrrrreplace_input_of_nodesrD) rAr5 bias_namer4rErGis_per_channelr>bias_initializeractual_bias_namenew_bias_initializers r0quantize_bias_tensorz!QDQQuantizer.quantize_bias_tensorrs  & & * *9 5 LL*9+5rs $(#=#=iVW#=#X ND77 4H ++I6 ' 4::3I3I3KL  # OOoi[8NO P   % %j.D.D.J.JJLbLbLjLj-k k LL?9+5Z[ \ $ -- -$(#8#89I#J 388  JJ - -i9II; W LL9)JO_N``ab c3C9jZegk2l./r/c |jsyt|}tjtj}d}tj |j tjtj |jdztjz } |j} d} |stj|jtj dtj} tj|j tj dtj} tjtj| tj| }|d|zz| z }tj |jtj}tj |jtj}||z}||krK|dkDrF||z }tjd |d |d |j d ||z}|j#| }d } | |fS|j$r!t'|j$dk(r|j$d}t)|D]}tj||}|d|zz| z }tj |jtj}tj ||jtj}||z}||ks|dkDs||z }tjd|d|d|d |j d ||z}|j#| ||<d } | |fS)aI Checks if the bias scale (input_scale * weight_scale) that we intend to use is too small. A bias scale that is too small leads to quantized bias values that fall outside the range of a int32 and have to be clipped, which decreases accuracy. If this function detects such a scenario, the weight_scale value will be increased to prevent this from happening. Although the adjustment method and amount differs, the idea to adjust the weight's scale came from the following reference: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/optimize/quantization_utils.cc#L252 :param input_scale: The input's scale. :param weight_scale: The weight scale to potentially adjust. :param weight_name: The weight initializer's name. Used for logging. :param bias_tp: The bias ONNX initializer. :param is_per_channel: True if the bias and weight are quantized per-channel. :return: A tuple with a bool indicating if the weight's scale was adjusted and the new weight scale. FNgqh?dtyper Frg@gzIncreasing scale for weight `z` by the ratio z to ensure bias input `z` has a valid scale.TzIncreased scale[z] for weight `z ` by ratio )sizer#npiinfoint32arraymaxfloat64minrminimummaximumabsitemrrrastypeshapelenrange)rA input_scale weight_scalerEbias_tprbias_float_data int32_infomultiplicative_epsilonqrangeweight_scale_dtypeupdated_an_elemrminrmaxabsmaxbias_smallest_valid_scaleinput_scale_fp64weight_scale_fp64bias_candidate_scaleratio new_scale num_elemsi bias_rmaxs r0#_adjust_weight_scale_for_int32_biasz0QDQQuantizer._adjust_weight_scale_for_int32_biass2  /8XXbhh' !'*.. ;bhhz~~XYGYacakak>ll)//::o113RXXarzz5RSD::o113RXXarzz5RSDZZt bffTl;F(>#,(ORX(X %!xx (8(8(:"**M  "):):)bjj#Q $&HH\!_-A-A-C2::$V!'7:K'K$(+DDK_beKe58LLELL*1#^K= TYSZ[118>RT!2E 9I&/&6&67I&JLO&*O! +$ ,,r/c |jry|jjD]\}}|j|jvs0|j|j vs|j |jvrP|j|jj|j}|j |j}tj|dtjj|j}|j|j }|d}|tj j"tj j$fvr2|d}|j'rI|d} |j)dddu} |j+|| |j t-||j.j1| \} } | s| |d<y)a3 Iterates through all bias inputs that should be quantized to int32. If the intended bias scale (equal to input_scale * weight_scale) is too small, this function will increase the associated weight's scale to ensure the bias does not overflow the int32 range when quantized. NrXr quant_typerYr>)r~rritemsr4rrqrErrRr5rasarrayrhelpertensor_dtype_to_np_dtyper@rINT8rrrtrrrr) rAr bias_info input_qparams input_inforweight_quant_paramsweight_quant_typeweight_zero_pointrrdid_update_weight_scalenew_weight_scales r0,_adjust_weight_quant_params_for_bias_tensorsz9QDQQuantizer._adjust_weight_quant_params_for_bias_tensorss  8 8 $($9$9$?$?$A& @ Iy$$D,D,DD''t/G/GG((0M0MM!44Y5I5IJ[[\e\o\opM11)2F2FGJ**g&dkk.R.RS]SgSg.hK#'"?"? @U@U"V  3L A  )9)9)>)>@P@P@V@V(WW,? ,M  $$&':7'CL044VTB$NN9=8`8`%%Y (>(>(@A 9 5 #%5'/?#G,M& @r/c:|jj|yr;)rsappend)rAnodes r0 remove_nodezQDQQuantizer.remove_node&s ##D)r/cN|jj|jyr;)r remove_nodesrs)rAs r0rzQDQQuantizer.remove_nodes)s  4 45r/c|jjD]}|j|rht||}|j |j D]=}||j vrg|j |<|j |j|?|jtk(s|jD]}||j|<|j|_ |j|j|j!|j"r|j%|j'|j(s|jj+t,|jj_t0|jj_|j4t6k(r |jj9t6d|jjS)Nr )rnodesshould_quantize_noder$quantizeinputryr op_typer outputrz_calc_initializer_quant_paramsrr _quantize_normal_tensors_quantize_sharing_param_tensorsrw_quantize_bias_tensorsrrvclean_initializersr producer_namerproducer_versionr|rset_opset_import)rAr  op_quantizerrs r0quantize_modelzQDQQuantizer.quantize_model,sJJ$$& DD((.1$= %%'#'::QK"$*L*LLJL::;G66{CJJ4PQ||.#';;DK?CD// <D D)-(K(K(M% 99; %%' ,,.     ' ' ) ** JJ ) ) +)5 &,7 )    * JJ ' ' 1 5zzr/c||jvr|j|j|j|jt|jj |dk(rn|jj |sS|jj |s8|jj||||jvr |j|=yy)Nr TF) rrKrrinput_name_to_nodesis_graph_outputis_graph_inputreplace_output_of_all_nodesrq)rAupstream_output_namers r0try_replacing_upstream_outputz*QDQQuantizer.try_replacing_upstream_outputKs 433 3((5??G(()=>HHPDJJ2245IJKqPJJ../CDJJ--.BC JJ 2 23G U#t'?'??,,-ABr/c||jd}|r||d<tjjt|||g|g|fi|} |j j | gy)zI Creates a QuantizeLinear node and adds it to the model. r>domainrN)r|rr make_noderr add_nodes) rAq_inputq_outputquant_node_name scale_namezp_namer>rkwargs qlinear_nodes r0_create_q_nodezQDQQuantizer._create_q_nodeZsj+/$:L:L!M #-F< {{,,  j' * J      l^,r/c||jd}|r||d<tjjt|||g|g|fi|} |j j | gy)zK Creates a DequantizeLinear node and adds it to the model. r*rN)r|rrr,r rr-) rAdq_input dq_outputdequant_node_namer1r2r>rr3 dequant_nodes r0_create_dq_nodezQDQQuantizer._create_dq_nodessj+/$:L:L!M #-F< {{,,  z7 + K      l^,r/c | |jd} | r| | d<tjjt|||g|g|fi| } tjjt |||g|g|fi| } |j j| | gy)Nr*r)r|rrr,rr rr-)rAr.r/r0r7r8r9r1r2r>rr3r4r:s r0_create_qdq_nodeszQDQQuantizer._create_qdq_nodess+/$:L:L!M #-F< {{,,  j' * J     {{,,  z7 + K      lL9:r/c |j}||jvry|j|}|jd}|jdd}|j ||}d}t |}|j j|||jrat|} |j|| t|| |t||jj|jj|| nt||d|d|d||} |j j!| | j}|j#| j|t||jj|jj|| t%|||jj|jjt&j(| } t+| dd|j|<y) a Adds Q/DQ nodes for an initializer. If `self.add_qdq_pair_to_weight` is true, creates the sequence (weight_f32 -> Q -> DQ -> ). Otherwise, this function quantizes the initializer and adds the sequence (weight_quant -> DQ ->). Nr>rr)rrrYrX)r>r)r>)rrrrt_make_scale_zp_initializersrrreplace_input_of_all_nodesrvrr=rrrXrYr!rr;rr Initializerr]) rA weight_protorE quant_paramsr>rscale_zp_initializers q_weight_nameweight_dequant_outputweight_quant_output quant_weightquantized_values r0_add_qdq_nodes_for_initializerz+QDQQuantizer._add_qdq_nodes_for_initializers #'' $22 2 +/+H+H+U  $$V,&**<; $ @ @l [$( 9+ F --k;PQ  & &#:+"F   " "# -#%";/%++00%0055% # 5\*\*W%% L JJ & &| 4(--M  !!%";/%++00%0055% ! )   ! ' ' , , ! , , 1 1  * *  1HY]_c0d  -r/c |jr||jvr t|j|dkDrt|j|}t|D]}d|dz}t ||z}t ||z} t ||z} t||z} |j||| || | |||j||} |jj| || |dk(st|| ||tj|} t| dd|j|<y|}t |}|jj!|r*t#|}|}|jj%||n|jj'|||j|t |t |t ||t|||t||||tj|} t| dd|j|<y)Nr _r scale_type)rxryrrrrrrr=rreplace_node_inputrrInputr]rr$rr&r@)rArr1r2r@num_dedicated_qdq_pairrpostfix tensor_name_quant_output_postfix"tensor_name_dequant_output_postfixquant_node_name_postfixdequant_node_name_postfixr rIr.r8s r0_add_qdq_pair_for_activationz)QDQQuantizer._add_qdq_pair_for_activations!  # #tAAAD66{CDqH%()K)KK)X%Y "12 qa!eW+3J;3WZa3a05N{5[^e5e2*:;*G'*Q',>{,Kg,U)&&4+46- 99+FqI --dKAcd6&4#:"*00#, 'O=TTceikoIzz))+60='  66{GL 55k9M  " "' 4 -' 4";/ -"(($ O5LO]acg4hD $ $[ 1r/c|jj|gDchc]}|j} }|jr4||jvr&t |j|dkDr t d| } | | }t } n| |z } t |t | k(} |jj|} |} | r't|} |jj|| t|}|j| |t|||t|}| r| s|}| r"||k7r|jj||| |j!||t#||||}| s/t|d}|j!||t#|d||t|d}|j||t|d||t|d}| r| r|}|r"||k7r|jj||||j!||t#|d||t%||||t&j(|}t%||||t&j(|}t+||||j,|<ycc}w)a Adds Q and DQ ops to a tensor whose quantized data type is converted. That is, some consumers may use the original data type from the producer, while other consumers use the converted data type. This is generally done by adding a sequence of ops that convert from one data type (e.g., uint8) to another (e.g., uint16). T_float ---> Quant(to u8) ---> Convert(to u16) ---> Dequant(to float) ---> T_float' where Convert(to u16) is equivalent to: ---> Dequant(to float) ---> Quant(to u16) ---> This function handles the following scenarios: 1) Tensor T is not a graph output; all consumers use the converted type ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 ---> 2) Tensor T is not a graph output; some consumers use the original type, others use the converted type ---> Q1 -+-> DQ1 ---> | +-> DQ1' ---> Q2 ---> DQ2 ---> 3) Tensor T is a graph output; all consumers use the converted type ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 -+-> | +-> 4) Tensor T is a graph output; some consumers use the original type, others use the converted type ---> Q1 -+-> DQ1 -+-> | | | +-> | +-> DQ1' ---> Q2 ---> DQ2 ---> 5) Tensor T is a graph output that is not consumed by any other nodes. ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 ---> r z|Do not currently support converted quant_types in TensorQuantOverrides when the `dedicated_qdq_pair` extra_option is enabledN_convert_convert_clonerM)ryrtrrxr ValueErrorsetrr$rr&rr5rrrr;rrrrPr]r)rArfirst_scale_name first_zp_namescale_data_typeconvert_scale_nameconvert_zp_nameconvert_recv_nodesr tensor_recv_nodesoriginal_recv_nodesall_use_convertedr$ first_q_inputfirst_q_outputfirst_dq_outputsecond_q_inputsecond_q_outputsecond_dq_outputoriginal_quantized_valueconverted_quantized_values r0%_add_qdq_ops_for_converted_activationz2QDQQuantizer._add_qdq_ops_for_converted_activation5s`483U3U3Y3YZegi3jk4TYYkk  # #tAAAD66{CDqHO  0  %!2 "%% "58J"J  23s;L7MM**44[A$ 2;?M JJ 2 2; N0=  >+;K+HJZ\i 4K@ #4)O ?k#A JJ - -k?L_ `  O-? -LN^`m ) 3{m84LMN  "k].#AB   2[M2JK     }H5 6    5 }H5MN 0*  "2k"A JJ - -k;KM_ `    +h7 8    $2      $ $& $  %3      $ $& % !1H $&?AS1   -SlsI7c |jjjD]\}}||jvr|jr#t ||j j}|r|j|nx||jvr|j|=x|j|}|std|d|j\|j||jjj |jj"j |j$n|j$|jjj$k(sJ|j'||jjj |jj"j |j$|jjj |jj"j |j(|j|=y)z} Adds Q/DQ ops to tensors (activations and weights) that have been marked for quantization by op quantizers. z4Quantization parameters are not specified for param zb. In static mode quantization params for inputs and outputs of nodes to be quantized are required.N)r@)rqcopyrrr?rrrrJrz"_make_tensor_scale_zp_initializersr[rKrWrJrXrrYr@rnrM)rAr tensor_infortensor_qparam_initializerss r0rz%QDQQuantizer._quantize_normal_tensorss)-(@(@(E(E(G(M(M(O0 : $Kd666((*; 8N8N8PQ 77 D #d&A&AA 44[A 151X1XYd1e.5(RS^R_` 2;;C99'6??EEJJ6??JJOO&1&;&; : +448R8[8[8a8a8k8kkkkBB'6??EEJJ6??JJOO'116@@FFKK6@@KKPP6KK,,[9a0 :r/c |jr|jjjD]\}}|j}|s|j|j vs/|j|=|j |jj |j}|j|r td||jvrtd|dd}d}||jvrD|j|}|jr)|j||jd}|j}|)|j||j |j"(|j%||j |j"|j&j(|j&j*|j,j*||jryy)a{ Adds Q/DQ ops to tensors that have been marked for quantization by op quantizers. Only operates on tensors that want to use the quantization parameter initializers from an upstream tensor. For example, a Transpose node's output tensor will typically want to use the same quantization parameter initializers as the Transpose node's input. zBQuantization parameter shared mode is not supported for weight yetz5Quantization parameter sharing is invalid for tensor z& because it has already been quantizedNrY)rqrprr=r4rrRr5is_input_a_initializerr[rzrrKr?rMrWr1r2rnrXr@rrY)rArrrquant_providerrIconverted_qparam_initsrM tensor_paramss r0rz,QDQQuantizer._quantize_sharing_param_tensorss&&,0,D,D,I,I,K,Q,Q,S. ( [!,!@!@!n&?&?4C[C[&[00=&*&>&>~?X?X&Y&j&j&00'O22;?()mnn"d&A&AA(ST_S`aDD.2*+/("d&>&>>(,(@(@(M (22595U5U +]-D-Dj624A3U3U0-599')C)C_E\E\BB'+66+33288BB288==2==BB0M. &&r/c .|jjD]w\}}||jvr|j||t ||j j }|j j||j|j}|jdk(rt|jts.tdt|jd|jt!|}t"j$j'd|j(g|g||j}n?|jdvr|j*t"j,j.t"j,j0t"j,j2hvrt5d|j*d|j(|j6|j8g}t!|}|j:;t"j$j'd ||g||j:|j< }nIt"j$j'd ||g||j< }nt5d |jd |j j?|zy)zq Adds DQ ops (or Cast) for bias tensors that have been marked for quantization by op quantizers. CastUnexpected type z for input=)rto)NDequantizeLinearzUnexpected quantize type z for DequantizeLinear.Nr}r*)r+zUnexpected operator type r) rrrrquantize_bias_staticrrrremove_initializerrJ node_typerr@intrrr4rrrr,q_name node_qtyperrBFLOAT16r RuntimeErrorr1r2r>r|add_node)rArrinit quant_valuer5r:inputss r0rz#QDQQuantizer._quantize_bias_tensors@sT%)$9$9$?$?$A1 . IyD444  % %i ; 4::+A+A+CDD JJ ) )$ /229=FFK$$."$..#6#&6tDNN7K6LKXaXlXlWo$pqq.y9 #{{44 ''(K"~~ 5  &&*DD))$$,,$$--$$**. ')B;CYCYBZZp'qrr%,,k.D.DkFYFYZ.y9 ##/#';;#8#8*" !(--#11 $9$L$(;;#8#8*" !#11 $9$L#%>{?T?T>WWX#YZZ JJ   -c1 .r/c>||jvxs||jvSr;)rqrrrs r0is_tensor_quantizedz QDQQuantizer.is_tensor_quantizedws#d666^+I^I^:^^r/c|jj|}|y|jj|ry|jj |}|j s|sy|r|j j||n|}|r#|jj|}|dd}t|j}t||\} }| stjd|d|d|yd|fS)a Checks if a given tensor is configured to be quantized per-channel. If so, also returns the channel axis. ORT only supports per-channel quantization on static weights (i.e., ONNX initializers). If the user did not provide tensor quantization overrides for this tensor, then the value of self.per_channel determines if the weight is to be quantized per-channel. Params: tensor_name: The name of the tensor to check. default_axis: The default channel axis. This method checks if the normalized axis is within bounds. Can be overridden via the extra_options 'QDQOpTypePerChannelSupportToAxis' and 'TensorQuantOverrides'. op_type: Optional, defaults to None. The operator type that is the only consumer of this weight. Used to access the extra option 'QDQOpTypePerChannelSupportToAxis'. Returns: A tuple (is_per_channel, axis) in which the first element indicates whether the tensor is quantized per-channel and the second element is the channel axis. The returned axis is only None if the tensor is not per-channel or the axis is out of bounds. rrr>zAxis z is out-of-range for weight 'z ' with rank T) initializersrtrhas_per_tensor_overrideshas_per_channel_overridesrr{get_per_channel_overridesrdimsr rr) rArrrweight_initializerhas_per_chan_overridesr>per_chan_overrides weight_rank axis_valids r0rz"QDQQuantizer.is_tensor_per_channelzs2"..22;?  %  & & ? ? L!%!Zat;;??Vgs !!%!OOPbcnnJ ,Z F 1155k4HG$0q1A<$P!;L;X$%67b^bbr/c Z||jvr#|j|jjS|j|j|j }|t d|jd|d|j|j|j }|t d|jd|d|j||||j\}}}}} } t||||tj|jdkDrdnd| | } t| dd|j|<|S) z] Quantized the bias. Zero Point == 0 and Scale == Input_Scale * Weight_Scale Nz9Unable to get valid quantization scale for weight input 'z' when quantizing bias 'z ' to int32.z2Unable to get valid quantization scale for input 'r r)rr)rrJrrrEr5r[r4quantize_bias_static_implrGrrrArr]) rArrrrquantized_bias_namequantized_bias_scale_namequantized_bias_zp_namebias_scale_datarrrIs r0r~z!QDQQuantizer.quantize_bias_staticsj 00 0++I6??FF F::9;P;PR[ReRef  KILaLaKbc))2 ;@  99):N:NPYPcPcd  DYEYEYDZ[))2 ;@   * *9krrr'r zWrong scale/zp shapesz,Scale and zero-point must have the same rank _zero_point_scalezUnexpected dtype=z for param_name=)rtboolrrrr make_tensorraveltolistrrrrfloat32rrrfloat16rr[rW)rA param_namerCinit_name_suffixrYrXzero_point_typer>r is_blockedzero_point_namer1init_zprN init_scales r0r?z(QDQQuantizer._make_scale_zp_initializerss",/ W%&|4'++F3&**<; *% D,U[[1AQ1F4#3EKK8HA8M4.> @P@P@R@Y@Y@[  ""7+ ;;"** $#//55J [[BJJ &#//77J0 =Mj^\] ][[,,ZU[[RWR]R]R_RfRfRhi  "":.%j'::r/c|j||jvrtjd|dy|j|}t|tst dt |d|d|j||j}|jr|j||jdnd}t|||jS)a Create and returns all scale/zero_point initializers for a given tensor. If the tensor is converted to a different quantization type, this function creates two pairs of zp/scale initializers. Otherwise, only one pair of zp/scale initializers is created. Nz$Quantization parameters for tensor:"z" not specifiedr{ for rrY) rrrrrIrrr?rJrKr[rM)rArrxoriginal_initsconverted_initss r0rqz/QDQQuantizer._make_tensor_scale_zp_initializerss  # # +{$BZBZ/Z LL? }O\ ]00= -)=>.tM/B.C5WXYZ Z99+}G]G]^&&  , ,[-:Q:QS] ^  ,NO]MoMoppr/c|j}d|vr|dj}d|vrd|vr |d|d}}n|tjjk(rt ||j d\}}n|jd|jd}|jd|jd}|jd|j}|jd d } t|| | \} } t||| | ||j\}}|jr<|tjjk(r|st||| | |j \}}t!|j#|j#| S)z Calculates quantization parameters (scale/zero-point) given a tensor's min/max range and optional user-provided overrides. rrXrYr rrr symmetricrF)rr)qminqmaxmin_real_rangerYrXr)rr<rr FLOAT8E4M3FNravg_stdrt range_valueis_activation_symmetricrrr#is_activation_restricted_asymmetricUINT8r"r squeeze) rA tensor_dataquant_overridesrzerorXrrrrrrs r0calc_quant_paramszQDQQuantizer.calc_quant_params4sj ** ? *(6BBJ o %,/*I),79Q%D 4++88 81*k>Q>QRS>TUKD%"&&v{/F/Fq/IJD"&&v{/F/Fq/IJD'++K9U9UVI*..~uEL0,bklJD$*4tT9dNaNabKD%77J$JZJZJ`J`<`ir6$TTEXEX e"T\\^5==?_ijjr/c|jiS|ji}|jD]}|j|}t|tst dt |d|d|j j|i}|j||}d}d}d|vr)|j||d}|djd}t|||||<|S)z Calculates quantization parameters (scale/zero-point) for all tensors in the graph using each tensor's min/max range and optional user-provided overrides. Nr{rr) default_valconvert recv_nodes) radjust_tensor_rangesrr rrrget_per_tensor_overridesrrtrI)rArrtdrrJrKrMs r0rz$QDQQuantizer.calc_graph_quant_paramsPs    %I !!# -- oK##K0Bb*-"248*E+PQ RSS"99RRS^lnRoO--b/BHI#' O+ 222y7QR '6y'A'E'El'S$/CHiYm/n  , o #"r/c  i}|jjD]\}}t||jj }|s.t |}t |j}|jtju}|r |jn |j}|jj|rb|j|} d| dvr| ddj}t|} d| dv} | sQt!t#j$| dd| t#j$| dd|j&|||<ng} g} | D]]}| j)t#j$|d| | j)t#j$|d|j&_| dd}t+||\}}|st-d|j.d |d |t!t#j$| t#j$| || ||<|jj1|ig} d| dvr| ddj}| dj1d|j2}|d u} | xs|r|j5|n |j6}| dj1d |}| dj1d|j8}d }d }||nd}|r{|j:dkDrlt+||\}}|st-d|j.d|d ||}t=||||j:|\}}t!|||||j:||<9| set?|jA||||jB| dj1d| dj1d\}}t!|||| ||<t+||\}}|st-d|j.d |d ||}|j|}g} g} tE|D]}|jG||}| r|t | kr| |ni}t?|jI||||jB|j1d|j1d\}}| j)|| j)|t#jJ| }t#jJ| }t!|||| ||<|S)ze Returns quantization parameters (scale/zero_point/quant_type) for all initializers. rrr>rYrrXrzWeight z# has a per-channel axis with value z that is out-of-bounds for rank )rYrXrr>Nrrz" has a block-wise axis with value )rYrXrr>rrr)rr rmin_override rmax_override)&rqrrrrr#rrr<r&r,rrroverrides_scale_zprr rrrr r r[rrtr>is_weight_symmetricrrrrrflattenrrtakerr)rArrrrrinitializer_datainitializer_rank is_weightr overrideszp_dtyperzero_points_list scales_listchan_overrides channel_axis is_axis_validnorm_channel_axisis_symmetric_default is_symmetricrrYrX block_axisnorm_block_axis channel_countrper_channel_datachannel_overrideschannel_zero_point channel_scales r0rz+QDQQuantizer._calc_initializer_quant_paramsms >@(,(@(@(F(F(HP  $K&{DJJ4J4J4LMK4[A "#3#9#9: $//3E3L3LLI.7**T=R=RJ**==kJ 77 D 9Q</!*1l!;!G!GJ/ ;!'9QN>T>TU#-8' 4 (*$"$K*3l(// 9UW_0`a#**288N74KScSiSi+jkl$-Q<#7L7ElTd7e4M#4((%k&6&6%77Z[gZhi66F5GI 8J#%88,<#= hh{3#-. 8' 43377 bTJIy|+&q\,7CC $Q<++FK4D4DEL)5N$2$8A((4tGcGc !%Q<++K9MNL$Q<++ND?&&}5 6 ZZ(89  ;/3E))% 4#K0WP d#"r/r;)rr3)rr3r4r3r5r3)ronnx.TensorProtorTr)g?) r np.ndarrayrrrEr3rrrrrTztuple[bool, np.ndarray | None])Nr)r.r3r/r3r0r3r1r3r2r3r> int | Nonerr)r7r3r8r3r9r3r1r3r2r3r>rrr)rr)rBr)rr3rrrz str | NonerTztuple[bool, int | None])rr3rQr3rTznp.ndarray | None)rr3rrDrTr3))rr3rCr rr3rTrW)rr3rTz#QDQTensorScaleZpInitializers | None)rr rzdict[str, Any]rTr )rTzdict[str, QDQTensorQuantParams])rTzdict[str, QuantizationParams])'r(r)r*rBrrr&r+rrrrrrrrr rrr!r(r5r;r=rJrWrnrrrrrrr~r?rqrrrr.r/r0r`r`sc&J ,EIVhVsVsy:X "Ty /mbN-N-!N- N- " N-  N- (N-`1@f*6 > , --- -  -  ---@ --- -  -  ---F;;@DeLAiF[ z4:l6p5.n_# /// / ! /bc0.#bZ\(;(;-?(;SV(; (;Tq.k8#:X#r/r`)7 __future__rr dataclassesrenumrtypingrnumpyrrrrrbase_quantizerr r calibrater quant_utilsr rrrrrrrrrrrrrrrrrrr r!r"r#registryr$r&r2r9rDrIrWr[r]r`r.r/r0rs #! &=!2)    ##   ff f&   ** * ff f$t#=t#r/