:j=5 8ddlmZddlZddlZddlZddlZddlZddlZddl m Z ddl Z ddl ZddlmZmZmZmZddlmZmZmZmZddlmZddlmZmZddlmZdd l m!Z!m"Z"ejFd ejH ejJe&Z'Gd d Z(Gdde(Z)Gdde(Z*Gdde(Z+Gdde(Z,Gdde(Z-Gdde(Z.dZ/GddZ0dBdZ1dCdZ2 dDdZ3d Z4Gd!d"Z5Gd#d$Z6Gd%d&Z7d'Z8d(Z9d)Z:e&d*k(re:Z;e;jxre'j{ej|e;j~Z@e;jZBe!e;jZCe;jreEe;jnd+ZDe;jreEe;jndZFejjeBr!e'jd,eBd-eJd,eBd-e;jr'e;jd.k(re'jd/d0e;_Ke je@ZOe;jd.k(r"e,e;je;jeDeF1ZRn=e;jd2k(r8e-e;je;je;jeCeDeFe;j3ZRne;jd4k(r e)eD5ZRne;jd6k(r e*eD5ZRne;jd7k(re+e;jeD8ZRne;jd9k(r|eCe!jk(re'jd:e!jZCe@ZOe;je;jd;k(rdZRneZd?e;je7eOe;je;je;je;jeR@Z]e]je]jjeBdAyy)E) annotationsN) GraphProto ModelProto NodeProto TensorProto)quantize_matmul_2bitsquantize_matmul_4bitsquantize_matmul_8bitsquantize_qdq_matmul_4bits)CalibrationDataReader) gptq_quantize rtn_quantize) ONNXModel) QuantFormatattribute_to_kwargz2%(asctime)s %(name)s [%(levelname)s] - %(message)s)formatlevelc.eZdZ d ddZy)WeightOnlyQuantConfigNc||_||_|r t|ndh|_|r t |nddd|_||_y)aqThis is the Base class for Weight Only blockwise quantization Configuration. Args: algorithm: weight only quantize algorithm name. quant_format: QuantFormat{QOperator, QDQ}. QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. op_types_to_quantize (optional): set of operator types to quantize. Default {MatMul} quant_axes (dict[str, int], optional): op:axis, which axis to quantize for an op. Default {MatMul: 0, Gather: 1} customized_weight_config: customized weight config for nodes if needed. It is dictionary with node name as key, and the value is a dict of customized config. MatMulrr )rGatherN) algorithm quant_formatsetop_types_to_quantizedict quant_axescustomized_weight_config)selfrrrrr s /home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/onnxruntime/quantization/matmul_nbits_quantizer.py__init__zWeightOnlyQuantConfig.__init__&sG0#(AUC(<$=\d[e!.8$z*VW>X(@%)NNN) rstrrrrtuple[str, ...] | Noner"tuple[tuple[str, int], ...] | Noner dict | None)__name__ __module__ __qualname__r#r$r"rr%sH 8<9=04 AA"A5 A 7 A #. Ar$rcDeZdZdejddf dfd ZxZS)RTNWeightOnlyQuantConfigNcv|tjk(sJd|i}t| d|||||_y)ap This is a class for round-to-nearest (RTN) algorithm Weight Only Quant Configuration. RTN is the most straightforward way to quantize weight using scale maps. Args: ratios: percentile of clip. Defaults to {}. quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. customized_weight_config: customized weight config for nodes if needed. It is dictionary with node name as key, and the value is a dict of customized config. z"RTN only supports QOperator formatNRTNrrrr r QOperatorsuperr#ratiosr!r5rrr __class__s r"r#z!RTNWeightOnlyQuantConfig.__init__FsP0{444Z6ZZ4 >F %!5%=   r$rr&r r(r)r*r+rr3r# __classcell__r7s@r"r.r.Es5 **7;04 "5 " #. ""r$r.cDeZdZdejddf dfd ZxZS)KQuantWeightOnlyQuantConfigNcv|tjk(sJd|i}t| d|||||_y)aF This is a class for k-quant algorithm Weight Only Quant Configuration. Args: ratios: percentile of clip. Defaults to {}. quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. z&k-quant only supports QOperator formatNk_quantr1r2r6s r"r#z$KQuantWeightOnlyQuantConfig.__init__lsP({444^6^^4 >F %!5%=   r$r8r9r;s@r"r=r=ks5 **7;04 5  #. r$r=cLeZdZddddddejdf dfd ZxZS)GPTQWeightOnlyQuantConfigNg{Gz?FTc |tjk(sJdt | d||||_||_||_||_||_||_ y)a This is a class for GPTQ algorithm Weight Only Quant Configuration. GPTQ algorithm provides more accurate quantization but requires more computational resources. Args: calibration_data_reader: a calibration data reader. It enumerates calibration data and generates inputs for the original model. percdamp: percent of the average Hessian diagonal to use for dampening. block_size (int, optional): channel number in one block to execute a GPTQ quantization iteration. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. z#GPTQ only supports QOperator formatGPTQ)rrrN) rr3r4r#calibration_data_readerpercdamp block_sizeactordermse perchannel) r!rErFrGrHrIrJrrr7s r"r#z"GPTQWeightOnlyQuantConfig.__init__shF{444[6[[4 %!5  (?$  $  $r$)rEzCalibrationDataReader | Nonerr&r9r;s@r"rArAsAAE  **7;/%!=/%5/%/%r$rAcHeZdZdddejddf dfd ZxZS)HQQWeightOnlyQuantConfigrBr Nc|tjk(sJdt| d|||||_||_||_y)a This is a class for HQQ algorithm Weight Only Quant Configuration. HQQ algorithm quant weight without needing calibrate data. Args: block_size (int, optional): channel number in one block to execute a HQQ quantization iteration. bits (int, optional): how many bits to represent weight. axis (int, optional): 0 or 1. which axis to quantize. https://arxiv.org/pdf/2309.15531.pdf quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. quant_axes (dict[str, int], optional): op:axis, which axis to quantize for an op. Default {MatMul: 0, Gather: 1} z"HQQ only supports QOperator formatHQQrrrrN)rr3r4r#rGbitsaxis)r!rGrQrRrrrr7s r"r#z!HQQWeightOnlyQuantConfig.__init__sT:{444Z6ZZ4 %!5!  %  r$)rr&rr'r9r;s@r"rLrLs;   **7;9=' 5 '7''r$rLc`eZdZdddejddddf dfd ZxZS)DefaultWeightOnlyQuantConfigrBFNrMc t |d|||||_||_||_||_||_|r|tjk(r tdyy)a  This is a class for weight only affine quantization configuration. Args: block_size (int, optional): channel number in one block to execute an affine quantization iteration. is_symmetric (bool, optional): whether quantize weight symmetrically. accuracy_level (int, optional): Accuracy level of the 4-bit quantized MatMul computation. Refer to the MatMulNBits contrib op's 'accuracy_level' attribute for details. (https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits) quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. quant_axes (dict[str, int], optional): op:axis, which axis to quantize for an op. Default {MatMul: 0, Gather: 1} bits (int, optional): number of bits per element after quantization. Default 4. DEFAULTrPzAQuantFormat.QOperator is not supported channel_wised_quantize yetN) r4r#rG is_symmetricrQaccuracy_levelchannel_wised_quantizerr3NotImplementedError) r!rGrWrXrrrrQrYr7s r"r#z%DefaultWeightOnlyQuantConfig.__init__sqD %!5!  %( ,&<# !lk6K6K&K%&ij j'L !r$)rGintrWboolrX int | Nonerr&rr'rQr[rYr\r9r;s@r"rTrTsz"%) **7;9=',.k.k.k# .k 5 .k7.k.k!%.k.kr$rTc2eZdZ dfd ZdZdZxZS)NVAWQWeightOnlyQuantConfigct ddl}ddlm}||_||_ ddlm}||_ dd lm}m } ||_| |_ t |-d tjdd |jj|jjj!rdnd} |j#|||ddd| ddddd } | |_||_y#t$rt dtddwxYw#t$rt dtddwxYw#t$rt d td dwxYw)a= Configuration for the nvidia_awq quantization method. Args: tokenizer_dir (str): pathof the tokenizer dir. dataset_name (str): Name of the dataset. cache_dir (str): Directory for caching. calibration_method (str): calib method for nvidia_awq. rN) DataLoaderzfError: The 'torch' library is required but not installed. Please install it using 'pip install torch'.z torch is not installed. Exiting.) load_datasetzlError: The 'datasets' library is required but not installed. Please install it using 'pip install datasets'.z#datasets is not installed. Exiting.) AutoConfig AutoTokenizerztError: The 'transformers' library is required but not installed. Please install it using 'pip install transformers'.z'transformers is not installed. Exiting. nvidia_awqrPcudacpu r iTFrB) dataset_name model_name cache_dir calib_size batch_sizerGdeviceuse_fp16use_buffer_shareadd_past_kv_inputsmax_calib_rows_to_loadadd_position_ids)torchtorch.utils.datara ImportErrorprintdatasetsrb transformersrcrdr4r#rQDQrnrf is_availableget_calib_inputsrEcalibration_method) r! tokenizer_dirrirkr}rtrarbrcrdrn calib_inputsr7s r"r#z#NVAWQWeightOnlyQuantConfig.__init__sm" L  3DJ(DO O - ,D  S >(DO!.D  "$!%  ""TZZ__-I-I-K6QVW,,%$"##&!-  (4$"4o L x @At K  L O ~ CD$ N  O S G GHd R  Ss!C C2D!C/2!D!D7c b|j} |} |} t|tr<| j||| j} | j||| j} | j | j d} |rJ| j jddz } | j| dk(d| j | d<|r|r | jn | j}| j\}}|j}|j|j|jz}}t!|j"D]q}| j%|||r|nd|||}| j%|||r|nd|||}| j'd|d|j d|d |j is| S) N)rndtype) input_idsattention_maskr r position_idszpast_key_values.z.keyz.value)rt isinstancelisttensorint64 contiguouslongcumsum masked_fill_float16float32shapemax_position_embeddingsnum_key_value_heads hidden_sizenum_attention_headsrangenum_hidden_layerszerosupdate)r!config input_ids_argattention_mask_argrqrnrorprsrtrrinputsr torch_dtypermsequence_lengthmax_sequence_length num_heads head_sizeipast_key past_values r"make_model_inputz+NVAWQWeightOnlyQuantConfig.make_model_inputns ! + mT * ]6 UI"\\*F> " +3%--K*3// 'J"("@"@ **""f&@&@@!I6334  ;;+;'!% '#[[+;'!% )  *1#T2H4G4G4I*1#V4j6K6K6M# 0 r$c 4|j} |j}|j}| j|d|d}|j|d|d}|j ddi|j |_|| ksJdd|vr&|ddd jt| }d }n d |vr |d d}d}ntd|d||d|}|j|ddd|}|j|}|d}|d}|j}|||d}|||d}t|jt|jk(sJt|t|k(sJ||z}g}t|D] \}}|j!|||dz k(s ng}t|D] \}}|j!|||dz k(s nt#d|dt|dt|dg}t|D]t} || }!|| }"|j%||!|"| ||| | }#|#j'D$%cic]$\}$}%|$|%j)j+&}#}$}%|j!|#vt#dt|d|Scc}%}$w)NT)use_auth_tokenrktrust_remote_code pad_tokenz[PAD]z8calib size should be no more than max_calib_rows_to_loadcnn cnn_dailymailz3.0.0train)namesplitarticlepilezmit-han-lab/pile-val-backup validation)rtextz dataset "z" not supportedpt)return_tensorspadding truncation max_lengthrrF)rmshuffler z/ --Quantize-Script-- number_of_batched_samples=z, batch-input-ids-list-len=z, batched_attention_mask= z0 --Quantize-Script-- number of batched inputs = )rcrdrbfrom_pretrainedadd_special_tokens eos_tokenrselectr ValueErrorbatch_encode_plustoralendataset enumerateappendrwritemsrgnumpy)&r!rirjrkrlrmrGrnrorprqrrrs auto_configauto_tokenizerrbr tokenizerdataset2column batch_encodedbatch_encoded_input_idsbatch_encoded_attention_mask data_loadercalib_dataloader_input_idscalib_dataloader_attention_masknumber_of_batched_samplesbatched_input_idsidxdatabatched_attention_maskbatched_inputs_listrrrr input_name torch_tensors& r"r|z+NVAWQWeightOnlyQuantConfig.get_calib_inputssS oo ++(( ,, tyTX- #22 tyTX3  $$k7%;<'11 33o5oo3 L #O'QXXY^_uYvwHF | ##$AVHFyoFG GF#KZ0!33 T4DU_4 &((0 "/ "<'45E'F$oo %01HU_in%o"*5 (Z+ '-556#>]>e>e:ffff-.#6U2VVVV$.*$<!"#=> IC  $ $T *0145  "$"#BC IC " ) )$ /0145  >?X>YZ((+,=(>'??XY\]sYtXuuw y !01 /A)!,I3A6N**"   F^d]i]i]klAY\j,"2"2"4":":"<<lFl  & &v . /" A#FYBZA[[]^_"" ms;)J)r./cacheawq_lite)r)r*r+r#rr|r:r;s@r"r_r_s!% N5`?Ba#r$r_cPt|tj||z z|k(SN)r[npceil)val1val2s r" is_divisiblers$ tbggdTk** +t 33r$c^eZdZ ddZe d d dZedZ d dZd dZy) HQQWeightOnlyQuantizerc||_yrrr!rs r"r#zHQQWeightOnlyQuantizer.__init__  r$Ncddl|dddddn|}|d|d|d |d f\}}} } |jr jn j} |j | } |j | }|j | }|ffd } d }t | D]}j | |z|zj|d|d }||z |z }| | |z |}j|| |z |zz |d}|| z}tj| |z j}|r t|tj |d||kr|}n~ ~~~||fS)Nrgffffff?g$@g)\(?)lp_normbetakappaitersrrrrc |dk(rLj|jjjj |d|z z zSj|jjjj |d|z j j |dz|dz zz zS)Nr ?g:0yE>)signnn functionalreluabspow)xrprts r" shrink_opz:HQQWeightOnlyQuantizer.optimize_weights..shrink_op8sAvzz!}uxx':':'?'? ! sUYz@Y'ZZZzz!}uxx':':'?'?IIaLC$J%))EIIaL4Ma-34z)*   c3d{r$c |jd|jdk(r|j}|j}|dvrA|jdz|z}t|D]}|ddxxx||d|||zzzcccyt d)NrrMrzOnly 2,4,8 bits are supported.)rT element_sizerrZ) pack_tensorori_int_tensorrQcompress_ratiojs r"pack_on_row_fast_248bitz.HQQWeightOnlyQuantizer.pack_on_row_fast_248bitTs   Q >#7#7#: :+--N%--K 9 (557!;tCN>* UAB>!2C^2C#DQR#TT U&&FG Gr$c<ddl}|j} | j} || ||zz |z} |dk(r+|jjj | d| fdd} n,|jjj | ddd| fdd} | j} |-|r+|dk(r| j d|gn| j |dg} |dur#| j| j}} d}n,| j|dd} | j|dd}d|zdz }d}||g}||| z z jd }|| z }|dk(jjdkDr|||dk(<||z jd }| |z}|r|j|}|r|j| |||| \}}|j| |z|zj|d|d}|j | j}d |z }|dk(r+|j | dd}|j | dd}n*|j d| d}|j d| d}~ ~ ~||j|j |j|j fS) Nrr constantrFTrrg@)max)rrrrrRr)rtrrrrpadreshapeminrrsumitemrr r[rr)r!rrQ channel_wise group_sizeoptimize round_zerorRrtweight ori_shapepad_lenr_min_maxmax_vmin_vrr min_max_axisrr s r"quantize_internalz(HQQWeightOnlyQuantizer.quantize_internalas LL  $* << J 19XX((,,Va\:qQFXX((,,VaAw5GUVWF   " :>!)V^^R$45&..ZdfhYiJjF 5 vzz|$DH::4:6q9D::4:6q9D4! %.$+&--#-6d{ A  " " $ ) ) +a /.3L* +\)00S09Euu} ;;t$D //vUQU_fmq/rKE4kk&5.4/066wqz71:Nkk% $$&e  19MM%(B/E<<a"-DMM"eBi0E<<E"I.D D$EHHV\\*DGGFLL,AAAr$c  |jdk(r tdddl}tj d|j d|j d}t||\}}|tj d|gStjj|}|j}t|d k7rxt|d krtj d |gS|dd } td | Dr"tj d t| |gS|j|d |d}nd}|j!|} |j"j%r| j#} |j&j(} |j+| j,| |j&j.\} } }| j1} | j1} |j1}d| z}|j3| jd| jd|zf|j4| j6}|j9|| | | j;j=}|j;j=}|jd}|jd}| j\}}|j&j.}||z}||zdz |z}|j|||}tjj?|j;j=}|j dztA| z|_|j D].}|j |k(s|j jC|ntjj?|}|j dz|_|jDjG||g|j d|j |j g}tjj?|}|j dz|_|jDjG|g|jI|j i}|j\}}||d<||d<| |d<|j&j.|d<tK|j d|}| t|nd}|duxr |duxs||k} | s|jLdn|jLddz}!tjNjP d||!g|j r|j dztA| znddd|}"|"g}#| rz|#jGtS|j d|||j r|j dztA| zn|j dztA| z|!|jLdtj d|j d|#S)  Target node: QOperator node: QDQ nodes: MatMul MatMulNBits DeQuantizeLinear -> MatMul Gather GatherBlockQuantized Gather, Gather, Gather (optional) -> DequantizeLinear If the node is target node with fp32 or fp16 const weight, quantize the weight to int4 and return the new nodes. If QOperator format, return the corresponding QOperator nodes. If QDQ format, return the corresdponging QDQ nodes. Gather (quantized data) + Gather (scales) + Gather (optional, zero points) -> DequantizeLinear is not supported yet because Gather does not support int4 data. rz/Gather quantization is not supported yet in HQQrNstart to quantize  ...r 2MatMul doesn't have const weight. Skip to quantizerz2HQQWeightOnlyQuantizer.quantize..+a16+cMatMul weight has non-unit batch dims %s; N-D batched quantization not supported. Skip to quantize.r)rQr"r)rrn_Q_scales _zero_pointsKNrQrG _pre_reshape com.microsoftroutputsrdomain a_input_nameb_original_shape target_graph name_prefixpre_reshape_output final_outputcomplete quantization of  MatMulNBits)*op_typerZrtloggerinforinputget_initializeronnx numpy_helperto_arrayrranyrr from_numpyrfr{rrQr-rrGrruint8rnrrgr from_arrayr%remove initializerextendr_get_static_rankoutputhelper make_node_build_nbits_output_reshape)$r!node graph_stackrtinput_bb_pbbs_graphb_arrayrKleading b_array_torchrQquant_weight_torch scales_torchzero_points_torch packed_size packed_torchscales zero_pointsrowscolsrG blob_sizek_blocksb_quantrV scales_tensor input_names zp_tensorkwargs a_static_rank rank_b_orig needs_reshapematmul_q_output matmul_q_node output_nodess$ r"quantizezHQQWeightOnlyQuantizer.quantizesr <<8 #%&WX X ( 489**Q-(+>h < KKL M6M##,,T2"==  A %#$q( Z[v &s+G+7++ (M v oo&6r&:B>T>T OO$4;;3I3I?U? ;L*;0::<#..0 -88:4i {{  % %a (*<*B*B1*E*T U++%,,# $$\3EtL!!#))+'++-335 #!))"- "(( d[[++ +- :%)j8#++D(IF ##..|/?/?/A/G/G/IJyy4'#d)3 ^^ EzzW$%%e,  ))44V< !YY2 ##Wm$<=zz!}gllM4F4FG %%00= ^3 ##YK09>>*]] ds s v#{{55|)A D /?/Kc*+QR (4o-4:O:nS`cnSn 0=$++a.4;;q>TbCb --  $%15T!CI-"    &    +!%A%5!)BF))T!1CI!=RVR[R[^bRbehimenRn'6!%Q    / {$?@r$)rrL)rNF)rz list[int]rRr[rr()rMT@TTr rgrrhlist[GraphProto]returnlist[NodeProto]) r)r*r+r# staticmethodr rr-rr,r$r"rrsw( "&2 2  2  22h H Hfg?BBEr$rctt|dz ddD]/}||}|jD]}|j|k(s||fccS1y)Nr r)NN)rrr`r)r graph_pathgidgraphrs r"rWrW*s\S_q("b1%3'' %F{{d"u}$ %% r$ctt|dz ddD]}||}t|jt|jzt|j zD]^}|j |k(s|jj}|jdr#t|jjccSyy)uReturn the static rank of a tensor if its shape is known, else None. Searches graph inputs, value_info, and outputs in the graph stack (inner-most graph first). A known shape requires ``HasField('shape')`` to be true on the tensor_type; the rank is then ``len(shape.dim)``. Individual dim sizes may still be symbolic — only the rank (dimension count) matters here. r rrN) rrrrV value_inforcrtype tensor_typeHasFieldrdim) tensor_namerrrvitts r"rbrb3sS_q("b13u{{#d5+;+;&<= rank(B_orig) where no leading-1 prepending is needed). All ops used are valid from opset 11 onward. Args: a_input_name: name of the activation input edge (A) feeding MatMulNBits. b_original_shape: the original (pre-squeeze) shape of B, e.g. ``(1, K, N)``. target_graph: graph proto to append helper initializers into. name_prefix: unique prefix for generated node/initializer names. pre_reshape_output: name of the MatMulNBits output edge (the input of the generated Reshape). final_output: name of the final edge produced by the generated Reshape (must match the original MatMul output edge). Returns: List of nodes to append to the consumer's ``output_nodes`` after the MatMulNBits node. Initializers are appended to ``target_graph`` in place. r__zero_one_two_one_vec_rank_b_n_vec _zero_vec _one_valuerrrr r_a_shape_a_shape_of_shape_a_rank _a_rank_eff _extra_raw _extra_count_extra_count_vec _extra_ones _a_rank_m1_a_rank_m1_vec_a_prefix_shape _target_shapeShape_shape_a_shape_of_a_shaper _gather_rankMax _max_rank_effSub_subr)Reshape_reshape_extraConstantOfShape _const_ones)r data_typedimsvals)rvalue_sub_one_reshape_rank_m1Slice_slice_a_prefixConcat_concat_target)rrR _reshape_out)rr`rarXrYr^rarrayrrdre make_tensorrINT64)rJrKrLrMrNrOrn_dimr init_zeroinit_oneinit_two init_one_vec init_rank_b init_n_vec init_zero_vecinit_one_value_templatea_shapea_shape_of_shapea_rank a_rank_eff extra_raw extra_countextra_count_vec extra_onesa_rank_minus_onea_rank_minus_one_veca_prefix_shape target_shapenodess r"rfrfFsH&'K R E cL(AG I6zH6zHz>Li-KXJ OM,.##    ( (!288)D9 ( U    ( (!288)D8 ( T    ( (!288)D8 ( T    ( (1#RXX)F\ ( Z    ( (+RXX)NU` ( a    ( (5')JQ[ ( \    ( (1#RXX)F] ( [  *nG.. ]F]"JL In$K,,O]"J<'//**N&L g ~yq:~V gy3C2D1ObKbc h)99(ExVWZhVhi efh%7*AP_L_` ek:%> RSV\R\] ei%;k]QRU[Q[\ i+|)DFW^_br^rs     L]"++)),%++SS *   efh%7:J9KRSV`R`a  (,7:N9OVWZlVl     m%9= I  &&     4 N%%      . N^#  Y2 Ef Lr$cb||z}|dz dz}|dz}tj|tj}||ddd<||ddd<|j||f}|j}|jd} | ddddzdz| ddddzz}|j tjS) NrMrrrr r)remptyr]rrastype) packedrvrwtotalhighlow int4_vals int4_matrixint4_matrix_transposedflats r"transpose_packed_int4_matrixrs 4KE aK4 D 4-Cbhh/IIaddOIaddO##T4L1K)]] " ) )" -DADqDzQ$&41:+< =F == ""r$ceZdZd dZd dZd dZeddZeddZeddZ e ddZ d dZ d d Z y )DefaultWeightOnlyQuantizerc||_yrrrs r"r#z#DefaultWeightOnlyQuantizer.__init__s  r$c |jj}d|z}t|jdk7r t d|j\}}|jj }||zdz |z}|jj tjk(r||zdz |z}||z} | |z } | dkDrtj|d| fdfd}tj|||fd } tj|||zdz |zfd } tj||f|j } |dk(r(t| || | ||||jjn!|dk(r't| || | ||||jjnt!| || | ||||jjn|jj"r|n|jj }||zdz |z}|d k(sJd tj||zdzdzd } tj||zdzdzd } tj||f|j } t%| || | ||||jj| | | fS) z54b/8b quantize fp32 weight to int4 using C++ kernels.rrz9Current int4 block quantization only supports 2D tensors!r r)rrrr]rrMz+QDQ format only support 4 bits quantization)rrQrrrrGrrr3rrrrrrWr r rYr )r! fp32weightqbitskpackrvrwrGryrx padded_rowsr'r zero_pointrts r"qbits_block_quantz,DefaultWeightOnlyQuantizer.qbits_block_quants|   U  z A %XY Y%% d[[++ :%)j8 ;; # #{'<'< <#e+a/E9I"Z/K!D(G{VVJ!Wv0F S XXtXy9IF48e+;a+?E*I"KSZ[JXXtX.j6F6FGFz%J JdTXT_T_TlTl!%J JdTXT_T_TlTl&J JdTXT_T_TlTl "&!C!CI_I_Jz)A-*.*r;r<r=rbfloat16r>rr?_DQ_QT _DQ_scalesrr@rArBrQrGrXrCrDrErFrI_output _transposed_DQ_zero_points)rRrG)rrGrr _matmul_Q Transpose)rrGpermrQ)DequantizeLinear)5rrQrWrINT8UINT8INT4UINT4rVrWrTrUir from_protorrrr[rrrrrrr ml_dtypesrrrr3serdeserialize_tensorTensorrrXrdrtobytesrzrYrr_r`rarYr^rrGrXrbrcrerf))r!rgrhrQqtyperib_tensorb_graph b_ndarrayrKrmrrrtrurzr{qdq_opt_for_intel_npu_enabledrvrwrVrr|r}r~rrr qop_outputmatmul_qbit_nodedq_input_namesdq_output_namestp_input_namestp_output_namesmatmul_input_namesqdq_matmul_outmatmul_output_names dq_kwargsdq_node matmul_nodetp_nodes) r"quantize_matmulz*DefaultWeightOnlyQuantizer.quantize_matmuls {{ 19(, (@(@K$$kFWFWE(, (@(@K$$kFWFWE**Q-+G[A'   KKL M6MMM(+113 $??  A %#$q( Z[v &s+G+7++ (M v !))*:2*>@PQS@TUI# ??j0 !((4I&*&<&S>rWdgrWrM/<Q$++a.SaBaJ#{{44 "# 04 TYY2dV,r&          0 1##/%)ZZ])9%,AETYY2dV%,>?N&||i78O-a01N.q1MABO 1 &C"YZI["  -TZZ]KHM3C3O#./UVK,D8smt>S>rWdgrWrM3@T[[^dkkRSnWeFeN#1"2 ;;++ KK33MM$55ufllKL_L_Lacg %%inn5##**I;7"JD$:&*kk&H&HddkkNdNdIkk++"%'3799TYY5/"   G++//)+7;yyTYY9TF!33b 0K -++//)+Q 0 ##Wg{$CD##Wk$:;##/%)ZZ])9%,DHIITYY5%?T\TaTafklpkqdrTr+9%)[[^  r$ctj|dd}tj|dd}tjtj|tj|kD||}|dz }tj|dk(d||z j j ddjtj}||fS)Nr TrRkeepdimsg ri) rrrwhererrcliprint8)rmax_valmin_valabs_maxrquantized_slices r"quant_slice_symmetricz0DefaultWeightOnlyQuantizer.quant_slice_symmetrics&&A5&&A5((266'?RVVG_DDFKKArRYYZ\ZbZbc ((5A:q$,2KLRRTYYZ[]_`gghjhphpqz11r$c|jd}t|dzdk7rtj|d}|ddddz|ddddzdzz}|j dS) z2Pack int8 data to int4 and store in uint8 ndarray.rrrNrr rMr])rrrrr)r data_flatquant_data_int4s r"pack_int8_to_int4z,DefaultWeightOnlyQuantizer.pack_int8_to_int4soLL$ y>A  " )Q/I$SqS>C/Yqt!t_s5Jq4PQ%%g..r$cd}|j|}d}t|jD]\}}||kr||z}||kDs||z}||zdz |z} t|j} | | |<|j|||f} t j || |f|j } |rt j |||fd} n4t j |||fd} t j || |fd}td||D]}t||z|}| dd||ddf}|rtj|\}}ntj|\}}}|| dd||ddf<||z}|| dd||dzddf<|r{dd||dzddf<tj| }d}|stj}| j| } || |fS)zXQuantize ndarray data to int4 using numpy, return (quantized data, scales, zero points).r rr)r]rN) rrrrrrrrrrr.r2r6)r quantize_axisrGrWmknrrry scales_shape data_reshapertquant_data_int8zero_point_int8end_idxslicequantized_slice_int8 scale_slicezero_point_slice_int8rr5zero_point_int4s r"quantize_ndarrayz+DefaultWeightOnlyQuantizer.quantize_ndarrays  JJ} %  + FAs= S]"S    NQ&:5DJJ' &. ]#||Q1I. 1h*$**=  hh1ay?O hh1ay@O hh8Q'7wGOq!Z( KA!j.!,G AgIq1E4N4d4dej4k1$k/EEeLI$k3H0DOAqy!O ,ZA(3F1a1q5k1$ %5J1A; 12 K$5FFW8JJ?[O -77r$c|jjtjk(sJd|jjrt j nt j}|jd}t||\}}|tjd|gStjj|}t|j }|jj"j%dd} |jj&} | |kr| | k\sJd| dk\r | dz | zdk(sJd| |z|z} |j)|| | |jj\} } } |jD].}|j*|k(s|jj-|ntj.j1|j*d z||j | j3d }tjj5| |j*d z}|j*|jd|j*g}|j6j9||g|jjstj.j1|j*d z|| j | j3d }|j;|j*|j6j9|g tj.j=|d }|| | d}tj.j@ d||jBdg|j*r|j*d znddd|}|gS#t>$rd}YdwxYw)z,Quantize weight data of Gather node to int4.z0Gather only supports QOperator format currently.rz4Gather doesn't have const weight. Skip quantization.rr z&Invalid quantize axis for Gather node.z#Invalid block size for Gather node._Q4Tr?r@rR) gather_axisr8rGrDrErF)GatherBlockQuantized)"rrrr3rWrrrrVrWrTrUrXrYrZrrrgetrGrFrr_rdrrr^r`rarget_node_attr_valuerrerc)r!rgrhrdata_argdata_tensorprotodata_graphproto data_ndarray data_rankr8rGquantized_datartrurVquantized_data_tensorprotoscales_tensorprotor|zp_tensorprotorJr~gather_q4_nodes r"quantize_gatherz*DefaultWeightOnlyQuantizer.quantize_gather#s5{{'';+@+@@tBtt@$(KK$<$<   +BSBS::a=,;Hk,R)/  # KKN O6M((112BC  **+  ..228Q? [[++ y(]yj-HrJrrHRj1n %Ba%GoJooH&2i? .2.C.C -T[[5M5M/ + %** EzzX%%%,,U3  &*[[%<%<  ! !E )5,2D2DnF\F\F^`d& ""..99&BRBWBWZcBcd166 1 GYG^G^_ ##**,FHZ+[\{{''![[44 %%6v||[M`M`MbdhN   ~22 3  ' ' . ./? @ ++99$GK '*$  .. " [[^$&*iiU"R"    % K s5 M,, M:9M:ctjd|jd|jj}|j dk(rW|dk(r?|jj tjk(rtjd|gS|j||}ny|j dk(rD|jjdk7rtjd|gS|j||}n&tjd |j d |gStjd |jd |jjd |S)r/r0r1rrz>MatMul only supports QOperator format for 8 bits quantization.rrMz)Gather only supports 4 bits quantization.zUnsupported operator z1 for weight only quantization. Skip quantization.rPz with z bits ...) rTrUrrrQrSrrrzerrorr"rX)r!rgrhrQresultss r"rz#DefaultWeightOnlyQuantizer.quantizecs  ( 489{{ <<8 #qyT[[55H ]^v **4=G \\X %{{1$ HIv **4=G LL0>op q6M / {&AQAQ@RR[\]r$N)rrT)rz npt.ArrayLiker)tuple[np.ndarray, np.ndarray, np.ndarray]r)r np.ndarrayrztuple[np.ndarray, np.ndarray])rr]rr\)rr]rr]) rr]r8r[rGr[rWr\rz0tuple[np.ndarray, np.ndarray, np.ndarray | None]) r)r*r+r#rr"rr.r2r6rFrXrr,r$r"rrs0,dwr&&22//58585858 58 : 5858n> @r$rc eZdZ ddZddZy)NVAWQWeightOnlyQuantizerc||_yrrrs r"r#z!NVAWQWeightOnlyQuantizer.__init__rr$c ddlm}tj d|j j}|||j j|}tj d|S#t$rtdtddwxYw) z Perform nvidia_awq quantization using ModelOpt's int4 quantize function. Args: model (ModelProto): The ONNX model to quantize. Returns: ModelProto: The quantized ONNX model. r)rzlPlease ensure that the 'modelopt' package is installed. Please install it using pip install nvidia_modelopt.zXmodelopt is not installed. Please install it using pip install nvidia_modelopt. Exiting.Nz#Starting nvidia_awq quantization...)r}rE"Completed nvidia_awq quantization.) modelopt.onnx.quantization.int4rrvrwrTrUrrEr})r!model quantize_int4rquantized_models r" quantize_awqz%NVAWQWeightOnlyQuantizer.quantize_awqs  Q  9:{{:: ( #{{==$0   89+  ~ j   s A))!B N)rr_)rdModelProto | strrr)r)r*r+r#rgr,r$r"r_r_s* !r$r_c eZdZdZddddddej ddddf d dZd dZdZd Z d Z y) MatMulNBitsQuantizera Target node: QOperator node: QDQ nodes: MatMul MatMulNBits DeQuantizeLinear -> MatMul Gather GatherBlockQuantized Gather, Gather, Gather (optional) -> DequantizeLinear Perform 2/4/8 bits quantization of constant weights for target nodes. If algo_config.quant_format is QOperator: - nodes are replaced by the corresponding QOperator nodes. - quantized weights are stored in the contrib ops. If algo_config.quant_format is QDQ: - the quantized weight is stored in a standard onnx node. For MatMul, it is DequantizeLinear. For Gather, it is the three Gathers, one for quantized data, one for scales and one for optional zero points. - The nodes are replaced by the corresponding QDQ nodes. - currently Gather is not supported in QDQ because Gather does not support int4 yet. Note: - for quantized gather, the memory usage of "DequantizeLinear + Gather" is the same as the original Gather during runtime. Therefor it is not recommended. - when a node is in nodes_to_exclude, and the node configuration in algo_config.customized_weight_config will be ignored. rMrBFNc |g}t|trttj|n t||_t|tr|nd|_||_||_||_ ||_ t||_ |r t|nd|_ d|_| t||||| | || } | |_t#|j dr|j jdvsJd| j$dk(rt'|j |_y| j$dk(rt)|j |_y| j$dk(rt+|j |_yy)N)rGrWrXrrrrQrYrQrz(Only support 2, 4 or 8 bits quantizationrOrVre)rr%rrXloadrd model_pathrQrGrWrXrnodes_to_excludenodes_to_includenode_quantizerrT algo_confighasattrrrrr_) r!rdrQrGrWrXrnrorrrrYrqs r"r#zMatMulNBitsQuantizer.__init__sY  #! 4>uc4JYtyy/0PYZ_P` #-eS#9%t $(, #$4 59I$4 5t"  6%)-)%9%'= K' 4##V ,##((I5 a7a a5  E )"89I9I"JD   " "i /".,s%UQ4::55a8D@Us'*rMsymasym)rQr"scheme) rdrrgrSallrVrGrWrqr rr)r!q4_node_configrgtemplate_config_q4keyrs` r"_generate_q4_node_configz-MatMulNBitsQuantizer._generate_q4_node_config'sJJ$$**// CD||z)U$**UU !&*oo+/+<+<%&*& ((AA II)9)9)R)RR*.*:*:*S*STXT]T]*^*d*d*f@JC"&88:? 23 7@1CN499- C r$cfd}i}jj|d<j}jj}tj d|d|dvrujj |d<||d< jD]}d ||< tdj jnjj|d |_ n|d k(rɉjj|d <jj|d <jj|d<jj|d<jj|d<d|d<|}t!dj jnjj||d|_ tj d|dy)u4b quantize a model with RTN or GPTQ algorithm. Please refer to https://github.com/intel/neural-compressor/blob/master/docs/source/quantization_weight_only.md for more details on weight only quantization using Intel® Neural Compressor. c3zKtjjj}|D]}|df ywr)copydeepcopyrqrE) data_readerrr!s r"inc_dataloaderz.inc_dataloaderBs:--(8(8(P(PQK# !Dj  !s8;NrXzstart to quantize model with z algorithm...)r0r?r5rfp32)rd weight_configrDrF blocksizerHrIrJr n_samples)rdr dataloaderz$complete quantization of model with z algorithm.r,)rXrrqrrTrUr5rnrrmrdrFrGrHrIrJr)r!rr~weight_only_node_configrr;rs` r"int4_quant_algoz$MatMulNBitsQuantizer.int4_quant_algo<s  !     *'+':':F# $"&"?"?"A$$..  3I;mLM * *#//66F8 "+F;  ** 4-3'* 4&)-)Ddoo$**JZJZ5DJ & !%!1!1!:!:F: "&"2"2"="=F; !%!1!1!:!:F:  ,,00F5M#'#3#3#>#>F< "$F; ')J&)-)Ddoo$**JZJZ5% DJ  :9+[QRr$cf|jjdvrD|jjg}|jjt j k(r|jjdd|jjt jk(sd|jjvr{|jj}|D]\}|jdvs|jdks"tjd|jj|jd^|j||jj!y|jjdk(rtj#d |j$j'|j(|jjn |j(|_tj#d t+|j|_|jj!y|j-y) N)rOrVrEr r)Nzai.onnxrDzThe opset of the input model is under 21 and doesn't support int4 data type. Force to update it to opset 21, but the generated model may not be a valid model.rez%Processing nvidia_awq quantization...rb)rqrrdrrrr3set_opset_importrzr opset_importrHversionrTwarningryclean_initializersrUrprgrmrr)r!rhropsets r"processzMatMulNBitsQuantizer.processns    % %); ;::++-.K,, 0E0EE ++OQ?,, ?8tO_O_OtOtCt#zz668 )FE||'<<QSASp 33ELL"E F  " "; / JJ ) ) +    ' '< 7 KK? @,,99$(OO$;   DJ KK< ="4::.DJ JJ ) ) +  "r$)rdrhrQr[rGr[rWr\rXr]rozlist[str] | Nonerr&rr'rYr\rqzWeightOnlyQuantConfig | None)rhr) r)r*r+__doc__rr3r#ryrrrr,r$r"rjrjs."%)-1 **7;9=',480M0M0M 0M  0M # 0M+0M50M70M!%0M20Md.`*0Sd #r$rjc&|jdvS)N)true1)lower)rs r"ort_convert_str_to_boolrs ;;=M ))r$cD|jd\}}|t|fS)N:)rr[)srrs r"parse_key_value_pairrs!JC E ?r$c tjd}|jddd|jddd|jd d d td |jddtgdd|jddtd|jdd dddt dd gd|jdd td|jddd d !|j d "|jd#d$td gd%&|jd'd$td d()|jd*d+td+d,gd-|jd.td$d/d0gd12|jd3td$d d45|jd6d7}|jd8td9d:;|jd|jd?td d@dAgdBC|jdDtdEdF;|jS)GNa Blockwise int4 quantization for MatMul 2D weight matrices. A weight matrix is partitioned into into blocks, where each block is a continguous subset inside each column. Each block is quantized into a set of 4b integers with a scaling factor and an optional offset. ) descriptionz --input_modelTzPath to the input model file)requiredhelpz--output_modelzPath to the output model filez --block_sizeFrhzBlock size for quantization)rdefaultrrz--quant_methodr)rhqqrtnr?gptqreuWthe algorithm used to quantize weight, rtn and gptq leverage Intel® Neural Compressor)rrchoicesrz--bitsrMz#the target bits to represent weight)rrrz --symmetric?zWIndicate whether to quantize the model symmetrically, symmetric is not supported by hqq)rrconstnargsrrrz--accuracy_levelzAccuracy level of the 4-bit quantized MatMul computation. Refer to the MatMulNBits contrib op's 'accuracy_level' attribute for details (https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits).)rrrz-vz --verbose store_true)raction)rz--nodes_to_exclude+zBSpecify the nodes to be excluded from quantization with node names)rrrrrz--nodes_to_includezKSpecify the specific nodes to be included from quantization with node names)rrrrz--quant_formatr3rzzQuantFormat {QOperator, QDQ}QOperator format quantizes the model with quantized operators directly.QDQ format quantize the model by inserting DeQuantizeLinear before the MatMul.z--op_types_to_quantizerrzPop_types_to_quantize {MatMul, Gather}. Operators to quantize. Default is MatMul.)rrrrz --quant_axeszKey-value pairs in op_type:axis_to_quantize separated by space.Specify the axis to quantize for an op. Default {MatMul:0, Gather:1}Example: --quant_axes MatMul:0 Gather:1)rrrrrez-Arguments specific to nvidia_awq quantizationz--calib_dataset_namerz/Name of the calibration dataset for nvidia_awq.)rrrz--tokenizer_dirzPath of the tokenizer dir.)rrrz--calibration_methodawqawq_clipz]^ Spq  J g  !#    )    # K   4     r$__main__rzfile z already existsrzAsymmetric is not supportted by hqq, will force to symmetric=FalseF)rGrQrrr)rGrWrXrrrrQr)rr?r)rGrrezFQOperator is not applicable to nvidia_awq. overriding the value to QDQrrr)rir~rkr}z!Unsupported quantization method: )rdrQrXrnrorqT)rrrztuple[TensorProto, GraphProto])rr%rrrr])rJr%rKtuplerLrrMr%rNr%rOr%rr)` __future__rrrloggingosr rr numpy.typingtypingnptrXonnx_irr  onnx.onnx_pbrrrronnxruntime.capi._pybind_staterr r r calibrater neural_compressorrr onnx_modelr quant_utilsrr basicConfigINFO getLoggerr)rTrr.r=rArLrTr_rrrWrbrfrrr_rjrrrargsrsetLevelDEBUG input_modelinput_model_path output_modeloutput_model_pathrrrrpathexistsrZ Exception symmetric quant_methodrrlrdrGrQ quant_configrXr3rzr}calib_dataset_namer~rkrrnroquantrsave_model_to_filer,r$r"rs0#   GG-:!8OW^WcWcd   8 $AA@#4#L"7D0% 50%f(4(V/k#8/kds#!6s#l4PPf&     F#&ffR ((V^#^#B*  l^ z