:j@#ddlZddlZddlZddlZddlmZddlZddl m Z m Z m Z m Z ddlmZddlmZddlmZej(eZGddZd Zed k(reZej4rej7ej8ej:Zej>Z ejBjEe r!ejGd e d e$d e d ejJeZ&ee&ejNejPejR Z*e*jWe*jLjYe dyy)N) GraphProto ModelProto NodeProto TensorProto)quantize_matmul_bnb4) ONNXModel)attribute_to_kwargceZdZdZdZdZddededefdZe d e e d e e e ffd Zd ej d ej$fd Zdede e d efdZde e fdZdZy)MatMulBnb4QuantizerzMPerform 4b quantization of constant MatMul weights using FP4 or NF4 data typerrNmodel quant_type block_sizec|xsg}|tjtjfvsJt||_||_||_t||_yN) r FP4NF4r r rrsetnodes_to_exclude)selfr rrrs /home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/onnxruntime/quantization/matmul_bnb4_quantizer.py__init__zMatMulBnb4Quantizer.__init__%sV+1r1557J7N7NOOOOu% $$ #$4 5 graph_pathreturnctt|dz ddD]/}||}|jD]}|j|k(s||fccS1y)Nr)NN)rangelen initializername)r!rgidgraphtensors r__get_initializerz%MatMulBnb4Quantizer.__get_initializer-s\Z1,b"5 )CsOE++ );;$&!5=( ) ) rfpweightc t|jdk7r td|jj }|j\}}||z}|j }||zdz |z}|dzdz}t j|d} t j||j} t| || ||j||| | fS)z4b quantize fp32/fp16 weightz9Current bnb4 block quantization only supports 2D tensors!ruint8)dtype) rshape ValueError transposecopyrnpzerosr*rr) rr& fpweight_trowscolsnumelr num_blocksquantized_numelpackedabsmaxs rbnb4_block_quantz$MatMulBnb4Quantizer.bnb4_block_quant6s x~~ ! #XY Y'')..0 ^^ dt __ j(1,;  19*/9*HNN;VZT__VZ\`arnode graph_stackcJ|jdk7r|Stjd|jd|j|jvr%tjd|jd|S|j d}t j||\}}|tjd|Stjj|}t|jdk7rtjd |S|j|\}}tjj|} |jd z| _|j D].} | j|k(s|j j| ntjj|} |jd z| _|j j#| | gi} |j\} }| | d <|| d <|j$| d<|j&| d<tj(j* d|j d| j| jg|j,dg|jr|jd znddd| }tjd|jd|S)zdIf the node is MatMul with fp32 const weight, quantize the weight with int4, and return the new nodeMatMulzstart to quantize z ...zexclude to quantize z$ as specified by nodes_to_exclude...rz2MatMul doesn't have const weight. Skip to quantizer(z)MatMul weight is not 2D. Skip to quantize_Bnb4_absmaxKNrrr com.microsoft)inputsoutputsr!domainzcomplete quantization of ) MatMulBnb4)op_typeloggerdebugr!rinputr %_MatMulBnb4Quantizer__get_initializeronnx numpy_helperto_arrayrr+r9 from_arrayremover extendrrhelper make_nodeoutput)rr:r;inputBBBs_graphB_arrayr7r8B_quantrK absmax_tensorkwargsr2r3matmul_bnb4_nodes r_bnb4_matmul_node_weightz,MatMulBnb4Quantizer._bnb4_matmul_node_weightLsN <<8 #K )$))D9: 99-- - LL/ {:^_ `KA);;FKP 8 9 LLM NK##,,Q/ w}}  " LLD EK..w7##..v6vv' ^^ EzzV#%%e,  ))44V< VVi/ ##Wm$<=]] ds s #|#|;;00  JJqM7<<1C1CD[[^$(, W$r"      0 4@Arc~g}|d}|jD]}|jDcgc]R}|jtjj k(s'|jtjj k(r|T}}|rVi}|jD]}|jtjj k(r9|j|j|j|j|i}n|jtjj k(rTg} |jD]4} |j| | j|j|g6|j| i}n t|}|j|tjj |j"|j$|j&fd|ji|}|j|j)|||j+d|jj||j-|Scc}w)Nrr!r:)r: attributetyperMAttributeProtoGRAPHGRAPHSappendgr!_process_subgraphgraphsrRr updaterSrTrHrKrUr^ ClearFieldpop) rr; new_nodesr#r:attr graph_attrsr\kvvaluesubgraphs rrgz%MatMulBnb4Quantizer._process_subgraphs BJJ OD!NN99 3 3 9 99TYY$J]J]JdJd=dK  NN&DyyD$7$7$=$==#**4662"ii)?)? )LMd&9&9&@&@@ "(, PH'..x8!LL$*@*@*M)NOP#ii//5MM"%&{{,,LL$**dkk@D MS   T::4M N7 O:   )$ ?sAH:c^|jjg}|jj}d}|D]}|jdk(sd}|s0|j t j jddg|j||jjy)NFrCTr) r r# opset_importrFrRrMrS make_opsetidrgclean_initializers)rr;rs has_ms_domainopsets rprocesszMatMulBnb4Quantizer.processszz'')* zz..0  ! %E||. $  %   !9!9/1!M N O {+ %%'rr)__name__ __module__ __qualname____doc__rrrintr staticmethodlistrtuplerrLnpt ArrayLiker/ndarrayr9rr^rgrxrrr r sW C C6j6c6s6D,<{T^G^A_  2:: ,5 Y5 T*EU5 Zc5 n$T*-=$L (rr ctjd}|jddd|jddd|jd d d tjtj gd |jdd dd|jddd d|j d |jddtd gd|jS)NaBlockwise FP4/NF4 quantization for MatMul 2D weight matrices. A weight matrix is partitioned into blocks, where each block is a contiguous subset inside the flattened transposed weight matrix. Each block is quantized into a set of 4b integers with an absolute value scaling factor. ) descriptionz --input_modelTzPath to the input model file)requiredhelpz--output_modelzPath to the output model filez --quant_typeFrz&Quantization data type. 0: FP4, 1: NF4)rdefaultchoicesrz --block_size@zVBlock size for blockwise quantization. Note: bnb.nn.Linear4bit only uses block_size=64)rrrz-vz --verbose store_true)raction)verbosez--nodes_to_exclude+zBSpecify the nodes to be excluded from quantization with node names)nargsrarrr) argparseArgumentParser add_argumentr rr set_defaultsstr parse_args)parsers rrrs  $ $F $=[\ (4>]^ $((*=*A*AB 5   e   kE,O &   Q     r__main__zfile z already exists)rT)-rloggingosnumpyr/ numpy.typingtypingrrM onnx.onnx_pbrrrronnxruntime.capi._pybind_stater onnx_modelr quant_utilsr getLoggerryrIr rargsrsetLevelDEBUG input_modelinput_model_path output_modeloutput_model_pathpathexistserror Exceptionloadr rrrquantrxsave_model_to_filerrrrs  GG?!+   8 $^(^(B$N z