:jF~ZddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZmZddlmZddlmZddlZddlmZmZddlmZmZmZdd lmZdd lmZm Z dd l!m"Z"m#Z# ddl$Z$ejLd k7rddl'Z'e"rdd l(m)Z)ddl*m+Z+ddl,m-Z-e#r ddl.Z.ddlm/Z/m0Z0e1ejdejfzZ4e1ejdejjzejfzdzZ6ddiddiddiddiddiddiddidZ7dZ8ddjse7judZ;Gdd ZGd%d&eZ?e@d'k(r.e=ZAd(eA_Bd)eA_Be?eAZCeCjyy#e%$rY5wxYw)*N)ArgumentParser Namespace) AsyncIterator) dataclassfield)Thread)Optional)AsyncInferenceClientChatCompletionStreamOutput) AutoTokenizerGenerationConfigPreTrainedTokenizer)BaseTransformersCLICommand)ServeArguments ServeCommand)is_rich_availableis_torch_availableWindows)Console)Live)Markdown)AutoModelForCausalLMBitsAndBytesConfigz .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?z$Which number is larger, 9.9 or 9.11?)llamacode helicopternumbersbirdssocksnumbers2a **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands (set generation settings, save chat, etc.) - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface ceZdZddeedeefdZdeedeee ffdZ defdZ d Z d efd Z d ed efd ZddefdZdededefdZy) RichInterfaceN model_name user_namecft|_|d|_n||_|d|_y||_y)N assistantuser)r_consoler$r%)selfr$r%s o/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/transformers/commands/chat.py__init__zRichInterface.__init__ws3   )DO(DO  #DN&DNstreamreturncK|jjd|jdt|jd5}d}|d{23d{}|jdj j }|s/tjdd|}||z }g}|jD]G}|j||jd r|jd 7|jd Itdj|jd }|j|d776 dddn #1swYnxYw|jjSw)Nz [bold blue]:)consolerefresh_per_secondrz <(/*)(\w*)>z\<\1\2\>z``` z z github-dark) code_themeT)refresh)r)printr$rchoicesdeltacontentresub splitlinesappend startswithrjoinstripupdate) r*r.livertokenoutputslineslinemarkdowns r+ stream_outputzRichInterface.stream_outputsE l4??*;2>? $--A >& 4$D%+|$ 4$ 4e--*0088&&gF OO--DLL&u- T* V,-$BGGEN$8$8$:}U Hd 3I ,$ 4|& 4& 4& 4P  sOAE,E D5 ED9D7D9CE7D99E; E,E  #E,c|jjd|jd}|jj|S)z!Gets user input from the console. [bold red]: )r)inputr%r8)r*rMs r+rMzRichInterface.inputs9 ##k$..1A$FG  r-c8|jjy)zClears the console.N)r)clearr*s r+rOzRichInterface.clears r-rc|jjd|jd||jjy)z%Prints a user message to the console.rLz>:[/ bold red] N)r)r8r%)r*rs r+print_user_messagez RichInterface.print_user_messages7 k$..)99I$PQ r-colorcz|jjd|d||jjy)z,Prints text in a given color to the console.z[bold ]Nr)r8)r*rrSs r+ print_colorzRichInterface.print_colors1 fUG1TF34 r-minimalc|jjt|rtnt|jjy)z'Prints the help message to the console.N)r)r8rHELP_STRING_MINIMAL HELP_STRING)r*rXs r+ print_helpzRichInterface.print_helps. HG%8UV r-generation_config model_kwargsc|jjd|d|r|jjd||jjd||jjy)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r5z[bold blue]Model kwargs: z [bold blue]NrV)r*r$r]r^s r+ print_statuszRichInterface.print_statussf 0 B?@  MM  ";L> J K k*;)<=> r-)NN)F)__name__ __module__ __qualname__r strr,rr tupleintrJrMrOrRrWboolr\r dictr`r-r+r#r#vs '8C= 'HSM ',-8R*S,X]^acf^fXg,\s s C $ s?O_cr-r#ceZdZUdZedddiZeeed<edddiZ eeed<eddd iZ eeed <ed dd iZ eed <edddiZ eeed<edddiZ eed<edddiZeeed<edddiZeed<edddiZeed<eddgddZeeed<eddgddZeeed <eddd!iZeed"<eddd#iZeeed$<eddd%iZeed&<eddd'iZeed(<ed)d*d+d)gdZeed,<eddd-iZeed.<ed/dd0iZeed1<ed2dd3iZeed4<d5Zy)6 ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r(zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathFz7Whether to show runtime warnings in the chat interface.verbosezPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.r]mainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisionautozDevice to use for inference.devicezA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rvbfloat16float16float32)rlr9 torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz%Interface the server will listen to..host@zPort the server will listen to.portc|j^|j|j|_y|j|jk7r&td|jd|jdyy)z(Only used for BC `torch_dtype` argument.Nz`torch_dtype` z and `dtype` zn have different values. `torch_dtype` is deprecated and will be removed in 4.59.0, please set `dtype` instead.)r{r| ValueErrorrPs r+ __post_init__zChatArguments.__post_init__%su    'zz!!-- !!TZZ/ $T%5%5$6mDJJ<PMM0 (r-)rarbrc__doc__rror rd__annotations__r(rprqrrrsrgr]rurwr{r|r}r~rrrrrrrfrrir-r+rkrksV). u )   ghD(3-$)IY@Z#[M8C=[%6&JhAijKj#(Im@n#oM8C=o%6;t2uvGTv(- 1  (x}  hiNC&:X1YZFCZ!&WA "K#!PA E8C=$)] ^t*/ r *#efL$efL$ %UFZhmotgu=vww!&uHm?n!o$okV=d4efD#fdf6W-XYD#Y r-rkargsct|S)z; Factory function used to chat with a local model. ) ChatCommand)rs r+chat_command_factoryr2s t r-ceZdZedefdZdZedefdZed!de de edefd Z ed!d e ede e fd Zd e ede fd Zde dedeee ffdZededede ede edeee eff dZede de efdZde dedeffdZdede dede ee eeffdede de e dee e ee ffdZdZd Zy)"rparserctf}|jd|}|jd}|jdtdd|jdtdd d |j t y) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_or_addressNz7Name of the pre-trained model or address to connect to.)typermrlgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rrmrlnargs)func)rk add_parseradd_argument_group add_argumentrd set_defaultsr)rr chat_parsergroups r+register_subcommandzChatCommand.register_subcommand:s)*'''P ../EF  +J   W      &: ;r-cz|j|j}|jds"|jds|jdrpd|_|jdk7s|jdk7r t d|jj dd\|_|_|j#t d d |_|j|_ts!ts|jr td ts td ts|jr td ||_ y)NhttphttpsrFruuLooks like you’ve set both a server address and a custom host/port. Please pick just one way to specify the server.:z\When connecting to a server, please specify a model name with the --model_name_or_path flag.TzYou need to install rich to use the chat interface. Additionally, you have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)zHYou need to install rich to use the chat interface. (`pip install rich`)zYou have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)) rr@ spawn_backendrrrrsplitrorr ImportErrorr)r*rnames r+r,zChatCommand.__init__[s+  - - 955Dv&$//'*BdooVaFb%*"99 +tyyD/@$J (,'I'I'P'PQTVW'X$ 49**2$v&*"*.*L*L' ",>,@TEWEWx #$hi i#%$*<*<7   r-r/ctjdk(rtjSt j tj jS)z)Returns the username of the current user.r)platformsystemosgetloginpwdgetpwuidgetuidpw_namerir-r+ get_usernamezChatCommand.get_usernames: ??  );;= << ,44 4r-Nrfilenameci}t||d<||d<|j}|Gtjd}|jd|d}t j j||}t jt j j|dt|d 5}tj||d dddt j j|S#1swY(xYw) z!Saves the chat history to a file.settings chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr1)indent)varsrqtimestrftimerrpathrAmakedirsdirnameopenjsondumpabspath)rrr output_dictfoldertime_strfs r+ save_chatzChatCommand.save_chats "&t* J&* N#!!  }}%89H<<=VH:USHww||FH5H BGGOOH-= (C  0A IIk1Q / 0wwx(( 0 0s +C++C4rpc|g}|Sd|dg}|S)zClears the chat history.rroler;ri)rprs r+clear_chat_historyzChatCommand.clear_chat_historys*  D &-@AD r-rc t|dk(riS|Dcic]/}d|jddzdz|jdd1}}|jDcic]*\}}||jdvr|jn|,}}}|jDcic]\}}||dk(rdn|}}}dtd t fd }|jDcic]\}}|||sd|dn|}}}d j |jDcgc] \}}|d |c}}}d |zdz}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd} tj|}|Scc}wcc}}wcc}}wcc}}wcc}}w#tj$r tdwxYw)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.r"=r)truefalseNonenullsr/cf|jd}|jdddjS)N-.r4r) removeprefixreplaceisdigit)rs r+ is_numberz3ChatCommand.parse_generate_flags..is_numbers,s#A99S"a(002 2r-z, z: {}z"null"z"true"rz"false"rz"[[z]"rUrzFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lensplititemslowerrdrgrArrloadsJSONDecodeErrorr) r*rflaggenerate_flags_as_dictkvrgenerate_flags_stringprocessed_generate_flagss r+parse_generate_flagsz ChatCommand.parse_generate_flagss@ ~ ! #I bp!pY]# 3(:":S"@$**S/RSBT"T!p!p OeNjNjNl" FJaAAGGI)::qwwy A" " OeNjNjNl!mda!qF{V"A!m!m 3 3 3 VlUqUqUs!tTQPQ!Yq\q1Xq"H!t!t !% CYC_C_Ca*b41aaS1#;*b c!$&; ;c A!6 = =h O 5 = =h O 5 = =i Q 5 = =dC H 5 = =dC H!6 = =c3 G '+zz2G'H $('W"q" "n"u +c"## D  s)4F>/G#G GG 'GG:model_generation_configc|jd|jvritjj|j}tjj |j}t j ||}nJt j |j}n*tj|}|jdiddd|j|j}|jdi|}||fS)zj Returns a GenerationConfig object holding the generation parameters for the CLI command. rT) do_samplemax_new_tokensri) r]rrrbasenamer from_pretrainedcopydeepcopyrCrr)r*rrrrr]parsed_generate_flagsr^s r+get_generation_parameterizationz+ChatCommand.get_generation_parameterizations  ! ! -$000''//$*@*@A77++D,B,BC$4$D$DWh$W!$4$D$DTE[E[$\! !% .E F  $  $ $ RTS'Q R!% 9 9$:M:M N/(//H2GH !,..r- tokenizerr] eos_tokens eos_token_idsc|j |j}n |j}g}|/|j|j|j d|7|j|j dDcgc] }t |c}t |dk(r|j|j||fScc}w)z:Retrieves the pad token ID and all possible EOS token IDs.,r) pad_token_id eos_token_idextendconvert_tokens_to_idsrrfrr?)rr]rrrall_eos_token_idstoken_ids r+parse_eos_tokenszChatCommand.parse_eos_tokenss  ) ) 1,99L,99L  !  $ $Y%D%DZEUEUVYEZ%[ \  $  $ $MDWDWX[D\%]c(m%] ^  !Q &  $ $%6%C%C D... &^s9B? model_argsc|jr:td|j|j|j|j}|S|j rtd}|Sd}|S)NT)rbnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)r)rrr|rrr)rquantization_configs r+get_quantization_configz#ChatCommand.get_quantization_configst  " ""4!'1'7'7$.$B$B*4*I*I'1'7'7 # #" $ $"4!#  #"#' ""r-rctj|j|j|j}|j dvr |j nt t|j }|j|}|j|j|d|d}tj|jfd|ji|}t |dd|j|j}||fS)N)revisionr})rvNrv)rr~r| device_mapr r} hf_device_map) r rmodel_name_or_path_positionalrur}r|getattrtorchrr~rtorw)r*rrr|r r^models r+load_model_and_tokenizerz$ChatCommand.load_model_and_tokenizer)s!11  . .(("44 #jjN: tzz@Z"::4@++#'#;#; #6  %44  . . BFBXBX \h  5/4 0 8HHT[[)Eir- user_input interfaceexamplesr^rcd}|dk(r-|j|j}|jn/|dk(r|jn|j drnt |j dkrR|j } t | dk(r| d} nd} |j||| } |jd| d d n|j d r|d dj} | j } | D] } d| vs|jd| dd nA|j| } |jdi| }|jdi|n |j drt |j dk(r|j d}||vrD|jg}|j||d|jd||ddnd|dt|jd}|j|d nM|dk(r|j!|j"||n)d}|jd|dd |j||||fS)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). Tz!clearz!helpz!saverNzChat saved in !green)rrSz!setr1rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr(rzExample z* not found in list of available examples: rz!status)r$r]r^F'z/' is not a valid command. Showing help message.ri)rrprOr\r@rrrrWrBrrCrRr?listkeysr`ro)r*rrrrr]r^r valid_command split_inputrnew_generate_flagsrparsed_new_generate_flagsnew_model_kwargs example_name example_errors r+handle_non_exit_user_commandsz)ChatCommand.handle_non_exit_user_commandsDs  !**4+=+=>D OO  7 "  "  " "7 +J4D4D4F0G!0K$**,K;1$&q>~~dD(;H  ! !z'C7 ! S  " "6 *",AB!5!5!7 !3!9!9!; * 8d?))FtfMAA$ * 8-1,E,EFX,Y)#;#4#;#;#X>W#X # ##7&67  " ": .3z7G7G7I3Ja3O%++-a0Lx'!,,Xl-CF-KL V8Nv8VWX|n,VW[\d\i\i\kWlVmmno%%=%F 9 $  " "22"3) # "M  ! !:,6e'fns ! t  "]$5|CCr-cJtj|jyN)asynciorun _inner_runrPs r+r0zChatCommand.runs DOO%&r-c K|jrt|jj|jj|jj |jj |jj|jj|jj|jj|jj|jjd }t|}t|j}d|_|j#|jj$dz|jj&z}|jjdk(rdn|jj}|j}|j(t*}n3t-|j(5}t/j0|}ddd|j2|j5} n |j2} t7j8|j$} |j;|| \} } t=|j$| } | j?|jA|jB}| jEd tG|d |jj4d{} | jI}|jKd r7|d k(rn|jM||| | | | \}}} } |r|jKdsZ|jOd|d|jQ|d| jS|d}| jU|d{}|jOd|ddddd{y#1swYxYw779#tV$rY-wxYw7%#1d{7swYyxYww)Nerror) rwr|r}r~rrrrrr log_level)targetT@rzhttp://localhost)r$r%)rXrrz!exit)rrrrr]r^rr!r(r)r]r)r. extra_bodyr'),rrrrwr|r}r~rrrrrrrrr0daemonstartrorurrDEFAULT_EXAMPLESryaml safe_loadr(rr rrr#rOrrpr\r rMr@r,r?chat_completionto_json_stringrJKeyboardInterrupt)r* serve_args serve_commandthreadrrrrrr(rr]r^rrclientrr%r. model_outputs r+r1zChatCommand._inner_runsA   'yy''iioo"&))"="=$(II$A$A!YY33!YY33$(II$A$A%)YY%C%CYY^^YY^^! J)4M=#4#45F FM LLN ,,s2TYY5M5MM%)YY^^{%B! yy    %'Hd(() -Q>>!, - 99 $$&D99D"2"B"B4CZCZ"[*.*N*NtUl*m'<!T-D-DPTU &&t'9'9: T*'4&$))..1A(BC' ' v%!*!2J",,S1%0!SWSuSu+5%)*3)12C-9%)TvTPD-1BL -J4I4I*4U$ V $KL#33#1B1Q1Q1S%*$4F*3)@)@)H#HLKK NOI' ' ' % - -$' F$I)M' ' ' ' sF9O;N C!O2N3O6N+9&NN+ 0NN+A NNN7N+8 ON)O N ON N&#N+%N&&N+)O+N=1N4 2N=9Or.)rarbrc staticmethodrrr,rdrrkr rr#rhrrr rerrrfrrrr rr#r,r0r1rir-r+rr9s Z##& ] uE[]jEj?k 6PDPDPD! PD sDcN*+ PD , PDPD4jPD tDz+T1 2PDh'Ur-r__main__z meta-llama/Llama-3.2-3b-Instructzhttp://localhost:8000)Er/rrrrr<stringrargparserrcollections.abcr dataclassesrr threadingrtypingr r;huggingface_hubr r transformersr r rtransformers.commandsrtransformers.commands.servingrrtransformers.utilsrrreadlinerrr rich.consoler rich.liver rich.markdownrrrrset ascii_letters whitespaceALLOWED_KEY_CHARSdigitsALLOWED_VALUE_CHARSr:rZrAr$r[r#rkrrrarrrr0rir-r+r\s   .)( L =FD  8?? !$& ,,v/@/@@A 6==(6+<+<?@A &YYx \\ \~yw,wt  z ?D)KD&)@D& t DHHJ M  s0F!!F*)F*