:j(J@dZddlmZddlmZddlmZddlZddlm Z ddl m Z ddl m Z erdd lmZddlZddlZndd lmZed Zed Ze eZdd ZGddej.j0ZGddZ d ddZy)aNotations in this Gaussian process implementation X_train: Observed parameter values with the shape of (len(trials), len(params)). y_train: Observed objective values with the shape of (len(trials), ). x: (Possibly batched) parameter value(s) to evaluate with the shape of (..., len(params)). cov_fX_fX: Kernel matrix X = V[f(X)] with the shape of (len(trials), len(trials)). cov_fx_fX: Kernel matrix Cov[f(x), f(X)] with the shape of (..., len(trials)). cov_fx_fx: Kernel scalar value x = V[f(x)]. This value is constant for the Matern 5/2 kernel. cov_Y_Y_inv: The inverse of the covariance matrix (V[f(X) + noise_var])^-1 with the shape of (len(trials), len(trials)). cov_Y_Y_inv_Y: `cov_Y_Y_inv @ y` with the shape of (len(trials), ). max_Y: The maximum of Y (Note that we transform the objective values such that it is maximized.) sqd: The squared differences of each dimension between two points. is_categorical: A boolean array with the shape of (len(params), ). If is_categorical[i] is True, the i-th parameter is categorical. ) annotations)Any) TYPE_CHECKINGN)*single_blas_thread_if_scipy_v1_15_or_newer) optuna_warn) get_logger)Callable) _LazyImportscipytorchctj|}tj|r|Stdtj|d}tj |tj |tjtj ||tjddtj |tjtj ||tj ddS)NzDClip non-finite values to the min/max finite values for GP fittings.r)axis) npisfiniteallranyclipwheremininfmax)valuesis_values_finite is_any_finites b/home/mcse/projects/srt_converter/srt-converter-venv/lib/python3.12/site-packages/optuna/_gp/gp.pywarn_and_convert_infr.s{{6* vv VWFF+!4M 77 rxx0@&"&&'QXY Z\_` rxx0@&266''RYZ []`a c0eZdZeddZeddZy)Matern52Kernelctjd|z}tj| }|d|z|zdzz}d|dzz|z}|j||S)a This method calculates `exp(-sqrt5d) * (1/3 * sqrt5d ** 2 + sqrt5d + 1)` where `sqrt5d = sqrt(5 * squared_distance)`. Please note that automatic differentiation by PyTorch does not work well at `squared_distance = 0` due to zero division, so we manually save the derivative, i.e., `-5/6 * (1 + sqrt5d) * exp(-sqrt5d)`, for the exact derivative calculation. Notice that the derivative of this function is taken w.r.t. d**2, but not w.r.t. d. g?g)r sqrtexpsave_for_backward)ctxsquared_distancesqrt5dexp_partvalderivs rforwardzMatern52Kernel.forward?shA 00199fW%5$44v=ABFQJ'(2 e$ rc(|j\}||zS)z Let x be squared_distance, f(x) be forward(ctx, x), and g(f) be a provided function, then deriv := df/dx, grad := dg/df, and deriv * grad = df/dx * dg/df = dg/dx. ) saved_tensors)r'gradr,s rbackwardzMatern52Kernel.backwardRs $$t|rN)r'rr( torch.Tensorreturnr2)r'rr0r2r3r2)__name__ __module__ __qualname__ staticmethodr-r1rrr r >s($rr ceZdZ d dZed dZd dZd dZ d ddZdddZ ddZ dd Z y) GPRegressorc||_||_||_||_||_|j d|j dz j |_|jjrT|jd|jfdkDjtj|jd|jf<d|_ d|_ ||_||_||_y)N.r)_is_categorical_X_train_y_train_X_all_y_all unsqueezesquare__squared_X_diffrtyper float64 _cov_Y_Y_chol_cov_Y_Y_inv_Yinverse_squared_lengthscales kernel_scale noise_var)selfis_categoricalX_trainy_trainrJrKrLs r__init__zGPRegressor.__init__]s .    ' 1 1" 58I8I"8M MVVX    # # %$$S$*>*>%>?#Ed5==!  d&:&:!: ;3737,H)("rcdtj|jjj j z S)Ng?)rr$rJdetachcpunumpy)rMs r length_scaleszGPRegressor.length_scalesws7RWWT>>EEGKKMSSUVVVrcR|j |jJdtj5|j j j j}dddtj|jjdxx|jjz cc<tjj|}t jj#|j$t jj#||j&j jdd}tj(||_tj(||_|j*j |_d|j*_|j.j |_d|j._|jj |_ d|j_y#1swYxYw)Nz(Cannot call cache_matrix more than once.rTlowerF)rHrIr no_gradkernelrSrTrUr diag_indicesr?shaperLitemlinalgcholeskyr solve_triangularTr@ from_numpyrJr0rK)rMcov_Y_Y cov_Y_Y_chol cov_Y_Y_inv_Ys r _cache_matrixzGPRegressor._cache_matrix{s!!)d.A.A.I 6 I]]_ ;kkm**,00288:G ;  3 3A 678DNN$..BUBUBW W > Xll++    " " $ * * ,.B.D.DD, ! ,.99+=+=>TWZ]`]b]bWb>b+c WXwx'(+. WXxx'(ii :B  55 NN LL ) ), 8I8O8O8QY] ) ^6 #--l;#..}=ii :B 1 X Xs +B0K99LNc||J|j}n| |j}|jdk(r||z n"|jd|jdz j }|j j r@|d|j fdkDjtj|d|j f<|j|j}tj||jzS)am Return the kernel matrix with the shape of (..., n_A, n_B) given X1 and X2 each with the shapes of (..., n_A, len(params)) and (..., n_B, len(params)). If x1 and x2 have the shape of (len(params), ), kernel(x1, x2) is computed as: kernel_scale * Matern52Kernel.apply( sqd(x1, x2) @ inverse_squared_lengthscales ) where if x1[i] is continuous, sqd(x1, x2)[i] = (x1[i] - x2[i]) ** 2 and if x1[i] is categorical, sqd(x1, x2)[i] = int(x1[i] != x2[i]). Note that the distance for categorical parameters is the Hamming distance. r#r<r=.r)rEr?ndimrCrDr>rrFr rGmatmulrJr applyrK)rMX1X2sqdsqdists rr[zGPRegressor.kernels :: :&&Cz]] ggl27 R0@2<##F+d.?.???rc p|j |jJd|jdk(}|s|n|jd}tj j |j||jx}|j}tj j|jtj j|jj|dddd}|rb|rJd|j||}||j|jdd z } | jd d jd n@|j}|tj j ||z } | jd |r"|j!d| j!dfS|| fS) a) This method computes the posterior mean and variance given the points `x` where both mean and variance tensors will have the shape of x.shape[:-1]. If ``joint=True``, the joint posterior will be computed. The posterior mean and variance are computed as: mean = cov_fx_fX @ inv(cov_fX_fX + noise_var * I) @ y, and var = cov_fx_fx - cov_fx_fX @ inv(cov_fX_fX + noise_var * I) @ cov_fx_fX.T. Please note that we clamp the variance to avoid negative values due to numerical errors. z+Call cache_matrix before calling posterior.r#rTF)upperleftz3Call posterior with joint=False for a single point.r<)dim1dim2r)rHrIrxrCr r_vecdotr[rArarbry transposediagonal clamp_min_rKsqueeze) rMxjointis_single_pointx_ cov_fx_fXmeanV cov_fx_fxvar_s r posteriorzGPRegressor.posteriors!!-$2E2E2Q 9 Q&&A+%Q1;;q>||"" B 0L#L9dNaNab LL ) )    LL ) )$*<*<*>*> QU\a ) b *  & ](] ]& B+Iqxx (;(;B(CDDD MMrM + 6 6s ;))Iu||229a@@D OOC 5D Qa1V4QU,Vrc|j}|jj|jtj j |}|jjj }tj j||jdddfddddf}d||zz}||zS)a This method computes the marginal log-likelihood of the kernel hyperparameters given the training dataset (X, y). Assume that N = len(X) in this method. Mathematically, the closed form is given as: -0.5 * log((2*pi)**N * det(C)) - 0.5 * y.T @ inv(C) @ y = -0.5 * log(det(C)) - 0.5 * y.T @ inv(C) @ y + const, where C = cov_Y_Y = cov_fX_fX + noise_var * I and inv(...) is the inverse operator. We exploit the full advantages of the Cholesky decomposition (C = L @ L.T) in this method: 1. The determinant of a lower triangular matrix is the diagonal product, which can be computed with N flops where log(det(C)) = log(det(L.T @ L)) = 2 * log(det(L)). 2. Solving linear system L @ u = y, which yields u = inv(L) @ y, costs N**2 flops. Note that given `u = inv(L) @ y` and `inv(C) = inv(L @ L.T) = inv(L).T @ inv(L)`, y.T @ inv(C) @ y is calculated as (inv(L) @ y) @ (inv(L) @ y). In principle, we could invert the matrix C first, but in this case, it costs: 1. 1/3*N**3 flops for the determinant of inv(C). 2. 2*N**2-N flops to solve C @ alpha = y, which is alpha = inv(C) @ y. Since the Cholesky decomposition costs 1/3*N**3 flops and the matrix inversion costs 2/3*N**3 flops, the overall cost for the former is 1/3*N**3+N**2+N flops and that for the latter is N**3+2*N**2-N flops. NF)rrg) r[radd_rLr r_r`logsumrar@)rMrdL logdet_partinv_L_y quad_parts rmarginal_log_likelihoodz#GPRegressor.marginal_log_likelihoods4++-/ LL ! !' *zz|'')--// ,,//4==D3IQV/WXY[\X\]Gg-. Y&&rc  jjd tjtjj j jjtjjjtjjjdzz gg}d  fd }t5tjj||ddd|i}dddj st#d|j$t'j(|j*}t'j,|d _t'j,| _ r%t'j.t&j0 nt'j,| dzz_ j3S#1swYxYw) Nr#gGz?ctj|jd}tj5tj|d_tj|_r%tjtjntj|dzz_ j z }|j|jdz}r|dk(sJdddj|jjjj!fS#1swYOxYw)NTrir#r)r rcrequires_grad_ enable_gradr%rJrKtensorrGrLrr1r0r^rSrTrU) raw_paramsraw_params_tensorlossraw_noise_var_graddeterministic_objective log_prior minimum_noisen_paramsrMs r loss_funcz1GPRegressor._fit_kernel_params..loss_func;s1 % 0 0 < K KD Q ""$ N49II>OPYQY>Z4[1$)II.?.I$J!/LLemmD#4X\#BCmS 44664H %6%;%;HqL%I"26HA6MMM N99; 1 6 6 = = ? C C E K K MM M N Ns C EETzl-bfgs-bgtol)jacmethodoptionszOptimization failed: ri)r np.ndarrayr3ztuple[float, np.ndarray])r?r]r concatenaterrJrSrTrUrKr^rLrr optimizeminimizesuccess RuntimeErrormessager rcrr%rrGrg) rMrrrrinitial_raw_paramsrresraw_params_opt_tensorrs ```` @r_fit_kernel_paramszGPRegressor._fit_kernel_params"s==&&q) ^^t88??AEEGMMOPFF4,,1134FF4>>..04-3GGH    N N"8 9 ..))"! *C {{!6s{{mDE E % 0 0 7,1II6KIX6V,W)!II&;H&EF' LLemm <+@A+N!OO   -  s /'G77H)rNr2rOr2rPr2rJr2rKr2rLr2r3None)r3r)r3r)rnr2ror2r3r)NN)r{torch.Tensor | Noner|rr3r2)F)rr2rboolr3z!tuple[torch.Tensor, torch.Tensor])r3r2) r%Callable[[GPRegressor], torch.Tensor]rfloatrrrrr3r:) r4r5r6rQpropertyrVrgrvr[rrrr8rrr:r:\s#$## # '3 # # # # #4WW#6#CLIM@%@2E@ @<#WJ"'H@8@@"& @  @  @rr:c Jtjjddztjd fd }|} ||}d} || fD]} t tj tj tj | j | j| jj||||cStjd| d|} | j| S#t$r } | } Yd} ~ d} ~ wwxYw) Nr#ric ttjtjtjddjdjdjS)Nr<rrNrOrPrJrKrL)r:r rcclone)XYdefault_kernel_paramsrNsr _default_gprz'fit_kernel_params.._default_gprrso ++N;$$Q'$$Q')>s)C)I)I)K.r288:+B/557   rr)rrrrz/The optimization of kernel parameters failed: z< The default initial kernel parameters will be used instead.)r3r:)r onesr]rGr:rcrJrKrLrrloggerwarningrg)rrrNrrr gpr_cacherrdefault_gpr_cacheerrorgpr_cache_to_usee default_gprrs``` @rfit_kernel_paramsres-"JJqwwqzA~U]]K  % N E'(9: $//?((+((+-=-Z-Z-::*44 ! #+(? ! $ NN :5'BF F.K  E sA:D  D"DD")rrr3r)Ng{Gz?)rrrrrNrrrrrrrrzGPRegressor | Nonerrr3r:)__doc__ __future__rtypingrrrUr"optuna._gp.scipy_blas_thread_patchroptuna._warningsroptuna.loggingrcollections.abcr r r optuna._importsr r4rrautogradFunctionr r:rr8rrrs&# Y(%(+  E  E H   U^^,,