\;junddlmZddlmZddlZddlmZddlmZgZ d dZ d d Z Gd d eZ y)) defaultdict)reduceN) framework) OptimizercV||\}}n||kr||fn||f\}}||zd||z z||z z z } | dz||zz } | dk\rf| j} ||kr|||z || z| z ||z d| zzz zz } n|||z || z| z ||z d| zzz zz } tt| ||S||zdz S)a]Cubic interpolation between (x1, f1, g1) and (x2, f2, g2). Use two points and their gradient to determine a cubic function and get the minimum point between them in the cubic curve. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp59: formula 3.59 Args: x1, f1, g1: point1's position, value and gradient. x2, f2, g2: point2's position, value and gradient. bounds: bounds of interpolation area Returns: min_pos: the minimum point between the specified points in the cubic curve. rrg@)sqrtminmax) x1f1g1x2f2g2bounds xmin_bound xmax_boundd1 d2_squared2min_poss _G:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/optimizer/lbfgs.py_cubic_interpolaters$!' J-/2X"bB8 J b1R=BG, ,BARIA~ ^^  8BGb2"r'AF:J(KLLGBGb2"r'AF:J(KLLG3w +Z88Z'3..c h|jj} |j}||||\} } d}tj| |}tj d|j |||f\}}}}d}d}|| kr| |||z|zzkDs |dkDr$| |k\r||g}|| g}|| jg}||g}ntj|| |zkr |g}| g}| g}d}n|dk\r||g}|| g}|| jg}||g}nj|d||z zz}|dz}|}t||||| |||f}|}| }| j}|}||||\} } |dz }| j |}|dz }|| kr|| k(r d|g}|| g}|| g}d}d|d krd nd \}}|s|| krtjd|dz | z| krnt|d|dd|d|d|d}d t|t|z z} tt||z |t|z | kr|s|t|k\s|t|krctj|t|z tj|t|z krt|| z }nt|| z}d}nd}nd}||||\} } |dz }| j |}|dz }| |||z|zzkDs| ||k\r5|||<| ||<| j|<|||<|d|dkrd nd \}}nrtj|| |zkrd}n1|||||z zdk\r ||||<||||<|||<||||<|||<| ||<| j|<|||<|s|| kr|}||} |} | | ||fS) ag Implements of line search algorithm that satisfies the strong Wolfe conditions using double zoom. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp60: Algorithm 3.5 (Line Search Algorithm). Args: obj_func: the objective function to minimize. ```` accepts a multivariate input and returns a scalar. xk (Tensor): the starting point of the iterates. alpha (Scalar): the initial step size. d (Tensor): search direction. loss (scalar): the initial loss grad (Tensor): the initial grad c1 (Scalar): parameter for sufficient decrease condition. c2 (Scalar): parameter for curvature condition. tolerance_change (Scalar): terminates if the change of function value/position/parameter between two iterations is smaller than this value. max_ls(int): max iteration of line search. alpha_max (float): max step length. Returns: loss_new (Scaler): loss of obj_func at final alpha. grad_new, (Tensor): derivative of obj_func at final alpha. alpha(Tensor): optimal step length, or 0. if the line search algorithm did not converge. ls_func_evals (Scaler): number of objective function called in line search process. Following summarizes the essentials of the strong Wolfe line search algorithm. Some notations used in the description: - `func` denotes the objective function. - `obi_func` is a function of step size alpha, restricting `obj_func` on a line. obi_func = func(xk + alpha * d), where xk is the position of k'th iterate, d is the line search direction(decent direction), and a is the step size. - alpha : substitute of alpha - a1 is alpha of last iteration, which is alpha_(i-1). - a2 is alpha of current iteration, which is alpha_i. - a_lo is alpha in left position when calls zoom, which is alpha_low. - a_hi is alpha in right position when calls zoom, which is alpha_high. Line Search Algorithm: repeat Compute obi_func(a2) and derphi(a2). 1. If obi_func(a2) > obi_func(0) + c_1 * a2 * obi_func'(0) or [obi_func(a2) >= obi_func(a1) and i > 1], alpha= zoom(a1, a2) and stop; 2. If |obi_func'(a2)| <= -c_2 * obi_func'(0), alpha= a2 and stop; 3. If obi_func'(a2) >= 0, alpha= zoom(a2, a1) and stop; a1 = a2 a2 = min(2 * a2, a2) i = i + 1 end(repeat) zoom(a_lo, a_hi) Algorithm: repeat aj = cubic_interpolation(a_lo, a_hi) Compute obi_func(aj) and derphi(aj). 1. If obi_func(aj) > obi_func(0) + c_1 * aj * obi_func'(0) or obi_func(aj) >= obi_func(a_lo), then a_hi <- aj; 2. 2.1. If |obi_func'(aj)| <= -c_2 * obi_func'(0), then alpha= a2 and stop; 2.2. If obi_func'(aj) * (a2 - a1) >= 0, then a_hi = a_lo a_lo = aj; end(repeat) reference: https://github.com/pytorch/pytorch rrdtypeFTg{Gz? )r)rr)rrg?) absr clonepaddledot to_tensorr rr )!obj_funcxkalphadlossgradgtdc1c2tolerance_changemax_lsd_normloss_newgrad_new ls_func_evalsgtd_newt_prevf_prevg_prevgtd_prevdonels_iterbracket bracket_f bracket_g bracket_gtdmin_stepmax_steptmpinsuf_progresslow_poshigh_posepss! r _strong_wolferI>spUUW[[]F ::222:"      h' !%b%3( ,,q/1 a F f&e*8$ 8$ N"+A,)B-"?VGXw' ::gaj71:- . 7:J J # AJ aL N AJ aL N  "S\CL01 s7|e#US\%9 :S @#g,!6%3w<:O::ec'l23fjjCL(7 L3.EL3.E!&!%"N%b%3( ,,q/1  rEzC// 09W--!&GH "*Ih "*.."2Ih $+K !#A,)A,6F GXzz'"rcCi/GH-0@@AQF$+G$4!&/&8 (#&/&8 (#(3G(< H% %GG !)Ig !)!1Ig #*K Qw'V G E!H!H Xum 33rceZdZdZ d fd ZdZdZdZdZdZ dZ d Z e jd Z d d ZxZS)LBFGSa The L-BFGS is a quasi-Newton method for solving an unconstrained optimization problem over a differentiable function. Closely related is the Newton method for minimization. Consider the iterate update formula: .. math:: x_{k+1} = x_{k} + H_k \nabla{f_k} If :math:`H_k` is the inverse Hessian of :math:`f` at :math:`x_k`, then it's the Newton method. If :math:`H_k` is symmetric and positive definite, used as an approximation of the inverse Hessian, then it's a quasi-Newton. In practice, the approximated Hessians are obtained by only using the gradients, over either whole or part of the search history, the former is BFGS, the latter is L-BFGS. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp179: Algorithm 7.5 (L-BFGS). Args: learning_rate (float, optional): learning rate .The default value is 1. max_iter (int, optional): maximal number of iterations per optimization step. The default value is 20. max_eval (int, optional): maximal number of function evaluations per optimization step. The default value is max_iter * 1.25. tolerance_grad (float, optional): termination tolerance on first order optimality The default value is 1e-5. tolerance_change (float, optional): termination tolerance on function value/parameter changes. The default value is 1e-9. history_size (int, optional): update history size. The default value is 100. line_search_fn (string, optional): either 'strong_wolfe' or None. The default value is strong_wolfe. parameters (list|tuple, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. The default value is None. weight_decay (float|WeightDecayRegularizer, optional): The strategy of regularization. \ It canbe a float value as coeff of L2 regularization or \ :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, \ the regularization setting here in optimizer will be ignored for this parameter. \ Otherwise, the regularization setting here in optimizer will take effect. \ Default None, meaning there is no regularization. grad_clip (GradientClipBase, optional): Gradient cliping strategy, it's an instance of \ some derived class of ``GradientClipBase`` . There are three cliping strategies \ ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , \ :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. Return: loss (Tensor): the final loss of closure. Examples: .. code-block:: python >>> import paddle >>> import numpy as np >>> paddle.disable_static() >>> np.random.seed(0) >>> np_w = np.random.rand(1).astype(np.float32) >>> np_x = np.random.rand(1).astype(np.float32) >>> inputs = [np.random.rand(1).astype(np.float32) for i in range(10)] >>> # y = 2x >>> targets = [2 * x for x in inputs] >>> class Net(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... w = paddle.to_tensor(np_w) ... self.w = paddle.create_parameter(shape=w.shape, dtype=w.dtype, default_initializer=paddle.nn.initializer.Assign(w)) ... ... def forward(self, x): ... return self.w * x ... >>> net = Net() >>> opt = paddle.optimizer.LBFGS(learning_rate=1, max_iter=1, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn='strong_wolfe', parameters=net.parameters()) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print('loss: ', loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... opt.step(closure) ... >>> for input, target in zip(inputs, targets): ... input = paddle.to_tensor(input) ... target = paddle.to_tensor(target) ... train_step(input, target) c ||dzdz}||_||_||_||_||_||_||_t|tjrtdt|ztt|_t|Ad|| | | t|j"dts|j"|_d|_yt'|j(D]\} } | d|_d|_y)Nz^parameters argument given to the optimizer should be an iterable of Tensors or dicts, but got ?) learning_rate parameters weight_decay grad_clipnamerparams)rPmax_itermax_evaltolerance_gradr1 history_sizeline_search_fn isinstancer%Tensor TypeErrortyperdictstatesuper__init___parameter_list_params enumerate _param_groups _numel_cache)selfrPrVrWrXr1rYrZrQrRrSrTidx param_group __class__s rrbzLBFGS.__init__s   !|q(H*    , 0(, j&-- 0<>B:>NO  !&  !%  $..q148//DL !%.d.@.@$A [*84 %B!rcxi}|jjD]\}}|j||id|iS)aReturns the state of the optimizer as a :class:`dict`. Return: state, a dict holding current optimization state. Its content differs between optimizer classes. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... opt.clear_grad() ... loss.backward() ... return loss ... ... opt.step(closure) ... >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> n_iter = 0 >>> while n_iter < 20: ... loss = train_step(inputs, targets) ... n_iter = opt.state_dict()["state"]["func_evals"] ... print("n_iter:", n_iter) ... r`)r`itemsupdate)rh packed_statekvs r state_dictzLBFGS.state_dictsD^ JJ$$&DAq   A ''&&rcl|jtd|jd|_|jS)Nc(||jzSN)numel)totalps rzLBFGS._numel..s!2rr)rgrrd)rhs r_numelz LBFGS._numels4    $ &2DLL!!D    rcg}|jD]a}|j&tj|j dg}n|jj dg}|j |ctj |dS)Nr"r)axis)rdr-r% zeros_likereshapeappendconcat)rhviewsrxviews r_gather_flat_gradzLBFGS._gather_flat_gradsnAvv~((+33RD9vv~~rd+ LL   }}U++rc <d}|jD]v}|jgk7rtd|jnd}tj|j ||||zj |j|z|}||z }x||jk(sJy)Nrc ||zSru)xys rryz!LBFGS._add_grad.. sArr)rdshaperr%assignaddr~rz)rhr* directionoffsetrxrvs r _add_gradzLBFGS._add_grad sA;<77b=F-qww7aE fv~6>>qwwG%O A eOF&&&rc\|jDcgc]}|jc}Scc}wru)rdr$)rhrxs r _clone_paramzLBFGS._clone_params$#'<<0>> import paddle >>> paddle.disable_static() >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print("loss:", loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... >>> opt.step(closure) func_evalsrn_iterrNr+r*old_ykold_skroH_diagprev_flat_grad prev_lossrOrg|=alr" strong_wolfez only 'strong_wolfe' is supportedc,j|||Sru)r)rr*r+rrhs rr(zLBFGS.step..obj_funcs#'#=#= 'E1$r)$r%no_grad enable_gradrPrVrWrXr1rZrYr` setdefaultrrr#r getnegr'r subtractmultiplyr&lenpoprrangerrr$r sum RuntimeErrorrrIr)%rhrrPrVrWrXr1rZrYr` orig_lossr, current_evalsropt_condr+r*rrrrrrrrsysnum_oldrqirbe_ir.r6x_initr(s%`` rstepz LBFGS.step%sR^^ *f((*73G ..M}}H}}H!00N#44 !00N,,LJJE   \1 -   Xq ) I#DM , 1 $ ..0I }}**,>H 7 < #AIIg&EYYx(FYYx(F4BYYx(F"YY'78N +.IF8#! h1$ ?a'! AFFB#--cIF"**>:A 6#3#3E#IJAqBEzv;,6"JJqM"JJqMFF1I a( a( #(+"$aeeAh"&kG5('+f|&;d tB" A"7Q;B7 &q a 02a5 81 aeeF1I"Q%,@&A1E8 #OOAv66A"7^%ay}}Q/"Q%7 aeeF1IA,F&GK,")%.__%6NMM)^< ?a'Cy}}':':'>ti'(+;;!H,X%C8#FE#J"E'N$E(O$E(OE$K$E(O&4E" #!*E+ gjK21ajs2C4W O"W 7WBW "(W W W  W*ctd)z}Empty method. LBFGS optimizer does not use this way to minimize ``loss``. Please refer 'Examples' of LBFGS() above for usage.zeLBFGS optimizer does not use this way to minimize loss. Please refer 'Examples' of LBFGS() for usage.)NotImplementedError)rhr,startup_programrQ no_grad_sets rminimizezLBFGS.minimizes" s  r) rONgHz>& .>dNNNNN)NNN)__name__ __module__ __qualname____doc__rbrrrzrrrrrrnon_static_onlyrr __classcell__)rks@rrKrK5sXx /!b3'j!, '1$]]@HL rrKru)g-C6?g?r) collectionsr functoolsrr%baser optimizerr__all__rrIrKrrrrsD$  !/X   t4nV IV r