\;j<:fddlZddlmZddlmZmZddlmZddlm Z ddl m Z gZ Gd d e Z y) N)_C_ops)core framework)no_grad) name_scope) Optimizerc^eZdZdZdZdZdZ d fd ZdZdZ dZ d Z d Z xZ S) Adamaxa The Adamax optimizer is implemented based on the Adamax Optimization in Section 7 of `Adam paper `_. The Adamax algorithm is a variant of the Adam algorithm based on the infinite norm, which makes the learning rate update algorithm more stable and simple. The parameter ``param_out`` update rule with gradient ``grad``: .. math:: t & = t + 1 moment\_out & = {\beta}_1 * moment + (1 - {\beta}_1) * grad inf\_norm\_out & = max({\beta}_2 * inf\_norm + \epsilon, |grad|) learning\_rate & = \frac{learning\_rate}{1 - {\beta}_1^t} param\_out & = param - learning\_rate * \frac{moment\_out}{inf\_norm\_out} Related paper: `Adam: A Method for Stochastic Optimization `_ The original paper does not have an ``epsilon`` attribute, it is added here for numerical stability to prevent the division by 0 error. Args: learning_rate (float|LRScheduler, optional): The learning rate used to update ``Parameter``. It can be a float value or a LRScheduler. The default value is 0.001. beta1 (float, optional): The exponential decay rate for the 1st moment estimates. The default value is 0.9. beta2 (float, optional): The exponential decay rate for the 2nd moment estimates. The default value is 0.999. epsilon (float, optional): A small float value for numerical stability. The default value is 1e-08. parameters (list|tuple, optional): List/Tuple of ``Tensor`` to update to minimize ``loss``. This parameter is required in dygraph mode. And you can specify different options for different parameter groups such as the learning rate, weight decay, etc, then the parameters are list of dict. Note that the learning_rate in parameter groups represents the scale of base learning_rate. The default value is None in static graph mode, at this time all parameters will be updated. weight_decay (float|WeightDecayRegularizer, optional): The strategy of regularization. It can be a float value as coeff of L2 regularization or :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, the regularization setting here in optimizer will be ignored for this parameter. Otherwise, the regularization setting here in optimizer will take effect. Default None, meaning there is no regularization. grad_clip (GradientClipBase, optional): Gradient clipping strategy, it's an instance of some derived class of ``GradientClipBase`` . There are three clipping strategies ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. **Notes**: **Currently, Adamax doesn't support sparse parameter optimization.** Examples: .. code-block:: python >>> import paddle >>> inp = paddle.uniform([10, 10], dtype="float32", min=-0.1, max=0.1) >>> linear = paddle.nn.Linear(10, 10) >>> inp = paddle.to_tensor(inp) >>> out = linear(inp) >>> loss = paddle.mean(out) >>> beta1 = paddle.to_tensor([0.9], dtype="float32") >>> beta2 = paddle.to_tensor([0.99], dtype="float32") >>> adam = paddle.optimizer.Adamax(learning_rate=0.1, ... parameters=linear.parameters(), ... beta1=beta1, ... beta2=beta2, ... weight_decay=0.01 ... ) >>> out.backward() >>> adam.step() >>> adam.clear_grad() >>> # Note that the learning_rate of linear_2 is 0.01. >>> linear_1 = paddle.nn.Linear(10, 10) >>> linear_2 = paddle.nn.Linear(10, 10) >>> inp = paddle.uniform(shape=[10, 10], min=-0.1, max=0.1) >>> out = linear_1(inp) >>> out = linear_2(out) >>> loss = paddle.mean(out) >>> adam = paddle.optimizer.Adamax( ... learning_rate=0.1, ... parameters=[{ ... 'params': linear_1.parameters() ... }, { ... 'params': linear_2.parameters(), ... 'weight_decay': 0.001, ... 'learning_rate': 0.1, ... 'beta1': 0.8 ... }], ... weight_decay=0.01, ... beta1=0.9 ... ) >>> out.backward() >>> adam.step() >>> adam.clear_grad() momentinf_norm beta1_pow_accc `|J|J|J|Jd|cxkrdkstdtdd|cxkrdkstdtdd|ks tdt | |||||d|_||_||_||_d|_i|_|||d |_ y) Nrr z.Invaild value of beta1, expect beta1 in [0,1).z.Invaild value of beta2, expect beta2 in [0,1).z.Invaild value of epsilon, expect epsilon >= 0.) learning_rate parameters weight_decay grad_clipnameadamaxF)beta1beta2epsilon) ValueErrorsuper__init__type_beta1_beta2_epsilon_multi_precision_master_weights _default_dict) selfrrrrrrrr __class__s `G:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/optimizer/adamax.pyrzAdamax.__init__s(((      """E~A~MN NMN NE~A~MN NMN NG|MN N '!%      %! cR|j}|j|r$tjjj }|j |j|||j |j|||j |j||jdgy)N)dtyper )rparam fill_valueshape) r)_is_dtype_fp16_or_bf16rVarDescVarTypeFP32_add_accumulator_moment_acc_str_inf_norm_acc_str_beta1_pow_acc_strr)r$p acc_dtypes r&_add_moments_powszAdamax._add_moments_powssGG  & &y 1 ,,11I d22AYG d44ayI (({{#  r'cHt|tr|j|}|D]}|j|jvr|j rc|j |jrH|j|}|j||jj|j|j |jr!|j stjd|j||jj|jy)NzAccumulating with FP16/BF16 in optimizer can lead to poor accuracy or slow convergence.Consider using multi_precision=True option of the Adam optimizer.) isinstancedict_update_param_groupr_already_create_accumulaterr!r-r)_create_master_weightr7addwarningswarn)r$blockrr5master_ps r&_create_accumulatorszAdamax._create_accumulatorss j$ '11*=JAvv999$$)D)DQWW)M55a8&&x00044QVV<++AGG4-- X  " "1 %  , , 0 0 8#r'c t|tjsJt|tr|j |}|j |j |d}|j |j|d}|jxr|j|dj}|r|j|djnd}|j |j|d}tjrStj |d|d|j#||||||j$|j&|j(| y|d|d|j#||||d}|d||d} |r ||d<|| d<|j$|j&|j(|d} |j+|j,|| | d } | S) Nrr )ParamGrad LearningRateMomentInfNormBeta1Pow)ParamOut MomentOut InfNormOut MasterParamMasterParamOut)rrrmulti_precisionTrinputsoutputsattrs stop_gradient)r9rBlockr:r;_get_accumulator_masterr2r3r!r-r)r"rr4in_dygraph_moderadamax__create_param_lrrrr append_opr) r$rAparam_and_gradr r find_master master_weightrrRrSrT adamax_ops r&_append_optimize_opzAdamax._append_optimize_ops%111 nd +!55nEN--  ."3 //  " "N1$5 ++ 0K0K 1  # #1   !2!7!7 8  44  # #^A%6   $ $ & NNq!q!%%n5    "(*&q) $ 5 5n E #) F+1-#&G (5}%,9()==#. E YY" (I r'c t|tjsJt|tr|D]\}}||jdurtj rd|j |j|}t5tj||jdd}|j|dddd|jjj||g5t!d5|j |j|}|j#dd|id|id|jid ddddddy|d D]]\}}||jdurtj r|j |j|}|j%d |j&d |_ t5tj||jdd}|j|dddd|jjj||g5t!d5|j |j|}|j%d |j&d |_ |j#dd|id|id|jid dddddd`y#1swYxYw#1swYxYw#1swYxYw#1swYxYw#1swYIxYw#1swYxYw) zUpdate Beta1 Power accumulatorNTgFrscaleXOutrQparamsr)r9rrVlistrUrXrWr4rrrbrcopy_rAprogram_optimized_guardrr[getr#)r$rAparameters_and_gradsr*gradrtmps r&_finish_updatezAdamax._finish_update#s%111 *D 13 t<5#6#6$#>,,.$($@$@//%M!$ll)4;;T&++C7 # ,,== !(+(,(D(D 33U) !($'#7%*M$:#*DKK"8*. ( , 46 4H= t<5#6#6$#>,,.$($@$@//%M#7":":!3!3G!<#DK!$ll)4;;T&++C7 # ,,== !(+(,(D(D 33U) ';&>&>#T%7%7%@' !($'#7%*M$:#*DKK"8*. (,! >)#,+0#,+sb5J1' K 3AJ>6K 5K* K16A'K%K11J; >K K  K K" %K. *K11K; c|jd|jd|_|jd|jd|_|jd|jd|_|jd}|S)Nrrrre)rjr#rrr )r$rs r&r;zAdamax._update_param_groupcsm nnWd.@.@.IJ  nnWd.@.@.IJ "y$2D2DY2OP ^^H- r')gMbP?g?g+?g:0yE>NNNN)__name__ __module__ __qualname____doc__r2r3r4rr7rCr`rnr; __classcell__)r%s@r&r r sYjVO"( ' R  90GR>@r'r )r?paddlerbaserr base.dygraphrbase.frameworkr optimizerr __all__r r'r&r|s,""' MYMr'