\;j_?lddlZddlZddlmZddlZddlZddlmZ ddlm Z ddl m Z m Z ddlmZddlmZmZmZmZmZmZmZmZmZddlmZdd lmZmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%d dl&m'Z'gZ(e)ejTjWddZ,ejZ ddZ.GddZ/y)N) defaultdict)_C_ops) parameter set_parameter)core) Variable_current_expected_placedefault_main_program device_guardin_dygraph_modein_dynamic_or_pir_mode in_pir_mode name_scope use_pir_api)L2Decay) framework unique_name)_get_no_grad_set_nameappend_backward) Parameter) LayerHelper LRScheduler$FLAGS_shard_bypass_dygraph_optimizercddlm}m}t}|jdk(sJd|j } |D]} | j | k(rJd|| || } ||jj}| j||\} } | j| | \}}g}| D]B}|| jj|j}|dk\sJ|j|D| jt!|| j#| t%|dk(r||fg}|Sg}t'|D]\}}|j|||f|S)Nr) Transform orig2primrzHThe append_backward_new interface is designed to process only one block.z@variable in loss_list should be in current block of main program)paddle.incubate.autograd.primxrrr num_blocks current_blockblock global_blockall_parameters linearize transposeopsindexopappend erase_opssorted erase_dotslen enumerate) loss_listparameter_list no_grad_set callbacks checkpointsdistop_contextrrprogramr#elad param_dotloss_dotloss_bar param_bar op_indexesvarop_indexparams_and_gradsiparams cG:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/optimizer/optimizer.pyappend_backward_newrE4sD"$GaRQR  ! ! #E HH  N M N  e 5 B --/>>@,,~yAIx,,x;HiJ ?yysvv.Hq= =   h '  LL #$MM) >a+Y78 !.1HAu  # #UIaL$9 :2 ceZdZdZej d-dZdZdZdZ dZ e jdZ e jd Zd Zd Ze jd Ze jd ZdZd.dZdZdZdZdZdZdZdZ d/dZdZdZdZdZ d0dZ! d0dZ" d-dZ#dZ$ d0d Z%d.d!Z& d.d"Z'd.d#Z(e jRd1d$Z*ej d2d%Z+d&Z,ej e jRd'Z-d(Z.d)Z/e jd*Z0e jd+Z1d,Z2y)3 OptimizeraWOptimizer Base class. Define the common interface of an optimizer. User should not use this class directly, but need to use one of it's implementation. Args: learning_rate (float|LRScheduler): The learning rate used to update ``Parameter``. It can be a float value or any subclass of ``LRScheduler`` . parameters (list|tuple, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. And you can specify different options for \ different parameter groups such as the learning rate, weight decay, etc, \ then the parameters are list of dict. Note that the learning_rate in paramter groups \ represents the scale of base learning_rate. \ The default value is None in static graph mode, at this time all parameters will be updated. weight_decay (float|WeightDecayRegularizer, optional): The strategy of regularization. \ It canbe a float value as coeff of L2 regularization or \ :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, \ the regularization setting here in optimizer will be ignored for this parameter. \ Otherwise, the regularization setting here in optimizer will take effect. \ Default None, meaning there is no regularization. grad_clip (GradientClipBase, optional): Gradient cliping strategy, it's an instance of \ some derived class of ``GradientClipBase`` . There are three cliping strategies \ ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , \ :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. Returns: Base class for optimizer. Examples: .. code-block:: python >>> # Take the subclass adam as an example >>> import paddle >>> linear = paddle.nn.Linear(10, 10) >>> inp = paddle.uniform(shape=[10, 10], min=-0.1, max=0.1) >>> out = linear(inp) >>> loss = paddle.mean(out) >>> adam = paddle.optimizer.Adam(learning_rate=0.1, ... parameters=linear.parameters()) >>> loss.backward() >>> adam.step() >>> adam.clear_grad() >>> #Take the subclass sgd as an example >>> #optimize parameters in linear_1 and linear2 in different options. >>> #Note that the learning_rate of linear_2 is 0.01. >>> linear_1 = paddle.nn.Linear(10, 10) >>> linear_2 = paddle.nn.Linear(10, 10) >>> inp = paddle.uniform(shape=[10, 10], min=-0.1, max=0.1) >>> out = linear_1(inp) >>> out = linear_2(out) >>> loss = paddle.mean(out) >>> sgd = paddle.optimizer.SGD( ... learning_rate=0.1, ... parameters=[{ ... 'params': linear_1.parameters() ... }, { ... 'params': linear_2.parameters(), ... 'weight_decay': 0.001, ... 'learning_rate': 0.1 ... }], ... weight_decay=0.01) >>> loss.backward() >>> sgd.step() >>> sgd.clear_grad() Nc|t|tjtjjfr#t dj t|t|tr t dt||_ nd|_ ||_ tjr|j td|ot|jdtsR|jD]C}t|ds|j t#j$d|j'znt|t(t*fst dt|z|9t|tj,j.j0s t dt|t(rt3||_n||_||_||_d|_|jr|t|jdtrA|jD] }d |vrJd |jdd dj<|_n|jdj<|_i|_tAd |_!d|_"g|_#i|_$i|_%|jL|_'|j4|j6d |_(g|_)|jrNt|jdtr1|jD]!}|jU|jW#n|j|_)d|_,|j[|_.i|_/ta|_1i|_2|jgy) Nzt`parameters` argument given to the optimizer should be an iterable of paddle Tensors, but got argument type is `{}`.zv`parameters` argument should not get dict type, if parameter groups is needed, please set `parameters` as list of dictzNparameters argument given to the Optimizer should not be None in dygraph mode.r regularizerzIf regularizer of a Parameter has been set by 'paddle.ParamAttr' or 'static.WeightNormParamAttr' already. The weight_decay[%s] in Optimizer will not take effect, and it will only be applied to other Parameters!z9learning rate should be float or LRScheduler, got %s herezE'grad_clip' should be an instance of GradientClipBase's derived classparamszYparams should be set in parameters if parameter groups are optimized in different optionsciSNrNrFrDz$Optimizer.__init__..srF) weight_decay grad_clip)4 isinstancepaddleTensorreager TypeErrorformattypedictlist_parameter_list_namerr AttributeErrorhasattrrJlogginginfo__str__floatrnnclipGradientClipBaserregularization _grad_clip_learning_rate_dtypedtype_learning_rate_mapr _accumulatorshelper_opti_name_list_accumulators_holder_param_device_map clear_gradclear_gradients _default_dict _param_groups_add_param_groupcopy_use_multi_tensor_create_multi_tensor_dict _param_dict_auxiliary_varsset_already_create_accumulater_master_weights_create_master_grad_states)self learning_rate parametersrPrQnamerC param_groups rD__init__zOptimizer.__init__s5  !*v}}djj6G6G&HITTZTZZ(U *d+' $( #3D #'D   $ $ &##+$d'!$"6"6q"94@!%!5!5#E=9 % 1 1 =#LL!K"."6"6"8!9 ""6-%)=>K}%&   i)H)HI[ lE *"),"7D ".D #+   $..q148#'#7#7K K/srs/$8#2215h?BHH "2215;; #% )4 !$&!!## //      Jt/C/CA/F$M#33 %%k&6&6&89 4"&!5!5D "&99;!+.5(! '')rFc i|_d|_y)NF) _master_grads _master_gradrs rDr~z$Optimizer._create_master_grad_states"s!rFc"||j|<yrM)rz)rkeyvals rD_set_auxiliary_varzOptimizer._set_auxiliary_var's$'S!rFc|jt|jnd}t|Dcgc]}gc}t|Dcgc]}gc}dScc}wcc}w)Nr)FP32_LODTensorFP16_LODTensor)rtr/range)rn_s rDrxz#Optimizer._create_multi_tensor_dict*s['+'9'9'EC"" #1+0848ar84+0848ar84  44s A Ac:|jj|dSrM)rzget)rrs rD_get_auxiliary_varzOptimizer._get_auxiliary_var1s##''T22rFci}|jjD]o\}}|jD]W\}}|||j<tjs*|j j ||jdz<Yqt|dr't|jdk7r|j|d<t|jtr|jj|d<|S)a Get state dict information from optimizer. It contain all the tensor used by optimizer. For Adam optimizer, contains beta1, beta2, momentum etc. If LRScheduler have been used, global_step will be include in state dict. If the optimizer never be called(minimize function), the state_dict is empty. Args: None Returns: state_dict(dict) : dict contains all the Tensor used by optimizer Examples: .. code-block:: python >>> import paddle >>> emb = paddle.nn.Embedding(10, 10) >>> adam = paddle.optimizer.Adam(0.001, parameters=emb.parameters()) >>> state_dict = adam.state_dict() .SCALE_VALUEr}rmaster_weights LR_Scheduler)rlitemsrris_compiled_with_xpu get_tensorget_xpu_scale_valuer^r/r}rRrhr state_dict)rrkv para_namevar_tmps rDrzOptimizer.state_dict4s, &&,,.DAq&'ggi" 7+2 7<<(,,. **,@@B ~5 '0/ 4* +4''(A-/3/C/C +, d)); 7)-)<)<)G)G)IJ~ &rFc t|jtr|jj|d|j }d|vr|j dd|vr't |dr |d|_|j d||_|jjD])\}}|jD]\}}|j|vsJd|jd|j}|j}tjr.|j!|j#|jdzdt%j&|}||j} t| t(rt%j&| } nxt| tj*j,rt%j&| } n>t| t$j.r| } n!t1dt3t5| d |j6| j6k(s6Jd j9|j|j6| j6|j:| j:k(s6Jd j9|j|j:| j:|j=| t?j@,y ) a: Load optimizer state dict. For Adam optimizer, contains beta1, beta2, momentum etc. If LRScheduler have been used, global_step will be changed. Args: state_dict(dict) : Dict contains all the Tensor needed by optimizer Return: None Examples: .. code-block:: python >>> import paddle >>> emb = paddle.nn.Embedding(10, 10) >>> layer_state_dict = emb.state_dict() >>> paddle.save(layer_state_dict, "emb.pdparams") >>> scheduler = paddle.optimizer.lr.NoamDecay( ... d_model=0.01, warmup_steps=100, verbose=True) >>> adam = paddle.optimizer.Adam( ... learning_rate=scheduler, ... parameters=emb.parameters()) >>> opt_state_dict = adam.state_dict() >>> paddle.save(opt_state_dict, "adam.pdopt") >>> opti_state_dict = paddle.load("adam.pdopt") >>> adam.set_state_dict(opti_state_dict) rrr}zoptimizer Tensor z not foundrzState dict type z not supprtzkParameter shape not match, Dygraph Parameter [ {} ] need tensor with shape {} but load tensor with shape {}zlParameter dtype not match, Dygraph Parameter [ {} ] need tensor with dtype {} but load tensor with dtype {}N)!rRrhrset_state_dictrvpopr^r}rorlrrvaluerrrset_xpu_scale_valuernparrayrrUrTndarray RuntimeErrorstrrXshaperWrjr{rr ) rrrrrrr?tensormodel_np load_para load_para_nps rDrzOptimizer.set_state_dict\so@ d)); 7    . .z./I J __& Z ' NN> * z )t./'12B'C$ NN+ ,$.!&&,,.DAq&'ggi" 7LLJ.@&w||nJ?@.mmo),,..."w||n'DdK88F+&w||4 i2#%88I#6L 4::+<+<=#%88I#6L 2::6#,L&*3tI+?*@ L NNl&8&88AHHMM8>><3E3E8 NNl&8&88BIIMM8>><3E3E8  <)J)J)LMQ'0/rFc|jSrM)rnrs rDget_opti_var_name_listz Optimizer.get_opti_var_name_lists###rFcfd}tjjj5|dddy#1swYyxYw)Nc 6 jtjn j}tjdk7r|tjk(s*tjdk7r#|tjk(rtj n|}t jtrj}trtjj}tjj}tjd}t!j}tjj#|5tj$j&j)|}tj*j,j/g|}|||j1}d|_t5||ddd|j7|t |tj*j8s|j_tjj#|5t=||g} dddd _d| _j|_ | |_!| jD|<yt |tFjHstjd}|j_jJjM|gdd|}tGj} j| _ || _!|jDtGj<t!j}jJjO|tj$j&j)|yyt jt r̉j} tr,t | tj*j8rytQ} t |tjRj,jTs)tj*j,jW|} tj*j,jY|gtjddtj$j&j[t!j jDtjj<yt | tFjHrytjj]tjdgt!j|d jDtGj<yy#1swYxYw#1swYBxYw) Nfloat16bfloat16rrT)rr persistable stop_gradientrj initializerF)rjrr trainablerrrrrjr)/rirSget_default_dtyperrfloat32rRrhr_global_learning_raterstaticdefault_startup_programr rgeneraterb program_guardrcrConstantpirr ParameterMetar$rrmove_parameters_fromOpResult _var_namerr lr_schedulerlr_varrkrrrmcreate_global_variableset_variable_initializerr baseDataTypeconvert_np_dtype_to_dtype_create_parameterConstantInitializercreate_global_var) _lr_dtyperstartup_program main_programlr_namelr_valuer paramete_meta init_resultrC main_proglrplacelr_dtypers rD do_createz9Optimizer._create_global_learning_rate..do_createsx;;&((*[[ 002i?%7002j@%8 $--{;335=&,mm&K&K&MO#)==#E#E#GL)22?CG$T%8%8%:;H44_E&,ii&;&;&D&D"*'E' )/ (E(E ) '2)?+G+G+I' 37 /%k7;F!55oF%ffjj.A.AB8?++5#]]88F$-gy"$EEG.2+,0)484G4G 1.3 +@E// =&fi.@.@A"-"6"6"G8?++5!%!C!C!("$(,*."+ "D"%.$B$B$D 151D1D .+1 (#//%::< %T%8%8%:;HKK88$*II$9$9$B$B"*%C%9=CHD//7//1=!"fjj&9&9: 7 9))V[[5E5E5N5NO &  J J$-!"%#JJOO<<"+"$!,!5!5o!F&+(. (=(=(Q(Q&+D,?,?&@)R) =//"MM>>@""i&8&89#MM;;!,!5!5o!F"$"'(;(;"<"+(, <//%::<;8eFE GFsA?VVV V)rSrrdygraph_guard_if_declarative)rrs` rD_create_global_learning_ratez&Optimizer._create_global_learning_rates4v p[[ " " ? ? A KB A As AA c ^t|ttfstdt |zt|j t r tdt||_|j}|trJt}tj|t|jt||j|yt!j"j%}|j'dd|gi|jt|jt|ddyy) a :api_attr: imperative Set the value of the learning rate manually in the optimizer. If the optimizer use LRScheduler, this API cannot be invoked, because it will lead to conflict. Args: value (float): the value of learning rate Returns: None Examples: .. code-block:: python >>> import paddle >>> linear = paddle.nn.Linear(10, 10) >>> adam = paddle.optimizer.Adam(0.1, parameters=linear.parameters()) >>> # set learning rate manually by python float value >>> lr_list = [0.2, 0.3, 0.4, 0.5, 0.6] >>> for i in range(5): ... adam.set_lr(lr_list[i]) ... lr = adam.get_lr() ... print("current lr is {}".format(lr)) current lr is 0.2 current lr is 0.3 current lr is 0.4 current lr is 0.5 current lr is 0.6 zGThe type of 'value' in optimizer.set_lr must be float, but received %s.zhoptimizer's learning rate can't be LRScheduler when invoke this API, because this will lead to conflict.N fill_constantOut)rjrrT)rXoutputsattrsr)rRintrbrVrXrhrrrr r rfull_rZrrjrr r$ append_op)rr current_lrrr$s rDset_lrzOptimizer.set_lr3sF%#u.Y;   d)); 7z $El//1  ! /1 ))*%L$$  )==?LLN &&("ZL1!+!1!1!%j&6&6!7!&u #''  "rFcdddlm}t||stdt |z||_y)a :api_attr: imperative Set the LRScheduler of the learning rate manually in the optimizer. If the optimizer already used LRScheduler previously, this API will set it be the new one. Args: scheduler (LRScheduler): the LRScheduler of learning rate Returns: None Examples: .. code-block:: python >>> import paddle >>> linear = paddle.nn.Linear(10, 10) >>> adam = paddle.optimizer.Adam(0.1, parameters=linear.parameters()) >>> # set learning rate manually by class LRScheduler >>> scheduler = paddle.optimizer.lr.MultiStepDecay(learning_rate=0.5, milestones=[2,4,6], gamma=0.8) >>> adam.set_lr_scheduler(scheduler) >>> lr = adam.get_lr() >>> print("current lr is {}".format(lr)) current lr is 0.5 >>> # set learning rate manually by another LRScheduler >>> scheduler = paddle.optimizer.lr.StepDecay(learning_rate=0.1, step_size=5, gamma=0.6) >>> adam.set_lr_scheduler(scheduler) >>> lr = adam.get_lr() >>> print("current lr is {}".format(lr)) current lr is 0.1 rrzZThe type of 'scheduler' in optimizer.set_lr_schduler must be LRScheduler, but received %s.N)paddle.optimizer.lrrrRrVrXrh)r schedulerrs rDset_lr_schedulerzOptimizer.set_lr_schedulerxs8J 4)[1l ?$ (rFcnt|jtr |jS|jS)a Get current learning rate of optimizer. If 'LRScheduler' is not used, the return value is all the same. If 'LRScheduler' is used, the return value is the current scheduled learing rete. Returns: float: The current learning rate of optimizer. Examples: .. code-block:: python >>> # train on default dynamic graph mode >>> import paddle >>> import numpy as np >>> emb = paddle.nn.Embedding(10, 3) >>> ## example1: LRScheduler is not used, return the same value is all the same >>> adam = paddle.optimizer.Adam(0.01, parameters = emb.parameters()) >>> for batch in range(10): ... input = paddle.randint(low=0, high=5, shape=[5]) ... out = emb(input) ... out.backward() ... print("Learning rate of step{}: {}".format(batch, adam.get_lr())) # 0.01 ... adam.step() Learning rate of step0: 0.01 Learning rate of step1: 0.01 Learning rate of step2: 0.01 Learning rate of step3: 0.01 Learning rate of step4: 0.01 Learning rate of step5: 0.01 Learning rate of step6: 0.01 Learning rate of step7: 0.01 Learning rate of step8: 0.01 Learning rate of step9: 0.01 >>> ## example2: StepDecay is used, return the scheduled learning rate >>> scheduler = paddle.optimizer.lr.StepDecay(learning_rate=0.5, step_size=2, gamma=0.1) >>> adam = paddle.optimizer.Adam(scheduler, parameters = emb.parameters()) >>> for batch in range(10): ... input = paddle.randint(low=0, high=5, shape=[5]) ... out = emb(input) ... out.backward() ... print("Learning rate of step{}: {}".format(batch, adam.get_lr())) # 0.5->0.05... ... adam.step() ... scheduler.step() Learning rate of step0: 0.5 Learning rate of step1: 0.5 Learning rate of step2: 0.05 Learning rate of step3: 0.05 Learning rate of step4: 0.005000000000000001 Learning rate of step5: 0.005000000000000001 Learning rate of step6: 0.0005000000000000001 Learning rate of step7: 0.0005000000000000001 Learning rate of step8: 5.000000000000001e-05 Learning rate of step9: 5.000000000000001e-05 >>> # train on static graph mode >>> paddle.enable_static() >>> main_prog = paddle.static.Program() >>> start_prog = paddle.static.Program() >>> with paddle.static.program_guard(main_prog, start_prog): ... x = paddle.static.data(name='x', shape=[None, 10]) ... z = paddle.static.nn.fc(x, 100) ... loss = paddle.mean(z) ... scheduler = paddle.optimizer.lr.StepDecay(learning_rate=0.5, step_size=2, gamma=0.1) ... adam = paddle.optimizer.Adam(learning_rate=scheduler) ... adam.minimize(loss) >>> exe = paddle.static.Executor() >>> exe.run(start_prog) >>> for batch in range(10): ... print("Learning rate of step{}: {}".format(batch, adam.get_lr())) # 0.5->0.05->0.005... ... out = exe.run(main_prog, feed={'x': np.random.randn(3, 10).astype('float32')}) ... scheduler.step() Learning rate of step0: 0.5 Learning rate of step1: 0.5 Learning rate of step2: 0.05 Learning rate of step3: 0.05 Learning rate of step4: 0.005000000000000001 Learning rate of step5: 0.005000000000000001 Learning rate of step6: 0.0005000000000000001 Learning rate of step7: 0.0005000000000000001 Learning rate of step8: 5.000000000000001e-05 Learning rate of step9: 5.000000000000001e-05 )rRrhrbrs rDget_lrzOptimizer.get_lrs0l d))5 1&& &&&( (rFc|=trtj}ntjj}|j j |dS)zC get global decayed learning rate :return: N)r rr rSrrkr)rr7s rDrzOptimizer._global_learning_ratesH ? #88: --<<>&&**7D99rFctd)zFappend optimize operator to block and return all the added optimize_opzcClass "Optimizer" connot be used directly as an optimizer, please use its subclasses such as "Adam")NotImplementedError)rr#param_and_grads rD_append_optimize_opzOptimizer._append_optimize_op s! u  rFc|d}t|dr|jd}t|ttj j fr|S|dk(r|jStsjtjjjd5tjd5|j|zcdddcdddStjd5|j|zcdddS|jS#1swYnxYw dddy#1swYyxYw#1swYyxYw)Nr optimize_attrr?T) is_with_optscale_with_param_lr)r^rrRrrSrrrrrr _lr_schedule_guardrr)rrrCparam_lrs rD_create_param_lrzOptimizer._create_param_lrs!q! 5/ ***?;H(Xvzz/B/B$CDs?5577&=#]]??ATT(,U$//1$(#=#=#?(#J'112GH#'#=#=#?(#JIH--/ /IHs0D.0D D.+D:D! D..D7:Ec|j|jvr|j|j}|S|j|}trt j |d}nt jrt j |d}||_nt|jtsJtjj||jddd}|jjj}|j!dd|gid|gi|j"t$j&j(j*d ||j|j<|S) NrrTrcastXr)in_dtype out_dtype)rXinputsrr)rr}_gen_master_weight_var_namerrSrrr rRrmrrrrrr$rrjrVarDescVarTypeFP32)rrCr?var_namer#s rD_create_master_weightzOptimizer._create_master_weight.s3 ::-- -&&uzz2C8 577>H}kk%3**,kk%3#!$++{;;;mm55!++# $ 6 33@@B%>"SEN$)KK%)\\%9%9%>%>  03D  , rFcJ|jdz}tj|S)N _fp32_master)rrr)rrCr s rDrz%Optimizer._gen_master_weight_var_nameNs!::.##H--rFc |j|jsJ|j|jvr|j|j}|S|jdz}t j |}|j j||jdd|j|j|j}||j|j<|S)Nrrr)rrrrj lod_levelris_data) _is_dtype_fp16_or_bf16rjrrrrr# create_varrrrr)rgradr?r s rD_create_master_gradzOptimizer._create_master_gradRs**4::666 99** *$$TYY/C yy>1H"++H5H**''jj.. ,, (C-0D  tyy ) rFcy)zCreate all accumulators needed by the parameters Args: block: the block in which the loss tensor is present parameters: list of parameter tensors for the optimizer NrN)rr#rs rD_create_accumulatorszOptimizer._create_accumulatorse rFcy)aFinish any custom updates needed before completing an optimization step Args: block: the block in which the loss tensor is present parameters: list of parameter tensors for the optimizer Returns: None NrN)rr#parameters_and_gradss rD_finish_updatezOptimizer._finish_updatens rFc ~|j|jdz|z}||jvrf|j|j|vrKtjr|j||jSt d|d|j| |j }|jdz|z}tj|}|jj|||j|j}trntjjj!|xs |j"||tj$j&j)t+|} nt-|j.t0sJ|j.j3|d|xs |j"tj4j6j8|d} t r|d k(st-|tj:rftj<sRt?j@| | j tCt+|| j"tj:nbtE|5|j.jG| tj$j&j)t+|dddtjrtI|jJd kDr||jJvs Jd |d | jM|jJjO|tj<r<| jQjS|jJjU|d zd| |j||j<| S#1swYxYw)a|Utility function to add an accumulator for a parameter Args: block: the block in which the loss tensor is present name: name of the accumulator param: parameter tensor for which accumulator is to be added dtype: data type of the accumulator tensor fill_value: value to initialize the accumulator tensor Nr Accumulator z already exists for parameter rrT)rrrjrXrbelong_to_optimizercpurzOptimizer set error, z should in state dictrr)+r\rlrrr Exceptionrrrrnr+_get_device_for_paramrrSrrrrjrcrrrbrRrmrrr r  LOD_TENSORCPUPlacerrrrr rr/ro set_valuerrrr) rrrCrj fill_valuerrXdevicer r?s rD_add_accumulatorzOptimizer._add_accumulator{s& :: !::#d*D D&& & d0066((*))$/ ;;tf$B5::,O  =KKE::#d*''1 ##H- >// ;F =**//22$"II11:: +; 3Cdkk;7 77++44 *u{{\\))44$( 5C !u_ 64==(I224 IIj)*IIMMO "&)KK88$*II$9$9$B$B"' "3%C%9*((*t001A5 D$=$==O.xj8MNO=MM$";";"?"?"IJ002(<< 5599 (> 94 034 , 1*)s ?AN33N<c|j|jdz|z}||jvs|j|j|vrtd|d|j|j||jS)aUtility function to fetch an accumulator for a parameter Args: name: name of the accumulator param: parameter tensor for which accumulator is to be fetched Returns: accumulator tensor for the parameter rr does not exist for parameter )r\rlrr!)rrrCs rD_get_accumulatorzOptimizer._get_accumulators :: !::#d*D ** *zz!3!3D!99tf$B5::,O !!$' 33rFcf|j|jdz|z}|jxr|j|j}|r|j|j n|}|j }||j vs||j |vrtd|d||j ||S)a Utility function to fetch an accumulator for a parameter Args: name: name of the accumulator param: parameter variable for which accumulator is to be fetched Returns: accumulator variable for the parameter rrr*)r\_multi_precisionrrjr}rrlr!)rrrC find_master target_param target_names rD_get_accumulator_masterz!Optimizer._get_accumulator_masters :: !::#d*D++ 0K0K KK1 1..z: 3rFcDd}||jvr|j|}|SrM)rp)rr8r's rDr"zOptimizer._get_device_for_params* // /++J7F rFc  tjj}|}tjj}|j|jk7rA|j dk7sJdtjj |j }t|j}t|jj|_ |j|jr|jjdvrt|jd|dk(rt|jd|dk(rt!|t"r;|dk(sJ|j%||Dcgc]}|dj&s|dc}|nG|j)||j%||dDcgc]}|dj&s|dc}|tj*r|j-|||n|j/||g}|D]@} | dj&r| d |j1| d|j1| d B|dj2j4j7|5t9d 5|j;|dj<} t?| 5|j-|||d d d d d d d d d n!tj*s)t!|t@r|dn|} |j/| |t!|t"rdtBjDjjG5|jI||Dcgc]}|dj&s|dc}d d d n{|jK} | dDcgc]}|dj&s|dc}| d<tBjDjjG5|jI|| d d d tj*r9|jMd } | r9t!| tNjPjRr|jUd d nt!| tNjPjRr|jUd dt!|t"r3|D],} | d  | dj&dus|jW|| .n|dD]k} | d  | dj&dusi}| |d<|jY|j[Dcic] \}}|dk7r||c}}|jW||mn|D]} | d  | dj2j4j7| 5t9d 5| dj&durD|j;| dj<} t?| 5|jW|| }d d d d d d d d d |j]||t|j}|j_||Scc}wcc}w#1swYxYw#1swYxYw#1swYfxYwcc}w#1swYmxYwcc}w#1swYxYwcc}}w#1swYxYw#1swYxYw#1swYZxYw)Add optimization operators to update gradients to tensors. Args: parameters_and_grads(list(tuple(Tensor, Tensor))): a list of (tensor, gradient) pair to update. Returns: return_op_list: a list of operators that will complete one step of optimization. This will include parameter update ops, global step update ops and any other custom ops required by subclasses to manage their internal state. zFcurrent block is not global_block, but it doesn't have backward block.)MomentumAdamrrrrKparam_group_idxrN optimizer found_infTF)0rr r$r"idxbackward_block_idxblocksr/r(r __class____name__rmrrwryrRrZ_multi_tensor_initr_update_param_groupr _append_optimize_multi_tensor_opr:r+r#r7_optimized_guardrr"rr rYrSrrrrvrrrUrTrrupdaterr _slice_ops)rrrBr$r7r"startpparam_grad_listrr'params_grads_device_mapparams_acc_dictrDparam_grad_dictrr optimize_opends rD_create_optimization_passz#Optimizer._create_optimization_pass#s4!557DDF # !668FFH    0 0 000B6 XW X6$99;BB00LL$$%!$.."9"9:  ))+  ! !dnn&=&=B ' D$$%56GHAM(()9:?KL2D9*a///++$&:%9#$Q4#5#5aD%9 (,,-AB++$&:(%C%C#$Q4#5#5aD%C (((*55 ($36 --(, #%&:N*1-;;*1-9'..~a/@A'..~a/@A ';%Q'--55FF#k*!778J8O8OPF%f-==(0,;>.+,,.""6=)2-( --+\.5[[**GGI--$&:%9#$Q4#5#5aD%9JI#7";";"=-X6-6Q4--aD6-) [[**GGI--lOLJ((* 33K@ !)TZZ->->?// TB!)TZZ->->?// UC!"6=.BN-a08 (-a0>>%G $ 8 8$0.!" /C/C8.LN-a08 (-a0>>%G24?,""#&&uc22K>.-+*2JI- JI6%&("6!5 /.s3Y ;Y Y,*Y?YYY,#Y=4Y8 Y=6Z ?Z9Z  Z:';Z."Z" 5Z.=Z:Y YY) $Y,,Y58Y==ZZ"Z+ 'Z..Z7 3Z::[ c tjj}|}t|j}|j t |tr|dn|}|j||t |tr4|j||Dcgc]}|djr|dc}nJ|j}|dDcgc]}|djs|dc}|d<|j||t |tr2|D],} | d | djdus|j|| .ns|dD]k} | d | djdusi} | | d<| j|jD cic] \} } | dk7r| | c} } |j|| m|j!||t|j} |j#|| Scc}wcc}wcc} } w)r=rKrrF)rr r$r/r(rrRrYr:rZrrrvrrNrrrO)rrrBr$r7rPrSrQrTrrUrrrWs rD_pir_create_optimization_passz'Optimizer._pir_create_optimization_passs. !557DDF # L$$% ))+.5 ! *% %%&=|L *D 1  % %3N3!1Q4;M;M13N  3779O)2)2At))!2)OH %  % %lO D *D 1"6!!$,!!$22e;,,\>J #7 #7x"@!!$,!!$22e;&(O0>OH-#**)=(B(B(D(D1 H}qD(D,,\?K#A" L*>?,""#&&uc22MO)*sG/ (G/ G4G9 cd}tjrn|j||}|j|j|_tjr_|r|n |j }g}t jj|} t| D]\} } |  |j|| | f |S|&tjjjg}nt|t sJ|j"j$} t'j(|j*dk(s Jdj-|j*|r|n |j }tj.j1| |5t3r|;| j5j7} | Dcgc]}|j8dur|}}g}tj:j<j?|||} t| D]\} } |  |j|| | f n+ddl m!}|rtE|g|||}ntG||||}ddd|Scc}w#1swYSxYw)a The first part of ``minimize``, do auto-diff to append backward operations for the current program. Args: loss (Tensor): ``loss`` tensor to run optimizations. startup_program (Program, optional): :ref:`api_paddle_static_Program` for initializing parameters in ``parameters``. The default value is None, at this time :ref:`api_paddle_static_default_startup_program` will be used. parameters (list, optional): List of ``Tensor`` or ``Tensor.name`` to update to minimize ``loss``. The default value is None, at this time all parameters will be updated. no_grad_set (set, optional): Set of ``Tensor`` or ``Tensor.name`` that don't need to be updated. The default value is None. callbacks (list, optional): list of callable objects to run when appending backward operator for one parameter. The default value is None. Return: list: list of (param, grad) tensor pairs, param is ``Parameter``, grad is the gradient value corresponding to the parameter. Examples: .. code-block:: python >>> import paddle >>> x = paddle.arange(26, dtype="float32").reshape([2, 13]) >>> linear = paddle.nn.Linear(13, 5) >>> # This can be any optimizer supported by dygraph. >>> adam = paddle.optimizer.Adam(learning_rate = 0.01, ... parameters = linear.parameters()) >>> out = linear(x) >>> out.backward() >>> adam.step() >>> adam.clear_grad() NrzThe number of elements of loss should be 1, but the current loss.shape is {}, whose number of elements is not 1. Maybe that you should call paddle.mean to process the current loss.F) no_grad_varsr) prim_enabled)$rr _get_no_grad_setrirjr[rrU get_all_gradsr0r+rSrcrderror_clip_callbackrRrZr#r7rprodrrWrrrr$r%rautograd ir_backwardrpaddle.incubate.autograd.utilsr]rEr)rlossrrr3r4act_no_grad_setr2 params_gradsgradsr)rr7program_all_paramsrCr]s rDbackwardzOptimizer.backward+s^X  $ $ & "33D+FO ;; **DK  $ $ &+5Z4;O;ONLJJ,,^(EF 0\U #YY^^??@ !)T222jj((G774::&!+ VV\V\JJW + ,6Z4;O;ON,,WoF=%-$002AAC+ *<*);$22e;");'* $&L"OO77<<n?=E(1'7 t+(//1F0MN(8L#~':!FNOY( (7 ./9( 7G</*G<s&>> import paddle >>> inp = paddle.uniform([10, 10], dtype="float32", min=-0.1, max=0.1) >>> linear = paddle.nn.Linear(10, 10) >>> out = linear(inp) >>> loss = paddle.mean(out) >>> optimizer = paddle.optimizer.Adam(learning_rate=0.1, ... parameters=linear.parameters()) >>> params_grads = optimizer.backward(loss) >>> optimizer.apply_gradients(params_grads) _sortedc |djSNr)r)xs rDrOz+Optimizer.apply_gradients..s adiirF)r) r^r-rgrSrcrdappend_gradient_clip_opsappend_regularization_opsrfrX)rrg optimize_opss rDapply_gradientszOptimizer.apply_gradientss6tY'!,4GHL ?? &??<8L!99>>BB<PL55 $-- 55lC rFc^tjrtrytrtj j tj jtj j5t|tr:|j|j|}|j||j}n7|d}|||d|d<|j|d|j|d<tr|j||}n|j!||}ddd|S|dk(sJ|j"j$}tj j ||5|j'|}ddd|S#1swYSxYw#1swYSxYw)a Second part of `minimize`, appending optimization operators for given `params_grads` pairs. Args: loss (Tensor): loss tensor to run optimizations. startup_program (Program): startup_program for initializing parameters in `parameters`. params_grads (list): list of (param, grad) pair to do optimization. Returns: list: A list of operators appended to the current program. NrQrKrAr)rr g_shard_bypass_dygraph_optimizerr rSrrr rrRrZrgrqrfrrZrXr#r7rs)rrerrgrBrQrrr7s rD_apply_optimizezOptimizer._apply_optimizes  $ $ &+K  ! #,, 224 557lD12'+|'D #'#A#A$d&9&9$L!-[ 9I ,1:(22 X..2-K-K$X.0C0C.L*=#'#E#E$o$F$L$(#A#A$o$B$L3B #a' ''jj((G,,WoF#33LA GCBGs?B3F9F"F"F,c|&t|drt|dr|j||Sd}fd}|||}t|dr*|j|j|||j}n|||||j}|Jtrt j ||gS|}|j tjjjk(r|jj|jtjz|j|j|j tjjj"}d||gi}d|gi}|jj%d|||S) zpCreate and add backward regularization Operators Function helper of append_regularization_ops. NrJc0|}|j|jk7ryjxrj|j}|r3tjdk7rj|j }|S|j |j}|Srn)rjr-rr/r}rastype)rCrr/r.rs rDget_target_paramzBOptimizer._create_regularization_of_grad..get_target_param s L{{djj())A33EKK@3t';';#<#A#'#7#7 #CL $)<< #;L rF)rrjrrrXrrsum)rXrr)r^rJr#r radd_nrXrr r  SELECTED_ROWSrrkNewGradSuffixrjrrr#r) rrCrrfregularization_termrznew_gradrrs ` rD_create_regularization_of_gradz(Optimizer._create_regularization_of_gradsp <E=1E=1e6G6G6O&K" !- 5- (U->->-J"'"3"3E4"L   '"0djj"I "... ! #<<': ;< <HyyDLL00>>>  ::00T%8%8%::++++#oo--88 1D"567Fxj)G JJ eFG LOrFcNg}tjs tr2|D]+\}}|j|||}|j ||f-|Sd}tj d5|D]\}}|s6|j *|(d}tjd|jz|jjj||g5|j|||}|j ||fddd ddd|S#1swYxYw#1swY|SxYw)abCreate and add backward regularization Operators Creates and adds backward regularization operators in the BlockDesc. This will add gradients of the regularizer function to the gradients of the parameters and return these modified gradients. This is the same as implementing weight decay in optimizers for regularization. Args: parameters_and_grads: A list of (parameters, gradients) pairs that need to be regularized. regularization: A global regularizer. If the parameter is not set. It will be applied with regularizer. Returns: list[(Variable, Variable)]: list of (parameters, gradients) \ pair with the regularized gradient Raises: Exception: Unknown regularization type FrfNTzIf regularizer of a Parameter has been set by 'base.ParamAttr' or 'base.WeightNormParamAttr' already. The Regularization[%s] in Optimizer will not take effect, and it will only be applied to other Parameters!) rr rrr+rrJr_r`rar#r7rM)rrrfrArCrrrepeate_regularizers rDrqz#Optimizer.append_regularization_ops9s7.  $ $ &+-3 t>>4!''(9: 42 '#( %%&67#7KE4/!--9*6.2+ I,4467 ,,==udmL#'#F#F!4$)//0AB ML$88$  ML8$ s%*A(D'D9 DD DD$ct|}|jjjj }|Dchc]}|j dus|j }}|j||Scc}w)NT)rr#r7r$r%rrrN)rrer3rrCparam_no_trainables rDr^zOptimizer._get_no_grad_setmsv+K8 ZZ''446EEG $. $.5%2E2E2MEJJJ   -.  s A9A9c\g}|jt|jdts0|jD] }|jr|j |"n9|j D]*}|dD] }|jr|j |",|D]}|j |y)a Clear the gradients of all optimized parameters for model. If not, new gradient will accumulat on previous gradient. There are two method to clear grad: set_to_zero or delete grad. Args: set_to_zero (bool, optional): If set grads to zero or not, default is True. Returns: None Examples: .. code-block:: python >>> import paddle >>> a = paddle.arange(26, dtype="float32").reshape([2, 13]) >>> linear = paddle.nn.Linear(13, 5) >>> # This can be any optimizer supported by dygraph. >>> adam = paddle.optimizer.Adam(learning_rate = 0.01, ... parameters = linear.parameters()) >>> out = linear(a) >>> out.backward() >>> adam.step() >>> adam.clear_grad() NrrK)r[rRrYrr+rtclear_gradient)r set_to_zero param_listrQrs rDrqzOptimizer.clear_gradxs>    'z   #T0 ))%%a(* $11 $X.A??"))!,/ 2 A  [ )rFct|ttjjfsJd|r|n |j }|j ||||}|j|||}||fS)a Add operations to minimize ``loss`` by updating ``parameters``. Args: loss (Tensor): A ``Tensor`` containing the value to minimize. startup_program (Program, optional): :ref:`api_paddle_static_Program` for initializing parameters in ``parameters``. The default value is None, at this time :ref:`api_paddle_static_default_startup_program` will be used. parameters (list, optional): List of ``Tensor`` or ``Tensor.name`` to update to minimize ``loss``. The default value is None, at this time all parameters will be updated. no_grad_set (set, optional): Set of ``Tensor`` or ``Tensor.name`` that don't need to be updated. The default value is None. Returns: tuple: tuple (optimize_ops, params_grads), A list of operators appended by minimize and a list of (param, grad) tensor pairs, param is ``Parameter``, grad is the gradient value corresponding to the parameter. In static graph mode, the returned tuple can be passed to ``fetch_list`` in ``Executor.run()`` to indicate program pruning. If so, the program will be pruned by ``feed`` and ``fetch_list`` before run, see details in ``Executor``. Examples: .. code-block:: python >>> import paddle >>> linear = paddle.nn.Linear(10, 10) >>> input = paddle.uniform(shape=[10, 10], min=-0.1, max=0.1) >>> out = linear(input) >>> loss = paddle.mean(out) >>> beta1 = paddle.to_tensor([0.9], dtype="float32") >>> beta2 = paddle.to_tensor([0.99], dtype="float32") >>> adam = paddle.optimizer.Adam(learning_rate=0.1, ... parameters=linear.parameters(), ... weight_decay=0.01) >>> loss.backward() >>> adam.minimize(loss) >>> adam.clear_grad() zThe loss should be an Tensor.)rrr3)rrg)rRrrSrrr[rjrv)rrerrr3r2rgrrs rDminimizezOptimizer.minimizes\ 8VZZ001  + * + (2t7K7K}} +%# % ++ / , \))rFctjjjj }t |j dtrJd|j Dchc]}|jc}|Dcgc]}|js|}}ttfd|}|Dcgc]}||jf}}|j|}ycc}wcc}wcc}w)zW In declarative mode, we forward `call step` to `call apply_gradients` rzQOnly list of parameters is supported while using optimizer in @paddle.jit.static.c<|jvxr t|dS)Nr)rr^)roselected_paramss rDrOz-Optimizer._declarative_step..s!&&O3J68JJrFN)rSrr r$r%rRr[rYrrrZfilterrrs)rrKrCrrgrrrs @rD_declarative_stepzOptimizer._declarative_steps MM . . 0 = = ? N N P    #T  _ ^ _ 483G3GH3G%5::3GH)/C5??e C J  :DD + D++L9 ICEs.C%C*C*<C/c .tjjjjr|j yt |j dtsjg}|j D]C}|jr|j!|j}|j||fE|jdd|dyt|j D]\}}td}|dD]F}|jr|j!|j}|dj||fH|j|jDcic]\}}|dk7s ||c}}|jdd||ycc}}w)a Execute the optimizer and update parameters once. Returns: None Examples: .. code-block:: python >>> import paddle >>> a = paddle.arange(26, dtype="float32").reshape([2, 13]) >>> linear = paddle.nn.Linear(13, 5) >>> # This can be any optimizer supported by dygraph. >>> adam = paddle.optimizer.Adam(learning_rate = 0.01, ... parameters = linear.parameters()) >>> out = linear(a) >>> out.backward() >>> adam.step() >>> adam.clear_grad() Nr)rerrgrBcgSrMrNrNrFrDrOz Optimizer.step..,s2rFrK)rSrdygraphin_to_static_moderrRrtrYr _grad_ivarr+rvr0rrNr)rrgrCgrad_varrErrrs rDstepzOptimizer.steps0 ;;   # # 5 5 7  " " $ $,,Q/6L++&&##%1$//1H ''(9: ,   $) ! ! %.d.@.@$A [*:6 (2E** '')5#(#3#3#5$X.55uh6GH 3 ##&1&7&7&9K&9daQ(]QT&9K$$$(!-$' %%BLs  F ,F c|d}t|tr|g|d<n)t|tr tdt ||d<|j j D]\}}|j||t}|jD]}|jt|d!|jt|ds td|dD]K}|d}t|tr t|} n|} | |_|jdd|j d<M|jj#|y)z Add a param group to parameter_list. Args: param_group (dict): The group of Tensors to be optimzed with different optimization options. rKz`optimizer parameters should be in ordered collections,but received set, please use list instead.z7some parameters appear in more than one parameter grouprPrrN)rRrr{rVrZrsr setdefaultrtrN isdisjoint ValueErrorrbrrJrrr+) rrrKrr param_setgrouprCrPrfs rDruzOptimizer._add_param_group=sIX& fi (%+HK !  $=  %)LK !&&,,.DAq  " "1a (/E ''E   Sx1 2(##C H(=$>?I !*E&~6L,.!(!6!- .E 3>??4E   0+ !!+.rFcy)z Update the param group with new entry Args: parameters (dict): The extra group of Tensors to be optimzed with different optimization options. Only used in child class. NrN)rrs rDrKzOptimizer._update_param_groupjrrFcy)a All parameters used for optimizer (such as: parameters, master_weight, velocity_acc for momentum) calculations are grouped into a python list by data type (float16, float32). This function will be overridden in the corresponding optimizer file. Args: target_block: the block in which the loss tensor is present parameters: list of parameter tensors for the optimizer NrN)rr7rrBs rDrJzOptimizer._multi_tensor_initss rFcy)zM For Multi Tensor, append optimize merged_operator to block. NrN)rr7rrBs rDrLz*Optimizer._append_optimize_multi_tensor_oprrFct|tjjtjfsJdt|tjjrP|tjjj k(xs'|tjjj k(S|tjjk(xs|tjjk(S)z check the dtype is fp16 or the dtype is bf16 :param dtype: instance of core.VarDesc.VarType :return: True if dtype is one of fp16 or bf16, False otherwise zIThe dtype should be an instance of core.VarDesc.VarType or core.DataType.) rRrr r rFP16BF16FLOAT16UINT16)rrjs rDrz Optimizer._is_dtype_fp16_or_bf16s  DLL(($--8  W V W  eT\\11 2--2226DLL00555  ...O%4==;O;O2O rF)NNNNrM)NgNNN)r)T)NNN)3rI __module__ __qualname____doc__imperative_baseno_gradrr~rrxrr dygraph_onlyrrrrrrrrrrr rrrrr(r+r1r:r"rXrZrjrsrvrrqr^non_static_onlyrqrrrrurKrJrLrrNrFrDrHrHds2GR_  q*q*f" ( 3%%NUNUNn$zxBBH+(+(ZY)v : 06@.&   " aF4*54 56}3@56G3X kZ*ZDE4l=@482 h ,*,*\_GK>*>*@:*_<<|+/Z       rFrH)NNNNN)0r_os collectionsrnumpyrrSpaddle.autogradrbrrpaddle._pir_opsrr paddle.baserpaddle.base.frameworkrr r r r r rrrpaddle.regularizerrrrr base.backwardrrbase.frameworkrbase.layer_helperrrr__all__renvironrru static_onlyrErHrNrFrDrs # )4   ')B&+ #&JJNN91=$   ,,^uurF