o lh4$@sddlZddlZddlZddlZddlmZmZddlm Z ddl m Z dgZ e dZeZdadadadd Z  dddZGdddeZdS)N)ProcessPoolExecutorEXTRA_QUEUED_CALLS) cpu_count) get_contextget_reusable_executorcCs8tt}td7a|WdS1swYdS)zEnsure that each successive executor instance has a unique, monotonic id. The purpose of this monotonic id is to help debug and test automated instance creation. rN)_executor_lock_next_executor_id) executor_idr Y/var/www/html/env/lib/python3.10/site-packages/joblib/externals/loky/reusable_executor.py_get_next_executor_ids $r Fautor c  Cstt} |dur|dur| dur| j}nt}n |dkr$td|t|tr-t|}|dur;| dkr;tdt ||||||| d} | durft j d|t} | attf|| d | a} n`|d krn| tk}| jjsx| jjsx|s| jjrd } n | jjrd } nd } t j d|| | jd|dda} atdd|i| WdSt j d| j| |Wd| SWd| S1swY| S)aReturn the current ReusableExectutor instance. Start a new instance if it has not been started already or if the previous instance was left in a broken state. If the previous instance does not have the requested number of workers, the executor is dynamically resized to adjust the number of workers prior to returning. Reusing a singleton instance spares the overhead of starting new worker processes and importing common python packages each time. ``max_workers`` controls the maximum number of tasks that can be running in parallel in worker processes. By default this is set to the number of CPUs on the host. Setting ``timeout`` (in seconds) makes idle workers automatically shutdown so as to release system resources. New workers are respawn upon submission of new tasks so that ``max_workers`` are available to accept the newly submitted tasks. Setting ``timeout`` to around 100 times the time required to spawn new processes and import packages in them (on the order of 100ms) ensures that the overhead of spawning workers is negligible. Setting ``kill_workers=True`` makes it possible to forcibly interrupt previously spawned jobs to get a new instance of the reusable executor with new constructor argument values. The ``job_reducers`` and ``result_reducers`` are used to customize the pickling of tasks and results send to the executor. When provided, the ``initializer`` is run first in newly spawned processes with argument ``initargs``. The environment variable in the child process are a copy of the values in the main process. One can provide a dict ``{ENV: VAL}`` where ``ENV`` and ``VAR`` are string literals to overwrite the environment variable ``ENV`` in the child processes to value ``VAL``. The environment variables are set in the children before any module is loaded. This only works with with the ``loky`` context and it is unreliable on Windows with Python < 3.6. NTrz+max_workers must be greater than 0, got {}.forkz4Cannot use reusable executor with the 'fork' context)contexttimeout job_reducersresult_reducers initializerinitargsenvz&Create a executor with max_workers={}.) max_workersr rbrokenshutdownzarguments have changedz[Creating a new executor with max_workers={} as the previous instance cannot be reused ({}).)wait kill_workersrz.Reusing existing executor with max_workers={}.r )r _executor _max_workersr ValueErrorformat isinstance STRING_TYPErget_start_methoddictmputildebugr_executor_kwargs_ReusablePoolExecutor_flagsrrr_resize)rrrrreuserrrrrexecutorkwargsr reasonr r r r(s,    6  : ::csNeZdZ    dfdd ZfddZdd Zd d Zfd d ZZS)r*Nrr c s0tt|j||||||| | d||_||_dS)N)rrrrrrrr)superr*__init__r _submit_resize_lock) selfsubmit_resize_lockrrrr rrrrr __class__r r r2s  z_ReusablePoolExecutor.__init__csH|jtt|j|g|Ri|WdS1swYdSN)r3r1r*submit)r4fnargsr/r6r r r9s $z_ReusablePoolExecutor.submitc Cst|j|dur td||jkr WddS|jdur+||_ WddS||j)t|j}t dd|D}||_t ||D]}|j dqKWdn1s^wYt |j|kr~|jjs~tdt |j|kr~|jjrn|t|j}tdd|Dstdtdd|DrWddSWddS1swYdS)Nz&Trying to resize with max_workers=Nonecss|]}|VqdSr8is_alive.0pr r r sz0_ReusablePoolExecutor._resize..MbP?cSsg|]}|qSr r<r>r r r sz1_ReusablePoolExecutor._resize..)r3r r_queue_management_thread_wait_job_completion_processes_management_locklist _processesvaluessumrange _call_queueputlenr+rtimesleep_adjust_process_countall)r4r processesnb_children_alive_r r r r,sD     "z_ReusablePoolExecutor._resizecCs\t|jdkrtdttjd|j t|jdkr,t dt|jdksdSdS)z8Wait for the cache to be empty before resizing the pool.rz\Trying to resize an executor with running jobs: waiting for jobs completion before resizing.z7Executor {} waiting for jobs completion before resizingrBN) rN_pending_work_itemswarningswarn UserWarningr&r'r(r!r rOrP)r4r r r rEs z*_ReusablePoolExecutor._wait_job_completioncs(dtt}tt|j|||ddS)N) queue_size)rrr1r* _setup_queues)r4rrr[r6r r r\s  z#_ReusablePoolExecutor._setup_queues) NNNrNNNr N) __name__ __module__ __qualname__r2r9r,rEr\ __classcell__r r r6r r*s " r*) NNrFrNNNr N)rOrW threadingmultiprocessingr&process_executorrrbackend.contextrbackendr__all__typer#RLockr r rr)rrr*r r r r s(    i