Downloader¶
- class parfive.Downloader(max_conn=5, max_splits=5, progress=True, overwrite=False, config=None)[source]¶
Bases:
objectDownload files in parallel.
- Parameters
max_conn (
int) – The number of parallel download slots.max_splits (
int) – The maximum number of splits to use to download a file (server dependant).progress (
bool) – IfTrueshow a main progress bar showing how many of the total files have been downloaded. IfFalse, no progress bars will be shown at all.overwrite (
Union[bool,Literal[‘unique’]]) – Determine how to handle downloading if a file already exists with the same name. IfFalsethe file download will be skipped and the path returned to the existing file, ifTruethe file will be downloaded and the existing file will be overwritten, if'unique'the filename will be modified to be unique.config (
Optional[SessionConfig,None]) – A config object containing more complex settings for thisDownloaderinstance.
Attributes Summary
The total number of files already queued for download.
Methods Summary
download()Download all files in the queue.
enqueue_file(url[, path, filename, overwrite])Add a file to the download queue.
retry(results)Retry any failed downloads in a results object.
Download all files in the queue.
simple_download(urls, *[, path, overwrite])Download a series of URLs to a single destination.
Attributes Documentation
- queued_downloads¶
The total number of files already queued for download.
Methods Documentation
- download()[source]¶
Download all files in the queue.
- Returns
parfive.Results– A list of files downloaded.
Notes
This is a synchronous version of
run_download, anasyncioevent loop will be created to run the download (in it’s own thread if a loop is already running).
- enqueue_file(url, path=None, filename=None, overwrite=None, **kwargs)[source]¶
Add a file to the download queue.
- Parameters
url (
str) – The URL to retrieve.path (
Union[str,PathLike,None]) – The directory to retrieve the file into, ifNonedefaults to the current directory.filename (
Union[str,Callable[[str,Optional[ClientResponse,None]],PathLike],None]) – The filename to save the file as. Can also be a callable which takes two arguments the url and the response object from opening that URL, and returns the filename. (Note, for FTP downloads the response will beNone.) IfNonethe HTTP headers will be read for the filename, or the last segment of the URL will be used.overwrite (
Union[bool,Literal[‘unique’],None]) – Determine how to handle downloading if a file already exists with the same name. IfFalsethe file download will be skipped and the path returned to the existing file, ifTruethe file will be downloaded and the existing file will be overwritten, if'unique'the filename will be modified to be unique. IfNonethe value set when constructing theDownloaderobject will be used.kwargs (
dict) – Extra keyword arguments are passed toaiohttp.ClientSession.requestoraioftp.Client.contextdepending on the protocol.
- retry(results)[source]¶
Retry any failed downloads in a results object.
Note
This will start a new event loop.
- Parameters
results (
parfive.Results) – A previous results object, the.errorsproperty will be read and the downloads retried.- Returns
parfive.Results– A modified version of the inputresultswith all the errors from this download attempt and any new files appended to the list of file paths.
- async run_download()[source]¶
Download all files in the queue.
- Returns
parfive.Results– A list of files downloaded.
- classmethod simple_download(urls, *, path='./', overwrite=None)[source]¶
Download a series of URLs to a single destination.
- Parameters
urls (iterable) – A sequence of URLs to download.
path (
pathlib.Path, optional) – The destination directory for the downloaded files. Defaults to the current directory.overwrite (
bool, optional) – Overwrite the files at the destination directory. IfFalsethe URL will not be downloaded if a file with the corresponding filename already exists.
- Returns
parfive.Results– A list of files downloaded.