tools/lib/filereader.py

import os

if "COMMA_PARALLEL_DOWNLOADS" in os.environ:
  from tools.lib.url_file_parallel import URLFileParallel as URLFile
else:
  from tools.lib.url_file import URLFile  # type: ignore


def FileReader(fn, debug=False):
  if fn.startswith("http://") or fn.startswith("https://"):
    return URLFile(fn, debug=debug)
  else:
    return open(fn, "rb")
Speedup URLFile (#1888) * add parallel download support to URLFile * make python 3.8 happy * Fix chunk size * Automatic number of threads * No daemon threads in unlogger * Cache length * dont touch old filereader * Remove debug info * remove debug script * Ignore type old-commit-hash: c70700758d9e3227a157d2e9fdcb9d3204b433c2 2020-07-20 17:10:08 +02:00			`import os`

			`if "COMMA_PARALLEL_DOWNLOADS" in os.environ:`
			`from tools.lib.url_file_parallel import URLFileParallel as URLFile`
			`else:`
			`from tools.lib.url_file import URLFile # type: ignore`

merge in tools old-commit-hash: 29ac3da7b84426c6764150cb233b9c7bab1446d8 2020-01-17 10:07:22 -08:00
Tools: Storage API (#1161) * filereader * support URLs in filereader, logreader * unused * use route files api; add auth file * Implement browser auth * Update readme, fix up cache paths * Add tests, clear token on 401 * Factor out URLFile * space old-commit-hash: c4af05868ba82d4295e3f508e8477f2f6f898834 2020-02-24 21:24:54 -05:00			`def FileReader(fn, debug=False):`
			`if fn.startswith("http://") or fn.startswith("https://"):`
			`return URLFile(fn, debug=debug)`
			`else:`
			`return open(fn, "rb")`