tools/lib/filereader.py

import os
from tools.lib.url_file import URLFile

DATA_PREFIX = os.getenv("DATA_PREFIX", "http://data-raw.internal/")

def FileReader(fn, debug=False):
  if fn.startswith("cd:/"):
    fn = fn.replace("cd:/", DATA_PREFIX)
  if fn.startswith("http://") or fn.startswith("https://"):
    return URLFile(fn, debug=debug)
  return open(fn, "rb")
eliminate xx filereader (#23514) * eliminate xx filereader * framereader cleanup 2022-01-13 15:56:54 -08:00			`import os`
remove parallel url downloader 2020-09-21 10:32:08 +02:00			`from tools.lib.url_file import URLFile`
Speedup URLFile (#1888) * add parallel download support to URLFile * make python 3.8 happy * Fix chunk size * Automatic number of threads * No daemon threads in unlogger * Cache length * dont touch old filereader * Remove debug info * remove debug script * Ignore type 2020-07-20 17:10:08 +02:00
eliminate xx filereader (#23514) * eliminate xx filereader * framereader cleanup 2022-01-13 15:56:54 -08:00			`DATA_PREFIX = os.getenv("DATA_PREFIX", "http://data-raw.internal/")`

Tools: Storage API (#1161) * filereader * support URLs in filereader, logreader * unused * use route files api; add auth file * Implement browser auth * Update readme, fix up cache paths * Add tests, clear token on 401 * Factor out URLFile * space 2020-02-24 21:24:54 -05:00			`def FileReader(fn, debug=False):`
eliminate xx filereader (#23514) * eliminate xx filereader * framereader cleanup 2022-01-13 15:56:54 -08:00			`if fn.startswith("cd:/"):`
fix filereader cd:/ replacement 2022-01-13 16:09:56 -08:00			`fn = fn.replace("cd:/", DATA_PREFIX)`
Tools: Storage API (#1161) * filereader * support URLs in filereader, logreader * unused * use route files api; add auth file * Implement browser auth * Update readme, fix up cache paths * Add tests, clear token on 401 * Factor out URLFile * space 2020-02-24 21:24:54 -05:00			`if fn.startswith("http://") or fn.startswith("https://"):`
			`return URLFile(fn, debug=debug)`
FrameIterator that uses 1+ GB less RAM (#21687) 2021-07-22 12:08:56 -07:00			`return open(fn, "rb")`