tools/lib/filereader.py

import os
from openpilot.tools.lib.url_file import URLFile

DATA_ENDPOINT = os.getenv("DATA_ENDPOINT", "http://data-raw.comma.internal/")

def FileReader(fn, debug=False):
  if fn.startswith("cd:/"):
    fn = fn.replace("cd:/", DATA_ENDPOINT)
  if fn.startswith(("http://", "https://")):
    return URLFile(fn, debug=debug)
  return open(fn, "rb")
eliminate xx filereader (#23514) * eliminate xx filereader * framereader cleanup old-commit-hash: d5f4fdb1dbda7cfabdcc9d2c9342636e56388716 2022-01-13 15:56:54 -08:00			`import os`
add openpilot prefix to imports (#29498) * add openpilot prefix to imports * more * more * fix docs * fix linter * bump submodules * fix patched tests * update dynamic imports * debug * Revert "debug" This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b. * fix pm test old-commit-hash: a9626f95b69af19306143fc4def02fb5769405d2 2023-08-20 20:49:55 -07:00			`from openpilot.tools.lib.url_file import URLFile`
Speedup URLFile (#1888) * add parallel download support to URLFile * make python 3.8 happy * Fix chunk size * Automatic number of threads * No daemon threads in unlogger * Cache length * dont touch old filereader * Remove debug info * remove debug script * Ignore type old-commit-hash: c70700758d9e3227a157d2e9fdcb9d3204b433c2 2020-07-20 17:10:08 +02:00
Update DATA_ENDPOINT default to data-raw.comma.internal (#25597) old-commit-hash: acbb230e0a2887971d906d9ae057ba0a7db44316 2022-08-29 13:31:43 -07:00			`DATA_ENDPOINT = os.getenv("DATA_ENDPOINT", "http://data-raw.comma.internal/")`
eliminate xx filereader (#23514) * eliminate xx filereader * framereader cleanup old-commit-hash: d5f4fdb1dbda7cfabdcc9d2c9342636e56388716 2022-01-13 15:56:54 -08:00
Tools: Storage API (#1161) * filereader * support URLs in filereader, logreader * unused * use route files api; add auth file * Implement browser auth * Update readme, fix up cache paths * Add tests, clear token on 401 * Factor out URLFile * space old-commit-hash: c4af05868ba82d4295e3f508e8477f2f6f898834 2020-02-24 21:24:54 -05:00			`def FileReader(fn, debug=False):`
eliminate xx filereader (#23514) * eliminate xx filereader * framereader cleanup old-commit-hash: d5f4fdb1dbda7cfabdcc9d2c9342636e56388716 2022-01-13 15:56:54 -08:00			`if fn.startswith("cd:/"):`
filereader better name for cd:/ replacement old-commit-hash: a6b46c311e41875612e6d28f6978f4e1d1e20e46 2022-01-13 18:21:19 -08:00			`fn = fn.replace("cd:/", DATA_ENDPOINT)`
Ruff: flake8 pie (misc lints) (#29318) flake8 pie old-commit-hash: af3ed37c5cd8823f793aec60e1c74760cf795bd9 2023-08-10 15:28:51 -07:00			`if fn.startswith(("http://", "https://")):`
Tools: Storage API (#1161) * filereader * support URLs in filereader, logreader * unused * use route files api; add auth file * Implement browser auth * Update readme, fix up cache paths * Add tests, clear token on 401 * Factor out URLFile * space old-commit-hash: c4af05868ba82d4295e3f508e8477f2f6f898834 2020-02-24 21:24:54 -05:00			`return URLFile(fn, debug=debug)`
FrameIterator that uses 1+ GB less RAM (#21687) old-commit-hash: 8f1806602c4e4a70e073bf1abd8f61206864d652 2021-07-22 12:08:56 -07:00			`return open(fn, "rb")`