Commit Graph

5 Commits

Author SHA1 Message Date
Adeeb Shihadeh
c64a8356fe move to github container registry + CI speedup (#19621)
old-commit-hash: 59d7c5e64a
2020-12-31 14:57:01 -08:00
Willem Melching
1a2893345a add check for partial response code (#2282)
old-commit-hash: 529e9a23ea
2020-10-07 12:14:22 +02:00
Greg Hogan
3ea69f614e eliminate round-trip reading entire file (#2275)
* eliminate round-trip reading entire file

* only if reading entire file

* print headers in errors

* fix end
old-commit-hash: 2293a6a36c
2020-10-05 16:59:19 -07:00
grekiki
398796a010 FileReader: cache files locally (#2170)
* framework for updates

* File caching

* update .gitignore

* update .gitignore

* Store in home tmp directory

* fix bugs in old code, add tests

* add test

* revert bug

* Fixed some bugs, more testing

* doing some style fixes

* Update caching with lenghts

* doing some style fixes

* doing some style fixes

* doing some style fixes

* subrepositories

* try to fix the tests

* change submodules back

* doing some style fixes

* if we don't cache, we don't cache length either

* fix curls

* make sure tests run in ci

* Cleanup tests

* Use int for cache flag

* fix linter

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 05b64257df
2020-09-17 13:03:34 +02:00
Willem Melching
9c291968f6 Speedup URLFile (#1888)
* add parallel download support to URLFile

* make python 3.8 happy

* Fix chunk size

* Automatic number of threads

* No daemon threads in unlogger

* Cache length

* dont touch old filereader

* Remove debug info

* remove debug script

* Ignore type
old-commit-hash: c70700758d
2020-07-20 17:10:08 +02:00