[FEAT] Native listing of http URLs

[jaychia]

Adds capabilities for our HTTP source to perform file listing

Logic is roughly adapted from fsspec's HTTPFileSystem in order to maintain some parity with how it behaves.

Current differences in behavior:

1. We don't support fsspec's simple_links=True option, which would search for any text starting with http(s):// and that aren't encased in a HTML <a href=...> tag.
2. When pointed at a single file, fsspec returns an empty list but we detect whether the file is HTML and return a list with just the file itself if we see that it is not HTML.

Closes: #1440

[codecov]

Codecov Report

Merging #1405 (62b11d3) into main (97845d3) will not change coverage.
Report is 1 commits behind head on main.
The diff coverage is n/a.

❗ Current head 62b11d3 differs from pull request most recent head c459e94. Consider uploading reports for the commit c459e94 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1405   +/-   ##
=======================================
  Coverage   74.61%   74.61%           
=======================================
  Files          60       60           
  Lines        6034     6034           
=======================================
  Hits         4502     4502           
  Misses       1532     1532