View Proxy Requests #18
Comments
|
Hi. Using update: v0.20.2 adds these log statements, which are displayed when using |
|
thank you! |
|
@warren-bank Updated but still having issues. I think I have it pinpointed but not sure if I am right or not with my theory. Are you on discord or any platform to discuss my current issue? |
|
Sorry, no.. I'm not on any social/chat sites.. I'm fairly anti-social, but if you'd like to post the URL for a manifest that isn't proxying well.. I'll give it a quick look and identify the issue. |
|
As it turns I don't think it is an issue with the manifest.. I have been overlooking the fact that it is encrypted with Sample-AES encryption. If you still want to look, I can paste it in here |
|
@warren-bank This is the url FWIW dcs-live-uc1.mp.lura.live/server/play/m10rvaXGcos8a0gx/rendition.m3u8?track=video-3&anvsid=m177626904-nae80564d887cda81645dfbafb7b9bbdf&ts=1667246194 |
|
observations..
|
|
I thought that was the key but was unsure as I couldn’t find and documentation on the manifest parameters for Apple FairPlay DRM. I know that certain DRM is troublesome according to most… any browsers that you could point me towards that would be compatible? I’ll keep searching in the mean time.
|
|
I'd guess that Safari would be able to play this.. since:
but.. I don't know for certain; |
|
@warren-bank I just put it in my iPhone and it played an Ad that was currently live in the player on my test device. See the pasted manifest & headers below. What I can assume from this is that it played the "targeting" .ts video fine because that did not have DRM (see key value is also now set to "none"). Once it kicked back into the content delivered from the 0300801-mp-lura-live.ctl domain was when I got a black screen. HTTP/1.1 200 OK #EXTM3U |
|
v0.20.3 is pushed to npm and updates the regex to prevent any change to the KEYFORMAT. so.. to summarize.. if the manifest uses FairPlay DRM (w/ |
|
PS: the structure of that long manifest can be summarized as.. takeaways:
|
|
is there a way to include a cookie in the http request? if I wanted to add |
|
yes, but not on a per-manifest basis.. the cookie would be sent with all outbound requests:
|
|
update: I thought it would be a good idea to support adding request headers (or more generally.. request options) that are conditional on the URL being requested.. so I just released v0.21.0 which adds 2x new hooks. The readme includes a description, but here's a super fast example: // hooks.js
module.exports = {
add_request_options: (url, is_m3u8) => ({headers: {
"x-abc": "foo"
}}),
add_request_headers: (url, is_m3u8) => ({
"x-xyz": "bar"
})
}Of course, these functions aren't applying any conditional logic.. but you can see how easy it would be to do so. Falsy return values are ignored. For the purpose of adding headers, only add_request_headers is needed; |
|
Is it possible instead of a header flag to append every URL request with my cookie? I have not been able to get it functioning yet when using the header to set the cookie. The host requires that hmac cookie when requesting the master.m3u8 file AND when requesting the .ts segments. I realized after some testing that I could append the value at the end of the url like so...
or
... and it would give me access to the requested endpoint |
|
well.. if we're no-longer talking about adding conditional headers.. but rather, we now want to add conditional querystring parameters to URLs.. you could use the "redirect" hook: // hooks.js
module.exports = {
redirect: (url) => {
if (url.toLowerCase().indexOf('://some.host.com/') >= 0) {
const querystring = 'hdntl=exp=1667601490~acl=%2f*~data=hdntl~hmac=cd6a3355902f727bed499565caa7e0564c97ea6b8456adc498e7653258397532'
url += (url.indexOf('?') >= 0) ? '&' : '?'
url += querystring
}
return url
}
}I should note that the value of |
|
Yeah I noticed it appeared in other links to work that way. Not sure why a cookie parameter could also be passed in that method for authentication (seems like a vulnerability to me) but it works 🤷🏼♂️
|
|
In reference to a separate site from above...
That is a snip from a manifest.m3u8 file. The fragments are constantly concatinated to the file rather than reloading a new set each time. Is there a method for hls proxy to recognize that and keep a base url but add that to it? |
|
that format doesn't look familiar to me.. |
|
so the video fragment is located at
With a validated hmac in the request, it will respond with the video segment (fragment) as the response. The endpoint which returns that is slightly different at
|
|
well.. if the video host responds to the request: wget -O '2477251416062000.ts' 'https://domain.akamaized.net/ch12/c12b13f7-32c3-4280-a71f-23ed88237a2a/c12b13f7-32c3-4280-a71f-23ed88237a2a.ism/exp=1667833586~acl=%2f*~data=hdntl,3cad4b4c-eb4f-4603-bfb1-5e5beba4ea0f~hmac=8e774d06f7edbc645f1ab3c9053f9d563227d04b6c87d63bf3704fa971bb8ca4/QualityLevels(4749952)/Fragments(video=2477251416062000,format=m3u8-aapl-v3,audiotrack=audio)'with: and the .m3u8 uses a relative URL to reference this path: then the way that the .m3u8 is (currently) parsed to extract URLs won't match these relative URLs.. the hooks that are (currently) available also can't help in this situation, having said that.. it might be a good idea to add one. // hooks.js
module.exports = {
modify_m3u8_content: (m3u8_content, m3u8_url) => {
const base_url = m3u8_url.replace(/[^\/]*$/, '')
return m3u8_content.replace(
/^(Fragments\([^\)]+\))$/mg,
(m0, m1) => base_url + m1
)
}
} |
|
actually.. that still wouldn't solve the problem.. |
|
though.. this might work; // hooks.js
module.exports = {
modify_m3u8_content: (m3u8_content, m3u8_url) => {
const base_url = m3u8_url.replace(/[^\/]*$/, '')
return m3u8_content.replace(
/^(Fragments\([^\)]+\))$/mg,
(m0, m1) => base_url + m1.replace(/[,]/g, '%2C')
)
}
} |
|
test to confirm: {
const regexs = {}
regexs.urls = new RegExp('(^|(?<!(?:KEYFORMAT=))[\\s\'"])((?:https?:/)?/)?((?:[^\\?#/\\s\'"]*/)+?)?([^\\?#,/\\s\'"]+?)(\\.[^\\?#,/\\.\\s\'"]+(?:[\\?#][^\\s\'"]*)?)?([\\s\'"]|$)', 'img')
let m3u8_content = `
#EXTINF:6.006000,no-desc
Fragments(video=2477251416062000,format=m3u8-aapl-v3,audiotrack=audio)
`
const m3u8_url = 'https://domain.akamaized.net/ch12/c12b13f7-32c3-4280-a71f-23ed88237a2a/c12b13f7-32c3-4280-a71f-23ed88237a2a.ism/exp=1667833586~acl=%2f*~data=hdntl,3cad4b4c-eb4f-4603-bfb1-5e5beba4ea0f~hmac=8e774d06f7edbc645f1ab3c9053f9d563227d04b6c87d63bf3704fa971bb8ca4/QualityLevels(4749952)/index.m3u8'
{
const base_url = m3u8_url.replace(/[^\/]*$/, '')
m3u8_content = m3u8_content.replace(
/^(Fragments\([^\)]+\))$/mg,
(m0, m1) => base_url + m1.replace(/[,]/g, '%2C')
)
}
m3u8_content.replace(regexs.urls, function(match, head, abs_path, rel_path, file_name, file_ext, tail) {
console.log(JSON.stringify({match, head, abs_path, rel_path, file_name, file_ext, tail}, null, 2))
})
void(0)
}output: {
"match": "\nhttps://domain.akamaized.net/ch12/c12b13f7-32c3-4280-a71f-23ed88237a2a/c12b13f7-32c3-4280-a71f-23ed88237a2a.ism/exp=1667833586~acl=%2f*~data=hdntl,3cad4b4c-eb4f-4603-bfb1-5e5beba4ea0f~hmac=8e774d06f7edbc645f1ab3c9053f9d563227d04b6c87d63bf3704fa971bb8ca4/QualityLevels(4749952)/Fragments(video=2477251416062000%2Cformat=m3u8-aapl-v3%2Caudiotrack=audio)\n",
"head": "\n",
"abs_path": "https://",
"rel_path": "domain.akamaized.net/ch12/c12b13f7-32c3-4280-a71f-23ed88237a2a/c12b13f7-32c3-4280-a71f-23ed88237a2a.ism/exp=1667833586~acl=%2f*~data=hdntl,3cad4b4c-eb4f-4603-bfb1-5e5beba4ea0f~hmac=8e774d06f7edbc645f1ab3c9053f9d563227d04b6c87d63bf3704fa971bb8ca4/QualityLevels(4749952)/",
"file_name": "Fragments(video=2477251416062000%2Cformat=m3u8-aapl-v3%2Caudiotrack=audio)",
"tail": "\n"
} |
|
v0.22.0 was just pushed to npm.. and it adds this new hook method. I tested the examples given earlier.. and they work as expected. Here is a more complete example, which uses the //hooks.js
module.exports = {
modify_m3u8_content: (m3u8_content, m3u8_url) => {
const base_url = m3u8_url.replace(/[^\/]*$/, '')
return m3u8_content.replace(
/^(Fragments\([^\)]+\))$/mg,
(m0, m1) => base_url + m1.replace(/[,]/g, '%2C')
)
},
redirect: (url) => {
return url.replace(
/^(.+\/)(Fragments\([^\)]+\))$/,
(m0, m1, m2) => m1 + m2.replace(/%2C/g, ',')
)
}
} |
|
So for hook usage in my case - I would create the hooks.js file & then call that with --hooks modify_m3u8_content in the command to start hlsd? |
|
|
|
@warren-bank Had to take a break last week but I am getting back to my project here. After doing some digging it looks like the platform originating the stream is utilizing Azure Media Service with Apple HLS v3 https://learn.microsoft.com/en-us/azure/media-services/previous/media-services-deliver-content-overview <-- is a great reference. If you have time could you check that and let me know if hls-proxy would still be suitable for my project with this delivery? Edit: Also I found out that the Manifest(video,format=m3u8-aapl-v3,audiotrack=audio,filter=hls) url will respond if .m3u8 is appended. Not sure if that helps in any way |
|
@warren-bank you can also find a test endpoint that microsoft hosts here. You can see the first manifest returns the manifest with quality/bandwidth values... That can be bypassed as it can be hardcoded. The next m3u8 file to grab is located at... The last request is for a fragment from the The difference between this test environment and mine is that the test urls have a defined end of list while the streamed content in the environment I am working with has no predetermined end of list. |
|
@warren-bank Is this still being followed? |
|
Admittedly, I haven't given this issue any thought. I just did some reading.. on these URLs.. which are produced by videos hosted by Microsoft Azure Media Services.. and inspecting the manifest that you referenced: As I see it, I have 2 options:
Option 2 sounds like a good idea. Of course, it could also completely break the app.. if not careful. I'll probably play around with this idea over the next few days. |
|
notes.. |
|
I have created a companion script that will actively run in the background along side hls. It's functionality is to get an active session token (hmac) for the endpoints that require them. I haven't implemented database storage at the time for those values but that is the plan. As of right now they are just written to json objects and stored locally. |
|
Feasibly that companion script could be used alongside a hook I guess? To set parts of the url as a variable that requests those values from the local storage? As far as detection goes, the only way I see it possible would be for you to detect ".ism" within the link and use a function dedicated to the azure media services delivery platform |
|
quick update.. I finally gave this repo some attention and did a ton of internal housekeeping, refactoring, and rewriting. I like where it is now.. so unless I'm told that I broke something.. I'll probably leave it alone (at v3.0.0) for now. regarding the Azure/PlayReady manifests.. the proxy (with its completely new manifest parser) can now successfully detect and encode its URLs. That said, I couldn't easily find a video player that supports PlayReady.. but the stream plays the same for me (ie: artifacts on the screen because the stream is scrambled) when using the proxy as it does when playing the stream directly (in VLC on Win64, and in ExoPlayer on Android). So.. if you have a video player that supports PlayReady.. I'm fairly certain it'll work through the proxy. |
|
https://ampdemo.azureedge.net/ looks to be a good demo site. I will play around with the new update today! Is it the same usage command to start up? |
|
Disregard previous question about usage. May need a little debugging if this is not an issue solely on my end. If my top level manifest link is
Then the next phase is a bandwidth determination. Since those bandwidth levels are static, could we hardcode that value to say something like
Which would now make our Fragment manifest...
The proxy in 3.0.0 does fine when it is directly calling a Fragment URL I.e. But if I call the full fragment manifest it just returns that manifest rather than parsing those fragments into a fetched stream |
|
the only change to user space is that file extensions appended to the encoded URL are now very important, whereas previously they were optional. previously, the as far as your comment about video "fragments".. the manifest is modified so the url to each individual video segment (ie: "fragment") is rewritten to be directed through the proxy (with the original url being base64 encoded and a and when each video segment is requested through the proxy by a client.. it is retrieved (either by requesting it from the origin server or by obtaining it from the internal cache) and returned in the response. |
|
Maybe I had its functionality misconstrued but what I thought the proxy did would be to tunnel all traffic through a central location thus it would turn Into Right? |
|
OK, yes.. that is correct. Here is a quick example to walk through its behavior using the example (master) manifest: The corresponding encoded URLs to proxy these endpoints are: Where the modified manifests are: manifest-master.m3u8manifest-child.m3u8 |
|
well.. actually "yes" and "no".. your example has problems. this: ..is not a valid URL to make a request to the proxy as the readme describes, the format of a proxied URL is: proxy_url='http://127.0.0.1:8080'
video_url='https://amssamples.streaming.mediaservices.windows.net/b6822ec8-5c2b-4ae0-a851-fd46a78294e9/ElephantsDream.ism/QualityLevels(4678229)/Fragments(video=600000000,format=m3u8-aapl-v3,audiotrack=AAC_und_ch2_56kbps)'
file_extension='.ts'
hls_proxy_url="${proxy_url}/"$(echo -n "$video_url" | base64 --wrap=0)"$file_extension"though, I personally always use this webpage to do all of the hard work for me. notes:
|
|
I see. I will give it another test run on my end tomorrow. Looks like it was an error on my end as I did not know the file extensions were required as of latest update |
|
Working great! Looks like the website uses user-agent strings as a form of security validation. If I wanted to write a hook to fetch the user agent from a json file, what would that look like in my command usage? and would the hook look like this...
|
|
unless you need to configure different user-agent request headers for different video hosts (which is unlikely), you could just configure a global value that is included in all requests: hlsd --useragent 'Chrome/100' ...but.. if you really do need to apply logic on a per-request basis, then either of these 2x hooks could be used:
..the signatures of each hook is described in the readme |
When I am using the proxy to load a m3u8 playlist, my end client ends with a blank screen. Is there a way to view the requests & responses from the http server that hls-proxy runs on to see what the requests & response codes look like in real time? I have charles/burp but clearly do not get the requests being made from the http server itself.
This blank screen is only happening on a certain stream. Others come through without issue. Thanks!
The text was updated successfully, but these errors were encountered: