Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated for access to current API #240

Closed
wants to merge 7 commits into from
Closed

Updated for access to current API #240

wants to merge 7 commits into from

Conversation

skittlesvampir
Copy link

@skittlesvampir skittlesvampir commented Aug 21, 2024 •
edited
Loading

I'm not that experienced with JavaScript and especially Firefox addon development, so feel give criticism if my code should be inadequate.

I've tested this code with 5 different libby audiobooks and they all downloaded flawlessly. This pull request should resolve #206 completely, for now.

@skittlesvampir
Updated for access to current API
00bb0f7 
I'm not that experienced with JavaScript and especially Firefox addon development,
so feel give criticism if my code should be inadequate.
@skittlesvampir
Copy link
Author

Also, is there any possibility of bringing this addon back to the mozilla store?

@jahlborn jahlborn mentioned this pull request Aug 27, 2024
Open
@melangeaddict
Copy link

melangeaddict commented Aug 27, 2024 •
edited
Loading

Edit: I was trying to load the wrong path. I loaded the correct one in /dist/manifest.json and it started working! Well done. :-)

I pulled your changes, built them, and attempted to run the extension. Clicking the 'Download' button didn't result in any actions, though. Were there any special commands to build besides running 'make'?

@zorin1
Copy link
Contributor

zorin1 commented Aug 30, 2024

I was able to download one book with the changes. Seems to work BUT I'm not sure if the length of the mp3 is set correctly. Not sure if this has to do with the program or something else going on. mp3split is not getting the length of the mp3 correctly. When I load in the cue file to mp3split it does not get the last few mins of the audiobook. I had to manually delete the the last line in the mp3split split file and add it with the right ending. I need to try with more audiobooks.

@zorin1
Copy link
Contributor

zorin1 commented Aug 30, 2024

I tried another audiobook and it worked fine. It must have just been something messed up with that one book. Hopefully, someone else can try a few books.

@skittlesvampir
Copy link
Author

It must have just been something messed up with that one book.

Could you please try to download the mentioned book using the workflow described by one of the odmpy users:
ping/odmpy#69 (comment)

I must admit that I don't really have the skills to investigate mp3split problems but maybe the issue is on Libby's site? But if you can confirm, that the Overdrive Download Console results in a correct file, I might be able to have another look.

@poehlert
Copy link

So I'm seeing your parsing code in extractBookJson on 174 fail when splitting on '--' fails. I'm not sure why, but a small percentage of books I'm seeing having a different encoding. An example "spine" element:

{
      "path": "%7B788690A8-7408-42EB-854B-382782723FBF%7DFmt425-Part02.mp3?cmpt=eyJzcGluZ-22e0e21503d6cfedd5b66bb1e525561670b6815fSI6MX0%3D-",
      "media-type": "audio/mpeg",
      "audio-duration": 4082.20725,
      "audio-bitrate": 64,
      "-odread-spine-position": 1,
      "-odread-file-bytes": 32657658,
      "-odread-original-path": "{788690A8-7408-42EB-854B-382782723FBF}Fmt425-Part02.mp3"
    },

My guess is in the originating code, these are used for padding and aren't intended to be delimiters? That might make the pad on the first field to be - or --, and the pad on the second field may be empty or -. I was thinking a regex to parse the path may be more useful?

/(.*\.mp3)\?cmpt=([\w+\/%]*)--?([\w%]*)-?/

Would match on the 3 groups you need and cover both forms (and potentially make additional forms easier to deal with if more encodings are found)

poehlert
poehlert reviewed Aug 31, 2024
View reviewed changes
src/background.ts Outdated
const path_split = path.split("?cmpt=");
const parameter = path_split[1].split("--");
parameter[0] = btoa(`{"spine":${i}}`);
parameter[1] = parameter[1].substring(0,40);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This right here fails as not all paths have -- as I noted in my description.

@skittlesvampir
Copy link
Author

@poehlert I added another commit which should address your issue, but my libby doesn't work right now for some reason ("you can't open this magazine using your card", but it's an audiobook?!). Could you please test if this works?

@zorin1
Copy link
Contributor

zorin1 commented Sep 2, 2024

I tested the new patch with 2 audio books, seems to work fine.

@zorin1
Copy link
Contributor

zorin1 commented Sep 2, 2024

It must have just been something messed up with that one book.

Could you please try to download the mentioned book using the workflow described by one of the odmpy users: ping/odmpy#69 (comment)

I must admit that I don't really have the skills to investigate mp3split problems but maybe the issue is on Libby's site? But if you can confirm, that the Overdrive Download Console results in a correct file, I might be able to have another look.

I was using Kim Stanley Robinson Aurora. When I use the overdrive app it breaks it up into individual tracks. With the Libby Downloader it is all in one mp3 files. On linux, I'm using mp3splt-gtk 0.9.2 to split the mp3 file. It cuts off the last part of the last chapter before the credits. If I listen to the big mp3 file that was downloaded, it is fine. The problem seems to be with the mp3splt program where it is not calculating the length of the mp3 files. If I manually change the last splitpoint then it works.

Chapter 7 starts at 14:28.56 and ends at 16:56:03.

mp3splt is not calculating the end length of 1016 mins and 3 seconds.

MediaInfo show it to be 16 h 56 min.
VLC show it as 16:56:21

The issue has to be with mp3splt

@zorin1
Copy link
Contributor

zorin1 commented Sep 2, 2024

It must have just been something messed up with that one book.

Could you please try to download the mentioned book using the workflow described by one of the odmpy users: ping/odmpy#69 (comment)
I must admit that I don't really have the skills to investigate mp3split problems but maybe the issue is on Libby's site? But if you can confirm, that the Overdrive Download Console results in a correct file, I might be able to have another look.

I was using Kim Stanley Robinson Aurora. When I use the overdrive app it breaks it up into individual tracks. With the Libby Downloader it is all in one mp3 files. On linux, I'm using mp3splt-gtk 0.9.2 to split the mp3 file. It cuts off the last part of the last chapter before the credits. If I listen to the big mp3 file that was downloaded, it is fine. The problem seems to be with the mp3splt program where it is not calculating the length of the mp3 files. If I manually change the last splitpoint then it works.

Chapter 7 starts at 14:28.56 and ends at 16:56:03.

mp3splt is not calculating the end length of 1016 mins and 3 seconds.

MediaInfo show it to be 16 h 56 min. VLC show it as 16:56:21

The issue has to be with mp3splt

I tried mp3splt from the command line and it worked fine. Go figure. Just one of those things.

@skittlesvampir
Copy link
Author

When I use the overdrive app it breaks it up into individual tracks. With the Libby Downloader it is all in one mp3 files.

Have you tried disabling "Parse Chapters" that downloads individual files, although I'm not sure if each file corresponds to one chapter

@zorin1
Copy link
Contributor

zorin1 commented Sep 4, 2024

@poehlert @skittlesvampir Do you know of an audiobooks that was failing before the latest patch? I would like to use it for a test to see if I get any issues. I don't have a lot of audiobooks to test with. I think the two PRs are good enough to be approved. The current code is not working at all.

@bookbonobo Do you think you can accept the 2 PRs?

@melangeaddict
Copy link

I pulled your latest changes and downloaded 3 books with no issues. +1 from me.

@JCroese
Copy link

JCroese commented Sep 7, 2024 •
edited
Loading

I tried both commits, and with most books they work, however I found one that does not work with both.

If I look at the URL from the networks tab of the web develope toold I find the following URL for the audio source:
[dewey part] %7BC4DC40D4-5799-494D-89E5-3E207576F914%7DFmt425-Part01.mp3?cmpt=eyJzcGluZSI6MH0%3D--99a7f9d1b137072ae9e62bb76e4574ef2ce08e59

commit: 00bb0f7 gives the flowing error in the pop-up and the extension debugging page:
"TypeError: n[1] is undefined"

here's an example spine element:

{"path":"%7BC4DC40D4-5799-494D-89E5-3E207576F914%7DFmt425-Part01.mp3?cmpt=eyJzcGluZ-99a7f9d1b137072ae9e62bb76e4574ef2ce08e59SI6MH0%3D-",
"media-type":"audio/mpeg",
"audio-duration":3228.55175,
"audio-bitrate":64,
"-odread-spine-position":0,
"-odread-file-bytes":25828414,
"-odread-original-path":"{C4DC40D4-5799-494D-89E5-3E207576F914}Fmt425-Part01.mp3"}

commit: 1141790 gives the following error in the popup:
"Loading state failed"
and in the extension debugging page:
"DOMException: The buffer passed to decodeAudioData contains an unknown content type."

here's an example spine element:
{"path":"%7BC4DC40D4-5799-494D-89E5-3E207576F914%7DFmt425-Part01.mp3?cmpt=eyJzcGluZ-99a7f9d1b137072ae9e62bb76e4574ef2ce08e59SI6MH0%3D-",
"media-type":"audio/mpeg",
"audio-duration":3228.55175,
"audio-bitrate":64,
"-odread-spine-position":0,
"-odread-file-bytes":25828414,
"-odread-original-path":"{C4DC40D4-5799-494D-89E5-3E207576F914}Fmt425-Part01.mp3"}

As you can see the spine elements get decoded the same way. I noticed that it seems compared to the correct URL the end of the spine one seems scrambeled.
original: eyJzcGluZ SI6MH0%3D- -99a7f9d1b137072ae9e62bb76e4574ef2ce08e59
order: 1 : 2 : 3
spine: eyJzcGluZ -99a7f9d1b137072ae9e62bb76e4574ef2ce08e59 SI6MH0%3D-
order: 1 : 3 : 2

Hope this helps

@brad
Copy link

brad commented Sep 9, 2024 •
edited
Loading

Just commenting to add more data points. I used this branch to download 1 4 books, worked almost perfectly. I had one download failure but I had a poor WiFi connection at the time. I'm assuming that was the cause because it work perfectly on the second try closer to the router

@skittlesvampir
Copy link
Author

@JCroese I can't find anything unusual with the data you posted. Could you please send me the MP3 requests from the network tab, both with and without using the extension? My hypothesis is, that the URLs are generated correctly but another error is happening elsewhere in the code.

@mokrates
Copy link

I can second that this works

@skittlesvampir
Copy link
Author

@eric10k93 The request I'm looking for begins with "dewey-", then has something like "{AEBTD-3243E-122EE}.mp3?cmpt=". I need that request once from downloading with the extension and once how it gets loaded during regular playback.

@skittlesvampir
Copy link
Author

Unrelated, literally every audiobook that I borrow gives the following error when opening:

Access to this magazine is not allowed with your active library card.

I can listen on overdrive.com in the browser but I can access them over libbyapp.com, not even on my phone. I have tried logging out and in again and verifying my card. Have I been shadowbanned?

JCroese
JCroese reviewed Sep 15, 2024
View reviewed changes
src/background.ts Outdated
const parameter_1 = btoa(`{"spine":${i}}`);
const parameter_2 = path_split[4].substring(0, 40);

const parameter_full = encodeURIComponent(`${parameter_1}${path_split[3]}${parameter_2}`);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part fails as all paths eventually need the "--" in there. For those who have the url "scrambeled" in the decoded spine the regex makes path_split[3] sometimes "-" and other times "--"

I would suggest the following edits:

171 const regex = /(.*\.mp3)\?cmpt=([\w+/%]*)(--?)([\w%]{40})([\w%]*-?)/;
...
176 const parameter_full = encodeURIComponent(`${parameter_1}--${path_split[4]}`);

having the regex in line 171 check for 40 characters avoids the need of line 174
the edit of line 176 fixes the problem however it hard codes the "--" part of the url. There might be a neater and more dynamic way of fixing this. e.g. by checking the contents of path_split[3] or the existence of path_split[5]

Copy link

@JCroese JCroese Sep 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skittlesvampir, I found where the code went wrong

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@poehlert can you check if this would fix the issue with the book you couldn't download as well?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Skittlesvampir, I was able to download 3 books.

@JCroese
Copy link

JCroese commented Sep 15, 2024 •
edited
Loading

@eric10k93 and @skittlesvampir I started a code review that solves this issue 5 days ago, I just forgot to submit it... sorry, I'm not that familiar with github.

But as I'm quite new to this let me share what I think I learned last week and how I found where the error was. and if I made a mistake please correct me!

So apparently there are (at least?) two kinds of web developer tools windows. One that shows stuff that's going on with the webpage (that's the one that opens with F12) and one that shows what's going on with extension (you can open this via add-ons manager -> Debug Add-ons -> the "inspect" button from the add-on).

When looking at the Networks tab of the "F12 tools" you can search for .mp3 and you'll find a (list of ) link(s) with the following format:
[Dewey part]/[long string 1]Fmt425-Partxx.mp3?cmpt=[longs string 2]--[long string 3]

I think/guess "long string 1" is a book identifier, "long string 2" some account/library/application identifier, and "long string 3" might be some kind of key/pass/cookie etc. code.

When you look at the "Inspect tools" in the Console tab, you can read the messages the extension logs/print. and in the "Network tab" you can see the web request send/received by the extension

Now to some of the code:

in background.ts the function "handleBookMetaWebRequest" (~line 208) catches the response of some specific web requests including the last "case"/"else if" which is the one that contains the ToC en Spine meta data.

Before issue #206 the response of this request was a json file which contained all of this information without encoding/encryption (I guess?). However now part of this json file has a large set of data which is encoded.

The "extractBookJson"(~line 102) function decodes this data to get the ToC en Spine data back, it follows the steps proposed by username1233414125 [How (s)he figured out these steps is still a mystery to me if someone knows the explanation of this I'd love to hear it]

next the "handleTitle" function (~line 131) extracts the required data from the ToC and Spine. In line 136 this function logs some of the source json to the console. when you look at this printed information some times the mp3 urls are shown as:

[Dewey part]/[long string 1]Fmt425-Partxx.mp3?cmpt=[long string 2]--[long string 3]

other times it shows:

[Dewey part]/[long string 1]Fmt425-Partxx.mp3?cmpt=[first part of long string 2]-[long string 3][last part of long string 2]-

in commit 00bb0f7 this second type failed because the code was looking for "--" and since this wasn't there it couldn't find al the parts needed to build the url.

then in commit 1141790 this second type gave a error because even though with the new regex it did find "long string 3" however when building the url it used only "-" instead of "--"which gave an invalid url.

I found this because the networks tab of the "Inspect tools" showed a url that was build like this"

[Dewey part]/[long string 1]Fmt425-Partxx.mp3?cmpt=[long string 2]-[long string 3]

So changing "${path_split[3]}" from line 176 into "--" will fix the problem. As path_split[3] will have the value '-' for the second case and thus generate an incomplete/incorrect url

@zorin1
Copy link
Contributor

zorin1 commented Sep 19, 2024

@JCroese What was the code change that you made? I only see 2 commits or are you waiting for @skittlesvampir to make the change?

fixed incorrect url generation for mp3 files whose
5bfa933 
path info is "scrambled" in de decoded spine
@JCroese
Copy link

JCroese commented Sep 19, 2024

@zorin1 to avoid duplicating pull request, I made one for the change to @skittlesvampir's repository, when he accepts, he can add that commit to this pull request.

@skittlesvampir
Merge pull request #1 from JCroese/main
5736149 
fixed incorrect url generation for mp3 files whose
@skittlesvampir
Copy link
Author

@JCroese Sorry, I didn't have much time lately and didn't see your pull request. It's merged now.

@mjpking
Copy link

mjpking commented Sep 26, 2024 via email

@JCroese
Copy link

JCroese commented Sep 26, 2024 •
edited
Loading

@eric10k93 it seems like you encountered yet another way of scrambling the url, not yet handled by the previous regex. Because for this url version there is no "-" or "--" in the middle, path_split[4] will be empty and thus the error is raised.

As I have no smart idea right now how to edit the regex, to include this option as well, adding an if statement to catch this version serves as a quick fix. commited here: bc100df

tested on books with all 3 url versions

Jared Croese and others added 2 commits September 27, 2024 10:37
Reviewed and updated regex
a8f3673
@skittlesvampir
Merge pull request #2 from JCroese/main
5328d24 
Added if statement to handle urls ending with "--"
@JCroese
Copy link

JCroese commented Oct 2, 2024

Is it just me or did they change the place of the spine information again? Cause for me the extension is currently not working, the script fails when trying to find the embedded encoded string

@mjpking
Copy link

mjpking commented Oct 4, 2024 via email

@zorin1
Copy link
Contributor

zorin1 commented Oct 8, 2024

@poehlert @skittlesvampir @JCroese Anyone figure out a solution yet?

@JCroese
Copy link

JCroese commented Oct 8, 2024 •
edited
Loading

@zorin1 nothing solid yet. The only thing I might have found is that the html response of the https://dewey-[string1].listen.libbyapp.com/?m=[string2] request has a <script> section which half way through has some encoded data starting with "window.eData=[" that might be some type of comma separated values, However I have know clue at all what type of encoding was used.

@skittlesvampir
Copy link
Author

Looks to me like they just straight blocked the Firefox browser... I keep getting the error "access to this magazine is not allowed with your active library card"... funny error, since they are audiobooks, not magazines, and I can access everything fine on Edge browser.

EDIT: Scratch that... I get the error regardless of browser.

@eric10k93 is there any way I can contact you in private?

@flyswimr
Copy link

On mine it is all browsers, appears to be blocked at the library card level, can you confirm?

Looks to me like they just straight blocked the Firefox browser... I keep getting the error "access to this magazine is not allowed with your active library card"... funny error, since they are audiobooks, not magazines, and I can access everything fine on Edge browser.
EDIT: Scratch that... I get the error regardless of browser.

@eric10k93 is there any way I can contact you in private?

I just made sure my email is public on my profile. Feel free to contact me there first.

@skittlesvampir
Copy link
Author

@flyswimr yes, can confirm. my discord is deeznutsbrother#2906 if you want to contact me

@flyswimr
Copy link

flyswimr commented Dec 3, 2024

@skittlesvampir I don't have discord..seems like the only "fix" is to physically go to the library and request a new library number. Libby/Overdrive support either doesn't know or plays dumb, and often times refers one back to their library for help. I drove to mine, told/showed them what's happening using libbyapp.com and mentioned that a friend was told by Overdrive to do just what I was requesting (a new library card number). They obliged and am back working. But not using any downloader. Re-recording using audacity is my current best solution. We all just want to put books on smaller devices like Sandisk/help older relatives (that don't want to use their phones either), that's it.

@skittlesvampir
Copy link
Author

@flyswimr damn, that seems bad
since all my changes are obsolete again, i will close this pull requests
further efforts seem dangerous and utopian, i advise everyone to be careful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mjpking mjpking mjpking left review comments

@JCroese JCroese JCroese left review comments

@poehlert poehlert poehlert left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Loading State Failed
9 participants
@skittlesvampir @melangeaddict @zorin1 @poehlert @JCroese @brad @mokrates @mjpking @flyswimr