-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Opusfilter filters #47
Conversation
One thing I noticed was that opusfilter effectively loads all its dependencies on start. For long running processes this makes sense (detect import errors early on etc.) but for this runner it is a bit annoying. Maybe there is an easy way to hook into importlib and only load a proxy module for some modules, and only when accessing of any of its components is noticed we load the actual module. |
Is it some specific dependencies that cause trouble? I guess we could try to use dynamic import for those. |
I'm not sure which bit of it is causing it. I'm using importlib's LazyLoader at the moment, and that helps to shave it down to half but start-up time on my laptop is still 2.5 seconds. I haven't looked at hooking up some measuring callbacks to lazyloader. I also imagine there are off-the-shelf tools out there that can help us measure and break down import times per package. |
Well, there was quite a few, with Edit: Now it's in release 2.6.0. |
They need a bit more logic to work well
Because there is no way to type the difference between an empty string and `None`, we should just treat them the same. I should also do this for the other types probably.
a82be5e
to
86b742f
Compare
Doens't work on macOS right now because mingruimingrui/fast-mosestokenizer#4
# Conflicts: # frontend/src/store/filtersteps.js # main.py
Haven't tested it yet because opuscleaner/empty-train isn't a proper package yet
in theory
# Conflicts: # frontend/yarn.lock # opuscleaner/config.py # opuscleaner/server.py # requirements.txt
Add opusfilters as choices to the list of included filters.