Roadmap for freqUI (Freqtrade UI)

[xmatthias]

Roadmap / Next steps:

• Finish implementation of a control interface supporting all options telegram supports
  • blacklist - add element to it done in 5afe652
  • blacklist - remove element (needs to be implemented in main bot first) 9f4c6c6
• Develop deployment mechansim to nicely work together with freqtrade
  • with docker (seperate container, or build into freqtrade itself?)
  • without docker (add the output of the build step to the freqtrade CI and include it in releases, to deliver it via api server from freqtrade itself?) Deploy FreqUI into webserver freqtrade#4285
• Include simple plotting methods (Echarts pairhistory #106)
  • UI will use Apache eCharts - a high performance, multi-purpose graphing library Displaying graphs in frequi #21
• add config editor (needs good consideration first - has security implications as it would expose API-keys via the API)
• support backtesting (incl. graphical result analysis) (requires several changes to freqtrade structure - @xmatthias in theory done - bot part still missing) Visualize backtesting report in frequi #247
• support starting bot in trading mode (requires major changes to freqtrade structure - @xmatthias working on it, but will take a while)
• multi bot control / dashboard (add multi bot support #478)

...

More points will be added to the above list as they pop up.

Please keep discussion in this issue to a minimum to keep it easily readable, but open a new issue if there is something to discuss around one of the above points.

Also - please let me know if you start working on one of the above points so we can avoid duplicated work.

[gaugau3000]

Hi,

How to you see the split between control/monitor/analyse running bots and backtest ?
Because for backtest you can run a temporary webserver on your computer that is able to connect to freqtrade.
But for control/monitor/analyse you should prefer deploy it on a server and see all you bot instances :-) everywhere.

They have both different needs in term of security, plot maybe... and do a different job

Should it be in the same project ?

Have a nice day.

[xmatthias]

Controlling multiple bots from one UI instance is currently not scope of this UI. It may be possible in the future - but for now, it'll be one UI instance per bot.

My final goal is to deploy this project within the freqtrade server (so you run freqtrade webserver - and can then start trading / start backtesting / whatever in that bot instance (the trade part does already work in a test branch of mine...).

Alternatively, you start it as usual freqtrade trade ... - and go to localhost:8080/ui (or something like this) - and get presented with the (deployed) version of this UI.

A "dashboard" version controlling to multiple bots at the same time is currently not planned - but instead a "one dashboard / UI per bot" approach.

[romanarranz]

add config editor (needs good consideration first - has security implications as it would expose API-keys via the API)

Can you elaborate more on this feature? Would that be an implementation of an editor used to set up the JSON config files for a specific bot? If that is the case what are your concerns about the security issues of having their API keys..., as long as the frequi instance prevents external access with an auth layer and such configurations are encrypted with private user keys would be difficult to get them I believe

Moreover, as an alternative, uploading the config file could cover that use case too right, thinking as a short term solution 🤔

[xmatthias]

well at the moment, we only expose a selected number of config options through the API to prevent secret leakage - so not all settings are available in frequi (not even in read-only).

That said - the UI is the least of the problem in this regard - as it's unclear what method should be taken on the backend to persist the settings. Writing a config-file is an option - but what if 2 or more config-files are used (which is supported) - which file would you store it to? the first? second, or into an additional one?

---

Yes, it's protected by a mandatory auth layer - but the problem is the deployment - as i will assume that 70-80% of the users (most users are non-technical, and with a simple UI this will increase) will not deploy frequi / the API correctly.

Every auth-method is susceptible to attacks if not properly secured (at least via TLS) - in addition to this, freqtrade does support both basic and JWT auth for all endpoints - and supporting basic auth doesn't quite increase security.
Also, in my opinion, the API (and frequi) should be only exposed to localhost, and access should go through a SSH tunnel or a VPN Tunnel - but as said above, most of the poeple will not follow this recommendation, either because it's difficult to do, or because it's inconvenient (think access from the phone).

However, we'll always have the requirement to run the python backend "first", so i think it'll be fine to have certain options only available there (for example API keys and such) - and only the "rest" of the config can be made via the UI.

---

An additional problem is the "overridable" settings - which CAN be configured in via cli flag, via configuration, or via strategy (and prevelance is in this sequence) - so you can in theory break a strategy (accidentally or on purpose) by overwriting a stoploss to something it's not designed to work with.

---

none of the above are insurmountable problems - but most are problems that need good thought - and most of the work is in the backend on this topic - creating a quick form to allow setting some settings is not a huge deal - but to process them correctly in the backend might be.

Also - keep in mind that most config-changes will require a bot-restart (depending on the options) - and changing them "on the fly" can cause inconsistencies - so it'd either need to be a "writes directly to file WITHOUT reading them immediately" - followed by a /reload_conf call, or the settings can only be changed while the bot is in a certain mode.

that said - i'm playing with a "webserver" mode - which starts only the API (without the remainder of the bot) - and then starts the bot via UI - which might be a solution to this.
however, i'm not trusting that implementation just yet - so it's kinda far from prime-time - and other things are more important than this mode at the moment.

[romanarranz]

OK, I understand your concerns. But putting the auth layer aside for the time being as this repo is in the early stages. I think the config should be an entity by itself like creating a new bot, then the user would be able to edit/tweak that config if so.

However, we'll always have the requirement to run the python backend "first", so i think it'll be fine to have certain options only available there (for example API keys and such) - and only the "rest" of the config can be made via the UI.

Agreed here. Does that mean there should be some ENV variables to setup such keys before a freqtrade bot run, keeping the config file simple enough just to tweak the features of the bot?

An additional problem is the "overridable" settings - which CAN be configured in via cli flag, via configuration, or via strategy (and prevelance is in this sequence) - so you can in theory break a strategy (accidentally or on purpose) by overwriting a stoploss to something it's not designed to work with.

From the UI point of view this would not apply I believe as each time the config is edited the whole set of settings would be used on the next run. But I can consider a scenario where the user defined some cli flag and gets confused if he edits the config on the UI and bot does not reflect that behavior..., would you say this is out of the scope of the UI?

Maybe adding an operational API on top of freqtrade could solve this issue, so what the bot runs is what is defined on the config, and that will be used as a snapshot to launch a bot.

[xmatthias]

From the UI point of view this would not apply I believe as each time the config is edited the whole set of settings would be used on the next run. But I can consider a scenario where the user defined some cli flag and gets confused if he edits the config on the UI and bot does not reflect that behavior..., would you say this is out of the scope of the UI?

Leaving cli arguments aside for a moment, we'd still have config + strategy settings.
Forcing configuration variables here will be a problem - as most settings (stoploss, ROI, ...) are heavily strategy-dependent, and by overwriting this you will break the strategy's functioning - and most of these settings are currently only kept in the strategy (by default) - with the possibility to overwrite it from the configuration.

---

Agreed here. Does that mean there should be some ENV variables to setup such keys before a freqtrade bot run, keeping the config file simple enough just to tweak the features of the bot?

No, i see it running as freqtrade webserver --config webserverconfig.json - which then is a regular freqtrade config file.
As said, i have a sample of this running (but not really trustworthy yet as it involves a few hacks to support both modes) - and i think it works quite well for now (this is the "operational API" you're talking about).

Using env-variables for everything that's needed is most likely not going to work, as the webserver alone requires quite some configurations already - and this "webserverconfig" object will give you the defaults in your UI.
WIthout that, you'd have to reconfigure everything from scratch the moment you relanuch the backend.

---

What will work is to then pass a (reduced) set of the freqtrade configuration via API when you start a mode (say backtesting or trading) - which is then used as a 2nd configuration when that mode is started, which will overwrite the relevant existing settings from the webserver config - and obviously it then loads the strategy, which sets whatever isn't set in the configuration (config overwrites strategy...).

The problem I'm conceptually struggling with this is how to persist these config changes done in the UI.

Sure, we can write a 2nd configuration file - but assuming docker and not knowing how people mount stuff, it might be tricky to write the configuration file reliably, as we don't know if the initial configuration is writable (we shouldn't overwrite the initial configuration anyway i think) - we however also don't know wether user_data is writable (you can mount each folder seperately - certain folders in RO if necessary)... - but we also don't want users to have to do the changes done in the UI every time ...

Using the database would be another option (that must be writable) - but what about backtesting mode? we don't use a database for this - and adding that will break hyperopt (even initializing the database breaks hyperopt ...) - so that's not really an option either.

It would also make the configuration situation a lot worse, as now i have: database configuration, config files + strategy (+ cli options) - so it'll become very confusing and intransparent where a configuration actually comes from.

[TheJoeSchr]

@xmatthias:

It would also make the configuration situation a lot worse, as now i have: database configuration, config files + strategy (+ cli options) - so it'll become very confusing and intransparent where a configuration actually comes from.

I brought this up before, but I still thinking using an .env file for saving and loading credentials (and maybe other "adminstrative" values like e.g. frequi stuff) and splitting it therefor from config.json would be a good thing. I'm not sure how canonical that approach is in the python world (pretty much standard with node-js crowd imho), but there is a package for that: https://pypi.org/project/python-dotenv/

About it clashing with the .env FOLDER some use for venv or similar, it could be checked if file/folder or named something like .env.defaults just to be safe

[xmatthias]

@JoeSchr absolutely wrong issue ...
and it's not applicable to frequi either - as the credentials MUST be stored clientside, and cannot be injected from the server either.

[TheJoeSchr]

On Wed, 3 Mar 2021 18:53:12 +0100, Matthias <[email protected]> said:

@JoeSchr absolutely wrong issue ...

sorry about that, just reminded me of it while browsing through the repo for the first time

and it's not applicable to frequi either - as the credentials MUST be stored clientside, and cannot be injected from the server either.

which credentials? from the exchanges? Or for frequi? I suppose you mean frequi, but reading through this issues, I thought it was about not sending/exposing exchange API keys by making it editable via frequi Anyways I will try to make myself more acquainted with the codebase before posting any further suggestions...

[xmatthias]

well THIS is in particular about frequi.
For frequi, having environment variables does not make sense - as in theory, i can run one bot to serve the ui, while logging in into a completely different bot. It's all stored in the browser - not in the backend (technically you could stop the serving of frequi after you loaded it - but eventually it'll try to reload and will tell you it's no longer there).

for freqtrade - yes - exposing credentials via api is a big problem, and the reason i'm currently not looking into a "config editor" further.

[TheJoeSchr]

ah, I understand.I didn't get that frequi starts it's own server and then just "proxies" the request over the rest api to the "real" freqtrade-server. I thought it was started by freqtrade directly and modified it's db/runtime/etc directly while the only thing send to the frontend would be the config and changes thereof.

I have quite a lot experience with vue and graphql, so let me know if you need any help with the frontend, maybe it will lessen your workload

[xmatthias]

well, it's served by freqtrade (uvicorn) if you use freqtrade install-ui.

However, it's a vue page - which in the end is a client side web application that you basically (mostly) download once, and then it's just running in the tab you got open, and from there it's using the API to get data.

[TheJoeSchr]

yeah I understood. I'm mostly working with nuxt to support server side rendering, so I was not immediatly thinking along the lines of deploying vue only as client-side script. makes total sense now On Wed, 3 Mar 2021 20:17:36 +0100, Matthias <[email protected]> said:

…

well, it's served by freqtrade (uvicorn) if you use `freqtrade install-ui`. However, it's a vue page - which in the end is a client side web application that you basically (mostly) download once, and then it's just running in the tab you got open, and from there it's using the API to get data. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #6 (comment)