new feature requests - FRITZ!Smart Gateway - support for Zigbee Devices #90

Peter1980FFM · 2023-12-31T13:07:56Z

Hallo I use the - FRITZ!Smart Gateway - in Home Assistant i see, that some Devices not Fully Supportet.
support for zigbee units (see chapter 1.2 in avm api docs)

If you need some Examples write me Back

Peter1980FFM · 2023-12-31T14:44:07Z

AVM_Zigbee_Devices.xlsx
Add the Device Information from the not correct working Devices

Peter1980FFM · 2024-01-07T09:51:49Z

@mib1185 i just create the Json file from my Fritzbox
overview.json
Groups are Devices 82 - 84
Device 81 is a Window Sensor in Homeassistant Device not get the Batterie Value
Device 79 is a Fritz Dect 440 Device in Home assistant are the 4 Buttons missing
Device 78 is aFritz Dect 400 Button in Home Assistan it only show the Batterie not the Buttons
Device 75 is a On/Off Button in Home Assistan it is not shown
Device 73 is a Zigbee Repeater and not show in Homeassistant
Device 69 i a Zigbee One Button and not show in Homeassitant
Device 66/67 is a Open/Close Button in Home Assistan it is not shown
Device 64 is a shutter in Home Assistan it is not shown in Home Assistant Batterie is Missing
Devices 50/51/52 aber one power strip AVM Make 3 Devices but all work in Home Assistant
Device 34 / 36 / 38 / 40 / 42 / 44 are only CCT Ligts but in Home Assistant show as Full collor Devices in Fritzbox CCT
Devices 9-12 and 4-7 are 4 ch Zigbee Devices and Each Channel is a Singel Entry

I have seen in the Json File the AVM makes for Every Button on Zigbee Device a One Device in the Json file
I hope i could help a Littel bit.

i have Order a Fritz DECT 350 Sensor but Hermes is not abel to Deliver so i can´t test it and a new oder is not posible its sold out yeah
If you need further support contact me

mib1185 · 2024-01-14T13:23:23Z

i've started to do some pre-work with #91, #94 and #95. there is a further PR #96 in my queue which will improve the interoperability with zigbee devices (and maybe other dect devices as well)

Peter1980FFM · 2024-01-14T14:48:25Z

If you need the json for the DECT 350 let me know

mib1185 · 2024-01-14T18:59:10Z

@Peter1980FFM Are you willing and able to help with some development tests in your HA installation?
If yes, I would like to provide you a custom version of the fritzbox integration which relies on the current development (open PRs above) so you could test if this will improve the compatibility with your zigbee devices.
Feel free to contact me via discord (my name is mib1185)

Peter1980FFM · 2024-01-14T19:48:58Z

@mib1185 i dont have a discord account but yes i can help

mib1185 · 2024-01-14T20:32:33Z

ok, than let's go with the very first step

1. create a backup !!!

2. now install the custom fritzbox component:

via HACS:

add https://github.com/mib1185/hass_fritzbox.git as custom repository
you should find the "Custom AVM FRITZ!SmartHome - only for development purpose" integration
install it and restart HA

or manually:

copy the whole custom fritzbox into your /config/custom_components folder (should be /config/custom_components/fritzbox afterwards)
restart HA

as soon as you got the custom fritzbox component installed, please restart HA, download and provide the diagnostics data and check if there are any noticeable improvements.

expected improvements:

missing battery values should be shown
buttons will still not be added (because the Fritz!Box api is pull only), but the possible sensors (eq. temperature and humidity) should be shown
same goes for repeaters - their possible sensors should be shown, but not the repeater itself

btw. creating a discord account lasts only about 5 minutes, has no need for using real life names and is absolutely for free and communication and sharing ob debug files is much easier/faster.

Peter1980FFM · 2024-01-15T06:04:09Z

I cant check it dont can bei Installed

mib1185 · 2024-01-15T08:01:49Z

hi @Peter1980FFM is there anything in the log? usually you do not need to start any new config flow, because this custom component will just replace the HA core integrated one, so use the already existing configuration

Peter1980FFM · 2024-01-15T08:37:03Z

After reboot

and want to be reboot and than comes the Message above.
i junst switch on debug in the logs i found
Home Assistant Core
Error setting up integration fritzbox - received exception
09:34:40 – (FEHLER) util/package.py

2024-01-15 09:34:40.632 ERROR (MainThread) [homeassistant.bootstrap] Error setting up integration fritzbox - received exception
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 563, in from_name
return Distribution.from_name(distribution_name)
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 565, in from_name
importlib.metadata.PackageNotFoundError: No package metadata was found for git+https://github.com/mib1185/[email protected]
git+https://github.com/mib1185/[email protected]
raise InvalidRequirement(str(e)) from e
git+https://github.com/mib1185/[email protected]
req = Requirement(urlparse(package).fragment)
raise InvalidRequirement(str(e)) from e
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 563, in from_name
return Distribution.from_name(distribution_name)
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 565, in from_name
importlib.metadata.PackageNotFoundError: No package metadata was found for git+https://github.com/mib1185/[email protected]
git+https://github.com/mib1185/[email protected]
raise InvalidRequirement(str(e)) from e
git+https://github.com/mib1185/[email protected]
req = Requirement(urlparse(package).fragment)
raise InvalidRequirement(str(e)) from e

mib1185 · 2024-01-15T18:21:40Z

@Peter1980FFM i've uploaded a new version of the custom component. would be great if you could test this new one

Peter1980FFM · 2024-01-15T18:39:23Z

@mib1185
Logger: homeassistant.bootstrap
Source: util/package.py:48
First occurred: 19:37:50 (1 occurrences)
Last logged: 19:37:50

Error setting up integration fritzbox - received exception
Traceback (most recent call last):
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 563, in from_name
return next(cls.discover(name=name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/util/package.py", line 40, in is_installed
distribution(package)
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 981, in distribution
return Distribution.from_name(distribution_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/metadata/init.py", line 565, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for git+https://github.com/mib1185/[email protected]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/packaging/requirements.py", line 35, in init
parsed = _parse_requirement(requirement_string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_parser.py", line 64, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_parser.py", line 82, in _parse_requirement
url, specifier, marker = _parse_requirement_details(tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_parser.py", line 126, in _parse_requirement_details
marker = _parse_requirement_marker(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_parser.py", line 147, in _parse_requirement_marker
tokenizer.raise_syntax_error(
File "/usr/local/lib/python3.11/site-packages/packaging/_tokenizer.py", line 165, in raise_syntax_error
raise ParserSyntaxError(
packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
git+https://github.com/mib1185/[email protected]
^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/util/package.py", line 44, in is_installed
req = Requirement(package)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/requirements.py", line 37, in init
raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
git+https://github.com/mib1185/[email protected]
^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/packaging/requirements.py", line 35, in init
parsed = _parse_requirement(requirement_string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_parser.py", line 64, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_parser.py", line 73, in _parse_requirement
name_token = tokenizer.expect(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_tokenizer.py", line 140, in expect
raise self.raise_syntax_error(f"Expected {expected}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/_tokenizer.py", line 165, in raise_syntax_error
raise ParserSyntaxError(
packaging._tokenizer.ParserSyntaxError: Expected package name at the start of dependency specifier

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/setup.py", line 138, in async_setup_component
return await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/setup.py", line 243, in _async_setup_component
await async_process_deps_reqs(hass, config, integration)
File "/usr/src/homeassistant/homeassistant/setup.py", line 466, in async_process_deps_reqs
await requirements.async_get_integration_with_requirements(
File "/usr/src/homeassistant/homeassistant/requirements.py", line 52, in async_get_integration_with_requirements
return await manager.async_get_integration_with_requirements(domain)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/requirements.py", line 168, in async_get_integration_with_requirements
await self._async_process_integration(integration, done)
File "/usr/src/homeassistant/homeassistant/requirements.py", line 183, in _async_process_integration
await self.async_process_requirements(
File "/usr/src/homeassistant/homeassistant/requirements.py", line 248, in async_process_requirements
await self._async_process_requirements(name, missing)
File "/usr/src/homeassistant/homeassistant/requirements.py", line 280, in _async_process_requirements
installed, failures = await self.hass.async_add_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/requirements.py", line 110, in _install_requirements_if_missing
if pkg_util.is_installed(req) or _install_with_retry(req, kwargs):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/util/package.py", line 48, in is_installed
req = Requirement(urlparse(package).fragment)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/requirements.py", line 37, in init
raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier

mib1185 · 2024-01-15T18:57:17Z

this is still the old version "0.0.1.dev0" 🤔
how did you install it? if hacs, please try to search for updates, so that "0.0.1.dev1" will be shown

Peter1980FFM · 2024-01-15T19:15:06Z

Look not so good

home-assistant_fritzbox_2024-01-15T19-14-36.194Z.log

mib1185 · 2024-01-15T19:38:03Z

ok, next try ... a new version -> "0.0.1.dev2" ... it contains now the just fresh released 0.6.10

Peter1980FFM · 2024-01-15T19:52:12Z

looks a littel bit better

some Device are 2 time in the List now
Groups are show unknown

home-assistant_fritzbox_2024-01-15T19-51-49.953Z.log

mib1185 · 2024-01-15T19:59:43Z

Please download and provide the diagnostics data.
The unknown model is expected for groups, since they do not have a model.

mib1185 · 2024-01-15T20:00:54Z

Further please also provide screenshot from the devices detailed page (eq. both "Esszimmer Fenster unten")

Peter1980FFM · 2024-01-15T20:07:00Z

config_entry-fritzbox-76a34de47a331963a19f9a3241f29eef.json (7).txt

mib1185 · 2024-01-15T20:29:48Z

mehh ... we need some kind of identifiers migration within the already existing devicees ... will work on it and come back with an updated version ^^

mib1185 · 2024-01-15T20:55:26Z

are there any of these duplicated devices, where both has entities?
were there any duplicated device before?

If you have console access (eq. via the terminal add-on or ssh) than it would be great if you could capture the actual device registry data like follows

change into /config/.storage directory

$ cd /config/.storage

get the id of the current fritzbox configuration entry

$ cat core.config_entries | jq '.data.entries[] | select(.domain == "fritzbox" and .source != "ignore") | .entry_id'
"4c042584ff6711eaa7a09baa9efee9fd"

now get all device registry entries connected to this configuration entry (replace 4c042584ff6711eaa7a09baa9efee9fd with the id from above) and provide all the output

$ cat core.device_registry | jq '.data.devices[] | select(.config_entries[0] == "4c042584ff6711eaa7a09baa9efee9fd")'

mib1185 · 2024-01-15T21:34:15Z

ok, here we are -> new version "0.0.1.dev3" with identifiers migration logic.

the duplicated devices may remain, but all entities should now be attached to the "correct" device (usually this with a correct firmware version information). if this is the case, please delete these "empty" devices and restart HA ... they should not re-occur

Peter1980FFM · 2024-01-15T21:46:40Z

i can not delet the unknown devices

mib1185 · 2024-01-15T21:52:45Z

please open the detail pages of these devices and provide some screenshots.
on the top right corner of the detail page, you should find the delete option via the edit (pencil) dialog.

what ist the current overall state? are the devices and entities now as expected?

Peter1980FFM · 2024-01-15T22:02:00Z

Deletion not work I dont know why, just go back to my backup an Install from git again, now it looks better,
some Device miss the Firmware Information but the rest looks good

mib1185 · 2024-01-15T22:03:59Z

Could you please download and provide the diagnostics data again?
and provide some screenshots of the devices with missing/wrong firmware infomration?

Peter1980FFM · 2024-01-16T06:32:30Z

Good Morning 3 Examples

config_entry-fritzbox-76a34de47a331963a19f9a3241f29eef.json (6).txt

mib1185 · 2024-01-18T19:55:38Z

hi @Peter1980FFM
sorry for late response, was busy these days.
i've checked the last diagnostics data and the screenshots - ich think we are near to get this finished and ready 👍
Would be great if you could provide a fresh set of debug logs - because the fritzbox integration is now used as custom component the settings needs to be changed a bit like follows in your configuration.yaml:

logger:
  default: info
  logs:
    custom_components.fritzbox: debug
    pyfritzhome: debug

Peter1980FFM · 2024-01-19T08:28:31Z

home-assistant_fritzbox_2024-01-19T08-26-40.548Z.log
Hi now the new Debug, i think i see somethingin the Debug Note, the main Device have the Firmware but the Datatpoints not

mib1185 · 2024-01-19T08:51:30Z

the main Device have the Firmware but the Datatpoints not

correct, that's why i tried to create the DeviceInfo in the fritzbox component based on if it is the main device or a sub unit. But as long as the main device do not provide any data which are consumed for any sensor entity, the DeviceInfo (with correct firmware) is not created ... anyhow this will lead to a breaking change in HA for the fritzbox component, so i'm, working on it to make this breaking change as less impact and "user friendly" as possible.

mib1185 · 2024-01-21T14:03:42Z

Hi @Peter1980FFM
unfortunately there is no acceptable way to automatically migrate all device ids and their possible usage in automations and scripts, so it will now create so called repair issues, so that the user is informed when automations and scripts needs to be changed by the user
this is now included in the new "0.0.1.dev4" version, would be great if you could test it again - if possible with one your previous backups

Peter1980FFM · 2024-01-21T14:49:52Z

@mib1185
got the error
Logger: homeassistant.config_entries
Source: config_entries.py:406
First occurred: 15:46:58 (1 occurrences)
Last logged: 15:46:58

Error setting up entry 192.168.122.24 for fritzbox
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 406, in async_setup
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/fritzbox/init.py", line 105, in async_setup_entry
hass.data[DATA_INSTANCES][automation.DOMAIN]
TypeError: unsupported operand type(s) for +: 'EntityComponent' and 'EntityComponent'

mib1185 · 2024-01-21T15:22:56Z

mehhh ... forgot to upload the latest code ... new version "0.0.1.dev5" is now ready

Peter1980FFM · 2024-01-21T15:42:34Z

better :-)

home-assistant_fritzbox_2024-01-21T15-42-12.935Z.log

Peter1980FFM · 2024-01-21T15:43:44Z

The gray of the switch are ok the Device is not Connect now to the Gateway christmal is Over ;-)

Peter1980FFM · 2024-01-21T15:48:09Z

@mib1185 i found a Error when i want to update my Automations

mib1185 · 2024-01-21T15:53:41Z

there should be repair issues for every replaced device, which was used in any automation or script

Peter1980FFM · 2024-01-21T16:07:01Z

@mib1185 not Realy staple
i reboot home Assist

Logger: homeassistant.config_entries
Source: config_entries.py:406
First occurred: 16:54:51 (1 occurrences)
Last logged: 16:54:51
home-assistant_fritzbox_2024-01-21T16-06-24.628Z.log

Error setting up entry 192.168.122.24 for fritzbox
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 406, in async_setup
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/fritzbox/init.py", line 102, in async_setup_entry
automation_component: EntityComponent[automation.AutomationEntity] = hass.data[
^^^^^^^^^^
KeyError: 'automation'

After Manual Rebbot the Integration

mib1185 · 2024-01-21T16:26:05Z

mehhh 😖 the automations seems not to be loaded in the first place, so the detection of issues couldn't work 🙄

Peter1980FFM · 2024-01-21T16:48:31Z

May the Automatic Repair is not the Solution :-( if you have more than one Entry in the Automation it only takes the Last one and Correct them but it not work i have to del it an make it new

mib1185 · 2024-01-21T17:04:45Z

there is no automatic repair of scripts and automations - this is not possible.
It will just inform the user, that there are automations and scripts (_ the names of these automations and scripts are also shown_) to be fixed by the user manually.

Peter1980FFM · 2024-01-21T20:15:50Z

thank you good to know i will whait for the final releas or is anything to test in future?

mib1185 · 2024-01-21T21:36:53Z

unfortunately even the final release will not have any automatic fixes. nevertheless I still need to solve the last issue you observed (btw many thanks for all your testing yet 🤝). Would be great if you could do some further tests as soon as I've I new development version ready

robertkleinschuster · 2024-07-25T06:45:38Z

What is the current state of Zigbee device support?

I would like to use Philips Hue Switches (4 Buttons) and Motion Sensors (that include a Motion, Brightness and Temperature Sensor)

In Home Assistant Philips Lightbulbs Seem to work fine but the Switches only Show up as Battery Level Sonsors and the Motion Sensors Show up as Battery and Alarm Sensors

BTW I am a software developer myself and lerning Python at the moment, i think with a bit of an introduction to the code i could help improve support for those things but I don't quite understand how the different device classes are instaciated yet from just looking at the code on GitHub.

Maybe i can checkout the repo and step debug the code to see whats going on with those devices.

mib1185 · 2024-07-25T14:11:34Z

Hi @robertkleinschuster
the Zigbee support is basically there, but the API of the Fritzbox does not provide push updates, so every device, which needs instant updates or reactions to be proper usable (eq. buttons and motion sensors) are still not supported.
This is a limitation of the http api of the Fritzbox itself, those only AVM could change it, by adding some push update mechanism to their api.

Peter1980FFM mentioned this issue Dec 31, 2023

AVM FRITZ!SmartHome not show groups home-assistant/core#106766

Closed

new feature requests - FRITZ!Smart Gateway - support for Zigbee Devices #90

new feature requests - FRITZ!Smart Gateway - support for Zigbee Devices #90

Comments

Peter1980FFM commented Dec 31, 2023

Peter1980FFM commented Dec 31, 2023

Peter1980FFM commented Jan 7, 2024

mib1185 commented Jan 14, 2024 • edited Loading

Peter1980FFM commented Jan 14, 2024

mib1185 commented Jan 14, 2024

Peter1980FFM commented Jan 14, 2024

mib1185 commented Jan 14, 2024 • edited Loading

1. create a backup !!!

2. now install the custom fritzbox component:

Peter1980FFM commented Jan 15, 2024

mib1185 commented Jan 15, 2024

Peter1980FFM commented Jan 15, 2024 • edited Loading

mib1185 commented Jan 15, 2024

Peter1980FFM commented Jan 15, 2024

mib1185 commented Jan 15, 2024

Peter1980FFM commented Jan 15, 2024 • edited Loading

mib1185 commented Jan 15, 2024

Peter1980FFM commented Jan 15, 2024

mib1185 commented Jan 15, 2024

mib1185 commented Jan 15, 2024

Peter1980FFM commented Jan 15, 2024

mib1185 commented Jan 15, 2024

mib1185 commented Jan 15, 2024 • edited Loading

mib1185 commented Jan 15, 2024 • edited Loading

Peter1980FFM commented Jan 15, 2024

mib1185 commented Jan 15, 2024 • edited Loading

Peter1980FFM commented Jan 15, 2024

mib1185 commented Jan 15, 2024

Peter1980FFM commented Jan 16, 2024 • edited Loading

mib1185 commented Jan 18, 2024

Peter1980FFM commented Jan 19, 2024

mib1185 commented Jan 19, 2024

mib1185 commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

mib1185 commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

mib1185 commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

mib1185 commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

mib1185 commented Jan 21, 2024

Peter1980FFM commented Jan 21, 2024

mib1185 commented Jan 21, 2024

robertkleinschuster commented Jul 25, 2024

mib1185 commented Jul 25, 2024

mib1185 commented Jan 14, 2024 •

edited

Loading

mib1185 commented Jan 14, 2024 •

edited

Loading

Peter1980FFM commented Jan 15, 2024 •

edited

Loading

Peter1980FFM commented Jan 15, 2024 •

edited

Loading

mib1185 commented Jan 15, 2024 •

edited

Loading

mib1185 commented Jan 15, 2024 •

edited

Loading

mib1185 commented Jan 15, 2024 •

edited

Loading

Peter1980FFM commented Jan 16, 2024 •

edited

Loading