netatmo driver don't work anymore

[odo125]

Hi
I posted this on Github in https://github.com/weewx/weewx under issues. There it was closed, because of not beeing a supported weewx driver. I was told to contact the author.

My netatmo-driver don't work anymore since 11th of July 2023.
I think something on the netatmo api has changed. I get the following error:

"netatmo: netatmo-client: failed attempt 1 of 5 to get data: HTTP Error 400
netatmo: netatmo-client: url: https://api.netatmo.net/oauth2/token data: b'grant_type=password&client_id=64146554164654464554654&client_secret=456545465464644654874841579874468&username=........................."
(client_id and client_secret was changed by me in this posting)

Maybee "grant_type=password" is not allowed anymore.
Can you tell me if this can get fixed. Thanks in advance.

[MrClever]

Netatmo has deprecated the "client credential" grant type (ie, grant_type=password) as per https://dev.netatmo.com/apidocumentation/oauth#client-credential

The method was officially deprecated in October 2022 but appears to have actually taken effect in July 2023. Netatmo's suggestion is to pivot to the authorization code grant type.

However, a casual perusal of the code suggests this is a slightly more involved change than my sleep-deprived brain can handle right now. Maybe @matthewwall can take a look?

[bucolo]

you can use this fork : https://github.com/Buco7854/weewx-netatmo

you must generate a refresh_token in https://dev.netatmo.com/apps/
and change weewx.conf with :
[netatmo]

client_secret = xxxxxxxxxx
driver = user.netatmo
client_id = yyyyyyyyyyyyyy
refresh_token = zzzzzzzzz

[odo125]

Thanks a lot bucolo for updating the code. My Station: https://www.harrynet.mine.nu/weewx/sofa/ is up and running again.

[MrClever]

you can use this fork : https://github.com/Buco7854/weewx-netatmo

you must generate a refresh_token in https://dev.netatmo.com/apps/

Thanks for taking the time to code a fix! Unfortunately I'm seeing the same problem (HTTP 400). Any ideas?

/var/log/syslog (formatted for clarity):

Jul 16 hh:mm:ss myhost weewxd: netatmo: netatmo-client: failed attempt 1 of 5 to get data: HTTP Error 400: 
Jul 16 hh:mm:ss myhost weewxd: netatmo: netatmo-client: waiting 10 seconds before retry
Jul 16 hh:mm:ss myhost weewxd: netatmo: netatmo-client:
    url: https://api.netatmo.net/oauth2/token
        data:
            b'grant_type=refresh_token&
              refresh_token=5a-blah-blah-42%C794-more-blah-51&
              client_id=5a-lots-of-randomness-35&
              client_secret=cP-more-randomness-Fv'
         hdr:
            {'Content-Type': 'application/x-www-form-urlencoded;charset=utf-8'}

Exerpt from /etc/weewx.conf:

[netatmo]
    driver = user.netatmo
    poll_interval = 180
    mode = cloud
    device_id = 70:xx:xx:xx:xx:70
    username = me@mydomain
    client_id = 5a-lots-of-randomness-35
    client_secret = cP-more-randomness-Fv
    refresh_token = 5a-blah-blah-42|94-more-blah-51

[bucolo]

Have you try to recreate a new app in https://dev.netatmo.com/apps/createanapp#form ?

for me it's always ok

you can delete username (and device_id i think)

[MrClever]

@bucolo - tried using an existing app with new tokens and also a brand-new app with new tokens. I’ve enabled all the relevant scopes and even a token with R+W access to all scopes. No dice, still getting 400 errors in the logs.

This is very odd :-/

[bucolo]

It's very strange indeed. I do not see. I didn't write the fork
You can write an issue directly on https://github.com/Buco7854/weewx-netatmo

[Buco7854]

you can use this fork : https://github.com/Buco7854/weewx-netatmo

you must generate a refresh_token in https://dev.netatmo.com/apps/

Thanks for taking the time to code a fix! Unfortunately I'm seeing the same problem (HTTP 400). Any ideas?

/var/log/syslog (formatted for clarity):

Jul 16 hh:mm:ss myhost weewxd: netatmo: netatmo-client: failed attempt 1 of 5 to get data: HTTP Error 400: 
Jul 16 hh:mm:ss myhost weewxd: netatmo: netatmo-client: waiting 10 seconds before retry
Jul 16 hh:mm:ss myhost weewxd: netatmo: netatmo-client:
    url: https://api.netatmo.net/oauth2/token
        data:
            b'grant_type=refresh_token&
              refresh_token=5a-blah-blah-42%C794-more-blah-51&
              client_id=5a-lots-of-randomness-35&
              client_secret=cP-more-randomness-Fv'
         hdr:
            {'Content-Type': 'application/x-www-form-urlencoded;charset=utf-8'}

Exerpt from /etc/weewx.conf:

[netatmo]
    driver = user.netatmo
    poll_interval = 180
    mode = cloud
    device_id = 70:xx:xx:xx:xx:70
    username = me@mydomain
    client_id = 5a-lots-of-randomness-35
    client_secret = cP-more-randomness-Fv
    refresh_token = 5a-blah-blah-42|94-more-blah-51

Try to make the request in a tool like postman or reqbin (if you dont want to download anything). Pass the same params (the one that you can see in the error message) and use the application/x-www-form-urlencoded content type.
You will probably see some description about was exactly was wrong with the request.

[danitus13]

Hi, I did this update.
In the new Netatmo file that I installed, is there also the change for the accumulation of rain?

[Buco7854]

Hi, I did this update.
In the new Netatmo file that I installed, is there also the change for the accumulation of rain?

If it did work before it will work now, I changed nothing more than the authentication.

[MrClever]

@Buco7854
FIXED - I simply replaced the Netatmo-generated refresh token with the access token and voila. All working again! Weird.

[Buco7854]

@Buco7854
FIXED - I simply replaced the Netatmo-generated refresh token with the access token and voila. All working again! Weird.

This is indeed pretty weird. The API clearly states it should take a refresh_token in.

[MrClever]

Agreed - I've coded some integrations for my solar system that make some "informed" guesses regarding efficiency by interrogating my weather station via Netatmo's servers too. In short, this shouldn't work...but it does 🤷🏻‍♂️ Thanks for the work on pivoting the authentication method though! You rock 👍

[danitus13]

Ciao, ho fatto questo aggiornamento.
Nel nuovo file Netatmo che ho installato c'è anche la modifica per l'accumulo di pioggia?

Se funzionava prima funzionerà ora, non ho cambiato nient'altro che l'autenticazione.

The fork you created is missing this rain fix, which is not in the original fork

#18

Could it be integrated?

[Buco7854]

could you try the following, I'm unable to do so for now.

https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test

Click on code, download the zip, upload it to your server, rename it to weewx-netatmo.zip and start to follow installation instruction from the second point.

[bucolo]

I replaced the old fix (v 0.14) with the new one (v0.17) . it works !
Thank you very much Buco7854

[danitus13]

potresti provare quanto segue, per ora non sono in grado di farlo.

https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test

Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

ok thanks Buco78, the rain is coming on Saturday, I text it and let you know if it's ok

[Buco7854]

potresti provare quanto segue, per ora non sono in grado di farlo.

https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test

Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

ok thanks Buco78, the rain is coming on Saturday, I text it and let you know if it's ok

Ok, thanks 😀

[dg9bja]

potresti provare quanto segue, per ora non sono in grado di farlo.
https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test
Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

ok thanks Buco78, the rain is coming on Saturday, I text it and let you know if it's ok

Ok, thanks 😀

Thanks for your work. Tried this fix and got following
Netatmo real vs weewx:
4.2 - 7.2
0.9 - 0.9
1.0 - 3.1
Weewx summay shows 11.2 instead of 6.1

[Buco7854]

potresti provare quanto segue, per ora non sono in grado di farlo.
https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test
Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

ok thanks Buco78, the rain is coming on Saturday, I text it and let you know if it's ok

Ok, thanks 😀

Thanks for your work. Tried this fix and got following
Netatmo real vs weewx:
4.2 - 7.2
0.9 - 0.9
1.0 - 3.1
Weewx summay shows 11.2 instead of 6.1

How was it without rain fix?

[dg9bja]

Last version I used was a fork from jkrasinger. Did work until this week and needed that thing with refresh token.
Should I try your master version?

[Buco7854]

Last version I used was a fork from jkrasinger. Did work until this week and needed that thing with refresh token. Should I try your master version?

What I basically did is just merge jkrasinger branch into mine. I had to do it manually since there was some conflict due to my previous changes. I double checked and it seems I unintentionally didnt delete a line that put the data into the collector before the rain data was sent (even if it was sent again right after, that could possibly change the result). It is the only thing that was changed and I will not be able to help further if the latest version of the rain_fix doesnt work. I recommend you to retry with the latest fix, and if it doesn't match your expectations then just compare the master and rain_fix branch to see if there is at least some improvements because the latest fix should work.

Please let me know if the fix works so that I can put it into the master branch.

[Buco7854]

Do not forget to download the zip from the "Code" button, dont use the link provided in the README since it point to the master branch (planning on changing the README)

Nevermind, I changed the link for now. You can just follow rain_fix_test branch README instructions.

[danitus13]

potresti provare quanto segue, per ora non sono in grado di farlo.

https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test

Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

Hi Buco, tonight I tested with the rain that came. Still underestimates.
Accumulation Netatmo 5.9mm
Accumulation Weewx 4.3mm

[Buco7854]

potresti provare quanto segue, per ora non sono in grado di farlo.

https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test

Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

Hi Buco, tonight I tested with the rain that came. Still underestimates.
Accumulation Netatmo 5.9mm
Accumulation Weewx 4.3mm

Did you download the zip again? there was some change.

[bucolo]

it's good now, the accumulations of yesterday and this night are good. you can change the version number for more clarity.

[danitus13]

potresti provare quanto segue, per ora non sono in grado di farlo.

https://github.com/Buco7854/weewx-netatmo/tree/rain_fix_test

Fai clic sul codice, scarica lo zip, caricalo sul tuo server, rinominalo in weewx-netatmo.zipe inizia a seguire le istruzioni di installazione dal secondo punto.

I downloaded the one from 3 days ago. I'll try to put the new one on and I'll tell you how the test goes

[jult]

you can use this fork : https://github.com/Buco7854/weewx-netatmo

you must generate a refresh_token in https://dev.netatmo.com/apps/ and change weewx.conf with : [netatmo]

client_secret = xxxxxxxxxx
driver = user.netatmo
client_id = yyyyyyyyyyyyyy
refresh_token = zzzzzzzzz

When I generate a token for [ ] read station, I get both an Access token and a Refresh token.
Don't we need to do this in weewx.conf then?

[netatmo]
driver = user.netatmo
client_secret = xxxxxxxxxx
client_id = yyyyyyyyyyyy
access_token = zzzzzzzzz
refresh_token = tttttttttt

[Buco7854]

you can use this fork : https://github.com/Buco7854/weewx-netatmo

you must generate a refresh_token in https://dev.netatmo.com/apps/ and change weewx.conf with : [netatmo]

client_secret = xxxxxxxxxx
driver = user.netatmo
client_id = yyyyyyyyyyyyyy
refresh_token = zzzzzzzzz

When I generate a token for [ ] read station, I get both an Access token and a Refresh token.
Don't we need to do this in weewx.conf then?

[netatmo]
driver = user.netatmo
client_secret = xxxxxxxxxx
client_id = yyyyyyyyyyyy
access_token = zzzzzzzzz
refresh_token = tttttttttt

No need since with the refresh token you can get the access one, and since we need to know when the token will expire we request one on startup to get this information thats why you dont need to provide one yourself.

[dg9bja]

Last version I used was a fork from jkrasinger. Did work until this week and needed that thing with refresh token. Should I try your master version?

What I basically did is just merge jkrasinger branch into mine. I had to do it manually since there was some conflict due to my previous changes. I double checked and it seems I unintentionally didnt delete a line that put the data into the collector before the rain data was sent (even if it was sent again right after, that could possibly change the result). It is the only thing that was changed and I will not be able to help further if the latest version of the rain_fix doesnt work. I recommend you to retry with the latest fix, and if it doesn't match your expectations then just compare the master and rain_fix branch to see if there is at least some improvements because the latest fix should work.

Please let me know if the fix works so that I can put it into the master branch.

Looks good now after two checks. Thank you.
If there is anyone with an exception "list index out of range" let me know. Sometimes the response misses data and that exception is thrown (regarding rain data). I have a little fix that is just a line in code.

[Buco7854]

I replaced the master branch, thanks for the feedbacks.

[Buco7854]

Just added the fix for the lacking data thing should work properly now.

[danitus13]

confirm you, everything is ok. It recorded the correct rain. Thank you very much

[WiggyToo]

First of all many thanks to @bucolo \ @Buco7854 for providing this branch and the fix for Netatmo Authentication.

I first noticed a lack of data on Sep 11th after a power cut at home, so up to this point the old Authentication method was working, or maybe the connection had not expired.

I ended up updating Debian, Python and Weewx to try to track down a problem I had after I installed this fix, namely an error as below.

Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__: Caught unrecoverable exception:
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****  No module named queue
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****  Traceback (most recent call last):
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 148, in main
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****      engine = weewx.engine.StdEngine(config_dict)
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 81, in __init__
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****      self.setupStation(config_dict)
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 108, in setupStation
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****      __import__(driver)
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****    File "/usr/share/weewx/user/netatmo.py", line 28, in <module>
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****      import queue
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****  ImportError: No module named queue
Sep 18 09:30:11 Weather weewx[21139] CRITICAL __main__:     ****  Exiting.

I tracked this down to line 28 in netatmo.py - not sure if this is just my setup but as it looked like the try and except lines were there to resolve a Python 2 \ Python 3 possible issue I just commented out line 28 and let the next bit of code sort it out.

# import queue
import sys
try:
    import queue as Queue                                      # Python 3
except:
    import Queue                                        # Python 2

At this point I could then see the authentication error:

Sep 18 00:05:44 Weather weewxd: netatmo: netatmo-client: url: https://api.netatmo.com/oauth2/token data: client_secret=[blah blah]&grant_type=refresh_token&client_id=[blah blah]&refresh_token=[blah blah] hdr: {'Content-Type': 'application/x-www-form-urlencoded;charset=utf-8'}
Sep 18 00:05:44 Weather weewxd: netatmo: netatmo-client: failed attempt 1 of 5 to get data: HTTP Error 400:

I'd like to thank @Buco7854 again for the pointer to reqbin which was a great way to test the credentials, as usual it was a typo with the client id that was causing the issue.

For others new to reqbin and this stuff, it was just a case of putting

https://api.netatmo.com/oauth2/token

in the URL box, selecting "Form URL Encoded" in the type dropdown and pasting the right text in the box below

grant_type=refresh_token
refresh_token=[blah blah]
client_id=[blah blah]
client_secret=[blah blah]

Select POST and click SEND, response should look like this

{
    "scope": ["read_station"],
    "access_token": "[blah blah]",
    "refresh_token": "[blah blah]",
    "expires_in": 10800,
    "expire_in": 10800
}

When I had the wrong data I just got a error code 10023.

So thanks for all the help, I do have a gap in my data now - anyone know how to get the old data in there?

https://www.wiggytoo.co.uk/reports/

Regards,
Gav

[WiggyToo]

More on this thread for later changes

Buco7854#2