Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subsys: net: download_client rework #17481

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+6,574 −482
Open

subsys: net: download_client rework #17481

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions .github/test-spec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
- "lib/nrf_modem_lib/**/*"
- "lib/sms/**/*"
- "subsys/dfu/dfu_target/**/*"
- "subsys/net/lib/download_client/**/*"
- "subsys/net/lib/downloader/**/*"
- "subsys/net/lib/fota_download/**/*"
- "subsys/net/lib/ftp_client/**/*"
- "subsys/net/lib/nrf_cloud/**/*"
Expand All @@ -51,7 +51,7 @@
- "lib/modem_info/**/*"
- "lib/modem_key_mgmt/**/*"
- "lib/date_time/**/*"
- "subsys/net/lib/download_client/**/*"
- "subsys/net/lib/downloader/**/*"
- "subsys/net/lib/fota_download/**/*"
- "subsys/net/lib/lwm2m_client_utils/**/*"
- "subsys/dfu/dfu_target/**/*"
Expand Down Expand Up @@ -81,7 +81,7 @@
- "include/modem/nrf_modem_lib.h"
- "include/modem/pdn.h"
- "include/modem/sms.h"
- "include/net/download_client.h"
- "include/net/downloader.h"
- "include/secure_services.h"
- "lib/at_host/**/*"
- "lib/at_monitor/**/*"
Expand All @@ -102,7 +102,7 @@
- "samples/cellular/pdn/**/*"
- "samples/cellular/sms/**/*"
- "samples/cellular/udp/**/*"
- "subsys/net/lib/download_client/**/*"
- "subsys/net/lib/downloader/**/*"

"CI-lwm2m-test":
- "include/modem/lte_lc*.h"
Expand All @@ -114,8 +114,8 @@
- "samples/cellular/lwm2m_carrier/**/*"
- "include/modem/at_monitor.h"
- "lib/at_monitor/**/*"
- "include/net/download_client.h"
- "subsys/net/lib/download_client/**/*"
- "include/net/downloader.h"
- "subsys/net/lib/downloader/**/*"
- "lib/sms/**/*"
- "include/modem/sms.h"
- "lib/pdn/**/*"
Expand Down Expand Up @@ -147,7 +147,7 @@
- "subsys/mgmt/**/*"
- "subsys/pcd/*"
- "subsys/net/lib/*fota*/**/*"
- "subsys/net/lib/download_client/**/*"
- "subsys/net/lib/downloader/**/*"
- "tests/subsys/bootloader/**/*"
- "tests/subsys/dfu/**/*"

Expand Down Expand Up @@ -433,7 +433,7 @@
- "include/modem/nrf_modem_lib.h"
- "include/modem/pdn.h"
- "include/modem/sms.h"
- "include/net/download_client.h"
- "include/net/downloader.h"
- "include/net/fota_download.h"
- "include/net/nrf_cloud.h"
- "include/net/nrf_cloud_agnss.h"
Expand All @@ -457,7 +457,7 @@
- "lib/sms/**/*"
- "lib/supl/**/*"
- "samples/cellular/modem_shell/**/*"
- "subsys/net/lib/download_client/**/*"
- "subsys/net/lib/downloader/**/*"
- "subsys/net/lib/fota_download/**/*"
- "subsys/net/lib/nrf_cloud/**/*"
- "subsys/net/lib/rest_client/**/*"
Expand Down Expand Up @@ -509,7 +509,7 @@
- "include/drivers/flash/**/*"
- "include/flash_map_pm.h"
- "include/modem/**/*"
- "include/net/download_client.h"
- "include/net/downloader.h"
- "include/net/fota_download.h"
- "include/net/nrf_cloud*"
- "include/net/rest_client.h"
Expand All @@ -531,7 +531,7 @@
- "subsys/bootloader/**/*"
- "subsys/caf/**/*"
- "subsys/dfu/**/*"
- "subsys/net/lib/download_client/**/*"
- "subsys/net/lib/downloader/**/*"
- "subsys/net/lib/fota_download/**/*"
- "subsys/net/lib/nrf_cloud/**/*"
- "subsys/net/lib/nrf_provisioning/**/*"
Expand Down
8 changes: 8 additions & 0 deletions CODEOWNERS
View file
Open in desktop
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,7 @@
/doc/nrf/libraries/networking/azure_*.rst @nrfconnect/ncs-cia-doc
/doc/nrf/libraries/networking/coap_utils.rst @nrfconnect/ncs-terahertz-doc
/doc/nrf/libraries/networking/download_client.rst @nrfconnect/ncs-modem-doc
/doc/nrf/libraries/networking/downloader.rst @nrfconnect/ncs-modem-doc
/doc/nrf/libraries/networking/fota_download.rst @nrfconnect/ncs-pluto-doc
/doc/nrf/libraries/networking/ftp_client.rst @nrfconnect/ncs-iot-oulu-tampere-doc
/doc/nrf/libraries/networking/icalendar_parser.rst @nrfconnect/ncs-doc-leads
Expand Down Expand Up @@ -335,7 +336,10 @@
/include/mgmt/ @nrfconnect/ncs-pluto
/include/modem/ @nrfconnect/ncs-modem
/include/mpsl/ @nrfconnect/ncs-dragoon
/include/net/ @nrfconnect/ncs-co-networking
/include/net/azure_* @nrfconnect/ncs-cia
/include/net/download_client* @nrfconnect/ncs-modem
/include/net/downloader* @nrfconnect/ncs-modem
/include/net/nrf_cloud_* @nrfconnect/ncs-nrf-cloud
/include/net/wifi_credentials.h @nrfconnect/ncs-cia
/include/nfc/ @nrfconnect/ncs-co-drivers @nrfconnect/ncs-si-muffin
Expand Down Expand Up @@ -743,6 +747,8 @@
/subsys/net/lib/mqtt_helper/ @nrfconnect/ncs-cia
/subsys/net/lib/azure_* @nrfconnect/ncs-cia
/subsys/net/lib/aws_* @nrfconnect/ncs-cia
/subsys/net/lib/download_client* @nrfconnect/ncs-modem
/subsys/net/lib/downloader/ @nrfconnect/ncs-modem
/subsys/net/lib/ftp_client/ @nrfconnect/ncs-iot-oulu
/subsys/net/lib/hostap_crypto/ @krish2718 @jukkar @vivekuppunda
/subsys/net/lib/icalendar_parser/ @lats1980
Expand Down Expand Up @@ -852,6 +858,8 @@
/tests/subsys/mpsl/ @nrfconnect/ncs-dragoon
/tests/subsys/net/lib/aws_*/ @nrfconnect/ncs-cia
/tests/subsys/net/lib/azure_iot_hub/ @nrfconnect/ncs-cia
/tests/subsys/net/lib/downloader/ @nrfconnect/ncs-modem
/tests/subsys/net/lib/download_client/ @nrfconnect/ncs-modem
/tests/subsys/net/lib/fota_download/ @nrfconnect/ncs-pluto
/tests/subsys/net/lib/lwm2m_*/ @nrfconnect/ncs-iot-oulu
/tests/subsys/net/lib/mqtt_helper/ @nrfconnect/ncs-cia
Expand Down
3 changes: 1 addition & 2 deletions applications/asset_tracker_v2/boards/native_sim.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,7 @@ CONFIG_NET_CONFIG_MY_IPV4_GW="192.0.2.2"
# FOTA
CONFIG_FOTA_DOWNLOAD=n
CONFIG_DFU_TARGET=n
CONFIG_DOWNLOAD_CLIENT=n
CONFIG_DOWNLOAD_CLIENT_HTTP_FRAG_SIZE_1024=n
CONFIG_DOWNLOADER=n

# MCUBOOT
CONFIG_BOOTLOADER_MCUBOOT=n
Expand Down
2 changes: 1 addition & 1 deletion applications/asset_tracker_v2/doc/asset_tracker_v2_description.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ This application uses the following |NCS| libraries and drivers:
* :ref:`lib_date_time`
* :ref:`lte_lc_readme`
* :ref:`modem_info_readme`
* :ref:`lib_download_client`
* :ref:`lib_downloader`
divipillai marked this conversation as resolved.
Show resolved Hide resolved
* :ref:`lib_fota_download`
* :ref:`caf_leds`

Expand Down
7 changes: 5 additions & 2 deletions applications/asset_tracker_v2/overlay-carrier.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ CONFIG_PDN=y
# AT Monitor is used by PDN library
CONFIG_AT_MONITOR=y

# Download client for DFU
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=230
# Downloader for DFU
CONFIG_DOWNLOADER_MAX_FILENAME_SIZE=230

# Modem info
CONFIG_MODEM_INFO_BUFFER_SIZE=512
Expand All @@ -36,3 +36,6 @@ CONFIG_DFU_TARGET=y

# Asserts
CONFIG_ASSERT_VERBOSE=n

# IPv4 (IPv6 is enabled by default)
CONFIG_NET_IPV4=y
8 changes: 3 additions & 5 deletions applications/asset_tracker_v2/prj.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,9 @@ CONFIG_FCB=y
# FOTA
CONFIG_FOTA_DOWNLOAD=y
CONFIG_DFU_TARGET=y
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_HTTP_FRAG_SIZE_1024=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
CONFIG_DOWNLOAD_CLIENT_BUF_SIZE=2300
CONFIG_DOWNLOAD_CLIENT_MAX_HOSTNAME_SIZE=128
CONFIG_DOWNLOADER=y
CONFIG_DOWNLOADER_STACK_SIZE=4096
CONFIG_DOWNLOADER_MAX_HOSTNAME_SIZE=128

# Flash - Used in FOTA, settings and storage for P-GPS.
CONFIG_FLASH=y
Expand Down
2 changes: 1 addition & 1 deletion applications/serial_lte_modem/doc/slm_description.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -684,7 +684,7 @@ This application uses the following |NCS| libraries:
* :ref:`lib_ftp_client`
* :ref:`sms_readme`
* :ref:`lib_fota_download`
* :ref:`lib_download_client`
* :ref:`lib_downloader`
* :ref:`lib_nrf_cloud`
* :ref:`lib_nrf_cloud_agnss`
* :ref:`lib_nrf_cloud_pgps`
Expand Down
5 changes: 4 additions & 1 deletion applications/serial_lte_modem/overlay-carrier.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ CONFIG_FP_HARDABI=y
CONFIG_PDN=y

# Download client for DFU
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=230
CONFIG_DOWNLOADER_MAX_FILENAME_SIZE=230

# Modem info
CONFIG_MODEM_INFO_BUFFER_SIZE=512
Expand All @@ -28,3 +28,6 @@ CONFIG_LWM2M_CARRIER_SETTINGS=y

# DFU target library
CONFIG_DFU_TARGET=y

# IPv4 (IPv6 is enabled by default)
CONFIG_NET_IPV4=y
6 changes: 3 additions & 3 deletions applications/serial_lte_modem/prj.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ CONFIG_HTTP_PARSER_URL=y
CONFIG_FOTA_DOWNLOAD=y
CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
CONFIG_DFU_TARGET=y
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=2048
CONFIG_DOWNLOADER=y
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are seeing random failures with application fota update, where the fota package is downloaded successfully, but after the reset, the new package is not updated and mcuboot_swap_type() is BOOT_SWAP_TYPE_NONE.

The issue is random, but can only take place if RTT-traces are activated, when the reset is performed (with AT#XRESET).

Only takes place with 9151 (and likely 9161), but not with 9160.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It takes place with nRF9161 as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We verified that the image is uploaded to the secondary slot correctly, so this issue is not directly related to downloader. Rather, timings are changed a bit, which likely causes the issue. After the image is uploaded, the device apparently resets, which causes bootloader to revert the upgrade and load the old application image instead.

CONFIG_DOWNLOADER_STACK_SIZE=4096
CONFIG_DOWNLOADER_MAX_FILENAME_SIZE=2048
CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_IMG_MANAGER=y

Expand Down
4 changes: 2 additions & 2 deletions applications/serial_lte_modem/src/slm_at_fota.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,10 @@
LOG_MODULE_REGISTER(slm_fota, CONFIG_SLM_LOG_LEVEL);

/* file_uri: scheme://hostname[:port]path[?parameters] */
#define FILE_URI_MAX CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE
#define FILE_URI_MAX CONFIG_DOWNLOADER_MAX_FILENAME_SIZE
#define SCHEMA_HTTP "http"
#define SCHEMA_HTTPS "https"
#define URI_HOST_MAX CONFIG_DOWNLOAD_CLIENT_MAX_HOSTNAME_SIZE
#define URI_HOST_MAX CONFIG_DOWNLOADER_MAX_HOSTNAME_SIZE
#define URI_SCHEMA_MAX 8
#define ERASE_POLL_TIME 2

Expand Down
2 changes: 1 addition & 1 deletion doc/nrf/app_dev/device_guides/nrf91/nrf91_building.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ To perform a FOTA update, complete the following steps:
See :ref:`upgradable_bootloader` for more information.

#. Make the binary file (or files) available for download.
Upload the serialized :file:`.cbor` binary file or files to a web server that is compatible with the :ref:`lib_download_client` library.
Upload the serialized :file:`.cbor` binary file or files to a web server that is compatible with the :ref:`lib_downloader` library.

The full FOTA procedure depends on where the binary files are hosted for download.

Expand Down
2 changes: 1 addition & 1 deletion doc/nrf/libraries/bin/lwm2m_carrier/app_integration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ It provides an abstraction of the following modules:
.. lwm2m_osal_mod_list_start
* :ref:`at_monitor_readme`
* :ref:`lib_download_client`
* :ref:`lib_downloader`
* :ref:`sms_readme`
* :ref:`pdn_readme`
* :ref:`lib_dfu_target`
Expand Down
2 changes: 1 addition & 1 deletion doc/nrf/libraries/bin/lwm2m_carrier/requirements.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Following are some of the requirements and limitations of the application while
* For example, setting :kconfig:option:`CONFIG_LWM2M_CARRIER_SERVER_SEC_TAG` to 42 uses the security tag range 43 to 46 instead of 25 to 28.

* The CA certificates that are used for out-of-band FOTA must be provided by the application.
Out-of-band FOTA updates are done by the :ref:`lib_download_client`.
Out-of-band FOTA updates are done by the :ref:`lib_downloader`.
Although the certificates are updated as part of the |NCS| releases, you must check the requirements from your carrier to know which certificates are applicable.

* The LwM2M carrier library uses the following NVS record key range: ``0xCA00`` to ``0xCAFF``.
Expand Down
16 changes: 8 additions & 8 deletions doc/nrf/libraries/networking/aws_fota.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Creating a FOTA job
#. Click the uploaded image file :file:`app_update.bin` and copy the *Object URL* without the *https://* prefix and folder path.
#. Create a text file (job document) with content as in the snippet, replacing the following data:

* *protocol* with either `http` or `https`.
* *protocol* with either ``http`` or ``https``.
* *host_url* with the *Object URL* copied in the previous step (for example, ``examplebucket.s3.eu-central-1.amazonaws.com``).
* *file_path* with the path and file name (for example, ``app_update.bin``).

Expand Down Expand Up @@ -105,10 +105,10 @@ Configure the following parameters when using this library:
* :kconfig:option:`CONFIG_AWS_FOTA_PAYLOAD_SIZE` - Sets the maximum payload size for AWS IoT job messages.
* :kconfig:option:`CONFIG_AWS_FOTA_DOWNLOAD_SECURITY_TAG` - Sets the security tag to be used in case of HTTPS downloads.

Additionally, configure the :ref:`lib_download_client` library:
Additionally, configure the :ref:`lib_downloader` library:

* :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_MAX_HOSTNAME_SIZE` - Sets the maximum length of the host name for the download client.
* :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE` - Sets the maximum length of the file name for the download client.
* :kconfig:option:`CONFIG_DOWNLOADER_MAX_HOSTNAME_SIZE` - Sets the maximum length of the host name for the library.
* :kconfig:option:`CONFIG_DOWNLOADER_MAX_FILENAME_SIZE` - Sets the maximum length of the file name for the library.

.. _aws_fota_implementation:

Expand All @@ -133,7 +133,7 @@ The following sequence diagram shows how a firmware over-the-air update is imple
AWS IoT jobs
============

The implementation uses a job document like the following (where *protocol* is either `http` or `https`, *bucket_name* is the name of your bucket and *file_name* is the name of your file) for passing information from `AWS IoT jobs`_ to the device:
The implementation uses a job document like the following (where *protocol* is either ``http`` or ``https``, *bucket_name* is the name of your bucket and *file_name* is the name of your file) for passing information from `AWS IoT jobs`_ to the device:

.. parsed-literal::
:class: highlight
Expand Down Expand Up @@ -191,17 +191,17 @@ Presigned URLs

When using the presigned URLs, you might need to increase the value of the following Kconfig options to accommodate the long file name and payload size of the presigned URL and the secure download of the image:

* :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE`.
* :kconfig:option:`CONFIG_DOWNLOADER_MAX_FILENAME_SIZE`.
* :kconfig:option:`CONFIG_AWS_FOTA_PAYLOAD_SIZE`.
* :kconfig:option:`CONFIG_MBEDTLS_HEAP_SIZE` - If running Mbed TLS on the application core (Wi-Fi® builds).

Limitations
***********

* If the :kconfig:option:`CONFIG_AWS_FOTA_DOWNLOAD_SECURITY_TAG` Kconfig option is not configured but HTTPS is selected as the protocol, the update job fails.
For further information about HTTPS support, refer to :ref:`the HTTPS section of the download client documentation <download_client_https>`.
For further information about HTTPS support, refer to :ref:`the HTTPS section of the Downloader library documentation <downloader_https>`.
* The library requires a Content-Range header to be present in the HTTP response from the server.
This limitation is inherited from the :ref:`lib_download_client` library.
This limitation is inherited from the :ref:`lib_downloader` library.

API documentation
*****************
Expand Down
8 changes: 4 additions & 4 deletions doc/nrf/libraries/networking/azure_fota.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,16 @@ Configure the following parameters when using this library:
* :kconfig:option:`CONFIG_AZURE_FOTA_TLS` - Enables HTTPS for downloads. By default, TLS is enabled and currently, the transport protocol must be configured at compile time.
* :kconfig:option:`CONFIG_AZURE_FOTA_SEC_TAG` - Sets the security tag for TLS credentials when using HTTPS as the transport layer. See :ref:`azure_iot_hub_flash_certs` for more details.

Additionally, configure the :ref:`lib_download_client` library:
Additionally, configure the :ref:`lib_downloader` library:

* :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_MAX_HOSTNAME_SIZE` - Sets the maximum length of the host name for the download client.
* :kconfig:option:`CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE` - Sets the maximum length of the file name for the download client.
* :kconfig:option:`CONFIG_DOWNLOADER_MAX_HOSTNAME_SIZE` - Sets the maximum length of the host name for the library.
* :kconfig:option:`CONFIG_DOWNLOADER_MAX_FILENAME_SIZE` - Sets the maximum length of the file name for the library.

Limitations
***********

The library requires a ``Content-Range`` header to be present in the HTTP response from the server.
This limitation is inherited from the :ref:`lib_download_client` library.
This limitation is inherited from the :ref:`lib_downloader` library.

API documentation
*****************
Expand Down
5 changes: 5 additions & 0 deletions doc/nrf/libraries/networking/download_client.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ Download client
:local:
:depth: 2

.. note::

The :ref:`lib_download_client` library has been deprecated and it will be removed in one of the future releases.
Use the :ref:`lib_downloader` library instead.

The download client library can be used to download files from an HTTP or a CoAP server.

Overview
Expand Down
Loading
Loading