Getting long-term access to LANCE / for data ingest

[mccabete]

@wildintellect here are some specific data ingest needs we have.

We have been getting data daily from:

Near Real Time (Via LANCE)
VNP14IMGTDL - https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/FIRMS/suomi-npp-viirs-c2/Global/
VJ114IMGTDL - https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/FIRMS/noaa-20-viirs-c2/Global/

We would love a way to handle tokening that's better than "let it fail and then bug someone to change it".

Also from @scoffiel

The other type of permanent token that would be really helpful is permanent AWS credentials for streaming the VIIRS Level 2 swaths (VNP14IMG, VJ114IMG) from LAADS via EarthData Cloud because currently they expire every hour:

from earthaccess import Auth, Store, DataCollections, DataGranules
auth = Auth() 
auth.login(strategy="netrc")
store = Store(auth)
fs = store.get_s3fs_session('LAADS') #daac or provider name

I use the following s3 paths:
s3://lp-prod-protected/VJ114IMG.002
s3://lp-prod-protected/VNP14IMG.002

Getting credentials for buckets is described here: https://data.lpdaac.earthdatacloud.nasa.gov/s3credentialsREADME
index

[scoffiel]

Following up to note that there has been some discussion on Slack about the second half of this (s3 credentials), and next steps are to try accessing them via maap-py with the "maap-data-reader" profile for permanent credentials instead of via earthaccess. As far as we know, earthaccess does not handle auto-refreshing via the get_s3fs_session method, but it could. Chuck is going to open a new issue in earthaccess if it doesn't already exist

[chuckwondo]

Following up to note that there has been some discussion on Slack about the second half of this (s3 credentials), and next steps are to try accessing them via maap-py with the "maap-data-reader" profile for permanent credentials instead of via earthaccess. As far as we know, earthaccess does not handle auto-refreshing via the get_s3fs_session method, but it could. Chuck is going to open a new issue in earthaccess if it doesn't already exist

See nsidc/earthaccess#765

[wildintellect]

The most basic example of our direct bucket method is:

import xarray as xr
import fsspec

sample_file = 's3://lp-prod-protected/VJ114IMG.002/VJ114IMG.A2024198.0942.002.2024198150736/VJ114IMG.A2024198.0942.002.2024198150736.nc'
s3_fsspec = fsspec.filesystem("s3", profile="maap-data-reader")
test = xr.open_dataset(s3_fsspec.open(sample_file), engine="h5netcdf", phony_dims='sort')

The above code will work in a DPS job.

For the ADE This requires you setup an AWS config file as noted in #1001

mkdir /projects/.aws && cat >> /projects/.aws/config <<CONFIG
[profile maap-data-reader]
region = us-west-2
role_arn = {arn}
credential_source = Ec2InstanceMetadata
CONFIG

There is an open ticket to clarify in the MAAP documentation.

[mccabete]

@eorland FYI this may be relevant to your discussion with Otmar.

[wildintellect]

Regular product (VJ114IMG_NRT)

https://lpdaac.usgs.gov/products/vj114imgv002/ is available via LPDAAC on EarthDataCloud (S3). Unclear how long after the NRT it gets published. This dataset is available via MAAP EDL authenticated methods and direct S3 reader Role. #1032 has a code example.

NRT ( 1 to 3 hrs) (VJ114IMG_NRT)

MAAP federated tokens appears to work.

from maap.maap import MAAP
maap = MAAP(maap_host='api.maap-project.org')
results = maap.searchGranule(
     cmr_host='cmr.earthdata.nasa.gov',
     short_name=sn)
print(len(results))
results[0].getData(".")

Could also be done with EarthAccess, but would need secrets stored cc: @sujen1412
AFAIK EarthAccess doesn't support Auth Tokens yet, I think there's an open ticket? @chuckwondo

import earthaccess
earthaccess.login(strategy="interactive")
product = "VJ114IMG_NRT"
granules = earthaccess.DataGranules().short_name(product).get(10)
data_links = [granule.data_links(access="DATA") for granule in granules]
print(data_links)

Other Notes

Trying to better understand what this means

Use of the VJ103MODLL data product is required to apply accurate geolocation information to the VNP14IMG Science Datasets (SDS).

[chuckwondo]

@scoffiel, yes, s3 stuff is expiring hourly, which is why I created the earthaccess ticket I linked earlier in this discussion, which you may have missed, as it's only a single line (nsidc/earthaccess#765). That ticket is to enhance earthaccess to auto-refresh the s3 creds.

Using https, however, your EDL user token will be used instead of s3 creds, and your EDL user tokens are good for 60 days.

[chuckwondo]

Hang on a sec. I just noticed that your output shows https URLs, not S3 URLs, so I'm confused as to why you're seeing hourly expiry. Your code above is not actually using the fs variable, so you're not using S3 at all. In what environment did you run that code above?

[wildintellect]

@scoffiel let me clarify, you should use different methods for these.

• NRT (LAADS) there is no S3 access, you need to use an EDL method maap-py or earthaccess
• Non NRT comes from LP DAAC, there is S3 access, use the maap-data-reader role. You could use the same method as the NRT if you want to only have 1 method, but use the LP DAAC urls for better reliability.

[scoffiel]

@chuckwondo @wildintellect I've spent a lot more time on this the past couple days trying to better understand the differences between s3 and https, and the multiple ways to get credentials and use these packages. A couple main findings:

• Yes, Chuck you're right in your previous comment - my most recently shared code wasn't actually using an s3 session, just https, and that piece of code wasn't creating an hourly expiry issue. The source of the expiry was a different chunk of code (next bullet). Sorry for the confusion!
• The full story is that I'm working with multiple datasets at once - Level 1 VIIRS data (eg., VNP02IMG, VNP03IMG) come from LAADS and Level 2 VIIRS data (eg., VNP14IMG) come from LPDAAC. I think I've figured out that switching between those DAACs for s3 access was breaking my login/authentication via earthaccess. Does this sound right / are you two aware of this type of issue?
• Alex, the maap-data-reader role seems to work great, except not on the L1 LAADS data. Is it not set up yet for LAADS or am I doing something wrong?
• Alex I also see what you mean now about NRT data from LAADS not having S3 access. I think I can still stream the VNP14IMG_NRT files via https, but for the larger VNP03IMG_NRT files I would need to EDL or maap getData download them (I'm not there yet - just working with the archive sets).

If you both have the time I'd appreciate you looking over my notebook (here and here on MAAP: /projects/shared-buckets/coffield/scripts/fetch_VIIRS_cloud.ipynb) where I tried to demonstrate all these things, to see if my interpretation is correct.

To summarize - I think I'm good to go for the time being, aside from not being able to use the maap method for the L1 LAADS data (VNP02IMG, VNP03IMG). I'll keep doing earthaccess for now and re-logging in between datasets, but keeping an eye on Chuck's ticket.

[wildintellect]

@scoffiel

• NRT(LAADS) data is only available via https, and yes with EarthAccess or maap-py you can access that (Streaming might not be the right term). Downloading on demand is often a good approach.
• Yes switching between DAACs using the same EDL session could cause issues, guaranteed for s3 because s3 actually issues you an AWS key good only at 1 DAAC for an hour, you probably want a fresh session each new DAAC you connect to. And you can talk to @chuckwondo about how to auto-refresh if your code is going to spend more than an hour downloading. There's also an issue if you make too many session requests in a short amount of time, i.e. 1000 jobs/threads all ask for separate tokens instead of sharing.
• As I said before and this is documented, maap-reader-role only works with 5 DAACs, LAADS is not currently on the list. https://docs.maap-project.org/en/latest/technical_tutorials/access/direct_access.html - we can try to request to get this working now that there's a use case. The same goes for MAAP-PY methods, the API has not been configured for LAADS but we can change that. cc: @bsatoriu
• URLs returned from your EarthData Searches/EarthAccess/MAAP-py show which DAAC a file is coming from, eg. s3://prod-lads/VNP02IMG/VNP02IMG.A2024201.1142.002.2024201185155.nc is LAADS, lp-protected would be LPDAAC, etc.. the metadata also states "dataCenter": "LAADS", and there's a flag in searches if you want to limit to cloud hosted.

Most of these things can be figure out by using https://search.earthdata.nasa.gov/ to understand where files are coming from and which access method you want to use.

Looking at your code, you are correct for https only files I do not recommend xarray.open , but rather that you download the file first to a local directory like /tmp ...

The HTTP server doesn't appear to support range requests
That's the advantage of using S3 when it's available, it is much easier to read only parts of files instead of the whole thing.

FYI for the FIRMS txt files, https://firms.modaps.eosdis.nasa.gov/api/area/ the API docs seem straight forward. Register for a token, supply with API requests (URLs)