Initial conditions#

aifs-modal supports four sources of initial conditions, selected with the ic_source= argument of run_forecast. All sources write identical zarr group layouts into the shared Modal IC Volume (aifs-ics), so any source can feed any forecast.

ic_source

Coverage

Credentials

Notes

"ifs-arraylake" (default)

~ last 2 weeks

ArrayLake account + dataset subscription

Brightband ECMWF IFS dataset on the Earthmover ArrayLake marketplace. Fastest and most reliable.

"ifs-ekd"

~ last 2 years

none

ECMWF open-data S3 archive (AWS), via earthkit-data. No credentials needed, but frequent 503 Slow Down errors make it unreliable.

"era5-arco"

1979 — ~5 days ago

none

ERA5 reanalysis on GCS; ingestion runs on Modal in us-central1 for in-region bandwidth.

"era5-cds"

1979 — ~5 days ago

~/.cdsapirc

ERA5 from the Copernicus CDS, via earthkit-data.

Ingestion is triggered automatically as part of run_forecast(..., ic_source=...) — already-ingested dates are skipped. For ifs-arraylake and era5-arco, run_forecast dispatches to a dedicated co-located Modal function (ingest_ifs_arraylake in us-east, ingest_era5_arco in us-central1); for ifs-ekd and era5-cds it runs inline. Each source can also be invoked separately, which is useful to pre-populate a date window in advance.

Note

ERA5 publication lags real time by about 5 days, so the ERA5 sources are suitable for hindcasts and reforecasts but not for operational use.

import datetime as dt

import aifs_modal

# from arraylake/earthmover
ic_source_repo = "martibosch/ecmwf-ifs-hres-ics-open"

_today_00z = dt.datetime.now(dt.UTC).replace(hour=0, minute=0, second=0, microsecond=0)

# IFS sources (arraylake + ekd): use 2 days ago — safely within all rolling windows
ifs_date = _today_00z - dt.timedelta(days=2)

# ERA5 sources lag real time by ~5 days; use 7 days back to be safe
era5_date = _today_00z - dt.timedelta(days=7)

print(f"IFS  date : {ifs_date.isoformat()}")
print(f"ERA5 date : {era5_date.isoformat()}")

1. Brightband IFS on ArrayLake (ifs-arraylake), default#

The ECMWF IFS initial conditions are ingested from the Brightband dataset on the Earthmover ArrayLake marketplace, which provides cloud-native zarr access — significantly faster and more reliable than the open-data archive (see ingestion benchmark). The dataset is updated 7 hours after each synoptic hour (00/06/12/18 UTC) and kept as a rolling window of approximately 15 days; dates older than that are not available.

This is the default source. It requires:

  1. A free ArrayLake account.

  2. A free subscription to the Brightband dataset on the marketplace — see getting started.

  3. Your ARRAYLAKE_API_TOKEN exported in the environment as well as set up as a Modal secret named “arraylake-api-token” with the ARRAYLAKE_API_TOKEN key set to the token value.

The dataset is hosted on Cloudflare R2 ENAM (US East); ingest_ifs_arraylake runs in Modal us-east for in-region bandwidth. Pass ic_source="ifs-arraylake" (or omit ic_source=) and ic_source_repo= to run_forecast to trigger automatic in-region ingestion. The cells below show standalone ingestion on Modal (recommended) and locally.

# standalone, on Modal in us-east — writes to the Modal IC Volume
with aifs_modal.app.run():
    aifs_modal.ingest_ifs_arraylake.remote(
        (ifs_date - dt.timedelta(hours=6)).isoformat(),
        ifs_date.isoformat(),
        ic_source_repo=ic_source_repo,
    )

2. ECMWF open-data (ifs-ekd), no-credentials fallback#

Operational IFS analysis from the ECMWF open-data S3 archive on AWS. No credentials are needed, but the public shared bucket frequently returns 503 Slow Down errors under load, which can stall ingestion for minutes at a time. Use this only as a fallback when ArrayLake credentials are unavailable.

Coverage is an operational rolling window corresponding roughly to the last two years. The cell below shows standalone ingestion; pass ic_source="ifs-ekd" to run_forecast to use it automatically.

3. ERA5 from ARCO-ERA5 (era5-arco)#

Analysis-Ready, Cloud-Optimized ERA5 (ARCO-ERA5) on Google Cloud Storage. Anonymous read access. Use this for hindcasts and reforecasts on dates outside the open-data archive’s rolling window.

The store is hosted in us-central1, so ingestion is fastest co-located there. aifs-modal exposes a Modal-side function ingest_era5_arco that runs in us-central1 for in-region bandwidth; pass ic_source="era5-arco" to run_forecast for automatic in-region ingestion. The cells below show standalone ingestion on Modal (recommended for batch runs) and locally.

# standalone, on Modal in us-central1 — writes to the Modal IC Volume
with aifs_modal.app.run():
    aifs_modal.ingest_era5_arco.remote(
        (era5_date - dt.timedelta(hours=6)).isoformat(),
        era5_date.isoformat(),
    )

4. ERA5 from CDS (era5-cds)#

ERA5 reanalysis from the Copernicus Climate Data Store via earthkit-data. Same time coverage as ARCO-ERA5. Requires a valid CDS API configuration — either ~/.cdsapirc or CDSAPI_URL / CDSAPI_KEY environment variables.

Generally slower than ARCO (GRIB over HTTPS rather than cloud-native zarr); useful when you don’t have GCP egress or want CDS’s exact byte-for-byte values.

See also#