{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": "# Initial conditions\n\n`aifs-modal` supports four sources of initial conditions, selected with the `ic_source=` argument of `run_forecast`. All sources write identical zarr group layouts into the shared Modal IC Volume (`aifs-ics`), so any source can feed any forecast.\n\n| `ic_source` | Coverage | Credentials | Notes |\n| --- | --- | --- | --- |\n| `\"ifs-arraylake\"` *(default)* | ~ last 2 weeks | ArrayLake account + dataset subscription | Brightband ECMWF IFS dataset on the Earthmover ArrayLake marketplace. Fastest and most reliable. |\n| `\"ifs-ekd\"` | ~ last 2 years | none | ECMWF open-data S3 archive (AWS), via earthkit-data. No credentials needed, but frequent `503 Slow Down` errors make it unreliable. |\n| `\"era5-arco\"` | 1979 — ~5 days ago | none | ERA5 reanalysis on GCS; ingestion runs on Modal in `us-central1` for in-region bandwidth. |\n| `\"era5-cds\"` | 1979 — ~5 days ago | `~/.cdsapirc` | ERA5 from the Copernicus CDS, via earthkit-data. |\n\nIngestion is triggered automatically as part of `run_forecast(..., ic_source=...)` — already-ingested dates are skipped. For `ifs-arraylake` and `era5-arco`, `run_forecast` dispatches to a dedicated co-located Modal function (`ingest_ifs_arraylake` in `us-east`, `ingest_era5_arco` in `us-central1`); for `ifs-ekd` and `era5-cds` it runs inline. Each source can also be invoked separately, which is useful to pre-populate a date window in advance.\n\n```{note}\nERA5 publication lags real time by about 5 days, so the ERA5 sources are suitable for hindcasts and reforecasts but not for operational use.\n```"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "\n",
    "import aifs_modal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2",
   "metadata": {},
   "outputs": [],
   "source": "# from arraylake/earthmover\nic_source_repo = \"martibosch/ecmwf-ifs-hres-ics-open\"\n\n_today_00z = dt.datetime.now(dt.UTC).replace(hour=0, minute=0, second=0, microsecond=0)\n\n# IFS sources (arraylake + ekd): use 2 days ago — safely within all rolling windows\nifs_date = _today_00z - dt.timedelta(days=2)\n\n# ERA5 sources lag real time by ~5 days; use 7 days back to be safe\nera5_date = _today_00z - dt.timedelta(days=7)\n\nprint(f\"IFS  date : {ifs_date.isoformat()}\")\nprint(f\"ERA5 date : {era5_date.isoformat()}\")"
  },
  {
   "cell_type": "markdown",
   "id": "3",
   "metadata": {},
   "source": "## 1. Brightband IFS on ArrayLake (`ifs-arraylake`), default\n\nThe ECMWF IFS initial conditions are ingested from the [Brightband dataset on the Earthmover ArrayLake marketplace](https://app.earthmover.io/marketplace/697162921880507a6587c31b), which provides cloud-native zarr access — significantly faster and more reliable than the open-data archive (see [ingestion benchmark](a03-ingestion-benchmark.ipynb)). The dataset is updated 7 hours after each synoptic hour (00/06/12/18 UTC) and kept as a rolling window of approximately 15 days; dates older than that are not available.\n\nThis is the default source. It requires:\n\n1. A *free* [ArrayLake account](https://app.earthmover.io).\n2. A *free* subscription to the Brightband dataset on the marketplace — see [getting started](https://docs.earthmover.io/marketplace/data-users#getting-started).\n3. Your `ARRAYLAKE_API_TOKEN` exported in the environment as well as set up as [a Modal secret](https://modal.com/docs/guide/secrets) named \"arraylake-api-token\" with the `ARRAYLAKE_API_TOKEN` key set to the token value.\n\nThe dataset is hosted on Cloudflare R2 ENAM (US East); `ingest_ifs_arraylake` runs in Modal `us-east` for in-region bandwidth. Pass `ic_source=\"ifs-arraylake\"` (or omit `ic_source=`) and `ic_source_repo=` to `run_forecast` to trigger automatic in-region ingestion. The cells below show standalone ingestion on Modal (recommended) and locally."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": [],
   "source": "# standalone, on Modal in us-east — writes to the Modal IC Volume\nwith aifs_modal.app.run():\n    aifs_modal.ingest_ifs_arraylake.remote(\n        (ifs_date - dt.timedelta(hours=6)).isoformat(),\n        ifs_date.isoformat(),\n        ic_source_repo=ic_source_repo,\n    )"
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {},
   "source": "## 2. ECMWF open-data (`ifs-ekd`), no-credentials fallback\n\nOperational IFS analysis from the [ECMWF open-data S3 archive](https://www.ecmwf.int/en/forecasts/datasets/open-data) on AWS. No credentials are needed, but the public shared bucket frequently returns `503 Slow Down` errors under load, which can stall ingestion for minutes at a time. Use this only as a fallback when ArrayLake credentials are unavailable.\n\nCoverage is an operational rolling window corresponding roughly to the last two years. The cell below shows standalone ingestion; pass `ic_source=\"ifs-ekd\"` to `run_forecast` to use it automatically.\n\n## 3. ERA5 from ARCO-ERA5 (`era5-arco`)\n\n[Analysis-Ready, Cloud-Optimized ERA5](https://github.com/google-research/arco-era5) (ARCO-ERA5) on Google Cloud Storage. Anonymous read access. Use this for hindcasts and reforecasts on dates outside the open-data archive's rolling window.\n\nThe store is hosted in `us-central1`, so ingestion is fastest co-located there. `aifs-modal` exposes a Modal-side function `ingest_era5_arco` that runs in `us-central1` for in-region bandwidth; pass `ic_source=\"era5-arco\"` to `run_forecast` for automatic in-region ingestion. The cells below show standalone ingestion on Modal (recommended for batch runs) and locally."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# standalone, on Modal in us-central1 — writes to the Modal IC Volume\n",
    "with aifs_modal.app.run():\n",
    "    aifs_modal.ingest_era5_arco.remote(\n",
    "        (era5_date - dt.timedelta(hours=6)).isoformat(),\n",
    "        era5_date.isoformat(),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {},
   "source": [
    "## 4. ERA5 from CDS (`era5-cds`)\n",
    "\n",
    "ERA5 reanalysis from the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu) via earthkit-data. Same time coverage as ARCO-ERA5. Requires a valid CDS API configuration — either `~/.cdsapirc` or `CDSAPI_URL` / `CDSAPI_KEY` environment variables.\n",
    "\n",
    "Generally slower than ARCO (GRIB over HTTPS rather than cloud-native zarr); useful when you don't have GCP egress or want CDS's exact byte-for-byte values.\n",
    "\n",
    "## See also\n",
    "\n",
    "- [Initial-conditions ingestion benchmark](a03-ingestion-benchmark.ipynb) — wall-clock comparison of the four backends on the same date range and target bucket."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (Pixi)",
   "language": "python",
   "name": "pixi-kernel-python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  },
  "pixi-kernel": {
   "environment": "user-guide"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}