SPLap 2.1.1 · engineering notes

Importing a race into SPLap

SPLap ships with one real session already loaded (2025 Italian Grand Prix, Monza, race window 14:00–14:13:30 UTC, from OpenF1) plus the labelled synthetic test fixture. Any other historical session can be imported three ways. All three produce the same canonical events (docs/DATA_SCHEMA.md) in one index (default splap), keyed by an idempotent import_id, and the Pit Wall reads them through the same backend as the packaged session.

No provider key is needed: OpenF1 is open. No credential is ever placed in browser JavaScript, source control or logs.

1. Connect Data wizard (browser)

Open Connect Data → Import Historical Race (/app/splap/connect_data?source=openf1).

1. Season – seasons are listed from OpenF1 meetings.
2. Grand Prix – meetings of that season, newest first.
3. Session – Practice/Qualifying/Sprint/Race; Race is preselected.
4. Capability preview – the wizard probes drivers, laps, pit, race_control, weather and a 10-second location/car_data sample and shows what the provider actually has for this session (counts, sample rate, missing channels). Nothing is written yet.
5. Profile and window –
6. Quick – location (positions on the map) only, no car_data; smallest.
7. Engineering – location + car_data (speed, gear, throttle, brake, DRS) for every driver.
8. Selected drivers – engineering channels for the drivers you tick.
9. The window (UTC HH:MM:SS, defaults to the race start) bounds the telemetry fetch. Laps, stints, pits, race control, weather, overtakes and positions are always fetched for the whole session; intervals from two minutes before the window. The estimate box shows expected events, ingest volume and provider requests.
10. Index – type an index name and Check. The wizard tells you whether it exists and whether your role can write. Create index uses data/indexes if your role allows it. Indexed events use the app's props.conf for timestamps, so the sourcetypes must be splap:* (the wizard enforces this).
11. Import – the wizard first computes the import_id and refuses if that id is already in the index (tick force re-import only if you really want duplicates), then fetches at ≤ 2.5 requests/s, derives the circuit geometry and 500 ms replay frames in the browser, and streams NDJSON batches to /services/receivers/stream?index=…&sourcetype=splap:… using your own Splunk Web session and CSRF form key. Each batch is retried up to four times; Cancel stops after the current batch. The log panel lists every provider request and every write.
12. Verify and open – counts per sourcetype for this import_id straight from the index, first/last timestamp, circuit summary, then Open in Pit Wall (pit_wall?index=<index>&session_key=<key>).

Import status events (splap:import_status, phases start/complete/failed) are written alongside the data and listed at the bottom of Connect Data.

If the browser cannot reach OpenF1

Corporate proxies often block api.openf1.org. Install Doctor → Provider connectivity tells you. Use a bundle or the CLI instead; the app itself never needs internet.

2. Offline race bundle

Produce a bundle on any machine with internet:

python3 tools/splap_import.py fetch --session 9912 \
    --window 2025-09-07T14:00:00 2025-09-07T14:13:30 --profile engineering \
    --out bundles/monza_2025 --zip
python3 tools/splap_import.py validate --bundle bundles/monza_2025

The folder (and .zip) contains manifest.json, session.json, drivers.json, laps.ndjson.gz, telemetry.ndjson.gz, location.ndjson.gz, race_events.ndjson.gz, replay_frames.json.gz, track.geojson, team_palettes.json, validation.json, ATTRIBUTION.md and checksums.json (SHA-256 of every file; readers refuse a bundle whose checksums do not match). Format details: docs/DATA_SCHEMA.md § Offline bundle.

Load it either in the browser (Connect Data → Load SPLap race bundle, pick the folder; files are decompressed with DecompressionStream and verified against checksums.json before anything is written) or with the CLI:

SPLUNK_PASSWORD='…' python3 tools/splap_import.py ingest --bundle bundles/monza_2025.zip \
    --splunk https://splunk.example:8089 --user admin --index splap --create-index --verify

The committed bundles/monza_2025_window/ is the exact dataset the packaged demo was built from and is what the unit tests run against.

3. CLI end to end

python3 tools/splap_import.py seasons                    # 2023 … 2026
python3 tools/splap_import.py meetings 2025              # meeting_key, name, circuit, dates
python3 tools/splap_import.py sessions --meeting 1268    # session_key 9912 = Race
python3 tools/splap_import.py fetch --session 9912 --window 2025-09-07T14:00:00 2025-09-07T14:13:30 --out bundles/monza_2025
python3 tools/splap_import.py ingest --bundle bundles/monza_2025 --hec https://splunk.example:8088 --index splap
python3 tools/splap_import.py doctor                     # python version, fastf1 presence, cache, provider reachability

Options: --profile quick|engineering|selected|full, --no-telemetry, --step-ms 250..500 (replay frame period), --cache DIR (raw OpenF1 responses are cached as JSON so re-runs are offline and deterministic), --force (re-import an existing import_id), --no-frames (skip splap:replay_frame events — only for analysis-plane use; the Pit Wall needs replay frames and reports "No replay frames for this session" without them).

Credentials: --password/--token/--hec-token or SPLUNK_PASSWORD / SPLUNK_TOKEN / SPLUNK_HEC_TOKEN. They are used for the request only and never written anywhere. --verify prints per-sourcetype counts after ingest.

FastF1 (optional)

If the fastf1 package is installed, splapdata.providers_extra.fastf1_fetch(year, gp, session) returns canonical lap, car_data, location, weather and race_control events from the FastF1 timing feed at 240 ms resolution. It is a Python-only path (FastF1 has no browser equivalent) and produces no session/driver/position/interval events itself, so today it is a supplement to an OpenF1 bundle rather than a replacement. See docs/DATA_SOURCES.md.

4. Making a session the packaged demo

fixture writes the three lookups (splap_monza_2025_events/telemetry/frames.csv.gz), the static replay assets under appserver/static/sessions/<name>/, the registry entry in appserver/static/data/sessions.json, and appends the transforms.conf lookup stanzas if they are missing:

python3 tools/splap_import.py fixture --session 9912 --window 2025-09-07T14:00:00 2025-09-07T14:13:30 --name monza_2025

Registry entries with "data_mode": "synthetic" must carry SYNTHETIC in their label; the tests enforce it.

5. Idempotence and re-imports

import_id = "imp_" + sha256(json.dumps({"p": provider, "s": str(session_key), "profile": profile, "w": window}, sort_keys=True))[:16]. The browser (importer.js, pure-JS SHA-256 when WebCrypto is unavailable) and the CLI produce the same id for the same inputs — tests/test_splapdata.py pins imp_789c180679bb8775 for the 14:05–14:08 quick window. Before any write, both check index=<idx> import_id=<id> sourcetype=splap:session and stop if it exists. Changing the window or profile is a new import_id, so the same session can coexist at different resolutions; the Pit Wall's session picker (Look for imported sessions) lists each session_key once (all frames for that session key are read, so keep one import per session per index unless you use --force deliberately).

To remove an import: | delete on index=splap import_id=<id> (requires the can_delete role) or clean the index.

6. Troubleshooting