The Dublinbikes feed tells you how many bikes are at each station right now. It doesn’t keep history. So if you want to ask “how do weather and time of day change how many bikes are free?”, you first need a pipeline that collects the data, keeps it safe and tells you when something breaks.
I built one, end to end, using only free tiers. The code is on GitHub and the project page has a live dashboard.
1. Collect: small, boring and append-only
A Python script runs on GitHub Actions every 15 minutes. It reads the Dublinbikes GBFS feed (all 115 stations) and, once an hour, the Met Éireann observations for Dublin Airport. Both are open data under CC BY 4.0.
Each run writes a new Parquet file, partitioned by date, to a public Hugging Face dataset. Nothing is ever overwritten, so a bad run can’t damage history.
Two lessons from this step:
- GitHub’s own scheduler is unreliable for frequent jobs. My 15-minute schedule didn’t fire once in its first hour. A free external timer (cron-job.org) that calls GitHub’s API fixed it.
- Fail loudly, then stop. After three failed runs in a row the workflow switches itself off and sends me a Telegram message, instead of failing silently for days.
2. Model: dbt on DuckDB
Every morning, dbt builds the models inside a DuckDB file:
- Staging: proper types, UTC timestamps (the weather comes in Irish local time) and de-duplication. Each hourly weather file repeats the whole day so far, so only the latest copy of each observation is kept.
- Intermediate: one row per station per hour, with average, minimum and maximum bikes, and the share of the hour the station was empty or full.
- Marts: a small star schema.
fct_station_hoursits in the middle, withdim_station(name, location, capacity) anddim_hour(Dublin local time, weekday and that hour’s weather).
The fact table is incremental: each run only recomputes the last three hours. It is also idempotent: running it twice gives exactly the same table. That matters, because reruns are the first thing you reach for when something goes wrong.
DuckDB writes everything to a single file, so I store it in a second Hugging Face dataset between runs. That’s how the incremental table “remembers” what it already built without a database server.
3. Test: 14 checks, every run
dbt runs 14 data tests on every build:
- keys are unique and never empty;
- every row in the fact table links to a station and an hour;
- bike and dock counts are never negative;
- freshness: the newest snapshot is less than 2 hours old;
- volume: at least 75% of the expected snapshots arrived in the last 24 hours.
The last two catch the real-world failures: a feed that goes quiet, or a collector that silently stops. If any check fails, I get a Telegram alert.
The data also taught me something: one “station” in the feed is a placeholder that never has a bike. Left in, it tops every “most often empty” list. The dashboard now leaves out stations that never had a bike in the period.
4. Publish: a dashboard with no backend
Each run publishes the tables as Parquet, a run log and a small dashboard.json summary. The project page fetches that file in the browser and draws the chart. There’s no server to keep running, and the dashboard updates as soon as the morning run finishes. The dbt docs and lineage graph are published as a free static page.
What it costs
€0: public-repo GitHub Actions minutes, free Hugging Face datasets and a static Space, and one free cron-job.org job.
What I’d do differently
- Compact the small files. About 100 small Parquet files a day is fine for now, but over a year I’d merge them into daily files.
- Add a “late data” test per station, not just overall, to spot a single broken dock.
What’s next
The pipeline is the easy part. The question needs weeks of data across different weather before any answer is honest. Once that has built up, I’ll publish the analysis and write up what the data says.