← All projects

Dublin bikes + weather data pipeline

A data engineering portfolio project by Malav Naik: a scheduled Python collector, dbt + DuckDB models with freshness and volume checks, and a live dashboard of Dublinbikes availability and weather.

  • Python
  • GitHub Actions
  • dbt
  • DuckDB
  • Parquet
  • Hugging Face Datasets
  • GBFS
  • Met Éireann

Code on GitHub →

The problem

Public city data is published live and then gone: the Dublinbikes feed only shows right now. To answer a question like “how do weather and time of day change how many bikes are free?” you need history, collected reliably and modelled so it can be trusted.

What I built

An end-to-end pipeline that runs on free tiers only:

  • a collector that saves a snapshot of all Dublinbikes stations every 15 minutes, plus the hourly Dublin weather, without ever overwriting old data;
  • dbt + DuckDB models that turn raw snapshots into a small star schema, with automatic data checks;
  • a live dashboard (below) that reads the newest results straight from the published data.

How it works

Flow: Dublinbikes and Met Éireann feeds are collected every 15 minutes into raw Parquet files on Hugging Face; a daily dbt run on DuckDB builds staging, hourly and star-schema tables with tests; the results feed the dashboard, and failures alert on Telegram Dublinbikes feed (GBFS)Met Éireann weather Every 15 min: raw Parquet, never overwritten Daily: dbt + DuckDB models and data checks Live dashboardTelegram alert if a check fails
  1. Collect: a Python job on GitHub Actions, started every 15 minutes, saves each station’s bikes and free docks, and once an hour the Dublin Airport weather, as date-partitioned Parquet files in a public Hugging Face dataset. After three failures in a row it switches itself off and messages me on Telegram.
  2. Model: every morning dbt builds, in DuckDB: staging views (types, time zones, de-duplication) → an hourly summary per station → a star schema: fct_station_hour with dim_station and dim_hour (local time, weekday and that hour’s weather). The fact table is incremental and idempotent: each run recomputes only the last few hours, and rerunning gives the same result. See the dbt docs and lineage graph.
  3. Check: 14 automatic data tests run every time: keys unique and never empty, every fact linked to its dimensions, counts never negative, freshness (newest data under 2 hours old) and volume (at least 75% of the expected snapshots in the last day). A failure sends me a Telegram alert.
  4. Publish: the tables, a run log and the dashboard summary go to a second Hugging Face dataset. Cost: €0.

Live dashboard

Loading the latest data…

What’s next

The question needs weeks of data, across different weather, before any answer is honest. Once enough has built up, I’ll publish an analysis notebook and write up what the data says.

Data and credits