← Back to selected work
Data engineering / full-stackPublic source

Biblio Explorer

Turning a large bibliographic dataset into searchable, comparable research profiles.

Team project · Data workflows, application development and testingPython / Flask / MySQL / JavaScript / D3.js / pytest
The context

A university team project integrating DBLP publication data with iCORE conference rankings and journal-ranking data. The ETL pipeline processes around 2.5 million publication records into a unified relational schema.

View repository ↗
How it connects

The workflow.

  1. 01DBLP & rankings
  2. 02Python ETL
  3. 03MySQL + Flask
  4. 04JavaScript / D3.js

Data is cleaned and matched before the dashboard exposes search, profiles and comparisons through REST endpoints.

Engineering details

Inside the work.

01

ETL and relational modeling

Cleaned and transformed publication records, modeled authors through many-to-many relationships, and matched venue names to ranking datasets. Publications without a venue match are retained so author and year analytics do not silently lose data.

02

APIs backed by analytical SQL

Flask Blueprints organize conference, journal, author and year endpoints. MySQL connection pooling and SQL views support the dashboard’s analytical queries.

03

Interactive exploration

JavaScript, HTML/CSS and D3.js power autocomplete search, year-range filters, profile tables and interactive multi-venue comparison charts.

04

Validation and continuous checks

The pytest suite covers API contracts, ETL behavior, venue matching and frontend layout. GitHub Actions runs pytest and flake8 checks; API rate limiting and environment-driven debug configuration support safer application behavior.

What I took from it.

  • Preserving unmatched records makes coverage limitations explicit instead of hiding them.
  • Database-level aggregation can keep an analytical frontend focused on presentation.
  • Testing data transformations is essential when a plausible-looking chart can still contain incorrect data.
See for yourself

Source & documentation.

Architecture & setup↗Automated tests↗ETL pipeline↗CI workflow↗