← Back to selected workData engineering / full-stackPublic source
Biblio Explorer
Turning a large bibliographic dataset into searchable, comparable research profiles.
Team project · Data workflows, application development and testingPython / Flask / MySQL / JavaScript / D3.js / pytest
The contextA university team project integrating DBLP publication data with iCORE conference rankings and journal-ranking data. The ETL pipeline processes around 2.5 million publication records into a unified relational schema.
View repository ↗ Data engineering / full-stack↗
Publication records processed2.5M+
ETLClean & match venues
APIProfiles & search
D3.jsInteractive comparisons
Explore the work ↗How it connects
The workflow.
- 01DBLP & rankings
- 02Python ETL
- 03MySQL + Flask
- 04JavaScript / D3.js
Data is cleaned and matched before the dashboard exposes search, profiles and comparisons through REST endpoints.
Engineering details
Inside the work.
01ETL and relational modeling
Cleaned and transformed publication records, modeled authors through many-to-many relationships, and matched venue names to ranking datasets. Publications without a venue match are retained so author and year analytics do not silently lose data.
02APIs backed by analytical SQL
Flask Blueprints organize conference, journal, author and year endpoints. MySQL connection pooling and SQL views support the dashboard’s analytical queries.
03Interactive exploration
JavaScript, HTML/CSS and D3.js power autocomplete search, year-range filters, profile tables and interactive multi-venue comparison charts.
04Validation and continuous checks
The pytest suite covers API contracts, ETL behavior, venue matching and frontend layout. GitHub Actions runs pytest and flake8 checks; API rate limiting and environment-driven debug configuration support safer application behavior.
What I took from it.
- Preserving unmatched records makes coverage limitations explicit instead of hiding them.
- Database-level aggregation can keep an analytical frontend focused on presentation.
- Testing data transformations is essential when a plausible-looking chart can still contain incorrect data.
See for yourself
Source & documentation.