ETL Pipeline: From heterogeneous bibliographic data to a unified schema

[marioloskovic55-jpg]

Function of the project

The goal of this pull request is to add a complete Data Extraction, Transformation, and Loading (ETL) pipeline to bibliometrix-python, so that we have clean, standardized data from the source and can proceed with analysis.
The work starts from a problem present in bibliometrix-python: it only works with data downloaded from Web of Science (WoS), while it encounters problems with those downloaded from PubMed, Scopus or Dimensions because the column names are different.
The created pipeline acts as a universal translator: wherever the data comes from, the output will always be standard so that it can be understood by all analytical functions.

Problems found in the current implementation

-There was no function in bibliometrix-python that would load and standardize the data (there is convert2df, but in the R version).

-There was no clear structure to the transformation logic, making it difficult to maintain or extend.

-Crashes were caused by incorrectly storing data in the wrong formats (for example, in the case of authors AU or cited references CR, the lists were incorrectly stored as strings)

-Many functions crashed when handling missing values (NaN, None)

-The data was pre-determined to come from Web of Science, so if it came from other sources the code would crash.

-There was no block that validated the data, but it was directly passed to the analytical functions without checking the formatting.

Building

5 modules have been created within the www/services/etl folder, each with a single, specific purpose.

1. Mapping (www/services/etl/mappings/)

Different sources use different names for the same information: for example, if the title of an article is indicated with "Title" in Scopus and Dimensions, in Web of Science it is indicated with "TI".

I created a dictionary file for each source that provides a table:

• scopus_mapping.py — maps Scopus column names to WoS standard tags
• dimensions_mapping.py — maps Dimensions column names to WoS standard tags
• pubmed_mapping.py — maps PubMed field tags to WoS standard tags
• openalex_mapping.py — maps OpenAlex field names to WoS standard tags
  These dictionaries are just lookup tables used by the transformer.

2. Transformer (www/services/etl/transformer.py)

This module has 5 steps:

• Renames columns according to the name used in WoS
• Applies list format to columns containing string formats instead of the correct format (such as authors "AU")
• Applies integer format to columns containing strings instead of integers (such as number of citations "TC") and sets missing data to 0
• Fills in missing columns if a source doesn't provide one, always outputs all 24 mandatory columns, using empty default values where data is missing
• Calculates the "SR" (Short Reference), a unique identifier for each article, automatically from the other columns. It consists of: "FirstAuthorSurname, PublicationYear, JournalName".

3. Validator (www/services/etl/validator.py)

The validator checks the data before providing it to the analytical functions. Specifically, it performs three checks:

1. It verifies that all 24 predefined columns are present.
2. It verifies that all columns are free of Nan and None.
3. It verifies that the format of all column data is correct.
   The validator prints a clear [OK] or [FAIL] for each check.

4. Api Retriever (www/services/etl/api_retriever.py)

This module retrieves data without the need for manual downloading, but via internet.
It draws sources from two free public databases:

1. OpenAlex - a search query is sent (for example, "machine learning") and output articles in JSON format. The module ensures that the results are displayed in 25-page spreads, limits the query rate to once every 0.5 seconds, and automatically retries up to a maximum of 3 times.
2. PubMed - the US biomedical health database. In this case, we first search for article IDs and then download the corresponding full records. The output is in MEDLINE text format and is parsed line by line.

5. Standardizer (www/services/etl/standardizer.py)

This is the equivalent of the convert2df function in R and is the entry point of the pipeline.
You only need to call this function.
It works in two modes:

• File mode (Base level) - load a manually exported file:

from www.services.etl.standardizer import convert2df
df = convert2df(source="scopus", filepath="scopus_export.csv")

• API mode (Advanced level) - retrieve data automatically:

df = convert2df(source="openalex", query="machine learning", max_results=100)
df = convert2df(source="pubmed", query="deep learning", max_results=50)

Internally, convert2df() coordinates all the other modules in order:
Extract -> Transform -> Validate. The user does not need to know anything about the internal modules..

Modifications to Existing Files

www/services/__init__.py

Originally, the file imported all modules using wildcards, which caused problems because numerous additional installations (such as selenium, pyvis, etc.) were required when only the ETL pipeline was needed. Therefore, the imports were replaced with a comment so that it is still possible to import modules if necessary.

Validation results

The pipeline passed validation in tests from 3 different sources:

• Scopus CSV file (20 articles) - all columns present, no missing data, and all typing correct. OK
• OpenAlex "machine learning" API query (10 articles) - passed the same checks. OK
• PubMed "machine learning" API query (10 articles) - passed the same checks. OK
  All three sources passed full validation without any crashes.

Demo notebook

The demo_etl_pipeline.ipynb file contains a step-by-step process starting from the raw pre-transformation data and displaying the standardized outputs after each stage.

[marioloskovic55-jpg]

Fixes applied to existing functions

During testing, a systematic bug was found in 38 files in the www/services/ and functions/ directories and has been fixed.

Bug 1

The original files used df.get() and df.set(M) as if calling a pre-existing custom class. But df is a standard pandas DataFrame, so they were called without existing, causing crashes.

Fix 1

• All df.get() statements have been replaced with df.copy()
• In metatagextraction.py, df.set(M) has been replaced with return M

Bug 2

Several functions in functions/ had several missing imports from the standard library.

Fix 2

• get_maininformations.py: added import time, import pandas as pd, and from www.services.metatagextraction import metaTagExtraction
• get_annualproduction.py: added import pandas as pd, import plotly.express as px, import plotly.graph_objects as go

Testing

After the corrections, the following functions were tested using standardized Scopus data and worked correctly:

• get_maininformations
• get_annualproduction

[marioloskovic55-jpg]

Additional fixing applied

Bug 3

After importing pandas and plotly into all functions, there were other missing imports

Fix 3

• All functions in functions/: added import plotly.express as px and import plotly.graph_objects as go
• get_lotkalaw.py: added import numpy as np
• get_frequentwords.py: added from collections import Counter
• get_countriesproduction.py: added import geopandas as gpd and from www.services.metatagextraction import metaTagExtraction
• get_bradfordlaw.py: added import numpy as np
• get_sourcesproduction.py: added from www.services.cocmatrix import cocMatrix

Bug 4

In addition to the failed imports, get_sourceproduction also encountered problems with cocMatrix. These arose from the fact that it directly called the split() function without first converting the values ​​to strings, causing AttributeErrors when interfacing with the ints contained in the "PY" (publication year) column.

Fix 4

• cocmatrix.py added str(x) cast before .split(sep)

Testing

Other functions have been tested with standardized Scopus data, always successfully:

• get_countriesproduction
• get_frequentwords
• get_lotkalaw
• get_averagecitations
• get_bradfordlaw
• get_sourcesproduction