Deep Learning Examines Academic Publications
In the realm of organizational theory, researchers are leveraging advanced Natural Language Processing (NLP) tools to streamline the process of reading and analyzing vast amounts of academic literature. A recent approach, dubbed as computational literature review, employs deep learning models to tackle this challenge more efficiently.
One such model gaining traction is the Longformer, a variant of the Transformer model from the HuggingFace Transformers library. In a groundbreaking study, researchers trained Longformer on Microsoft Azure's machine learning servers, using a labeled dataset of around 700 articles in organizational theory.
The team compared Longformer's performance with two other Transformer models: BERT and SciBERT. By using Longformer, the researchers were able to process longer sequences than BERT and SciBERT, which is crucial given the length of academic articles. This capability allowed Longformer to minimize computational burdens by performing only relevant comparisons.
To address the class imbalance in the hand-labeled data, the researchers oversampled the training data. They also considered using stratified sampling during data preprocessing. K-fold cross-validation was employed to provide a more generalizable measure of model performance.
The models were trained to determine which of three perspectives (demographic, relational, or cultural) are used in an academic text. The researchers found that the demographic perspective was the most common in articles in organizational sociology, while the relational perspective was the most common in management and organizational behavior.
The best model, using Longformer, had relatively low values for all hyperparameters except dropout rate. In the researchers' case, Longformer proved to be more accurate, yielding a 2% or so improvement in accuracy compared to BERT.
Looking ahead, the researchers are considering continuing to train Longformers with larger token lengths and using them to analyze citation patterns and the changing lexica representing perspectives in organizational theory. They are also considering removing the abstract or the metadata information during preprocessing.
For those interested in exploring this topic further, recent papers or reviews using keywords such as "Longformer computational literature review sociology," "Longformer vs BERT management studies," or "Longformer performance literature mining" can provide valuable insights. Exploring repositories like arXiv for studies up to 2025, including the linked papers on Longformer architecture and domain adaptation, can also offer a wealth of information.
In conclusion, Longformer is a promising tool for computational literature reviews in organizational theory. Its ability to process long sequences efficiently sets it apart from traditional Transformer models like BERT and SciBERT. By understanding and comparing the performance of these models, researchers can develop more effective methods for reading and tracking organizational theory in the disciplines of sociology and management studies.
Data-and-cloud-computing played a significant role in the study, as the researchers utilized Microsoft Azure's machine learning servers to train their model. The technology of longformer, a variant of the Transformer model from the HuggingFace Transformers library, was enhanced by artificial-intelligence to process longer sequences of academic articles more efficiently, thereby minimizing computational burdens.
