Evaluating journals’ yearly impact with
Simon S. LI1,2 & Fred Y. YE1,2
1School of Information Management, Nanjing University, Nanjing 210023, China
2Jiangsu Key Lab of Data Engineering & Knowledge Service, Nanjing University, Nanjing 210023, China
):25-38, Received: Apr. 20, 2015
Revised: May 28, 2015
Accepted: Jun. 1, 2015
This work is jointly supported by the National Natural Science Foundation of China (Grant No.: 71173187)
and the China Major Key Project of National Social Science Foundation (Grant No.: 12&ZD221). The
authors would like to thank the anonymous reviewers for their constructive comments and editors for
S. S. Li (firstname.lastname@example.org) collected the data, analyzed the data, and wrote the first draft.
F. Y. Ye (email@example.com, corresponding author) revised the manuscript and edited the final
version of the paper.
"Purpose: Applying SSCI journals of library and information science (LIS) as the research sample, we explore the feasibility of measuring academic journals' yearly social impact by using altmetric indicators.
Design/methodology/approach: Using a sample of 66 SSCI journals in LIS published in 2013, statistics regarding journal mentions in social media and other online tools were retrieved from Altmetric.com and meanwhile citation data was also collected from JCR and Scopus. Based on the method of principal component analysis, data was analyzed for associations between the altmetric and traditional metrics to demonstrate the effect of altmetric indicators on measuring academic journals' yearly impact.
Findings: The Spearman's rank correlation test results show that altmetric indicators and traditional citation counts were significantly correlated, indicating that altmetrics can be used to measure a journal's yearly social impact.
Research limitations: The time frame of data collected from Altmetric.com may not be consistent with that of JCR and Scopus citation data.
Practical implications: A new method is provided based on altmetrics for evaluating the social impact of academic journals, which can be applied to design new indicators of short-term journal impact.
Originality value: In this paper, we have established a method for evaluating the social impact of academic journals based on altmetric indictors. Altmetrics can be complementary to traditional citation metrics in assessing a journal's impact within a year or even in a shorter period of time."
The emergence of social media tools such as Twitter and Facebook has made it
easier for researchers to engage with the public and disseminate their research more
widely than ever. When an increasing number of scholars are using social media
tools in their professional communication in the Web2.0 era, it remains a challenge
to measure the impact of research in social media. Alternative metrics (altmetrics)
provides a new opportunity for the study of informetrics in the Web2.0 era.
Torres-Salinas et al.
defined altmetrics as the “creation and study of new
indicators for the analysis of academic activity based on Web2.0” and altmetric
metrics such as mentions in blogs may be a valid measure of the use and the impact
of scientific publications. Qiu & Yu[3,4]
pointed out that altmetric is an aggregator
of online attention to scholarly papers, which provides an intuitive understanding
of both social and academic influence of research findings. Altmetrics has been
recently applied in scientific contexts. A large number of publishers such as PLoS
ONE have adopted altmetric measures to assess the online impact of scientific
literature. In June, 2014, the American National Information Standards Organization
developed a draft altmetrics standard
The study of altmetrics, however, is still in its initial stage. On the one hand,
researchers are investigating potential use of altmetrics as a source of impact
assessment. You et al.
used the method of principal component analysis to build
a model to measure the impact of research articles with the data from the Mendeley
platform. Their study suggested that the research influence measured by the social
media tools is related to traditional citation-based impact. Hammarfelt
the altmetric coverage and influence of journals and books in humanities published
by the Swedish universities in 2012. He pointed out that altmetrics could evolve
into a valuable tool for evaluating research in humanities. Zahedi et al.
randomly 20,000 publications from the Web of Science, analyzed their presence
and distribution in the social media, and found a moderate Spearman correlation
= 0.49) between Mendeley readership counts and citation indicators. Alhoori
studied altmetrics based on the country-level impact and concluded that
altmetrics can be used to evaluate the impact of research activities of all countries.
They found significant correlation between country-level altmetrics and several
traditional bibliometric measures.
On the other hand, altmetric indicators’ effect on impact assessment has been
found to be rather limited. Costas et al.
reported positive but relatively weak
correlation between altmetric indicators and citations. Therefore, they considered
altmetric indicators only as a supplementary tool of citation analysis. Ortega
studied the correlation between altmetric indicators and traditional citation counts at the author level and they also found a very weak correlation between altmetrics
and citations. Due to their limitations, altmetric indicators have not yet been as
widely accepted as traditional citation metrics
. As Ortega
has pointed out,
altmetrics cannot be a substitute of citation counts.
The existing research into altmetrics focuses more on the evaluation
of social impact at the article level rather than at the journal level
. To the
scientometrics community, however, it is important to assess academic journals’
impact with bibliometric measures. One limitation of using traditional bibliometric
indicators to measure journal impact is that accumulation of citations takes time
and therefore it is challenging to assess a journal’s impact in an immediate way.
Compared with traditional bibliometrics, al tmetrics has the potential to provide
information about a journal’s social impact in a timely manner since mentions of
articles can be tracked in social media and online tools even before the articles
are formally published. Whether the altmetrics can be used to measure a journal’s
social impact in a shorter period of time has been an issue worthy of great attention.
This paper attempts to explore the feasibility of evaluating academic journals’
yearly impact by using data collected from Altmetric.com. To this end, the method
of principal component analysis is used to explore the relationship between online
readership and traditional citation counts.
2 Data collection and selection of altmetric indicators
2.1 Data collection
Altmetric.com① is committed to developing altmetric tools and providing related
services. It tracks the attention that scholarly articles and datasets receive online
by pulling in data from 3 main sources in real-time: 1) social media like Twitter,
Google+, Pinterest and blogs, 2) traditional media such as news outputs and
government documents, and 3) online reference managers like Mendeley and
CiteULike. It indicates the amount of attention each research output receives with
Altmetric score, which is calculated by assigning weights to each source tracked by
Using a sample of 66 journals in library and information science (LIS) indexed
in the Social Sciences Citation Index (SSCI), we collected data from 13 sources
for mentions of these journals in 2014 and the annual Altmetric score for the
related articles from Altmetric.com (download time: December 14, 2014). Table 1
shows partial data of the sample. Specific steps of data collection are summarized
• Step 1: Co llect the ISSNs of SS CI journals in the field of LIS from Journal
Citation Reports (JCR)②;
• Step 2: Search for journals based on the ISSNs on Altmetric.com③, and collect
data within one-year period. Raw data mainly includes the title of article,
digital object identifier (DOI), source journals, Altmetric score and the number
of mentions in each source. After standardizing all journal titles, we import the
data into MySQL database for statistical analysis. For each journal, we calculate
the total article mentions in the 13 data sources traced by Altmetric.com, the
total Altmetric score, and the total number of articles published in 2014;
• Step 3: Find the latest JCR impact factor, JCR total cites and JCR immediacy
index④ of the 66 SSCI journals;
• Step 4: Download citation data published by Scopus, such as source normalized
impact per paper (SNIP)⑤
, SCImago journal rank (SJR)⑥
, Scopus impact
Partial data about SSCI journals in LIS field
Note: Full journal titles are listed in Appendix I. Article number: The total number of journal articles mentioned by Altmetric.com; Total score: Total Altmetric score within one-year period; JCR TC: JCR total cites; JCR IF: JCR impact factor; JCR II: JCR immediacy index; SNIP: Source normalized impact per paper; SJR: SCImago journal rank; Scopus IF: Scopus impact factor.
③http://www.altmetric.com/login.php. Users need to apply for access to Altmetric.com.
2.2 Selection of altmetric indicators
The data sources from which we collected data were quite different in medium
nature and the number of users. We found mentions in social media and other online tools were not evenly distributed (Table 2). In addition, the altmetric indicators are
related with one another as a person may forward a microblog post, and meanwhile
he or she may include that article in the online reference managers. Considering
the uneven distribution of data, we analyzed the correlation among 13 altmetric
indicators using Spearman’s rank correlation test. Table 3 displays the correlation
results among the altmetric indicators.
Distribution of mentions of journals in social media and online tools
3 Journal impact evaluation with PCA
3.1 Principal component analysis
This paper uses principal component analysis (PCA) to evaluate the influence
of academic journals, analyzing the correlation between comprehensive principal
component scores and traditional citation indicators to avoid subjective evaluation
and eliminate the impact of correlation between data.
Principal component analysis, as a multivariate statistical method, deals with
high dimensional data by reducing it to a smaller dimension. It aims at finding
a few linear combinations of variables, called principal components, to explain as
much of the variance in the data as possible
. These new variables, the identified
principal components, are low dimensional, unrelated, and cannot be directly
measured. One of the advantages of using PCA for evaluating journals is that the
weight of indicators is assigned more objectively[15,16,17]
. Song et al.
dimensions of scientific evaluation with PCA using article-level metric data sample
of 1,390 articles on physics, chemistry, sociology, and immunology from PLoS
PCA consists of the following steps[15,19]
Spearman’s rank correlation coefficient among altmetric indicators
Note: Reddit: Reddit threads; Google+: Google+ authors; F1000: F1000 reviews; Pinterest: Pinterest posts; News: News outlets; Facebook: Facebook walls; Weibo: Sina Weibo users; Peer: Peer review sites; Policy: Policy documents; Mendeley: Mendeley readers; CiteULike: CiteULike readers. ** Statistically significant at the 99% confidence level (double sided). *Statistically significant at the 95% confidence level (double sided).
• Step 1: Standardization of raw data. The raw data (i.e. matrix X) consists of
sample and p
dimensional vector, and its element xij
transformation to obtain the standardization matrix Z = [zij
, which is
calculated with Eq. (1).
In Eq. (1),
indicate the mean and standard deviation of j
data in matrix X, respectively. They are calculated using Eqs. (2) and (3).
• Step 2: Calculation of the covariance matrix S of the normalized matrix Z
with Eq. (4).
In Eq. (4), the covariance sij
is computed with Eq. (5).
In Eq. (5),
indicates the mean of j
column data in matrix Z and is calculated
by using Eq. (6).
• Step 3: Computation of eigenvalue λi of matrix S and corresponding unit
orthogonal eigenvectors ai. For the characteristic equation |S – λIp| = 0
of matrix S, we find p characteristic roots. The eigenvalues of matrix S
are calculated and represented as λ1 ~ λp. The first m larger feature values
λ1 ≥ λ2 ≥ … ≥ λm are the variances of m principal components, and λi
corresponds to the unit eigenvector ai, which is also the factor between principal
component Fi and the original p dimensional vector Xj.
• Step 4: Determination of the number of principal com ponents. The variance
contribution rate of each principal component is calculated with Eq. (7):
The accumulated variance contribution rate of m
principal components is
computed with Eq. (8):
Choose the smallest m value so that G(m) is equal or greater than 80%, which
means sufficient information reflects the original variables.
• Step 5: Computation of component matrix. Principal component load l
reflects the correlation between the principal component Fi
and the original
, and it is computed with Eq. (9).
In Eq. (9), ai
corresponding unit orthogonal eigenvectors.
• Step 6: Calculation of the principal component score with Eq. (10).
In Eq. (10), aki
= 1,2,…, p
) indicates the k
dimensional elements of
= 1,2,…, p
) indicates k
column vectors of the normalized
• Step 7: Calculation of comprehensive evaluation (F
) of m
with Eq. (11).
3.2 Suitability of the data for PCA
Kaiser-Mayer-Olkin(KMO)-Bartlett was conducted to make sure the data was
suitable for principal components analysis. KMO value ranges between 0 and 1 and
the value close to 1 means there are more common factors for a group of variables,
meeting the requirement for PCA analysis. The KMO value for our data sample
was 0.768, which was close to 1. PCA analysis requires the Bartlett test of sphericity
is statistically significant. The probability associated with the Bartlett test for
our data sample was less than 0.05. KMO-Bartlett test results indicate we satisfied
the basic requirement for PCA analysis.
3.3 Data processing
Data processing was conducted by SPSS statistics software. First, the data in
Table 1 was imported into the SPSS 19.0 and standardized. For example: Reddit
threads index corresponds to the standardized index Zscore: Reddit threads.
Table 4 shows each journal’s Zscore for each indicator.
Partial data of journals’ Zcore for each data source
Note: Reddit: Reddit threads; Google+: Google+ authors; F1000: F1000 reviews; Pinterest: Pinterest posts; News: News outlets; Facebook: Facebook walls; Weibo: Sina Weibo users; Peer: Peer review sites; Policy: Policy documents; Mendeley: Mendeley readers; CiteULike: CiteULike readers.
Second, we computed eigenvalues, feature vector and total variance explained
and the result was displayed in Table 5. It is observed that the first 3 principal components with initial eigenvalues greater than 1 explain roughly 80% of the total
variability in the standardized data. As a result, selection of the first 3 principal
components is a reasonable way to reduce data dimensions.
Total variance explained
Third, we calculated comprehensive principal component scores. Component
matrix refers to factor loadings matrix of principal components with each factor
loading value indicating the relationship between each variable and principal
components. Table 6 shows the component matrix.
In Table 6, the first principal component includes the variables of Reddit threads,
bloggers, tweeters, Google+ authors, F1000 reviews, Pinterest posts, news outlets,
Facebook walls, Mendeley readers and CiteULike readers. We can substitute one
component variable for this combination of variables in further analyses. The
variables Sina Weibo users and peer review sites are included in the second principal
component. It can substitute this combination of variables in further analyses. The
third principal component includes policy documents, indicating that it reflects the
basic information of the indicator. In short, the extracted 3 principal components
can basically reflect the information of all the indicators, and can be used as new
variables to replace the original 13 variables.
Finally, using each principal component score (F1, F2, F3) computed with
Eq. (10) and their weights, we calculated the comprehensive principal component
score F of each journal with Eq. (11). The weight was calculated as variance
contribution rate against accumulated variance contribution rate. The final results
are presented in Table 7.
3.4 Correlation between altmetric indicators and citation counts
Spearman’s rank correlation test was conducted to analyze the correlation between
the journals’ yearly comprehensive principal component score F
and traditional journal evaluation indicators, and the results were summarized in Table 8. It is noted
that F has a significant correlation with all the other indicators. This shows that the
comprehensive principal component score can be used for evaluation of journals’
yearly social impact, and altmetrics can be considered as a supplement to traditional
Partial data of factor score, component score, and comprehensive principal component score
Note: FAC: Factor score; F
1: The score of the first principal component; F
2: The score of the second principal component; F
3: The score of the third principal component; F
: The comprehensive principal component score.
Spearman’s rank correlation coefficient between the evaluation indicators
Note: JCR TC: JCR total cites; JCR IF: JCR impact factor; JCR II: JCR immediacy index; SNIP: Source normalized impact per paper; SJR: SCImago journal rank; Scopus IF: Scopus impact factor. ** Statistically significant at the 99% confidence level (double sided).
Using the data of Altmetric.com, we extracted related citation data of 66 SSCI
journals from social media and other online media within one year’s time. We
calculated comprehensive principal component scores to evaluate journals’ social
impact based on the method of principal component analysis (PCA). As an objective
method to identify patterns in data, PCA is suitable for the study of altmetrics.
Spearman’s rank correlation analysis shows that altmetrics correlate significantly
with traditional measures and they can be used to supplement traditional bibliometric
indicators in reflecting the multidimensional nature of scholarly impact in an
immediate way. Hopefully, our findings will encourage more research into altmetrics
as complements to traditional citation measures in assessing academic journals’
yearly social impact.
While traditional bibliometrics could not address the issue of evaluation of shortterm
journal impact, altmetrics provides an alternative. But it should be noted that
related indicators need to be further studied and improved.
Our study has several limitations. First, tools such as Altmetric.com can track a
limited number of social media and part of data is not provided in a standardized
format, which will affect the accuracy of statistics. In addition, the Altmetric score
represents a weighted count of the amount of attention to a research output, but
whether the weight is assigned in an objective way needs to be further studied.
This study tried to assess academic journals’ yearly social impact, but it is still a
challenging task to evaluate journals’ quarterly or monthly impact, or even weekly
and daily impact due to the difficulties of data collection of altmetric indicators at
present. But theoretically it is possible to measure academic journals’ shorter-term
social impact when comprehensive data in standardized formats can be available.
This paper is confined to the discussion of evaluation of social impact of SSCI
journals in the field of library and information science, and this method needs to be
applied to evaluate social impact of journals in other subject areas in the future to
verify its effectiveness.
Abbreviations of journal titles
Rousseau, R., & Ye, F.Y. A multi-metric approach for research evaluation. Chinese Science Bulletin, 2013, 58(26): 3288-3290. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1007/s11434-013-5939-3
Torres-Salinas, D., Cabezas-Clavijo, Á., & Jiménez-Contreras, E. Altmetrics: New indicators for scientific communication in Web2.0. Comunicar, 2013, 21(41): 53-60. Retrieved on June2, 2015, from http://www.revistacomunicar.com/index.php?contenido=detalles&numero=41&articulo=41-2013-05
Qiu, J.P., & Yu, H.Q. Some basic problems in advancing the development of Altmetrics. The Journal of Library Science in China (in Chinese), 2015, 41(1): 4-15. Retrieved on June 2,2015, from http://lib.cqvip.com/QK/93480A/201501/663336037.html
Qiu, J.P., & Yu, H.Q. The putting forward process and research progress of altmetrics. Library and Information Service (in Chinese), 2013, 57(19): 5-12. Retrieved on June 2, 2015, from http://lib.cqvip.com/qk/92987X/201319/47666303.html
Mazov, N.A., & Gureev, V.N. Alternative approaches to assessing scientific results. Herald of the Russian Academy of Sciences, 2015, 85(1): 26-32. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1134%2FS1019331615010116
You, Q.B., Wei, B., & Tang, S.H. Evaluation model construction to evaluate article's influence based on altmetrics. Library and Information Service (in Chinese), 2014, 58(22): 5-11. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz201422001.aspx
Ham marfelt, B. Using altmetrics for assessing research impact in the humanities. Scientometrics,2014, 101(2): 1419-1430. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1007/s11192-014-1261-3
Zah edi, Z., Costas, R., & Wouters, P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics,2014, 101(2): 1491-1513. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1007/s11192-014-1264-0
Alh oori, H., Furuta, R., & Tabet, M., et al. Altmetrics for country-level research assessment. In Tuamsuk, K., Jatowt, A., & Rasmussen, E. (Eds.), The Emergence of Digital Libraries- Research and Practices. Berlin: Springer, 2014: 59-64. Retrieved on June 2, 2015, from http://link.springer.com/chapter/10.1007/978-3-319-12823-8_7
Cos tas, R., Zahedi, Z., & Wouters, P. Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology. Retrieved on June 2, 2015, from http://onlinelibrary.wiley.com/doi/10.1002/asi.23309/abstract;jsessionid=B4C8A712F1583CDF02D1C52E4847E847.f02t04
Ort ega, J.L. Relationship between altmetric and bibliometric indicators across academic social sites: The case of CSIC's members. Journal of Informetrics, 2015, 9(1): 39-49. Retrieved on June 2, 2015, from http://www.sciencedirect.com/science/article/pii/S1751157714000984
Bri gham, T.J. An introduction to altmetrics. Medical Reference Services Quarterly, 2014,33(4): 438-447. Retrieved on June 2, 2015, from http://www.tandfonline.com/doi/abs/10.1080/02763869.2014.957093?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed
Liu, C.L. Scientometrics in the Web 2.0 environment: Altmetrics. Library and Information Service (in Chinese), 2012, 56(14): 52-56. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz201214012.aspx
Zha, X.J. Information analysis (in Chinese). Wuhan: Wuhan University Press, 2011: 167.
Wan g, Y.B. New way to determine core journal—Major component analysis. Journal of the China Society for Scientific and Technical Information (in Chinese), 1998, 17(5): 395-398. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_qbxb199805012.aspx
Guan, J. A comprehensive evaluation of core periodicals collected at libraries—Principal component Analysis. Library and Information Service (in Chinese), 2003, 2: 40-43. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz200302008.aspx
Zhang, H., Zhao, H.X., & Liu, Y.P., et al. Evaluation method on sci-tech journals based principle component analysis. Acta Editologica, 2008, 20(1): 87-90. Retrieved on June 2,2015, from http://d.wanfangdata.com.cn/Periodical_bjxb200801038.aspx
Son g, L.P., Wang, J.F., & Liu, R. Study on science evaluation dimensions based on the principal component: A case study on PLoS ONE. Library and Information Service (in Chinese), 2014, 58(17): 119-124. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz201417018.aspx
Wan g, D.H. Multivariate statistical analysis and SPSS applications (in Chinese). Shanghai: East China University of Science and Technology Press, 2010: 187-203.