Evaluating journals’ yearly impact with altmetric indicators
Simon S. LI1,2E-mail to the corresponding author & Fred Y. YE1,2Corresponding authorE-mail to the corresponding author
1School of Information Management, Nanjing University, Nanjing 210023, China
2Jiangsu Key Lab of Data Engineering & Knowledge Service, Nanjing University, Nanjing 210023, China
2015, 8(2):25-38, Received: Apr. 20, 2015 Revised: May 28, 2015 Accepted: Jun. 1, 2015
This work is jointly supported by the National Natural Science Foundation of China (Grant No.: 71173187) and the China Major Key Project of National Social Science Foundation (Grant No.: 12&ZD221). The authors would like to thank the anonymous reviewers for their constructive comments and editors for helpful corrections.
S. S. Li (chqlee87@gmail.com) collected the data, analyzed the data, and wrote the first draft. F. Y. Ye (yye@nju.edu.cn, corresponding author) revised the manuscript and edited the final version of the paper.

"Purpose: Applying SSCI journals of library and information science (LIS) as the research sample, we explore the feasibility of measuring academic journals' yearly social impact by using altmetric indicators.

Design/methodology/approach: Using a sample of 66 SSCI journals in LIS published in 2013, statistics regarding journal mentions in social media and other online tools were retrieved from Altmetric.com and meanwhile citation data was also collected from JCR and Scopus. Based on the method of principal component analysis, data was analyzed for associations between the altmetric and traditional metrics to demonstrate the effect of altmetric indicators on measuring academic journals' yearly impact.

Findings: The Spearman's rank correlation test results show that altmetric indicators and traditional citation counts were significantly correlated, indicating that altmetrics can be used to measure a journal's yearly social impact.

Research limitations: The time frame of data collected from Altmetric.com may not be consistent with that of JCR and Scopus citation data.

Practical implications: A new method is provided based on altmetrics for evaluating the social impact of academic journals, which can be applied to design new indicators of short-term journal impact.

Originality value: In this paper, we have established a method for evaluating the social impact of academic journals based on altmetric indictors. Altmetrics can be complementary to traditional citation metrics in assessing a journal's impact within a year or even in a shorter period of time."

1 Introduction
The emergence of social media tools such as Twitter and Facebook has made it easier for researchers to engage with the public and disseminate their research more widely than ever. When an increasing number of scholars are using social media tools in their professional communication in the Web2.0 era, it remains a challenge to measure the impact of research in social media. Alternative metrics (altmetrics) provides a new opportunity for the study of informetrics in the Web2.0 era.
Torres-Salinas et al.[2] defined altmetrics as the “creation and study of new indicators for the analysis of academic activity based on Web2.0” and altmetric metrics such as mentions in blogs may be a valid measure of the use and the impact of scientific publications. Qiu & Yu[3,4] pointed out that altmetric is an aggregator of online attention to scholarly papers, which provides an intuitive understanding of both social and academic influence of research findings. Altmetrics has been recently applied in scientific contexts. A large number of publishers such as PLoS ONE have adopted altmetric measures to assess the online impact of scientific literature. In June, 2014, the American National Information Standards Organization developed a draft altmetrics standard[5].
The study of altmetrics, however, is still in its initial stage. On the one hand, researchers are investigating potential use of altmetrics as a source of impact assessment. You et al.[6] used the method of principal component analysis to build a model to measure the impact of research articles with the data from the Mendeley platform. Their study suggested that the research influence measured by the social media tools is related to traditional citation-based impact. Hammarfelt[7] analyzed the altmetric coverage and influence of journals and books in humanities published by the Swedish universities in 2012. He pointed out that altmetrics could evolve into a valuable tool for evaluating research in humanities. Zahedi et al.[8] collected randomly 20,000 publications from the Web of Science, analyzed their presence and distribution in the social media, and found a moderate Spearman correlation (r = 0.49) between Mendeley readership counts and citation indicators. Alhoori et al.[9] studied altmetrics based on the country-level impact and concluded that altmetrics can be used to evaluate the impact of research activities of all countries. They found significant correlation between country-level altmetrics and several traditional bibliometric measures.
On the other hand, altmetric indicators’ effect on impact assessment has been found to be rather limited. Costas et al.[10] reported positive but relatively weak correlation between altmetric indicators and citations. Therefore, they considered altmetric indicators only as a supplementary tool of citation analysis. Ortega[11] studied the correlation between altmetric indicators and traditional citation counts at the author level and they also found a very weak correlation between altmetrics and citations. Due to their limitations, altmetric indicators have not yet been as widely accepted as traditional citation metrics[12]. As Ortega[11] has pointed out, altmetrics cannot be a substitute of citation counts.
The existing research into altmetrics focuses more on the evaluation of social impact at the article level rather than at the journal level[13]. To the scientometrics community, however, it is important to assess academic journals’ impact with bibliometric measures. One limitation of using traditional bibliometric indicators to measure journal impact is that accumulation of citations takes time and therefore it is challenging to assess a journal’s impact in an immediate way. Compared with traditional bibliometrics, al tmetrics has the potential to provide information about a journal’s social impact in a timely manner since mentions of articles can be tracked in social media and online tools even before the articles are formally published. Whether the altmetrics can be used to measure a journal’s social impact in a shorter period of time has been an issue worthy of great attention. This paper attempts to explore the feasibility of evaluating academic journals’ yearly impact by using data collected from Altmetric.com. To this end, the method of principal component analysis is used to explore the relationship between online readership and traditional citation counts.
2 Data collection and selection of altmetric indicators
2.1 Data collection
Altmetric.com is committed to developing altmetric tools and providing related services. It tracks the attention that scholarly articles and datasets receive online by pulling in data from 3 main sources in real-time: 1) social media like Twitter, Google+, Pinterest and blogs, 2) traditional media such as news outputs and government documents, and 3) online reference managers like Mendeley and CiteULike. It indicates the amount of attention each research output receives with Altmetric score, which is calculated by assigning weights to each source tracked by Altmetric.com.
Using a sample of 66 journals in library and information science (LIS) indexed in the Social Sciences Citation Index (SSCI), we collected data from 13 sources for mentions of these journals in 2014 and the annual Altmetric score for the related articles from Altmetric.com (download time: December 14, 2014). Table 1 shows partial data of the sample. Specific steps of data collection are summarized as follows:
• Step 1: Co llect the ISSNs of SS CI journals in the field of LIS from Journal Citation Reports (JCR);
• Step 2: Search for journals based on the ISSNs on Altmetric.com, and collect data within one-year period. Raw data mainly includes the title of article, digital object identifier (DOI), source journals, Altmetric score and the number of mentions in each source. After standardizing all journal titles, we import the data into MySQL database for statistical analysis. For each journal, we calculate the total article mentions in the 13 data sources traced by Altmetric.com, the total Altmetric score, and the total number of articles published in 2014;
• Step 3: Find the latest JCR impact factor, JCR total cites and JCR immediacy index of the 66 SSCI journals;
• Step 4: Download citation data published by Scopus, such as source normalized impact per paper (SNIP), SCImago journal rank (SJR), Scopus impact factor and h-index.
Table 1    Partial data about SSCI journals in LIS field

Note: Full journal titles are listed in Appendix I. Article number: The total number of journal articles mentioned by Altmetric.com; Total score: Total Altmetric score within one-year period; JCR TC: JCR total cites; JCR IF: JCR impact factor; JCR II: JCR immediacy index; SNIP: Source normalized impact per paper; SJR: SCImago journal rank; Scopus IF: Scopus impact factor.
① http://www.altmetric.com/
② http://www.webofknowledge.com/JCR/JCR
③http://www.altmetric.com/login.php. Users need to apply for access to Altmetric.com.
④ http://www.webofknowledge.com/JCR/JCR
⑤ http://www.journalindicators.com/
⑥ http://www.scimagojr.com/
⑦ http://www.scimagojr.com/
2.2 Selection of altmetric indicators
The data sources from which we collected data were quite different in medium nature and the number of users. We found mentions in social media and other online tools were not evenly distributed (Table 2). In addition, the altmetric indicators are related with one another as a person may forward a microblog post, and meanwhile he or she may include that article in the online reference managers. Considering the uneven distribution of data, we analyzed the correlation among 13 altmetric indicators using Spearman’s rank correlation test. Table 3 displays the correlation results among the altmetric indicators.
Table 2    Distribution of mentions of journals in social media and online tools

3 Journal impact evaluation with PCA
3.1 Principal component analysis
This paper uses principal component analysis (PCA) to evaluate the influence of academic journals, analyzing the correlation between comprehensive principal component scores and traditional citation indicators to avoid subjective evaluation and eliminate the impact of correlation between data.
Principal component analysis, as a multivariate statistical method, deals with high dimensional data by reducing it to a smaller dimension. It aims at finding a few linear combinations of variables, called principal components, to explain as much of the variance in the data as possible[14]. These new variables, the identified principal components, are low dimensional, unrelated, and cannot be directly measured. One of the advantages of using PCA for evaluating journals is that the weight of indicators is assigned more objectively[15,16,17]. Song et al.[18] analyzed major dimensions of scientific evaluation with PCA using article-level metric data sample of 1,390 articles on physics, chemistry, sociology, and immunology from PLoS ONE website.
PCA consists of the following steps[15,19]:
Table 3    Spearman’s rank correlation coefficient among altmetric indicators

Note: Reddit: Reddit threads; Google+: Google+ authors; F1000: F1000 reviews; Pinterest: Pinterest posts; News: News outlets; Facebook: Facebook walls; Weibo: Sina Weibo users; Peer: Peer review sites; Policy: Policy documents; Mendeley: Mendeley readers; CiteULike: CiteULike readers. ** Statistically significant at the 99% confidence level (double sided). *Statistically significant at the 95% confidence level (double sided).
• Step 1: Standardization of raw data. The raw data (i.e. matrix X) consists of n sample and p dimensional vector, and its element xij conducts standard transformation to obtain the standardization matrix Z = [zij]n×p, which is calculated with Eq. (1).

In Eq. (1), and sj indicate the mean and standard deviation of j column data in matrix X, respectively. They are calculated using Eqs. (2) and (3).

• Step 2: Calculation of the covariance matrix S of the normalized matrix Z with Eq. (4).

In Eq. (4), the covariance sij is computed with Eq. (5).

In Eq. (5), indicates the mean of j column data in matrix Z and is calculated by using Eq. (6).

• Step 3: Computation of eigenvalue λi of matrix S and corresponding unit orthogonal eigenvectors ai. For the characteristic equation |S – λIp| = 0 of matrix S, we find p characteristic roots. The eigenvalues of matrix S are calculated and represented as λ1 ~ λp. The first m larger feature values λ1λ2 ≥ … ≥ λm are the variances of m principal components, and λi corresponds to the unit eigenvector ai, which is also the factor between principal component Fi and the original p dimensional vector Xj.
• Step 4: Determination of the number of principal com ponents. The variance contribution rate of each principal component is calculated with Eq. (7):

The accumulated variance contribution rate of m principal components is computed with Eq. (8):

Choose the smallest m value so that G(m) is equal or greater than 80%, which means sufficient information reflects the original variables.
• Step 5: Computation of component matrix. Principal component load l(Fi, Xj) reflects the correlation between the principal component Fi and the original variable Xj, and it is computed with Eq. (9).

In Eq. (9), ai indicates λi corresponding unit orthogonal eigenvectors.
• Step 6: Calculation of the principal component score with Eq. (10).

In Eq. (10), aki (k = 1,2,…, p) indicates the k dimensional elements of vectors ai, Zk (k = 1,2,…, p) indicates k column vectors of the normalized matrix Z.
• Step 7: Calculation of comprehensive evaluation (F ) of m principal components with Eq. (11).

3.2 Suitability of the data for PCA
Kaiser-Mayer-Olkin(KMO)-Bartlett was conducted to make sure the data was suitable for principal components analysis. KMO value ranges between 0 and 1 and the value close to 1 means there are more common factors for a group of variables, meeting the requirement for PCA analysis. The KMO value for our data sample was 0.768, which was close to 1. PCA analysis requires the Bartlett test of sphericity is statistically significant. The probability associated with the Bartlett test for our data sample was less than 0.05. KMO-Bartlett test results indicate we satisfied the basic requirement for PCA analysis.
3.3 Data processing
Data processing was conducted by SPSS statistics software[19]. First, the data in Table 1 was imported into the SPSS 19.0 and standardized. For example: Reddit threads index corresponds to the standardized index Zscore: Reddit threads. Table 4 shows each journal’s Zscore for each indicator.
Table 4    Partial data of journals’ Zcore for each data source

Note: Reddit: Reddit threads; Google+: Google+ authors; F1000: F1000 reviews; Pinterest: Pinterest posts; News: News outlets; Facebook: Facebook walls; Weibo: Sina Weibo users; Peer: Peer review sites; Policy: Policy documents; Mendeley: Mendeley readers; CiteULike: CiteULike readers.
Second, we computed eigenvalues, feature vector and total variance explained and the result was displayed in Table 5. It is observed that the first 3 principal components with initial eigenvalues greater than 1 explain roughly 80% of the total variability in the standardized data. As a result, selection of the first 3 principal components is a reasonable way to reduce data dimensions.
Table 5    Total variance explained

Third, we calculated comprehensive principal component scores. Component matrix refers to factor loadings matrix of principal components with each factor loading value indicating the relationship between each variable and principal components. Table 6 shows the component matrix.
In Table 6, the first principal component includes the variables of Reddit threads, bloggers, tweeters, Google+ authors, F1000 reviews, Pinterest posts, news outlets, Facebook walls, Mendeley readers and CiteULike readers. We can substitute one component variable for this combination of variables in further analyses. The variables Sina Weibo users and peer review sites are included in the second principal component. It can substitute this combination of variables in further analyses. The third principal component includes policy documents, indicating that it reflects the basic information of the indicator. In short, the extracted 3 principal components can basically reflect the information of all the indicators, and can be used as new variables to replace the original 13 variables.
Finally, using each principal component score (F1, F2, F3) computed with Eq. (10) and their weights, we calculated the comprehensive principal component score F of each journal with Eq. (11). The weight was calculated as variance contribution rate against accumulated variance contribution rate. The final results are presented in Table 7.
3.4 Correlation between altmetric indicators and citation counts
Spearman’s rank correlation test was conducted to analyze the correlation between the journals’ yearly comprehensive principal component score F and traditional journal evaluation indicators, and the results were summarized in Table 8. It is noted that F has a significant correlation with all the other indicators. This shows that the comprehensive principal component score can be used for evaluation of journals’ yearly social impact, and altmetrics can be considered as a supplement to traditional bibliometric indicators.
Table 6    Component matrix

Table 7    Partial data of factor score, component score, and comprehensive principal component score

Note: FAC: Factor score; F1: The score of the first principal component; F2: The score of the second principal component; F3: The score of the third principal component; F: The comprehensive principal component score.
Table 8    Spearman’s rank correlation coefficient between the evaluation indicators

Note: JCR TC: JCR total cites; JCR IF: JCR impact factor; JCR II: JCR immediacy index; SNIP: Source normalized impact per paper; SJR: SCImago journal rank; Scopus IF: Scopus impact factor. ** Statistically significant at the 99% confidence level (double sided).
4 Conclusions
Using the data of Altmetric.com, we extracted related citation data of 66 SSCI journals from social media and other online media within one year’s time. We calculated comprehensive principal component scores to evaluate journals’ social impact based on the method of principal component analysis (PCA). As an objective method to identify patterns in data, PCA is suitable for the study of altmetrics. Spearman’s rank correlation analysis shows that altmetrics correlate significantly with traditional measures and they can be used to supplement traditional bibliometric indicators in reflecting the multidimensional nature of scholarly impact in an immediate way. Hopefully, our findings will encourage more research into altmetrics as complements to traditional citation measures in assessing academic journals’ yearly social impact.
While traditional bibliometrics could not address the issue of evaluation of shortterm journal impact, altmetrics provides an alternative. But it should be noted that related indicators need to be further studied and improved.
Our study has several limitations. First, tools such as Altmetric.com can track a limited number of social media and part of data is not provided in a standardized format, which will affect the accuracy of statistics. In addition, the Altmetric score represents a weighted count of the amount of attention to a research output, but whether the weight is assigned in an objective way needs to be further studied.
This study tried to assess academic journals’ yearly social impact, but it is still a challenging task to evaluate journals’ quarterly or monthly impact, or even weekly and daily impact due to the difficulties of data collection of altmetric indicators at present. But theoretically it is possible to measure academic journals’ shorter-term social impact when comprehensive data in standardized formats can be available.
This paper is confined to the discussion of evaluation of social impact of SSCI journals in the field of library and information science, and this method needs to be applied to evaluate social impact of journals in other subject areas in the future to verify its effectiveness.
Appendix I:    Abbreviations of journal titles

1 Rousseau, R., & Ye, F.Y. A multi-metric approach for research evaluation. Chinese Science Bulletin, 2013, 58(26): 3288-3290. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1007/s11434-013-5939-3. DOI:10.1007/s11434-013-5939-3
2 Torres-Salinas, D., Cabezas-Clavijo, Á., & Jiménez-Contreras, E. Altmetrics: New indicators for scientific communication in Web2.0. Comunicar, 2013, 21(41): 53-60. Retrieved on June2, 2015, from http://www.revistacomunicar.com/index.php?contenido=detalles&numero=41&articulo=41-2013-05. DOI:10.3916/C41-2013-05
3 Qiu, J.P., & Yu, H.Q. Some basic problems in advancing the development of Altmetrics. The Journal of Library Science in China (in Chinese), 2015, 41(1): 4-15. Retrieved on June 2,2015, from http://lib.cqvip.com/QK/93480A/201501/663336037.html. DOI:10.13530/j.cnki.jlis.150001
4 Qiu, J.P., & Yu, H.Q. The putting forward process and research progress of altmetrics. Library and Information Service (in Chinese), 2013, 57(19): 5-12. Retrieved on June 2, 2015, from http://lib.cqvip.com/qk/92987X/201319/47666303.html. DOI:10.7536/j.issn.0252-3116.2013.19.001
5 Mazov, N.A., & Gureev, V.N. Alternative approaches to assessing scientific results. Herald of the Russian Academy of Sciences, 2015, 85(1): 26-32. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1134%2FS1019331615010116. DOI:10.1134/S1019331615010116
6 You, Q.B., Wei, B., & Tang, S.H. Evaluation model construction to evaluate article's influence based on altmetrics. Library and Information Service (in Chinese), 2014, 58(22): 5-11. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz201422001.aspx. DOI:10.13266/j.issn.0252-3116.2014.22.001
7 Ham marfelt, B. Using altmetrics for assessing research impact in the humanities. Scientometrics,2014, 101(2): 1419-1430. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1007/s11192-014-1261-3. DOI:10.1007/s11192-014-1261-3
8 Zah edi, Z., Costas, R., & Wouters, P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics,2014, 101(2): 1491-1513. Retrieved on June 2, 2015, from http://link.springer.com/article/10.1007/s11192-014-1264-0. DOI:10.1007/s11192-014-1264-0
9 Alh oori, H., Furuta, R., & Tabet, M., et al. Altmetrics for country-level research assessment. In Tuamsuk, K., Jatowt, A., & Rasmussen, E. (Eds.), The Emergence of Digital Libraries- Research and Practices. Berlin: Springer, 2014: 59-64. Retrieved on June 2, 2015, from http://link.springer.com/chapter/10.1007/978-3-319-12823-8_7. DOI:10.1007/978-3-319-12823-8_7
10 Cos tas, R., Zahedi, Z., & Wouters, P. Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology. Retrieved on June 2, 2015, from http://onlinelibrary.wiley.com/doi/10.1002/asi.23309/abstract;jsessionid=B4C8A712F1583CDF02D1C52E4847E847.f02t04. DOI:10.1002/asi.23309
11 Ort ega, J.L. Relationship between altmetric and bibliometric indicators across academic social sites: The case of CSIC's members. Journal of Informetrics, 2015, 9(1): 39-49. Retrieved on June 2, 2015, from http://www.sciencedirect.com/science/article/pii/S1751157714000984. DOI:10.1016/j.joi.2014.11.004
12 Bri gham, T.J. An introduction to altmetrics. Medical Reference Services Quarterly, 2014,33(4): 438-447. Retrieved on June 2, 2015, from http://www.tandfonline.com/doi/abs/10.1080/02763869.2014.957093?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed. DOI:10.1080/02763869.2014.957093
13 Liu, C.L. Scientometrics in the Web 2.0 environment: Altmetrics. Library and Information Service (in Chinese), 2012, 56(14): 52-56. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz201214012.aspx
14 Zha, X.J. Information analysis (in Chinese). Wuhan: Wuhan University Press, 2011: 167.
15 Wan g, Y.B. New way to determine core journal—Major component analysis. Journal of the China Society for Scientific and Technical Information (in Chinese), 1998, 17(5): 395-398. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_qbxb199805012.aspx. DOI:10.3969/j.issn.1000-0135.1998.05.012
16 Guan, J. A comprehensive evaluation of core periodicals collected at libraries—Principal component Analysis. Library and Information Service (in Chinese), 2003, 2: 40-43. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz200302008.aspx. DOI:10.3969/j.issn.0252-3116.2003.02.008
17 Zhang, H., Zhao, H.X., & Liu, Y.P., et al. Evaluation method on sci-tech journals based principle component analysis. Acta Editologica, 2008, 20(1): 87-90. Retrieved on June 2,2015, from http://d.wanfangdata.com.cn/Periodical_bjxb200801038.aspx
18 Son g, L.P., Wang, J.F., & Liu, R. Study on science evaluation dimensions based on the principal component: A case study on PLoS ONE. Library and Information Service (in Chinese), 2014, 58(17): 119-124. Retrieved on June 2, 2015, from http://d.wanfangdata.com.cn/Periodical_tsqbgz201417018.aspx. DOI:10.13266/j.issn.0252-3116.2014.17.018
19 Wan g, D.H. Multivariate statistical analysis and SPSS applications (in Chinese). Shanghai: East China University of Science and Technology Press, 2010: 187-203.