Abstract: Although different studies have shown that ChatGPT 4o, ChatGPT 4o-mini and Gemini 1.5 Flash have a moderate ability to estimate the quality of academic research, there are many technical challenges with researching the value of Large Language Models (LLMs) for this task. These challenges include the amount of resources (money or computing power and specialist equipment) to conduct large scale experiments, the time taken to run some tests, and the likely continued evolution of LLMs and the emergence of new ones. These all reduce the ability of researchers to compare models and strategies, compared to pre-LLM artificial intelligence studies. This talk will discuss these issues and ask questions about the expectations that are reasonable for this type of research.

Mike Thelwall is a Professor of Data Science in the Information School at the University of Sheffield in the UK. He primarily investigates quantitative methods to support research evaluation, including artificial intelligence, citation analysis and altmetrics. He is currently investigating the extent to which Large Language Models can support research evaluation tasks. His books include: Quantitative Methods in Research Evaluation Citation Indicators, Altmetrics, and Artificial Intelligence. He is an associate editor of the Journal of the Association for Information Science and Technology and sits on five other editorial boards, including Scientometrics and Quantitative Science Studies.

Sessions

The workshop will be held on June 23, 2025, and specific activities include keynote, paper presentations and a panel session.

June 23, 10:00am -13:00pm， Location: ISTC
Time	Content	Speaker	Session Chair
10:00am-10:10am	Openning Remark of AII-EEKE2025	General Chairs Yi Zhang, Chengzhi Zhang, and Philipp Mayr and the Organisation Committees Wei Lu, Ying Ding, Arho Suominen, and Haihua Chen.
10:10am-11:00am	Keynote : Large Language Models for Research Quality Evaluation: Technical Challenges	Mike Thelwall	Chair: Yi Zhang
Coffee Break
11:10am -13:00pm	Session 1: Technology Evolution and Research Evaluation
11:10am -11:35am	An innovative topic modeling method integrating large language models for topic recognition and evolutionary analysis	Jiamin Wang, Tao Zhang, Xiao Zhou and Jiming Hu	Chair: Philipp Mayr
11:35am -12:00am	Interdisciplinarity and scientific output quality in AI‐for‐Science: Implications from a Large-Scale Literature Analysis of AI4S	Weiyu Duan, Ying Guo and Jiaqi Wei
Coffee Break
12:10pm -12:35pm	Research on Technology Evolution Analysis Method from the “Problem-Solution-Effect” (P-S-E) Three-Dimensional Perspective	Zhanyi Zhao, Ruiyu Yang, Jiatang Luo, Lili Zheng, Lucheng Lv and Yajuan Zhao
12:35pm -13:00pm	Measuring Technology Diffusion Dynamics Using Patent Full-Text Data and Machine Learning	Alex Jie Yang, Star Zhao, Yi Bu and Sanhong Deng
Coffee Break
June 23, 14:00pm -17:00pm， Location: ISTC
14:00pm-15:35pm	Session 2: Application of Large Language Models
14:00pm-14:25pm	Chatting with Papers: A Hybrid Approach Using LLMs and Knowledge Graphs	Vyacheslav Tykhonov, Han Yang, Philipp Mayr, Jetze Touber and Andrea Scharnhorst	Chair: Mengjia Wu
14:25pm-14:40pm	LLM-based Approaches to Canonical Reference Extraction in Academic Texts: Initial Results	Luisa Ripoll-Alberola
Coffee Break
14:50pm-15:05pm	PromptSight: Forecasting Emerging Technologies via Iterative Self-Prompting in Large Language Models	Alexander Sternfeld, Andrei Kucharavy, Dimitri Percia David, Julian Jang-Jaccard and Alain Mermou
15:05pm-15:20pm	Using large language models in literature screening for bibliometrics and reviews: a case study on health communication gap	Ni Cheng, Heng Dong, Xuan Han, Beibei Tan and Shuzhen Zhu
Coffee Break
15:30pm-16:30pm	Session 3: Panel Discussion	Yi Zhang, Chengzhi Zhang, Philipp Mayr, Robin Haunschild, and Lin Zhang
16:30pm-16:45pm	Greeting Notes of AII-EEKE2025	General Chairs Yi Zhang, Chengzhi Zhang, and Philipp Mayr and the Organisation Committees Wei Lu, Ying Ding, Arho Suominen, and Haihua Chen.
16:45pm	End of workshop

Submission Information

All submissions must be written in English, following the CEUR-ART style and should be submitted as PDF files to EasyChair.

• Regular papers:  10 pages for full papers and 4 pages for short papers exclusive of unlimited pages for references.
• Poster & demonstration: We welcome submissions detailing original, early findings, works in progress a17:00pmnd industrial applications of knowledge entities extraction ande evaluation for a special poster session, possibly with a 2-minute presentation in the main session. Some research track papers will also be invited to the poster track instead, although there will be no difference in the final proceedings between poster and research track submissions. These papers should follow the same format as the research track papers but can be shorter (2 pages for poster and demo papers).

Submit a paper

All submissions will be reviewed by at least two independent reviewers. Please be aware of the fact that at least one author per paper needs to register for the workshop and attend the workshop to present the work. In case of no-show the paper (even if accepted) will be deleted from the proceedings and from the program.

Workshop proceedings will be deposited online in the CEUR workshop proceedings publication service. This way the proceedings will be permanently available and citable (digital persistent identifiers and long term preservation).

Special Issue

Accepted submissions will be invited to submit to our special issue in Scientometrics. More detailed information about this special issue can be visited at: https://eeke-workshop.github.io/2025/si-aii-eeke.html.

Important Dates

All dates are Anywhere on Earth (AoE).

Deadline for submission: April 30, 2025
Notification of acceptance: May 15, 2025
Camera-ready:May 30, 2025
Workshop: June 24, 2025

General Chairs

Yi Zhang (yi.zhang@uts.edu.au) is an Associate Professor at the Australian Artificial Intelligence Institute, University of Technology Sydney. He holds dual Ph.D. degrees in Management Science & Engineering and in Software Engineering. His research interests align with intelligent bibliometrics - incorporating artificial intelligence and data science techniques with bibliometric indicators for broad science, technology & innovation studies. He is the recipient of the 2019 Discovery Early Career Researcher Award granted by the Australian Research Council. He serves as the Executive Editor for Technological Forecasting & Social Change, Associate Editor for the IEEE Transactions on Engineering Management and Scientometrics, and the Specialty Chief Editor for Frontiers in Research Metrics and Analytics. (https://www.uts.edu.au/staff/yi.zhang)

Chengzhi Zhang (zhangcz@njust.edu.cn) is a professor of Department of Information Management, Nanjing University of Science and Technology, China. He received his PhD degree of Information Science from Nanjing University, China. He has published more than 100 publications, including JASIST, IPM, LISR, TFSC, Aslib JIM, JOI, OIR, SCIM, ACL, NAACL, etc. His current research interests include scientific text mining, knowledge entity extraction and evaluation, social media mining. He serves as Editorial Board Member and Managing Guest Editor for 10 international journals (Patterns, IPM, SCIM, OIR, Aslib JIM, TEL, JDIS, DIM, DI, etc.) and PC members of several international conferences in fields of natural language process and scientometrics. (https://chengzhizhang.github.io/)

Philipp Mayr ( philipp.mayr@gesis.org) is a team leader at the GESIS - Leibniz-Institute for the Social Sciences department Knowledge Technologies for the Social Sciences (WTS). He received his PhD in applied informetrics and information retrieval from the Berlin School of Library and Information Science at Humboldt University Berlin. He has published in top conferences and prestigious journals in the areas informetrics, information retrieval and digital libraries. His research group focuses on methods and techniques for interactive information retrieval and data set search. He was the main organizer of the BIR workshops at ECIR 2014-2021 and the BIRNDL workshops at JCDL 2016 and SIGIR 2017-2019. (https://philippmayr.github.io/)

Organization Committee

Wei Lu (weilu@whu.edu.cn) is a professor of School of Information Management and director of Information Retrieval and Knowledge Mining Center, Wuhan University. He received his PhD degree of Information Science from Wuhan University, China. His current research interests include information retrieval, text mining, QA etc. He has papers published on SIGIR, Information Sciences, JASIT, Journal of Information Science etc. He serves as diverse roles (e.g., Associate Editor, Editorial Board Member, and Managing Guest Editor) for several journals. (http://39.103.203.133/member/4)

Ying Ding (ying.ding@austin.utexas.edu)is Bill & Lewis Suit professor at School of Information, University of Texas at Austin. She has been involved in various NIH, NSF and European-Union funded projects. She has published 240+ papers in journals, conferences, and workshops, and served as the program committee member for 200+ international conferences. She is the co-editor of book series called Semantic Web Synthesis by Morgan & Claypool publisher, the co-editor-in-chief for Data Intelligence published by MIT Press and Chinese Academy of Sciences, and serves as the editorial board member for several top journals in Information Science and Semantic Web. Her current research interests include data-driven science of science, AI in healthcare, Semantic Web, knowledge graph, data science, scholarly communication, and the application of Web technologies. (https://yingding.ischool.utexas.edu/)

Arho Suominen (Arho.Suominen@vtt.fi) is principal scientist at the VTT Technical Research Centre of Finland and Industrial professor at Tampere University (Finland). Dr. Suominen’s research focuses on qualitative and quantitative assessment of innovation systems with a special focus on quantitative methods. His prior research has been funded by the European Commission via H2020, Academy of Finland, Finnish Funding Agency for Technology, Turku University Foundation and the Fulbright Center Finland. Through the Fulbright program, he worked as Visiting Scholar at the School of Public Policy at the Georgia Institute of Technology. Dr. Suominen has a Doctor of Science (Tech.) degree from the University of Turku and holds an Officers basic degree from the National Defence University of Finland. (https://cris.vtt.fi/en/persons/arho-suominen)

Haihua Chen (haihua.chen@unt.edu)is an assistant professor in the Departmentof Information Science at the University of North Texas. He has expertise in applied data science, natural language processing, information retrieval, and text mining. He co-authored more than 40 publications in academic venues in both information science and computer science. He is serving as co-editor for The Electronic Library, the guest editor of Information Discovery & Delivery and Frontiers in Big Data special issues, and the reviewer for 14 peer reviewed journals and several international conferences. (https://iia.ci.unt.edu/haihua-chen/)

Programme Committee

• Alireza Abbasi, University of New South Wales (Canberra)
  
• Iana Atanassova, CRIT, Université de Bourgogne Franche-Comté
  
• Marc Bertin, Université Claude Bernard Lyon 1
  
• Katarina Boland, GESIS - Leibniz Institute for the Social Sciences
  
• Yi Bu, Peking University
  
• Guillaume Cabanac, IRIT - Université Paul Sabatier Toulouse 3
  
• Caitlin Cassidy, Search Technology Inc
  
• Chong Chen, Beijing Normal University
  
• Guo Chen, Nanjing University of Science and Technology
  
• Hongshu Chen, Beijing Institute of Technology
  
• Gong Cheng, Nanjing University
  
• Jian Du, Peking University
  
• Edward Fox, Virgina Tech
  
• Ying Guo, China University of Political Science and Law
  
• Arash Hajikhani, VTT Technical Research Centre of Finland
  
• Jiangen He, The University of Tennessee
  
• Zhigang Hu, South China Normal University
  
• Bolin Hua, Peking University
  
• Lu Huang, Beijing Institute of Technology
  
• Ying Huang, Wuhan University
  
• Yong Huang, Wuhan University
  
• Yuya Kajikawa, Tokyo University of Technology
  
• Vivek Kumar Singh, Banaras Hindu University, Varanasi, U.P., India
  
• Chenliang Li, Wuhan Univerisity
  
• Kai Li, University of Tennessee
  
• Chao Lu, Hohai University
  
• Shutian Ma, Tencent
  
• Jin Mao, Wuhan University
  
• Xianling Mao, Beijing Institute of Technology
  
• Chao Min, Nanjing University
  
• Wolfgang Otto, GESIS - Leibniz-Institute for the Social Sciences
  
• Xuelian Pan, Nanjing University
  
• Dwaipayan Roy, GESIS - Leibniz-Institute for the Social Sciences
  
• Philipp Schaer, TH Köln (University of Applied Sciences)
  
• Mayank Singh, Indian Institute of Technology Gandhinagar
  
• Bart Thijs, ECOOM, MSI, K.U.Leuven
  
• Suppawong Tuarob, Mahidol University
  
• Dongbo Wang, Nanjing Agricultural University
  
• Xuefeng Wang Beijing Institute of Technology
  
• Yuzhuo Wang, Anhui University
  
• Dietmar Wolfram, University of Wisconsin-Milwaukee
  
• Jian Wu, Old Dominion University
  
• Mengjia Wu, University of Technology Sydney
  
• Tianxing Wu, Southeast University
  
• Xiaolan Wu, Nanjing Normal University
  
• Yanghua Xiao, Fudan University
  
• Jian Xu, Sun Yat-sen university
  
• Shuo Xu, Beijing University of Technology
  
• Erjia Yan, Drexel University
  
• Heng Zhang, Central China Normal University
  
• Jinzhu Zhang, Nanjing University of Science and Technology
  
• Xiaojuan Zhang, Southwest University
  
• Yingyi Zhang, Soochow University
  
• Zhixiong Zhang, National Science Library, Chinese Academy of Sciences
  
• Qingqing Zhou, Nanjing Normal University
  
• Yongjun Zhu, Yonsei University
  

References

1. Chang, X., Zheng, Q. (2008). Knowledge Element Extraction for Knowledge-Based Learning Resources Organization. In: Leung, H., Li, F., Lau, R., Li, Q. (eds) Advances in Web Based Learning – ICWL 2007. ICWL 2007. Lecture Notes in Computer Science, vol 4823. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-78139-4_10
2. Ying, D., Min, S., Jia, H., Qi, Y., Erjia, Y., Lili, L., Tamy, C. entitymetrics: measuring the impact of entities. Plos One, 2013, 8(8), e71416. https://doi.org/10.1371/journal.pone.0071416
3. Zhang, C., Mayr, P., Lu, W., & Zhang, Y. (2022). JCDL2022 workshop: extraction and evaluation of knowledge entities from scientific documents (EEKE2022). In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries (JCDL '22). Association for Computing Machinery, New York, NY, USA, Article 54, 1–2. https://doi.org/10.1145/3529372.3530917
4. Zhang, Y., Zhang, C., Mayr, P., & Suominen, A. An editorial of “AI + informetrics”: multi-disciplinary interactions in the era of big data. Scientometrics 127, 6503–6507 (2022). https://doi.org/10.1007/s11192-022-04561-w

Links

• Workshop on Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE)
• Workshop on AI + Informetrics (AII)
  

Past Proceedings & Journal Special Issues

Proceedings can be accessed at http://ceur-ws.org/. 

• Proceedings of EEKE-AII 2024
• Proceedings of EEKE-AII 2023
• Proceedings of EEKE 2022
• Proceedings of EEKE 2021
• Proceedings of EEKE2020
• Proceedings of AII 2021

We have organized the related special issues on the topic of extraction and evaluation of knowledge entities in the following journals:

• Aslib Journal of Information Management
• Scientometrics
• Journal of Data and Information Science
• Data and Information Management

We have organized the related special issues on the topic of AI + Informetrics in the following journals:

• Information Processing and Managerment
• Journal of Informetrics
• Scientometrcis