Koli Calling 2024 Trip Report
Koli Calling 2024 Trip Report
Thursday: Arriving at Koli
I caught the IC1 train at 6:54 AM from Helsinki Central Station with VR. As someone who pays close attention to app usability, I was genuinely impressed with Finland’s train service (VR) app. Not only do they offer free WiFi on long-distance routes and maintain impressive punctuality, but their app also has an incredible UX/UI: I could see if someone was sitting next to me, access a help chat, track my route, and activate tickets (when activated, the conductor doesn’t even need to stop by).
An amusing incident occurred during the journey when a postdoc from the University of Helsinki, who was also heading to the conference, spotted me reading the proceedings. He posted in the Discord: “One laptop with an ACM paper open spotted in the train.” I turned around, and we ended up having coffee while discussing our projects.
Leo and I having a coffee while talking on the way to Koli
I arrived at Koli on a bus that was waiting for us in Joensuu - everything was incredibly well-organized. Koli is a special place, situated in Koli National Park, a location of great historical and cultural significance for the Finnish people. It has inspired artists, writers, and composers like Jean Sibelius, who found creativity in its beauty. Today, Koli remains an emblem of Finnish identity and a reminder of the relationship between humans and their natural environment.
First of all, they welcomed us with lunch with these views.
First day lunch view at Koli
Opening session
Then we dove right in. Juho Leinonen and Andreas Mühling did the introduction with a nice tribute to Brett Becker, a researcher who passed away very recently, a great loss for the CSEd research community.
Then they gave us some interesting data: this conference is organized jointly by the Leibniz Institute for Science and Mathematics Education / Kiel University (Germany), Aalto University, and the University of Eastern Finland (Finland). This year there are 74 participants, of which 61 are in person, from all over the world (Germany, USA, Netherlands, Switzerland…).
Session 1 (Regular): Problem Solving and Understanding – Session Chair Ilkka Jormanainen
After the introduction, we started the sessions which are the conference’s strong point. In the first one about problem solving and understanding, it was Eduardo Carneiro from Utrecht’s turn who told us about how students reason when refactoring code. Using the “think-aloud” method, they investigated students’ actions and reasoning while improving code.
Investigating Student Reasoning in Method-Level Code Refactoring: A Think-Aloud Study
The second session was given by Imke de Jong from Utrecht as well and was about how Computational Thinking (CT) skills are used in solving Bebras tasks.
Use of Computational Thinking Skills when solving Bebras Tasks: a Think-aloud Study
The last one was by Joey Bevilacqua from Lugano about understanding programming expressions through the analysis of university exams.
Assessing the Understanding of Expressions: A Qualitative Study of Notional-Machine-Based Exam Questions
Session 2: Doctoral Consortium Lightning Talks and Posters – Session Chair Nick Falkner
We finished with PhD student presentations about various research projects focused on improving programming education, including mobile escape rooms, interactive visualizations, industrial code quality tools, and applications in secondary school mathematics.
Photo Credits: Ilkka Jormanainen
One could go to each poster asking questions, and the research covered from K-12 education to university level, emphasizing different aspects like computational thinking, mental models, and digital humanities.
Two particularly caught my attention:
The first one was by Naaz from the University of Toronto, who presented a way to improve novice programmers’ understanding through interactive visualizations and multiple representations.
Exploring the Impact of Multiple Representations in Introductory Programming: A Pilot Study
The second one also blew my mind: it’s about integrating static analysis with large language models (LLMs). Their research shows that combining LLMs with static analysis allows significantly improving the quality of suggestions provided to students.
Bringing Industry-Grade Code Quality and Practices into Software Engineering Education (Doctoral Consortium)
The evening ended with a wine tasting session led by Nick Falkner.
Wine tasting session with Nick Falkner
And I had a great conversation about life with Max Barth from ETH.
Talking about different topics over a glass of wine
Did we have a Northern Light? Kind of.
Photo Credits: Ilkka Jormanainen
Day Two: Friday
Keynote and Discussion – Session Chair Juho Leinonen
So let me start from the beginning of the day: Lauri Malmi, from Aalto University, did a very interesting review of Koli Calling’s 24-year history. In his presentation, he covered the conference’s origins, its evolution, the meaning of its name, as well as data about participants, publications, and other aspects.
Lauri Malmi: 24 years of Koli Calling
That journey served as a basis for making several thought-provoking reflections about the development of the computing education area over time, addressing the topics that have been subject to research: K-12, tools for university students, university courses, curriculum, stakeholders, research, and other topics. It was an opportunity to reflect on how the field has advanced and where it might be heading.
Session 3 (Regular): Student Conceptions – Session Chair Hieke Keuning
The first block of sessions of the day was about student conceptions and had four papers.
The first was by Marco Hartmann, from ETH, who talked about the prevalence of misconceptions in programming among primary education students with xLogo and Scratch, touching on topics like sequentiality, loops, conditional structures, and the concept of ‘superbug’ (attributing interpretative capability to the computer).
Prevalence of Programming Misconceptions in Primary School Students
The second talk was by Marko Schmellenkamp, from Ruhr-Universität Bochum, who addressed the types of errors in context-free grammars among secondary school students (10-12 years old).
Exploring Error Types in Formal Languages Among Students of Upper Secondary Education
The third talk was given by Tina Vrieler from Uppsala and was about applying the CSC framework to understand factors that facilitate children’s decisions to participate in code clubs: teachers, family members, friends, video games, caregivers.
Applying the Computer Science Capital Framework to Understand Factors Facilitating Children’s Decision to Participate in Code Clubs
The last session of the block explored, through Bostjan Bubnic from the University of Maribor, whether metacognition (behavior) can predict success in solving object-oriented programming problems.
Can metacognition predict your success in solving problems? An exploratory case study in programming
Session 4 (Panel): Generative AI – Session Chair Otto Seppälä
The second block of sessions of the day was a panel discussion on Generative AI led by Otto Seppälä from Aalto.
Before my session, Isaac, a Costa Rican professor at Utrecht who specializes in computational thinking and GenAI, and Hieke from Utrecht talked about students’ perceptions (concerns) and use of generative AI tools (strategies, main uses) for programming in different computer science courses.
Then two researchers from the University of Helsinki talked about the unrestricted use of LLMs in a software projects course and student perceptions about learning and its impact on course performance.
Finally, it was my turn, where I shared a use case with 15 CS1 students who completed three programming tasks with the possibility of generative AI assistance and the learnings from this coexistence with AI.
Delivering my first conference presentation and panel
This was special: I gave my first talk at a conference of this type and, additionally, I participated in my first panel, where we talked about the impact of GenAI on CSEd. I’ve uploaded the video of my presentation to YouTube, in case you’re curious and want to take a look: Watch here
The talk seems to have been quite well received based on the feedback I got at the end.
Lauri Malmi gave us a great moment at the piano while we had our afternoon snack.
Learning Analytics
The last block of sessions was about Learning Analytics.
The first one was with Fabian, super nice German guy I’ve talked quite a bit with during the conference, about a matrix factorization topic.
Estimating and Differentiating Programming Skills from Unstructured Coding Exercises via Matrix Factorization
The second was given by my train colleague Leo Leppänen about flow analysis in student admission itineraries. Now I’m almost an expert in the Finnish educational system and its admissions hehe.
Flowthrough Analysis on Student Intake Paths
Cyrille gave us a fantastic session about comparing responses generated by beginner programmers and AI, pattern detection, clustering.
Sieving Coding Assignments Over Submissions Generated by AI and Novice Programmers
Finally, Elizaveta Artser, from JetBrains, presented Rhubarb, a custom clustering tool for selecting diverse Python solutions to show in MOOCs.
Clustering MOOC Programming Solutions to Diversify Their Presentation to Students
To end the day, there were board games and a party with a band included (apparently they were famous in Finland).
Photo Credits: Ilkka Jormanainen
Photo Credits: Ilkka Jormanainen
Saturday:
Session 6 (Regular): Teaching Tools – Session Chair Leo Leppänen
The first block today was dedicated to teaching tools. I enjoyed it a lot, I think there’s a great opportunity here.
We started with a block-based testing framework for Scratch presented by Ute Heuer from Passau (Germany). Very solid.
A Block-Based Testing Framework for Scratch
The second was about a tool created for LogEx problems and Parsons for equivalence demonstrations in logic by Ebrahim Rahimi, from Open University (Netherlands). I loved his way of explaining Parsons problems and his tool demo.
Parsons Problems for Equivalence Proofs in Logic
The third was about a tool for direct manipulation for teaching basic and advanced software testing presented by Maximilian Georg Barth from ETH.
A Direct Manipulation Programming Environment for Teaching Introductory and Advanced Software Testing
The last paper of this block fascinated me with its approach. The basic idea is to combine LLMs and static analysis to generate progressive hints in programming tasks. Thank you, Anastasiia, great presentation!
One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks
Session 7 (Panel): Teaching Approaches – Session Chair Brian Dorn
The next block of the conference was panel style and focused on pedagogical approaches.
The first session was about cognitive load, increasing self-efficacy, and promoting conceptual understanding in basic programming, by Noah Cowit.
Designing for Interdisciplinary Transfer to Reduce Intrinsic Cognitive Load, Increase Self-Efficacy, and Promote Conceptual Understanding in Introductory Programming
The second was about the limited impact that spatial skills training has on performance in a non-programming CS course and the factors that impact those skills to improve them.
The Effect (or Lack Thereof) of Spatial Skills Training in a Mid-Major Computing Course
The third session was given by Max Sölch, a super nice guy doing his PhD at TUM (Munich), and presented a tool to make exams more computerized under the Bring Your Own Device (BYOD) framework. His great team has created Artemis
Transforming Computer-Based Exams with BYOD: An Empirical Study
We finished with a panel discussing topics like the time it takes to adopt the discussed practices, how to make the tool fair and work for everyone (an idea from Noah was representing different groups in the sample, for example), whether some programming languages are more appropriate for these pedagogical approaches, or how to make solutions work for all types of computers (and make it more equitable).
Teaching Approaches Panel
Then we enjoyed a Koli walk with spectacular views (if it weren’t for the fog). Still, a gift.
Koli walk
Great guy Max. We really hit it off.
Selfie with Max
Another of the stars I met was Fabian who works at Volkswagen but also dabbles in the field. He won best paper at the conference!
Selfie with Fabian
Session 8: Posters (including afternoon coffee) – Session Chair Andreas Mühling
We closed with some final posters. The winners for best in-person poster voting were: Nick Falkner and Ute Heuer. Congrats guys!
One of the winners for best poster
Session 9 (Panel): New Didactic Approaches – Session Chair Lauri Malmi
First was Anna Sollazzo’s turn: Towards a theory of humanistic computer science and its teaching. This talk broke me with the depth of the topic (I think everyone) but was revealing because it made us think about the intersection of both disciplines and about what it means to be human… BIG TOPIC.
Towards a Theory of Humanistic Computing and How to Teach It
Carsten Schulte closed with a very well-presented and quite theoretical session with new perspectives on the future of Computing Education: exploratory models in teaching and learning.
New Perspectives on the Future of Computing Education: Teaching and Learning Explanatory Models
After the closing dinner, we also had the Koli Calling 2024 awards where the best paper winner was announced.
It went to Fabian Pfütsch and Frank Höppner with the paper “Estimating and Differentiating Programming Skills from Unstructured Coding Exercises via Matrix Factorization”. Congratulations, Fabian Pfütsch and Frank Höppner! I’m so happy about this well-deserved award, colleagues. Since I was having dinner next to them, I just took their picture:
Fabian and Frank receiving their well-deserved award
Final Reflections
Finally, I’d like to tell you the best things I take away from my first experience at an academic conference:
- I realized the breadth and dimension the field has.
- I demystified professors and researchers, seeing them as approachable and collaborative.
- I came away with a better approach to structuring each section of a paper.
- Unlike industry conference talks, the sessions at this conference were excellent because they underwent a rigorous selection process beforehand. Usually, the first authors were present, so they had deep knowledge of the study and could answer even the most challenging questions.
- The experience benefited me in several ways: it supported my personal projects, strengthened my doctorate application, and helped me connect with potential collaborators.
- I thoroughly enjoyed every conversation. Being a conference held in a small, crowded classroom, it fostered an intimate atmosphere that encouraged lively and positive discussions. It also allowed for easy and direct interaction with all attendees. Koli brought together people who shared a genuine enthusiasm for the same topics, creating a delightful feedback loop of energy and ideas.
I hope this post helps you experience a bit of what it was like.