CS 567 Winter 2024
Empirical Lab Studies (Quantitative) in SE and HCI

Instructor: Dr. Burnett
Office: KEC 3051
E-Mail: burnett@eecs.oregonstate.edu

Dr. Burnett's Office Hours are listed on my home page

Course Description

The course is about empirical methods in understanding humans' use of tools, environments, and practices with software and/or software development. There are many possibilities about empirical methods, and we can't cover them all. Thus, this course will focus on a method that is very useful for certain kinds of questions in HCI/SE, but is not well understood in computer science: how to scientifically conduct and analyze (statistically oriented) laboratory studies with human participants.

What the course is:

• This course will cover how you go about designing, preparing for, running, analyzing, and writing-for-publication quantitative (statistical) lab experiments with human participants. This is an end-to-end coverage of the entire process, and will put you in a position to conduct lab studies of your own with human participants.

• This is not a statistics course, although we will cover a couple of basic stats. If what you really want is a statistics-for-experiments course, I recommend the Stats 511 and/or Stats 515 courses, which are excellent, and are specifically targeted to non-stats grad students.
• This course will not produce data suitable for analysis using AI/ML techniques. If you want that kind of course, you should take one of our Data Mining courses instead.

Course objectives

1. Choose when a (statistically oriented) lab study is the right choice of empirical work.
2. Design, conduct, and gather data in such lab studies...
3. ... according to accepted ethical principles of dealing with human subjects.
4. Analyze data in lab studies using quantitative (statistical) methods.
5. Report quantitative lab study empirical work in research publications.

Prerequisites

Contrary to what the catalog or scheduling system might say, the only prerequisite for this course is grad standing in Computer Science.

How the course will be conducted, method of instruction

You'll actually do a lab study of some programming language, tool, or practice, with a team. You and your teammates will choose your project together.

• Any idea within the scope of this course will be fine; it does not have to relate to your research or thesis activity. However, it is allowed to relate to your research if you want it to (read on).
• Optionally, this can be a study you need for your research, with the IRB approval already in place very early in the course (so it is probably something your advisor has been planning to do for awhile).
• Optionally, what you do in this class could turn out to be a pilot in preparation for a "real" study related to your thesis and/or research.

I'm anticipating little or no programming in this course. There will be some lectures by me; about half of the classes will be like this. The other half of the class will be more studio/discussion style, with teams discussing and critiquing each others' work, and based on team presentations in which the class jointly provides feedback on some aspect of a team's case study. In short, it will be highly interactive.

Textbooks

• Required, ISBN/SKU 9781848000438, Shull et al., Guide to Advanced Empirical Software Engineering, Springer, 2008.

We will also have selected readings from other sources, but you don't have to buy those.

I have high expectations, and expect performance worthy of graduate students in computer science. Thus, in this class, "A" does not mean "adequate" or "nothing wrong" -- it means "excellent". For an A, you should expect to dig deep and get the most you can out of the class.

Resources:

• Canvas site for this course
• The last pre-Canvas website for this class (2014 version): The schedule, etc., for this year will be similar to that one.

Margaret M. Burnett
Date of last update: Jan. 19, 2024