CS 259: Conducting effectful tasks with agents (a programming languages approach)

Course ID: CS259
Instructor: Konstantinos Kallas (he/they) (kkallas@ucla.edu) (Website)
Meeting Time & Location: Tuesdays/Thursdays 4:00-5:50PM -- TBD
Office Hours: By appointment

Synopsis

This graduate-level course focuses on how to program and describe effectful programs, i.e., programs that perform actions that affect their surrounding system, in the context of agents. Topics that we will cover include programmability and language design, integrating black-box external components, specification inference, correctness checking, and benchmarks and workloads for such systems.

The class is heavily focused on research. A key goal of it is to collectively come up with important characteristics and aspects of how to correctly develop effectful programs in the context of agents. This will be done as a final project that students will do in groups. Students will split up in groups from the start of the class and will collaborate on designing parts of the language. We will use the final lecture of each section to discuss and summarize what we have seen so far, and try identifying principles for the final framework. Each group will also make a final report and presentation about the component they end up working on.

By the end of the course, students should be able to:

The size of the class is very limited and requires background in a combination of programming languages, software systems, and agentic systems. Since it is a research heavy class, PhD students will get a priority. If you are interested in the class and haven't managed to enroll, come to the first class and talk to the instructor.

Prerequisites

Having taken courses on systems or programming languages topics. Having programmed using agentic systems and having written programs that affect the surrounding environment. A key prerequisite is to be interested in doing research! Some possibly relevant courses are: Distributed Systems (CS 134), Compilers (CS 132), Programming Languages (CS 131), Operating Systems (CS 111), Web Applications (CS 144), Parallel and Distributed Computing (CS 133), Data Management Systems (CS 143), and Networks (CS 118).

Grading

Response papers (20%); Participation/Discussion (40%); Project (40%).

Response papers

Students will prepare an insightful critique of the assigned papers due at the beginning of class. These response papers should address the following three axes: problem statement, key insight, evaluation. Read this post for more context on important axes of systems papers. The response papers can be as short as 0.5 pages and no longer than 1 page.

Participation/Discussion

This course focuses heavily on class participation and discussion. Students are expected to read the assigned papers and prepare for discussions.

Project

Students will team up in groups and conduct research on each of the course sections, and then keep working on this research until the final project in the end of the class, when they will have to prepare a report and presentation.

Reading list

We will be reading papers from conferences like: SOSP, PLDI, OSDI, ASPLOS, NSDI, EuroSys, POPL, ATC, OOPSLA, ICFP, and SoCC.

We will be reading papers in the following categories: programmability and language design; integrating black-box external components; specification inference and correctness checking; and benchmarks/workloads.

Programmability and Language Design

What kind of features do we want in a language for writing effectful programs in the context of LLM agents? Do we want a complex language like real-world Python or Shell? Or do we want no language at all? Some example papers we will cover: React, SGLang, Smoosh, CaMeL.

Integrating Black-box external components

How do we integrate external components like Unix utilities and networked services? How can we reason about these components? Do we reimplement them internally? If not, how do we determine their behaviors? What is an abstraction for this? Example papers we will read: Harp, Caruca, and Synthesis from Partial Traces.

Specification inference and correctness checking

What is the best way to generate specifications that capture the user intent? Do we aim for full or partial specifications? Do we expect users to write assertions or do we expect them to use templates? Can we infer specifications automatically? Given the specifications, how do we check that they are satisfied? Ahead of time or at runtime? Example papers we will read: AgentPex, Conseca, SaSh, try.

Benchmarks and Workloads

What are the relevant tasks, benchmarks, and workloads that we can use to evaluate an effectful program development framework? Example papers we will read: OSWorld, TerminalBench, tau^2 bench.