Side-Eye shares some of its basic capabilities with traditional debuggers, but differs from them both in its mode of usage, and in the underlying technology. Philosophically, Side-Eye is focused on server software running in production environments, services with larger footprints (multiple microservices executing different binaries on multiple machines), and teams of people debugging a service over time. A traditional debugger has a narrower focus stemming from the desktop software era — a single person debugging a single process that they can pause and resume at will, with every session starting from scratch every time.

A debugger like Delve provides features like breakpoints and reading variables out of a target process’ memory. You generally use Delve either through its command line interface, or through your editor. Every Delve session starts with a “blinking cursor” — the debugger says nothing until you ask it very specific questions. The usual interaction is that you stop the process, think for a while, look at some data, resume it, and maybe stop it again later to look at more data.

Side-Eye has some of the same technical abilities to interact with processes that are not particularly cooperating with the debugger, but they are packaged differently:

1. You’re generally not debugging a single process (although you certainly can use it on a single process); you’re debugging many different processes at once, running on different machines.
2. You benefit from your colleagues having previously used Side-Eye to instrument the programs you’re looking at: all the data that they’ve asked for is automatically collected for you to peruse.
3. Side-Eye is all about raising the level of abstraction that you’re supposed to operate on when debugging your service. Data is collected automatically for you across goroutines and functions so that you don’t have to ask for it variable by variable. Then, data can be further organized into SQL tables which can act as “reports”. For example, when debugging a server, you might build a table of all the requests or queries that are currently executing. Many debugging sessions might start by looking at this table instead of wandering aimlessly through goroutines. The oldest running request might be interesting, for example, at which point you might drop from that request’s details onto lower level things — for example by navigating to that query’s goroutine to look at its exact state of execution.
4. Being focused on dynamic, heavily concurrent and parallel software, Side-Eye has particular features for exploring goroutine stack traces: every Side-Eye snapshot includes the stack traces of all the goroutines and they can be inspected graphically.
5. Side-Eye does not “pause” the target processes for any measurable amount of time (capturing a snapshot takes on the order of a millisecond), and there’s certainly no human in the loop while a target process is paused. You interact with your service by taking a snapshot, letting the service continue unperturbed, and then later maybe taking another snapshot asking for more data.

At a technical level, Side-Eye uses the same debug information produced by the compiler that Delve does in order to find the locations of variables and make sense of a process’ memory. However, the actual interaction with a target process works differently. Most debuggers use the ptrace() system call to interact with the target process in a piecemeal way: one ptrace() call to read a pointer value from the target memory space, then another call to dereference that pointer, etc. Side-Eye works differently, for much greater efficiency: based on the data that Side-Eye was asked to collect, it uses the debug info to generate a eBPF probe that it injects into the target process. See more details below.