How it works

Side-Eye is made out of a control-plane that we (Data Ex Machina) run as a hosted service, and an agent that you need to run on every machine where any processes to be monitored are executing (instead of running the agent, a library can also be imported into your program). When you ask for a snapshot of a bunch of processes, the Side-Eye control plane looks at your specification of what data to collect, asks the agents to retrieve the debug information of all the binaries corresponding to the processes to snapshot, and dynamically generates two programs for each one of the binaries: an eBPF probe that will be injected into the target processes and a specification for how to interpret the raw binary data produced by the probe.

eBPF is a Linux kernel technology that allows, among many other things, for code to be injected into a running process. The kernel verifies the injected code to be “safe” – it is guaranteed to not run for very long and to not cause any side-effects for the host program. The generated Side-Eye eBPF probes know how do to a couple of things:

• iterate through all the live goroutines and walk their stacks
• while walking the stacks, recognize functions for which any data is to be collected
• for such a function, locate the required variables in memory depending on the program counter corresponding to that function
• read the respective variables from memory and copy them to an output buffer
• besides collecting full variables, Side-Eye can also “evaluate” expressions like foo.bar.baz consisting of accessing struct fields and dereferencing pointers, recursively
• the probe understand a various go data structures: for example, it can dereference interfaces and figure out what type hides behind the interface box

Before triggering the probe inside each target process, the agent briefly pauses the execution of all the threads from that process; they will be restarted once the probe finishes. This ensures that the data that will be collected from the (stalks of the) different goroutine is coherent. The probe runs, walks all the stacks, collects all the requested data, and streams it to the local agent. All the agents communicate all the collected data to a central server, where the data is interpreted according to the specification generated from the debug information. The binary data is now turned into human-readable JSON which includes, for example, the names of the fields of structs.

Finally, the data retrieved from the different agents (pertaining to the processes running on those respective machines) is bundled together into one snapshot.

eBPF is a Linux-specific technology, so our agent only works on Linux. Snapshots can also be collected from applications running on macOS if these programs import the Side-Eye client library. In this case, the library “acts as the agent” – it communicates with the Side-Eye control plane receiving snapshot commands that include a “program” instructing the library what memory to read and send back to Side-Eye.

Dynamic events

For a snapshot, an eBPF probe executes once per target process and collects data across all stack frames in all the goroutines. In contrast, dynamic events work by triggering a probe every time the respective events happen (i.e. every time the functions corresponding to the events are called). Side-Eye generates an eBPF probe for every function. These probes share a lot of the inner workings with the snapshot probe: they are produced based on the debug information of the binaries involved and the generated code knows how to find the requested variables in memory and how to evaluate the requested expressions by recursively dereferencing pointers. Probes are injected into the target processes and enabled for the tracing duration; as the respective functions are called, the probes collect the requested data and stream it out to the agent, which then streams it out to the central server which decodes it and stores it.

eBPF is a Linux kernel technology that allows, among many other things, for code to be injected into a running process. The kernel verifies the injected code to be “safe” – it is guaranteed to not run for very long and to not cause any side-effects for the host program. The generated Side-Eye eBPF probes know how do to a couple of things:

• iterate through all the live goroutines and walk their stacks
• while walking the stacks, recognize functions for which any data is to be collected
• for such a function, locate the required variables in memory depending on the program counter corresponding to that function
• read the respective variables from memory and copy them to an output buffer
• besides collecting full variables, Side-Eye can also “evaluate” expressions like foo.bar.baz consisting of accessing struct fields and dereferencing pointers, recursively
• the probe understand a various go data structures: for example, it can dereference interfaces and figure out what type hides behind the interface box

Before triggering the probe inside each target process, the agent briefly pauses the execution of all the threads from that process; they will be restarted once the probe finishes. This ensures that the data that will be collected from the (stalks of the) different goroutine is coherent. The probe runs, walks all the stacks, collects all the requested data, and streams it to the local agent. All the agents communicate all the collected data to a central server, where the data is interpreted according to the specification generated from the debug information. The binary data is now turned into human-readable JSON which includes, for example, the names of the fields of structs.

Finally, the data retrieved from the different agents (pertaining to the processes running on those respective machines) is bundled together into one snapshot.

eBPF is a Linux-specific technology, so our agent only works on Linux. Snapshots can also be collected from applications running on macOS if these programs import the Side-Eye client library. In this case, the library “acts as the agent” – it communicates with the Side-Eye control plane receiving snapshot commands that include a “program” instructing the library what memory to read and send back to Side-Eye.

Dynamic events

For a snapshot, an eBPF probe executes once per target process and collects data across all stack frames in all the goroutines. In contrast, dynamic events work by triggering a probe every time the respective events happen (i.e. every time the functions corresponding to the events are called). Side-Eye generates an eBPF probe for every function. These probes share a lot of the inner workings with the snapshot probe: they are produced based on the debug information of the binaries involved and the generated code knows how to find the requested variables in memory and how to evaluate the requested expressions by recursively dereferencing pointers. Probes are injected into the target processes and enabled for the tracing duration; as the respective functions are called, the probes collect the requested data and stream it out to the agent, which then streams it out to the central server which decodes it and stores it.