Welcome back, process wranglers and interrupt dodgers.

If you're still here, you've already accepted the truth: the kernel doesn't babysit, the CPU doesn't forgive, and the abstractions will fail you.

In this part, we tear apart some of the most seductive lies in systems programming – time, CPU usage, and signal handling. Buckle up. We're past page faults now. We're in the zone where physics and illusion blur.

Also see Part I, Part II, and Part III.

Lesson 16: System Time Lies – Always

"The only real time is TSC. Everything else is a distorted reflection."
– Ren, low-jitter engineer, syncs clocks across 20,000 machines with nanosecond precision

The gettimeofday() you love? It lies. NTP adjustments, leap seconds, time drift, clocksource jitter – it's all fake. Even clock_gettime(CLOCK_MONOTONIC) can get warped on suspend/resume.

You want real time? Use the TSC – the Time Stamp Counter. It ticks with the CPU, unadjusted, uncaring. It's raw. It's fast. It doesn't lie.

Protip: Calibrate your TSC. Check invariance. Use it for performance timing only – never for wall clocks. And never trust time from a userspace library without proof.

Lesson 17: CPU Usage Is a Story You Tell Yourself

"Your process is idle. But it's using 30% CPU. Because you're polling like a maniac."
– Jules, perf wizard, wrote a dashboard that revealed the truth behind "idle loops"

The CPU usage you see in top or htop? It's not real. It's a summation of what the kernel thinks your process wanted to do. If you're busy-looping on a shared memory flag? Congrats – you're eating CPU for no reason.

On the flip side, a thread doing real work in short bursts might show as using "0%". Because it did everything between sample points.

Protip: Use perf top, not top. Count instructions, not time. And know the difference between busy and productive.

Lesson 18: Signals Are the Spawn of Chaos

"Signals are like interrupts from a drunk kernel."
– Aya, realtime systems dev, replaced signals with pipes everywhere

POSIX signals sound great: asynchronous notifications, process control, alarm bells. But they're unsafe, unstructured, and mostly un-debuggable.

What functions are safe in a signal handler? Almost none. Did your signal interrupt a malloc()? You're now in undefined behavior land. Did you install a signal handler in a multithreaded process? Welcome to the land of unpredictable race conditions.

Protip: Don't use SIGALRM. Use timerfd_create(). Don't use SIGIO. Use epoll. Treat signals like you treat global variables – legacy cruft.

Lesson 19: Preemption Will Break You

"The kernel will yank your thread off the core – mid-move – unless you beg it not to."
– Omar, realtime audio engineer, once debugged a 300-microsecond jitter caused by preemptive scheduling

You think your thread is running? It might be – until the scheduler decides to "balance load", move you to another CPU, trash your cache, and insert a 1ms stall.

Unless you've explicitly pinned threads (sched_setaffinity), locked memory (mlockall), and disabled preemption (SCHED_FIFO), you have no guarantees.

Protip: Pin your hot threads. Use perf sched to see preemption. And remember: Linux is not realtime until you make it.

Lesson 20: Deterministic I/O Is a Fight

"Want to guarantee every packet goes out in under 100µs? Better start building your own OS."
– El, embedded network engineer, built TCP-like logic on raw Ethernet frames

Most I/O in modern systems is buffered, batched, delayed. write() doesn't write. send() doesn't send. They queue. The real delivery is handled later – by interrupts, daemons, NAPI loops, hidden queues.

If you're writing software that needs deterministic I/O – think HFT, robotics, audio processing – you're up against a monster. You must disable coalescing. Tune every buffer. Trace every latency source.

Protip: Use setsockopt() like a scalpel. Disable Nagle. Tune SO_RCVBUF, SO_SNDBUF, and TCP_NODELAY. And trace with packet timestamps – not logs.

Hacker Meditation: The System Is an Illusion

Every number you read – time, CPU, memory – is a guess. Every behavior – scheduling, I/O, cache access – is probabilistic.

You don't program a machine. You program an illusion of a machine, built on 100 layers of trickery, performance hacks, and legacy trade-offs.

The deeper you go, the more you see: There is no "system". There is only behavior.

Coming Up in Part V

  • Cache prefetching: the silent performance killer
  • Lockless queues: design or die
  • NUMA: the architecture that hates you
  • DMA: when the hardware does it better
  • When your CPU is doing nothing and still losing

Update: Part V is live!

As always, drop your tales from the trench – the weirder the better. This is a living logbook of reality at the lowest level.

Until next time. Stay low, stay locked, stay hacker.

P.S. Want this printed out of /dev/random and sorted later with awk? I can do that too.