% Writing an Operating System in 1,000 Lines, v0.1-alpha % Seiya Nuta

Operating System in 1,000 Lines

Hey there! In this book, we're going to build a small operating system from scratch, step by step.

You might get intimidated when you hear OS or kernel development, the basic functions of an OS (especially the kernel) are surprisingly simple. Even Linux, which is often cited as a huge open-source software, was only 8,413 lines in version 0.01. Today's Linux kernel is overwhelmingly large, but it started with a tiny codebase, just like your hobby project.

We'll implement basic context switching, paging, user mode, a command-line shell, a disk device driver, and file read/write operations in C. Sounds like a lot, however, it's only 1,000 lines of code!

One thing you should remember is, it's not as easy as it sounds. The tricky part of creating your own OS is debugging. You can't do printf debugging until you implement it. You'll need to learn different debugging techniques and skills you've never needed in application development. Especially when starting "from scratch", you'll encounter challenging parts like boot process and paging. But don't worry! We'll also learn "how to debug an OS" too!

The tougher the debugging, the more satisfying it is when it works. Let's dive into exciting world of OS development!

You can download the implementation examples from GitHub.

This book is available under the CC BY 4.0 license. The implementation examples and source code in the text are under the MIT license.

The prerequisites are familiarity with the C language and with a UNIX-like environment. If you've run gcc hello.c && ./a.out, you're good to go!

This book was originally written as an appendix of my book Design and Implementation of Microkernels (written in Japanese).

Happy OS hacking!

title: Getting Started

Getting Started

This book assumes you're using a UNIX or UNIX like OS such as macOS or Ubuntu. If you're on Windows, install Windows Subsystem for Linux (WSL2) and follow the Ubuntu instructions.

Install development tools

macOS

Install Homebrew and run this command to get all tools you need:

brew install llvm lld qemu

Also, you need to add LLVM binutils to your PATH:

$ export PATH="$PATH:$(brew --prefix)/opt/llvm/bin"
$ which llvm-objcopy
/opt/homebrew/opt/llvm/bin/llvm-objcopy

Ubuntu

Install packages with apt:

sudo apt update && sudo apt install -y clang llvm lld qemu-system-riscv32 curl

Also, download OpenSBI (think of it as BIOS/UEFI for PCs):

curl -LO https://github.com/qemu/qemu/raw/v8.0.4/pc-bios/opensbi-riscv32-generic-fw_dynamic.bin

Warning

When you run QEMU, make sure opensbi-riscv32-generic-fw_dynamic.bin is in your current directory. If it's not, you'll see this error:

qemu-system-riscv32: Unable to load the RISC-V firmware "opensbi-riscv32-generic-fw_dynamic.bin"

Other OS users

If you are using other OSes, get the following tools:

bash: The command-line shell. Usually it's pre-installed.

tar: Usually it's pre-installed. Prefer GNU version, not BSD.

clang: C compiler. Make sure it supports 32-bit RISC-V CPU (see below).

lld: LLVM linker, which bundles complied object files into an executable.

llvm-objcopy: Object file editor. It comes with the LLVM package (typically llvm package).

llvm-objdump: A disassembler. Same as llvm-objcopy.

llvm-readelf: An ELF file reader. Same as llvm-objcopy.

qemu-system-riscv32: 32-bit RISC-V CPU emulator. It's part of the QEMU package (typically qemu package).

Tip

To check if your clang supports 32-bit RISC-V CPU, run this command:

$ clang -print-targets | grep riscv32
    riscv32     - 32-bit RISC-V

You should see riscv32. Note pre-installed clang on macOS won't show this. That's why you need to install another clang in Homebrew's llvm package.

Setting up a Git repository (optional)

If you're using a Git repository, use the following .gitignore file:

/disk/*
!/disk/.gitkeep
*.map
*.tar
*.o
*.elf
*.bin
*.log
*.pcap

You're all set! Let's start building your first operating system!

RISC-V 101

Just like web browsers hide the differences between Windows/macOS/Linux, operating systems hide the differences between CPUs. In other words, operating system is a program which controls the CPU to provide an abstraction layer for applications.

In this book, I chose RISC-V as the target CPU because:

The specification is simple and suitable for beginners.

It's a trending ISA (Instruction Set Architecture) in recent years, along with x86 and Arm.

The design decisions are well-documented throughout the spec and they are fun to read.

We will write an OS for 32-bit RISC-V. Of course you can write for 64-bit RISC-V with only a few changes. However, the wider bit width makes it slightly more complex, and the longer addresses can be tedious to read.

QEMU virt machine

Computers are composed of various devices: CPU, memory, network cards, hard disks, and so on. For example, although iPhones and Raspberry Pis use Arm CPUs, it's natural to consider them as different computers.

In this book, we support the QEMU virt machine (documentation) because:

Even though it does not exist in the real world, it's simple and very similar to real devices.

You can emulate it on QEMU for free. You don't need to buy a physical hardware.

When you encounter debugging issues, you can read QEMU's source code, or attach a debugger to the QEMU process to investigate what's wrong.

RISC-V assembly 101

RISC-V, or RISC-V ISA (Instruction Set Architecture), defines the instructions that the CPU can execute. It's similar to APIs or programming language specifications for programmers. When you write a C program, the compiler translates it into RISC-V assembly. Unfortunately, you need to write some assembly code to write an OS. But don't worry! Assembly is not as difficult as you might think.

Tip

Try Compiler Explorer!

A useful tool for learning assembly is Compiler Explorer, an online compiler. As you type C code, it shows the corresponding assembly code.

By default, Compiler Explorer uses x86-64 CPU assembly. Specify RISC-V rv32gc clang (trunk) in the right pane to output 32-bit RISC-V assembly.

Also, it would be interesting to specify optimization options like -O0 (optimization off) or -O2 (optimization level 2) in the compiler options and see how the assembly changes.

Assembly language basics

Assembly language is a (mostly) direct representation of machine code. Let's take a look at a simple example:

addi a0, a1, 123

Typically, each line of assembly code corresponds to a single instruction. The first column (addi) is the instruction name, also known as the opcode. The following columns (a0, a1, 123) are the operands, the arguments for the instruction. In this case, the addi instruction adds the value 123 to the value in register a1, and stores the result in register a0.

Registers

Registers are like temporary variables in the CPU, and they are way faster than memory. CPU reads data from memory into registers, does arithmetic operations on registers, and writes the results back to memory/registers.

Here are some common registers in RISC-V:

Register | ABI Name (alias) | Description
pc | pc | Program counter (where the next instruction is)
x0 | zero | Hardwired zero (always reads as zero)
x1 | ra | Return address
x2 | sp | Stack pointer
x5 - x7 | t0 - t2 | Temporary registers
x8 | fp | Stack frame pointer
x10 - x11 | a0 - a1 | Function arguments/return values
x12 - x17 | a2 - a7 | Function arguments
x18 - x27 | s0 - s11 | Temporary registers saved across calls
x28 - x31 | t3 - t6 | Temporary registers

Tip

Calling convention:

Generally, you may use CPU registers as you like, but for the sake of interoperability with other software, how registers are used is well defined - this is called the calling convention.

For example, x10 - x11 registers are used for function arguments and return values. For human readability, they are given aliases like a0 - a1 in the ABI. Check the spec for more details.

Memory access

Registers are really fast, but they are limited in number. Most of data are stored in memory, and programs reads/writes data from/to memory using the lw (load word) and sw (store word) instructions:

lw a0, (a1)  // Read a word (32-bits) from address in a1
             // and store it in a0. In C, this would be: a0 = *a1;

sw a0, (a1)  // Store a word in a0 to the address in a1.
             // In C, this would be: *a1 = a0;

You can consider (...) as a pointer dereference in C language. In this case, a1 is a pointer to a 32-bits-wide value.

Branch instructions

Branch instructions change the control flow of the program. They are used to implement if, for, and while statements,

    bnez    a0, <label>   // Go to <label> if a0 is not zero
    // If a0 is zero, continue here

<label>:
    // If a0 is not zero, continue here

bnez stands for "branch if not equal to zero". Other common branch instructions include beq (branch if equal) and blt (branch if less than). They are similar to goto in C, but with conditions.

Function calls

jal (jump and link) and ret (return) instructions are used for calling functions and returning from them:

    li  a0, 123      // Load 123 to a0 register (function argument)
    jal ra, <label>  // Jump to <label> and store the return address
                     // in the ra register.

    // After the function call, continue here...

// int func(int a) {
//   a += 1;
//   return a;
// }
<label>:
    addi a0, a0, 1    // Increment a0 (first argument) by 1

    ret               // Return to the address stored in ra.
                      // a0 register has the return value.

Function arguments are passed in a0 - a7 registers, and the return value is stored in a0 register, as per the calling convention.

Stack

Stack is a Last-In-First-Out (LIFO) memory space used for function calls and local variables. It grows downwards, and the stack pointer sp points to the top of the stack.

To save a value into the stack, decrement the stack pointer and store the value (aka. push operation):

    addi sp, sp, -4  // Move the stack pointer down by 4 bytes
                     // (i.e. stack allocation).

    sw   a0, (sp)    // Store a0 to the stack

To load a value from the stack, load the value and increment the stack pointer (aka. pop operation):

    lw   a0, (sp)    // Load a0 from the stack
    addi sp, sp, 4   // Move the stack pointer up by 4 bytes
                     // (i.e. stack deallocation).

Tip

In C, stack operations are generated by the compiler, so you don't have to write them manually.

CPU modes

CPU has multiple modes, each with different privileges. In RISC-V, there are three modes:

Mode | Overview
M-mode | Mode in which OpenSBI (i.e. BIOS) operates.
S-mode | Mode in which the kernel operates, aka. "kernel mode".
U-mode | Mode in which applications operate, aka. "user mode".

Privileged instructions

Among CPU instructions, there are types called privileged instructions that applications (user mode) cannot execute. In this book, we use the following privileged instructions:

Opcode and operands | Overview | Pseudocode
csrr rd, csr | Read from CSR | rd = csr;
csrw csr, rs | Write to CSR | csr = rs;
csrrw rd, csr, rs | Read from and write to CSR at once | tmp = csr; csr = rs; rd = tmp;
sret | Return from trap handler (restoring program counter, operation mode, etc.) |
sfence.vma | Clear Translation Lookaside Buffer (TLB) |

CSR (Control and Status Register) is a register that stores CPU settings. The list of CSRs can be found in RISC-V Privileged Specification.

Tip

Some instructions like sret do some somewhat complex operations. To understand what actually happens, reading RISC-V emulator source code might be helpful. Particularly, rvemu is written in a intuitive and easy-to-understand way (e.g. sret implementation).

Inline assembly

In following chapters, you'll encounter special C language syntax like this:

uint32_t value;
__asm__ __volatile__("csrr %0, sepc" : "=r"(value));

This is "inline assembly", a syntax for embedding assembly into C code. While you can write assembly in a separate file (.S extension), using inline assembly is generally preferred because:

You can use C variables within the assembly. Also, you can assign the results of assembly to C variables.

You can leave register allocation to the C compiler. That is, you don't have to manually write the preservation and restoration of registers to be modified in the assembly.

How to write inline assembly

Inline assembly is written in the following format:

__asm__ __volatile__("assembly" : output operands : input operands : clobbered registers);

Part | Description
__asm__ | Indicates it's an inline assembly.
__volatile__ | Tell the compiler not to optimize the "assembly" code.
"assembly" | Assembly code written as a string literal.
output operands | C variables to store the results of the assembly.
input operands | C expressions (e.g. 123, x) to be used in the assembly.
clobbered registers | Registers whose contents are destroyed in the assembly. If forgotten, the C compiler won't preserve the contents of these registers and would cause a bug.

Output and input operands are comma-separated, and each operand is written in the format constraint (C expression). Constraints are used to specify the type of operand, and usually =r (register) for output operands, and r for input operands.

Output and input operands can be accessed in the assembly using %0, %1, %2, etc., in order starting from the output operands.

Examples

uint32_t value;
__asm__ __volatile__("csrr %0, sepc" : "=r"(value));

This reads the value of the sepc CSR using the csrr instruction, and assigns it to the value variable. %0 corresponds to the value variable.

__asm__ __volatile__("csrw sscratch, %0" : : "r"(123));

This writes 123 to the sscratch CSR, using the csrw instruction. %0 corresponds to the register containing 123 (r constraint), and it would actually look like:

li    a0, 123        // Set 123 to a0 register
csrw  sscratch, a0   // Write the value of a0 register to sscratch register

Although only the csrw instruction is written in the inline assembly, the li instruction is automatically inserted by the compiler to satisfy the "r" constraint (value in a register). It's super convenient!

Tip

Inline assembly is a compiler-specific extension not included in the C language specification. You can check detailed usage in the GCC documentation. However, it takes time to understand because constraint syntax differs depending on CPU architecture, and it has many complex functionalities.

For beginners, I recommend to search for real-world examples. For instance, HinaOS and xv6-riscv are good references.

What we will implement

Before starting to build an OS, let's quickly get an overview of the features we will implement.

Features in 1K LoC OS

In this book, we will implement the following major features:

Multitasking: Switch between processes to allow multiple applications to share the CPU.

Exception handler: Handle events requiring OS intervention, such as illegal instructions.

Paging: Provide an isolated memory address space for each application.

System calls: Allow applications to call kernel features.

Device drivers: Abstract hardware functionalities, such as disk read/write.

File system: Manage files on disk.

Command-line shell: User interface for humans.

Features not implemented

The following major features are not implemented in this book:

Interrupt handling: Instead, we will use a polling method (periodically check for new data on devices), also known as busy waiting.

Timer processing: Preemptive multitasking is not implemented. We'll use cooperative multitasking, where each process voluntarily yields the CPU.

Inter-process communication: Features such as pipe, UNIX domain socket, and shared memory are not implemented.

Multi-processor support: Only single processor is supported.

Source code structure

We'll build from scratch incrementally, and the final file structure will look like this:

├── disk/     - File system contents
├── common.c  - Kernel/user common library: printf, memset, ...
├── common.h  - Kernel/user common library: definitions of structs and constants
├── kernel.c  - Kernel: process management, system calls, device drivers, file system
├── kernel.h  - Kernel: definitions of structs and constants
├── kernel.ld - Kernel: linker script (memory layout definition)
├── shell.c   - Command-line shell
├── user.c    - User library: functions for system calls
├── user.h    - User library: definitions of structs and constants
├── user.ld   - User: linker script (memory layout definition)
└── run.sh    - Build script

Tip

In this book, "user land" is sometimes abbreviated as "user". Consider it as "applications", and do not confuse it with "user account"!

Booting the Kernel

When a computer is turned on, the CPU initializes itself and starts executing the OS. OS initializes the hardware and starts the applications. This process is called "booting".

What happens before the OS starts? In PCs, BIOS (or UEFI in modern PCs) initializes the hardware, displays the splash screen, and loads the OS from the disk. In QEMU virt machine, OpenSBI is the equivalent of BIOS/UEFI.

Supervisor Binary Interface (SBI)

The Supervisor Binary Interface (SBI) is an API for OS kernels, but defines what the firmware (OpenSBI) provides to an OS.

The SBI specification is published on GitHub. It defines useful features such as displaying characters on the debug console (e.g., serial port), reboot/shutdown, and timer settings.

A famous SBI implementation is OpenSBI. In QEMU, OpenSBI starts by default, performs hardware-specific initialization, and boots the kernel.

Let's boot OpenSBI

First, let's see how OpenSBI starts. Create a shell script named run.sh as follows:

$ touch run.sh
$ chmod +x run.sh

#!/bin/bash
set -xue

# QEMU file path
QEMU=qemu-system-riscv32

# Start QEMU
$QEMU -machine virt -bios default -nographic -serial mon:stdio --no-reboot

QEMU takes various options to start the virtual machine. Here are the options used in the script:

-machine virt: Start a virt machine. You can check other supported machines with the -machine '?' option.

-bios default: Use the default firmware (OpenSBI in this case).

-nographic: Start QEMU without a GUI window.

-serial mon:stdio: Connect QEMU's standard input/output to the virtual machine's serial port. Specifying mon: allows switching to the QEMU monitor by pressing Ctrl+A then C.

--no-reboot: If the virtual machine crashes, stop the emulator without rebooting (useful for debugging).

Tip

In macOS, you can check the path to Homebrew's QEMU with the following command:

$ ls $(brew --prefix)/bin/qemu-system-riscv32
/opt/homebrew/bin/qemu-system-riscv32

Run the script and you will see the following banner:

$ ./run.sh

OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : riscv-virtio,qemu
Platform Features         : medeleg
Platform HART Count       : 1
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 10000000Hz
...

OpenSBI displays the OpenSBI version, platform name, features, number of HARTs (CPU cores), and more for debugging purposes.

When you press any key, nothing will happen. This is because QEMU's standard input/output is connected to the virtual machine's serial port, and the characters you type are being sent to the OpenSBI. However, no one reads the input characters.

Press Ctrl+A then C to switch to the QEMU debug console (QEMU monitor). You can exit QEMU by q command in the monitor:

QEMU 8.0.2 monitor - type 'help' for more information
(qemu) q

Tip

Ctrl+A has several features besides switching to the QEMU monitor (C key). For example, pressing the X key will immediately exit QEMU.

C-a h    print this help
C-a x    exit emulator
C-a s    save disk data back to file (if -snapshot)
C-a t    toggle console timestamps
C-a b    send break (magic sysrq)
C-a c    switch between console and monitor
C-a C-a  sends C-a

Linker script

A linker script is a file which defines the memory layout of executable files. Based on the layout, the linker assigns memory addresses to functions and variables.

Let's create a new file named kernel.ld:

ENTRY(boot)

SECTIONS {
    . = 0x80200000;

    .text :{
        KEEP(*(.text.boot));
        *(.text .text.*);
    }

    .rodata : ALIGN(4) {
        *(.rodata .rodata.*);
    }

    .data : ALIGN(4) {
        *(.data .data.*);
    }

    .bss : ALIGN(4) {
        __bss = .;
        *(.bss .bss.* .sbss .sbss.*);
        __bss_end = .;
    }

    . = ALIGN(4);
    . += 128 * 1024; /* 128KB */
    __stack_top = .;
}

Here are the key points of the linker script:

The entry point of the kernel is the boot function.

The base address is 0x80200000.

The .text.boot section is always placed at the beginning.

Each section is placed in the order of .text, .rodata, .data, and .bss.

The kernel stack comes after the .bss section, and its size is 128KB.

.text, .rodata, .data, and .bss sections mentioned here are data areas with specific roles:

Section | Description
.text | This section contains the code of the program.
.rodata | This section contains constant data that is read-only.
.data | This section contains read/write data.
.bss | This section contains read/write data with an initial value of zero.

Let's take a closer look at the syntax of the linker script. First, ENTRY(boot) declares that the boot function is the entry point of the program. Then, the placement of each section is defined within the SECTIONS block.

The *(.text .text.*) directive places the .text section and any sections starting with .text. from all files (*) at that location.

The . symbol represents the current address. It automatically increments as data is placed, such as with *(.text). The statement . += 128 * 1024 means "advance the current address by 128KB". The ALIGN(4) directive ensures that the current address is adjusted to a 4-byte boundary.

Finally, __bss = . assigns the current address to the symbol __bss. In C language, you can refer to a defined symbol using extern char symbol_name.

Tip

Linker scripts offer many convenient features, especially for kernel development. You can find real-world examples on GitHub!

Minimal kernel

We're now ready to start writing the kernel. Let's start by creating a minimal one! Create a C language source code file named kernel.c:

typedef unsigned char uint8_t;
typedef unsigned int uint32_t;
typedef uint32_t size_t;

extern char __bss[], __bss_end[], __stack_top[];

void *memset(void *buf, char c, size_t n) {
    uint8_t *p = (uint8_t *) buf;
    while (n--)
        *p++ = c;
    return buf;
}

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    for (;;);
}

__attribute__((section(".text.boot")))
__attribute__((naked))
void boot(void) {
    __asm__ __volatile__(
        "mv sp, %[stack_top]\n" // Set the stack pointer
        "j kernel_main\n"       // Jump to the kernel main function
        :
        : [stack_top] "r" (__stack_top) // Pass the stack top address as %[stack_top]
    );
}

Let's explore the key points one by one:

The kernel entry point

The execution of the kernel starts from the boot function, which is specified as the entry point in the linker script. In this function, the stack pointer (sp) is set to the end address of the stack area defined in the linker script. Then, it jumps to the kernel_main function. It's important to note that the stack grows towards zero, meaning it is decremented as it is used. Therefore, the end address (not the start address) of the stack area must be set.

boot function attributes

The boot function has two special attributes. The __attribute__((naked)) attribute instructs the compiler not to generate unnecessary code before and after the function body, such as a return instruction. This ensures that the inline assembly code is the exact function body.

The boot function also has the __attribute__((section(".text.boot"))) attribute, which controls the placement of the function in the linker script. Since OpenSBI simply jumps to 0x80200000 without knowing the entry point, the boot function needs to be placed at 0x80200000.

extern char to get linker script symbols

At the beginning of the file, each symbol defined in the linker script is declared using extern char. Here, we are only interested in obtaining the addresses of the symbols, so using char type is not that important.

We can also declare it as extern char __bss;, but __bss alone means "the value at the 0th byte of the .bss section" instead of "the start address of the .bss section". Therefore, it is recommended to add [] to ensure that __bss returns an address and prevent any careless mistakes.

.bss section initialization

In the kernel_main function, the .bss section is first initialized to zero using the memset function. Although some bootloaders may recognize and zero-clear the .bss section, but we initialize it manually just in case. Finally, the function enters an infinite loop and the kernel terminates.

Let's run!

Add a kernel build command and a new QEMU option (-kernel kernel.elf) to run.sh:

#!/bin/bash
set -xue

QEMU=qemu-system-riscv32

# Path to clang and compiler flags
CC=/opt/homebrew/opt/llvm/bin/clang  # Ubuntu users: use CC=clang
CFLAGS="-std=c11 -O2 -g3 -Wall -Wextra --target=riscv32 -ffreestanding -nostdlib"

# Build the kernel
$CC $CFLAGS -Wl,-Tkernel.ld -Wl,-Map=kernel.map -o kernel.elf \
    kernel.c

# Start QEMU
$QEMU -machine virt -bios default -nographic -serial mon:stdio --no-reboot \
    -kernel kernel.elf

Tip

You can check the file path of the Homebrew version of clang on macOS with the following command:

$ ls $(brew --prefix)/opt/llvm/bin/clang
/opt/homebrew/opt/llvm/bin/clang

The specified clang options (CFLAGS variable) are as follows:

Option | Description
-std=c11 | Use C11.
-O2 | Enable optimizations to generate efficient machine code.
-g3 | Generate the maximum amount of debug information.
-Wall | Enable major warnings.
-Wextra | Enable additional warnings.
--target=riscv32 | Compile for 32-bit RISC-V.
-ffreestanding | Do not use the standard library of the host environment (your development environment).
-nostdlib | Do not link the standard library.
-Wl,-Tkernel.ld | Specify the linker script.
-Wl,-Map=kernel.map | Output a map file (linker allocation result).

-Wl, means passing options to the linker instead of the C compiler. clang command does C compilation and executes the linker internally.

Your first kernel debugging

When you run run.sh, the kernel enters an infinite loop. There are no indications that the kernel is running correctly. But don't worry, this is quite common in low-level development! This is where QEMU's debugging features come in.

To get more information about the CPU registers, open the QEMU monitor and execute the info registers command:

QEMU 8.0.2 monitor - type 'help' for more information
(qemu) info registers

CPU#0
 V      =   0
 pc       80200014  ← Address of the instruction to be executed (Program Counter)
 ...
 x0/zero  00000000 x1/ra    8000a084 x2/sp    80220018 x3/gp    00000000  ← Values of each register
 x4/tp    80033000 x5/t0    00000001 x6/t1    00000002 x7/t2    00000000
 x8/s0    80032f50 x9/s1    00000001 x10/a0   80220018 x11/a1   87e00000
 x12/a2   00000007 x13/a3   00000019 x14/a4   00000000 x15/a5   00000001
 x16/a6   00000001 x17/a7   00000005 x18/s2   80200000 x19/s3   00000000
 x20/s4   87e00000 x21/s5   00000000 x22/s6   80006800 x23/s7   8001c020
 x24/s8   00002000 x25/s9   8002b4e4 x26/s10  00000000 x27/s11  00000000
 x28/t3   616d6569 x29/t4   8001a5a1 x30/t5   000000b4 x31/t6   00000000

Tip

The exact values may differ depending on the versions of clang and QEMU.

pc 80200014 shows the current program counter, the address of the instruction being executed. Let's use the disassembler (llvm-objdump) to narrow down the specific line of code:

$ llvm-objdump -d kernel.elf

kernel.elf:     file format elf32-littleriscv

Disassembly of section .text:

80200000 <boot>:  ← boot function
80200000: 37 05 22 80   lui     a0, 524832
80200004: 13 05 85 01   addi    a0, a0, 24
80200008: 2a 81         mv      sp, a0
8020000a: 6f 00 60 00   j       0x80200010 <kernel_main>
8020000e: 00 00         unimp

80200010 <kernel_main>:  ← kernel_main function
80200010: 73 00 50 10   wfi
80200014: f5 bf         j       0x80200010 <kernel_main>  ← pc is here

Each line corresponds to an instruction. Each column represents:

The address of the instruction.

Hexadecimal dump of the machine code.

Disassembled instructions.

pc 80200014 means the currently executed instruction is j 0x80200010. This confirms that QEMU has correctly reached the kernel_main function.

Let's also check if the stack pointer (sp register) is set to the value of __stack_top defined in the linker script. The register dump shows x2/sp 80220018. To see where the linker placed __stack_top, check kernel.map file:

     VMA      LMA     Size Align Out     In      Symbol
       0        0 80200000     1 . = 0x80200000
80200000 80200000       16     4 .text
...
80200016 80200016        2     1 . = ALIGN ( 4 )
80200018 80200018    20000     1 . += 128 * 1024
80220018 80220018        0     1 __stack_top = .

Alternatively, you can also check the addresses of functions/variables using llvm-nm:

$ llvm-nm kernel.elf
80200010 t .LBB0_1
00000000 N .Lline_table_start0
80220018 T __stack_top
80200000 T boot
80200010 T kernel_main

The first column is the address where they are placed (VMA). You can see that __stack_top is placed at 0x80220018. This confirms that the stack pointer is correctly set in the boot function. Nice!

As execution progresses, the results of info registers will change. If you want to temporarily stop the emulation, you can use the stop command in the QEMU monitor:

(qemu) stop             ← The process stops
(qemu) info registers   ← You can observe the state at the stop
(qemu) cont             ← The process resumes

Now you've successfully written your first kernel!

Hello World!

In the previous chapter, we successfully booted our first kernel. Although we could confirm it works by reading the register dump, it still felt somewhat unsatisfactory.

In this chapter, let's make it more obvious by outputting a string from the kernel.

Say "hello" to SBI

In the previous chapter, we learned that SBI is an "API for OS". To call the SBI to use its function, we use the ecall instruction:

#include "kernel.h"

extern char __bss[], __bss_end[], __stack_top[];

struct sbiret sbi_call(long arg0, long arg1, long arg2, long arg3, long arg4,
                       long arg5, long fid, long eid) {
    register long a0 __asm__("a0") = arg0;
    register long a1 __asm__("a1") = arg1;
    register long a2 __asm__("a2") = arg2;
    register long a3 __asm__("a3") = arg3;
    register long a4 __asm__("a4") = arg4;
    register long a5 __asm__("a5") = arg5;
    register long a6 __asm__("a6") = fid;
    register long a7 __asm__("a7") = eid;

    __asm__ __volatile__("ecall"
                         : "=r"(a0), "=r"(a1)
                         : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(a4), "r"(a5),
                           "r"(a6), "r"(a7)
                         : "memory");
    return (struct sbiret){.error = a0, .value = a1};
}

void putchar(char ch) {
    sbi_call(ch, 0, 0, 0, 0, 0, 0, 1 /* Console Putchar */);
}

void kernel_main(void) {
    const char *s = "\n\nHello World!\n";
    for (int i = 0; s[i] != '\0'; i++) {
        putchar(s[i]);
    }

    for (;;) {
        __asm__ __volatile__("wfi");
    }
}

Also, create a new kernel.h file and define the return value structure:

#pragma once

struct sbiret {
    long error;
    long value;
};

We've newly added the sbi_call function. This function is designed to call OpenSBI as specified in the SBI specification. The specific calling convention is as follows:

Chapter 3. Binary Encoding

All SBI functions share a single binary encoding, which facilitates the mixing of SBI extensions. The SBI specification follows the below calling convention.

An ECALL is used as the control transfer instruction between the supervisor and the SEE.

a7 encodes the SBI extension ID (EID),

a6 encodes the SBI function ID (FID) for a given extension ID encoded in a7 for any SBI extension defined in or after SBI v0.2.

All registers except a0 & a1 must be preserved across an SBI call by the callee.

SBI functions must return a pair of values in a0 and a1, with a0 returning an error code. This is analogous to returning the C structure

struct sbiret {
    long error;
    long value;
};

-- "RISC-V Supervisor Binary Interface Specification" v2.0-rc1

Tip

"All registers except a0 & a1 must be preserved across an SBI call by the callee" means that the callee (OpenSBI side) must not change the values of except a0 and a1. In other words, from the kernel's perspective, it is guaranteed that the registers (a2 to a7) will remain the same after the call.

The register and __asm__("register name") used in each local variable declaration asks the compiler to place values in the specified registers. This is a common idiom in system call invocations (e.g., Linux system call invocation process).

After preparing the arguments, the ecall instruction is executed in inline assembly. When this is called, the CPU's execution mode switches from kernel mode (S-Mode) to OpenSBI mode (M-Mode), and OpenSBI's processing handler is invoked. Once it's done, it switches back to kernel mode, and execution resumes after the ecall instruction.

The ecall instruction is also used when applications call the kernel (system calls). This instruction behaves like a function call to the more privileged CPU mode.

To display characters, we can use Console Putchar function:

5.2. Extension: Console Putchar (EID #0x01)

  long sbi_console_putchar(int ch)

Write data present in ch to debug console.

Unlike sbi_console_getchar(), this SBI call will block if there remain any pending characters to be transmitted or if the receiving terminal is not yet ready to receive the byte. However, if the console doesn’t exist at all, then the character is thrown away.

This SBI call returns 0 upon success or an implementation specific negative error code.

-- "RISC-V Supervisor Binary Interface Specification" v2.0-rc1

Console Putchar is a function that outputs the character passed as an argument to the debug console.

Try it out

Let's try your implementation. You should see Hello World! if it works:

$ ./run.sh
...

Hello World!

Tip

Life of Hello World:

When SBI is called, characters will be displayed as follows:

The kernel executes ecall instruction. The CPU jumps to the M-mode trap handler (mtvec register), which is set by OpenSBI during startup.

After saving registers, the trap handler written in C is called.

Based on the eid, the corresponding SBI processing function is called.

The device driver for the 8250 UART (Wikipedia) sends the character to QEMU.

QEMU's 8250 UART emulation implementation receives the character and sends it to the standard output.

The terminal emulator displays the character.

That is, by calling Console Putchar function is not a magic at all - it just uses the device driver implemented in OpenSBI!

printf function

We've successfully printed some characters. The next item is implementing printf function.

printf function takes a format string, and the values to be embedded in the output. For example, printf("1 + 2 = %d", 1 + 2) will display 1 + 2 = 3.

While printf bundled in the C standard library has a very rich set of features, let's start with a minimal version. Specifically, we'll implement a printf that supports three format specifiers: %d (decimal), %x (hexadecimal), and %s (string).

Since we'll use printf in applications too, let's create a new file common.c for code shared between the kernel and userland.

Here's the implementation of the printf function:

#include "common.h"

void putchar(char ch);

void printf(const char *fmt, ...) {
    va_list vargs;
    va_start(vargs, fmt);

    while (*fmt) {
        if (*fmt == '%') {
            fmt++; // Skip '%'
            switch (*fmt) { // Read the next character
                case '\0': // '%' at the end of the format string
                    putchar('%');
                    goto end;
                case '%': // Print '%'
                    putchar('%');
                    break;
                case 's': { // Print a NULL-terminated string.
                    const char *s = va_arg(vargs, const char *);
                    while (*s) {
                        putchar(*s);
                        s++;
                    }
                    break;
                }
                case 'd': { // Print an integer in decimal.
                    int value = va_arg(vargs, int);
                    if (value < 0) {
                        putchar('-');
                        value = -value;
                    }

                    int divisor = 1;
                    while (value / divisor > 9)
                        divisor *= 10;

                    while (divisor > 0) {
                        putchar('0' + value / divisor);
                        value %= divisor;
                        divisor /= 10;
                    }

                    break;
                }
                case 'x': { // Print an integer in hexadecimal.
                    int value = va_arg(vargs, int);
                    for (int i = 7; i >= 0; i--) {
                        int nibble = (value >> (i * 4)) & 0xf;
                        putchar("0123456789abcdef"[nibble]);
                    }
                }
            }
        } else {
            putchar(*fmt);
        }

        fmt++;
    }

end:
    va_end(vargs);
}

It's surprisingly concise, isn't it? It goes through the format string character by character, and if we encounter a %, we look at the next character and perform the corresponding formatting operation. Characters other than % are printed as is.

For decimal numbers, if value is negative, we first output a - and then get its absolute value. We then calculate the divisor to get the most significant digit and output the digits one by one.

For hexadecimal numbers, we output from the most significant nibble (a hexadecimal digit, 4 bits) to the least significant. Here, nibble is an integer from 0 to 15, so we use it as the index in string "0123456789abcdef" to get the corresponding character.

va_list and related macros are defined in the C standard library's <stdarg.h>. In this book, we use compiler builtins directly without relying on the standard library. Specifically, we'll define them in common.h as follows:

#pragma once

#define va_list  __builtin_va_list
#define va_start __builtin_va_start
#define va_end   __builtin_va_end
#define va_arg   __builtin_va_arg

void printf(const char *fmt, ...);

We're simply defining these as aliases for the versions with __builtin_ prefixed. They are builtin features provided by the compiler (clang) itself (Reference: clang documentation). The compiler will handle the rest appropriately, so we don't need to worry about it.

Now we've implemented printf. Let's add a "Hello World" from the kernel:

#include "kernel.h"
#include "common.h"

void kernel_main(void) {
    printf("\n\nHello %s\n", "World!");
    printf("1 + 2 = %d, %x\n", 1 + 2, 0x1234abcd);

    for (;;) {
        __asm__ __volatile__("wfi");
    }
}

Also, Add common.c to the compilation targets:

$CC $CFLAGS -Wl,-Tkernel.ld -Wl,-Map=kernel.map -o kernel.elf \
    kernel.c common.c

Now, let's try! You will see Hello World! and 1 + 2 = 3, 1234abcd as shown below:

$ ./run.sh

Hello World!
1 + 2 = 3, 1234abcd

The powerful ally "printf debugging" has joined your OS!

C Standard Library

In this chapter, let's implement basic types and memory operations, as well as string manipulation functions. In this book, for the purpose of learning, we'll create these from scratch instead of using C standard library.

Tip

The concepts introduced in this chapter are very common in C programming, so ChatGPT would provide solid answers. If you struggle with implementation or understanding any part, feel free to try asking it or ping me.

Basic types

First, let's define some basic types and convenient macros in common.h:

typedef int bool;
typedef unsigned char uint8_t;
typedef unsigned short uint16_t;
typedef unsigned int uint32_t;
typedef unsigned long long uint64_t;
typedef uint32_t size_t;
typedef uint32_t paddr_t;
typedef uint32_t vaddr_t;

#define true  1
#define false 0
#define NULL  ((void *) 0)
#define align_up(value, align)   __builtin_align_up(value, align)
#define is_aligned(value, align) __builtin_is_aligned(value, align)
#define offsetof(type, member)   __builtin_offsetof(type, member)
#define va_list  __builtin_va_list
#define va_start __builtin_va_start
#define va_end   __builtin_va_end
#define va_arg   __builtin_va_arg

void *memset(void *buf, char c, size_t n);
void *memcpy(void *dst, const void *src, size_t n);
char *strcpy(char *dst, const char *src);
int strcmp(const char *s1, const char *s2);
void printf(const char *fmt, ...);

Most of these are available in the standard library, but we've added a few useful ones:

paddr_t: A type representing physical memory addresses.

vaddr_t: A type representing virtual memory addresses. Equivalent to uintptr_t in the standard library.

align_up: Rounds up value to the nearest multiple of align. align must be a power of 2.

is_aligned: Checks if value is a multiple of align. align must be a power of 2.

offsetof: Returns the offset of a member within a structure (how many bytes from the start of the structure).

align_up and is_aligned are useful when dealing with memory alignment. For example, align_up(0x1234, 0x1000) returns 0x2000. Also, is_aligned(0x2000, 0x1000) returns true, but is_aligned(0x2f00, 0x1000) is false.

The functions starting with __builtin_ used in each macro are Clang-specific extensions (built-in functions). See Clang built-in functions and macros.

Tip

These macros can also be implemented in C without built-in functions. The pure C implementation of offsetof is particularly interesting ;)

Memory operations

Next, we implement the following memory operation functions.

The memcpy function copies n bytes from src to dst:

void *memcpy(void *dst, const void *src, size_t n) {
    uint8_t *d = (uint8_t *) dst;
    const uint8_t *s = (const uint8_t *) src;
    while (n--)
        *d++ = *s++;
    return dst;
}

The memset function fills the first n bytes of buf with c. This function has already been implemented in Chapter 4 for initializing the bss section. Let's move it from kernel.c to common.c:

void *memset(void *buf, char c, size_t n) {
    uint8_t *p = (uint8_t *) buf;
    while (n--)
        *p++ = c;
    return buf;
}

Tip

*p++ = c; does pointer dereferencing and pointer manipulation in a single statement. For clarity, it's equivalent to:

*p = c;    // Dereference the pointer
p = p + 1; // Advance the pointer after the assignment

This is an idiom in C.

String operations

Let's start with strcpy. This function copies the string from src to dst:

char *strcpy(char *dst, const char *src) {
    char *d = dst;
    while (*src)
        *d++ = *src++;
    *d = '\0';
    return dst;
}

Warning

The strcpy function continues copying even if src is longer than the memory area of dst. This can easily lead to bugs and vulnerabilities, so it's generally recommended to use alternative functions instead of strcpy. Never use it in production!

For simplicity, we'll use strcpy in this book, but if you have the capacity, try implementing and using an alternative function (strcpy_s) instead.

Next function is the strcmp function. It compares s1 and s2 and returns:

Condition | Result
s1 == s2 | 0
s1 > s2 | Positive value
s1 < s2 | Negative value

int strcmp(const char *s1, const char *s2) {
    while (*s1 && *s2) {
        if (*s1 != *s2)
            break;
        s1++;
        s2++;
    }

    return *(unsigned char *)s1 - *(unsigned char *)s2;
}

Tip

The casting to unsigned char * when comparing is done to conform to the POSIX specification.

The strcmp function is often used to check if two strings are identical. It's a bit counter-intuitive, but the strings are identical when !strcmp(s1, s2) is true (i.e., when the function returns zero):

if (!strcmp(s1, s2))
    printf("s1 == s2\n");
else
    printf("s1 != s2\n");

Kernel Panic

A kernel panic occurs when the kernel encounters an unrecoverable error, similar to the concept of panic in Go or Rust. Have you ever seen a blue screen on Windows? Let's implement the same concept in our kernel to handle fatal errors.

The following PANIC macro is the implementation of kernel panic:

#define PANIC(fmt, ...)                                                        \
    do {                                                                       \
        printf("PANIC: %s:%d: " fmt "\n", __FILE__, __LINE__, ##__VA_ARGS__);  \
        while (1) {}                                                           \
    } while (0)

It prints where the panic occurred, it enters an infinite loop to halt processing. We define it as a macro here. The reason for this is to correctly display the source file name (__FILE__) and line number (__LINE__). If we defined this as a function, __FILE__ and __LINE__ would show the file name and line number where PANIC is defined, not where it's called.

This macro also uses two idioms:

The first idiom is the do-while statement. Since it's while (0), this loop is only executed once. This is a common way to define macros consisting of multiple statements. Simply enclosing with { ...} can lead to unintended behavior when combined with statements like if (see this clear example). Also, note the backslash (\) at the end of each line. Although the macro is defined over multiple lines, newline characters are ignored when expanded.

The second idiom is ##__VA_ARGS__. This is a useful compiler extension for defining macros that accept a variable number of arguments (reference: GCC documentation). ## removes the preceding , when the variable arguments are empty. This allows compilation to succeed even when there's only one argument, like PANIC("booted!").

Let's try it

Let's try using PANIC. You can use it like printf:

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    PANIC("booted!");
    printf("unreachable here!\n");
}

Try in QEMU and confirm that the correct file name and line number are displayed, and that the processing after PANIC is not executed (i.e., "unreachable here!" is not displayed):

$ ./run.sh
PANIC: kernel.c:46: booted!

Blue screen in Windows and kernel panics in Linux are very scary, but in your own kernel, don't you think it is a nice feature to have? It's a "crash gracefully" mechanism, with a human-readable clue.

Exception

Exception is a CPU feature that allows the kernel to handle various events, such as invalid memory access (aka. page faults), illegal instructions, and system calls.

Exception is like a hardware-assisted try-catch mechanism in C++ or Java. Until CPU encounters the situation where kernel intervention is required, it continues to execute the program. The key difference from try-catch is that the kernel can resume the execution from the point where the exception occurred, as if nothing happened. Doesn't it sound like cool CPU feature?

Exception can also be triggered in kernel mode and mostly they are fatal kernel bugs. If QEMU resets unexpectedly or the kernel does not work as expected, it's likely that an exception occurred. I recommend to implement an exception handler early to crash gracefully with a kernel panic. It's similar to adding an unhandled rejection handler as the first step in JavaScript development.

Life of an exception

In RISC-V, an exception will be handled as follows:

CPU checks the medeleg register to determine which operation mode should handle the exception. In our case, OpenSBI has already configured to handle U-Mode/S-mode exceptions in S-Mode's handler.

CPU saves its state (registers) into various CSRs (see below).

The value of the stvec register is set to the program counter, jumping to the kernel's exception handler.

The exception handler saves general-purpose registers (i.e. the program state), and handles the exception.

Once it's done, the exception handler restores the saved execution state and calls the sret instruction to resume execution from the point where the exception occurred.

The CSRs updated in step 2 are mainly as follows. The kernel's exception determines necessary actions based on the CSRs:

Register Name | Content
scause | Type of exception. The kernel reads this to identify the type of exception.
stval | Additional information about the exception (e.g., memory address that caused the exception). Depends on the type of exception.
sepc | Program counter at the point where the exception occurred.
sstatus | Operation mode (U-Mode/S-Mode) when the exception has occurred.

Exception Handler

Now let's write your first exception handler! Here's the entry point of the exception handler to be registered in the stvec register:

__attribute__((naked))
__attribute__((aligned(4)))
void kernel_entry(void) {
    __asm__ __volatile__(
        "csrw sscratch, sp\n"
        "addi sp, sp, -4 * 31\n"
        "sw ra,  4 * 0(sp)\n"
        "sw gp,  4 * 1(sp)\n"
        "sw tp,  4 * 2(sp)\n"
        "sw t0,  4 * 3(sp)\n"
        "sw t1,  4 * 4(sp)\n"
        "sw t2,  4 * 5(sp)\n"
        "sw t3,  4 * 6(sp)\n"
        "sw t4,  4 * 7(sp)\n"
        "sw t5,  4 * 8(sp)\n"
        "sw t6,  4 * 9(sp)\n"
        "sw a0,  4 * 10(sp)\n"
        "sw a1,  4 * 11(sp)\n"
        "sw a2,  4 * 12(sp)\n"
        "sw a3,  4 * 13(sp)\n"
        "sw a4,  4 * 14(sp)\n"
        "sw a5,  4 * 15(sp)\n"
        "sw a6,  4 * 16(sp)\n"
        "sw a7,  4 * 17(sp)\n"
        "sw s0,  4 * 18(sp)\n"
        "sw s1,  4 * 19(sp)\n"
        "sw s2,  4 * 20(sp)\n"
        "sw s3,  4 * 21(sp)\n"
        "sw s4,  4 * 22(sp)\n"
        "sw s5,  4 * 23(sp)\n"
        "sw s6,  4 * 24(sp)\n"
        "sw s7,  4 * 25(sp)\n"
        "sw s8,  4 * 26(sp)\n"
        "sw s9,  4 * 27(sp)\n"
        "sw s10, 4 * 28(sp)\n"
        "sw s11, 4 * 29(sp)\n"

        "csrr a0, sscratch\n"
        "sw a0, 4 * 30(sp)\n"

        "mv a0, sp\n"
        "call handle_trap\n"

        "lw ra,  4 * 0(sp)\n"
        "lw gp,  4 * 1(sp)\n"
        "lw tp,  4 * 2(sp)\n"
        "lw t0,  4 * 3(sp)\n"
        "lw t1,  4 * 4(sp)\n"
        "lw t2,  4 * 5(sp)\n"
        "lw t3,  4 * 6(sp)\n"
        "lw t4,  4 * 7(sp)\n"
        "lw t5,  4 * 8(sp)\n"
        "lw t6,  4 * 9(sp)\n"
        "lw a0,  4 * 10(sp)\n"
        "lw a1,  4 * 11(sp)\n"
        "lw a2,  4 * 12(sp)\n"
        "lw a3,  4 * 13(sp)\n"
        "lw a4,  4 * 14(sp)\n"
        "lw a5,  4 * 15(sp)\n"
        "lw a6,  4 * 16(sp)\n"
        "lw a7,  4 * 17(sp)\n"
        "lw s0,  4 * 18(sp)\n"
        "lw s1,  4 * 19(sp)\n"
        "lw s2,  4 * 20(sp)\n"
        "lw s3,  4 * 21(sp)\n"
        "lw s4,  4 * 22(sp)\n"
        "lw s5,  4 * 23(sp)\n"
        "lw s6,  4 * 24(sp)\n"
        "lw s7,  4 * 25(sp)\n"
        "lw s8,  4 * 26(sp)\n"
        "lw s9,  4 * 27(sp)\n"
        "lw s10, 4 * 28(sp)\n"
        "lw s11, 4 * 29(sp)\n"
        "lw sp,  4 * 30(sp)\n"
        "sret\n"
    );
}

Here are some key points:

sscratch register is used as a temporary storage to save the stack pointer at the time of exception occurrence, which is later restored.

Floating-point registers are not used within the kernel, and thus there's no need to save them here. Generally, they are saved and restored during thread switching.

The stack pointer is set in the a0 register, and the handle_trap function is called. At this point, the address pointed to by the stack pointer contains register values stored in the same structure as the trap_frame structure described later.

Adding __attribute__((aligned(4))) aligns the function's starting address to a 4-byte boundary. This is because the stvec register not only holds the address of the exception handler but also has flags representing the mode in its lower 2 bits.

Note

The entry point of exception handlers is one of most critical and error-prone parts of the kernel. Reading the code closely, you'll notice that original values of general-purpose registers are saved onto the stack, even sp by using sscratch.

If you accidentally overwrite a0 register, it can lead to hard-to-debug problems like "local variable values change for no apparent reason". Save the program state perfectly not to spend your precious Saturday night debugging!

In the entry point, the following handle_trap function is called to handle the exception in our favorite C language:

void handle_trap(struct trap_frame *f) {
    uint32_t scause = READ_CSR(scause);
    uint32_t stval = READ_CSR(stval);
    uint32_t user_pc = READ_CSR(sepc);

    PANIC("unexpected trap scause=%x, stval=%x, sepc=%x\n", scause, stval, user_pc);
}

It reads why the exception has occurred, and triggers a kernel panic for debugging purposes.

Let's define the various macros used here in kernel.h:

#include "common.h"

struct trap_frame {
    uint32_t ra;
    uint32_t gp;
    uint32_t tp;
    uint32_t t0;
    uint32_t t1;
    uint32_t t2;
    uint32_t t3;
    uint32_t t4;
    uint32_t t5;
    uint32_t t6;
    uint32_t a0;
    uint32_t a1;
    uint32_t a2;
    uint32_t a3;
    uint32_t a4;
    uint32_t a5;
    uint32_t a6;
    uint32_t a7;
    uint32_t s0;
    uint32_t s1;
    uint32_t s2;
    uint32_t s3;
    uint32_t s4;
    uint32_t s5;
    uint32_t s6;
    uint32_t s7;
    uint32_t s8;
    uint32_t s9;
    uint32_t s10;
    uint32_t s11;
    uint32_t sp;
} __attribute__((packed));

#define READ_CSR(reg)                                                          \
    ({                                                                         \
        unsigned long __tmp;                                                   \
        __asm__ __volatile__("csrr %0, " #reg : "=r"(__tmp));                  \
        __tmp;                                                                 \
    })

#define WRITE_CSR(reg, value)                                                  \
    do {                                                                       \
        uint32_t __tmp = (value);                                              \
        __asm__ __volatile__("csrw " #reg ", %0" ::"r"(__tmp));                \
    } while (0)

The trap_frame struct represents the program state saved in kernel_entry. READ_CSR and WRITE_CSR macros are convenient macros for reading and writing CSR registers.

The last thing we need to do is to tell the CPU where the exception handler is located. It's done by setting the stvec register in the kernel_main function:

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    WRITE_CSR(stvec, (uint32_t) kernel_entry); // new
    __asm__ __volatile__("unimp"); // new

In addition to setting the stvec register, it executes unimp instruction. it's a pseudo instruction which triggers an illegal instruction exception.

Note

unimp is a "pseudo" instruction.

According to RISC-V Assembly Programmer's Manual, the assembler translates unimp to the following instruction:

csrrw x0, cycle, x0

This reads and writes the cycle register into x0. Since cycle is a read-only register, CPU determines that the instruction is invalid and triggers an illegal instruction exception.

Let's try it

Let's try running it and confirm that the exception handler is called:

$ ./run.sh
Hello World!
PANIC: kernel.c:47: unexpected trap scause=00000002, stval=ffffff84, sepc=8020015e

According to the specification, when the value of scause is 2, it indicates an "Illegal instruction," meaning that program tried to execute an invalid instruction. This is precisely the expected behavior of the unimp instruction!

Let's also check where the value of sepc is pointing. If it's pointing to the line where the unimp instruction is called, everything is working correctly:

$ llvm-addr2line -e kernel.elf 8020015e
/Users/seiya/os-from-scratch/kernel.c:129

Memory Allocation

In this chapter, we'll implement a simple memory allocator.

Revisiting the linker script

Before implementing a memory allocator, let's define the memory regions to be managed by the allocator:

    . = ALIGN(4);
    . += 128 * 1024; /* 128KB */
    __stack_top = .;

    . = ALIGN(4096);
    __free_ram = .;
    . += 64 * 1024 * 1024; /* 64MB */
    __free_ram_end = .;
}

This adds two new symbols: __free_ram and __free_ram_end. This defines a memory area after the stack space. The size of the space (64MB) is an arbitrary value and . = ALIGN(4096) ensures that it's aligned to a 4KB boundary.

By defining this in the linker script instead of hardcoding addresses, the linker can determine the position to avoid overlapping with the kernel's static data.

Tip

Practical operating systems on x86-64 determine available memory regions by obtaining information from hardware at boot time (for example, UEFI's GetMemoryMap).

The world's simplest memory allocation algorithm

Let's implement a function to allocate memory dynamically. Instead of allocating in bytes like malloc, it allocates in a larger unit called "pages". 1 page is typically 4KB (4096 bytes).

Tip

4KB = 4096 = 0x1000 (hexadecimal). Thus, page-aligned addresses look nicely aligned in hexadecimal.

The following alloc_pages function dynamically allocates n pages of memory and returns the starting address:

extern char __free_ram[], __free_ram_end[];

paddr_t alloc_pages(uint32_t n) {
    static paddr_t next_paddr = (paddr_t) __free_ram;
    paddr_t paddr = next_paddr;
    next_paddr += n * PAGE_SIZE;

    if (next_paddr > (paddr_t) __free_ram_end)
        PANIC("out of memory");

    memset((void *) paddr, 0, n * PAGE_SIZE);
    return paddr;
}

PAGE_SIZE represents the size of one page. Define it in common.h:

#define PAGE_SIZE 4096

You will find the following key points:

next_paddr is defined as a static variable. This means, unlike local variables, its value is retained between function calls. That is, it behaves like a global variable.

next_paddr points to the start address of the "next area to be allocated" (free area). When allocating, next_paddr is advanced by the size being allocated.

next_paddr initially holds the address of __free_ram. This means memory is allocated sequentially starting from __free_ram.

__free_ram is placed on a 4KB boundary due to ALIGN(4096) in the linker script. Therefore, the alloc_pages function always returns an address aligned to 4KB.

If it tries to allocate beyond __free_ram_end, in other words, if it runs out of memory, a kernel panic occurs.

The memset function ensures that the allocated memory area is always filled with zeroes. This is to avoid hard-to-debug issues caused by uninitialized memory.

Isn't it simple? However, there is a big problem with this memory allocation algorithm: allocated memory cannot be freed! That said, it's good enough for our simple hobby OS.

Tip

This algorithm is known as Bump allocator or Linear allocator, and it's actually used in scenarios where deallocation is not necessary. It's an attractive allocation algorithm that can be implemented in just a few lines and is very fast.

When implementing deallocation, it's common to use a bitmap-based algorithm or use an algorithm called the buddy system.

Let's try memory allocation

Let's test the memory allocation function we've implemented. Add some code to kernel_main:

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    paddr_t paddr0 = alloc_pages(2);
    paddr_t paddr1 = alloc_pages(1);
    printf("alloc_pages test: paddr0=%x\n", paddr0);
    printf("alloc_pages test: paddr1=%x\n", paddr1);

    PANIC("booted!");
}

Verify that the first address (paddr0) matches the address of __free_ram, and that the next address (paddr1) matches an address 8KB after paddr0:

$ ./run.sh
Hello World!
alloc_pages test: paddr0=80221000
alloc_pages test: paddr1=80223000

$ llvm-nm kernel.elf | grep __free_ram
80221000 R __free_ram
84221000 R __free_ram_end

Process

A process is an instance of an application. Each process has its own independent execution context and resources, such as a virtual address space.

Note

Practical operating systems provide the execution context as a separate concept called a "thread". For simplicity, in this book we'll treat each process as having a single thread.

Process control block

The following process structure defines a process object. It's also known as "Process Control Block (PCB)".

#define PROCS_MAX 8       // Maximum number of processes

#define PROC_UNUSED   0   // Unused process control structure
#define PROC_RUNNABLE 1   // Runnable process

struct process {
    int pid;             // Process ID
    int state;           // Process state: PROC_UNUSED or PROC_RUNNABLE
    vaddr_t sp;          // Stack pointer
    uint8_t stack[8192]; // Kernel stack
};

The kernel stack contains saved CPU registers, return addresses (where it was called from), and local variables. By preparing a kernel stack for each process, we can implement context switching by saving and restoring CPU registers, and switching the stack pointer.

Tip

There is another approach called "single kernel stack". Instead of having a kernel stack for each process (or thread), there's only single stack per CPU. seL4 adopts this method.

This "where to store the program's context" issue is also a topic discussed in async runtimes of programming languages like Go and Rust. Try searching for "stackless async" if you're interested.

Context switch

Switching the process execution context is called "context switching". The following switch_context function is the implementation of context switching:

__attribute__((naked)) void switch_context(uint32_t *prev_sp,
                                           uint32_t *next_sp) {
    __asm__ __volatile__(
        // Save callee-saved registers onto the current process's stack.
        "addi sp, sp, -13 * 4\n" // Allocate stack space for 13 4-byte registers
        "sw ra,  0  * 4(sp)\n"   // Save callee-saved registers only
        "sw s0,  1  * 4(sp)\n"
        "sw s1,  2  * 4(sp)\n"
        "sw s2,  3  * 4(sp)\n"
        "sw s3,  4  * 4(sp)\n"
        "sw s4,  5  * 4(sp)\n"
        "sw s5,  6  * 4(sp)\n"
        "sw s6,  7  * 4(sp)\n"
        "sw s7,  8  * 4(sp)\n"
        "sw s8,  9  * 4(sp)\n"
        "sw s9,  10 * 4(sp)\n"
        "sw s10, 11 * 4(sp)\n"
        "sw s11, 12 * 4(sp)\n"

        // Switch the stack pointer.
        "sw sp, (a0)\n"         // *prev_sp = sp;
        "lw sp, (a1)\n"         // Switch stack pointer (sp) here

        // Restore callee-saved registers from the next process's stack.
        "lw ra,  0  * 4(sp)\n"  // Restore callee-saved registers only
        "lw s0,  1  * 4(sp)\n"
        "lw s1,  2  * 4(sp)\n"
        "lw s2,  3  * 4(sp)\n"
        "lw s3,  4  * 4(sp)\n"
        "lw s4,  5  * 4(sp)\n"
        "lw s5,  6  * 4(sp)\n"
        "lw s6,  7  * 4(sp)\n"
        "lw s7,  8  * 4(sp)\n"
        "lw s8,  9  * 4(sp)\n"
        "lw s9,  10 * 4(sp)\n"
        "lw s10, 11 * 4(sp)\n"
        "lw s11, 12 * 4(sp)\n"
        "addi sp, sp, 13 * 4\n"  // We've popped 13 4-byte registers from the stack
        "ret\n"
    );
}

switch_context saves the callee-saved registers onto the stack, switches the stack pointer, and then restores the callee-saved registers from the stack. In other words, the execution context is stored as temporary local variables on the stack. Alternatively, you could save the context in struct process, but this stack-based approach is beautifully simple, isn't it?

Callee-saved registers are registers that a called function must restore before returning. In RISC-V, s0 to s11 are callee-saved registers. Other registers like a0 are caller-saved registers, and already saved on the stack by the caller. This is why switch_context handles only part of registers.

The naked attribute tells the compiler not to generate any other code than the inline assembly. It should work without this attribute, but it's a good practice to use it to avoid unintended behavior especially when you modify the stack pointer manually.

Tip

Callee/Caller saved registers are defined in Calling Convention. Compilers generate code based on this convention.

Next, let's implement the process initialization function, create_process. It takes the entry point as a parameter, and returns a pointer to the created process struct:

struct process procs[PROCS_MAX]; // All process control structures.

struct process *create_process(uint32_t pc) {
    // Find an unused process control structure.
    struct process *proc = NULL;
    int i;
    for (i = 0; i < PROCS_MAX; i++) {
        if (procs[i].state == PROC_UNUSED) {
            proc = &procs[i];
            break;
        }
    }

    if (!proc)
        PANIC("no free process slots");

    // Stack callee-saved registers. These register values will be restored in
    // the first context switch in switch_context.
    uint32_t *sp = (uint32_t *) &proc->stack[sizeof(proc->stack)];
    *--sp = 0;                      // s11
    *--sp = 0;                      // s10
    *--sp = 0;                      // s9
    *--sp = 0;                      // s8
    *--sp = 0;                      // s7
    *--sp = 0;                      // s6
    *--sp = 0;                      // s5
    *--sp = 0;                      // s4
    *--sp = 0;                      // s3
    *--sp = 0;                      // s2
    *--sp = 0;                      // s1
    *--sp = 0;                      // s0
    *--sp = (uint32_t) pc;          // ra

    // Initialize fields.
    proc->pid = i + 1;
    proc->state = PROC_RUNNABLE;
    proc->sp = (uint32_t) sp;
    return proc;
}

Testing context switch

We have implemented the most basic function of processes - concurrent execution of multiple programs. Let's create two processes:

void delay(void) {
    for (int i = 0; i < 30000000; i++)
        __asm__ __volatile__("nop"); // do nothing
}

struct process *proc_a;
struct process *proc_b;

void proc_a_entry(void) {
    printf("starting process A\n");
    while (1) {
        putchar('A');
        switch_context(&proc_a->sp, &proc_b->sp);
        delay();
    }
}

void proc_b_entry(void) {
    printf("starting process B\n");
    while (1) {
        putchar('B');
        switch_context(&proc_b->sp, &proc_a->sp);
        delay();
    }
}

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    WRITE_CSR(stvec, (uint32_t) kernel_entry);

    proc_a = create_process((uint32_t) proc_a_entry);
    proc_b = create_process((uint32_t) proc_b_entry);
    proc_a_entry();

    PANIC("unreachable here!");
}

The proc_a_entry function and proc_b_entry function are the entry points for Process A and Process B respectively. After displaying a single character using the putchar function, they switch context to the other process using the switch_context function.

delay function implements a busy wait to prevent the character output from becoming too fast, which would make your terminal unresponsive. nop instruction is a "do nothing" instruction. It is added to prevent compiler optimization from removing the loop.

Now, let's try! The startup messages will be displayed once each, and then "ABABAB..." lasts forever:

$ ./run.sh

starting process A
Astarting process B
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABAQE

Scheduler

In the previous experiment, we directly called the switch_context function to specify the "next process to execute". However, this method becomes complicated when determining which process to switch to next as the number of processes increases. To solve the issue, let's implement a "scheduler", a kernel program which decides the next process.

The following yield function is the implementation of the scheduler:

Tip

The word "yield" is often used as the name for an API which allows giving up the CPU to another process voluntarily.

struct process *current_proc; // Currently running process
struct process *idle_proc;    // Idle process

void yield(void) {
    // Search for a runnable process
    struct process *next = idle_proc;
    for (int i = 0; i < PROCS_MAX; i++) {
        struct process *proc = &procs[(current_proc->pid + i) % PROCS_MAX];
        if (proc->state == PROC_RUNNABLE && proc->pid > 0) {
            next = proc;
            break;
        }
    }

    // If there's no runnable process other than the current one, return and continue processing
    if (next == current_proc)
        return;

    // Context switch
    struct process *prev = current_proc;
    current_proc = next;
    switch_context(&prev->sp, &next->sp);
}

Here, we introduce two global variables. current_proc points to the currently running process. idle_proc refers to the idle process, which is "the process to run when there are no runnable processes". The idle_proc is created at startup as a process with process ID -1, as shown below:

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    printf("\n\n");

    WRITE_CSR(stvec, (uint32_t) kernel_entry);

    idle_proc = create_process((uint32_t) NULL);
    idle_proc->pid = -1; // idle
    current_proc = idle_proc;

    proc_a = create_process((uint32_t) proc_a_entry);
    proc_b = create_process((uint32_t) proc_b_entry);

    yield();
    PANIC("switched to idle process");
}

The key point of this initialization process is current_proc = idle_proc. This ensures that the execution context of the boot process is saved and restored as that of the idle process. During the first call to the yield function, it switches from the idle process to process A, and when switching back to the idle process, it behaves as if returning from this yield function call.

Lastly, modify proc_a_entry and proc_b_entry as follows to call the yield function instead of directly calling the switch_context function:

void proc_a_entry(void) {
    printf("starting process A\n");
    while (1) {
        putchar('A');
        yield();
    }
}

void proc_b_entry(void) {
    printf("starting process B\n");
    while (1) {
        putchar('B');
        yield();
    }
}

If "A" and "B" are printed as before, it works perfectly!

Changes in the exception handler

In the exception handler, it saves the execution state onto the stack. However, since we now use separate kernel stacks for each process, we need to update it slightly.

First, set the initial value of the kernel stack for the currently executing process in the sscratch register during process switching.

void yield(void) {
    /* omitted */

    __asm__ __volatile__(
        "csrw sscratch, %[sscratch]\n"
        :
        : [sscratch] "r" ((uint32_t) &next->stack[sizeof(next->stack)])
    );

    // Context switch
    struct process *prev = current_proc;
    current_proc = next;
    switch_context(&prev->sp, &next->sp);
}

Since the stack pointer extends towards lower addresses, we set the address at the sizeof(next->stack)th byte as the initial value of the kernel stack.

The modifications to the exception handler are as follows:

void kernel_entry(void) {
    __asm__ __volatile__(
        // Retrieve the kernel stack of the running process from sscratch.
        "csrrw sp, sscratch, sp\n"

        "addi sp, sp, -4 * 31\n"
        "sw ra,  4 * 0(sp)\n"
        "sw gp,  4 * 1(sp)\n"
        "sw tp,  4 * 2(sp)\n"
        "sw t0,  4 * 3(sp)\n"
        "sw t1,  4 * 4(sp)\n"
        "sw t2,  4 * 5(sp)\n"
        "sw t3,  4 * 6(sp)\n"
        "sw t4,  4 * 7(sp)\n"
        "sw t5,  4 * 8(sp)\n"
        "sw t6,  4 * 9(sp)\n"
        "sw a0,  4 * 10(sp)\n"
        "sw a1,  4 * 11(sp)\n"
        "sw a2,  4 * 12(sp)\n"
        "sw a3,  4 * 13(sp)\n"
        "sw a4,  4 * 14(sp)\n"
        "sw a5,  4 * 15(sp)\n"
        "sw a6,  4 * 16(sp)\n"
        "sw a7,  4 * 17(sp)\n"
        "sw s0,  4 * 18(sp)\n"
        "sw s1,  4 * 19(sp)\n"
        "sw s2,  4 * 20(sp)\n"
        "sw s3,  4 * 21(sp)\n"
        "sw s4,  4 * 22(sp)\n"
        "sw s5,  4 * 23(sp)\n"
        "sw s6,  4 * 24(sp)\n"
        "sw s7,  4 * 25(sp)\n"
        "sw s8,  4 * 26(sp)\n"
        "sw s9,  4 * 27(sp)\n"
        "sw s10, 4 * 28(sp)\n"
        "sw s11, 4 * 29(sp)\n"

        // Retrieve and save the sp at the time of exception.
        "csrr a0, sscratch\n"
        "sw a0,  4 * 30(sp)\n"

        // Reset the kernel stack.
        "addi a0, sp, 4 * 31\n"
        "csrw sscratch, a0\n"

        "mv a0, sp\n"
        "call handle_trap\n"

The first csrrw instruction is a swap operation in short:

tmp = sp;
sp = sscratch;
sscratch = tmp;

Thus, sp now points to the kernel (not user) stack of the currently running process. Also, sscratch now holds the original value of sp (user stack) at the time of the exception.

After saving other registers onto the kernel stack, We restore the original sp value from sscratch and save it onto the kernel stack. Then, calculate the initial value of sscratch and restore it.

The key point here is that each process has its own independent kernel stack. By switching the contents of sscratch during context switching, we can resume the execution of the process from the point where it was interrupted, as if nothing had happened.

Tip

We've implemented the context switching mechanism for the "kernel" stack. The stack used by applications (so-called user stack) will be allocated separately from the kernel stack. This will be implemented in later chapters.

Appendix: Why do we reset the stack pointer?

In the previous section, you might have wondered why we need to switch to the kernel stack by tweaking sscratch.

This is because we must not trust the stack pointer at the time of exception. In the exception handler, we need to consider the following three patterns:

An exception occurred in kernel mode.

An exception occurred in kernel mode, when handling another exception (nested exception).

An exception occurred in user mode.

In case (1), there's generally no problem even if we don't reset the stack pointer. In case (2), we would overwrite the saved area, but our implementation triggers a kernel panic on nested exceptions, so it's OK.

The problem is with case (3). In this case, sp points to the "user (application) stack area". If we implement it to use (trust) sp as is, it could lead to a vulnerability that crashes the kernel.

Let's experiment with this by running the following application after completing all the implementations up to Chapter 17 in this book:

// An example of applications
#include "user.h"

void main(void) {
    __asm__ __volatile__(
        "li sp, 0xdeadbeef\n"  // Set an invalid address to sp
        "unimp"                // Trigger an exception
    );
}

If we run this without applying the modifications from this chapter (i.e. restoring the kernel stack from sscratch), the kernel hangs without displaying anything, and you'll see the following output in QEMU's log:

epc:0x0100004e, tval:0x00000000, desc=illegal_instruction <- unimp triggers the trap handler
epc:0x802009dc, tval:0xdeadbe73, desc=store_page_fault <- an aborted write to the stack  (0xdeadbeef)
epc:0x802009dc, tval:0xdeadbdf7, desc=store_page_fault <- an aborted write to the stack  (0xdeadbeef) (2)
epc:0x802009dc, tval:0xdeadbd7b, desc=store_page_fault <- an aborted write to the stack  (0xdeadbeef) (3)
epc:0x802009dc, tval:0xdeadbcff, desc=store_page_fault <- an aborted write to the stack  (0xdeadbeef) (4)
...

First, an invalid instruction exception occurs with the unimp pseudo-instruction, transitioning to the kernel's trap handler. However, because the stack pointer points to an unmapped address (0xdeadbeef), an exception occurs when trying to save registers, jumping back to the beginning of the trap handler. This becomes an infinite loop, causing the kernel to hang. To prevent this, we need to retrieve a trusted stack area from sscratch.

Another solution is to have multiple exception handlers. In the RISC-V version of xv6 (a famous educational UNIX-like OS), there are separate exception handlers for cases (1) and (2) (kernelvec) and for case (3) (uservec). In the former case, it inherits the stack pointer at the time of the exception, and in the latter case, it retrieves a separate kernel stack. The trap handler is switched when entering and exiting the kernel.

Tip

In Fuchsia, an OS developed by Google, there was a case where an API allowing arbitrary program counter values to be set from the user became a vulnerability. Not trusting input from users (applications) is an extremely important habit in kernel development.

Next Steps

We have now achieved the ability to run multiple processes concurrently, realizing a multi-tasking OS.

However, as it stands, processes can freely read and write to the kernel's memory space. It's super insecure! In the coming chapters, we'll look at how to safely run applications, in other words, how to isolate the kernel and applications.

Page Table

Memory management and virtual addressing

When a program accesses memory, CPU translates the specified address (virtual address) into a physical address. The table that maps virtual addresses to physical addresses is called a page table. By switching page tables, the same virtual address can point to different physical addresses. This allows isolation of memory spaces (virtual address spaces) and separation of kernel and application memory areas, enhancing system security.

In this chapter, we'll implement the hardware-based memory isolation mechanism.

Structure of virtual address

In this book, we use one of RISC-V's paging mechanisms called Sv32, which uses a two-level page table. The 32-bit virtual address is divided into a first-level page table index (VPN[1]), a second-level index (VPN[0]), and a page offset.

Try RISC-V Sv-32 Virtual Address Breakdown to see how virtual addresses are broken down into page table indices and offsets.

Here are some examples:

Virtual Address | VPN[1] (10 bits) | VPN[0] (10 bits) | Offset (12 bits)
0x1000_0000 | 0x040 | 0x000 | 0x000
0x1000_0000 | 0x040 | 0x000 | 0x000
0x1000_1000 | 0x040 | 0x001 | 0x000
0x1000_f000 | 0x040 | 0x00f | 0x000
0x2000_f0ab | 0x080 | 0x00f | 0x0ab
0x2000_f012 | 0x080 | 0x00f | 0x012
0x2000_f034 | 0x080 | 0x00f | 0x045

Tip

From the examples above, we can see the following characteristics of the indices:

Changing the middle bits (VPN[0]) doesn't affect the first-level index. This means page table entries for nearby addresses are concentrated in the same first-level page table.

Changing the lower bits doesn't affect either VPN[1] or VPN[0]. This means addresses within the same 4KB page are in the same page table entry.

This structure utilizes the principle of locality, allowing for smaller page table sizes and more effective use of the Translation Lookaside Buffer (TLB).

When accessing memory, CPU calculates VPN[1] and VPN[0] to identify the corresponding page table entry, reads the mapped base physical address, and adds offset to get the final physical address.

Constructing the page table

Let's construct a page table in Sv32. First, we'll define some macros. SATP_SV32 is a single bit in the satp register which indicates "enable paging in Sv32 mode", and PAGE_* are flags to be set in page table entries.

#define SATP_SV32 (1u << 31)
#define PAGE_V    (1 << 0)   // "Valid" bit (entry is enabled)
#define PAGE_R    (1 << 1)   // Readable
#define PAGE_W    (1 << 2)   // Writable
#define PAGE_X    (1 << 3)   // Executable
#define PAGE_U    (1 << 4)   // User (accessible in user mode)

Mapping pages

The following map_page function takes the first-level page table (table1), the virtual address (vaddr), the physical address (paddr), and page table entry flags (flags):

void map_page(uint32_t *table1, uint32_t vaddr, paddr_t paddr, uint32_t flags) {
    if (!is_aligned(vaddr, PAGE_SIZE))
        PANIC("unaligned vaddr %x", vaddr);

    if (!is_aligned(paddr, PAGE_SIZE))
        PANIC("unaligned paddr %x", paddr);

    uint32_t vpn1 = (vaddr >> 22) & 0x3ff;
    if ((table1[vpn1] & PAGE_V) == 0) {
        // Create the non-existent 2nd level page table.
        uint32_t pt_paddr = alloc_pages(1);
        table1[vpn1] = ((pt_paddr / PAGE_SIZE) << 10) | PAGE_V;
    }

    // Set the 2nd level page table entry to map the physical page.
    uint32_t vpn0 = (vaddr >> 12) & 0x3ff;
    uint32_t *table0 = (uint32_t *) ((table1[vpn1] >> 10) * PAGE_SIZE);
    table0[vpn0] = ((paddr / PAGE_SIZE) << 10) | flags | PAGE_V;
}

This function prepares the second-level page table, and fills the page table entry in the second level.

It divides paddr by PAGE_SIZE because the entry should contain the physical page number, not the physical address itself. Don't confuse the two!

Mapping kernel memory area

The page table must be configured not only for applications (user space), but also for the kernel.

In this book, the kernel memory mapping is configured so that the kernel's virtual addresses match the physical addresses (i.e. vaddr == paddr). This allows the same code to continue running even after enabling paging.

First, let's modify the kernel's linker script to define the starting address used by the kernel (__kernel_base):

ENTRY(boot)

SECTIONS {
    . = 0x80200000;
    __kernel_base = .;

Warning

Define __kernel_base after the line . = 0x80200000. If the order is reversed, the value of __kernel_base will be zero.

Next, add the page table to the process struct. This will be a pointer to the first-level page table.

struct process {
    int pid;
    int state;
    vaddr_t sp;
    uint32_t *page_table;
    uint8_t stack[8192];
};

Lastly, map the kernel pages in the create_process function. The kernel pages span from __kernel_base to __free_ram_end. This approach ensures that the kernel can always access both statically allocated areas (like .text), and dynamically allocated areas managed by alloc_pages:

extern char __kernel_base[];

struct process *create_process(uint32_t pc) {
    /* omitted */

    // Map kernel pages.
    uint32_t *page_table = (uint32_t *) alloc_pages(1);
    for (paddr_t paddr = (paddr_t) __kernel_base;
         paddr < (paddr_t) __free_ram_end; paddr += PAGE_SIZE)
        map_page(page_table, paddr, paddr, PAGE_R | PAGE_W | PAGE_X);

    proc->pid = i + 1;
    proc->state = PROC_RUNNABLE;
    proc->sp = (uint32_t) sp;
    proc->page_table = page_table;
    return proc;
}

Switching page tables

Let's switch the process's page table when context switching:

void yield(void) {
    /* omitted */

    __asm__ __volatile__(
        "sfence.vma\n"
        "csrw satp, %[satp]\n"
        "sfence.vma\n"
        "csrw sscratch, %[sscratch]\n"
        :
        // Don't forget the trailing comma!
        : [satp] "r" (SATP_SV32 | ((uint32_t) next->page_table / PAGE_SIZE)),
          [sscratch] "r" ((uint32_t) &next->stack[sizeof(next->stack)])
    );

    switch_context(&prev->sp, &next->sp);
}

We can switch page tables by specifying the first-level page table in satp. Note that we divide by PAGE_SIZE because it's the physical page number.

sfence.vma instructions added before and after setting the page table serve two purposes:

To ensure that changes to the page table are properly completed (similar to a memory fence).

To clear the cache of page table entries (TLB).

Tip

When the kernel starts, paging is disabled by default (the satp register is not set). Virtual addresses behave as if they match physical addresses.

Testing paging

let's try it and see how it works!

$ ./run.sh

starting process A
Astarting process B
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABAB

The output is exactly the same as in the previous chapter (context switching). There's no visible change even after enabling paging. To check if we've set up the page tables correctly, let's inspect it with QEMU monitor!

Examining page table contents

Let's look at how the virtual addresses around 0x80000000 are mapped. If set up correctly, they should be mapped so that (virtual address) == (physical address).

QEMU 8.0.2 monitor - type 'help' for more information
(qemu) stop
(qemu) info registers
 ...
 satp     80080253
 ...

You can see that satp is 0x80080253. According to the specification (RISC-V Sv32 mode), interpreting this value gives us the first-level page table's starting physical address: (0x80080253 & 0x3fffff) * 4096 = 0x80253000.

Next, let's inspect the contents of the first-level page table. We want to know the second-level page table corresponding to the virtual address 0x80000000. QEMU provides commands to display memory contents (memory dump). xp command dumps memory at the specified physical address. Dump the 512th entry because 0x80000000 >> 22 = 512. Since each entry is 4 bytes, we multiply by 4:

(qemu) xp /x 0x80253000+512*4
0000000080253800: 0x20095001

The first column shows the physical address, and the subsequent columns show the memory values. We can see that some non-zero values are set. The /x option specifies hexadecimal display. Adding a number before x (e.g., /1024x) specifies the number of entries to display.

Tip

Using the x command instead of xp allows you to view the memory dump for a specified virtual address. This is useful when examining user space (application) memory, where virtual addresses do not match physical addresses, unlike in our kernel space.

According to the specification, the second-level page table is located at (0x20095000 >> 10) * 4096 = 0x80254000. Let's dump the entire second-level table (1024 entries):

(qemu) xp /1024x 0x80254000
0000000080254000: 0x00000000 0x00000000 0x00000000 0x00000000
0000000080254010: 0x00000000 0x00000000 0x00000000 0x00000000
0000000080254020: 0x00000000 0x00000000 0x00000000 0x00000000
0000000080254030: 0x00000000 0x00000000 0x00000000 0x00000000
...
00000000802547f0: 0x00000000 0x00000000 0x00000000 0x00000000
0000000080254800: 0x2008004f 0x2008040f 0x2008080f 0x20080c0f
0000000080254810: 0x2008100f 0x2008140f 0x2008180f 0x20081c0f
0000000080254820: 0x2008200f 0x2008240f 0x2008280f 0x20082c0f
0000000080254830: 0x2008300f 0x2008340f 0x2008380f 0x20083c0f
0000000080254840: 0x200840cf 0x2008440f 0x2008484f 0x20084c0f
0000000080254850: 0x200850cf 0x2008540f 0x200858cf 0x20085c0f
0000000080254860: 0x2008600f 0x2008640f 0x2008680f 0x20086c0f
0000000080254870: 0x2008700f 0x2008740f 0x2008780f 0x20087c0f
0000000080254880: 0x200880cf 0x2008840f 0x2008880f 0x20088c0f
...

The initial entries are filled with zeros, but values start appearing from the 512th entry (254800). This is because __kernel_base is 0x80200000, and VPN[1] is 0x200.

We've manually read memory dumps up, but QEMU actually provides a command that displays the current page table mappings in human-readable format. If you want to do a final check on whether the mapping is correct, you can use the info mem command:

(qemu) info mem
vaddr    paddr            size     attr
-------- ---------------- -------- -------
80200000 0000000080200000 00001000 rwx--a-
80201000 0000000080201000 0000f000 rwx----
80210000 0000000080210000 00001000 rwx--ad
80211000 0000000080211000 00001000 rwx----
80212000 0000000080212000 00001000 rwx--a-
80213000 0000000080213000 00001000 rwx----
80214000 0000000080214000 00001000 rwx--ad
80215000 0000000080215000 00001000 rwx----
80216000 0000000080216000 00001000 rwx--ad
80217000 0000000080217000 00009000 rwx----
80220000 0000000080220000 00001000 rwx--ad
80221000 0000000080221000 0001f000 rwx----
80240000 0000000080240000 00001000 rwx--ad
80241000 0000000080241000 001bf000 rwx----
80400000 0000000080400000 00400000 rwx----
80800000 0000000080800000 00400000 rwx----
80c00000 0000000080c00000 00400000 rwx----
81000000 0000000081000000 00400000 rwx----
81400000 0000000081400000 00400000 rwx----
81800000 0000000081800000 00400000 rwx----
81c00000 0000000081c00000 00400000 rwx----
82000000 0000000082000000 00400000 rwx----
82400000 0000000082400000 00400000 rwx----
82800000 0000000082800000 00400000 rwx----
82c00000 0000000082c00000 00400000 rwx----
83000000 0000000083000000 00400000 rwx----
83400000 0000000083400000 00400000 rwx----
83800000 0000000083800000 00400000 rwx----
83c00000 0000000083c00000 00400000 rwx----
84000000 0000000084000000 00241000 rwx----

The columns represent, in order: virtual address, physical address, size (in hexadecimal bytes), and attributes.

Attributes are represented by a combination of r (readable), w (writable), x (executable), a (accessed), and d (written), where a and d indicate that the CPU has "accessed the page" and "written to the page" respectively. They are auxiliary information for the OS to keep track of which pages are actually being used/modified.

Tip

For beginners, debugging page table can be quite challenging. If things aren't working as expected, refer to the "Appendix: Debugging paging" section.

Appendix: Debugging paging

Setting up page tables can be tricky, and mistakes can be hard to notice. In this appendix, we'll look at some common paging errors and how to debug them.

Forgetting to set the paging mode

Let's say we forget to set the mode in the satp register:

    __asm__ __volatile__(
        "sfence.vma\n"
        "csrw satp, %[satp]\n"
        "sfence.vma\n"
        :
        : [satp] "r" (((uint32_t) next->page_table / PAGE_SIZE)) // Missing SATP_SV32!
    );

However, when you run the OS, you'll see that it works as usual. This is because paging remains disabled and memory addresses are treated as physical addresses as before.

To debug this case, try info mem command in the QEMU monitor. You'll see something like this:

(qemu) info mem
No translation or protection

Specifying physical address instead of physical page number

Let's say we mistakenly specify the page table using a physical address instead of a physical page number:

    __asm__ __volatile__(
        "sfence.vma\n"
        "csrw satp, %[satp]\n"
        "sfence.vma\n"
        :
        : [satp] "r" (SATP_SV32 | ((uint32_t) next->page_table)) // Forgot to shift!
    );

In this case, info mem will print no mappings:

$ ./run.sh

QEMU 8.0.2 monitor - type 'help' for more information
(qemu) stop
(qemu) info mem
vaddr    paddr            size     attr
-------- ---------------- -------- -------

To debug this, dump registers to see what the CPU is doing:

(qemu) info registers

CPU#0
 V      =   0
 pc       80200188
 ...
 scause   0000000c
 ...

According to llvm-addr2line, 80200188 is the starting address of the exception handler. The exception reason in scause corresponds to "Instruction page fault".

Let's take a closer look at what's specifically happening by examining the QEMU logs:

$QEMU -machine virt -bios default -nographic -serial mon:stdio --no-reboot \
    -d unimp,guest_errors,int,cpu_reset -D qemu.log \  # new!
    -kernel kernel.elf

Invalid read at addr 0x253000800, size 4, region '(null)', reason: rejected
riscv_cpu_do_interrupt: hart:0, async:0, cause:0000000c, epc:0x80200580, tval:0x80200580, desc=exec_page_fault
Invalid read at addr 0x253000800, size 4, region '(null)', reason: rejected
riscv_cpu_do_interrupt: hart:0, async:0, cause:0000000c, epc:0x80200188, tval:0x80200188, desc=exec_page_fault
Invalid read at addr 0x253000800, size 4, region '(null)', reason: rejected
riscv_cpu_do_interrupt: hart:0, async:0, cause:0000000c, epc:0x80200188, tval:0x80200188, desc=exec_page_fault

Here are what you can infer from the logs:

epc, which indicates the location of the page fault exception, is 0x80200580. llvm-objdump shows that it points to the instruction immediately after setting the satp register. This means that a page fault occurs right after enabling paging.

All subsequent page faults show the same value. The exceptions occurred at 0x80200188, points to the starting address of the exception handler. Because this log continues indefinitely, the exceptions (page fault) occurs when trying to execute the exception handler.

Looking at the info registers in QEMU monitor, satp is 0x80253000. Calculating the physical address according to the specification: (0x80253000 & 0x3fffff) * 4096 = 0x253000000, which does not fit within a 32-bit address space. This indicates that an abnormal value has been set.

To summarize, you can investigate what's wrong by checking QEMU logs, register dumps, and memory dumps. However, the most important thing is to "read the specification carefully." It's very common to overlook or misinterpret it.

Application

In this chapter, we'll prepare the first application executable to run on our kernel.

Memory layout

In the previous chapter, we implemented isolated virtual address spaces using the paging mechanism. Let's consider where to place the application in the address space.

Create a new linker script (user.ld) that defines where to place the application in memory:

ENTRY(start)

SECTIONS {
    . = 0x1000000;

    /* machine code */
    .text :{
        KEEP(*(.text.start));
        *(.text .text.*);
    }

    /* read-only data */
    .rodata : ALIGN(4) {
        *(.rodata .rodata.*);
    }

    /* data with initial values */
    .data : ALIGN(4) {
        *(.data .data.*);
    }

    /* data that should be zero-filled at startup */
    .bss : ALIGN(4) {
        *(.bss .bss.* .sbss .sbss.*);

        . = ALIGN(16);
        . += 64 * 1024; /* 64KB */
        __stack_top = .;

       ASSERT(. < 0x1800000, "too large executable");
    }
}

It looks pretty much the same as the kernel's linker script, isn't it? The key difference is the base address (0x1000000) so that the application doesn't overlap with the kernel's address space.

ASSERT is an assertion which aborts the linker if the condition in the first argument is not met. Here, it ensures that the end of the .bss section, which is the end of the application memory, does not exceed 0x1800000. This is to ensure that the executable file doesn't accidentally become too large.

Userland library

Next, let's create a library for userland programs. For simplicity, we'll start with a minimal feature set to start the application:

#include "user.h"

extern char __stack_top[];

__attribute__((noreturn)) void exit(void) {
    for (;;);
}

void putchar(char c) {
    /* TODO */
}

__attribute__((section(".text.start")))
__attribute__((naked))
void start(void) {
    __asm__ __volatile__(
        "mv sp, %[stack_top] \n"
        "call main           \n"
        "call exit           \n"
        :: [stack_top] "r" (__stack_top)
    );
}

The execution of the application starts from the start function. Similar to the kernel's boot process, it sets up the stack pointer and calls the application's main function.

We prepare the exit function to terminate the application. However, for now, we'll just have it perform an infinite loop.

Also, we define the putchar function that the printf function in common.c refers to. We'll implement this later.

Unlike the kernel's initialization process, we don't clear the .bss section with zeros. This is because the kernel guarantees that it has already filled it with zeros (in the alloc_pages function).

Tip

Allocated memory regions are already filled with zeros in typical operating systems too. Otherwise, the memory may contain sensitive information (e.g. credentials) from other processes, and it could lead to a critical security issue.

Lastly, prepare a header file (user.h) for the userland library:

#pragma once
#include "common.h"

__attribute__((noreturn)) void exit(void);
void putchar(char ch);

First application

It's time to create the first application! Unfortunately, we still don't have a way to display characters, we can't start with a "Hello, World!" program. Instead, we'll create a simple infinite loop:

#include "user.h"

void main(void) {
    for (;;);
}

Building the application

Applications will be built separately from the kernel. Let's create a new script (run.sh) to build the application:

OBJCOPY=/opt/homebrew/opt/llvm/bin/llvm-objcopy

# Build the shell (application)
$CC $CFLAGS -Wl,-Tuser.ld -Wl,-Map=shell.map -o shell.elf shell.c user.c common.c
$OBJCOPY --set-section-flags .bss=alloc,contents -O binary shell.elf shell.bin
$OBJCOPY -Ibinary -Oelf32-littleriscv shell.bin shell.bin.o

# Build the kernel
$CC $CFLAGS -Wl,-Tkernel.ld -Wl,-Map=kernel.map -o kernel.elf \
    kernel.c common.c shell.bin.o

The first $CC call is very similar to the kernel build script. Compile C files and link them with the user.ld linker script.

The first $OBJCOPY command converts the executable file (in ELF format) to raw binary format. A raw binary is the actual content that will be expanded in memory from the base address (in this case, 0x1000000). The OS can prepare the application in memory simply by copying the contents of the raw binary. Common OSes use formats like ELF, where memory contents and their mapping information are separate, but in this book, we'll use raw binary for simplicity.

The second $OBJCOPY command converts the raw binary execution image into a format that can be embedded in C language. Let's take a look at what's inside using the llvm-nm command:

$ llvm-nm shell.bin.o
00000000 D _binary_shell_bin_start
00010260 D _binary_shell_bin_end
00010260 A _binary_shell_bin_size

The prefix _binary_ is followed by the file name, and then start, end, and size. These are symbols that indicate the beginning, end, and size of the execution image, respectively. In practice, they are used as follows:

extern char _binary_shell_bin_start[];
extern char _binary_shell_bin_size[];

void main(void) {
    uint8_t *shell_bin = (uint8_t *) _binary_shell_bin_start;
    printf("shell_bin size = %d\n", (int) _binary_shell_bin_size);
    printf("shell_bin[0] = %x (%d bytes)\n", shell_bin[0]);
}

This program outputs the file size of shell.bin and the first byte of its contents. In other words, you can treat the _binary_shell_bin_start variable as if it contains the file contents, like:

char _binary_shell_bin_start[] = "<shell.bin contents here>";

_binary_shell_bin_size variable contains the file size. However, it's used in a slightly unusual way. Let's check with llvm-nm again:

$ llvm-nm shell.bin.o | grep _binary_shell_bin_size
00010454 A _binary_shell_bin_size

$ ls -al shell.bin   ← note: do not confuse with shell.bin.o!
-rwxr-xr-x 1 seiya staff 66644 Oct 24 13:35 shell.bin

$ python3 -c 'print(0x10454)'
66644

The first column in the llvm-nm output is the address of the symbol. This 10454 hexadecimal number matches the file size, but this is not a coincidence. Generally, the values of each address in a .o file are determined by the linker. However, _binary_shell_bin_size is special.

The A in the second column indicates that the address of _binary_shell_bin_size is a type of symbol (absolute) that should not be changed by the linker. That is, it embeds the file size as an address.

By defining it as an array of an arbitrary type like char _binary_shell_bin_size[], _binary_shell_bin_size will be treated as a pointer storing its address. However, since we're embedding the file size as an address here, casting it will result in the file size. This is a common trick (or a dirty hack) that exploits the object file format.

Lastly, we've added shell.bin.o to the clang arguments in the kernel compiling. It embeds the first application's executable into the kernel image.

Disassemble the executable

In disassembly, we can see that the .text.start section is placed at the beginning of the executable file. The start function should be placed at 0x1000000 as follows:

$ llvm-objdump -d shell.elf

shell.elf:  file format elf32-littleriscv

Disassembly of section .text:

01000000 <start>:
 1000000: 37 05 01 01   lui a0, 4112
 1000004: 13 05 05 26   addi    a0, a0, 608
 1000008: 2a 81         mv  sp, a0
 100000a: 19 20         jal 0x1000010 <main>
 100000c: 29 20         jal 0x1000016 <exit>
 100000e: 00 00         unimp

01000010 <main>:
 1000010: 01 a0         j   0x1000010 <main>
 1000012: 00 00         unimp

01000016 <exit>:
 1000016: 01 a0         j   0x1000016 <exit>

User Mode

In this chapter, we'll run the application we created in the previous chapter.

Extracting the executable file

In executable file formats like ELF, the load address is stored in its file header (program header in ELF). However, since our application's execution image is a raw binary, we need to prepare it with a fixed value like this:

// The base virtual address of an application image. This needs to match the
// starting address defined in `user.ld`.
#define USER_BASE 0x1000000

Next, define symbols to use the embedded raw binary in shell.bin.o:

extern char _binary_shell_bin_start[], _binary_shell_bin_size[];

Also, update the create_process function to start the application:

void user_entry(void) {
    PANIC("not yet implemented");
}

struct process *create_process(const void *image, size_t image_size) {
    /* omitted */
    *--sp = 0;                      // s3
    *--sp = 0;                      // s2
    *--sp = 0;                      // s1
    *--sp = 0;                      // s0
    *--sp = (uint32_t) user_entry;  // ra (changed!)

    uint32_t *page_table = (uint32_t *) alloc_pages(1);

    // Map kernel pages.
    for (paddr_t paddr = (paddr_t) __kernel_base;
         paddr < (paddr_t) __free_ram_end; paddr += PAGE_SIZE)
        map_page(page_table, paddr, paddr, PAGE_R | PAGE_W | PAGE_X);

    // Map user pages.
    for (uint32_t off = 0; off < image_size; off += PAGE_SIZE) {
        paddr_t page = alloc_pages(1);

        // Handle the case where the data to be copied is smaller than the
        // page size.
        size_t remaining = image_size - off;
        size_t copy_size = PAGE_SIZE <= remaining ? PAGE_SIZE : remaining;

        // Fill and map the page.
        memcpy((void *) page, image + off, copy_size);
        map_page(page_table, USER_BASE + off, page,
                 PAGE_U | PAGE_R | PAGE_W | PAGE_X);
    }

We've modified create_process to take the pointer to the execution image (image) and the image size (image_size) as arguments. It copies the execution image page by page for the specified size and maps it to the process' page table. Also, it sets the jump destination for the first context switch to user_entry. For now, we'll keep this as an empty function.

Warning

If you map the execution image directly without copying it, processes of the same application would end up sharing the same physical pages. It ruins the memory isolation!

Lastly, modify the caller of the create_process function and make it create a user process:

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);

    printf("\n\n");

    WRITE_CSR(stvec, (uint32_t) kernel_entry);

    idle_proc = create_process(NULL, 0); // updated!
    idle_proc->pid = -1; // idle
    current_proc = idle_proc;

    // new!
    create_process(_binary_shell_bin_start, (size_t) _binary_shell_bin_size);

    yield();
    PANIC("switched to idle process");
}

Let's try it and check with the QEMU monitor if the execution image is mapped as expected:

(qemu) info mem
vaddr    paddr            size     attr
-------- ---------------- -------- -------
01000000 0000000080265000 00001000 rwxu---
01001000 0000000080267000 00010000 rwxu---

We can see that the physical address 0x80265000 is mapped to the virtual address 0x1000000 (USER_BASE). Let's take a look at the contents of this physical address. To display the contents of physical memory, use xp command:

(qemu) xp /32b 0x80265000
0000000080265000: 0x37 0x05 0x01 0x01 0x13 0x05 0x05 0x26
0000000080265008: 0x2a 0x81 0x19 0x20 0x29 0x20 0x00 0x00
0000000080265010: 0x01 0xa0 0x00 0x00 0x82 0x80 0x01 0xa0
0000000080265018: 0x09 0xca 0xaa 0x86 0x7d 0x16 0x13 0x87

It seems some data is present. Check the contents of shell.bin to confirm that it indeed matches:

$ hexdump -C shell.bin | head
00000000  37 05 01 01 13 05 05 26  2a 81 19 20 29 20 00 00  |7......&*.. ) ..|
00000010  01 a0 00 00 82 80 01 a0  09 ca aa 86 7d 16 13 87  |............}...|
00000020  16 00 23 80 b6 00 ba 86  75 fa 82 80 01 ce aa 86  |..#.....u.......|
00000030  03 87 05 00 7d 16 85 05  93 87 16 00 23 80 e6 00  |....}.......#...|
00000040  be 86 7d f6 82 80 03 c6  05 00 aa 86 01 ce 85 05  |..}.............|
00000050  2a 87 23 00 c7 00 03 c6  05 00 93 06 17 00 85 05  |*.#.............|
00000060  36 87 65 fa 23 80 06 00  82 80 03 46 05 00 15 c2  |6.e.#......F....|
00000070  05 05 83 c6 05 00 33 37  d0 00 93 77 f6 0f bd 8e  |......37...w....|
00000080  93 b6 16 00 f9 8e 91 c6  03 46 05 00 85 05 05 05  |.........F......|
00000090  6d f2 03 c5 05 00 93 75  f6 0f 33 85 a5 40 82 80  |m......u..3..@..|

Hmm, it's hard to understand in hexadecimal. Let's disassemble the machine code to see if it matches the expected instructions:

(qemu) xp /8i 0x80265000
0x80265000:  01010537          lui                     a0,16842752
0x80265004:  26050513          addi                    a0,a0,608
0x80265008:  812a              mv                      sp,a0
0x8026500a:  2019              jal                     ra,6                    # 0x80265010
0x8026500c:  2029              jal                     ra,10                   # 0x80265016
0x8026500e:  0000              illegal
0x80265010:  a001              j                       0                       # 0x80265010
0x80265012:  0000              illegal

It calculates/fills the initial stack pointer value, and then calls two different functions. If we compare this with the disassembly results of shell.elf, we can confirm that it indeed matches:

$ llvm-objdump -d shell.elf | head -n20

shell.elf:      file format elf32-littleriscv

Disassembly of section .text:

01000000 <start>:
 1000000: 37 05 01 01   lui     a0, 4112
 1000004: 13 05 05 26   addi    a0, a0, 608
 1000008: 2a 81         mv      sp, a0
 100000a: 19 20         jal     0x1000010 <main>
 100000c: 29 20         jal     0x1000016 <exit>
 100000e: 00 00         unimp

01000010 <main>:
 1000010: 01 a0         j       0x1000010 <main>
 1000012: 00 00         unimp

Transition to user mode

To run applications, we use a CPU mode called user mode, or in RISC-V terms, U-Mode. It's surprisingly simple to switch to U-Mode. Here's how:

#define SSTATUS_SPIE (1 << 5)

// ↓ __attribute__((naked)) is very important!
__attribute__((naked)) void user_entry(void) {
    __asm__ __volatile__(
        "csrw sepc, %[sepc]        \n"
        "csrw sstatus, %[sstatus]  \n"
        "sret                      \n"
        :
        : [sepc] "r" (USER_BASE),
          [sstatus] "r" (SSTATUS_SPIE)
    );
}

The switch from S-Mode to U-Mode is done with the sret instruction. However, before changing the operation mode, it does two writes to CSRs:

Set the program counter for when transitioning to U-Mode in the sepc register. That is, where sret jumps to.

Set the SPIE bit in the sstatus register. Setting this enables hardware interrupts when entering U-Mode, and the handler set in the stvec register will be called.

Tip

In this book, we don't use hardware interrupts but use polling instead, so it's not necessary to set the SPIE bit. However, it's better to be clear rather than silently ignoring interrupts.

Try user mode

Now let's try it! That said, because shell.c just loops infinitely, we can't tell if it's working properly on the screen. Instead, let's take a look with the QEMU monitor:

(qemu) info registers

CPU#0
 V      =   0
 pc       01000010

It seems CPU is continuously executing 0x1000010. It appears to be working properly, but somehow it doesn't feel satisfying. So, let's see if we can observe behavior which is specific to U-Mode. Add one line to shell.c:

#include "user.h"

void main(void) {
    *((volatile int *) 0x80200000) = 0x1234; // new!
    for (;;);
}

This 0x80200000 is a memory area used by the kernel that is mapped on the page table. However, since it is a kernel page where the U bit in the page table entry is not set, an exception (page fault) should occur, and the kernel should panic. Let's try it:

$ ./run.sh

PANIC: kernel.c:71: unexpected trap scause=0000000f, stval=80200000, sepc=0100001a

The 15th exception (scause = 0xf = 15), it corresponds to "Store/AMO page fault". It seems the expected exception happened! Also, the program counter in sepc points to the line we added to shell.c:

$ llvm-addr2line -e shell.elf 0x100001a
/Users/seiya/dev/os-from-scratch/shell.c:4

Congrats! You've successfully executed your first application! Isn't it surprising how easy it is to implement user mode? The kernel is very similar to an application - it just has a few more privileges.

System Call

In this chapter, we will implement "system calls" that allow applications to invoke kernel functions. Time to Hello World from the userland!

User library

Invoking system call is quite similar to the SBI call implementation we've seen before:

int syscall(int sysno, int arg0, int arg1, int arg2) {
    register int a0 __asm__("a0") = arg0;
    register int a1 __asm__("a1") = arg1;
    register int a2 __asm__("a2") = arg2;
    register int a3 __asm__("a3") = sysno;

    __asm__ __volatile__("ecall"
                         : "=r"(a0)
                         : "r"(a0), "r"(a1), "r"(a2), "r"(a3)
                         : "memory");

    return a0;
}

The syscall function sets the system call number in the a3 register and the system call arguments in the a0 to a2 registers, then executes the ecall instruction. The ecall instruction is a special instruction used to delegate processing to the kernel. When the ecall instruction is executed, an exception handler is called, and control is transferred to the kernel. The return value from the kernel is set in the a0 register.

The first system call we will implement is putchar, which outputs a character, via system call. It takes a character as the first argument. For the second and subsequent unused arguments are set to 0:

#define SYS_PUTCHAR 1

void putchar(char ch) {
    syscall(SYS_PUTCHAR, ch, 0, 0);
}

Handle ecall instruction in the kernel

Next, update the trap handler to handle ecall instruction:

#define SCAUSE_ECALL 8

void handle_trap(struct trap_frame *f) {
    uint32_t scause = READ_CSR(scause);
    uint32_t stval = READ_CSR(stval);
    uint32_t user_pc = READ_CSR(sepc);
    if (scause == SCAUSE_ECALL) {
        handle_syscall(f);
        user_pc += 4;
    } else {
        PANIC("unexpected trap scause=%x, stval=%x, sepc=%x\n", scause, stval, user_pc);
    }

    WRITE_CSR(sepc, user_pc);
}

Whether the ecall instruction was called can be determined by checking the value of scause. Besides calling the handle_syscall function, we also add 4 (the size of ecall instruction) to the value of sepc. This is because sepc points to the program counter that caused the exception, which points to the ecall instruction. If we don't change it, the kernel goes back to the same place, and the ecall instruction is executed repeatedly.

System call handler

The following system call handler is called from the trap handler. It receives a structure of "registers at the time of exception" that was saved in the trap handler:

void handle_syscall(struct trap_frame *f) {
    switch (f->a3) {
        case SYS_PUTCHAR:
            putchar(f->a0);
            break;
        default:
            PANIC("unexpected syscall a3=%x\n", f->a3);
    }
}

It determines the type of system call by checking the value of the a3 register. Now we only have one system call, SYS_PUTCHAR, which simply outputs the character stored in the a0 register.

Test the system call

You've implemented the system call. Let's try it out!

Do you remember the implementation of the printf function in common.c? It calls the putchar function to display characters. Since we have just implemented putchar in the userland library, we can use it as is:

void main(void) {
    printf("Hello World from shell!\n");
}

You'll see the charming message on the screen:

$ ./run.sh
Hello World from shell!

Congratulations! You've successfully implemented the system call! But we're not done yet. Let's implement more system calls!

Receive characters from keyboard (getchar system call)

Our next goal is to implement a shell. To do that, we need to be able to receive characters from the keyboard.

SBI provides an interface to read "input to the debug console". If there is no input, it returns -1:

long getchar(void) {
    struct sbiret ret = sbi_call(0, 0, 0, 0, 0, 0, 0, 2);
    return ret.error;
}

The getchar system call is implemented as follows:

#define SYS_GETCHAR 2

int getchar(void) {
    return syscall(SYS_GETCHAR, 0, 0, 0);
}

int getchar(void);

void handle_syscall(struct trap_frame *f) {
    switch (f->a3) {
        case SYS_GETCHAR:
            while (1) {
                long ch = getchar();
                if (ch >= 0) {
                    f->a0 = ch;
                    break;
                }

                yield();
            }
            break;
        /* omitted */
    }
}

The implementation of the getchar system call repeatedly calls the SBI until a character is input. However, simply repeating this prevents other processes from running, so we call the yield system call to yield the CPU to other processes.

Note

Strictly speaking, SBI does not read characters from keyboard, but from the serial port. It works because the keyboard (or QEMU's standard input) is connected to the serial port.

Write a shell

Let's write a shell with a simple command hello, which displays Hello world from shell!:

void main(void) {
    while (1) {
prompt:
        printf("> ");
        char cmdline[128];
        for (int i = 0;; i++) {
            char ch = getchar();
            putchar(ch);
            if (i == sizeof(cmdline) - 1) {
                printf("command line too long\n");
                goto prompt;
            } else if (ch == '\r') {
                printf("\n");
                cmdline[i] = '\0';
                break;
            } else {
                cmdline[i] = ch;
            }
        }

        if (strcmp(cmdline, "hello") == 0)
            printf("Hello world from shell!\n");
        else
            printf("unknown command: %s\n", cmdline);
    }
}

It reads characters until a newline comes, and checks if the entered string matches the command name.

Warning

Note that on the debug console, the newline character is ('\r').

Let's try typing hello command:

$ ./run.sh

> hello
Hello world from shell!

Your OS is starting to look like a real OS! How fast you've come this far!

Process termination (exit system call)

Lastly, let's implement exit system call, which terminates the process:

#define SYS_EXIT    3

__attribute__((noreturn)) void exit(void) {
    syscall(SYS_EXIT, 0, 0, 0);
    for (;;); // Just in case!
}

#define PROC_EXITED   2

void handle_syscall(struct trap_frame *f) {
    switch (f->a3) {
        case SYS_EXIT:
            printf("process %d exited\n", current_proc->pid);
            current_proc->state = PROC_EXITED;
            yield();
            PANIC("unreachable");
        /* omitted */
    }
}

The system call changes the process state to PROC_EXITED, and calls yield to give up the CPU to other processes. The scheduler will only execute processes in PROC_RUNNABLE state, so it will never return to this process. However, PANIC macro is added to cause a panic in case it does return.

Tip

For simplicity, we only mark the process as exited (PROC_EXITED). If you want to build a practical OS, it is necessary to free resources held by the process, such as page tables and allocated memory regions.

Add the exit command to the shell:

        if (strcmp(cmdline, "hello") == 0)
            printf("Hello world from shell!\n");
        else if (strcmp(cmdline, "exit") == 0)
            exit();
        else
            printf("unknown command: %s\n", cmdline);

You're done! Let's try running it:

$ ./run.sh

> exit
process 2 exited
PANIC: kernel.c:333: switched to idle process

When the exit command is executed, the shell process terminates via system call, and there are no other runnable processes remaining. As a result, the scheduler will select the idle process and cause a panic.

Disk I/O

In this chapter, we will implement a device driver for the virtio-blk, a virtual disk device. While virtio-blk does not exist in real hardware, it shares the very same interface as a real one.

Virtio

Virtio is a device interface standard for virtual devices (virtio devices). In other words, it is one of the APIs for device drivers to control devices. Like you use HTTP to access web servers, you use virtio to access virtio devices. Virtio is widely used in virtualization environments such as QEMU and Firecracker.

Virtqueue

Virtio devices have a structure called a virtqueue. As the name suggests, it is a queue shared between the driver and the device. In a nutshell:

A virtqueue consists of the following three areas:

Name | Written by | Content | Contents
Descriptor Area | Driver | A table of descriptors: the address and size of the request | Memory address, length, index of the next descriptor
Available Ring | Driver | Processing requests to the device | The head index of the descriptor chain
Used Ring | Device | Processing requests handled by the device | The head index of the descriptor chain

Each request (e.g., a write to disk) consists of multiple descriptors, called a descriptor chain. By splitting into multiple descriptors, you can specify scattered memory data (so-called Scatter-Gather IO) or give different descriptor attributes (whether writable by the device).

For example, when writing to a disk, virtqueue will be used as follows:

The driver writes a read/write request in the Descriptor area.

The driver adds the index of the head descriptor to the Available Ring.

The driver notifies the device that there is a new request.

The device reads a request from the Available Ring and processes it.

The device writes the descriptor index to the Used Ring, and notifies the driver that it is complete.

For details, refer to the virtio specification. In this implementation, we will focus on a device called virtio-blk.

Enabling virtio devices

Before writing a device driver, let's prepare a test file. Create a file named lorem.txt and fill it with some random text like the following:

$ echo "Lorem ipsum dolor sit amet, consectetur adipiscing elit. In ut magna consequat, cursus velit aliquam, scelerisque odio. Ut lorem eros, feugiat quis bibendum vitae, malesuada ac orci. Praesent eget quam non nunc fringilla cursus imperdiet non tellus. Aenean dictum lobortis turpis, non interdum leo rhoncus sed. Cras in tellus auctor, faucibus tortor ut, maximus metus. Praesent placerat ut magna non tristique. Pellentesque at nunc quis dui tempor vulputate. Vestibulum vitae massa orci. Mauris et tellus quis risus sagittis placerat. Integer lorem leo, feugiat sed molestie non, viverra a tellus." > lorem.txt

Also, attach a virtio-blk device to QEMU:

$QEMU -machine virt -bios default -nographic -serial mon:stdio --no-reboot \
    -d unimp,guest_errors,int,cpu_reset -D qemu.log \
    -drive id=drive0,file=lorem.txt,format=raw,if=none \            # new
    -device virtio-blk-device,drive=drive0,bus=virtio-mmio-bus.0 \  # new
    -kernel kernel.elf

The newly added options are as follows:

-drive id=drive0: Defines disk named drive0, with lorem.txt as the disk image. The disk image format is raw (treats the file contents as-is as disk data).

-device virtio-blk-device: Adds a virtio-blk device with disk drive0. bus=virtio-mmio-bus.0 maps the device into a virtio-mmio bus (virtio over Memory Mapped I/O).

Define C macros/structs

First, let's add some virtio-related definitions to kernel.h:

#define SECTOR_SIZE       512
#define VIRTQ_ENTRY_NUM   16
#define VIRTIO_DEVICE_BLK 2
#define VIRTIO_BLK_PADDR  0x10001000
#define VIRTIO_REG_MAGIC         0x00
#define VIRTIO_REG_VERSION       0x04
#define VIRTIO_REG_DEVICE_ID     0x08
#define VIRTIO_REG_QUEUE_SEL     0x30
#define VIRTIO_REG_QUEUE_NUM_MAX 0x34
#define VIRTIO_REG_QUEUE_NUM     0x38
#define VIRTIO_REG_QUEUE_ALIGN   0x3c
#define VIRTIO_REG_QUEUE_PFN     0x40
#define VIRTIO_REG_QUEUE_READY   0x44
#define VIRTIO_REG_QUEUE_NOTIFY  0x50
#define VIRTIO_REG_DEVICE_STATUS 0x70
#define VIRTIO_REG_DEVICE_CONFIG 0x100
#define VIRTIO_STATUS_ACK       1
#define VIRTIO_STATUS_DRIVER    2
#define VIRTIO_STATUS_DRIVER_OK 4
#define VIRTIO_STATUS_FEAT_OK   8
#define VIRTQ_DESC_F_NEXT          1
#define VIRTQ_DESC_F_WRITE         2
#define VIRTQ_AVAIL_F_NO_INTERRUPT 1
#define VIRTIO_BLK_T_IN  0
#define VIRTIO_BLK_T_OUT 1

// Virtqueue Descriptor area entry.
struct virtq_desc {
    uint64_t addr;
    uint32_t len;
    uint16_t flags;
    uint16_t next;
} __attribute__((packed));

// Virtqueue Available Ring.
struct virtq_avail {
    uint16_t flags;
    uint16_t index;
    uint16_t ring[VIRTQ_ENTRY_NUM];
} __attribute__((packed));

// Virtqueue Used Ring entry.
struct virtq_used_elem {
    uint32_t id;
    uint32_t len;
} __attribute__((packed));

// Virtqueue Used Ring.
struct virtq_used {
    uint16_t flags;
    uint16_t index;
    struct virtq_used_elem ring[VIRTQ_ENTRY_NUM];
} __attribute__((packed));

// Virtqueue.
struct virtio_virtq {
    struct virtq_desc descs[VIRTQ_ENTRY_NUM];
    struct virtq_avail avail;
    struct virtq_used used __attribute__((aligned(PAGE_SIZE)));
    int queue_index;
    volatile uint16_t *used_index;
    uint16_t last_used_index;
} __attribute__((packed));

// Virtio-blk request.
struct virtio_blk_req {
    uint32_t type;
    uint32_t reserved;
    uint64_t sector;
    uint8_t data[512];
    uint8_t status;
} __attribute__((packed));

Note

__attribute__((packed)) is a compiler extension that tells the compiler to pack the struct members without padding. Otherwise, the compiler may add hidden padding bytes and driver/device may see different values.

Next, add utility functions to kernel.c for accessing MMIO registers:

uint32_t virtio_reg_read32(unsigned offset) {
    return *((volatile uint32_t *) (VIRTIO_BLK_PADDR + offset));
}

uint64_t virtio_reg_read64(unsigned offset) {
    return *((volatile uint64_t *) (VIRTIO_BLK_PADDR + offset));
}

void virtio_reg_write32(unsigned offset, uint32_t value) {
    *((volatile uint32_t *) (VIRTIO_BLK_PADDR + offset)) = value;
}

void virtio_reg_fetch_and_or32(unsigned offset, uint32_t value) {
    virtio_reg_write32(offset, virtio_reg_read32(offset) | value);
}

Warning

Accessing MMIO registers are not same as accessing normal memory. You should use volatile keyword to prevent the compiler from optimizing out the read/write operations. In MMIO, memory access may trigger side effects (e.g., sending a command to the device).

Map the MMIO region

First, map the virtio-blk MMIO region to the page table so that the kernel can access the MMIO registers. It's super simple:

struct process *create_process(const void *image, size_t image_size) {
    /* omitted */

    for (paddr_t paddr = (paddr_t) __kernel_base;
         paddr < (paddr_t) __free_ram_end; paddr += PAGE_SIZE)
        map_page(page_table, paddr, paddr, PAGE_R | PAGE_W | PAGE_X);

    map_page(page_table, VIRTIO_BLK_PADDR, VIRTIO_BLK_PADDR, PAGE_R | PAGE_W); // new

Virtio device initialization

The initialization process is detailed in the virtio specification:

3.1.1 Driver Requirements: Device Initialization The driver MUST follow this sequence to initialize a device:

Reset the device.

Set the ACKNOWLEDGE status bit: the guest OS has noticed the device.

Set the DRIVER status bit: the guest OS knows how to drive the device.

Read device feature bits, and write the subset of feature bits understood by the OS and driver to the device. During this step the driver MAY read (but MUST NOT write) the device-specific configuration fields to check that it can support the device before accepting it.

Set the FEATURES_OK status bit. The driver MUST NOT accept new feature bits after this step.

Re-read device status to ensure the FEATURES_OK bit is still set: otherwise, the device does not support our subset of features and the device is unusable.

Perform device-specific setup, including discovery of virtqueues for the device, optional per-bus setup, reading and possibly writing the device’s virtio configuration space, and population of virtqueues.

Set the DRIVER_OK status bit. At this point the device is “live”.

You might be overwhelmed by lengthy steps, but don't worry. A naive implementation is very simple:

struct virtio_virtq *blk_request_vq;
struct virtio_blk_req *blk_req;
paddr_t blk_req_paddr;
unsigned blk_capacity;

void virtio_blk_init(void) {
    if (virtio_reg_read32(VIRTIO_REG_MAGIC) != 0x74726976)
        PANIC("virtio: invalid magic value");
    if (virtio_reg_read32(VIRTIO_REG_VERSION) != 1)
        PANIC("virtio: invalid version");
    if (virtio_reg_read32(VIRTIO_REG_DEVICE_ID) != VIRTIO_DEVICE_BLK)
        PANIC("virtio: invalid device id");

    // 1. Reset the device.
    virtio_reg_write32(VIRTIO_REG_DEVICE_STATUS, 0);
    // 2. Set the ACKNOWLEDGE status bit: the guest OS has noticed the device.
    virtio_reg_fetch_and_or32(VIRTIO_REG_DEVICE_STATUS, VIRTIO_STATUS_ACK);
    // 3. Set the DRIVER status bit.
    virtio_reg_fetch_and_or32(VIRTIO_REG_DEVICE_STATUS, VIRTIO_STATUS_DRIVER);
    // 5. Set the FEATURES_OK status bit.
    virtio_reg_fetch_and_or32(VIRTIO_REG_DEVICE_STATUS, VIRTIO_STATUS_FEAT_OK);
    // 7. Perform device-specific setup, including discovery of virtqueues for the device
    blk_request_vq = virtq_init(0);
    // 8. Set the DRIVER_OK status bit.
    virtio_reg_write32(VIRTIO_REG_DEVICE_STATUS, VIRTIO_STATUS_DRIVER_OK);

    // Get the disk capacity.
    blk_capacity = virtio_reg_read64(VIRTIO_REG_DEVICE_CONFIG + 0) * SECTOR_SIZE;
    printf("virtio-blk: capacity is %d bytes\n", blk_capacity);

    // Allocate a region to store requests to the device.
    blk_req_paddr = alloc_pages(align_up(sizeof(*blk_req), PAGE_SIZE) / PAGE_SIZE);
    blk_req = (struct virtio_blk_req *) blk_req_paddr;
}

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);
    WRITE_CSR(stvec, (uint32_t) kernel_entry);

    virtio_blk_init(); // new

Virtqueue initialization

Virtqueues also need to be initialized. Let's read the specification:

The virtual queue is configured as follows:

Select the queue writing its index (first queue is 0) to QueueSel.

Check if the queue is not already in use: read QueuePFN, expecting a returned value of zero (0x0).

Read maximum queue size (number of elements) from QueueNumMax. If the returned value is zero (0x0) the queue is not available.

Allocate and zero the queue pages in contiguous virtual memory, aligning the Used Ring to an optimal boundary (usually page size). The driver should choose a queue size smaller than or equal to QueueNumMax.

Notify the device about the queue size by writing the size to QueueNum.

Notify the device about the used alignment by writing its value in bytes to QueueAlign.

Write the physical number of the first page of the queue to the QueuePFN register.

Here's a simple implementation:

struct virtio_virtq *virtq_init(unsigned index) {
    // Allocate a region for the virtqueue.
    paddr_t virtq_paddr = alloc_pages(align_up(sizeof(struct virtio_virtq), PAGE_SIZE) / PAGE_SIZE);
    struct virtio_virtq *vq = (struct virtio_virtq *) virtq_paddr;
    vq->queue_index = index;
    vq->used_index = (volatile uint16_t *) &vq->used.index;
    // 1. Select the queue writing its index (first queue is 0) to QueueSel.
    virtio_reg_write32(VIRTIO_REG_QUEUE_SEL, index);
    // 5. Notify the device about the queue size by writing the size to QueueNum.
    virtio_reg_write32(VIRTIO_REG_QUEUE_NUM, VIRTQ_ENTRY_NUM);
    // 6. Notify the device about the used alignment by writing its value in bytes to QueueAlign.
    virtio_reg_write32(VIRTIO_REG_QUEUE_ALIGN, 0);
    // 7. Write the physical number of the first page of the queue to the QueuePFN register.
    virtio_reg_write32(VIRTIO_REG_QUEUE_PFN, virtq_paddr);
    return vq;
}

This function allocates a memory region for a virtqueue, and tells the its physical address to the device. The device will use this memory region to read/write requests.

Tip

What drivers do in the initialization process is to check device capabilities/features, allocating OS resources (e.g., memory regions), and setting parameters. Isn't it similar to handshakes in network protocols?

Sending I/O requests

We now have an initialized virtio-blk device. Let's send an I/O request to the disk. I/O requests to the disk is implemented by "adding processing requests to the virtqueue" as follows:

// Notifies the device that there is a new request. `desc_index` is the index
// of the head descriptor of the new request.
void virtq_kick(struct virtio_virtq *vq, int desc_index) {
    vq->avail.ring[vq->avail.index % VIRTQ_ENTRY_NUM] = desc_index;
    vq->avail.index++;
    __sync_synchronize();
    virtio_reg_write32(VIRTIO_REG_QUEUE_NOTIFY, vq->queue_index);
    vq->last_used_index++;
}

// Returns whether there are requests being processed by the device.
bool virtq_is_busy(struct virtio_virtq *vq) {
    return vq->last_used_index != *vq->used_index;
}

// Reads/writes from/to virtio-blk device.
void read_write_disk(void *buf, unsigned sector, int is_write) {
    if (sector >= blk_capacity / SECTOR_SIZE) {
        printf("virtio: tried to read/write sector=%d, but capacity is %d\n",
              sector, blk_capacity / SECTOR_SIZE);
        return;
    }

    // Construct the request according to the virtio-blk specification.
    blk_req->sector = sector;
    blk_req->type = is_write ? VIRTIO_BLK_T_OUT : VIRTIO_BLK_T_IN;
    if (is_write)
        memcpy(blk_req->data, buf, SECTOR_SIZE);

    // Construct the virtqueue descriptors (using 3 descriptors).
    struct virtio_virtq *vq = blk_request_vq;
    vq->descs[0].addr = blk_req_paddr;
    vq->descs[0].len = sizeof(uint32_t) * 2 + sizeof(uint64_t);
    vq->descs[0].flags = VIRTQ_DESC_F_NEXT;
    vq->descs[0].next = 1;

    vq->descs[1].addr = blk_req_paddr + offsetof(struct virtio_blk_req, data);
    vq->descs[1].len = SECTOR_SIZE;
    vq->descs[1].flags = VIRTQ_DESC_F_NEXT | (is_write ? 0 : VIRTQ_DESC_F_WRITE);
    vq->descs[1].next = 2;

    vq->descs[2].addr = blk_req_paddr + offsetof(struct virtio_blk_req, status);
    vq->descs[2].len = sizeof(uint8_t);
    vq->descs[2].flags = VIRTQ_DESC_F_WRITE;

    // Notify the device that there is a new request.
    virtq_kick(vq, 0);

    // Wait until the device finishes processing.
    while (virtq_is_busy(vq))
        ;

    // virtio-blk: If a non-zero value is returned, it's an error.
    if (blk_req->status != 0) {
        printf("virtio: warn: failed to read/write sector=%d status=%d\n",
               sector, blk_req->status);
        return;
    }

    // For read operations, copy the data into the buffer.
    if (!is_write)
        memcpy(buf, blk_req->data, SECTOR_SIZE);
}

A request is sent in the following steps:

Construct a request in blk_req. Specify the sector number you want to access and the type of read/write.

Construct a descriptor chain pointing to each area of blk_req (see below).

Add the index of the head descriptor of the descriptor chain to the Available Ring.

Notify the device that there is a new pending request.

Wait until the device finishes processing (aka busy-waiting or polling).

Check the response from the device.

Here, we construct a descriptor chain consisting of 3 descriptors. We need 3 descriptors because each descriptor has different attributes (flags) as follows:

struct virtio_blk_req {
    // First descriptor: read-only from the device
    uint32_t type;
    uint32_t reserved;
    uint64_t sector;

    // Second descriptor: writable by the device if it's a read operation (VIRTQ_DESC_F_WRITE)
    uint8_t data[512];

    // Third descriptor: writable by the device (VIRTQ_DESC_F_WRITE)
    uint8_t status;
} __attribute__((packed));

Because we busy-wait until the processing is complete every time, we can simply use the first 3 descriptors in the ring. However, in practice, you need to track free/used descriptors to process multiple requests simultaneously.

Try it out

Lastly, let's try disk I/O. Add the following code to kernel.c:

    virtio_blk_init();

    char buf[SECTOR_SIZE];
    read_write_disk(buf, 0, false /* read from the disk */);
    printf("first sector: %s\n", buf);

    strcpy(buf, "hello from kernel!!!\n");
    read_write_disk(buf, 0, true /* write to the disk */);

Since we specify lorem.txt as the (raw) disk image, its contents should be displayed as-is:

$ ./run.sh

virtio-blk: capacity is 1024 bytes
first sector: Lorem ipsum dolor sit amet, consectetur adipiscing elit ...

Also, the first sector is overwritten with the string "hello from kernel!!!":

$ head lorem.txt
hello from kernel!!!
amet, consectetur adipiscing elit ...

Congratulations! You've successfully implemented a disk I/O driver!

Tip

As you might notice, device drivers are just "glue" between the OS and devices. Drivers don't control the hardware directly; drivers communicate with other software running on the device (e.g., firmware). Devices and their software, not the OS driver, will do the rest of the heavy lifting, like moving disk read/write heads.

File System

You've done a great job so far! You've implemented a process, a shell, memory management, and a disk driver. Let's finish up by implementing a file system.

Tar as file system

In this book, we'll take an interesting approach to implement a file system: using a tar file as our file system.

Tar is an archive format that can contain multiple files. It contains file contents, filenames, creation dates, and other information necessary for a file system. Compared to common file system formats like FAT or ext2, tar has a much simpler data structure. Additionally, you can manipulate the file system image using the tar command which you are already familiar with. Isn't it an ideal file format for educational purposes?

Tip

Nowadays, tar is used as a ZIP alternative, but originally it was born as sort of file system for magnetic tape. We can use it as a file system as we do in this chapter, however, you'll notice that it is not suitable for random access. The design of FAT file system would be fun to read.

Create a disk image (tar file)

Let's start by preparing the contents of our file system. Create a directory called disk and add some files to it. Name one of them hello.txt:

$ mkdir disk
$ vim disk/hello.txt
$ vim disk/meow.txt

Add a command to the build script to create a tar file and pass it as a disk image to QEMU:

(cd disk && tar cf ../disk.tar --format=ustar ./*.txt)                          # new

$QEMU -machine virt -bios default -nographic -serial mon:stdio --no-reboot \
    -d unimp,guest_errors,int,cpu_reset -D qemu.log \
    -drive id=drive0,file=disk.tar,format=raw,if=none \                         # modified
    -device virtio-blk-device,drive=drive0,bus=virtio-mmio-bus.0 \
    -kernel kernel.elf

The tar command options used here are:

cf: Create tar file.

--format=ustar: Create in ustar format.

Tip

The parentheses (...) create a subshell so that cd doesn't affect in other parts of the script.

Tar file structure

A tar file has the following structure:

+----------------+
|   tar header   |
+----------------+
|   file data    |
+----------------+
|   tar header   |
+----------------+
|   file data    |
+----------------+
|      ...       |

In summary, a tar file is essentially a series of "tar header" and "file data" pair, one pair for each file. There are several types of tar formats, but we will use the ustar format (Wikipedia).

We use this file structure as the data structure for our file system. Comparing this to a real file system would be very interesting and educational.

Reading the file system

First, define the data structures related to tar file system in kernel.h:

#define FILES_MAX      2
#define DISK_MAX_SIZE  align_up(sizeof(struct file) * FILES_MAX, SECTOR_SIZE)

struct tar_header {
    char name[100];
    char mode[8];
    char uid[8];
    char gid[8];
    char size[12];
    char mtime[12];
    char checksum[8];
    char type;
    char linkname[100];
    char magic[6];
    char version[2];
    char uname[32];
    char gname[32];
    char devmajor[8];
    char devminor[8];
    char prefix[155];
    char padding[12];
    char data[];      // Array pointing to the data area following the header
                      // (flexible array member)
} __attribute__((packed));

struct file {
    bool in_use;      // Indicates if this file entry is in use
    char name[100];   // File name
    char data[1024];  // File content
    size_t size;      // File size
};

In our file system implementation, all files are read from the disk into memory at boot. FILES_MAX defines the maximum number of files that can be loaded, and DISK_MAX_SIZE specifies the maximum size of the disk image.

Next, let's read the whole disk into memory in kernel.c:

struct file files[FILES_MAX];
uint8_t disk[DISK_MAX_SIZE];

int oct2int(char *oct, int len) {
    int dec = 0;
    for (int i = 0; i < len; i++) {
        if (oct[i] < '0' || oct[i] > '7')
            break;

        dec = dec * 8 + (oct[i] - '0');
    }
    return dec;
}

void fs_init(void) {
    for (unsigned sector = 0; sector < sizeof(disk) / SECTOR_SIZE; sector++)
        read_write_disk(&disk[sector * SECTOR_SIZE], sector, false);

    unsigned off = 0;
    for (int i = 0; i < FILES_MAX; i++) {
        struct tar_header *header = (struct tar_header *) &disk[off];
        if (header->name[0] == '\0')
            break;

        if (strcmp(header->magic, "ustar") != 0)
            PANIC("invalid tar header: magic=\"%s\"", header->magic);

        int filesz = oct2int(header->size, sizeof(header->size));
        struct file *file = &files[i];
        file->in_use = true;
        strcpy(file->name, header->name);
        memcpy(file->data, header->data, filesz);
        file->size = filesz;
        printf("file: %s, size=%d\n", file->name, file->size);

        off += align_up(sizeof(struct tar_header) + filesz, SECTOR_SIZE);
    }
}

In this function, we first use the read_write_disk function to load the disk image into a temporary buffer (disk variable). The disk variable is declared as a static variable instead of a local (stack) variable. This is because the stack has limited size, and it's preferable to avoid using it for large data areas.

After loading the disk contents, we sequentially copy them into the files variable entries. Note that the numbers in the tar header are in octal format. It's very confusing because it looks like decimals. The oct2int function is used to convert these octal string values to integers.

Lastly, make sure to call the fs_init function after initializing the virtio-blk device (virtio_blk_init) in kernel_main:

void kernel_main(void) {
    memset(__bss, 0, (size_t) __bss_end - (size_t) __bss);
    WRITE_CSR(stvec, (uint32_t) kernel_entry);
    virtio_blk_init();
    fs_init();

    /* omitted */
}

Test file reads

Let's try! It should print the file names and their sizes in disk directory:

$ ./run.sh

virtio-blk: capacity is 2560 bytes
file: world.txt, size=0
file: hello.txt, size=22

Writing to the disk

Writing files can be implemented by writing the contents of the files variable back to the disk in tar file format:

void fs_flush(void) {
    // Copy all file contents into `disk` buffer.
    memset(disk, 0, sizeof(disk));
    unsigned off = 0;
    for (int file_i = 0; file_i < FILES_MAX; file_i++) {
        struct file *file = &files[file_i];
        if (!file->in_use)
            continue;

        struct tar_header *header = (struct tar_header *) &disk[off];
        memset(header, 0, sizeof(*header));
        strcpy(header->name, file->name);
        strcpy(header->mode, "000644");
        strcpy(header->magic, "ustar");
        strcpy(header->version, "00");
        header->type = '0';

        // Turn the file size into an octal string.
        int filesz = file->size;
        for (int i = sizeof(header->size); i > 0; i--) {
            header->size[i - 1] = (filesz % 8) + '0';
            filesz /= 8;
        }

        // Calculate the checksum.
        int checksum = ' ' * sizeof(header->checksum);
        for (unsigned i = 0; i < sizeof(struct tar_header); i++)
            checksum += (unsigned char) disk[off + i];

        for (int i = 5; i >= 0; i--) {
            header->checksum[i] = (checksum % 8) + '0';
            checksum /= 8;
        }

        // Copy file data.
        memcpy(header->data, file->data, file->size);
        off += align_up(sizeof(struct tar_header) + file->size, SECTOR_SIZE);
    }

    // Write `disk` buffer into the virtio-blk.
    for (unsigned sector = 0; sector < sizeof(disk) / SECTOR_SIZE; sector++)
        read_write_disk(&disk[sector * SECTOR_SIZE], sector, true);

    printf("wrote %d bytes to disk\n", sizeof(disk));
}

In this function, a tar file is built in the disk variable, then written to the disk using the read_write_disk function. Isn't it simple?

Design file read/write system calls

Now that we have implemented file system read and write operations, let's make it possible for applications to read and write files. We'll provide two system calls: readfile for reading files and writefile for writing files. Both take as arguments the filename, a memory buffer for reading or writing, and the size of the buffer.

#define SYS_READFILE  4
#define SYS_WRITEFILE 5

int readfile(const char *filename, char *buf, int len) {
    return syscall(SYS_READFILE, (int) filename, (int) buf, len);
}

int writefile(const char *filename, const char *buf, int len) {
    return syscall(SYS_WRITEFILE, (int) filename, (int) buf, len);
}

int readfile(const char *filename, char *buf, int len);
int writefile(const char *filename, const char *buf, int len);

Tip

It would be interesting to read the design of system calls in general operating systems and compare what has been omitted here. For example, why do read(2) and write(2) system calls in Linux take file descriptors as arguments, not filenames?

Implement system calls

Let's implement the system calls we defined in the previous section.

struct file *fs_lookup(const char *filename) {
    for (int i = 0; i < FILES_MAX; i++) {
        struct file *file = &files[i];
        if (!strcmp(file->name, filename))
            return file;
    }

    return NULL;
}

void handle_syscall(struct trap_frame *f) {
    switch (f->a3) {
        /* omitted */
        case SYS_READFILE:
        case SYS_WRITEFILE: {
            const char *filename = (const char *) f->a0;
            char *buf = (char *) f->a1;
            int len = f->a2;
            struct file *file = fs_lookup(filename);
            if (!file) {
                printf("file not found: %s\n", filename);
                f->a0 = -1;
                break;
            }

            if (len > (int) sizeof(file->data))
                len = file->size;

            if (f->a3 == SYS_WRITEFILE) {
                memcpy(file->data, buf, len);
                file->size = len;
                fs_flush();
            } else {
                memcpy(buf, file->data, len);
            }

            f->a0 = len;
            break;
        }
        default:
            PANIC("unexpected syscall a3=%x\n", f->a3);
    }
}

File read and write operations are mostly the same, so they are grouped together in the same place. The fs_lookup function searches for an entry in the files variable based on the filename. For reading, it reads data from the file entry, and for writing, it modifies the contents of the file entry. Lastly, the fs_flush function writes to the disk.

Warning

For simplicity, we are directly referencing pointers passed from applications (aka. user pointers), but this poses security issues. If users can specify arbitrary memory areas, they could read and write kernel memory areas through system calls.

File read/write commands

Let's read and write files from the shell. Since the shell doesn't implement command-line argument parsing, we'll implement readfile and writefile commands that read and write a hardcoded hello.txt file for now:

        else if (strcmp(cmdline, "readfile") == 0) {
            char buf[128];
            int len = readfile("hello.txt", buf, sizeof(buf));
            buf[len] = '\0';
            printf("%s\n", buf);
        }
        else if (strcmp(cmdline, "writefile") == 0)
            writefile("hello.txt", "Hello from shell!\n", 19);

It's easy peasy! However, it causes a page fault:

$ ./run.sh

> readfile
PANIC: kernel.c:561: unexpected trap scause=0000000d, stval=01000423, sepc=8020128a

Let's dig into the cause. According the llvm-objdump, it happens in strcmp function:

$ llvm-objdump -d kernel.elf
...

80201282 <strcmp>:
80201282: 03 46 05 00   lbu     a2, 0(a0)
80201286: 15 c2         beqz    a2, 0x802012aa <.LBB3_4>
80201288: 05 05         addi    a0, a0, 1

8020128a <.LBB3_2>:
8020128a: 83 c6 05 00   lbu     a3, 0(a1) ← page fault here (a1 has 2nd argument)
8020128e: 33 37 d0 00   snez    a4, a3
80201292: 93 77 f6 0f   andi    a5, a2, 255
80201296: bd 8e         xor     a3, a3, a5
80201298: 93 b6 16 00   seqz    a3, a3

Upon checking the page table contents in QEMU monitor, the page at 0x1000423 (with vaddr = 01000000) is indeed mapped as a user page (u) with read, write, and execute (rwx) permissions:

QEMU 8.0.2 monitor - type 'help' for more information
(qemu) info mem
vaddr    paddr            size     attr
-------- ---------------- -------- -------
01000000 000000008026c000 00001000 rwxu-a-

Let's dump the memory at the virtual address (x command):

(qemu) x /10c 0x1000423
01000423: 'h' 'e' 'l' 'l' 'o' '.' 't' 'x' 't' '\x00' 'r' 'e' 'a' 'd' 'f' 'i'
01000433: 'l' 'e' '\x00' 'h' 'e' 'l' 'l' 'o' '\x00' '%' 's' '\n' '\x00' 'e' 'x' 'i'
01000443: 't' '\x00' 'w' 'r' 'i' 't' 'e' 'f'

If the page table settings are incorrect, the x command will display an error or contents in other pages. Here, we can see that the page table is correctly configured, and the pointer is indeed pointing to the string "hello.txt".

In that case, what could be the cause of the page fault? The answer is: SUM bit in sstatus CSR is not set.

Accessing user pointers

In RISC-V, the behavior of S-Mode (kernel) can be configured through sstatus CSR, including SUM (permit Supervisor User Memory access) bit. When SUM is not set, S-Mode programs (i.e. kernel) cannot access U-Mode (user) pages.

Tip

This is a safety measure to prevent unintended references to user memory areas. Incidentally, Intel CPUs also have the same feature named "SMAP (Supervisor Mode Access Prevention)".

Define the position of the SUM bit as follows:

#define SSTATUS_SUM  (1 << 18)

All we need to do is to set the SUM bit when entering user space:

__attribute__((naked)) void user_entry(void) {
    __asm__ __volatile__(
        "csrw sepc, %[sepc]\n"
        "csrw sstatus, %[sstatus]\n"
        "sret\n"
        :
        : [sepc] "r" (USER_BASE),
          [sstatus] "r" (SSTATUS_SPIE | SSTATUS_SUM) // updated
    );
}

Tip

I explained that "the SUM bit was the cause", but you may wonder how you could find this on your own. It is indeed tough - even if you are aware that a page fault is occurring, it's often hard to narrow down. Unfortunately, CPUs don't even provide detailed error codes. The reason I noticed was, simply because I knew about the SUM bit.

Here are some debugging methods for when things don't work "properly":

Read the RISC-V specification carefully. It does mention that "when the SUM bit is set, S-Mode can access U-Mode pages."

Read QEMU's source code. The aforementioned page fault cause is implemented here. However, this can be as challenging or more so than reading the specification thoroughly.

Ask LLMs. Not joking. It's becoming your best pair programmer.

Troubleshooting is one of the major reasons why building an OS from scratch is a time sink and OS implementers are prone to giving up. However, more you overcome these challenges, the more you'll learn and ... be super happy!

Testing file reads/writes

Let's try reading and writing files again. readfile should display the contents of hello.txt:

$ ./run.sh

> readfile
Can you see me? Ah, there you are! You've unlocked the achievement "Virtio Newbie!"

Let's also try writing to the file. Once it's done the number of bytes written should be displayed:

> writefile
wrote 2560 bytes to disk

Now the disk image has been updated with the new contents. Exit QEMU and extract disk.tar. You should see the updated contents:

$ mkdir tmp
$ cd tmp
$ tar xf ../disk.tar
$ ls -alh
total 4.0K
drwxr-xr-x  4 seiya staff 128 Jul 22 22:50 .
drwxr-xr-x 25 seiya staff 800 Jul 22 22:49 ..
-rw-r--r--  1 seiya staff  26 Jan  1  1970 hello.txt
-rw-r--r--  1 seiya staff   0 Jan  1  1970 meow.txt
$ cat hello.txt
Hello from shell!

You've implemented a key feature "file system"! Yay!

Outro

Congratulations! You've completed the book. You've learned how to implement a simple OS kernel from scratch. You've learned about the basic concepts of operating systems, such as booting CPU, context switching, page table, user mode, system call, disk I/O, and file system.

Although it's less than 1000 lines, it must have been quite challenging. This is because you built the core of the core of the core of a kernel.

For those who are not satisfied yet and want to continue with something, here are some next steps:

Add new features

In this book, we implemented the basic features of a kernel. However, there are still many features that can be implemented. For example, it would be interesting to implement the following features:

A proper memory allocator that allows freeing memory.

Interrupt handling. Do not busy-wait for disk I/O.

A full-fledged file system. Implementing ext2 would be a good start.

Network communication (TCP/IP). It's not hard to implement UDP/IP (TCP is somewhat advanced). Virtio-net is very similar to virtio-blk!

Read other OS implementations

The most recommended next step is to read the implementation of existing OSes. Comparing your implementation with others is very educational.

My favorite is this RISC-V version of xv6. This is a UNIX-like OS for educational purposes, and it comes with an explanatory book (in English). It's recommended for those who want to learn about UNIX-specific features like fork(2).

Another one is my project Starina, a microkernel-based OS written in Rust. This is still very experimental, but would be interesting for those who want to learn about microkernel architecture and how Rust shines in OS development.

Feedback is very welcome!

If you have any questions or feedback, please feel free to ask on GitHub, or send me an email if you prefer. Happy your endless OS programming!