Concurrent Modification and Execution of Instructions (CMODX) for RISC-V Linux¶

CMODX is a programming technique where a program executes instructions that were modified by the program itself. Instruction storage and the instruction cache (icache) are not guaranteed to be synchronized on RISC-V hardware. Therefore, the program must enforce its own synchronization with the unprivileged fence.i instruction.

However, the default Linux ABI prohibits the use of fence.i in userspace applications. At any point the scheduler may migrate a task onto a new hart. If migration occurs after the userspace synchronized the icache and instruction storage with fence.i, the icache on the new hart will no longer be clean. This is due to the behavior of fence.i only affecting the hart that it is called on. Thus, the hart that the task has been migrated to may not have synchronized instruction storage and icache.

There are two ways to solve this problem: use the riscv_flush_icache() syscall, or use the PR_RISCV_SET_ICACHE_FLUSH_CTX prctl() and emit fence.i in userspace. The syscall performs a one-off icache flushing operation. The prctl changes the Linux ABI to allow userspace to emit icache flushing operations.

As an aside, “deferred” icache flushes can sometimes be triggered in the kernel. At the time of writing, this only occurs during the riscv_flush_icache() syscall and when the kernel uses copy_to_user_page(). These deferred flushes happen only when the memory map being used by a hart changes. If the prctl() context caused an icache flush, this deferred icache flush will be skipped as it is redundant. Therefore, there will be no additional flush when using the riscv_flush_icache() syscall inside of the prctl() context.

prctl() Interface¶

Call prctl() with PR_RISCV_SET_ICACHE_FLUSH_CTX as the first argument. The remaining arguments will be delegated to the riscv_set_icache_flush_ctx function detailed below.

int riscv_set_icache_flush_ctx(unsigned long ctx, unsigned long scope)¶: Enable/disable icache flushing instructions in userspace.

Parameters

unsigned long ctx: Set the type of icache flushing instructions permitted/prohibited in userspace. Supported values described below.
unsigned long scope: Set scope of where icache flushing instructions are allowed to be emitted. Supported values described below.

Description

Supported values for ctx:

PR_RISCV_CTX_SW_FENCEI_ON: Allow fence.i in user space.
PR_RISCV_CTX_SW_FENCEI_OFF: Disallow fence.i in user space. All threads in a process will be affected when scope == PR_RISCV_SCOPE_PER_PROCESS. Therefore, caution must be taken; use this flag only when you can guarantee that no thread in the process will emit fence.i from this point onward.

Supported values for scope:

PR_RISCV_SCOPE_PER_PROCESS: Ensure the icache of any thread in this process
is coherent with instruction storage upon migration.
PR_RISCV_SCOPE_PER_THREAD: Ensure the icache of the current thread is
coherent with instruction storage upon migration.

When scope == PR_RISCV_SCOPE_PER_PROCESS, all threads in the process are permitted to emit icache flushing instructions. Whenever any thread in the process is migrated, the corresponding hart’s icache will be guaranteed to be consistent with instruction storage. This does not enforce any guarantees outside of migration. If a thread modifies an instruction that another thread may attempt to execute, the other thread must still emit an icache flushing instruction before attempting to execute the potentially modified instruction. This must be performed by the user-space program.

In per-thread context (eg. scope == PR_RISCV_SCOPE_PER_THREAD) only the thread calling this function is permitted to emit icache flushing instructions. When the thread is migrated, the corresponding hart’s icache will be guaranteed to be consistent with instruction storage.

On kernels configured without SMP, this function is a nop as migrations across harts will not occur.

Example usage:

The following files are meant to be compiled and linked with each other. The modify_instruction() function replaces an add with 0 with an add with one, causing the instruction sequence in get_value() to change from returning a zero to returning a one.

cmodx.c:

#include <stdio.h>
#include <sys/prctl.h>

extern int get_value();
extern void modify_instruction();

int main()
{
        int value = get_value();
        printf("Value before cmodx: %d\n", value);

        // Call prctl before first fence.i is called inside modify_instruction
        prctl(PR_RISCV_SET_ICACHE_FLUSH_CTX_ON, PR_RISCV_CTX_SW_FENCEI, PR_RISCV_SCOPE_PER_PROCESS);
        modify_instruction();
        // Call prctl after final fence.i is called in process
        prctl(PR_RISCV_SET_ICACHE_FLUSH_CTX_OFF, PR_RISCV_CTX_SW_FENCEI, PR_RISCV_SCOPE_PER_PROCESS);

        value = get_value();
        printf("Value after cmodx: %d\n", value);
        return 0;
}

cmodx.S:

.option norvc

.text
.global modify_instruction
modify_instruction:
lw a0, new_insn
lui a5,%hi(old_insn)
sw  a0,%lo(old_insn)(a5)
fence.i
ret

.section modifiable, "awx"
.global get_value
get_value:
li a0, 0
old_insn:
addi a0, a0, 0
ret

.data
new_insn:
addi a0, a0, 1

The Linux Kernel

Contents

This Page

Concurrent Modification and Execution of Instructions (CMODX) for RISC-V Linux¶

prctl() Interface¶