I was looking everywhere around the net for an article or sample code describing how to issue system calls on x86-64 architecture on Mac OS X Leopard but I couldn't find anything.
So, I decided to find it out myself. I spent yesterday night digging around XNU kernel, FreeBSD kernel and Mac OS X Libc source code and disassembling and debugging to find it out.
I want to sum it up and write down what I found along with Linux syscall convention.
OK, let's get started. We'll use "yasm" with Intel syntax as our assembler.
Linux
x86 32-bit convention with "int 0x80": System call number is stored in eax. Parameters are passed using registers ebx, ecx, edx, esi, edi, ebp. Return value is stored in eax, edx. Stack is not used. System call numbers can be found in "unistd_32.h".
Example:
SECTION .data
hello db 'Hello world!', 10
hellolen equ $ - hello
SECTION .text
global _start
_start:
mov eax, 4 ; SYS_write
mov ebx, 1 ; stdout
mov ecx, hello ; string
mov edx, hellolen ; string length
int 0x80 ; issue system call
mov eax, 1 ; SYS_exit
xor ebx, ebx ; exit code
int 0x80 ; issue system call
Test: yasm -f elf a.asm; ld a.o; ./a.out
x86 32-bit convention with "sysenter": It's possible in newer releases of Linux kernel to use Intel architecture's fast system call facility by means of "sysenter" and "sysexit" instructions. I managed to use it successfully, but I'm not very sure how it should be used. It has something to do with stack and requires more prepration in userspace before system call as "sysenter" doesn't do much itself. Since I'm not sure about it, I won't talk about it here.
UPDATE: I figured out why the kernel doesn't use the rcx register to pass forth parameter of the system call like a function call. The reason is, as documented in the Intel manual, the fact that syscall instruction puts lower 32 bit of RFLAGS in R11 and also saves RIP value to RCX. Therefore, previous RCX value is discarded and it can't be used as a syscall parameter. As a result, both Darwin and Linux use R10 to pass the forth parameter.
amd64 64-bit convention with "syscall": There's a new instruction supported by Intel 64 architecture CPUs running in long mode used to issue system calls. Notice that system call numbers are changed in this mode and are specified in "unistd_64.h". In this mode, you store the system call number in rax register, and parameters are passed just like you were calling a C function (rdi, rsi, rdx, r10, r8, r9) except forth parameter which is passed via r10 instead of rcx. Return value will be placed in rax, rdx. Again stack is not used.
Example:
SECTION .data
hello db 'Hello world!', 10
hellolen equ $ - hello
SECTION .text
global _start
_start:
mov rax, 1 ; SYS_write
mov rdi, 1 ; stdout
mov rsi, hello ; string
mov rdx, hellolen ; string length
syscall ; issue system call
mov rax, 60 ; SYS_exit
xor rdi, rdi ; exit code
syscall ; issue system call
Test: yasm -f elf64 a.asm; ld a.o; ./a.out
Mac OS X
Darwin is a Mach and BSD based operating system. Therefore, its kernel, XNU, supports BSD and Mach system calls. In 32-bit x86 architecture, it seems that there are different interrupts for different system calls (e.g. int 0x80 is for BSD syscalls and int 0x81 seems to be for Mach syscalls). I'll stick to BSD syscalls in this article. By the way, 32-bit mode seems to support sysenter/sysexit too, but I haven't tried.
x86 32-bit convention with "int 0x80": Just like other BSD based operating systems, such as FreeBSD, parameters are passed on the stack. Just note that it requires a 4 byte empty space on the stack. This can be achieved by manually adjusting the stack, pushing a dword or by wrapping int 0x80 instruction in a separate function and calling the function instead. Return value will be stored in eax, and the caller has the responsibility of removing the parameters from the stack. You have to push parameters in the reverse order to allow the kernel to retrieve them in the correct order. System call numbers are in "sys/syscall.h".
Example:
SECTION .data
hello db 'Hello world!', 10
hellolen equ $ - hello
SECTION .text
global start
start:
push dword hellolen ; string length
push dword hello ; string
push dword 1 ; stdout
mov eax, 4 ; SYS_write
sub esp, 4 ; 4 bytes scratch space
int 0x80 ; issue system call
add esp, 16 ; clean up the stack
mov eax, 1 ; SYS_exit
push dword 0 ; exit code
sub esp, 4 ; 4 bytes scratch space
int 0x80 ; issue system call
Test: yasm -f macho a.asm; ld a.o; ./a.out
x86_64 64-bit convention with "syscall": This is the thing I was searching for in the source code. It's basically similar to Linux amd64 system call convention. Unfortunately, I had trouble testing this method since kernel panics (see last post) made me reboot the system anytime I did something wrong. The key difference between Mac OS X and Linux in this syscall method is system call numbers. Since Mac OS X has to handle more than just BSD syscalls (It should be able to handle Mach calls too) and unlike interrupts, there's only one syscall instruction, it had to somehow differentiate between them. To do so, it uses the highest byte of the eax. Therefore, first 32 bits of rax remain unused, the next 8 bit (highest byte of eax) will show what kind of system call you're going to use (2 for BSD, 1 for Mach, for instance). The next 24 bits (low 3 bytes of rax) will indicate syscall number, which you'll find in "sys/syscall.h".
This code snippet is from XNU source code, "osfmk/mach/i386/syscall_sw.h":
#define SYSCALL_CLASS_NONE 0 /* Invalid */
#define SYSCALL_CLASS_MACH 1 /* Mach */
#define SYSCALL_CLASS_UNIX 2 /* Unix/BSD */
#define SYSCALL_CLASS_MDEP 3 /* Machine-dependent */
#define SYSCALL_CLASS_DIAG 4 /* Diagnostics */
Example: (essentially the same as Linux with a different syscall number)
SECTION .data
hello db 'Hello world!', 10
hellolen equ $ - hello
SECTION .text
global start
start:
mov rax, 0x2000004 ;(SYSCALL_CLASS_UNIX<<24)|SYS_write
mov rdi, 1 ; stdout
mov rsi, qword hello ; string
mov rdx, hellolen ; string length
syscall ; issue system call
mov rax, 0x2000001 ;(SYSCALL_CLASS_UNIX<<24)|SYS_exit
xor rdi, rdi ; exit code
syscall ; issue system call
Test: yasm -f macho64 a.asm; ld a.o; ./a.out
The interesting thing I noticed when using yasm is that in Mac OS X, when I run a yasm-assembled program with empty .data section or without a .data section at all and used the syscall instruction, dyld prints out an error message and the program exits. I've no idea why a dynamic linker should interfere with such a small independent program. I don't know much about Mach-O file format but it seems to be the reason dyld complains.