1# Building `sys/unix`
2
3The sys/unix package provides access to the raw system call interface of the
4underlying operating system. See: https://godoc.org/golang.org/x/sys/unix
5
6Porting Go to a new architecture/OS combination or adding syscalls, types, or
7constants to an existing architecture/OS pair requires some manual effort;
8however, there are tools that automate much of the process.
9
10## Build Systems
11
12There are currently two ways we generate the necessary files. We are currently
13migrating the build system to use containers so the builds are reproducible.
14This is being done on an OS-by-OS basis. Please update this documentation as
15components of the build system change.
16
17### Old Build System (currently for `GOOS != "linux"`)
18
19The old build system generates the Go files based on the C header files
20present on your system. This means that files
21for a given GOOS/GOARCH pair must be generated on a system with that OS and
22architecture. This also means that the generated code can differ from system
23to system, based on differences in the header files.
24
25To avoid this, if you are using the old build system, only generate the Go
26files on an installation with unmodified header files. It is also important to
27keep track of which version of the OS the files were generated from (ex.
28Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes
29and have each OS upgrade correspond to a single change.
30
31To build the files for your current OS and architecture, make sure GOOS and
32GOARCH are set correctly and run `mkall.sh`. This will generate the files for
33your specific system. Running `mkall.sh -n` shows the commands that will be run.
34
35Requirements: bash, go
36
37### New Build System (currently for `GOOS == "linux"`)
38
39The new build system uses a Docker container to generate the go files directly
40from source checkouts of the kernel and various system libraries. This means
41that on any platform that supports Docker, all the files using the new build
42system can be generated at once, and generated files will not change based on
43what the person running the scripts has installed on their computer.
44
45The OS specific files for the new build system are located in the `${GOOS}`
46directory, and the build is coordinated by the `${GOOS}/mkall.go` program. When
47the kernel or system library updates, modify the Dockerfile at
48`${GOOS}/Dockerfile` to checkout the new release of the source.
49
50To build all the files under the new build system, you must be on an amd64/Linux
51system and have your GOOS and GOARCH set accordingly. Running `mkall.sh` will
52then generate all of the files for all of the GOOS/GOARCH pairs in the new build
53system. Running `mkall.sh -n` shows the commands that will be run.
54
55Requirements: bash, go, docker
56
57## Component files
58
59This section describes the various files used in the code generation process.
60It also contains instructions on how to modify these files to add a new
61architecture/OS or to add additional syscalls, types, or constants. Note that
62if you are using the new build system, the scripts/programs cannot be called normally.
63They must be called from within the docker container.
64
65### asm files
66
67The hand-written assembly file at `asm_${GOOS}_${GOARCH}.s` implements system
68call dispatch. There are three entry points:
69```
70 func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)
71 func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr)
72 func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)
73```
74The first and second are the standard ones; they differ only in how many
75arguments can be passed to the kernel. The third is for low-level use by the
76ForkExec wrapper. Unlike the first two, it does not call into the scheduler to
77let it know that a system call is running.
78
79When porting Go to a new architecture/OS, this file must be implemented for
80each GOOS/GOARCH pair.
81
82### mksysnum
83
84Mksysnum is a Go program located at `${GOOS}/mksysnum.go` (or `mksysnum_${GOOS}.go`
85for the old system). This program takes in a list of header files containing the
86syscall number declarations and parses them to produce the corresponding list of
87Go numeric constants. See `zsysnum_${GOOS}_${GOARCH}.go` for the generated
88constants.
89
90Adding new syscall numbers is mostly done by running the build on a sufficiently
91new installation of the target OS (or updating the source checkouts for the
92new build system). However, depending on the OS, you may need to update the
93parsing in mksysnum.
94
95### mksyscall.go
96
97The `syscall.go`, `syscall_${GOOS}.go`, `syscall_${GOOS}_${GOARCH}.go` are
98hand-written Go files which implement system calls (for unix, the specific OS,
99or the specific OS/Architecture pair respectively) that need special handling
100and list `//sys` comments giving prototypes for ones that can be generated.
101
102The mksyscall.go program takes the `//sys` and `//sysnb` comments and converts
103them into syscalls. This requires the name of the prototype in the comment to
104match a syscall number in the `zsysnum_${GOOS}_${GOARCH}.go` file. The function
105prototype can be exported (capitalized) or not.
106
107Adding a new syscall often just requires adding a new `//sys` function prototype
108with the desired arguments and a capitalized name so it is exported. However, if
109you want the interface to the syscall to be different, often one will make an
110unexported `//sys` prototype, and then write a custom wrapper in
111`syscall_${GOOS}.go`.
112
113### types files
114
115For each OS, there is a hand-written Go file at `${GOOS}/types.go` (or
116`types_${GOOS}.go` on the old system). This file includes standard C headers and
117creates Go type aliases to the corresponding C types. The file is then fed
118through godef to get the Go compatible definitions. Finally, the generated code
119is fed though mkpost.go to format the code correctly and remove any hidden or
120private identifiers. This cleaned-up code is written to
121`ztypes_${GOOS}_${GOARCH}.go`.
122
123The hardest part about preparing this file is figuring out which headers to
124include and which symbols need to be `#define`d to get the actual data
125structures that pass through to the kernel system calls. Some C libraries
126preset alternate versions for binary compatibility and translate them on the
127way in and out of system calls, but there is almost always a `#define` that can
128get the real ones.
129See `types_darwin.go` and `linux/types.go` for examples.
130
131To add a new type, add in the necessary include statement at the top of the
132file (if it is not already there) and add in a type alias line. Note that if
133your type is significantly different on different architectures, you may need
134some `#if/#elif` macros in your include statements.
135
136### mkerrors.sh
137
138This script is used to generate the system's various constants. This doesn't
139just include the error numbers and error strings, but also the signal numbers
140and a wide variety of miscellaneous constants. The constants come from the list
141of include files in the `includes_${uname}` variable. A regex then picks out
142the desired `#define` statements, and generates the corresponding Go constants.
143The error numbers and strings are generated from `#include <errno.h>`, and the
144signal numbers and strings are generated from `#include <signal.h>`. All of
145these constants are written to `zerrors_${GOOS}_${GOARCH}.go` via a C program,
146`_errors.c`, which prints out all the constants.
147
148To add a constant, add the header that includes it to the appropriate variable.
149Then, edit the regex (if necessary) to match the desired constant. Avoid making
150the regex too broad to avoid matching unintended constants.
151
152### internal/mkmerge
153
154This program is used to extract duplicate const, func, and type declarations
155from the generated architecture-specific files listed below, and merge these
156into a common file for each OS.
157
158The merge is performed in the following steps:
1591. Construct the set of common code that is idential in all architecture-specific files.
1602. Write this common code to the merged file.
1613. Remove the common code from all architecture-specific files.
162
163
164## Generated files
165
166### `zerrors_${GOOS}_${GOARCH}.go`
167
168A file containing all of the system's generated error numbers, error strings,
169signal numbers, and constants. Generated by `mkerrors.sh` (see above).
170
171### `zsyscall_${GOOS}_${GOARCH}.go`
172
173A file containing all the generated syscalls for a specific GOOS and GOARCH.
174Generated by `mksyscall.go` (see above).
175
176### `zsysnum_${GOOS}_${GOARCH}.go`
177
178A list of numeric constants for all the syscall number of the specific GOOS
179and GOARCH. Generated by mksysnum (see above).
180
181### `ztypes_${GOOS}_${GOARCH}.go`
182
183A file containing Go types for passing into (or returning from) syscalls.
184Generated by godefs and the types file (see above).
View as plain text