Arroyo.cc / tech / ocaml / DevNotes: OpenGL programming in OCaml

I'm going to try starting a new section of my blog that is meant for raw, unpolished notes, logged while I'm working on something. They are mostly un-edited, and just meant as a way for myself to track and revisit my own work. A lab notebook of sorts. I find that writing things down helps me to think, helps me stay motivated, and keeps me more organized.

I'm making them publicly available on the off chance that my ramblings or struggles could be useful to someone else.

OpenGL programming in OCaml

Over the past ~year I've really become fond of OCaml as a programming language. I tried using it for an advent of code challenge and really appreciated the expressiveness of its type system, its practicality, its performance, and its community. So naturally, I want to write a game from scratch in it :).

Workspace tour

I spend a lot of time and effort trying to remove tedium from my workflow. I feel it's important, because these are side projects that I do on my own time and expense. There is no external motivation like a paycheck for me to work on these things, so I try to minimize the boring tasks that would wear down my motivation over time. I find it challenging to finish projects, so this is a real concern.

My first goal was to get some pixels on the screen, nothing more. This is the result. On the left side is the text editor acme from plan9port, which I am running on Linux. I still use Xorg instead of Wayland, because acme's mouse warping does not work in Xwayland. Maybe at some point I will port devdraw (the display server for plan9port) to wayland and include mouse warping. My window manager is dwm.

In the top right is a terminal window, and in the bottom right is feh running

feh --auto-zoom --scale-down --auto-reload _build/default/output.tga

from an acme window. It will watch the targa image file _build/default/output.tga, relative to my project directory, and reload it whenever it is written to, fitting it to the current window dimensions. My OCaml project is built with dune, the de-facto standard build system for OCaml (though there are still a decent number of people who still use good old make). Here is my dune file:

(executable
 (name raymarch)
 (libraries tgls.tgl4 tsdl fmt)
 (preprocess (pps ppx_blob))
 (preprocessor_deps frag.glsl vert.glsl))

(rule
 (target output.tga)
 (action
  (with-stdout-to %{target}
   (run ./raymarch.exe))))

Starting from the top, I define an executable, raymarch. It depends on the Tgls and Tsdl libraries, providing bindings to OpenGL for hardware-accelerated graphics, and SDL2 for interfacing with the window manager, respectively. I also use the Fmt library which provides a nicer pretty-printing API than the standard library's Format module.

I run the ppx_blob preprocessor, which converts the following lines:

let frag = [%blob "frag.glsl"]
let vert = [%blob "vert.glsl"]

into

let frag = "<content of the file frag.glsl>"
let vert = "<content of the file vert.glsl>"

allowing me to keep my shader programs in separate files, while still embedding them in my binary. They are so simple that I could just define it in my OCaml file, but I am thinking further down the line.

Next, I define a rule stanza that generates the image file that is being watched by feh. My program is designed to write a tga image to standard output. It does this by rendering to a RenderBuffer, then using glReadPixels to copy the pixels, and finally writes the data out in TGA format. I chose TGA because it is so simple; an 18-byte header followed by RGB (well, BGR) data.

  let hdr = Bytes.make 18 (Char.chr 0) in

  Bytes.set_int8 hdr 2 uncompressed_rgb;
  Bytes.set_int16_le hdr 8 x_origin;
  Bytes.set_int16_le hdr 10 y_origin;
  Bytes.set_int16_le hdr 12 width;
  Bytes.set_int16_le hdr 14 height;
  Bytes.set_int8 hdr 16 bits_per_pixel;

  output_bytes stdout hdr;
  output_bytes stdout pixel_data

In another acme window, I am running

win dune build -w output.tga

This will rebuild the output.tga target (which is being watched by feh) every time it detects a change to one of its dependencies. So, every time I change one of the source files, the image output is updated. This provides me with a nice feedback loop, where I can immediately see the consequences of any change I make.

This setup, of building an image file and watching it with an image viewer, is something I picked up while following the Raytracing in one weekend book, using OCaml. That book describes a CPU-based ray tracer which writes its output to a PPM image file. It was very helpful getting immediate feedback with every change I made. It allowed me to experiment and kept me motivated to finish the project.

Since I am using OpenGL and SDL now, you might wonder why I don't just open a window and write directly to that. While I will probably do that in the future, there are a few benefits to generating an image file instead;

So while I am still working on just rendering graphics, I will probably stick to this approach, and change it once I start accepting input and need an event loop.

From my experience using Go, I've come to appreciate automated code formatting. There is a code formatter for OCaml, called ocamlformat. I wrote a program, acme-autoformat, which runs the content of an acme window through a process whenever the Put command is executed. Then I run a script, OcamlFmt, which looks like this:

#!/bin/sh
exec /usr/local/bin/acme-autoformat -r '\.ml[ily]?$' \
	-- ocamlformat --name='{{.Basename}}' --enable-outside-detected-project -

This script will automatically detect writes to any OCaml files in acme and run ocamlformat on them, so I get automatic, instant code formatting, and feedback on syntax errors.

I run all of these "helper" programs in their own acme windows by prefixing them with the win command:

win dune build -w output.tga
win feh --auto-zoom --scale-down --auto-reload _build/default/output.tga
win OcamlFmt

I could run them directly, without a window, by middle clicking on the commands. Their output would go to the global +Errors window. While running them in their own windows has the benefit of keeping their output separate, the main reason I run them in separate windows is because I can dump acme's current state to a file using the Dump command, and when I load that file (acme.dump by default) using acme's -l flag, acme will correctly start all of these processes for me when it recreates the windows. Without running them under the win command, they would not be started. This gives me the ability to easily save and resume my work if I need to close acme (for example if I need to reboot my machine).

I keep various commands like the win commands above in files called guide. If they are relevant to a specific project like this one, the guide file will live in the current directory. If they are useful no matter what I'm doing, I put them in the directory /acme/edit.

Notes on the Ocaml program

The current program is very basic. The rendering call is

Gl.draw_arrays Gl.triangles 0 6

and the vertex data is

let vertices =
  [
    (* top left triangle *)
    -1, -1;
    -1, +1;
    +1, +1;

    (* bottom right triangle *)
    -1, -1;
    +1, +1;
    +1, -1
  ]

Together with the simple pass-through vertex shader (which you can see in the screenshot above), it simply covers the whole screen with two triangles. As a result, the fragment shader will run for every pixel on the screen. In effect, I am recreating shadertoy.

The fragment shader is very basic; it just uses the pixel position and some trig functions to choose a color, giving us a pretty gradient. From this dev loop I can experiment with different rendering techniques like ray tracing and ray marching.

I will have to figure out how to send the scene data to the GPU. With a traditional polygon-based renderer, the most obvious approach is to use a vertex buffer filled with a triangle mesh for the scene. If I want to do my rendering work in the fragment shader, I need to decide on a format for scene data. I will need to survey a few renderers to see what they do.

There was a ton of boilerplate involved with setting up this very simple program. I think this is pretty common for graphics programming, since the APIs need to be sufficiently low-level and flexible enough to support a wide variety of hardware and use cases. I could save a lot of work by using an existing engine. However I think going through this process of writing something from scratch will help me appreciate what is useful in an engine and make it easier to choose one in the future.

The Tgls OpenGL bindings are more or less a 1:1 binding from OCaml to OpenGL. As a result, it is not very strongly typed; most OpenGL functions work with integers and arrays. For example, glBindVertexArray accepts an integer, which should be returned by glGenVertexArrays. However, in the C library, there is nothing to stop you from passing an arbitrary integer to this function, or an integer returned by some other function such as glGenBuffers. The OCaml bindings reflect this;

val gl_bind_vertex_array : int -> unit

You could envision a binding that used Ocaml's type system to prevent incorrect usage, for example:

type vertex_array_id = private int
val gen_vertex_array : int -> vertex_array_id array
val bind_vertex_array : vertex_array_id -> unit

However, it is not clear to me whether the OpenGL API is consistent enough to apply this technique generally. Moreover, these bindings are generated automatically using a structured specification, and I do not think the specification provides enough detail to indicate that the inputs of one function come from the outputs of another function. In short, it would be a ton of effort to generate bindings like this, it would probably require some heuristics (like matching Gen* functions to Bind* functions), and would almost certainly add maintenance burden to keeping up with OpenGL version updates. I don't think it's worth the tradeoff.

The program as it stands is very "scrappy", and was written quickly to get the pixels on the screen. There is plenty of opportunity to add abstractions that get rid of boilerplate and improve readability. Here's what's top of mind for me;

An "object manager"

Most OpenGL "objects", like buffers, vertex arrays, framebuffers, and the like follow a pretty consistent pattern:

glGen<Object>s N -- allocate N <object>s
glDelete<Object>s N [objects] -- free these <object>s

glCreate<Object> -- allocate 1 <object>
glDelete<Object> -- free 1 <object>

where between the allocation and deallocation, you can manipulate the object. Currently my program is simple and short-lived enough that I'm not too worried about leaking these resources, since they will be cleaned up when the program exits anyway. However, for a long-running game I don't want to leak these resources.

Ocaml allows for the creation of binding operators that can be used to bracket the execution of code. I could use this to automatically allocate and deallocate resources within the scope of a function. For example,

module R : sig
  (** resource manager **)
  type 'a t = { create: unit -> 'a; destroy: 'a -> unit }
  val bind : 'a t -> ('a -> 'b) -> 'b
end

let ( let* ) = R.bind in
let* frag = R.{
  create=fun () -> Gl.(create_shader fragment_shader);
  delete=Gl.delete_shader;
}
in
(* code using resources goes here *)

Better error handling

Currently error checking is very tedious and optional; you call the Gl.get_error function which returns the value of the most recently encountered error, as an integer enumeration. For other operations, like compiling a shader or linking a shader program, you call the Gl.get_<resource>iv function with the appropriate property argument to check the result, then use the Gl.get_<resource>_info_log function to get any details.

It is common in Ocaml to make error checking mandatory by using an Option or Result type. For example, if a function returns a result, you must pattern match it like so:

match (fn args) with
| Ok v -> (* successful case *)
| Error e -> (* failed case *)

and if you leave out the Error case, you will get a compile-time error. You can combine this with a binding operator:

let ( let* ) = Result.bind in
let* v = fn args in
(* successful case *)

Doing this will make your function return a result type which callers would have to check or bubble up to their callers. I would like to follow this convention where possible. Similar to the resource management example above, I should be able to provide an abstraction that allows us to adapt the existing API into one that requires error checking. Alternatively or in combination with this approach, in places where I do not want to allocate a result type, or where the only viable way to handle an error is to exit the program, I can raise an exception.

While there are performance implications to using a Result type, I do not think they will be relevant, because most of this code will be run once while setting up the rendering pipeline. We will not do this in code that runs every frame. My performance objective will be to minimize allocations in per-frame code, and avoid major heap allocations altogether. Ocaml has a garbage collector, which is generally considered to be very good, but any garbage collection can be a source of random delay, which is not good for code that needs to run every 15 milliseconds. I may have to get creative to achieve this goal, and I look forward to the challenge. This will be an opportunity for me to get more familiar with Ocaml and performance profiling. I want the end result to run well on my modest computer which uses an integrated GPU.

Next steps

From here I will probably shoot for a few different milestones roughly in order of difficulty:

At that point I will pause and re-evaluate what I want to do. I see these projects as more of a warm-up exercise to get me familiar with graphics programming, of which which I've done precious little. I may end up going in a completely different direction. This experience may also lead me to pursue some other graphical, but not gaming-related projects I've had on the back-burner for some time:

Inspiration

Like many, my imagination has been captured by voxels. I like the idea of constructing a world that, like our own, is made up of solid objects rather than polygonal surfaces. I watched this technical breakdown of the game Teardown which I found fascinating. I am inspired by games like Dwarf Fortress, Minecraft, and roguelikes.

I want to explore different ways to represent a world and how they affect gameplay and storytelling. As a simple example, imagine a key-based puzzle that you would commonly see in older Resident Evil games: in order to access the secret lab and kill the abomination, you must:

Here are some alternative "real-world" solutions:

  1. Burn down the blue house and walk into the basement.
  2. Grab a shovel and dig a hole into the lab.
  3. Dig a hole into the lab and flood it with a water hose, drowning the boss.

Any one of these solutions could be scripted by the game designer. But there could be hundreds of other solutions that the designer never thought of.

Now imagine that not only are these solutions available to you, but any number of physically-based counter-solutions are available to the game's AI. An intelligent enemy could, for example, smear poison on a door knob. Or start a fire outside to trap you in the building. A giant worm could dig out the foundation and cause the building you're in to collapse. A strong enemy could throw you through a wall into a dangerous room, down a pit, or uproot the tree you're hiding in.