This recipe will use the GLSL shading language with version 3.3. It assumes that all the vertices are stored in Vertex Buffer Object (VBO). The vertex shader program is applied on every vertex that is contained within VBO.

To prepare the vertex shader, you'll need to create the shader object first:

local shader_stage = gl_enum.GL_VERTEX_SHADER
local shader_object = gl.CreateShader(shader_stage)

The shader programs code can be stored in a text file or you can submit it directly as a string value. This recipe will use the latter method. The following source code will define the basic vertex shader:

local shader_source = [[
//Requires GLSL 3.3 at least
#version 330

//Input variables – vertex attributes
layout (location = 0) in vec3 VertexPosition;
layout (location = 1) in vec4 VertexColor;
layout (location = 2) in vec2 VertexTexCoord;

//Output variables for later shader stages
out VertexData {
  vec4 Color;
  vec2 TexCoord;
} outData;

//Application variable
uniform mat4 matrix;

//Entry function for vertex shader
void main(){
  gl_Position = matrix * vec4(VertexPosition.xyz, 1.0);
  outData.Color = vec4(VertexColor.rgba);
  outData.TexCoord = vec2(VertexTexCoord.st);
}
]]

Now you can load and compile this source code into the shader object:

gl.ShaderSource(shader_object, shader_source)
gl.CompileShader(shader_object)

Be sure to always check for the compilation status. The production version of the game should use at least some kind of message logging mechanism, so you can store error messages into the bug report file, which is always handy. In order to store the messages, use the following code:

local status = gl.GetShaderiv(shader_object, gl_enum.GL_COMPILE_STATUS)
if status == gl_enum.GL_FALSE then
  local compilation_status = gl.GetShaderInfoLog(shader_object)
  error("Vertex Shader compilation failed: "..compilation_status)
end

After these steps, you can finally link the vertex shader with the shader program.

It's recommended to specify the required shader specification version at the beginning of the shader source code. This is done with preprocessor macro:

#version VERSION_NUMBER

The version number is always in the form of three digits. For example, for GLSL version 1.5, one would use a number 150. The good thing is that OpenGL shaders are backwards compatible. This way you can use older GLSL specifications even on newer graphic cards.

The input variables for the vertex shader can have two forms. You can use either the uniform variables or the vertex attributes stored in VBO. This recipe uses the vertex attributes with layout specification. Each vertex attribute layout number represents a VBO identifier. This way the GLSL shader knows what VBO to use:

layout (location = 0) in vec3 VertexPosition;

Optionally, layouts can be set explicitly in Lua with the following code:

local attribute_name = "VertexPosition"
gl.BindAttribLocation(shader_program, layout_index, attribute_name)

The vertex shader has to pass results to the next stage. The output variables can be specified in two ways. The first one uses direct output variable specification:

out vec4 VertexColor;

The second way is more preferred as it offers a certain level of consistency:

out VertexData {
  vec4 Color;
  vec2 TexCoord;
} outData;

This is also called as an interface block. Interface blocks are shared between shader stages. However, this will work only if the interface block shares the same interface name, variable name, and also their order and types have to be the same. Notice that the interface block name VertexData is specified right after our qualifier. The local interface name outData is valid only in the local context. You can refer to these variables as if you were using C structures. Therefore, to set the vertex color, you would use the following code:

outData.Color = vec4(...);

You may also omit the local interface name. In that case, you can refer to the interface variables in this fashion:

Color = vec4(...);

The uniform variables are specified with the qualifier uniform:

uniform mat4 matrix;

The last and the most important part of vertex shader is the main function. This sample does simple matrix transformation on the vertex position:

gl_Position = matrix * vec4(VertexPosition.xyz, 1.0);

It takes three coordinates x, y, and z in the form of a vector with three elements. This vector is extended to contain four elements with the vec4 type declaration. Notice that the forth element is set to 1. This is because the matrix multiplication rule must be applied. The A x B matrix can be multiplied only with matrix B x C. This will result in A x C matrix. In this case, you are using 4 x 4 matrix and you multiply it with the 4 x 1 matrix. Vectors with N elements can be seen as matrices with the size of N x 1. The result of this is a 4 x 1 matrix or a vector with four elements.

The other attributes such as the vertex color of texture coordinates are passed unchanged:

outData.Color = vec4(VertexColor.rgba);
outData.TexCoord = vec2(VertexTexCoord.xy);

The vector data type in GLSL can contain 2, 3, or 4 components. As you've already seen, components are accessed by their names x, y, z and w. This is also called swizzling. That's because you can use any combination of components as long as you maintain the correct output data type. Therefore, the following code is completely valid:

vec2 vector1;
vec3 vector2 = vector1.xxy;
vec4 vector3 = vector2.zwyx;
vec4 vector4 = vector1.xxxx;

You can use swizzling even on the left side (also known as l-value) of the value assignment:

vec4 vector1;
vector1.xz = vec2(1.0, 2.0);

Alternatively, you can use color component names r, g, b, and a; or even texture coordinate names s, t, p, and q.