Introduction

Versions

This documentation is on the latest version of Astra and will store a version for every x.0.0 release. The other versions will be listed below here when available.

About

ArkForge Astra is an easy-to-use, fault-tolerant, extendible, and high performant web server runtime targeting Lua (5.1 upto 5.4), LuaJIT (LuaJIT and LuaJIT 5.2), and Luau, and built upon Rust. Astra takes advantages of the Rust's performance, correctness and memory safety offering, and with a rich standard library making full use of the Tokio async runtime, to write fault-tolerant and no-build servers with ease.

Currently Astra is used within the ArkForge for some internal and external products. However being very young and early project, it lacks battle testing. Even so, it does not mean you cannot build software with it, as the base, Rust and Lua, is already mature and Astra is a thin wrapper over it.

Philosophy

The goal is to have the cake and eat it too. Obtaining the low-level advantages of Rust while having the iteration, ease and development speed of Lua. This way you can both have a small runtime and iterate over your products and servers with a very simple CI setup that ships in seconds, or even direct SSH. This is also called no-build, as there is no building and packaging stage required.

In an ideal world, Astra will be able to handle majority of the common use cases, which are basic REST servers that uses a PostgreSQL or SQLite DB if needed, single server instance, and manage hundreds of thousands of users per second upon it.

Astra's development style is to be as minimalist and simple as we can afford. Simplicity means decreasing as many steps as possible between the core developers and someone completely new to the project being able to pick it up and start changing it to their needs. However we do add complexity when it is required as well. Keeping the minimalistic development style also means we use minimal number of tools, and if we do use a tool, it should not be too foreign from the source. The result of this is an output of a single binary, a single lua prelude that includes batteries along with it; still having all of the goodies that the Lua language provides.

Getting Started

Dev environment

For development, we recommend visual studio code (or alternative) along with the lua extension.

After your setup is complete, you will want to obtain a prebuilt binary of the runtime from the releases page. Alternatively you can get it with wget as well. For example for the latest version with LuaJIT VM:

wget https://github.com/ArkForgeLabs/Astra/releases/latest/download/astra-luajit-linux-amd64

There are also windows binaries available if you are working on Windows, however we mostly assume your web server code will likely run linux, hence more support is geared towards it. Although the final code that runs should run well, agnostic of the platform it was written it (with the exception of OS specific additions).

After getting your binary on linux, you'll want to change permissions to make it executable:

chmod +x astra-luajit-linux-amd64

Alternatively you can also install through cargo tool, if you have it installed:

cargo install lua-astra

Each release likely contains updates to the packaged bundle lua code that contains definitions you might need during development which you can omit and ignore during production release. You can obtain them through:

./astra-luajit-linux-amd64 export-bundle

which will create a new file called astra_bundle.lua. You can include it in your code to get intellisense and references for the available functions. The import will be ignored during runtime as Astra will use it's own packaged bundle internally instead. There are some pure lua utilities for example table validation, ... which are also included in the bundle by default, but you can ignore them if you wish.

Interal dev environment

If you want to extend or modify Astra, you will need Rust and some form of C compiler such as clang or gcc to be installed on the latest versio. You will also need to make sure a C linker, Cargo, clippy and rust-analyzer components are also installed. These components often are packaged alongside the basic installation. Your IDE may depend on whichever you are comfortable with as Rust have amazing support everywhere.

Then you'll need to clone the repository:

git clone https://github.com/ArkForgeLabs/Astra

And from there, in the src folder, you can begin your contribution and development.

HTTP Server

Astra offers HTTP1/2 web server through the axum project. On the Lua's side, the server holds configuration and route details, which are then sent to the Rust for running them. Since it is running on Tokio, it can take advantage of all available resources automatically, making it easy for vertical scaling. Throughout this documentation, the word server is used to describe an HTTP web server table on the Lua's side. You can create one as such:

-- create a new server with
local server = Astra.http.server.new()

-- run the server with
server:run()

Configuration

Astra can be configured in a few ways for runtime. As of now there is no native TLS/SSL support and needs a reverse proxy such as Caddy to handle that. Check Deployment for more information.

However every configuration option will be available at the server instead. For example, changing the compression, port and hostname is as such:

-- configure the server with
server.compression = false
server.port = 8000
server.hostname = "0.0.0.0"

You can also configure other languages that compiles to Lua such as Fennel. Astra's api is for pure Lua however, so it will be up to you to make type definitions and make sure it can call the right functions and tables.

Routes

The server holds all of the route details. The routes are loaded at the start of the runtime and cannot be dynamically modified later on. There are also methods within the server that makes it easy to add new routes. For example:

-- A simple GET index route with text return
server:get("/", function()
    return "hello from default Astra instance! " .. Astra.version
end)

The syntax are as follows:

server:ROUTE_TYPE(ROUTE_PATH, CALLBACK);

-- Where callback is:
function(request?, response?);

The following route types are supported as of now:

  • GET
  • POST
  • PUT
  • PATCH
  • PARSE
  • DELETE
  • OPTIONS
  • TRACE

All lowercase and snake_case when calling with astra of course. There are two additional ones available:

  • STATIC_DIR
  • STATIC_FILE

Which does as expected, serves a file or directory over a route.

Route Logic

Each route function needs a callback which contains a route's logic. This callback function optionally can have two arguments: request and response respectively, and may optionally have a return.

Interally requests and responses are each a struct in Rust initialized but not parsed/deserialized beforehand. This is to save performance overhead of serialization. However its content and be modified or accessed through their methods. We will discuss them later on.

Return types of the callback can optionally be either empty, string, or a table. The table responses are parsed in Rust and serialized to JSON, and then returned. Empty responses does not include any content. Responses, or lack of them, are by default sent with status code of 200.

Requests

Requests are provided as the first argument of the route callbacks as a table (not deseralized). Each request in the route callbacks can be accessed through its methods. The following methods are available:

  • body: Body
  • headers: table<string, string>
  • uri: string
  • queries: table<any, any>
  • method: string
  • multipart: Multipart

where Body has:

  • text: string
  • json: table

and where Multipart has:

  • save_file(file_path: string | nil)

Example:

server:get("/", function(req)
    -- access the headers
    pretty_print(req:headers())

    -- print the body as text
    print(req:body():text())
end)

Responses

Responses are the second argument provided in the route callback. They allow you to modify the response to the way you want. Each response has the default 200 OK status along content header based on your response. The following methods are available:

  • set_status_code(status_code: number)
  • set_header(key: string, value: string)
  • remove_header(key: string)
  • get_headers(): table<string, string>

Example:

server:get("/", function(req, res)
    -- set header code
    res:set_status_code(300)
    -- set headers
    res:set_header("header-key", "header-value")

    return "Responding with Code 300 cuz why not"
end)

The headers, as stated, will include content type when sending to user, but can be changed while setting the type yourself.

Cookies

Cookies allow you to store data on each HTTP request, if supported. Astra does not currently support signed and private cookies. You can create a new cookie by getting it from a request:

server:get("/", function(request)
    local cookie = request:new_cookie("key", "value")

    return "HEY"
end)

You can also get a previously set cookie:

local cookie = request:get_cookie("key")

After modification or creation, they will have no effect unless you set them to the response

response:set_cookie("key", cookie)

And similary, remove them with

response:remove_cookie("key")

Each cookie contains extra details and functions which are as follows:

set_name(cookie: Cookie, name: string)
set_value(cookie: Cookie, value: string)
set_domain(cookie: Cookie, domain: string)
set_path(cookie: Cookie, path: string)
set_expiration(cookie: Cookie, expiration: number)
set_http_only(cookie: Cookie, http_only: boolean)
set_max_age(cookie: Cookie, max_age: number)
set_permanent(cookie: Cookie)
get_name(cookie: Cookie): string?
get_value(cookie: Cookie): string?
get_domain(cookie: Cookie): string?
get_path(cookie: Cookie): string?
get_expiration(cookie: Cookie): number?
get_http_only(cookie: Cookie): boolean?
get_max_age(cookie: Cookie): number?

Deployment

You can follow the steps covered in Configuration to setup the Astra itself.

Astra does not support TLS/SSL as of yet, but may support by the 1.0 release. However generally a reverse proxy service is recommended for deployment. We recommend Caddy as it is easy to setup and use, especially for majority of our, and hopefully your, usecases. What caddy also does is automatically fetching TLS certificates for your domain as well which is always a good idea. You can install caddy through your system's package manager.

Then open a new file with the name Caddyfile with the following content:

your_domain.tld {
    encode zstd gzip
    reverse_proxy :8080
}

and change your_domain.tld to your domain, and :8080 to the port you have set for your server. After this, make sure your 443 and 80 ports are open through your firewall. For a linux server running ufw you can open them by:

sudo ufw allow 80
sudo ufw allow 443

And finally run the caddy:

caddy run

Make sure your server is running before that. That is pretty much it for the basic deployment.

Fault Tolerance

Astra ensures fault tolerance through several methods internally and offers guidence on how you can ensure it on the Lua's endpoint as well.

Fault-tolerance essentially describes the ability to tolerate crashing errors and continue execution whereas otherwise caused the server to shut down. In Astra's internal case, this is ensured by removing all of the crashing points and handle every error that could occur during runtime. This was achieved through denying unwraps and expects throughout the codebase for the runtime. However there are still crashes on startup for cases that needs to be looked into, such as Lua parsing and errors, bad path, and/or system issues such as port being unavailable or unauthorized for Astra.

In Lua however, the errors are usually crash by default, which are still tolerated with Astra and does not shutdown the server. To handle the errors as values, where it allows you to ensure the server does not crash and the issues are handled, you can use features such as the pcall. This is always recommended over any other method. For Astra's case, there are usually chained calls that each can faily on their own as well, hence wrapping them in lambda functions or individually pcall wrapping them always is a good idea.

HTTP Client

Sometimes your server needs to access other servers and make an HTTP Request, for these cases Astra provides a HTTP Client function:

-- By default its always a GET request
local response = Astra.http.request("https://example.com/"):execute()
pretty_print(response:status_code())
pretty_print(response:headers())
pretty_print(response:remote_address())
pretty_print(response:body():text())

The Astra.http.request function returns a HTTPClientRequest object which can be further modified to the needs before execution. The way to do these modification is through chained setters.

local request_client = Astra.http.request("https://example.com")
-- - Method. You can pick between one of these:
--   - GET,
--   - POST,
--   - PUT,
--   - PATCH,
--   - DELETE,
--   - HEAD,
  :set_method("POST")
  :set_header("key", "value")
  :set_headers({ key = "value" })
  :set_form("key", "value")
  :set_forms({ key = "value" })
  :set_body("THE CONTENT OF THE BODY")
  :set_json({ key = "value" })
  :set_file("/path/to/file")
  -- You can also execute as an async task
  :execute_task(function (result) end)

Templating

Astra supports jinja-like templating through Tera. It is incredibly performent, feature rich and easy to use.

-- can also pass no arguments to make an empty templating engine
local templates = Astra.new_templating_engine("examples/templates/**/*.html")
-- Exclude files
templates:exclude_templates({ "base.html" })
-- Add some data for the templates
templates:context_add("count", 5)

-- You can also add functions to be used within the templates
-- Example within templates: { test(key="value") }
template_engine:add_function("test", function (args)
    pprint(args)
    return "YEE HAW"
end)

There are two ways of templating in Astra:

Static serve

This is where your templates are compiled and ran at the start of your server. The data for these templates do not change once compiled.

server:templates(templates)

Partial Hydration

This method allows you to include dynamic data and render them yourself.

local count = 0
server:get("/hydrate", function(request, response)
    -- your dynamic data
    count = count + 1
    template_engine:context_add("count", count)
    -- response type
    response:set_header("Content-Type", "text/html")
    -- render the template
    return template_engine:render("index.html")
end)

Importing

The Lua's default module and importing system has proven to have some issues with our approaches, namely async. Because of this, the import function have been introduced. The import function, imports the given module relative to the runtime binary, similar to how GOlang and Python imports work. The import order of the modules also affect the imported data, in case the module data is shared across. For example:

-- module A.lua:
return { value = 0 }

-- module B.lua:
local a = import("A")
a.value = 2

-- main.lua
local a = import("A")
print(a.value) -- value = 0
local b = import("B")
print(a.value) -- value = 2

File IO

Astra provides some File IO functionality to help extend the standard library. Which contains the IO functions. The current list is as follows:

  • get_metadata
  • read_dir
  • get_current_dir
  • get_script_path
  • get_separator
  • change_dir
  • exists
  • create_dir
  • create_dir_all
  • remove
  • remove_dir
  • remove_dir_all

They are fairly self explanitory and does not require further details. Example usage:

pprint(Astra.io.get_script_path())

Schema Validation

Sometimes during development, your server likely recieves structured data such as JSON from outside. You likely also have a structure in mind for them. For these cases to validate that the structures are correct and to confidently go through them without risk of errors, hopefully, you can use the schema validation utility.

Schema Validation essentially is a function that returns true if a given table is of a given structure. The structure is defined as a separate table that has the field names along the types and requirements. For example:

-- Your schema
local schema = {
    -- Type names along their types and requirements
    id = { type = "number" },
    name = { type = "string", required = false }
}
-- Your actual data
local example = { id = "123", name = 456 }
-- Check the validation
local is_valid, err = Astra.validate_table(example, schema)
assert(not is_valid, "Validation failed: expected validation to fail")

Almost all of the native lua types are accounted for. Deeply nesting is obviously supported as well:

local schema = {
    user = {
        type = "table",
        schema = {
            profile = {
                type = "table",
                schema = {
                    id = { type = "number" },
                    name = { type = "string" }
                }
            }
        }
    }
}
local example = {
    user = {
        profile = {
            name = "John",
        },
    },
}
local is_valid, err = Astra.validate_table(example, schema)
assert(is_valid, "Validation failed: " .. tostring(err))

As well as arrays:

local schema = {
    -- normal single type array
    numbers = { type = "array", array_item_type = "number" },
    strings = { type = "array", array_item_type = "string" },
    -- table array
    entries = {
        type = "array",
        schema = {
            id = { type = "number" },
            text = { type = "string" }
        }
    }
}

local tbl = {
    numbers = { 1, 2, 3 },
    strings = { "a", "b", "c" },
    entries = {
        {
            id = 123,
            text = "hey!"
        },
        {
            id = 456,
            text = "hello!"
        }
    }
}

local is_valid, err = Astra.validate_table(tbl, schema)
assert(is_valid, "Validation failed: " .. tostring(err))

Crypto

During development of your web servers, you might need some cryptography functionality such as hashing and encoding. For these cases, Astra provides commonly used cryptographic functions to ease up development.

Hashing

Currently Astra provides SHA2 and SHA3 (both 256 and 512 variants) hashing functions.

Astra.crypto.hash("sha2_512", "MY INPUT")

Base64

Astra also provides encoding and decoding of base64 strings, including URL safe variants:

local input = "MY VERY COOL STRING"

local encoded = Astra.crypto.base64.encode(input)
print(encoded)

local decoded = Astra.crypto.base64.decode(encoded)
print(decoded)

JSON

Often you will have to deal with a medium of structured data between your server and the clients. This could be in form of JSON, YAML, e.t.c. Astra includes some utilities to serialize and deserialize these with native Lua structures.

For JSON, the Astra.json.encode() and Astra.json.decode() methods are available which converts JSON data from and into Lua tables.

SQL Driver

If your server requires access to an SQL database such as PostgreSQL and SQLite, Astra provides utilities for basic connection and querying.

-- connect to your db
local db = database_connect("postgres", "postgres://astra_postgres:password@localhost/astr_database")

-- You can execute queries to the database along with optional parameters
db:execute([[
    CREATE TABLE IF NOT EXISTS test (id SERIAL PRIMARY KEY, name TEXT);

    INSERT INTO TABLE test(name) VALUES ('Astra');
]], {});

-- And finally query either one which returns a single result or
local result = db:query_one("SELECT * FROM test;", {});

-- query all from tables which returns an array as result
-- which also supports parameters for protection against SQL injection attacks
local name = "Tom"
local result = db:query_all("INSERT INTO test (name) VALUES ($1)", {name});

pretty_print(result)

Async Tasks

The spawn_task function spawns a new Async Task internally within Astra. An async task is a non-blocking block of code that runs until completion without affecting the rest of the software's control flow. Internally Astra runs these tasks as Tokio tasks which are asynchronous green threads. There are no return values as they are not awaited until completion nor joined. The tasks accept a callback function that will be run.

These are useful for when you do not wish to wait for something to be completed, such as making an HTTP request to an API that may or may not fail but you do not want to make sure of either. For example, telemetry or marketing APIs where it can have delays because of volume.

An example of async task:

spawn_task(function ()
    print("RUNNING ON ASYNC GREENTHREAD")
end)

print("RUNNING ON MAIN SYNC THREAD")

The tasks return a TaskHandler as well which has a single method: abort. This will kill the running task, even if it isn't finished.

Additionally two more task types are also available:

-- Runs in a loop with a delay
local task_id = spawn_interval(function ()
    print("I AM LOOPING");
end, 2000)

-- Runs once after the given delay in milliseconds
spawn_timeout(function ()
    print("I AM RUNNING ONLY ONCE.")
    print("Time to abort the interval above")
    -- cancel the interval task
    task_id:abort()
end, 5000)

note

The interval code runs immediately and then the delay happens before the loop starts again. In contrast the timeout's delay happen first before the code runs.

Utilities

These are some of the smaller utilities and functionality that are usually globally available regardless of the Astra namespace:

Pretty Print

pprint function is a wrapper over print function that also lets you print tables as well as other values.

String Split

string.split splits a string according to the seperator provided and result into an array of chunks.

Dotenv

It is always a good idea to never include sensitive API keys within your server code. For these reasons we usually recommend using a .env file. Astra automatically loads them if they are present in the same folder into the environment, accessible through the os.getenv. You can also load your own file using the global Astra.dotenv_load function.

This is the load order of these files (They can overwrite the ones loaded previously):

  • .env
  • .env.production
  • .env.prod
  • .env.development
  • .env.dev
  • .env.test
  • .env.local

Structure

Astra is written in Rust and is built on top of Axum. Axum internally uses Tokio for its async runtime and Tower. Astra also have other dependencies to assist with utilities and functionality of the runtime itself. One of them is mlua which is the library used for the Lua runtime.

Astra's internal philosophy is to contain all routes in a single table array and use mlua UserData as much as possible for every other functionality. Since routes and server itself is the main point, utilities and functionality should be optional. Hence main.rs, routes.rs and common.rs are the main files that contain the essential details of the framework, while everything else is optional and extra. The utilities that require native system interopability must also be written in Rust. For example, the database drivers and the HTTP client.

note

If you wish to extend Astra with your own utilities written in Rust, check out this guide.

The source code structure is containing all of the sources in the src folder which divides between actual Rust code, lua folder which contains sources for the lua side and lua based utilities, and docs folder which contains sources for this documentation. The docs folder on the root only contains built doc files for usage with GitHub Pages.

There is also astra_build.lua which is a small script for automating different operations within Astra, such as automatic doc builds, lua packing, changelogs, ...

Extending Astra

Through Components

Your point of interest will be the src directory. If your aim is to extend the engine, you may look into the components folder upon which you can add a new file of your choice.

For extensions, you need a struct that implements LuaComponents trait implemented which gives an async lua runtime context. Within it you can create a new lua function that adds your function to the global context at runtime.

#![allow(unused)]
fn main() {
pub struct MyNewExtension {
    pub field: String,
}
impl crate::components::LuaComponents for MyNewExtension {
    async fn register_to_lua(lua: &mlua::Lua) -> mlua::Result<()> {
        let function = lua.create_async_function(|lua, param: mlua::Value| async move {
            /* Content */

            Ok(Self { field: "Hello!".to_string() })
        })?;

        lua.globals().set("my_extension", function)
    }
}
}

After this, open mod.rs folder, include your new extention's file, and within register_components call your extension's register_to_lua method.

#![allow(unused)]
fn main() {
    my_extension::MyNewExtension::register_to_lua(lua).await?;
}

If you wish to return a table with methods that your lua code can call, then you need to implement mlua's UserData trait:

#![allow(unused)]
fn main() {
impl UserData for MyNewExtension {
    fn add_methods<M: mlua::UserDataMethods<Self>>(methods: &mut M) {
        // Add your custom method that you can call within Lua
        methods.add_method(
            "sync_method",
            |lua, this, param: mlua::Value| {
                // Your code
                println!("{:param#?}");

                Ok(())
            },
        );

        // It can also be async
        methods.add_async_method(
            "async_method",
            |lua, this, param: mlua::Value| async move {
                // Your code
                println!("{:param#?}");

                Ok(())
            },
        );

        // As well as mutability
        methods.add_mut_method(
            "sync_method",
            |lua, this, param: String| {
                // Your code
                this.field = param.clone();
                println!("{:param#?}");

                Ok(())
            },
        );
    }
}
}