It’s a common requirement to inspect some persistent Lua data structures when troubleshooting online Lua applications, including those run by OpenResty or Nginx (in the case of Nginx, it’s still using OpenResty’s core component, the Lua Nginx modules). The Lua data structures might get embedded deep into some loaded Lua modules, referenced by some Lua functions via upvalues, etc. The traditional way of inspecting such Lua data is to expose special APIs that serialize and dump out the corresponding Lua data. But it’s much more flexible and efficient to examine the Lua data inside the running processes via dynamic tracing technologies.

OpenResty XRay provides a Lua-compatible language called YLua that supports writing such tools for inspecting any Lua data inside a running process using a 100% non-invasive way. This tutorial demonstrates a straightforward use case of dumping out the names of all loaded Lua modules in a running OpenResty and LuaJIT application. We’ll show more complex use cases with YLua in future tutorials.

System Environment

Here we use a Red Hat Enterprise Linux 8 system as an example. Any Linux distributions supported by OpenResty XRay should work equally fine, like Ubuntu, Debian, Fedora, Rocky, Alpine, etc.

We use an unmodified open-source OpenResty binary build as the target application. You can use any OpenResty or Nginx binaries, including those compiled by yourself. No special build options, plugins, or libraries are needed in your existing server installation or processes. It is the beauty of dynamic tracing technologies. It’s genuinely non-invasive.

We also have the OpenResty XRay’s Agent daemon running on the same system and have the command-line utilities from the openresty-xray-cli package installed and configured.

Names of Loaded Lua Modules

Using Standard Tools

The easiest way to dump the loaded Lua modules in a target process is to use the standard OpenResty XRay tool lj-loaded-modules:

Use the ps cmd to find the nginx worker process we want to analyze. Here it is 1046.

$ ps aux | grep nginx
1 S root     1059    1  0  80   0 - 73941 -      15:03 ?        00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx
5 S nobody   1046 1059  0  80   0 - 82123 -      15:03 ?        00:00:00 nginx: worker process

Now let’s run the standard analyzer lj-dump-loaded-mods with the orxray command-line utility from the openresty-xray-cli package. The -p option specifies the target process ID (or PID) to be analyzed.

# 4096 is the target process's PID
$ orxray analyzer run lj-loaded-modules -p 4096
Start tracing...
jit: table
math: table
coroutine: table
debug: table
...
Go to https://x5vrki.xray.openresty.com/targets/68/history/751715 for charts.

Using Custom YLua Tools

The standard tool is implemented in the YLua language under the hood. Let’s see how we can recreate the tool with YLua ourselves. We know that loaded Lua modules are always cached in the package.loaded in the LuaJIT VM (and the standard Lua 5.1 interpreter’s VM). The keys are the module names while the values are the module data. Now we can just create a new YLua source file with the name, say, my-dump-loaded-mods.ylua:

probe process.begin
    for k, v in pairs(package.loaded) do
        print(k, ": ", type(v))
    end
    exit()
end

Here the probe keyword defines a new probe handler for the probe-point process.begin, triggered when the tool is attached to the target processes¹. And then, we dump the Lua modules by iterating the package.loaded Lua table in YLua, which is precisely the same syntax as Lua. After printing out the module names and their data types, we call exit() to end the whole tracing process. And we can use the run-ylua command-line utility from the openresty-xray-cli package to run this YLua source file.

Inspecting an OpenResty or Nginx Lua Application

Use the ps cmd to find the nginx worker process we want to analyze. Here it is 1046.

$ ps aux | grep nginx
1 S root     1059    1  0  80   0 - 73941 -      15:03 ?        00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx
5 S nobody   1046 1059  0  80   0 - 82123 -      15:03 ?        00:00:00 nginx: worker process

Now let’s run the previous YLua source file with the run-ylua command-line utility from the openresty-xray-cli package. The -p option specifies the target process ID (or PID) to be analyzed.

$ run-ylua -p 1046 my-dump-loaded-mods.ylua
Start tracing...
resty.core.uri: table
resty.core.exit: table
resty.core.base64: table
resty.core.request: table
resty.core.response: table
string: table
pgmoon: table
openresty_org.controller: table
ndk: table
table: table
resty.core.utils: table
resty.lrucache: table
table.new: function
debug: table
table.clear: function
...
package: table
math: table
resty.core.base: table
jit.opt: table
ngx: table
_G: table
resty.core.var: table
resty.core.worker: table
resty.core.regex: table
resty.core.shdict: table
resty.core.time: table
resty.core.hash: table

We omitted most of the output for brevity. We can check out the total number by piping the output to the wc -l command:

$ run-ylua -p 1046 my-dump-loaded-mods.ylua | wc -l
Start tracing...
38

Most of the modules are of the table type. Usually, a Lua module is represented by a Lua table, after all. A few of the modules are unique in that they are of the function type. For example, table.new is a function typed module implemented specifically by LuaJIT.

• io, os, string, table, math, debug are standard Lua modules defined by the Lua 5.1 language.
• jit, bit, ffi, jit.opt, table.new, table.clear are standard LuaJIT modules.
• resty.*, coroutine, ndk, ngx are modules introduced by OpenResty.

Inspecting standalone LuaJIT applications

We can also inspect a standalone luajit process without Nginx or OpenResty involved.

First, find the PID of the luajit process as always:

$ ps aux | grep luajit
root     3371845  0.0  0.0   4428  2524 pts/3    S    22:17   0:00 luajit t.lua

And then, we use this PID, 3371845, to run the run-ylua command:

$ orxray analyzer run lj-loaded-modules -p 3371845
Start tracing...
jit: table
math: table
coroutine: table
debug: table
os: table
_G: table
package: table
string: table
jit.opt: table
bit: table
io: table
table: table

As we see, standalone luajit programs usually load many fewer Lua modules by default.

We can check out the total number by piping the output to the wc -l command:

$ orxray analyzer run lj-loaded-modules -p 3371845 | wc -l
Start tracing...
12

Running Directly in the Web Console

The user may choose to execute any of the tools covered in this tutorial directly in the web console of OpenResty XRay. They can even be triggered automatically upon interesting events like high CPU usage. The command-line utilities from the openresty-xray-cli are handy for demonstration purposes. And they are also easy to automate and integrate into other systems by the DevOps and SRE people.

Tracing Applications inside Containers

OpenResty XRay tools support tracing containerized applications transparently. Both Docker and Kubernetes (K8s) containers work transparently. Just as with normal application processes, the target containers do not need any applications or extra privileges. The OpenResty XRay Agent daemon should run outside the target containers (like in the host operating system directly or in its own privileged container).

Let’s see an example. We first check the container name or container ID with the docker ps command.

$ docker ps
CONTAINER ID   IMAGE                                       COMMAND                  CREATED         STATUS         PORTS     NAMES
4465297209d9   openresty/openresty:1.19.3.1-2-alpine-fat   "/usr/local/openrest…"   18 months ago   Up 3 minutes             angry_mclaren

Here the container name is angry_mclaren. We can then find out the target process’s PID in this container.

$ docker top angry_mclaren
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                3373047             3373026             0                   22:23               ?                   00:00:00            nginx: master process /usr/local/openresty/bin/openresty -g daemon off;
nobody              3373101             3373047             0                   22:23               ?                   00:00:00            nginx: worker process

The PID for the openresty worker process is 3373101. We then run the OpenResty XRay analyzer against this PID as usual.

$ orxray analyzer run lj-loaded-modules -p 3373101
Start tracing...
table: table
ngx: table
resty.core.var: table
table.new: function
resty.core.regex: table
resty.core.shdict: table
resty.core.time: table
...
table.clear: function
resty.core.worker: table
resty.lrucache: table
io: table

OpenResty XRay is also able to automatically detect long-running processes as “applications” of a particular type (like “OpenResty”, “Python”, etc.).

How The Tools are Implemented

All the tools are implemented in the Y language. OpenResty XRay executes them with either the Stap+² or eBPF³ backends of OpenResty XRay, both of which use the 100% non-invasive dynamic tracing technologies based on the Linux kernel’s uprobes and kprobes facilities. The YLua scripts are first compiled down to the Y language and then further down to the executable dynamic tracing tools.

We don’t require any collaborations from the target applications and processes. No log data or metrics data is used or needed. We directly analyze the running processes' process space in a strictly read-only way. And we also never inject any byte-code or other executable code into the target processes. It is 100% clean and safe.

The Overhead of the Tools

The dynamic-tracing tools demonstrated in this tutorial are very efficient and suitable for online execution.

When the tools are not running and actively sampling, the overhead on the system and the target processes are strictly zero. We never inject any extra code or plugins into the target applications and processes; thus, there’s no inherent overhead.

During sampling, the overhead on average request latency and throughput is usually not measurable for the tools covered in this tutorial. The tools here only do a one-off inspection anyway.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.

---

1. Note that this is very different from attaching through the ptrace() system call as in GDB. Here is the dynamic tracing attachment, which is much safer and faster. ↩︎

2. Stap+ is OpenResty Inc’s greatly enhanced version of SystemTap. ↩︎

3. This is actually the greatly enhanced version of OpenResty Inc.’s eBPF implementation called ORBPF. ↩︎