🔗 What is OpenResty XRay?

OpenResty XRay is a dynamic-tracing product analyzing various cloud and server applications in real-time. It treats running processes and containers as read-only databases and extracts essential information to troubleshoot performance issues, exceptions, errors, and security vulnerabilities. Armed with knowledge bases, inference engines, and hundreds of advanced analyzers, OpenResty XRay can diagnose and narrow down the root causes of deep issues without changing or affecting the target applications.

🔗 Who should use OpenResty XRay?

DevOps can use OpenResty XRay to quickly discover any issues or pinpoint the root causes of the problems and then forward the automatic reports and findings to the developers.

The development team can also utilize OpenResty XRay to check any regressions in everyday development or before new releases of the software products.

🔗 How to set up OpenResty XRay for my applications?

You need to install and run the OpenResty XRay Agent program on the servers with your applications or containers. The Agent program is a standalone daemon process that inspects all your applications on the same server. It also serves as a client to the OpenResty XRay web console.

There are several ways to install the OpenResty XRay Agent:

1. Install the RPM or Deb package repositories we provided for the corresponding Linux distro.
2. Install a big .tar.gz bundle containing all our RPM or Deb packages for your Linux distro.
3. Install our official Docker or Kubernetes (K8s) container images

One common mistake is to install the Agent inside your own containers. It will always fail because your containers almost always lack privileges or file system access for the Agent to inspect other processes and containers.

🔗 What operating systems do we support?

Our official Docker or K8s container images can run on any modern Linux distros with container support, as long as the Linux distros used inside those target application containers are supported.

We support most of the mainstream Linux distributions if you want to run OpenResty XRay's Agent service natively. Check the latest supported distro list for details.

If you have a new Linux distro to be supported natively, contact us at info@openresty.com!

In the future, we may support non-Linux operating systems like Android, macOS, *BSD, and Microsoft Windows.

🔗 Do you require very new Linux kernels?

No! We support old kernels like CentOS 7's 3.10 kernels. Older kernels like 2.6 in CentOS 6 are not supported, though.

🔗 What is the overhead of the Agent?

We try very hard to minimize the performance impact of our Agent program on your servers marked as "production." The Agent uses an event-triggered sampling approach so that it won't attempt to collect data all the time, just when there is a real need. When not sampling, all of your processes incur strictly 0 extra overhead. Thanks to the true noninvasive nature of dynamic tracing. And even when sampling, the overhead of most of our analyzers should be less than 5% for your applications. Even for those inherently more expensive tools, the system tries to automatically adapt the frequency and process scope of those analyzers to minimize the impact.

If you do not mark your servers as production in the web console, the system will assume they are development or testing boxes. Therefore, it will no longer be cautious about resource usage, and all the analyzers will run at full speed so that you can get results as soon as possible.

🔗 What data does the Agent collect?

The OpenResty XRay Agent program only collects essential data for profiling performance issues, troubleshooting stability and behavioral problems, and scanning security vulnerabilities. We never collect any potentially sensitive data from your business-level code. We also implement safeguards to avoid accidentally collecting sensitive data in the data source so that it won't go outside your applications' processes.

🔗 How do you handle the data collected by the Agent?

We protect our customer's data and privacy with great caution. It is our #1 priority. We always use encryption methods of industrial standards (like TLS) when transferring all the data to our web console. The storage and management of the data on our web console also conforming to all the SOC 2 guidelines. Only the most recent 2 weeks of raw data live in our web console (higher-level report data lives longer).. If some customers have even shorter data lifetime requirements, we can also accommodate that.

🔗 Do you sell or share my data?

No, never. We won't sell your data to any third parties. We may only share some audited and obfuscated data with some of our external contractors. And all our contractors must sign nondisclosure agreements (NDAs) with us.

🔗 Can I avoid uploading any data to your console?

Yes! We provide the on-premises version of OpenResty XRay for companies with stringent data handling policies. It is more expensive than the cloud version. For the on-prem version, the user deploys the web console in their own environments; thus, no data will ever be uploaded to OpenResty Inc's servers.

🔗 Do I need to restart my processes after installing the Agent?

No! One beauty of dynamic-tracing technology is that it is postmortem. You don't need to restart your applications or any of your existing processes for the newly installed OpenResty XRay Agent to analyze. We can do that because we don't need any new plugins, new startup options, or inject any code into your applications. It is spotless and noninvasive.

We understand that you may have some processes that entered some rare state and restarting them will make the problem disappear. This is why postmortem analysis and debugging are so critical.

🔗 How do I stop the Agent?

To stop the OpenResty XRay Agent service, just run the following shell command:

sudo systemctl stop orxray-agent

But this is temporary. If the system restarts or reboots, the Agent service will start again. To stop it permanently, you can also run the command after running the previous one:

sudo systemctl disable orxray-agent

For containerized Agent services, you can stop the Agent container instances with Docker or Kubernetes.

🔗 How do I restart the Agent?

To restart your Agent, just run the command

sudo systemctl restart orxray-agent

Wait a few minutes for the new instance of the Agent to take effect.

🔗 My Agent is not working; what should I do?

There are times when the Agent is not working correctly due to network or disk issues. In such cases, please check out the error log files in this directory:

/usr/local/orxray-agent/logs/

If you still need help with the errors in the error log files, feel free to contact us.

🔗 Can Agent analyze private software?

Yes, if you wrote the private software. For example, you should never analyze other vendors' proprietary software if you don't have their permission. But if you wrote the software or have access to the source code and permissions to compile it, then it is possible. And you can even use our Y language (ylang) to write advanced analyzers for your programs. Make sure you compile the private software with DWARF data (or "debug symbols"). For example, C/C++ programs should be compiled with the -g or -g3 option when gcc or g++ are used. Building with DWARF symbols incurs zero overhead to your programs' runtime performance, and most of the mainstream Linux distributions do this by default for their standard software packages.

DWARF data is not always required, however. Some elemental analyzers in OpenResty XRay do not need "debug symbols" to work. We used to help a customer to pinpoint a memory leak in a third-party proprietary library on the C/C++ function level. But make sure you conform to the licensing requirements of the third parties.

🔗 Does the Agent require installing C compiler toolchains?

Nope! Unlike those open-source dynamic tracing frameworks like SystemTap or eBPF/BCC, OpenResty XRay does not require installing any C or C++ compiler toolchains on the target machines. No gcc, no clang, nothing. Additionally, it does not require installing any Linux kernel header files.

🔗 Do I need to install the debug symbol packages on all my servers?

No. OpenResty XRay indexes all the debug symbols it sees in a remote database called package database. Whenever the Agent sees a binary executable file, it will automatically query the package database for its corresponding debug symbols.

We run crawlers that fetch the Internet for public software packages' debug symbols (or DWARF data) across the Internet all the time. And we also automatically index our users' own DWARF data and debug symbols. Each OpenResty XRay tenant has their own separate package database for privacy protection. If you compile the software yourself, then make sure you have an offline (or online) machine with all the software binaries and debug symbols installed and an OpenResty XRay Agent is running there (for a while at least). Once the DWARF data and binary programs are indexed in the user's package database, the user can get rid of the debug symbols on all her machines.

🔗 OpenResty XRay says my programs lack debug symbols; what should I do?

If the programs are compiled by yourself, try to find the debug symbol packages associated with your main packages (like *-debuginfo packages on an RPM-based distro and *-dbgsym.ddeb or *-dbgsym.deb files for a Deb-based distro). Install the debuginfo packages on one of your machines with an Agent installed and running, and wait for the Agent to index the debug symbols.

Suppose your programs were compiled without debug symbols in the first place, or you cannot find the corresponding debuginfo packages. In that case, you can try re-compiling your programs with debug symbols enabled (for C/C++ programs, pass the -g or -g3 option to the compiler command lines). Also, make sure your packaging scripts generate a debuginfo package (like *-debuginfo*.rpm or *-dbgsym.deb) instead of tossing the debug symbols away.

We understand finding debug symbols can be a hassle sometimes. We've been working on an AI system that can automatically rebuild DWARF data from existing DWARF data for similar but not identical binary programs. Once this is ready, OpenResty XRay will be able to thoroughly analyze a lot of open-source software binaries missing debug symbols.

It's worth mentioning that some elemental analyzers in OpenResty XRay do not require "debug symbols" to work. Indeed, we used to help a customer to pinpoint a memory leak in a third-party proprietary library on the C/C++ function level.

🔗 Can OpenResty XRay analyze applications not running on OpenResty?

Sure! OpenResty XRay is a general platform that can analyze any software. Okay, at least the software you control (like you have access to the DWARF debug symbols, source code, etc.).

However, OpenResty XRay currently analyzes OpenResty and Nginx tech stacks best. We have initial support for some versions of Python, Perl, PHP, and PostgreSQL. And we've been working on the Go and Java support as well. Tell us the open-source software you want OpenResty XRay to support soon!

🔗 Does OpenResty XRay need to change my applications in any way?

No! OpenResty XRay uses advanced dynamic-tracing technologies that do not require collaboration from the target applications. It is 100% noninvasive.

1. We do not require adding a plugin or module to your applications.
2. We do not require patching your applications' code.
3. We do not need special compilation or startup options (Okay, you do need the -g option when compiling C/C++ programs to emit DWARF data, but that's very standard practice).
4. We do not inject any code into your application processes.

Using OpenResty XRay analyzers won't crash or impair your applications' processes in any way since we add zero stuff to your processes. Zero.

🔗 Can I write my own analyzers?

Yes! You can use our Y language (or ylang) to write your own analyzers. Ylang is a large subset of the C programming language with our own extensions for dynamic tracing. We also provide the Ylua language, a subset of the Lua language with extensions, and YSQL, a SQL dialect.

🔗 Can I use my own Linux kernels?

Yes! But you need to contact our team so that we can index your kernels in the package database and test their compatibility with our product thoroughly. We do not support automatic indexing of custom or unknown kernels yet. We may consider adding this support in the future.

🔗 Can I use OpenResty XRay in a local VM atop VMWare/VirtualBox/KVM/etc.?

Yes! All such full-system virtualization technologies should work out of the box with OpenResty XRay. If not, please report to us.

Please note that installing our Agent inside your own Docker or K8s containers is not supported. But you can always use our official Docker and K8s container images for the Agent.

🔗 Can I run OpenResty XRay in a public cloud VM?

Yes! Like Microsoft Azure, Google Cloud (GCP), Amazon Cloud (AWS), or Digital Ocean. And virtualized Cloud servers are fine. It doesn't have to be bare metals.

🔗 Is there a personal version for individual use?

Yes! As long as the personal version is not used for any companies, governments, or other commercial purposes. We usually don't provide SLA for personal versions, though.