Skip to content

Add core dump documentation and improve docs structure and accuracy #903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hoffmaen merged 4 commits into from
Apr 27, 2026
Merged

Add core dump documentation and improve docs structure and accuracy #903

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6c83088
Add documentation for enabling core dumps
hoffmaen Apr 23, 2026
08074b6
Restructure README with documentation index and fix spelling and gram…
hoffmaen Apr 23, 2026
46606e6
Fix incorrect property name and config examples in docs
hoffmaen Apr 23, 2026
f4ddb80
Document enablement of core dumps with bpm-release > 1.4.29
hoffmaen Apr 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 21 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,18 @@
BOSH Release for haproxy
===========================
# BOSH Release for HAProxy

Questions? Pop in our [slack channel](https://cloudfoundry.slack.com/messages/haproxy-boshrelease/)!

This BOSH release is an attempt to get a more customizable/secure haproxy release than what
This BOSH release is an attempt to get a more customizable/secure HAProxy release than what
is provided in [cf-release](https://github.com/cloudfoundry/cf-release). It allows users to
blacklist internal-only domains, preventing potential Host header spoofing from allowing
unauthorized access of internal APIs. It also allows for better control over haproxy's
blocklist internal-only domains, preventing potential Host header spoofing from allowing
unauthorized access of internal APIs. It also allows for better control over HAProxy's
timeouts, for greater resiliency under heavy load.

Usage
-----
## Usage

To deploy this BOSH release:

```
```bash
git clone https://github.com/cloudfoundry-community/haproxy-boshrelease.git
cd haproxy-boshrelease

Expand All @@ -27,6 +25,16 @@ bosh deploy manifests/haproxy.yml \

To make alterations to the deployment you can use the `bosh deploy [-o operator-file.yml]` flag to provide [operations files](https://bosh.io/docs/cli-ops-files.html).

## Documentation

- [External Certificates](/docs/external_certs.md) - Using HAProxy with additional external certificates
- [Mutual TLS](/docs/mutual_tls.md) - Mutual TLS configuration
- [Rate Limiting](/docs/rate_limiting.md) - Client IP based rate limiting
- [Keepalived](/docs/keepalived.md) - Keepalived integration for high availability
- [Core Dumps](/docs/coredumps.md) - Enabling core dumps for HAProxy debugging
- [Dependency Updates](/docs/version-bumps.md) - How to bump dependency versions
- [Release Process](/docs/release-process.md) - How to create a new release

## Development

Feel free to contribute back to this via a pull request on a feature branch! Once merged, we'll
Expand All @@ -35,11 +43,11 @@ cut a new final release for you.
### Unit Tests and Linting

#### PR Validation
PRs will be automatically tested by https://concourse.arp.cloudfoundry.org/teams/main/pipelines/haproxy-boshrelease once a maintainer has labelled the PR with the `run-ci` label
PRs will be automatically tested by https://concourse.arp.cloudfoundry.org/teams/main/pipelines/haproxy-boshrelease once a maintainer has labeled the PR with the `run-ci` label

#### Local Test Execution
Unit/rspec Tests and linters can be run locally to verify correct functionality before pushing to the CI system.
If you change any erb logic in the jobs directory please add a corresponding test to `spec`.
Unit/RSpec tests and linters can be run locally to verify correct functionality before pushing to the CI system.
If you change any ERB logic in the jobs directory, please add a corresponding test to `spec`.

```bash
# install the necessary dependencies, once
Expand Down Expand Up @@ -68,12 +76,8 @@ bundle exec guard
```

#### Test Debugging
Unit/rspec Tests can also be debugged/stepped through when needed. See for example the [VSCode rdbg Ruby Debugger](https://marketplace.visualstudio.com/items?itemName=KoichiSasada.vscode-rdbg) extension. You can follow the "Launch without configuration" instructions for the extension, just set the "Debug command line" input to `bundle exec rspec <filepath>`.
Unit/RSpec tests can also be debugged/stepped through when needed. See for example the [VSCode rdbg Ruby Debugger](https://marketplace.visualstudio.com/items?itemName=KoichiSasada.vscode-rdbg) extension. You can follow the "Launch without configuration" instructions for the extension, just set the "Debug command line" input to `bundle exec rspec <filepath>`.

### Acceptance tests
### Acceptance Tests

See [acceptance-tests README](/acceptance-tests/README.md).

### Certificate reloads during runtime

See [external_certs README](/docs/external_certs.md)
67 changes: 67 additions & 0 deletions docs/coredumps.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Enabling Core Dumps for HAProxy

When debugging crashes or unexpected behavior in HAProxy, it can be useful to enable core dumps for post-mortem analysis.

## Required Changes

Enabling core dumps requires a few modifications to the BOSH release.

Depending on the BPM (BOSH Process Manager) version, BPM needs to be either configured or disabled:

### 1a. Disable BPM (BPM <= 1.4.29)

BPM <= 1.4.29 restricts the process environment in ways that prevent core dumps from being written. To work around this, the monit configuration must be changed to manage HAProxy directly via `haproxy_wrapper` instead of BPM:

- **Start program** in `jobs/haproxy/monit`: Change from `/var/vcap/jobs/bpm/bin/bpm start haproxy` to `/var/vcap/jobs/haproxy/bin/haproxy_wrapper`
- **Stop program** in `jobs/haproxy/monit`: Change from `/var/vcap/jobs/bpm/bin/bpm stop haproxy` to `/bin/bash -c 'kill $(cat /var/vcap/sys/run/haproxy/haproxy.pid)'`
- **PID file** in `jobs/haproxy/monit`: Change from `/var/vcap/sys/run/bpm/haproxy/haproxy.pid` to `/var/vcap/sys/run/haproxy/haproxy.pid`
- **PID file** in `jobs/haproxy/templates/drain.erb`: Change the `pidfile=` variable from `/var/vcap/sys/run/bpm/haproxy/haproxy.pid` to `/var/vcap/sys/run/haproxy/haproxy.pid`
- **PID file** in `jobs/haproxy/templates/reload.erb`: Change the `pidfile=` variable from `/var/vcap/sys/run/bpm/haproxy/haproxy.pid` to `/var/vcap/sys/run/haproxy/haproxy.pid`

### 1b. Set BPM core_file_size limit (BPM > 1.4.29)

BPM > 1.4.29 allows setting the core dump file size limit. In `jobs/haproxy/templates/bpm.yml`, add `core_file_size` to the existing `limits:` block (alongside `open_files`):

```yaml
limits:
open_files: <%= p("ha_proxy.max_open_files") %>
core_file_size: 1073741824
```

### 2. Configure the HAProxy wrapper script

The following must be added to `haproxy_wrapper.erb` before HAProxy is started:

```bash
ulimit -c unlimited # Allow unlimited core dump file size
ulimit -n 256000 # Ensure sufficient file descriptors
echo /var/vcap/data/haproxy/core.%e.%p.%t > /proc/sys/kernel/core_pattern
```

The core pattern places dumps in `/var/vcap/data/haproxy/` with the filename format `core.<executable>.<pid>.<timestamp>`.

### 3. Enable `set-dumpable` in HAProxy config

Add the `set-dumpable` directive to the `global` section in `haproxy.config.erb`. This is required because HAProxy drops privileges to the `vcap` user after startup, which by default causes the kernel to disable core dumps. `set-dumpable` calls `prctl(PR_SET_DUMPABLE, 1)` to re-enable them.

## Analyzing Core Dumps

Core dump files are written to `/var/vcap/data/haproxy/`.

To analyze a core dump, use `gdb` with the HAProxy binary and the core file:

```bash
gdb /var/vcap/packages/haproxy/bin/haproxy /var/vcap/data/haproxy/core.<executable>.<pid>.<timestamp>
```

Useful GDB commands once loaded:

```
bt # Print backtrace of the crashing thread
bt full # Print backtrace with local variables
info threads # List all threads
thread <N> # Switch to thread N
thread apply all bt # Print backtraces for all threads
```

> **Note:** For meaningful stack traces, HAProxy should be compiled with debug symbols. To enable this, add `DEBUG_CFLAGS="-g"` to the `make` command in `packages/haproxy/packaging`. Without debug symbols, the backtrace will show only addresses without function names.
28 changes: 14 additions & 14 deletions docs/external_certs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,35 @@
# Using HAProxy with additional External Certificates

By default, the HAproxy BOSH manifest contains all certificates to be used during runtime.
By default, the HAProxy BOSH manifest contains all certificates to be used during runtime.
The way to pass the certificates can be either via the `ha_proxy.ssl_pem` property that sets one chain for
the hostname HAproxy is running on. Or they can be passed via the `ha_proxy.crt_list` property, which is essentially
the hostname HAProxy is running on. Or they can be passed via the `ha_proxy.crt_list` property, which is essentially
a list of `ssl_pem` properties that allows to configure multiple entries for different hostnames using SNI.

## What are External Certificates and why are they needed?

If you are planning to use more than one certificate on your HAproxy you are most likely going to use the `ha_proxy.crt_list`
If you are planning to use more than one certificate on your HAProxy you are most likely going to use the `ha_proxy.crt_list`
property. The main use-case for this property is to register different trust configurations for different hosts.
For example, if your HAproxy services both the secure.example.com and www.example.com hosts, they both might have different
For example, if your HAProxy services both the secure.example.com and www.example.com hosts, they both might have different
requirements towards security. One could be using mTLS and a more secure certificate than the other or they could be using different CAs.

The problems start when you are using a very large `ha_proxy.crt_list` with dozens or even hundreds of entries while using BOSH to deploy them.
The way BOSH works is that all certificates will become part of the manifest during rendering and those certificates will then be extracted
from the manifest and onto the HAproxy disk during deployment. If the manifest becomes very large (> 20M) the time BOSH needs to render and deploy increases significantly. At the same time, providing your customers the capability to register custom domains and certificates tends to be a very dynamic process, i.e. you never know when a customer will register a domain and upload a certificate to deploy but you will want to deploy the new certificate as quickly as possible so the customer can use it right away. Using the given way, you'll end up deploying HAproxy all the time. The major downside of this is that every time you deploy HAproxy, there will be a brief moment where the old process exits and the new process has not yet started. This will drop all existing connections to HAproxy and any client connected at that moment will receive a disruption.
from the manifest and onto the HAProxy disk during deployment. If the manifest becomes very large (> 20M) the time BOSH needs to render and deploy increases significantly. At the same time, providing your customers the capability to register custom domains and certificates tends to be a very dynamic process, i.e. you never know when a customer will register a domain and upload a certificate to deploy but you will want to deploy the new certificate as quickly as possible so the customer can use it right away. Using the given way, you'll end up deploying HAProxy all the time. The major downside of this is that every time you deploy HAProxy, there will be a brief moment where the old process exits and the new process has not yet started. This will drop all existing connections to HAProxy and any client connected at that moment will receive a disruption.

How can this dilemma be solved? External certificates to the rescue!

## How does it all work?

The HAproxy BOSH release provides an additional property `ha_proxy.ext_crt_list` that enables the use of a second source of certificates.
When used, HAproxy will expect an additional `crt-list` file to be present in a specific folder (by default: `/var/vcap/jobs/haproxy/config/ssl/ext`).
If the file exists its contents will be merged with the existing certificates from the manifest before HAproxy is started.
Since the list of certificates is now provided by two decoupled sources, those sources need to be synchronized in order to avoid starting HAproxy with an incomplete set of certificates. During startup, HAproxy will wait for the second `crt-list` file to appear. This allows an external service (e.g. another BOSH release) to generate the file and place it in the directory where it is expected.
The HAProxy BOSH release provides an additional property `ha_proxy.ext_crt_list` that enables the use of a second source of certificates.
When used, HAProxy will expect an additional `crt-list` file to be present in a specific folder (by default: `/var/vcap/jobs/haproxy/config/ssl/ext`).
If the file exists its contents will be merged with the existing certificates from the manifest before HAProxy is started.
Since the list of certificates is now provided by two decoupled sources, those sources need to be synchronized in order to avoid starting HAProxy with an incomplete set of certificates. During startup, HAProxy will wait for the second `crt-list` file to appear. This allows an external service (e.g. another BOSH release) to generate the file and place it in the directory where it is expected.

At runtime, when a new certificate needs to be added, the external service can simply update the second `crt-list` file and trigger a [hitless reload](https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/) of HAproxy using the `/var/vcap/jobs/haproxy/bin/reload` command. No connections will be dropped.
At runtime, when a new certificate needs to be added, the external service can simply update the second `crt-list` file and trigger a [hitless reload](https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/) of HAProxy using the `/var/vcap/jobs/haproxy/bin/reload` command. No connections will be dropped.

Depending on your configuration, HAproxy will refuse to start without external certificates or it will continue without them after a timeout.
Depending on your configuration, HAProxy will refuse to start without external certificates or it will continue without them after a timeout.

## Configuring HAproxy to use External Certificates
## Configuring HAProxy to use External Certificates

The feature is controlled by these properties:
```
Expand All @@ -42,10 +42,10 @@ ha_proxy.ext_crt_list_file:
The location from which to load additional external certificates list
default: "/var/vcap/jobs/haproxy/config/ssl/ext/crt-list"
ha_proxy.ext_crt_list_timeout:
Timeout (in seconds) to wait for the external certificates list located at `ha_proxy.ext_crt_list_file` to appear during HAproxy startup
Timeout (in seconds) to wait for the external certificates list located at `ha_proxy.ext_crt_list_file` to appear during HAProxy startup
default: 60
ha_proxy.ext_crt_list_policy:
What to do if the external certificates list located at `ha_proxy.ext_crt_list_file` does not appear within the time
denoted by `ha_proxy.ext_crt_list_timeout`. Set to either 'fail' (HAproxy will not start) or 'continue' (HAproxy will start without external certificates)
denoted by `ha_proxy.ext_crt_list_timeout`. Set to either 'fail' (HAProxy will not start) or 'continue' (HAProxy will start without external certificates)
default: "fail"
```
Loading