diff --git a/docs/images/priv-connections-nlb-disable-inbound-rules-options.png b/docs/images/priv-connections-nlb-disable-inbound-rules-options.png new file mode 100644 index 00000000000..91fced5213a Binary files /dev/null and b/docs/images/priv-connections-nlb-disable-inbound-rules-options.png differ diff --git a/docs/images/priv-connections-target-group-basic.png b/docs/images/priv-connections-target-group-basic.png deleted file mode 100644 index a6c8c9fbbe1..00000000000 Binary files a/docs/images/priv-connections-target-group-basic.png and /dev/null differ diff --git a/docs/images/priv-connections-target-group-first-step.png b/docs/images/priv-connections-target-group-first-step.png new file mode 100644 index 00000000000..c884e4a0e55 Binary files /dev/null and b/docs/images/priv-connections-target-group-first-step.png differ diff --git a/docs/images/priv-connections-target-group-register-listeners.png b/docs/images/priv-connections-target-group-register-listeners.png new file mode 100644 index 00000000000..73e23097d73 Binary files /dev/null and b/docs/images/priv-connections-target-group-register-listeners.png differ diff --git a/docs/images/priv-connections-target-group-register-nlb.png b/docs/images/priv-connections-target-group-register-nlb.png deleted file mode 100644 index 72d9f2f65eb..00000000000 Binary files a/docs/images/priv-connections-target-group-register-nlb.png and /dev/null differ diff --git a/docs/private-networking/aws-console-setup.mdx b/docs/private-networking/aws-console-setup.mdx index 3082bf91aea..91fbc3d5a70 100644 --- a/docs/private-networking/aws-console-setup.mdx +++ b/docs/private-networking/aws-console-setup.mdx @@ -44,21 +44,40 @@ The target group is how the NLB will know where to forward traffic. AWS requires directly — if you need to expose a Lambda, put it behind an ALB and use the ALB target type. - + + On the **Specify group details** page (the first of two steps in AWS's target-group form), set: + - **Name**: e.g. `trigger-postgres-tg` - **Protocol**: TCP - **Port**: the port your resource listens on (5432 for Postgres, 6379 for Redis, 3306 for MySQL, etc.) - **VPC**: the VPC where your resource lives (this must match the VPC you'll use for the NLB) - **Health check protocol**: TCP - ![Target group basic configuration](/images/priv-connections-target-group-basic.png) + Click **Next** to move to the second step (registering targets). + + ![Target group first step — basic configuration](/images/priv-connections-target-group-first-step.png) - - Add the IP addresses of the resource. For RDS, look up the writer endpoint's IPs (`dig ` from inside the VPC). - For ElastiCache, use the primary endpoint IPs. + + On the **Register targets** page — the second step of the IP target-group flow — paste the + private IPs of your resource and set the port to the same value you picked above. Click + **Include as pending below**, then **Create target group**. + + ![Register targets in the target group](/images/priv-connections-target-group-register-listeners.png) + + + Both ElastiCache and RDS expose a DNS endpoint, not an IP, on their console pages. Find the + private IP behind the endpoint via the EC2 console: + + 1. Open **EC2 → Network & Security → Network Interfaces**. + 2. In the search bar, filter by **Description** with `ElastiCache` (or `RDSNetworkInterface` + for RDS). Optionally narrow further by **VPC ID** if you have several clusters. + 3. Read the **Primary private IPv4 address** column — that's the IP to register here. For + multi-node clusters or read replicas, each node has its own ENI and IP. - ![Register targets in the target group](/images/priv-connections-target-group-register-nlb.png) + You can also reach the same list from **VPC → Subnets → \ → Network + Interfaces tab**, which scopes the list to a single subnet. + RDS and ElastiCache endpoints' IP addresses can change after failover or maintenance. For long-lived @@ -68,9 +87,6 @@ The target group is how the NLB will know where to forward traffic. AWS requires - - Click **Create target group**. - ## Step 2: Create an internal Network Load Balancer @@ -111,6 +127,16 @@ The NLB is what PrivateLink exposes to Trigger.dev. It must be **internal** (not column shows **Active** before moving on. The endpoint service in the next step won't list the NLB until it's fully active. + + Once the NLB is **Active**, open it and go to its **Security** tab, then click **Edit**. If a + security group is attached, AWS enables **Enforce inbound rules on PrivateLink traffic** by + default — leaving it on can cause traffic from the Trigger.dev VPC Endpoint to be silently + dropped before reaching your listener. Uncheck **Enforce inbound rules on PrivateLink traffic** + and save. + + ![Uncheck Enforce inbound rules on PrivateLink traffic on the NLB](/images/priv-connections-nlb-disable-inbound-rules-options.png) + + @@ -256,6 +282,10 @@ such as the "Private link not found" wizard error. A few quick checks specific t - Confirm the listener port matches the port your task code is dialing. - Confirm the security group on your resource allows inbound traffic from the NLB or the VPC's private IP range. + - If the NLB itself has a security group attached, turn off **Enforce inbound rules on + PrivateLink traffic** on the load balancer. See [the troubleshooting + page](/private-networking/troubleshooting#connection-is-active-but-the-assigned-ip-is-not-reachable-from-tasks) + for details. - Try connecting from inside the VPC first (e.g., a bastion host) to rule out resource-side issues. diff --git a/docs/private-networking/troubleshooting.mdx b/docs/private-networking/troubleshooting.mdx index a2c07b63eff..62dd03c5f50 100644 --- a/docs/private-networking/troubleshooting.mdx +++ b/docs/private-networking/troubleshooting.mdx @@ -43,3 +43,36 @@ Trigger.dev cannot provision a VPC Endpoint until your endpoint service explicit For full setup instructions including this step, see [Setting up PrivateLink in the AWS Console](/private-networking/aws-console-setup). + +## Connection is Active but the assigned IP is not reachable from tasks + +If your private connection shows **Active** in the Trigger.dev dashboard and the NLB target group reports healthy targets, but tasks still cannot reach the assigned IP, the most common cause is that your Network Load Balancer is enforcing security group rules on PrivateLink traffic. + +When a security group is attached to an NLB, AWS exposes a separate setting called **Enforce inbound rules on PrivateLink traffic**. When this is **on**, the NLB applies its security group's inbound rules to traffic arriving from VPC endpoints — and the source IP it evaluates is the **private IP of the consumer's VPC endpoint network interface**, not an IP in your own VPC. Because that IP belongs to Trigger.dev's VPC and isn't known ahead of time, the SG rule almost never matches, and traffic is silently dropped at the NLB. + +### How to fix it + + + + Go to **EC2 → Load balancers** in the region where your NLB lives and select the load balancer + backing your endpoint service. + + + On the **Security** tab, click **Edit**. + + + Uncheck **Enforce inbound rules on PrivateLink traffic** and save. + + + This only changes how the NLB itself filters traffic. Authorization is still enforced by the + endpoint service's **Allow principals** list, so only AWS accounts you've explicitly + allow-listed can connect. + + + + + Re-run a task that dials the assigned private IP. The connection should now succeed. + + + +If you need to keep the enforcement on for compliance reasons, the alternative is to widen your NLB's security group inbound rule to `0.0.0.0/0` on the listener port. Allow-listing the consumer endpoint's CIDR is not practical because it lives in Trigger.dev's VPC and may change.