fix: avoid recursive signature instantiation

[lewis6991]

Fixes #1003

Summary

• Guard generic signature instantiation so recursive substituted return types keep nested signatures opaque.
• Preserve the same instantiation context through conditional branches.
• Add a regression for pairs(self) iterator methods used through a generic enum-like helper.

Tests

• cargo test -p emmylua_code_analysis test_pairs_self_iterator_method_does_not_recurse -- --nocapture
• cargo test -p emmylua_code_analysis for_range_var_infer_test -- --nocapture
• cargo test -p emmylua_code_analysis

[github-actions]

Code Review Summary

Changes Overview

This PR introduces a recursion guard mechanism to prevent infinite recursion when instantiating generic signatures that reference themselves (e.g., pairs(self) on a table containing the method).

Issues Found

1. Potential Performance Concern with RefCell Usage

• Location: type_substitutor.rs lines 20-21
• Issue: Using Rc<RefCell<HashSet>> for tracking signatures introduces runtime borrow checking overhead. While acceptable for this use case, consider if a simpler approach (e.g., recursion depth counter) would suffice.
• Suggestion: If recursion depth is bounded, a simple u32 counter with a max depth check might be more performant.

2. Missing Test Assertions

• Location: for_range_var_infer_test.rs lines 244-245
• Issue: The test test_pairs_self_iterator_method_does_not_recurse calls ws.expr_ty() but discards the results. The test only verifies no crash/panic occurs, but doesn't validate the inferred types.
• Suggestion: Add assertions to verify the inferred types are reasonable:

  let key_type = ws.expr_ty("key_out");
  let value_type = ws.expr_ty("value_out");
  assert!(!key_type.is_nil());
  assert!(!value_type.is_nil());

3. Inconsistent Error Handling

• Location: instantiate_signature in mod.rs lines 535-537
• Issue: When recursion is detected, the function returns LuaType::Signature(*signature_id) without any warning or error. This silently degrades type inference quality.
• Suggestion: Consider logging a debug warning or returning a more specific type (e.g., LuaType::Any) to make the limitation visible.

4. Missing Documentation

• Location: type_substitutor.rs - enter_signature method
• Issue: The purpose and behavior of the recursion guard is not documented.
• Suggestion: Add a doc comment explaining:
  • Why recursion can occur (self-referencing signatures)
  • What happens when recursion is detected (returns None)
  • The lifecycle of the guard (automatically removed on drop)

Positive Aspects

• ✅ Clean refactoring of instantiate_type_generic calls to use context-based version
• ✅ Proper use of RAII pattern with InstantiatingSignatureGuard and Drop implementation
• ✅ Good test coverage for the recursion scenario
• ✅ Minimal changes to existing logic

Recommendations

1. Add assertions to the test to validate type inference quality
2. Consider adding a debug log when recursion is detected
3. Document the recursion guard mechanism for future maintainers
4. Monitor performance impact of RefCell usage in hot paths

[gemini-code-assist]

Code Review

This pull request introduces a recursion guard mechanism in the type instantiation process to prevent infinite recursion when a signature refers to itself, such as in the case of pairs(self). It updates GenericInstantiateContext to track active signature instantiations using a HashSet and an RAII guard. The PR also refactors instantiate_conditional_generic.rs to use context-aware instantiation functions and adds a regression test to verify the fix. I have no further feedback to provide.