Warning etc. if threads=true while multithreading

[KnutAM]

The following example causes julia to quit silently on Windows 10 (but not on Windows Subsystem for Linux) if run with more than 1 thread.

using RecursiveFactorization, LinearAlgebra
num = 10
dim = 14
K_vec = [rand(dim,dim) + 5I for _ in 1:num]
b_vec = [rand(dim) for _ in 1:num]
lupivot_vec = [Vector{Int}(undef, dim) for _ in 1:num]

function run_recursive_test(Kv, bv, lupivotv)
    Threads.@threads for i in 1:length(Kv)
        K = Kv[i]
        b = bv[i]
        lupivot = lupivotv[i]
        LU = RecursiveFactorization.lu!(K,lupivot)
        ldiv!(LU, b)
    end
end

run_recursive_test(K_vec, b_vec, lupivot_vec)

Setting thread=false seems to solve it: LU = RecursiveFactorization.lu!(K,lupivot,Val{true}(),Val{false}()).
Should this be described somewhere, or did I miss some documentation?

[chriselrod]

Windows quitting silently is unexpected.
That not yielding great performance is expected, and should probably be documented.

[ChrisRackauckas]

Can confirm it crashes.

[KnutAM]

Sorry for not including the output of versioninfo() the first time, but now it seems to work with version v0.2.13

Current versioninfo

julia> versioninfo()
Julia Version 1.8.2
Commit 36034abf26 (2022-09-29 15:21 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_PKG_OFFLINE = false
  JULIA_PKG_USE_CLI_GIT = true

[chriselrod]

These arrays are too small for multithreading to be profitable. I'm surprised it ever did.
We adjusted the threading thresholds, which may explain the difference.

That is, try making the arrays much larger, and it'll probably still crash, as I don't think any underlying issues have been fixed.

[KnutAM]

Tried again on julia 1.7.3, then it crashes for dim=4 below.
For julia 1.8.2 it works fine up to dim=4096 (cannot go larger without getting OutOfMemoryError())

function run_recursive_test(num, dim)
   Kv = [rand(dim,dim) + 5I for _ in 1:num]
   bv = [rand(dim) for _ in 1:num]
   lupivotv = [Vector{Int}(undef, dim) for _ in 1:num]
   Threads.@threads for i in 1:length(Kv)
       K = Kv[i]
       b = bv[i]
       lupivot = lupivotv[i]
       LU = RecursiveFactorization.lu!(K,lupivot)
       ldiv!(LU, b)
   end
   return bv
end
foreach(dim->(println(dim); run_recursive_test(10, dim)), 2 .^ collect(2:2:12))

Julia 1.7 versioninfo

julia> versioninfo()
Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, tigerlake)
Environment:
  JULIA_PKG_OFFLINE = false
  JULIA_PKG_USE_CLI_GIT = true

(jl_1LkVGl) pkg> st
      Status `C:\Users\<user>\AppData\Local\Temp\jl_1LkVGl\Project.toml`
  [f2c3362d] RecursiveFactorization v0.2.13
  [37e2e46d] LinearAlgebra

[KnutAM]

Final update
Did some further digging, and I think the error occurs during compilation...

function foo(dim=4, dt=0)
    num = 4
    Kv = [rand(dim,dim) + 5I for _ in 1:num]
    bv = [rand(dim) for _ in 1:num]
    lupivotv = [Vector{Int}(undef, dim) for _ in 1:num]
    Threads.@threads for i in 1:length(Kv)
        sleep((Threads.threadid()-1)*dt)
        dt>0.1 && @info "Running thread $(Threads.threadid())"
        K = Kv[i]
        b = bv[i]
        lupivot = lupivotv[i]
        LU = RecursiveFactorization.lu!(K,lupivot)
        ldiv!(LU, b)
    end
    return bv
 end

Running first foo(4,0) causes a crash. Running first foo(4,5) (or any dt large enough to make compilation finish I guess), and then running foo(4,0) seems to work. It was interesting that the thread argument had any effect, since for the small matrices it is taking the _generic_lufact! path...
So this might not be a bug in RecursiveFactorization.jl after all, but it seems solved in 1.8.3 at least

[chriselrod]

Interesting investigation! It may be that on 1.7, running ThreadingUtilities.jl on multiple threads at the same time could result in one ccalling a not yet compiled function? I'm not sure what goes on with JITing parallel code, or how it normally ensures safety.