Calling of java methods in a .jar file fails, but listing methods works fine.

[RainerHeintzmann]

I am struggling with being able to call java routines, whos classes are inside a subfolder
view5d/
in the jar file. I tried in vain for days. After opening the library

V = @JavaCall.jimport view5d.View5D

works fine as do

listmethods(V)

or

listmethods(V,"Start5DViewerF")

yet when calling the function I always obtain a message that this method does not exist.
I also tried different version ("view5d/Start5FViewerF","view5d.Start5FViewerF","View5D.Start5FViewerF") and also calling a method like wait() with a much simpler signature, but the result is always the same.
The code can be accessed under:

https://github.com/RainerHeintzmann/View5D.jl
(.jar file included)
Any help would be greatly appreciated!

[mkitti]

Hi Rainer,

See if this works for you:

julia> using JavaCall

julia> begin
           JavaCall.init(["-Djava.class.path=$(joinpath(@__DIR__, "jars","View5D.jar"))"])
           V = @jimport view5d.View5D
           jArr = Vector{jfloat}
           myJArr = rand(jfloat, 5,5,5,5,5)
           myViewer = jcall(V, "Start5DViewerF", V, (jArr, jint, jint, jint, jint, jint), myJArr[:], 5, 5, 5, 5, 5)
       end
created data 5
JavaObject{Symbol("view5d.View5D")}(JavaCall.JavaLocalRef(Ptr{Nothing} @0x0000000067418ea0))

Mark Kittisopikul, Ph.D.
Janelia Scientific Computing

[RainerHeintzmann]

You are a star! This works like a charm saving me days of figuring this out! I will update the viewer to now work also in Julia very soon!
By the way, it would be cool to also add Nothing as the corresponding Java type for void. Or maybe just to avoid
long searches introduce jvoid=Nothing as a new type.
Any thoughts to add an automatic way to instantiate Julia methods? This would make everything so much simpler.
The dream-usage would be:

V = @jimport   view5d.View5D
createMethods(V,"Start5DViewer")
Start5DViewer(JuliaArray, size(1),size(2),size(3),size(4),size(5))

Rainer

[mkitti]

For void it depends on the context. For example the return type of System.out.println is void as used here:

julia> using JavaCall

julia> JavaCall.init()

julia> jls = @jimport java.lang.System
JavaObject{Symbol("java.lang.System")}

julia> out = jfield(jls, "out", @jimport java.io.PrintStream )
JavaObject{Symbol("java.io.PrintStream")}(JavaCall.JavaLocalRef(Ptr{Nothing} @0x0000000062b60830))

julia> jcall(out, "println", Nothing, (JString,), "Hello world")
Hello world

I guess the analog would be Cvoid which is Nothing.

About methods, one approach I am interested in is making JMethod (java.lang.reflect.Method) callable. We actually have a lot of information we can obtain.

julia> Start5DViewerF = listmethods(V,"Start5DViewerF")[2]
view5d.View5D Start5DViewerF(float[], int, int, int, int, int)

julia> Start5DViewerF
view5d.View5D Start5DViewerF(float[], int, int, int, int, int)

julia> params .|> p->jcall(p, "getTypeName", JString, ())
6-element Vector{String}:
 "float[]"
 "int"
 "int"
 "int"
 "int"
 "int"

julia> JavaCall.getreturntype(Start5DViewerF) |> p->jcall(p, "getTypeName", JString, ())
"view5d.View5D"

julia> Start5DViewerF(JuliaArray, size(1),size(2),size(3),size(4),size(5)) # ??

Hidden within JavaCall there is a subpackage called JProxies that attempted to do that:
https://github.com/JuliaInterop/JavaCall.jl/tree/master/JProxies

It could use some refactoring though.

[RainerHeintzmann]

That's great! I should give this a try.
However, coming back to the original problem, I uncovered something strange which was the reason why it took me a day to give up:
When I open a new julia REPL, cd to the View5D directory, everything works great.
Yet, when I am inside my "View5D.jl" module with "using JavaCall" there is something strange going on as I described in the very beginning. I now get the message ERROR: JavaCall.JavaCallError("Class Not Found view5d/View5D").
If I restart the repl and execute the JavaCal.init line in the REPL and then start my module, everything works great.
I checked: The two paths are identical for the JavaCall.init. No idea how to deal with this in my module.
Did you see a similar issue when contructing the wrapper for Napari?
I just uploaded a revised version of View5D.jl which now can do a lot more things, but in my hands it only works if I first execute the JavaCall.init() outside the module.

[mkitti]

I'll have to take a deeper look. A few things to consider:

1. There is a JavaCall.addClassPath that helps with class path management.
2. Also see JavaCall.getClassPath to query the actual class path recognized.
3. You may want to consider adding the class path from inside your module's __init__
4. Consider using pathof(@__MODULE__) to get the path to the module.

[mkitti]

I see the problem now. You want:

myPath = ["-Djava.class.path=$(joinpath(dirname(@__DIR__), "jars","View5D.jar"))"]

With your current code I was getting

myPath = ["-Djava.class.path=C:\\Users\\kittisopikulm\\.julia\\packages\\View5D\\kHfzG\\src\\jars\\View5D.jar"]

Your code was looking for the <packageRoot>/src/jars/View5D.jar relative to your package root, but you want <packageRoot>/jars/View5D.jar that is not in the src directory.
@__DIR__ gets you <packageRoot>/src.
dirname(@__DIR__) gets you <packageRoot>

[RainerHeintzmann]

I think the problem is different. This is copied from the current gitHub repository:
myPath = ["-Djava.class.path=$(joinpath(@__DIR__, "jars","View5D.jar"))"]
Which is what you suggested.
The effect really has to do with the VSCode IDE I am using. If I put an explicit path into the .julia/config/startup.jl, it works nicely by a REFL started from the Windows system, but it does not work in the VSCode REPL, even though it executes the same code.
Well, almost, as it seems to activate a different default environment.

[mkitti]

Note that my version has an extra dirname around @__DIR__. That's the critical change.

I'm also using VSCode. How are you loading the module to test it? Does the module live under .julia/dev? What is the output of using Pkg; Pkg.status("View5D") ?

[mkitti]

Here's the initial diagnostic I did with your package:

julia> using View5D

julia> using View5D
[ Info: Precompiling View5D [90d841e0-6953-4e90-9f3a-43681da8e949]

julia> View5D.view5d( rand(Float32, 5, 4, 3, 2, 1) )
myPath = ["-Djava.class.path=C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\src\\jars\\View5D.jar"]
ERROR: JavaCall.JavaCallError("Class Not Found view5d/View5D")
Stacktrace:
 [1] _metaclass(::Symbol) at C:\Users\kittisopikulm\.julia\dev\JavaCall\src\core.jl:409
 [2] metaclass at C:\Users\kittisopikulm\.julia\dev\JavaCall\src\core.jl:414 [inlined] 
 [3] jcall(::Type{JavaCall.JavaObject{Symbol("view5d.View5D")}}, ::String, ::Type{T} where T, ::NTuple{6,DataType}, ::Array{Float32,1}, ::Vararg{Any,N} where N) at C:\Users\kittisopikulm\.julia\dev\JavaCall\src\core.jl:251
 [4] view5d(::Array{Float32,5}, ::Nothing, ::Nothing) at C:\Users\kittisopikulm\.julia\dev\View5D\src\View5D.jl:175
 [5] view5d(::Array{Float32,5}) at C:\Users\kittisopikulm\.julia\dev\View5D\src\View5D.jl:152
 [6] top-level scope at REPL[2]:1

julia> using JavaCall

julia> JavaCall.getClassPath()
"C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\src\\jars\\View5D.jar"

julia> JavaCall.getClassPath() |> isfile
false

julia> JavaCall.getClassPath() |> dirname
"C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\src\\jars"

julia> JavaCall.getClassPath() |> dirname |> isdir
false

julia> JavaCall.getClassPath() |> dirname |> dirname
"C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\src"

julia> JavaCall.getClassPath() |> dirname |> dirname |> isdir
true

julia> JavaCall.getClassPath() |> dirname |> dirname |> dirname |> p->joinpath(p,"jars")
"C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\jars"

julia> JavaCall.getClassPath() |> dirname |> dirname |> dirname |> p->joinpath(p,"jars") |> isdir
true

julia> JavaCall.getClassPath() |> dirname |> dirname |> dirname |> p->joinpath(p,"jars") |> p->joinpath(p,"View5D.jar")
"C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\jars\\View5D.jar"

julia> JavaCall.getClassPath() |> dirname |> dirname |> dirname |> p->joinpath(p,"jars") |> p->joinpath(p,"View5D.jar") |> isfile
true

[RainerHeintzmann]

Thanks a lot. the current mast branch has version 82f3457
https://github.com/RainerHeintzmann/View5D.jl
I did massive modifications since this morning. Is it possible you used an old version?
My comment an hour ago referred to the current version.

[mkitti]

With https://github.com/mkitti/View5D.jl/blob/82f345760c1a9798db980d1e1fdb09c7960722e1/src/View5D.jl#L154
I am able to reproduce your error:

shell> git log -p -1
commit 82f345760c1a9798db980d1e1fdb09c7960722e1 (HEAD -> master, origin/master, origin/HEAD, mkitti/master)
Author: rheintzmann <heintzmann@gmail.com>
Date:   Tue Mar 30 18:30:26 2021 +0200

    documentation change

....
 
julia> using View5D
[ Info: Precompiling View5D [90d841e0-6953-4e90-9f3a-43681da8e949]

julia> View5D.view5d( rand(Float32, 5, 4, 3, 2, 1) )
myPath = ["-Djava.class.path=C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\src\\jars\\View5D.jar"]
ERROR: JavaCall.JavaCallError("Class Not Found view5d/View5D")
Stacktrace:
 [1] _metaclass(::Symbol) at C:\Users\kittisopikulm\.julia\dev\JavaCall\src\core.jl:409
 [2] metaclass at C:\Users\kittisopikulm\.julia\dev\JavaCall\src\core.jl:414 [inlined]
 [3] jcall(::Type{JavaCall.JavaObject{Symbol("view5d.View5D")}}, ::String, ::Type{T} where T, ::NTuple{6,DataType}, ::Array{Float32,1}, ::Vararg{Any,N} where N) at C:\Users\kittisopikulm\.julia\dev\JavaCall\src\core.jl:251
 [4] view5d(::Array{Float32,5}, ::Nothing, ::Nothing) at C:\Users\kittisopikulm\.julia\dev\View5D\src\View5D.jl:175
 [5] view5d(::Array{Float32,5}) at C:\Users\kittisopikulm\.julia\dev\View5D\src\View5D.jl:152
 [6] top-level scope at REPL[3]:1

In a new REPL, I am also able to fix it:

shell> git checkout dirname
Switched to branch 'dirname'
Your branch is up to date with 'mkitti/dirname'.

shell> git log -p -1
commit 34a1c2d25308906a6bb677301f81a74ed078c4bf (HEAD -> dirname, mkitti/dirname)
Author: Mark Kittisopikul <kittisopikulm@janelia.hhmi.org>
Date:   Tue Mar 30 16:49:37 2021 -0400

    Add dirname to myPath

diff --git a/src/View5D.jl b/src/View5D.jl
index f6aeea6..b13530c 100644
--- a/src/View5D.jl
+++ b/src/View5D.jl
@@ -151,7 +151,7 @@ julia> v3 = view5d(img3);
 function view5d(myArray :: AbstractArray, exitingViewer=nothing, gamma=nothing)
         if ! JavaCall.isloaded()
             # JavaCall.init(["-Djava.class.path=$(joinpath(@__DIR__, "View5D.jl","AllClasses"))"])
-            myPath = ["-Djava.class.path=$(joinpath(@__DIR__, "jars","View5D.jar"))"]
+            myPath = ["-Djava.class.path=$(joinpath(dirname(@__DIR__), "jars","View5D.jar"))"]
             @show myPath
             JavaCall.init(myPath)
             # JavaCall.init(["-Djava.class.path=$(joinpath(@__DIR__, "jars","view5d"))"])

julia> using View5D
[ Info: Precompiling View5D [90d841e0-6953-4e90-9f3a-43681da8e949]

julia> View5D.view5d( rand(Float32, 5, 4, 3, 2, 1) )
myPath = ["-Djava.class.path=C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\jars\\View5D.jar"]
created data 2
JavaCall.JavaObject{Symbol("view5d.View5D")}(JavaCall.JavaLocalRef(Ptr{Nothing} @0x000000003457ee98))

mkitti/View5D.jl@34a1c2d#diff-65fd30959efdc3bea2f43738e03f77584a68e5737b924733e704eeb1f563b0b4R154

I cannot reproduce the error after the change.

https://github.com/mkitti/View5D.jl/blob/34a1c2d25308906a6bb677301f81a74ed078c4bf/src/View5D.jl#L154

[RainerHeintzmann]

Brilliant! Sorry. I somehow missed the apparently crucial dirname() call, which I now included.
I think this is somehow behind most of the trouble I had. If this is not correct in the initialization then JavaCall is only partially agreeing with expectations.
Thank you so much for your great help!

[RainerHeintzmann]

Is there something special when it comes to supplying nested types to JavaCall?
I have no trouble exporting a Vector{Vector{Float64}}, in my export_marker_lists() routine, yet a am struggling with
the JavaCall in import_maker_lists() at the following lines:

    jfloatArrArr = Vector{Vector{jfloat}}
    jcall(myviewer, "ImportMarkerLists", Nothing, (jfloatArrArr,), marker_lists);

which yields

julia> import_marker_lists(a)
ERROR: MethodError: no method matching jvalue(::Vector{Vector{Float32}})
Closest candidates are:
  jvalue(::Integer) at C:\Users\pi96doc\.julia\packages\JavaCall\aVXyt\src\core.jl:179
  jvalue(::Float32) at C:\Users\pi96doc\.julia\packages\JavaCall\aVXyt\src\core.jl:180
  jvalue(::Float64) at C:\Users\pi96doc\.julia\packages\JavaCall\aVXyt\src\core.jl:181

Any idea what could be the problem here?

[mkitti]

Vector{Vector{Float32}} is difficult to deal with. The problem is that we need to protect the entire structure from Julia's garbage collection including the inner vectors for the duration of the call. We need to first convert all of the inner Vector{Float32} to JavaObjects and then pass the Vector{JavaObject} through. Let me see if I can figure out some code to accomplish that.

Let me see if I can come up with a snippet for the short term.

[mkitti]

This is a hack, but it works for now.

using JavaCall
using View5D
import View5D: import_marker_lists, get_viewer

JavaCall.metaclass(::Type{T}) where T <: AbstractVector = JavaCall.metaclass(Symbol(JavaCall.signature(T)))
JavaCall.javaclassname(::Type{T}) where T <: AbstractVector = JavaCall.signature(T)
JavaCall.signature(arg::Type{JavaObject{T}}) where {T <: AbstractVector} = string(JavaCall.javaclassname(T))

function import_marker_lists(marker_lists::Vector{Vector{T}}, myviewer=nothing) where {T}
    myviewer=get_viewer(myviewer)
    if T != Float32
        marker_lists = [convert.(Float32,marker_lists[n]) for n in 1:length(marker_lists)]
    end
    jfloatArrArr = Vector{JavaObject{Vector{jfloat}}}
    converted = JavaCall.convert_arg.(Vector{jfloat}, marker_lists)
    GC.@preserve converted begin
        jcall(myviewer, "ImportMarkerLists", Nothing, (jfloatArrArr,), [c[2] for c in converted]);
    end
    return
end

Demo:

julia> viewer = View5D.view5d( rand(Float32, 5, 4, 3, 2, 1) )
Initializing JavaCall with callpath: ["-Djava.class.path=C:\\Users\\kittisopikulm\\.julia\\dev\\View5D\\jars\\View5D.jar"]
created data 2
JavaObject{Symbol("view5d.View5D")}(JavaCall.JavaLocalRef(Ptr{Nothing} @0x000000000c9a7718))

julia> marker_lists = [rand(Float32,5) for i in 1:6]
6-element Vector{Vector{Float32}}:
 [0.7869251, 0.16082549, 0.84801865, 0.81543195, 0.72889245]
 [0.12232578, 0.7346517, 0.8028685, 0.9892626, 0.38214874]
 [0.2725159, 0.24991906, 0.6246897, 0.82423186, 0.7155572]
 [0.7211771, 0.05370474, 0.2918185, 0.28415167, 0.2144965]
 [0.3052622, 0.64955544, 0.09476519, 0.22565961, 0.23076034]
 [0.07435727, 0.23536158, 0.37190127, 0.00080156326, 0.9454497]

julia> import_marker_lists(marker_lists, viewer)