collaboration with Jolt JNI

[stephengold]

Hello, I'm the author of Jolt JNI, another open-source project providing Java Bindings for JoltPhysics. I'm wondering if it would make sense for us to somehow combine our efforts. Your thoughts?

[Morgoth398]

Hello,
sorry for the late response.

If there's a way for both of us to benefit from our combined efforts, then we can definitely try it. What kind of collaboration did you have in mind?

[stephengold]

To identify specific ways we might collaborate, it might help for us to first get a clearer picture of each other's motivations and constraints. That might explain why we took different approaches.

Depending what we find, we might combine into a single project or continue pursuing both approaches in parallel. If we continue in parallel, we might experiment with using each other's libraries and provide each other feedback on ease of use, performance, and bugs.

[Morgoth398]

I started this project during the development of my game engine. I wanted to include a physics library, but all existing ports or bindings I could find were either outdated or relied on JNI/JNA. My requirement was to use the FFM API exclusively, as it is clearly the better and easier API.

This does mean my bindings require at least Java 25 (and Java 26 on the Overhaul branch due to the use of LazyConstants) whereas your bindings support a lot more versions. This may stand in the way of combining the projects.

Another constraint on my side is my dependency on the C wrapper. All my config files for generating the bindings are tied to the symbol names defined in that wrapper's header file. Changing the wrapper would therefore result in a significant amount of work updating these config files.

[stephengold]

I started developing Jolt JNI to replace the existing physics system in the JMonkeyEngine game engine, which is based on JNI and Bullet Physics. I've strayed from that plan, to put it mildly.

For interfacing with plain C code, I agree FFM is easier than JNI. When I started working with Jolt Physics, I considered using FFM, but realized that would require a plain-C wrapper such as Amerkoleci's JoltC. JNI can interface directly and efficiently with C++ code, and the coding effort is comparable to that of creating a plain-C wrapper.

JoltC provides access to a limited subset of JoltPhysics, and according to its README, any feature request requires financial sponsorship. Jolt JNI is also incomplete, but it provides access to many features (including serialization, real-time type info, SIMD hair simulations, pulley/rack-and-pinion constraints, soft-body skinning, and Cosserat rods) that aren't yet accessible via JoltC. And unlike Amerkoleci, I'm open to pursuing un-sponsored feature requests from users.

Much of the complexity of interfacing between C++ and Java lies around memory management. Java/Kotlin programmers are generally not in the habit explicitly freeing memory. Jolt makes extensive use of reference counting, which is an awkward fit to the JVM's automated garbage collection. It took me several iterations to settle on a workable scheme for managing native memory in Jolt JNI. How is native memory freed in JoltPhysics-JavaFFM?

You say JoltPhysics-JavaFFM is undergoing an overhaul. What is the goal of that effort?

I can readily imagine that changing wrappers would entail significant effort.

[Morgoth398]

To answer your question about the memory management I'll just quote my README:

Memory Management

When creating an object (directly or indirectly), native memory will be allocated.

Structs will always be allocated with an arena. By default an automatic arena is used, but you can use any arena you want (even a global arena, but this may not be desirable) by using the appropriate constructor.

Other classes will allocate memory with a function call. When the object is created directly with a constructor call, a cleanup function will be attached to the memory segment. By default an automatic arena is used, but you can again use any arena you want. But this means you must not free the memory yourself, as it will be freed a second time when the arena is closed (does not apply for the global arena).

When an object is not created directly, e.g. with a method call, no cleanup function will be attached to the segment (unfortunately a side effect of generating all sources). That means you need to free the memory that this object holds manually. A workaround is to use the "raw methods" to allocate the memory and then attach the cleanup function manually.

BoxShapeSettings floorShapeSettings = new BoxShapeSettings(
		new Vec3().x(100).y(1).z(100),
		Jolt.DEFAULT_CONVEX_RADIUS);

// Object is created indirectly. No cleanup function will be attached
BoxShape floorShape = floorShapeSettings.createShape(); 

// Workaround
MemorySegment floorShapeSegment = BoxShapeSettings.createShape(floorShapeSettings.memorySegment());
BoxShape floorShape = new BoxShape(floorShapeSegment.reinterpret(Arena.ofAuto(), Shape::destroy));

Nevertheless, we still need to ensure on the Java side that all pointers passed to the native code stay alive as long as they are in use.

The overhaul is already finished. In my first version (the main branch), I wrote almost everything by hand. This of course took a huge amount of effort and a long time. The hard part was not really to write the actual bindings by hand, but rather the sheer quantity of code needed and the gathering of the required information from the header file (e.g. the extensive use of typedefs adds a lot of work). At some point I decided to use libclang to parse the header file and to extract all the required information. This allowed me to generate a rough structure of the classes that were nonetheless required to be finished off by hand. For the second, overhauled version (the Overhaul branch), I have generated nearly all sources using a generator I wrote. The generator uses libclang and FreeMarker to generate bindings from a given header file. When the config files are correct, the generator outputs already fully functional bindings that do not need to be changed in any way. So when something changes, I only need to update the config files, and this requires far less effort and work.

However, generating everything comes with a loss of flexibility: If a class deviates from the default structure, then I need to provide a custom template that handles this correctly. Because this may result in additional work, I have left it out most of the time. That's the reason that, for example, the class Vec3 has no constructor that takes three floats.

[stephengold]

I discovered there are 2 README files: one in each branch, and they are very different.
Even though I don't use Maven much, I managed to build the project and run the HelloWorld example on Linux.

I'm still thinking about ways we might collaborate.
One thing I'm unclear about: how interested are you in getting people to use JoltPhysics-JavaFFM? You made the effort to write a README file and request a link from https://github.com/jrouwe/JoltPhysics . There's plenty more you could to do to encourage people to use it.

[Morgoth398]

The README files are indeed very different. I have now made the Overhaul branch the default, as it is the version that should be used. Which version of the HelloWorld example did you build?

As for promoting my project, I am happy if people find it useful, but the link in the JoltPhysics repository is currently enough promotion for me. Additionally, I have no clear idea what other ways there are to encourage people to use it. What did you have in mind?

[Wr0ngPass]

@Morgoth398 I'd like to say a huge thank you! Your project is simply magical. I stumbled upon it by chance and fell in love with it :)

[stephengold]

Which version of the HelloWorld example did you build?

Originally I built the version in the "main" branch, but I've now tested both branches.

I have no clear idea what other ways there are to encourage people to use it. What did you have in mind?

Changing the default branch was going to be my first suggestion. I'm glad you've done that!

Now that you've changed the default branch, the next priority is to highlight the fact that the project is actively maintained and in a usable state. (There are many projects at GitHub that are in early development or broken or abandoned.) There are many ways to make your project stand out: a list of features, a list of dependent projects, screenshots, and/or video clips. Publishing a release might help. Or if you don't want to bother with releases, at least provide explicit build instructions.

More documentation would be a good thing. More code fragments and runnable sample apps.

Provide pre-compiled binaries for more platforms, starting with macOS. (How do you know it works with macOS if you haven't tried it?)

[stephengold]

Also, you could add "topics" keywords to the repository details at GitHub, so your project will show up in searches:

[Morgoth398]

I have now applied some of your suggestions. Thanks for the feedback!

More documentation would be a good thing.

This is indeed true. On the main branch I added all the documentation by hand, but as I now generate all the classes, the documentation is missing. Only documenation that is present in the header file will be added (more precisely the brief description that libclang extracts) but the c wrapper header does not contain any. For now I have not found a good and easy way to add "external" documentation into my generator.

More code fragments and runnable sample apps

If I find the time, I may port more Jolt examples. As these examples should have some graphical visualization (instead of just console output) I would probably create a separate project only for them. That way this project remains independent from any library used for rendering.

Provide pre-compiled binaries for more platforms, starting with macOS.

As I do not have any mac device and I am not familiar with GitHub Actions, I cannot provide the shared library.

(How do you know it works with macOS if you haven't tried it?)

I just assumed that when the wrapper was build correctly, the bindings would work. I have now used a more cautious wording in the README.

[stephengold]

I have remote access to a Mac M1, which I use for occasional testing. If you point me to your procedures for building the Linux native library, I could attempt to build macOS native libraries by hand (using similar procedures) and verify that they work. I could send you the results in a pull request. Would that interest you?

Building natives by hand isn't great for reproducibility. Going forward, I think it's best to build natives in a continuous-integration environment such as GitHub Actions. I might help you set that up. However, I should mention most of my recent build-automation experience is with Gradle; I'm not comfortable using Maven and/or Cmake.

Jolt Physics has its own graphics engine for visualization, based on Vulkan, but I doubt JoltC exposes any of that. For Jolt JNI, I created a simple OpenGL graphics engine called Sport-Jolt. It shouldn't be difficult to adapt Sport-Jolt for JoltPhysics-JavaFFM, assuming JoltPhysics-JavaFFM provides the sufficient Jolt-Physics functionality. Would that interest you?

[Morgoth398]

If we manage to build the shared libraries with GitHub Actions, it would be great if you could test the macOS ones. I do not think there is the need for you to build them by hand, testing is enough.

I only know the fundamentals of CMake. I can write a basic Cmake file and reliably build a project if a correct one is already provided, but writing more complex Cmake files or resolving errors I cannot do reliably.

JoltC exposes a DebugRenderer with only three methods (drawLine, drawTriangle and drawText3D) that the user can override. For those, I can write a small batch renderer using OpenGL without much effort, so you won't need to adapt your graphics engine. But thanks for the offer!

[stephengold]

First step for GitHub Actions is to get a continuous Maven build going. I've submitted PR #3 for that.
Next step will be to build the native library on Linux. You know better than I how to do it, but if you publish (or send) instructions, I can try to automate that.

[Morgoth398]

I have now merged your PR but the workflow run failed.

This is how to build a CMake project:

1. cd directory/containing/the/cmake/file
2. Run cmake -S . -B build.
   • -S specifies the directory containing the CMake file
   • -B specifies the directory to store all data needed for building
3. Run cmake --build build to actually build the project