How to show a backend bitmap inside a HaxeUI Component?

I’m trying to display a Heaps Bitmap inside HaxeUI, but can’t figure out how. The Bitmap in question is a shadow map visualization of a 3D scene, so it cannot be generated as a file or via HaxeUI itself, but rather comes from a part of the program which uses directly the backend framework.

In HaxeUI, there is the Image class, but that seems to be intended for resources loaded from files. There is also the Canvas class, but that seems to be for drawing vector graphics. However, I need to show an already existing Heaps Bitmap as a HaxeUI component.

I understand that haxeui-heaps uses the Heaps Bitmap object, it appears there as ImageDisplay.sprite. However, setting ImageDisplay.sprite does nothing. I tried to make an Image (or a simple Component) and set the sprite to point to my bitmap, but nothing shows up:

function setImage(component:haxe.ui.core.Component, bitmap:h2d.Bitmap)
{
  component.getImageDisplay().sprite = bitmap;
}

I feel like I’m missing something very obvious here. To generalize, my question is how to take the default 2D element of a particular backend framework, put it inside a HaxeUI component, and have it show up as part of the HaxeUI layout, get resized along with it and so on.

This is not really specific to Heaps either, the Bitmap in question could just as well be an OpenFL Sprite object and the question would be the same, how to show the basic 2D element of any backend framework as part of the HaxeUI component hierarchy.

I’m sorry if I overlooked something in the documentation, but I’ve probably missed something so obvious that I’ve been looking in the wrong place.

Hi :wave:

So, the haxeui component Image has a resource property, the type of this property is a Variant which can be a number of things, one of this things can be ImageData, which is a backend class, for haxeui-heaps, this is: https://github.com/haxeui/haxeui-heaps/blob/master/haxe/ui/backend/ImageData.hx, ie hxd.BitmapData. Is it possible to get that out of a h2d.Bitmap? (Note: component .icon properties are the same as .resource, ie, variants)

Cheers,
Ian

Thank you for looking into this quickly!

I don’t think it is possible to get a Heaps BitmapData out of a Bitmap. The problem is that hxd.BitmapData is a byte vector in regular memory, whereas h2d.Bitmap is a h2d.Tile already in GPU memory. Even if it were possible to convert a Bitmap to BitmapData, this would involve downloading the texture from the GPU memory.

However, as far as I understand, HaxeUI itself on Heaps is already GPU accelerated. ImageDisplay.sprite is a backend-specific GPU object (h2d.Bitmap on Heaps), and the resulting HaxeUI component hierarchy is thus a tree of GPU objects.

Therefore, my use case would be to take an application-specific h2d.Bitmap (or any comparable basic visual element of a backend framework) and display that inside the HaxeUI component tree. While usually you put the HaxeUI interface “inside” the main view of your application, in this case I would like to go also the other way round and put a part of the 3D application content (an h2d.Bitmap representing a texture) inside a HaxeUI component, for the purpose of visualizing the state of this texture as part of the UI.

OK, so i think i understand… basically, you want to put a h2d heaps object (h2d.Object, h2d.Bitmap etc) inside a haxeui component tree?

The first thing that comes to mind is “just do it”… in haxeui, all components eventually extend from a “ComponentSurface” class, in haxeui-heaps this is: https://github.com/haxeui/haxeui-heaps/blob/master/haxe/ui/backend/ComponentSurface.hx (h2d.Object)

So you should be able to just .add your h2d.Bitmap into a haxeui component like you would anything else. I say should because this isnt something ive tested so maybe there are some gotchas… also, one thing that would certainly be an issue is that haxeui wouldnt recognize it as a haxeui component (since it isnt) so you wouldnt get some layout features - but i think if you made the component big enough, it should be fine (again, should because its not something ive tested).

Ill have a play and see if anything i said above holds true. It may be worth creating some type of “ObjectWrapper” haxeui component (specific to haxeui-heaps) that could take any h2d.Object and wrap a haxeui component around it so you would get all the layout features… … :thinking:

Ill have a think / play. Let me know if ive totally misunderstood your intentions, or if you can immediately spot a flaw in my thinking :slight_smile:

Cheers,
Ian

So yeah, assuming i have understood correctly, it seems to work as i expected:

heaps-unsolicated-object

<vbox style="padding: 5px;">
    <hbox>
        <button text="Click Me!" id="button1" style="color: red;" />
        <button text="Click Me!" id="button2" style="color: green;" />
        <button text="Click Me!" onclick="this.text='Thanks!'" style="color: blue;" />
    </hbox>    

    <slider pos="0" onchange="theGrid.padding = this.pos;" />

    <grid id="theGrid" columns="3">
        <button text="TL" />
        <button text="T" width="100%" />
        <button text="TR" />

        <button text="L" width="100%" height="100%" />
        <box id="container" width="200" height="200" style="background-color:yellow"/>
        <button text="R" width="100%" height="100%" />

        <button text="BL" />
        <button text="B" width="100%" />
        <button text="BR" />
    </grid>
</vbox>
@:build(haxe.ui.ComponentBuilder.build("assets/main-view.xml"))
class MainView extends VBox {
    public function new() {
        super();
        var tile = Tile.fromColor(0xFF0000, Std.int(container.width - 20), Std.int(container.height - 20), 1);
        var bitmap = new Bitmap(tile);
        bitmap.x = 10;
        bitmap.y = 10;
        container.addChild(bitmap);
    }
}

Is this the sort of thing you meant?

Cheers,
Ian

1 Like

Thank you very much, that works! Using addChild to add the h2d.Bitmap to a VBox works, and the image appears in the UI. As you mentioned, the layout doesn’t detect it’s size automatically, but setting the VBox width/height manually works.

I think my mistake was that I tried to use a non-container component, first an Image, then just a Component, and tried to attach the h2d.Bitmap either via the ImageDisplay or by using addChild. I assume that addChild didn’t work with those components because they are not HaxeUI containers.

In any case, this is exactly what I was trying to do. My test program is basically the Heaps Lights sample, which shows a 2D visualization of the shadow map in the upper right corner. By being able to attach h2d.Bitmaps like that to HaxeUI components, the HaxeUI interface can be used to provide various views into the inner state of the GPU next to the UI controls modifying the 3D scene.

I guess attaching a “native view” like this within the HaxeUI interface might be a common enough use case that it would be nice if there was a component for it. I think it’s also a distinct use case from building your own custom HaxeUI component, where you usually define some complex interactive component for reuse.

On the other hand, the goal of such a “NativeImage” component would be just to put any framework-specific visual element (e.g. an h2d.Bitmap or an openfl.display.Sprite) into a HaxeUI component, and then have the HaxeUI component tree a) show it on screen in the right place and b) take its dimensions into account in the HaxeUI layout.

Yeah, i certainly think an “ObjectWrapper” could be useful. It would be specific to heaps, and you would need to populate the contents of the wrapper via code, something like:

<vbox>
    <object-wrapper id="wrapper1" />
    <object-wrapper id="wrapper2" />
</vbox>



wrapper1.object = myHeapsObject1;
wrapper2.object = myHeapsObject2;

It would be very specific to haxeui-heaps ofc, haxeui-openfl would have a SpriteWrapper, etc. Ill have a think, the component itself is certainly easy enough to write, id just like to see if maybe there is a better way.

My test program is basically the Heaps Lights sample, which shows a 2D visualization of the shadow map in the upper right corner. By being able to attach h2d.Bitmaps like that to HaxeUI components

Sounds interesting. Feel free to share screen shots… sounds cool

Cheers,
Ian

Or could the ObjectWrapper component be part of haxeui-core, but its ObjectWrapper.object member would be a typedef for a backend specific GPU 2D element? The HaxeUI backends would then define internal objects for displaying those. In any case, I don’t know enough about the division of work between haxeui-core and the backends to say whether that would cause issues compared to completely backend-specific ObjectWrapper components.

Please find below a screenshot of the test application. I’ll put the whole program here too in case it might be useful for anyone reading this thread. It is a very simple test, basically just a slightly modified Heaps Lights sample.

Heaps has a small UI system in its SampleApp class and a larger UI system called DomKit, but HaxeUI is more mature and better documented, so it’s nice to be able to use it with Heaps. In the test program, I’ve replaced the Heaps SampleApp UI with HaxeUI components. I’m using the manual update mode for HaxeUI, as I guess it is better to have it synchronized with the Heaps main loop.

The sample program is like this:

import haxe.ui.backend.BackendImpl;
import haxe.ui.components.CheckBox;
import haxe.ui.components.DropDown;
import haxe.ui.components.Label;
import haxe.ui.ComponentBuilder;
import haxe.ui.containers.VBox;
import haxe.ui.core.Screen;
import haxe.ui.Toolkit;
import hxd.App;
import hxd.Math;

class Lights extends App
{
	var lights : Array<h3d.scene.pbr.Light>;
	var movingObjects : Array<{ m : h3d.scene.Mesh, cx : Float, cy : Float, pos : Float, ray : Float, speed : Float }> = [];
	var curLight : Int = 0;
	var bitmap : h2d.Bitmap;
    var styles : DropDown;
    var shadows : DropDown;
    var dyn : CheckBox;
	var inf : Label;
	var dynCullingEnable = true;
    var controller : h3d.scene.CameraController;

	function addCullingCollider() {
		dynCullingEnable = true;
		for( o in s3d ) {
			if( o.cullingCollider != null ) continue;
			var absPos = o.getAbsPos();
			o.cullingCollider = new h3d.col.Sphere(absPos.tx, absPos.ty, absPos.tz, hxd.Math.max(o.scaleZ, hxd.Math.max(o.scaleX, o.scaleY)));
		}
	}

	function removeCullingCollider() {
		dynCullingEnable = false;
		for( o in s3d ) {
			o.cullingCollider = null;
		}
	}

	override function init() {
		super.init();

		s3d.camera.pos.set(100, 20, 80);
        controller = new h3d.scene.CameraController(s3d);
        controller.loadFromCamera();

		var prim = new h3d.prim.Grid(100,100,1,1);
		prim.addNormals();
		prim.addUVs();

		var floor = new h3d.scene.Mesh(prim, s3d);
		floor.material.castShadows = false;
		floor.x = -50;
		floor.y = -50;

		var box = new h3d.prim.Cube(1,1,1,true);
		box.unindex();
		box.addNormals();
		for( i in 0...50 ) {
			var m = new h3d.scene.Mesh(box, s3d);
			m.material.color.set(Math.random(), Math.random(), Math.random());
			m.material.color.normalize();
			m.scale(1 + Math.random() * 10);
			m.z = m.scaleX * 0.5;
			m.setRotation(0,0,Math.random() * Math.PI * 2);
			do {
				m.x = Std.random(80) - 40;
				m.y = Std.random(80) - 40;
			} while( m.x * m.x + m.y * m.y < 25 + m.scaleX * m.scaleX );
			m.material.getPass("shadow").isStatic = true;

			var absPos = m.getAbsPos();
			m.cullingCollider = new h3d.col.Sphere(absPos.tx, absPos.ty, absPos.tz, hxd.Math.max(m.scaleZ, hxd.Math.max(m.scaleX, m.scaleY)));
		}

		var sp = new h3d.prim.Sphere(1,16,16);
		sp.addNormals();
		for( i in 0...20 ) {
			var m = new h3d.scene.Mesh(sp, s3d);
			m.material.color.set(Math.random(), Math.random(), Math.random());
			m.material.color.normalize();
			m.scale(0.5 + Math.random() * 4);
			m.z = 2 + Math.random() * 5;
			var cx = (Math.random() - 0.5) * 20;
			var cy = (Math.random() - 0.5) * 20;

			var absPos = m.getAbsPos();
			m.cullingCollider = new h3d.col.Sphere(absPos.tx, absPos.ty, absPos.tz, hxd.Math.max(m.scaleZ, hxd.Math.max(m.scaleX, m.scaleY)));

			movingObjects.push({ m : m, pos : Math.random() * Math.PI * 2, cx : cx, cy : cy, ray : 8 + Math.random() * 50, speed : (0.5 + Math.random()) * 0.2 });
		}

		var pt = new h3d.scene.pbr.PointLight(s3d);
		pt.setPosition(0,0,15);
		pt.range = 40;
		pt.color.scale(20);

		var sp = new h3d.scene.pbr.SpotLight(s3d);
		sp.setPosition(-30,-30,30);
		sp.setDirection(new h3d.Vector(1,2,-5));
		sp.range = 70;
		sp.angle = 70;
		sp.color.scale(10);

		lights = [
			new h3d.scene.pbr.DirLight(new h3d.Vector(1,2,-5), s3d),
			pt,
			sp,
		];

		for( l in lights )
			l.shadows.mode = Static;
		s3d.computeStatic();
		for( l in lights )
			l.shadows.mode = Dynamic;

		for( l in lights )
			l.visible = false;
		lights[curLight].visible = true;

        // Define the UI
        Toolkit.init({manualUpdate: true}); // Init HaxeUI
        var root = new VBox();
        Screen.instance.root = root;
        s2d.add(root);
        root.addComponent(ComponentBuilder.fromString(
            '<vbox>
                <scrollview width="220" height="400">
                    <vbox>
                        <label text="Shadowmap" />
                        <vbox id="shadowMapBox" width="200" height="200" />
                        <label text="Style" />
                        <dropdown id="styles">
                            <data>
                                <item text="Directional" />
                                <item text="Point" />
                                <item text="Spot" />
                                <item text="All" />
                            </data>
                        </dropdown>
                        <label text="Shadows" />
                        <dropdown id="shadows">
                            <data>
                                <item text="Dynamic" />
                                <item text="Static" />
                                <item text="Mixed" />
                                <item text="None" />
                            </data>
                        </dropdown>
                        <checkbox id="dyn" text="DynCulling" selected="true" />
                        <label id="inf" />
                    </vbox>
                </scrollview>
            </vbox>'));
        
        // Turn off the 3D scene mouse rotation while within the UI area
        root.onMouseOver = e -> if (controller.parent != null) controller.remove();
        root.onMouseOut = e -> if (controller.parent == null) s3d.addChild(controller);
        
        // Connect the UI to the scene
        styles = root.findComponent("styles", DropDown);
        styles.onChange = e -> {
            if (styles.selectedIndex != curLight)
            {
                for( l in lights )
                    l.visible = false;
                curLight = styles.selectedIndex;
                if( curLight == lights.length ) {
                    for( l in lights )
                        l.visible = true;
                } else
                    lights[curLight].visible = true;
            }
        };
        
        var modes = ([Dynamic,Static,Mixed,None] : Array<h3d.pass.Shadows.RenderMode>);
        shadows = root.findComponent("shadows", DropDown);
        shadows.onChange = e -> {
            if (lights[0].shadows.mode != modes[shadows.selectedIndex])
                for( l in lights )
				    l.shadows.mode = modes[shadows.selectedIndex];
        };

        dyn = root.findComponent("dyn", CheckBox); 
        dyn.onChange = e -> {
            if (dyn.selected != dynCullingEnable)
			    (dynCullingEnable = dyn.selected) ? addCullingCollider() : removeCullingCollider();
        };

        inf = root.findComponent("inf", Label);

        // Add the shadow map view to the UI
		bitmap = new h2d.Bitmap(null, null);
		bitmap.scale(192 / 1024);
		bitmap.filter = h2d.filter.ColorMatrix.grayed();
        root.findComponent("shadowMapBox", VBox).addChild(bitmap);
	}

	override function update(dt:Float) {
        BackendImpl.update(); // Update HaxeUI

		for( m in movingObjects ) {
			m.pos += m.speed / m.ray;
			m.m.x = m.cx + Math.cos(m.pos) * m.ray;
			m.m.y = m.cy + Math.sin(m.pos) * m.ray;

			var cc = Std.downcast(m.m.cullingCollider, h3d.col.Sphere);
			if( cc != null ) {
				var absPos = m.m.getAbsPos();
				cc.x = absPos.tx;
				cc.y = absPos.ty;
				cc.z = absPos.tz;
				cc.r = hxd.Math.max(m.m.scaleZ, hxd.Math.max(m.m.scaleX, m.m.scaleY));
			}
		}
		var light = lights[curLight];
		var tex = light == null ? null : light.shadows.getShadowTex();
		bitmap.tile = tex == null || tex.flags.has(Cube) ? null : h2d.Tile.fromTexture(tex);
		inf.text = "Shadows Draw calls: "+ s3d.lightSystem.drawPasses;

		for( o in s3d ) {
			o.culled = false;
		}
	}

	static function main() {
		h3d.mat.MaterialSetup.current = new h3d.mat.PbrMaterialSetup();
		new Lights();
	}

}

It can be compiled like this:

haxe -lib heaps:1.10.0 -lib hlsdl:1.13.0 -lib haxeui-core:1.6.0 -lib haxeui-heaps:1.6.0 -D windowSize=1024x800 -debug -main Lights -hl Lights.hl

And the end result should look like this:

2 Likes

Or could the ObjectWrapper component be part of haxeui-core

thats a good point… the backend defines a ComponentSurface typdef, which in the case of haxeui-heaps is a h2d.Object, so i could have a class in core that uses that ComponentSurface directly… … :thinking:

(SurfaceWrapper sounds crappy though… maybe NativeDisplayWrapper… or something)

1 Like