Superclipping against framebuffer texture
I've been recently trying to do a demoeffect where I render a 3d cube in a backbuffer and have another CEL with the LRFORM flags on to capture the region of the framebuffer and render it in the visible screen as a zooming sprite. And I fell upon yet another hardware bug that of course doesn't replicate in any emulator. When the zooming CEL sprite grew so big that it touched the edges of the screen on the right side, suddenly the frame rate would reach something like 1FPS. This is not a performance issue, but a sudden lock of rendering when as I realized, enabling super clipping on an LRFORM CEL is problematic on the real hardware. I don't know why, but this is what I found after burning few more CDs to try various things.
I have talked about super clipping before, and it's a flag (or two) in a CEL that will enable faster clipping/early termination functions when a CEL polygon is clipped against the screen edges. Depending on the side it clips (left-right, up-down) different things will happen, like if a bitmap line is rendering and reaches the right side of the screen with a scaling vector going to the right, it will know to terminate early as it's pointless to continue. As this option seemed to be mostly free (not significant performance reduction for several sprites, whether you enable it or not) in my engine both for the polygon and the sprite (which are really using the same CCB struct, the polygon just has to manipulate the 3rd vector of the CEL, while the sprite doesn't) will by default enable these flags because it always seems like a gain.
But first time I realized one kind of CEL that you should not enable this, at least the Line Super Clipping (ALSC) which is the one that seems to create the problem. The Cel Super Clipping (ACSC) seems ok, however performance-wise seems to not do much compared to the ALSC. The docs even say that it's not ready yet, but maybe it was implemented in the hardware by that time, there are few cases where it makes a difference, but in most cases the ALSC will save you. Anyway,. better keep this off if your CEL that displays a feedback framebuffer texture is going to clip with the screen edges.
Why does this happen I don't know. It could be an internal bug of the CEL hardware or something that makes sense if you know how the CEL hardware works when the source is not a texture but the framebuffer. The special LRFORM flag will change the way a bitmap line is read. It will really be two bitmap lines, as the videoram structure follows a zig-zag pattern, read a 16bit pixel (X=0, Y=0) and then the pixel below it (X=0, Y=1), then the next pixel up (X=1, Y=0) then down (X=1, Y=1), etc. And that's how 320*16bit*2 = 1280 bytes of data might be read serially and probably passed to the two CEL parallel processors (which I know normally will each share odd/even bitmap scanlines and render them in parallel). It could be something messing up with this, the line superclipping early terminating the last scanlines and something doesn't synchronize well with the workers and it locks. Although I did another test and used two flags that enable/disable this parallelism or each one having to wait for the other to finish, and that didn't fix the problem. I can only assume here. For the practical programmer, they just have to not use superclipping in CELs that use the framebuffer texture as source. It's not a big issue, we can live without it. I am only looking at it out of curiosity of learning more of the 3DO quirks and maybe understanding how the CEL renderer works behind the metal.
I have uploaded a fixed version of the effects above: