ASDF now compiles my cl-sdl

The title says it all really. I "fixed" the cl-sdl library up to the point where I can do an asdf:compile-op without triggering the debugger. I had to install some extra SDL libraries, and I had to fix a Makefile for the stubs cl-sdl uses. And, of course, there is still that mysterious symbol redefinition I mentioned in the previous post. But that was it.

ASDF progress

Things have slightly slowed down, mainly because I have been focusing on something for work (an island-based Cobol parser, if you're curious). I have fixed the warning in the sdl component, and now asdf compiles it just fine. I still don't get where the warning is coming from though. It complains about a duplicate definition of "cd-status", but if that's so then I don't find the second definition. I do see both a foreign type and a foreign function named cd-status, but disambiguating those doesn't help. The warning only goes away when I rename the label for the foreign function. Anyway, I'm now planning to check the other parts of cl-sdl (I already did the sdl-img component, which required installing the sdl-image library). Once I've gone through all of them I'll add my own fixes and start testing the uffi-generator script for real.

ASDF weirdness

Why is it that

(asdf:operate 'asdf:compile-op :sdl)

fails, when

(compile-file "/path/to/cl-sdl/sdl/sdl.lisp")

succeeds ? The weird thing is that I see the same compilation of the same file except for a final:

;/path/to/cl-sdl/sdl/sdl.fasl written
; compilation finished in 0:00:00
WARNING:
COMPILE-FILE warned while performing # on
#<cl-source-file sdl="">.
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
;   caught 2 WARNING conditions
;   printed 20 notes

I know the file compiles just fine. Hell, even when things go wrong it still says it wrote out the fasl. So it must be something with asdf. Unfortunately, I have no idea where to start looking...

[edit] I read somewhere that "asdf regards a full warning as an error". There are indeed two warnings, so maybe that's what is tripping things up. Two options then: try to fix the warnings, or figure out how to change this behaviour in asdf...

[edit] Here is a message I found stating the "warning is error" behaviour. It's really counterintuitive; which is to say that every other compiler I have ever used treats warnings just as warnings. Ah well. It seems the best option is to fix the warnings, so I'll start looking into that.

Forking cl-sdl

Yes, the title says it all: I'm going to work on a custom version of cl-sdl. Hold your "shenanigans", because I will do it in the easiest way possible. First question is why do it at all ? Well, I want to integrate my sdl/aqua fix (see previous post) into cl-sdl, but that's not really a good reason. A much better reason is that cl-sdl is not being actively maintained anymore. The last release stems from end 2002, and the mailing lists are virtually dormant. OpenGL support is stuck at version 1.2, with a comment on 1.3 but no further development. This last fact is especially bothersome as I want to use any OpenGL functionality I have available on my machine, not just what cl-sdl is offering. The most important reason however, is to fix the installation problems I have had. I'll need this when I want to start sharing my engine with other people. If they can't even get the OpenGL binding going then I'll have a hard time getting them to test my real code. So how to do it easily and keep the effort at a minimum ? As for integrating the sdl/aqua fix and correcting the installation, I think that is just a one-time investment and I should just do it. The bigger bother is keeping the package to move along with new versions of OpenGL, but I think I can automate much of that process. You see, at the core of cl-sdl are UFFI bindings which basically offer a lispy description of whatever is in the OpenGL library. Someone needs to make this description and then you can use whatever is in that library. Really cool, but maintaining that description can be quite bothersome. It should however be possible to generate most of that description automatically. Simply parse the relevant header file and translate it into the equivalent UFFI code. This is what I have been experimenting with, and it seems to be quite doable. Sofar I have a tool which processes most of the gl.h header, and which generates a UFFI which is close enough to what is in cl-sdl to be almost indistinguishable; except that it now also has OpenGL 1.3 functionality. This is giving me good hopes of getting it working, and will mean that when there is a new OpenGL header I can just rerun my tool and I'll be instantly up-to-date. (As a possible bonus it might be possible to use the same generator script against other headers which will allow me to interface with many other interesting C libraries. There's nothing I need right away, but it might prove a time-saver in the future.) The todo now is to finish of a first version of the tool and then verify that its output still produces a working cl-sdl.

I got my fix

Isn't it beautiful ? Ready Lisp showing Aquamacs connecting to an sbcl repl which is running the basic cl-sdl test. Seems to work just fine, though I did notice that any messages printed by the app don't show up until after you close the window. This doesn't happen when running it straight in the terminal.

_NSAutoreleaseNoPool troubles

So, thanks to a quick reply from John Wiegley (who makes Ready Lisp), I was able to get sbcl running from within the terminal. If anyone else wants to know how to do this here is the magic incantation (assuming "/Applications/Ready Lisp.app/Contents" as the pwd):

./Resources/sbcl/sbcl --core ./Resources/sbcl/i386/lib/sbcl/sbcl.core-with-slime

If you're on powerpc you'll need to replace the i386 with powerpc. And I also suggest aliasing this command in your .bash_profile to make it friendlier. Back to my problem: when trying to run the sdl-demos I get the following (familiar) messages:

2008-05-11 09:13:30.048 sbcl[195:10b] *** _NSAutoreleaseNoPool(): Object 0x121a20 of class NSCFNumber autoreleased with no pool in place - just leaking

This is the same problem I saw when trying to get cl-sdl working on cmucl. It basically has to do with SDL and Cocoa. Cocoa needs special initialisation which SDL only provides for main methods. As there is no "main" method here, the initialisation does not happen and the demos don't work. So the next step will be to see if my old solution to this problem still works.

Technical problems already ?

Some initial background: my intent is to use a (possibly customised; but that's a future post) version of cl-sdl. This is basically a uffi binding to SDL, which provides the windowing, event-handling and, most importantly, OpenGL capabilities. I have already played with cl-sdl once from within cmucl (a Lisp implementation), doing a very basic heightmap renderer. Basically a port of NeHe tutorial number 34. Getting cmucl and cl-sdl to work together on my mac took some doing but it worked nevertheless. This time I have opted for sbcl rather than cmucl, for one simple reason: Ready Lisp. This is a complete package which has sbcl, aquamacs and slime working right out of the box. Pretty cool actually, and it saves me from having to set it all up myself. Anyway, as sbcl is related to cmucl I was pretty confident that I could get cl-sdl back up and running. Not so... When I try to install cl-sdl through asdf it downloads just fine, but during compilation I get an error saying that sdl-ext references sdl:get-video-info but that sdl does not define this. This is utter stupidity as sdl defines it just fine. My guess here is that there is a build-order problem. You see, it compiles sdl-ext before compiling sdl, and so yeah the symbol may not be there yet. So next I resort to doing manual compile-file invocations, doing things in a better order. Guess what, no complaints anymore. So something is definitely going wrong in this asdf. I didn't have this problem in cmucl... But still things go wrong. While I can now require sdl, opengl and sdl-demos without problems, when I actually try to run one of the demos slime loses the connection with the repl... What I guess might be going on this time is that the demo is trying to open a window, but that because of the way aquamacs/slime is connecting to the repl this might not be possible. So I would like to start sbcl up from within a terminal and try running it that way. This is what worked fine in cmucl. Of course, first I have to figure out where Ready Lisp is keeping its version of sbcl. Probably somewhere in its app package, but I'll have to look. So no joy yet. Nor any pretty pictures of triangles floating in mid-air. :-)

Challenge

I'm a self-confessed computer geek (see, I just confessed to it), and as with most such geeks I love programming. I will program anything and everything that interests me, and there lies the problem: there's too much of it, and I don't finish anything. I usually get stuck architecting and implementing stuff I don't end up using (look up "axe sharpening" and "yak shaving"). And then I slow down with the overall project, and I'm eventually left with another piece of abandonware. I'm tired of ending up with nothing to show for myself when I undertake these projects, so I'm issuing myself a challenge: over the course of the next year (with this post marking the start date) I will work on one project and one project only. Mimimal yack shaving, minimal features, and good-enough code will be the motto. The project I'm choosing is one for which I already have some experience, and for which I know there's a lot of good resources to help me out: a GPU-accelerated 3D engine (yes, I know there are loads of those out there, but there will be a slight twist to mine; keep on reading). Here are the features I'm going for (and these features only):

• Support for one format for indoor scenes (one of the BSP formats most likely).
• Support for one format for outdoor scenes (octrees, CLOD, or some such).
• Support for one format for characters (will likely be MD5, though I may go for one of the older formats).
• Support for one format for static models (Milkshape obj perhaps).
• Support for basic skyboxes (just to make the scenes complete).
• Support for mixing any of these in any way I want (I already have a comprehensive design document on how to do this).

For hardware acceleration I will be using OpenGL. No DirectX; I'm on a Mac so that's not an option. And I won't be setting up my engine so that I can later easily choose one or the other (I don't care about that anyway). Windowing will most likely make use of SDL, mostly because there's a lot of documentation and help for it, and because it will help me stay multi-platform. Now for the twist. I have started a similar project once with a friend where we used Java as the implementation language. I really don't want to use Java anymore. It was once good-enough for me, and it's still the language I'm most fluent in. But I've had enough. So instead I'm going to do my 3D engine in Lisp. ***Pause to let readers catch their breath. With me again ? OK.*** Honestly, I don't think this is much of a shocker, but I know some of you may find it strange anyway. Seen as I can't find any 3D engine in Lisp I'm guessing it is somehow not an obvious pick for most of you. Still, I'm hoping this will count as something new and interesting to more people than just me. As for art/assets, I will stick to what I can find freely online, though I will allow myself to reuse some of the assets I made for my Java engine. I can tell you now that this restriction will likely be even more of a challenge for me than the technical stuff. Even not doing anything on the design for this blog is making me itch... Just to be clear about this: I will consider my project a success when I'm one year further on and I have kept my focus on this project and this alone. That is: no other projects were started (though, of course, I won't count anything I'm doing for work, nor anything which I'm working on to help out someone else), and this one has not been abandoned. Not implementing all features will not count as a failure (as I have no idea how much time each feature will take nor how much time I will have overall), but spending too much effort on unlisted features will. Now for the blog-part and you, the reader. In order to keep myself focussed on my goal I'm setting a task of minimum of one post per week listing what I did or did not do during that week. You will get the right to call "shenanigans" whenever you feel I'm straying too far from the challenge. I'm hoping you won't need to but, then again, if I knew for sure I wouldn't be needing this blog/challenge anyway. And with all that, here's hoping I won't fall flat on my face. :-|