Koopa: new players

I never really expected this to happen but it did: Koopa, my little hobby Cobol parser, found some extra developers!

The Koopa team* welcomes Simon Sobisch and Luís Rodrigues. Simon has been very helpful in detecting bugs and suggesting improvements. In fact I had to start making use of the bug tracker just to keep up. :-) Meanwhile Luís offered a major contribution to the grammar, and is now frequently adding more and more directly to the code. Every commit he makes feels like a little present I get to open. :-)

So to both of them a big thank you!

* I.e., me. :-)

Koopa 1up: GUI update

Finally found something to post about again: Koopa! I spent the last week updating the GUI applications. Rather than having two separate ones (one for batch processing, one for visualising a single file) I have now integrated them both.

(JTabbedPanes look a bit ugly on the Mac, but I don't want to spend time changing that. It works, and that's enough for now.)

In addition you can now simply drag-and-drop files onto the Koopa GUI and it will try to parse them. And once parsed you can easily navigate from a selected part of some source code to the grammar rule which parsed it.

All this will hopefully make for more a more useful and usable application.

Koopa WIP: Cobol Preprocessor

I just added an experimental Cobol preprocessor to Koopa. So far all it does is expand copy books. There is no support for replacements of tokens in copy books, or of any other preprocessing statements. Still, even this basic support was enough for Koopa to be able to parse another three of the five remaining test files which are part of the testsuite.

If you're interested in playing with this (or in expanding it) grab the source code from the project page and start with the Cobol85PreprocessingTest class.

Koopa 2000+

It's been a little over a year since Koopa got its 1000th download, and now the counter has gone well past 2000. Yay!

Thanks again to everyone who downloaded Koopa and tried it out. And especially thanks to all the amazing contributors who actually helped Koopa grow!

Koopa 1up

For those of you keeping track: the Cobol grammar in Koopa just got a major update thanks to a contribution by Adrian Noguero. Koopa now parses more files in the testsuite than before, and overall coverage of the parsed files has gone up to 98.92%! There are lots of source files which score a perfect 100% coverage. That means that for those files Koopa is able to categorize all tokens.

Adrian: thanks man!

Scertify Code

I got a message in my mailbox today from Armel Gouriou from Tocea, about their Scertify Code product. They added support for auditing Cobol code. And guess what they chose as a basis for doing those analyses ? Koopa! They're even so nice to give my little project a great shout-out on their blog!

Coolest thing to happen to Koopa so far! :-)

Koopa 1000+

The Koopa download counter has gone past 1000!

We're not stopping there, of course. I just updated the code with another contribution from Peter Tang. And I'm also thinking about how to add support for parsing conditions (which is proving trickier than I expected).

Koopa 1up

I only just bragged about the first official contribution to Koopa, and here already is a second one. Thanks to Peter Tang Koopa now understands more of:

• EXEC SQL statements
• EXEC CICS statements

Thanks!

PS. Here's a teaser from Peter on what he's doing with Koopa:

We are able to run koopa against 20,000 Cobol programs plus another 20,000 copybooks within 4 hours. We have also adapted pig latin (from http://pig.apache.org) to include User Defined Functions to extract xml fragments from the results generated by Koopa on a Hadoop cluster for other correlation analysis.

It might be just me, but I find this extremely cool. To paraphrase a well-known slashdot quote: can you imagine a Hadoop cluster of Koopa ? :-)

Koopa 1up

Koopa got its first contribution today! Thanks to Adrián Noguero Mucientes' generosity we can now enjoy support for the following Cobol features:

• CONFIGURATION section
• INPUT-OUTPUT section
• LOCAL-STORAGE section
• RETURNING phrase on the procedure division
• SET statement
• INITIALIZE statement
• DISPLAY statement
• INSPECT statement
• COMPUTE statement

I guess Koopa now officialy lives up to the open-source spirit. :-)

Koopa 1up

If you're a Koopa developer the following should cheer your day: I reworked Koopa's build system so that you can compile everything from a simple 'ant clean all'. Up until now I have been relying on Eclipse and the ANTLR IDE to compile everything for me (minus a few tasks which I did automate). This made it hard for new developers to get going (yes, there actually are a few brave souls who gave it a try). The new system should take away that problem.

Look for the updated system in subversion on sourceforge.

Koopa 1up

I just updated the Koopa Cobol Parser with a new version. It now supports parsing Cobol code in either fixed format or free format. Don't know what I'm talking about ? Here are the definitions as found on COBOLPortal:

Fixed Format Source - Each COBOL source line consists of 80 characters which are divided into various fixed areas. There are restrictions on the syntax that can appear in each area.
Free Format Source - Each COBOL source line can consist of up to 250 bytes of characters. There is no restriction on where syntax may appear on the line.

I don't restrict lines to either 80 characters or 250 bytes (note the difference in units btw.), but other than that it should be as described.

The GUIs and the command line application have been updated to give the user the choice of which Cobol format to use. Default is fixed. Just pick what you want in the "Parser settings" menu, or by passing "--free-format" as an option on the command line.

As usual there is an executable jar available on the sourceforge project page.

Koopa 1up

I just updated the Koopa Cobol Parser with a new version. I have been updating the grammar a bit (it now covers ADD, SUBTRACT, MULTIPLY and DIVIDE), but the really new feature is the XPath querying in the GUI.

The XPath support was already in the code, but you couldn't access it in an easy way. With this new release you can now open up a dialog in the code visualiser and perform XPath queries on the full syntax tree. All results link back to the source code with a simple double click. So you can now, for instance, search for all string literals, all paragraphs, all tokens on line 100, the first four data definitions, the last section, etc.

On a totally unrelated note: today marks the two-year anniversary of if-false-panic. Yay! I may not have stuck with that 3D engine in Lisp, but I did stick with this blog, and I have gotten at least one project to the point where it's usable by the public (as a hint: see this post's title).

Koopa 1up

Finally some progress to report: an update of the Koopa Cobol Parser project. The main feature of the new release can be seen below: calculation of coverage.

The horizontal axis iterates over the 513 files in the test suite. The vertical axis shows you how much of that file Koopa actually parses (or, put differently, doesn't ignore). As you can now see, coverage is not bad at all. Consider me a happy developer.

I extended the batch parsing application with the ability to export its results to CSV, which you can then import into the favourite spreadsheet of your choice. So you can run this against any code you have lying around and get a nice visual in a reasonable amount of time.

Other feature: Koopa can now also correctly handle SKIP1, SKIP2, SKIP3 and EJECT statements. These are basically directives for printing the source code; they don't actually contribute to the semantics of the program. Makes you wonder how much Cobol code got physically printed over all these years...

As usual, the executable jar has been updated, so feel free to take Koopa for a spin.

Koopa 1up

I just updated the Koopa Cobol Parser project with the ability to export the syntax trees of the parsed Cobol code to XML. The output is very basic, but it has the full tree as it is seen by Koopa, including tokens in the water (i.e. which got skipped). No whitespace or line/column information though.

The executable jar has also been updated, so you can try before you buy.

Koopa Cobol Parser

I just uploaded the very first release of the Koopa Cobol Parser project on sourceforge. You can find the website for the project here.

If you're wondering "why?", then this project probably isn't for you. A short answer is that there exist no free, extensible, adaptable Cobol parsers which are able to handle real legacy Cobol code. Koopa tries to fill that gap.

While it is still very much an alpha release, there are some redeeming features:

• It makes extensive use of unit testing at the level of individual grammar rules. This helps with quick, detailed detection of problems.
• It includes an ANSI 85 testsuite, which it is able to process quite well. There are some failures, but these are quite reasonable.
• It has been run on over 1.5GB of industrial Cobol code, and again performs quite well.

So if you're looking for a no-strings-attached flexible Cobol parser, give Koopa a try!

PS. If you're wondering about the name, the thought process went something like: Cobol Parsing -> Co Pa -> Koopa...