Steve Carter on Software Development

THoTH version 2.4

2008-11-11T11:57:00.002-05:00

I've just released version 2.4 of THoTH. Read about it here.

Search, forum, audio

2006-07-27T11:04:00.000-04:00

I added the freeware version of Zoom search to my site, to see if I liked it. It's quite impressive, does a good job of indexing. I'll have to make the search capability more readily visible and see if my visitors use it. If so, I may upgrade to the commercial version.

I'm looking into phpBB to add a forum to my site. Looks easy enough, and flexible.

I added an Audio page to THoTH. So now users can not only see music notation for a song or musical exercise, but they can hear it. My plan is to record some of the musical exercises and make mp3 files available to THoTH users. Haven't figured out exactly how to distribute them yet. I think a zip package that would install the files and update the database through a script.

THoTH 2.0 Released

2006-04-23T14:32:00.000-04:00

I've released THoTH version 2.0 (See Overview). I think it's the only program currently on the market that can do chord/scale analysis of chord progressions from MusicXML. Currenlty I only have a couple of MusicXML files provided with the release. I'll be adding more as a free upgrade, and more after that for a small upgrade fee.

THoTH song form additions

2006-03-30T21:03:00.000-05:00

I've decided to add two tabs to the song form: Lessons, and Notation.

Lessons will contain a list of lesson files. Selecting one and clicking the View button will bring up the PDF. At first it will be my lesson PDF files. But the THoTH user will be able to add his own lesson files to the list. This might be helpful to music teachers and students.
The Notation tab will contain a list of notation files (Finale, Sibelius, Notion, etc.) I'll provide a few notation files for the tunes I've provided lessons for, such as "Blue Bossa" and "Body and Soul". Since this is for educational purposes, I think that will fall under the "Fair Use" section of the copyright law. The THoTH user will be able to add his own notation files to the list. Each row in the list, in addition to the file name, will have the name of the notation program that created the file, so selecting a file and clicking View will open the file in the appropriate program (assuming the program can be launched with a file name in the command line).

THoTH ready for next Beta

2006-03-19T19:51:00.000-05:00

I've finished the database analysis engine and worked it into THoTH. I created two test MusicXML file with Finale PrintMusic, and ran analysis in THoTH. The tunes are "Blue Bossa" and "Body and Soul". The analysis looks correct (although I still need to deal a better handling for the D/F# in the bridge of "Body and Soul".

So now the user can add a tune record in THoTH, fill in the name of the MusicXML file, and run an analysis. THoTH will display the analysis, save the analysis to the database, and update the tune record.

Once I've add more MusicXML files, run more analyses, and added more results to the analysis table, I'll be able to to meta-analysis across tunes. Like "what is the most common measure for the appearance of a IVm6 chord?"

THoTH analysis data-driven

2005-12-17T20:20:00.000-05:00

I've created an Analysis table and inserted rules for all dominant seventh chords, based on Bill Leavitt's approach. I can do a simple SELECT statement and get the analysis for any dom7 chord in any key. For example, I can ask, in effect, for the analysis of Ab7 in the key of Bb, and it will tell me bVII7 AbLydianb7 EbRMM (Real Melodic Minor).
So the methodology is in place. No it's just a matter of adding the other rules to the database. That will be a couple of hours work. At that point I should be able to parse a MusicXML file and output the analysis.

Firebird's selectable stored procedure made this task pretty simple.

Logical

2005-12-10T22:09:00.000-05:00

12/10/2005

I've been working on THoTH, translating the program logic into database logic. In the original program, all the rules for chord-scales were in one large "IF" statement, and now I'm moving them into the database. For readers who are confused by this, all it really means is that the rules are being moved from a text file to a database. The rules are based on Bill Leavitt's approach to chord-scale analysis, as he expressed them in his Modern Method for Guitar series.

As I was working on this, I was thinking, "What would Bill think of all this -- translating his chord-scale analysis into computer code?" He never had any involvement with personal computers -- they came into common use very near the end of his life. But he would have liked the logic of the computer. Bill was a very logical guy. His analysis was systematic, which is why it translates so well to computer logic. But Bill also understood exceptions. He used to say, "There is no such thing as 'never' or 'always' in music." I thought of that as I wrote the logic for transposing B-natural up a fifth. In most cases, if the root in question is named with a sharp, when we transpose up a fifth, we name with a sharp: up a fifth from C# is G#, not Ab. And if the root is natural, up a fifth is natural: up a fifth from A is E, not D##. But up a fifth from B is F#. We do this intuitively. But computers are not intuitive; we must tell them about the exceptions.

Testing testing

2005-12-04T22:00:00.000-05:00

THoTH is out to a few Beta testers, but I haven't been able to follow up very well recently, because I've been so busy, between my day job and gigs. I'm about to put out a new Beta build, and I'll contact all the current Beta testers and make sure they are actually testing.

Back to the MusicXML future

2005-06-28T22:52:00.000-04:00

I've implemented the start of MusicXML analysis in THoTH. I was using a component called SimpleXML to parse the MusicXML, but it provided no help and no examples. I switched to NativeXML, which has good help and good examples.

I used BrookWestSample.XML from Recordare as my test MusicXML file. I parse the file, pull out the chords, create a chord object for each one, with root, chord type, measure, and full chord name, then insert the chord object into a list. Then I can traverse the list for analysis and/or printing.

I need to map the MusicXML chord types to chord types that are more usefull to me. Then, with slight changes, I can use my analysis engine on the chord list, and, as I did in the previous version of THoTH, both update the song record in the database, and show the analysis.

The advantage is that a user who can export MusicXML from, say, Finale or Sibelius, can use the THoTH analysis. Still some work to do to make it user-friendly, but it's getting there.

THoTH Beta available

2005-06-25T09:02:00.000-04:00

THoTH is coming along nicely. I released a Beta to a few musicians who had contacted me asking about the application. (If anyone reading this blog is interested in being part of the Beta-test team, send me an email.)

Since the first Beta release, a few days ago, I've added some functionality: add a song, delete a song. I aslo modified the Find dialog so that hitting Enter opens the selected song.

THoTH resurrected yet again

2005-06-19T09:25:00.000-04:00

After a long hiatus, I'm back to work on this blog. More importantly, I'm back to work on THoTH.

I'm planning to release a Beta version to a few testers in the next week or two. It won't include the analytic engine, but it will provide a pretty good database, and a few analysis examples.

THoTH uses a very simple database, just one table for tunes. In the previous version, which I used in my teaching for several years, I used a Paradox table, and the Borland Database Engine. For the upcoming release, I've switched to a Firebird (Interbase) table, using the embedded Firebird server. So the end user will not have to bother with database installation. Just fire up the program, and all the data is available. I will have to include a backup menu option, so that the THoTH.fdb Firebird database gets backed up properly.

The Tunes table contains one row for each tune. It's completely denormalized. Each record has one field for each of the seven modes for the three scale types, one field for each of the 23 chord types, and so forth. For my purposes, this works fine. The display values for mode names, chord names, key names, style names are all stored in the ini file. An end user can change them -- I'm not sure at this point whether that's a good idea or not.

My analysis engine, up to this point, worked with MusicPrinter Plus files. The next version will work with MusicXML files. I haven't implemented that yet, though I've done some work on it. For the first Beta, I'll take a couple of dozen of my existing analysis output files, which are plain text, format them for HTML (as I do on my web site), and include them.

I need to start on the help files. Very little help is needed for the actual program operation; it's pretty simple. But I want the help to include explanations of the music theory behind the system. I need to give more thought to integrating this.

Harmonic Analysis Examples

2005-02-20T12:08:00.000-05:00

From web stats, it looks like not many users were finding the harmonic analysis examples on my web site, so I added a link to a new page. On the Lessons page, I now have a link: Harmonic (Modal) Analysis Examples.

I have a MySQL database on my site that contains the data that THoTH uses. For each tune: title, composer, key, chords, modes, etc. I wrote a PHP page that queries the databse for tunes that have a value of "Y" in the has_analysis field. This means that I've taken the analytic output of THoTH for that tune and massaged it (I'm rewriting THoTH and in the new version I won't need this step). The PHP calls another page the reads in the analysis text file and outputs formated HTML, then displays the result.

Progress?

2004-12-31T19:31:00.000-05:00

I was just reviewing my journal, and I read a comment about implementing some things in SUBMIT, using MySQL, that were similar to things I'd done years ago in BASIC using Paradox tables. I was surprised to find that it took so long. In some ways, progress doesn't help much. The problem with MySQL is that it doesn't (at least in the version I'm using) provide IN and nested queries. So what I did was export the data to Interbase, write the queries there, look at the sets I needed to create, then rewrite the queries using temporary tables in MySQL.

At some point I want to port SUBMIT to Firebird. I'm tired of workarounds. I'll keep MySQL on my laptop, because my Frogstory site uses PHP with MySQL tables, so I'll need a working environment.

I still have a shortcut on my desktop to the TIMEMAN application I wrote years ago in Delphi. It was my main time management program for many years, dating back to the days before PIM was invented. In many ways it's better than the Palm solution I use today. I use DateBk5 and ShadowPlan, along with the Palm desktop app, and some programs I wrote myself, but I still don't have some of the functionality I had in TIMEMAN. For example, I assigned timed tasks to projects and could get an instant report. Sure, in DateBk5 I can assign categories, but then I can't get a report on the laptop. The way I do it is pretty convoluted. I had written an app to pull data from the Palm datebook into Interbase and then report, but when the Palm file format changed, I gave up on that, and I now track time in MobileDB; I pump those files into Interbase (Firebird) and run reports. But it's still more steps.

I aslo found that when "upgrading" to OfficeXP from my academic verions of Office97, the MailMerge sections of SUBMIT stopped working. Word reports the files as being opened -- by me. Annoying. So I just diabled the automated merging and had the app simply open the file, then I merge "by hand". Annoying that I'd have to do that. WordStar was easier to manage.

As a kid, I used to see Ronald Reagan on TV, hosting some program sponsored by GE. He used to say, "At GE, progress is our most important product." Idiocy.

THoTH does HTML

2004-12-20T07:28:00.000-05:00

THoTH displays a text page with the analysis. Over the weekend I wrote a PHP script to convert the text to HTML. Much easier to read. I've added a few of these HTML pages to the search by mode function, so when the user clicks on a tune title, this HTML will pop up. Interesting project.

I modified THoTH to save the analysis to a text file, using the tune title, normalized, as the file name. Then I wrote a PHP page that opens the text file, adds some HTML tags, includes a brief explanation, and displays in the browser. Looks much better and easier to understand than plain text.

2004-12-09T23:14:00.000-05:00

The past two days, I've decided, once again, to commit more of my personal resources to THoTH.

When I first created THoTH, I knew it was ahead of its time. The question always was: When would THoTH's time arrive. Maybe the time is now.

2004-12-04T21:30:00.001-05:00

How to archive and index my Player's Journals. First, standardize the entry dates to dd/mm/yyyy. That will facilitate automated searching and indexing. For the initial pass at archivvng I'll manually extract sections by year. Create one HTML file for each year, leaving the current year in the main page. At the top of main page:
Archives: 2002 2003 2004 Index
with each year a hyperlink to that year's archive. Index will be a link to the index page.
Preparation for indexing. Create 2 CSS classes: entrydate and entrytitle. Then I can search for those, they mark the beginning of an entry.

Indexing. Rejected idea: make page a form and insert hidden fields with keywords. Do I want to hide the keywords? More readable if I do, easier to edit if I don't. There will be topics for entries where the keyword of the topic does not appear in the text. Example: an entry about Bill Laevitt. Topics: William G. Leavitt; music education; Berklee. Idea: a line or more of keywords, maybe grey and in 7-point type so unobtrusive, at the bottom of an entry. Then I could write a Perl script to create the index. But where do I put the anchors? On the keywords, I guess. But then the link in the index takes you to the bottom of the entry--not good. Maybe keywords at the top of the entry.
I can't remember seeing an index like this on the web. A table of contents is common, and some people even label those as index, but they are not indexes.
So if I had these keywords in place, how do I proceed? What text for the index links? In a book, it's page number. I could just use the entry date, but not very informative. Maybe entrydate and entrytitle. Example:
Berlee
11/12/2003 Blue Bossa
12/04/2004 Tribute to Bill

That's not bad. So how would I do this in Perl? Read an entry and then...
Hmmm. Maybe I want certain words to be hyperlinks into the index. The phrase "chord etude" might link to that index entry. That entry might contain a see also section, which contain a link to, say, William G. Leavitt - chord etudes.
It could be fun adding this index information to the journal!
The idea of linking a word to the index solves the problem of one word linking to many places. Ray Kurzweil's site has an elaborate many-to-many linking (The Brain) and I've seen other things like that on the web for links among web sites, but they are too intimidating for my purposes in this player's journal.
So, Perl for creating the index. Read a section - delimited by entrydate tag at start and hr at end. Build a date string from the date entry: 11/22/2004 20042211. Build a title string from the entrytitle: Tribute to Bill -> tribute_to_bill. Create an id: 20042211_tribute_to_bill. Insert an anchor in the journal page with that id. (at the entrydate text?) Write an index entry that is a link to that anchor. That way I can have many index entries linked to any given entry.
Try this on a test file. Create a file journal html with a few dummy entries. Write Perl to insert the anchors and to write the index file. This means reading the input file, adding the anchors to it, as it is output, and also writng to the index file for each entry as it is processed.
In the actual journal file, I might want a CSS class for the journal title, then I could use that as a dilimiter to indicate to Perl where to start processing.

Indexing Player's Journal

2004-12-04T21:30:00.000-05:00

How to archive and index Player's Journal. First, standardize the entry dates to dd/mm/yyyy. That will facilitate automated searching and indexing. For the initial pass at archivvng I'll manually extract sections by year. Create one HTML file for each year, leaving the current year in the main page. At the top of main page:
Archives: 2002 2003 2004 Index
with each year a hyperlink to that year's archive. Index will be a link to the index page.
Preparation for indexing. Create 2 CSS classes: entrydate and entrytitle. Then I can search for those, they mark the beginning of an entry. hr tag can mark the end of an entry. Currently I can have mulitiple entry titles withtin a date. I might need to change that to make indexing and linking easier
Indexing. Rejected idea: make page a form and insert hidden fields with keywords. Do I want to hide the keywords? More readable if I do, easier to edit if I don't. There will be topics for entries where the keyword of the topic does not appear in the text. Example: an entry about Bill Laevitt. Topics: William G. Leavitt; music education; Berklee. Idea: a line or more of keywords, maybe grey and in 7-point type so unobtrusive, at the bottom of an entry. Then I could write a Perl script to create the index. But where do I put the anchors? On the keywords, I guess. But then the link in the index takes you to the bottom of the entry--not good. Maybe keywords at the top of the entry.
I can't remember seeing an index like this on the web. A table of contents is common, and some people even label those as index, but they are not indexes.
So if I had these keywords in place, how do I proceed? What text for the index links? In a book, it's page number. I could just use the entry date, but not very informative. Maybe entrydate and entrytitle. Example:
Berlee
11/12/2003 Blue Bossa
12/04/2004 Tribute to Bill

That's not bad. So how would I do this in Perl? Read an entry and then --
Hmmm. Maybe I want certain words to be hyperlinks into the index. The phrase "chord etude" might link to that index entry. That entry might contain a see also section, which contain a link to, say, William G. Leavitt - chord etudes.
It could be fun adding this index information to the journal!
The idea of linking a word to the index solves the problem of one word linking to many places. Ray Kurzweil's site has an elaborate many-to-many linking (The Brain) and I've seen other things like that on the web for links among web sites, but they are too intimidating for my purposes in this player's journal.
So, Perl for creating the index. Read a section - delimited by entrydate tag at start and hr tag at end. Build a date string from the date entry: 11/22/2004 becomes 20042211. Build a title string from the entrytitle: Tribute to Bill becomes tribute_to_bill. Create an id: 20042211_tribute_to_bill. Insert an anchor in the journal page with that id. (at the entrydate text?) Write an index entry that is a link to that anchor. That way I can have many index entries linked to any given entry.
Try this on a test file. Create a file journal html with a few dummy entries. Write Perl to insert the anchors and to write the index file. This means reading the input file, adding the anchors to it, as it is output, and also writng to the index file for each entry as it is processed.
In the actual journal file, I might want a CSS class for the journal title, then I could use that as a dilimiter to indicate to Perl where to start processing.

SUBMIT through the years

2004-11-20T11:15:00.000-05:00

One of the first things I wanted to do with a computer was to track my writing submissions. I was submitting poems to literary magazines and articles to music magazines. DataStar, which came bundled with WordStar, was a pretty good flat-file database, for the time (this was about 1984). I quickly realized, however, that I needed to track one piece to many magazines and many pieces to one magazine.

I wrote a BASIC program that served my purposes. I used DataStar to manage the tables, and the BASIC program to manage the relations and to massage the data in preparation for MailMerge. That was the first version of SUBMIT. Over the years I rewrote in various versions of Pascal, and eventually Delphi. Today it is written in Delphi 5 and uses a MySQL database. I don't like MySQL, so my next step will be to port it to Interbase. I've been a long-time fan of Interbase.

At one point SUBMIT was written in Turbo Pascal (TP) 5.5. Object-oriented programming was pretty new at the time, and porting to Turbo Pascal was an interesting challenge. Turbo Pascal gradually evolved to Borland Pascal 7 (BP7) and SUBMIT evolved along with it. BP7 had a sample application called TVForms. It combined a data file with a form for viewing the file. I wrote a TP utility to pump the data out of DataStar and into my own modified version of TVForms. Because of the power of the Pascal language, I was able to simulate a relational database in TVForms. Screenshots.

About two years ago I wanted to learn more about Perl, and I thought of trying to get the data out of the TVForms version of SUBMIT and into a relational database, and writing a Delphi application around it.
That was an interesting learning experience. ... more to follow...

Task tracking task

2004-11-17T08:13:00.000-05:00

The never-ending task of task tracking. I have a Tiber Todo list in ShadowPlan. I can filter by checked or unchecked and by date range. But my time tracking is in MobileDB and notes in bugs are in various ShadowPlan lists. So it is still hard to answer the questions: What did I do this week? What bugs did I fix?

TODO: add pages on ShadowPlan, MobileDB. Link to them from this entry.

Some background

2004-11-14T21:29:00.000-05:00

In the early 1980's I was teaching music at Berklee College of Music in Boston. I had always had an interest in science, and personal computers were big science news at the time. I couldn't afford one, but I started reading about computers. I had always done a lot of writing, and the thought of a personal word processor interested me.

Then the Sanyo 555 came out: the first MS-DOS computer for under $1000. I bought one. It came with a bundle of software: WordStar, DataStar, CalcStar. It also came with BASIC, which is a good thing, because, given the limited capabilities of that early software, I really needed a way to make the computer do what I wanted.

New Software Development Weblog

2004-11-14T10:47:00.000-05:00

On these pages I'll be posting my thoughts, research, and notes on software development.