Wednesday, August 30, 2006

Transaction Log Testing

It always pays to test your codebase regularly and I have finally got to the point where I can test the transaction logging code and you'd be surprised with the amount of code I have had to write in order to get this far!
However it has also been such a surprise to find that most of the transaction log writing code worked without modification! The only part requiring work was the class responsible for providing a virtual file-system over a stream - this object is crucial for dividing the backing storage used for transaction logs into smaller chunks. It had a problem where it was reporting the stream position incorrectly which led to all plenty of unrelated problems...
So the writing of transaction log records is working but the recovery process still remains untested but in order to conclude that portion of the application I will need to get to a point where the cache writer is correctly saving logged pages and that means I will be retesting the buffer class state-machine which until recently had a few too many states!

Fields, Properties and Serialisation

In a bid to ease the task of writing row and index information to a given page I set out on a major mission (major due to the number of classes that would need modification) to revised the use of explicit member fields and change these into objects that can not only serialise themselves to and from a suitable backing store but also support the concept of being locked.

Locked fields support both read and write of their associated values whereas unlocked fields perform a dummy read of existing data instead of the corresponding write. This makes it easier to update a buffer when extent information of a distribution page changes without causing all distribution page extent changes to require an exclusive object lock - this alone will speed concurrent updates but will need further coordination plus additional LogEntry derived classes to deal with the information actually written in the event of a rollback operation.

As one can imagine with over 100 classes and almost one thousand fields and properties to update this was an erronous task to undertake...

The net result was a true simplification in the implementation of Page object persistence and as an added bonus the persistence of both table and index key information has become so simple I ended up removing classes - that is always a great feeling!

So after a good 50 hours of near continuous programming I have finally got the codebase back to a situation where it builds! It took another 12 hours to fix the variety of bugs and race-condition related problems before the creation of a database is actually working without problem!

No time to rest and relax - I also modified the LogEntry classes to incorporate the same mechanism for reading and writing themselves.

Sunday, August 06, 2006

Async Continues

Page splitting implementation has at long last been converted to a fully asynchronous operation and relatively painlessly too - I must be becoming something of an expert in writing implementations of IAsyncResult as it seems to be getting easier and easier although I must admit to wondering sometimes with all these async wrappers all over the place where the real thread is hiding doing the actual work - scary but ever so true...

It's easier doing this stuff than doing the "real" work I'm dreading - finishing off the index manager and table manager... What's worse is that I'm almost certain I'll need a final wrapper that sits on top of the Table Index Manager and the Table Page Manager and coordinates the actions of both.

It never ceases to amaze me to the sheer number of layers and wrappers this project seems to be creating - I once joked it was like peeling an onion but in actual fact it is more like building an onion!

Saturday, August 05, 2006

Magic Tables, Constraints and Columns

It's been frantic - as the low-level engine edges ever closer to completion my attention has been squarely focused on that mainstay of RDBMS systems known affectionately as tables!

As it happens the table implementation has not proved too difficult to implement thus far and currently support is in-place for the following;


  • Overflow table definition pages

  • Column constraints

  • Data page-splitting



The row block writer and row block reader objects have an initial implementation and the internal row organisation is finished!

The work required to finish the row block persistence can only continue when the Table Index Page/Table Index Manager logic has been completed. This will take a bit longer now that an implementation of clustered indices is required plus the Index Delete/Index Page Combine operations have yet to be written - yet more work!

Magic Tables, Constraints and Columns

It's been frantic - as the low-level engine edges ever closer to completion my attention has been squarely focused on that mainstay of RDBMS systems known affectionately as tables!

As it happens the table implementation has not proved too difficult to implement thus far and currently support is in-place for the following;


  • Overflow table definition pages

  • Column constraints

  • Data page-splitting



The row block writer and row block reader objects are complete and the internal row organisation is finished!

This work needs to be synchronised with ongoing work on the Table Index Manager to ensure we can add/update/delete rows along with appropriate index trees - yippee!

An old column object has been extended to provide serialisation support to both RowReader and RowWriter classes - as one might have expected - to centralise persistence logic and column capabilities. Hence when I get around to supporting User Defined Types or whatever the rest of the code should simply carry on working - famous last words I know...