Saturday, May 13, 2006

Lock Testing

Been busy testing the generic locking class and implementing the specialised derivatives to establish the lock-hierarchy and so far it all works - nice!
Added bonus has been getting the Visual Studio Team System testing framework going again as that is allowing stress testing and such madness to be initiated.

Incidentally it also occurred to me that the locking primatives used to get page level locks could be written to support asynchronous operation (they are fully synchronous at the moment) which might just eke a bit more performance out of them - not sure if I want to go through the pain
Need to reimplement testers for the buffer devices as actually testing the transactioning of a page may well take some time - I dunno I shall see.

The lock manager and the associated lock-owner-block/transaction owner blocks have been rejigged - they are easier to implement now that I'm thinking in terms of how they will be used rather than as the next layer out from the page/buffer implementation! That also made the page logic easier to implement so maybe there is something to learn here...

As ever there is no rest for the wicked and I've turned my attention to tables. The row reader needs careful consideration as this code will need to read both from the table pages themselves and from a result-set defined on search results. I need to think about that some more...

Friday, May 12, 2006

ACID Fundamentals - Locks

I'm wading around in the guts of transaction locking code and it is nothing short of a nightmare! I have a number of generic classes which do all the hard work these are then specialised by final classes for each lock type which deal with the specifics. These "specifics" amount to handling state transitions and determining compatable lock types from different transactions and even the state class is defined within the generic implementation - very clean and somewhat tidy.

Right now I cannot decide whether the escalation behaviour needs a way of being plugged into this generic implementation and I still don't know where to put contextual information such as "whether to hold a read lock until the end of a transaction" as it can't stay in the DatabasePage object...

While I investigate escalation options, I have split the Page lock into seperate sub-types;


  • Database Locks

  • Root Page Locks

  • Object Locks

  • Distribution Page Locks

  • Extent Locks

  • Schema Locks

  • Page Locks



Each of these locks has slightly differing state logic and this is the cleanest way of dealing with that.

Added a collection object to the transaction context which allows transaction context to track the objects which have outstanding locks during the lifespan of a transaction. This is very important when using ReadCommitted (with hold lock) isolation and above as these locks will be released after the commit/rollback has happened - hence the best place for that is the transaction context - not the lock manager!

Weekend should involve more major work on this project - it's Friday already and I should be sleeping!

Tuesday, May 09, 2006

Async Wrappers

The implementation is once again making more and more sense - proof of my own self-delusional state or perhaps proof that the project and the design are going in the right direction! Now fleshing out the data device initialisation/mounting code and that is proving not too strenuous. Need to get the root page and distribution page init code sorted - then I'll be able to see the log-writer do its work - I can't wait!

I was under the mistaken impression that these updates to promote asynchronous behaviour and removing swaths of class hierarchy were going to make the app a dash simpler but as I found out the stack trace during writes is actually longer than before - lots and lots of async wrapper objects the root of the issue - I may need to assign these wrappers from a pool of the things to keep the C# memory manager happy but then again this is what .NET is all about so I'll just flag it for now!

Taken a brief look at the index manager implementation - which still looks rather slick with it's generics all over the place and crossed another TODO off the list... I created an initial implementation of B-Trees operating over pages ages ago but the code was totally synchronous. I realised it needed some careful rework in order to get it to work efficiently with the BeginLoadPage/EndLoadPage APIs that have cropped up following the async conversions and today I can happily say I have solved it with some of the scariest code I've ever written!! Not scary for it's complexity - it is some of the most elegant encapsulated OO code you'll ever meet - no, what scared me was the fact that when I started I didn't actually think the task at hand was entirely possible - writing the B-Tree handler in the first place was nothing short of pain and misery...

Worse yet I still need to provide the B-tree implementation for the table index manager - similar but with the added complication of defining a class hierarchy for dealing with the different data-types I plan to support and the obvious headaches involved in doing string comparisons... I have never understood the various collations - ever...

Anyways I'm happy - my writing of asynchronous code has come of age - almost to the point where I am considering writing an article on just that! Watch this space for a URL...

Need to revisit the locking implementation as it currently needs too much information some of which is not present until the page is loaded - a situation I am keen to avoid...

Database Devices Cracked

Wahey! Good news!

Page level functionality is now tested and working. This includes the behaviour of the CheckPointer and the Log Writer although in the case of the latter the recovery process remains untested and uncharted waters!

Extensive debugging was been achieved since the installer classes were rewritten to emcompass the new class framework (the test harness utilising the installers was already in existence) and now the Buffer state-machine has been tested along side the asynchronous behaviour exposed by just about every buffer/page class available!

It's not been plain sailing though - the implementation of the Free Buffer Service had to ensure Data Page device buffers were marked transactional and log buffers were not... The Buffer state machine pattern also unearthed a number of peculiarities which caused the moving of some of the state switching logic and I also found out that the NestedContainer object does not forward calls to GetService to the owner component - so I had to write one that did in order to get my own proprietry routing chain to work!

Now that the Log Writer appears to be working - at least in an initial capacity (I've just had to changed the default log-page block size - it was far too large) my attention will now turn to initialisation of the primary data device and the setup of file-group primary devices which share more than a few attributes.

After that the real hard work begins!

Saturday, May 06, 2006

Buffer Internals

Finally the asynchronous support has been completed. :-D and as a result the BufferDevice hierarchy is far simpler and the PageDevice hierarchy is also far flatter.

So now I have two class hierarchies - one deals with buffers and can be considered the low-level API and the other deals with pages and can be considered the next level up the chain. The buffer handlers were surprisingly easy to write - more a question of moving code from various other classes which was made a nice change.

The page classes were also surprisingly straightforward especially given the fact that I chopped lots and lots of classes out!!

The real pain came when I decided to unravel the state machine for buffer objects - this turned into a week-long adventure but the result is an incredibly flexible finite-state-machine which ensures consistent buffer state transitions and proper state handling without littering the Buffer class with lots of boolean flags! This task was necessary in order to make the object fully asynchronous...

The buffer can now support the following async operations;

  • Read From Device Stream

  • Write To Device Stream

  • Write To Log Device



With this support in buffers and their corresponding devices complete, attention now turns to pages and their devices. There are a couple of loose ends which still need to be tied up with regard to hand off and lock acquisition - I also need to ensure the transaction handler will correctly unlock pages during the commit phase...

Finally the outer DatabaseDevice device can be completed with regard to recovery and the final mounting procedure before I once again fix the installer classes (and use them to test the initial portion of the codebase).