Friday, May 25, 2007

The Final Shove... I mean - push!

Okay the integration of the new memory manager and low-level streaming improvements is now almost finished - as you can imagine - if you change a set of low-level objects to a new set that function completely differently it may take a while to wade through all the error messages...

If I can find the time then I'll be able to commence testing early next week - if I don't find the time then I may well end up repeating myself next week/next month etc...

Monday, May 07, 2007

Low-Level Engineering

Well I've been so busy with so many other projects that I've not had the time to devote to the Audio Database of late!

All has not been lost and sometimes a break from a project means you can look at it with fresh eyes when you return - assuming of course that you do infact return at all...

My fresh eyes have been taking a critical look at the low-level file and buffer handling and I have implemented three rather important features which have interestingly enough led to a fourth rather radical change in the architecture. These three features are listed as follows (in no particular order);

  • Sparse Files
  • Overlapped I/O
  • Scatter/Gather I/O
  • Memory Allocation

Sparse Files are a feature of NTFS 5.1 which allow big files that contain mostly zeros to occupy only the space needed for the non-zero portions - clearly a feature every database file should be making use of!

Overlapped I/O is a system of reading and writing files that allows for asynchronous data-transfer. Without Overlapped I/O the implementation of the buffer cache manager would be much more difficult! This is because the cache manager takes care of reading and writing data and does so using a rather elegant algorithm.

Scatter/Gather file I/O is a method of reading and writing files that allows seperate non-contiguous buffers to be read and written to using overlapped I/O - this technology is important as both the Read Ahead manager and the cache manager use this technique to speed up data-transfer to and from the underlying file. This I/O technology places a number of demands on the caller however these requirements are easily dealt with and in almost all cases exactly what is required for a DBMS file system.

Memory Allocation had to change in order to properly support scatter/gather I/O and lead to an improvement in the way buffers are allocated and managed. To support the scatter/gather logic buffers must be sized according to the system page size which for most Win32 systems is 4096 bytes and the buffers must also be aligned on a page boundary. To satisfy the first requirement is simple - the second requirement however is surprisingly tricky. The other tricky aspect is dealing with .NET as the buffers need to be pinned and passed to the scatter/gather IO wrapped in yet another structure! As it turns out the solution involves turning the memory allocation scheme on it's head!

The Memory Manager

The Windows Virtual Memory APIs have been around for ages but one of the things they give you is page-aligned memory. One of the other things they give you is the ability to reserve blocks of memory. To implement the scatter/gather support both features are used - now a managed virtual memory manager takes care of buffer allocation by using the virtual memory functions to reserve the space needed for the buffer pool. The manager also tracks allocated buffers by maintaining a linked list of allocated buffers. Memory is only allocated when the buffer instance is requested and the requested buffer is taken from the reserved address space - hence the system can reserve say 32Mb of memory for data pages (that's 8192 pages of 8192 bytes incidentally) but the actual memory consumed is determined by the actual buffers currently in use. Nice!

The Advanced File Stream

Bringing the sparse files, overlapped I/O, scatter/gather I/O and virtual memory buffers together under one so-called roof is done with a new managed class derived directly from System.Stream. Unfortunately I had to derived directly from System.Stream rather than the more obvious System.FileStream because the later does not allow the creation of unbuffered streams or write-cache disabled streams (both requirements for using scatter/gather I/O) thankfully much of the code can be lifted directly from System.FileStream (I love Reflector) with the only changes being a changed set of constructors since we can only use scatter/gather on overlapped files and several of the other options are also fixed which simplifies things somewhat.

To support scatter/gather four new methods are added to the stream code;

  • BeginReadScatter / EndReadScatter
  • BeginWriteGather / EndWriteGather

No synchronous methods are provided - although these would simply call their asynchronous counterparts in any case.

The begin methods take the usual asynchronous method parameters of a callback object and a state object in addition to an array of virtual buffer objects that indicate the memory blocks to be persisted.

Buffer Changes

To integrate these changes into the existing database framework I have needed to make some rather drastic changes to the Buffer class used by the page classes for their internal persistence. Up until now the Buffer classes have been in control of their own loading and saving however this cannot continue - the loading and saving (possibly of multiple buffers) must now be controlled by an external object - this may well wind up being a scatter/gather helper object rather than the read/write request handler directly - the idea being that buffers and page-ids can be added to this mystical helper and when contiguous runs are detected then these can be cached for a single overlapped operation.

While I am breaking the internal buffer implementation it's probably the right time to look at splitting the implementation of transacted and non-transacted buffers into seperate classes - it's confusing enough as it is!

Conclusion

These changes will take a while to implement however the effort will be well worth it - the overall performance increment both from a memory usage viewpoint and outright I/O performance viewpoint will be staggering. The use of sparse file technology will optimise the disk space usage too!