Aaru Progress Report: November 2021

Welcome to another progress report!

.NET 6

We talked about .NET 6 last month, and now it has been finally released. Migration from the release candidates has been painful, and on re-testing all the compression and checksum algorithms, we found no difference from the final release to the release candidate we used for the graphs last month.

Integration of new native libraries

Small but useful, every part of the media image formats that used compression is now using one of the native algorithms we implemented last month.

This also allowed us to add support for more media image formats, specifically LZFSE compressed (ULFO) and LZMA compressed (ULMO) UDIF images.

Linear memory

Linear memory is basically memory that is accessed linearly, not by blocks.

This explanation will probably not clarify you in the least, so let me become a little more specific.

When we access the most common media, like a floppy, or a CD, we access it in blocks, that is, an aggrupation of bytes, like 512, or 2048. And we cannot access the data in between. It's not that we get 512 bytes, it's that we get 512 from 0, or 512 from 512, we cannot get 512 from 513 or 12 from 0.

Linear memory can be accessed at any position: 0, 1, 15, 613, doesn't matter. Or at least, if it matters, is only on performance, it's not really prevented.

For this reason, and to avoid confusion with linear tape (that is a block based media) we decided to call it byte addressable media.

And now you'll probably think you've never heard of it, but you've done. RAM is byte addressable media, so is a game cartridge, the chip that stores the firmware of your computer (or at least, it was), etc.

There is a lot!

But byte addressable media presents several problems.

The most important is, what is a byte? Nowadays that definition seems very simple, a byte is 8 bits. That was not always the case, there were 6-bit, 7-bit, 9-bit and even 11-bit bytes!!!

The second is that while the media itself is byte addressable, that doesn't necessarily mean it reads or writes one byte at a time. They are read and written in words, and that can be any number of bytes, but normally, a handful, far from the size of a block.

And the third is that, whatever is accessing the media, may need to work in words that are not the same size as the media itself. If you're old enough you may remember the 30-pin SIMM memory modules. And that you needed multiples of 4 of them for a 386DX, but only multiples of 2 of them for a 386SX.

That's because 30-pin SIMMs are read/write in 8-bit words (basically bytes) while the 386SX read/write in 16-bit words and the 386DX read/write in 32-bit words.

Unluckily the need for this also mean combinations of "this chip gives byte 0, this gives byte 2 and this gives byte 1". This is called interleaving.

And to make things worse, some equipment had a limitation on the maximum size of a media they can read, so they needed suplemental circuitry to help making some part of the media visible. These are called mappers, and were extensively used in many videogame consoles (like the NES/Famicom, or the Game Boy).

All of these shenanigans meant that adding support for byte addressable media in Aaru required some imagination.

So to fix this, we have designed a way to describe how the different parts of the media are assembled (the physical addressing), and how the equipment sees it (the virtual addressing), as well as describing the mapped and interleaving.

We added support for some formats that are self-describing and allow us to guess this information from the file, mostly some videogame console dumps, like Game Boy, Genesis, et al.

For the future we'll allow you to write a description file, in JSON format, so Aaru can recognize any file or combination of files you want.

But why adding this feature? The goal is clear. We want to support dumping hardware for videogame consoles, specially we want to support hardware that is still working, but so old that the original software that came with it is completely unusable in modern computers. We also want to support modern dumping hardware.

When we finish this feature, we'll add the format of the description file for our documentation.

Conclusion

The complications of linear memory have been intense, but everything is on the right path!

We want to thank all of our users for the bug reports they've sent us this month.

And special thanks to the N-Gage preservation group, that has been extensively preserving retail games, as well as prototypes, for the confidence and trust in our tool, our archiving methods, and our team.

We hope to keep working with the, and with the team behind the N-Gage emulator in the future!

Have a nice holiday ahead!