Quote:

Are there any plans to support memory mapped files now under OS4 ?

Plans, yes, but that's future plans.. as others already said, the problem is that a large file mapped into memory will take up a lot of virtual address space...

_________________
Thomas, the kernel guy

"I don't have a frigging clue. I'm norwegian" -- Ole-Egil

All opinions expressed are my own and do not necessarily represent those of Hyperion Entertainment

Quote:

Very interesting read. I hope they are not giving up any trade secretes, is this type of memory management unique to OS4 on the Amiga?

No, most of it is used elsewhere, too... For the core functionality (object caches), the older article holds references to the original papers.

_________________
Thomas, the kernel guy

"I don't have a frigging clue. I'm norwegian" -- Ole-Egil

All opinions expressed are my own and do not necessarily represent those of Hyperion Entertainment

@EntilZha

I have to thank you very much. And to the rest of the team working on AOS.
I am supose to do my final paperwork in college in subject "Architecture of Amiga OS", and if I go trough with it all this stuff could come very handy to me as reference.



edit: sorry for any typo, just came from coctail night

Last edited by Teddy on 15-Jul-2006 at 03:08 AM.

_________________
You can crack anything with your head, even the head itself.
--------------------------------
...proud AOS user since 1993
--------------------------------

Quote:

Other OSs have multiple address spaces, that's how they can deal with it.

Every system has it's limitation, even on multiple adress spaces. A 32bit system would most likely be limited by a 4GB per process space (or similar, bigger with 48bit addresses or 64bit). Still, would just be enough for one DVD. This could be limited to 512MB (or 720 or so) for now, for example.

Quote:

Loading a 1GB file into memory, on the other hand, would cost swap space and time.

It wouldn't. It would just eat up another 1GB adress space - and given the total space is less the max available space - these 1GB - you could simply "attach" it as an additional swap space. However, as mentioned, I'd limit the size to 512MB for now. It should be enough for most cases. I don't expect DVD like games any time soon on the Amiga (or don't limit it for games at all - you hardly run more then one game at one time).

Quote:

A mmap'ed file is like a swap file created on demand.

Right. That means the contents is loaded from disk only, when the page gets mapped into the physical mem and not else. Which also means, only data needed will be loaded but not everything - in opposition of pre-loading a bunch of files and then pick what you need (well, nobody would do that, though).

Quote:

More useful would be a C++ class which let you access a file via an overloaded [] operator, mimicing a mmapped file, without the problems of a mmapped files.

One could do that now. Why not? Would require some logic, and would be maybe easier, when the OS supports such a function. Also, little bit hard to access this C++ []operator from plain C or phyton

_________________
AmigaOne XE G3/933/VIA/FM801/R200 (fixed),
G3SE/600/Voodoo3/Sil680/RTL8139/SBLive! (noiseless!)
µA1-MK2/G3/933/R200/CMI8738
XE/G4 (broken 7450/800)

Nice article Thomas, keep up the good work.

_________________
See my blog and collection website! . https://www.blog.amigaguru.com

Quote:

Every system has it's limitation, even on multiple adress spaces. A 32bit system would most likely be limited by a 4GB per process space (or similar, bigger with 48bit addresses or 64bit). Still, would just be enough for one DVD. This could be limited to 512MB (or 720 or so) for now, for example.

A 512 MB limit would mean that 4 tasks mmap'ing a 512MB file would fill up the whole (usable) address space. On other systems, with multiple address spaces, you can have n processes mmap'ing huge files all at the same time.

You see, the problem isn't the size of the file per se, but rather the number of tasks that could be doing the same at any given time. In a shared environment any resources you take up is rendered unavailable to everyone else: this just doesn't happen on systems with multiple address spaces.

Quote:

It wouldn't. It would just eat up another 1GB adress space -

Read what I wrote again: "Loading a 1GB file into memory". I wasn't talking about mmap'ing there, but loading a file into memory.

Quote:

ne could do that now. Why not? Would require some logic, and would be maybe easier, when the OS supports such a function. Also, little bit hard to access this C++ []operator from plain C or phyton

Plain C is so passé... As for python, you could implement the same functionality there too: currently the mmap() function in python is implemented on top of the mmap() function (or equivalent) of the underlying OS, you could just reimplement it so that it offers the same interface, but doesn't use any underlying mmap().

Last edited by falemagn on 15-Jul-2006 at 09:14 AM.
Last edited by falemagn on 15-Jul-2006 at 09:14 AM.

_________________
“It is well enough that people of the nation do not understand our banking and monetary
system, for if they did, I believe there would be a revolution before tomorrow morning.”
~~ Henry Ford

@falemagn

OS4 has a MEMF_PRIVATE tag, which is defined as "memory which can only be seen by the allocating task". Presumably, the memory space could be divided into two areas; one for shared memory and one for private. If the private one is per-task, it can have its own address space, and memory mapped files become useful.

Edit: OS4 also introduces the concept of threads, which would share the memory map of the owner task, presumably, making private memory more useful.

Edit part 2: And the shared section wouldn't even need to be that big, since AmigaOS apps that don't know about or use private memory tend to be written for systems with 32, 64, 128 or 256Mb of RAM. I only have 256Mb in my AmigaOne, and that rarely gets filled, so the shared section could easily be that, or even 128Mb maybe.

Last edited by xeron on 15-Jul-2006 at 12:17 PM.
Last edited by xeron on 15-Jul-2006 at 10:44 AM.

_________________
Playstation Network ID: xeron6

Nice to see the OS improved and modernized. And nice to read this kind of articles that let the users informed. I hope there will be other surprises like this one !

@Xeron
You write that the new memory system introduces the concept of threads. Will this help unix compability or is this related to something completely different (Thinking of fork() and such) ?

@Desler

Well, it won't help too much with fork(). The problem with fork is that it creates a whole new instance of your running program, with all variables at the same location in memory, except that they are a copy of the variables in another address space (AFAIK, i'm actually not very knowledgable about the inner workings of UNIX).

The "thread" support I talk of is just the ability for Amiga programs to launch subtasks and tell AmigaOS that the subtasks actually "belong" to the program. This allows things like not letting the program quit until all child tasks are closed, and in the example i posted above, it would allow threads of a program to share the same private memory.

_________________
Playstation Network ID: xeron6

@Rogue

Thanks for the update, I think this will also answer those that have been wondering what you have been doing since update 4

Rewriting the memory allocation must have been a big task.

hopefully we will see another update released.

aldur

_________________
Aldur
------------------------------

Quote:

A 512 MB limit would mean that 4 tasks mmap'ing a 512MB file would fill up the whole (usable) address space. On other systems, with multiple address spaces, you can have n processes mmap'ing huge files all at the same time.

I understand that. And I don't say a current implementation would not have its limits. It sure would. But please stay reasonable. Games would heavenly benefit from mmaped files, and please tell me, how many games do you want to run at the same time ? (and the mmaped area will most likely be data, textures etc.)

Like Xeron explained, one possibility would be to use MEMF_PRIVATE mem on a complete separate page (2GB?), but that would partition the mem again.

I think, a current solution with limits and maybe some real per process addresses in a (way) later stage would be nice. But anyway, we have to wait if and when Hyperion might come up with something. I put my faith in them to give us something usefull.

And about C. Well, it's not. Even if you do C++ (especially in middle ware), useing a plain C interface gives you a huge benefit over e.g. eporting C++ interfaces (how to make sure an public shared interface knows about the VTab of a class used in a shared library ?). I agree, however, I can't live without writing code mainly in C++ any more.

_________________
AmigaOne XE G3/933/VIA/FM801/R200 (fixed),
G3SE/600/Voodoo3/Sil680/RTL8139/SBLive! (noiseless!)
µA1-MK2/G3/933/R200/CMI8738
XE/G4 (broken 7450/800)

@Jorge
I have to agree with Falemagn - what's the use of memory mapping when it can only be done by perhaps a couple of tasks (best case given memory fragmentation)? A multi-tasking OS (and all it's features) is suppose to be able to handle an arbitrary number of tasks, as long as the physical resources allow it. A chronically limited mmap sticks out like a sore thumb.

Your "Games benefit most from mmaping" is a straw-man argument (you only brought it up to knock it down - no-one else mentioned it that I saw). Plus I suspect that your argument is flawed anyway: What if a game wants to mmap several 1GB data files?

How about this? You have a DVD program which mmaps a 2GB iso image - it may have stolen 2GB of virtual memory, but actual memory usage is probably tiny, so the user has no idea the program is doing anything demanding. With the program still running (*), he then decides to run a new game, or some other demanding software that also happens to want to mmap a couple of 1GB files. At this point the new game/program will go POOF - there isn't sufficient virtual memory available. Which will likely surprise the user, because he thought OS4 was a multi-tasking OS, and it has tons of free memory available...


(* I'll leave it to your imagination as to whether the DVD program is actually doing something in the background, or whether the user is just leaving it running out of lazyness.)

_________________
Author of the PortablE programming language.
It is pitch black. You are likely to be eaten by a grue...

Where interesting read, nice to see some progress on the OS, AmigaOS most be one of the best OS outer when it comes to its small footprint, and how it utilizes the resources efficient, it’s a great os that that deservers more credit then it is given.

_________________
http://lifeofliveforit.blogspot.no/
Facebook::LiveForIt Software for AmigaOS

I most agree whit Falemagn that Memory Mapped IO, is mostly eats up virtual address space, and the benefits are not that big, the only thing it’s doing is automatically load in segments of data, so you don’t need to do It your self, the best argument for using it is to avoid coding file IO operations your self; that’s pure “laziness” in my option.

_________________
http://lifeofliveforit.blogspot.no/
Facebook::LiveForIt Software for AmigaOS

I think that memory-mapped files only make sense on a 64-bit or greater system. 64-bit addressing gives you an address-space large enough to do these things. 4GB is just not that big any more when it comes to data-files. We need 64-bit addressing first, before we start talking about implementing mmaped files.

Hans

_________________
http://hdrlab.org.nz/ - Amiga OS 4 projects, programming articles and more. Home of the RadeonHD driver for Amiga OS 4.x project.
https://keasigmadelta.com/ - More of my work.

Looks liek it's already been explained, never mind. :)

Last edited by billt on 17-Jul-2006 at 06:51 PM.
Last edited by billt on 17-Jul-2006 at 06:47 PM.

_________________
All glory to the Hypnotoad!

@ChrisH

"How about this? You have a DVD program which mmaps a 2GB iso image - it may have stolen 2GB of virtual memory"

How about this. You have a DVD program which knows about MEMF_PRIVATE, and allocates 2Gb of private virtual memory. This has stolen 2Gb of its own memory map (from the private memory range) and leaves other programs unaffected.

I admit that all this is much more useful in a 64bit operating system, BUT, even on AmigaOS this could be handy.

Lets say that you leave just 256Mb of memory available as shared memory, and 4Gb, minus that 256Mb and any reserved memory areas, for private per-task memory.

Right now, my A1 has only 256Mb of RAM, and no form of virtual memory, and it works great, so dividing the memory space this way (which would only affect legacy apps), wouldn't have a huge impact (or in my case, any impact). Presumably the division in memory space would be user settable but have a sensible default.

_________________
Playstation Network ID: xeron6

@xeron

Let's extend your example a bit further. Imagine now that you now have an 8GB iso (or a large lha archive of your data partition). Your DVD program attempts to mmap it to memory and fails because the 32-bit address space is too small. The DVD program now either just fails, in which case you can't write your file to disk, or, it has to use a backup method via normal I/O. In the latter case, you have to write disk I/O stuff anyway, so why bother having the mmapped method in the first place?

Your suggestion of MEMF_PRIVATE and private virtual address-spaces is good and will solve most of the problems others have suggested, but sadly 32-bit addressing is still in the way.

Hans

_________________
http://hdrlab.org.nz/ - Amiga OS 4 projects, programming articles and more. Home of the RadeonHD driver for Amiga OS 4.x project.
https://keasigmadelta.com/ - More of my work.