How can I tell if virtual memory is working on OS4.1? I believe that I have correctly configured the SWAP partition. Is the anything else that I need to do that I missed? I have gone into prefs/workbench and edited my Screen Title Format to add the word "Virtual" and then clicked on Type/Virtual, Show/Total, and Unit/MBytes. After doing that my screen title says "Virtual 256" which is not correct. My swap partition is about 10 GB. 256 MB is the amount of real memory on my system. Am I doing something wrong or not understanding the Amiga's concept of virtual memory? Once it is working, is there any neat demo that I might run like Hyperion's composite demo?

@Ned

I was adviced (not by an official person) to copy a file larger than the amount of RAM to RAM:. If it works SWAP is OK!

It kind of worked for me... It copied the file to RAM: ok but didn't release the RAM when deleted the file (free ram shoved just a couple of kb even when the file was removed from RAM:).

@Ned

I set up a SWAP partition as described in the AOS 4.1 Quickstart guide. It is little more than 1 GB. Then I copied 1.4GB file and my RAM disk was full when there were 1.2 GB in use of the RAM Disk, which must mean that my virtual memory works.
Basicly I don't noticed even if it was used at all.

_________________
Games, programs, reviews

@Ned

You are thinking of "virtual memory" = "disk memory" which isn't the definition used in OS4. I'm sure someone can give a good definition, or just search the forum, it has been discussed in the past. There is also no need to have a swap partition larger than 2GB as OS4.x wont be able to use more anyway as everything (real memory, PCI cards, disk memory) has to fit in the 4GB (32bit) address space.

@Menthos

Quote:

I was adviced (not by an official person) to copy a file larger than the amount of RAM to RAM:. If it works SWAP is OK! It kind of worked for me... It copied the file to RAM: ok but didn't release the RAM when deleted the file (free ram shoved just a couple of kb even when the file was removed from RAM:).

This sounds strange. Maybe there is some caching going on. I guess that an "avail FLUSH" should restore the memory, but indeed it does not (here I'm left with less than 1 Mb free). Adding an entry to this thread to give this more visibility...

Edit: it does not seem to be a bug, but just a smarter way of allocating memory - here is a simple test that I did:
1. after a reboot, I opened a shell and executed avail FLUSH, which returned 185.196.544 bytes free;
2. I copied a ~28 Mb file to RAM disk;
3. I executed avail FLUSH, which returned 158.597.120 bytes free;
4. I deleted the file from RAM;
5. I executed avail FLUSH, which returned again 158.597.120 bytes free;
6. I copied the file again;
7. I executed avail FLUSH, which returned again 158.597.120 bytes free.

In a nutshell, the memory gets re-used and is re-/de-allocated only when strictly needed.

saimo

Last edited by saimo on 17-Sep-2008 at 12:45 PM.
Last edited by saimo on 17-Sep-2008 at 12:45 PM.

_________________
RETREAM - retro dreams for Amiga, Commodore 64 and PC

@AmiDog

Damn, so no Blu-Ray downloads to Ram and unpack from there.

Hehe. VM isn't made for Amiga and never should be there in the first place. I got 2GB Ram in mine. And so far, i've only used 700MB of it. In one go.

@Speedy

Quote:

Damn, so no Blu-Ray downloads to Ram and unpack from there.

It makes no sense using the RAM disk that way. The peculiar RAM disk advantage over HD is precisely that accessing data stored in physical RAM is faster: when swapping kicks in, the HD again enters the picture, practically voiding that advantage (the whole process, moreover, just gets more complicated). In other words, you'd better work with files larger than your RAM disk can hold directly on the HD.

BTW: as general consideration, when I first read that the RAM disk could be swapped (and I did not receive my copy of AOS 4.1 yet) I thought it was odd: I gave it for granted that it would have been unswappable. Does it make any sense at all using swappable memory for it?

saimo

_________________
RETREAM - retro dreams for Amiga, Commodore 64 and PC

@saimo

In most systems, it is generally a bad idea to allow 'applications' (ram: is considered as an application) to decided if they want to use large, non-swappable areas. Applications do not have the full picture of the system, which a vm system has. If you use your ram drive for fast access of data, the data is likely to be accessed a lot. Then the vm system will notice that and make sure that it keeps things in ram until your usage frequency drops blow the usage frequency of another application. The reason why ram: is used to demonstrate the vm, is probably just because it is one of the very few apps that you can easily tell to eat a lot of memory. Most other apps are used to work within much tighter memory restrictions and will not demonstrate the increased amount of available memory as easily.

We can argue forever if you think that the user should be in control rather than the vm system, but for most users the above model provides the best average performance. If you really need to have your ram: in non-swappable ram then I guess that you can construct your own ram drive that only uses non-swappable ram. (Adding flags to the current ram: implementation for example) However, before going down that path one should realize that this is a highly special case and time is limited for the developers. Again, providing something that works good for most has highest priority.

regards,
Stefan

@Yabba

Quote:

In most systems, it is generally a bad idea to allow 'applications' (ram: is considered as an application) to decided if they want to use large, non-swappable areas. Applications do not have the full picture of the system, which a vm system has.If you use your ram drive for fast access of data, the data is likely to be accessed a lot. Then the vm system will notice that and make sure that it keeps things in ram until your usage frequency drops blow the usage frequency of another application.

I agree on everything (oh, well, except for "ram: is considered as an application", but that does not matter), but the point is that the distinctive characteristic of a RAM disk with respect to other disks is precisely that data is stored in RAM. RAM disk is used exactly when one does not want the hard disk (or any other device) to be involved for whatever reason. Once RAM disk is mapped to hard disk, it is deprived of its own nature, the reason why it was invented in first place is voided. And the user's needs are not met.
Please don't consider this just abstract phylosophy.

Quote:

The reason why ram: is used to demonstrate the vm, is probably just because it is one of the very few apps that you can easily tell to eat a lot of memory. Most other apps are used to work within much tighter memory restrictions and will not demonstrate the increased amount of available memory as easily.

I know, in fact it was just me suggesting Menthos to copy stuff to RAM

Quote:

We can argue forever if you think that the user should be in control rather than the vm system, but for most users the above model provides the best average performance. If you really need to have your ram: in non-swappable ram then I guess that you can construct your own ram drive that only uses non-swappable ram. (Adding flags to the current ram: implementation for example) However, before going down that path one should realize that this is a highly special case and time is limited for the developers.

It is not a personal need. I'm speaking in general. Any user that chooses to use the RAM disk does it expecting data to be in RAM - otherwise, he/she would not even choose the RAM disk.

Quote:

Again, providing something that works good for most has highest priority.

I can't see how providing a non-swappable RAM disk does not work fine for most. RAM disk is a resource that, like everything else, must be used correctly: using it to hold more than it can contain is indeed a misuse that goes against its principles and that, moreover, hardly gives any advantage.

saimo

_________________
RETREAM - retro dreams for Amiga, Commodore 64 and PC

@saimo

Quote:

It is not a personal need. I'm speaking in general. Any user that chooses to use the RAM disk does it expecting data to be in RAM - otherwise, he/she would not even choose the RAM disk.

That's not true. A user which uses the RAM disk mainly expects it to be fast and to be temporary (i.e. that it is lost when power is switched off). A user usually does not care where the data is stored.

And of course the RAM disk is stored in RAM. As long as there is enough RAM. But that's system internals. *Users* don't care about virtual or physical memory. If the RAM disk can be extended beyond the size of the physical memory, this is a benefit not a disadvantage.

Being swappable does not mean that memory *is* swapped. Memory pages will only be swapped if there is not enough physical memory available.

Bye,
Thomas

_________________
Email: thomas-rapp@web.de
Home: thomas-rapp.homepage.t-online.de

Thanks to all for the info. I guess I was looking at virtual memory from a different point of view. I'll give the idea of copying one or more big files to ram: to see what happens.

@Ned

You might also want to recover that 8GB of lost space on your HD too.

The maximum size of SWAP that can be addressed is 2GB, anything larger is totally unused.

Simon

_________________
Simon

Comments made by me on any public fora are not representative of, or on behalf of, any company I may have, or assumed by the reader to have, any association with.

Any comments are a personal opinion, and should be accepted as such.

@thomas

Quote:

It is not a personal need. I'm speaking in general. Any user that chooses to use the RAM disk does it expecting data to be in RAM - otherwise, he/she would not even choose the RAM disk.Quote: That's not true. A user which uses the RAM disk mainly expects it to be fast and to be temporary (i.e. that it is lost when power is switched off). A user usually does not care where the data is stored.

Speed and temporariness are only two advantages of the RAM disk. A user may make his choice based on different reasons. What if the user chose the RAM disk precisely to have the data in RAM, *not* on the HD? The RAM disk could be chosen precisely not to pester the HD. Here are the first examples that come to my mind:
- one could set the caches of web browsers to reside in RAM disk so that not only fetching data is quicker, but also HD stress is avoided;
- one could want to work on files while the HD is doing some stressful task (f.ex. deep data recovery);
- one could want to avoid the HD noise (f.ex. while sampling sounds).

In a nutshell: a RAM disk that uses only physical memory meets the requirements of anybody; a RAM disk that can be swapped may not meet the requirements of who, indeed, does not want the HD to be involved.

Your "usually" seems to agree on this.

Quote:

And of course the RAM disk is stored in RAM. As long as there is enough RAM. But that's system internals.

I beg to differ: the RAM disk is labelled "RAM" for a reason. The RAM disk is characterized by the fact that data resides in RAM and not on any other medium. That's the main feature of the RAM disk (speed and temporariness are only two consequences). Such a feature is completely exposed to the users exactly by means of such name. It is not about internals.

Quote:

*Users* don't care about virtual or physical memory. If the RAM disk can be extended beyond the size of the physical memory, this is a benefit not a disadvantage.

If somebody needs to process so much data from files that the RAM cannot contain it all, he/she had better access that data directly from HD: it is useless to have the data moved from one place to another of the HD, not to mention that such a mechanism could impair the policy of programs written precisely to handle large files (think of a video editing software, which already has its own policy of data handling in RAM: filling up the RAM with files surely does not help them; or just think of a file converter that uses its own internal buffers).

Quote:

Being swappable does not mean that memory *is* swapped. Memory pages will only be swapped if there is not enough physical memory available.

That's another story.

saimo

Edit: fixed a grammar mistake and added a missing word.
Edit: added "@thomas".

Last edited by saimo on 18-Sep-2008 at 10:15 AM.
Last edited by saimo on 18-Sep-2008 at 09:40 AM.

_________________
RETREAM - retro dreams for Amiga, Commodore 64 and PC

@saimo

Just so that you know, "avail flush" has no effect on OS4.1 - the FLUSH parameter has been depreciated, and does nothing at all. Its behaviour has been replaced with various automated mechanisms that flushing resources when necessary, in ramlib and such like.

_________________
IBrowse, AmiSSL and Warp Datatype Developer

@saimo

Quote:

In a nutshell, the memory gets re-used and is re-/de-allocated only when strictly needed.

Yes, this is indeed the case.

Just imagine you have a lot of unused RAM disk space... since ram-handler doesn't return the memory immediately, you have potentially an empty ram disk that has more than a GB of memory in use.

Now you do an avail flush...

The memory will ALL be reclaimed, resulting in severe disk activity until all is reclaimed.. and this is even the case if you just need 10 bytes...

So the memory system only reclaims memory when there is need for it. This leads to "incorrectly" displayed free memory in the Workbench titlebar, or with avail, but these figures are changing by the minute, anyway, and I think you'd rather have a fast response than exact memory figures.

And before anyone objects: Memory that is freed is not first paged in again. It's simply declared "invalid" in the page cache. However, memory pools in their current implementation have control structures inside their free memory, so pools result in paging when memory is freed (this will be fixed later, there wasn't enough time).

Avail flush is now a no-op. It was normally used for expunging libraries from memory, but that isn't necessary anymore since libraries are automatically expunged when the file is overwritten (i.e. ramlib watches opened files, and expunges the library when the file is written).

Quote:

thought it was odd: I gave it for granted that it would have been unswappable. Does it make any sense at all using swappable memory for it

Actually, it does make sense. RAM disk is potentially the biggest consumer of memory in the whole system (even a complex application like Blender currently uses some 100 MB of memory when I run it with medium complexity scenes... copy a large LHA to RAM disk, and it's bigger).

Therefore, you would potentially have the largest consumer of memory NOT being swappable. We decided that this wasn't a good situation. Just imagine that you can't start a program because there's still some files in RAM disk that you copied there an hour ago and didn't touch since then.

In addition, what is "RAM" ? If you run Quake and it runs out of memory on 4.1, it will start paging out memory to disk. So the *RAM* Quake uses is OK to swap, but the *RAM* ram disk uses isn't ?

My tests indicated that copying a large file to ram: and then copying it from ram: to ram: under a different name was faster than the first copy, even though the second copy was causing massive pager activity. Sure, the pager isn't as fast as it could be yet, but we'll be updating it in the future.

_________________
Thomas, the kernel guy

"I don't have a frigging clue. I'm norwegian" -- Ole-Egil

All opinions expressed are my own and do not necessarily represent those of Hyperion Entertainment

@EntilZha

Quote:

(i.e. ramlib watches opened files, and expunges the library when the file is written).

Does ramlib also notice when the library search path gets changed?

Most of the time, someone trying out a new library version is unlikely to just write over the old one...

@umisef

Quote:

Does ramlib also notice when the library search path gets changed?

To be honest, I don't know (wasn't done by me)

But there's also a tool called "Expunge" that will specifically expunge a given library.

_________________
Thomas, the kernel guy

"I don't have a frigging clue. I'm norwegian" -- Ole-Egil

All opinions expressed are my own and do not necessarily represent those of Hyperion Entertainment

@Futaura

Quote:

Just so that you know, "avail flush" has no effect on OS4.1 - the FLUSH parameter has been depreciated, and does nothing at all. Its behaviour has been replaced with various automated mechanisms that flushing resources when necessary, in ramlib and such like.

Thank you for the information
I was sure it did not work as with the old memory system, but it was all I could resort to

BTW: about time to update avail's documentation, then (it still says "The FLUSH option causes all unused fonts, catalogs, libraries and device modules to be expunged from memory).

saimo

_________________
RETREAM - retro dreams for Amiga, Commodore 64 and PC

@EntilZha

Before I go on, I'd like to make it clear (to you and anybody reading here) that in no way I'm saying VM is bad or its implementation is bad (VM done right is a good thing and AOS 4.1's implementation seems just fine). All I want to do is to point out what, in my opinion, is a mistake and to suggest a couple of things to solve it.

Quote:

Just imagine you have a lot of unused RAM disk space... since ram-handler doesn't return the memory immediately, you have potentially an empty ram disk that has more than a GB of memory in use. Now you do an avail flush... The memory will ALL be reclaimed, resulting in severe disk activity until all is reclaimed.. and this is even the case if you just need 10 bytes...

Sorry, I don't get it... disk activity for what? The premise is that memory is unused, so, unless swap space allocation is kept track of directly on the SWAP partition itself (which would seem a bit crazy to me and should involve little information anyway), all that would be needed is updating system structures in memory - at least, without knowing the system internals, that's all I can imagine.

Quote:

So the memory system only reclaims memory when there is need for it.

This is a policy I agree on.

Quote:

This leads to "incorrectly" displayed free memory in the Workbench titlebar, or with avail, but these figures are changing by the minute, anyway, and I think you'd rather have a fast response than exact memory figures.

I personally don't care about those figures - just think that I deactivated the WB bar since AOS 3.9 (or 3.5... cannot remember when this possibility has been introduced). But if you are telling me that currently there is no way to tell how much memory is actually in use, well, then that's a problem (not a big one, though) at least as regards spotting memory leaks and benchmarking system performance.

Quote:

And before anyone objects: Memory that is freed is not first paged in again. It's simply declared "invalid" in the page cache.

I'm no VM expert so, if you can, could you please elaborate on this? Do you mean to say that when a page of memory mapped to SWAP is freed, at some point it is supposed to be copied back to physical memory? If so, what for?

Quote:

Avail flush is now a no-op. It was normally used for expunging libraries from memory, but that isn't necessary anymore since libraries are automatically expunged when the file is overwritten (i.e. ramlib watches opened files, and expunges the library when the file is written).

This is defintely a nice thing
But what happens if the replaced library is still in use? I guess that expunging must be avoided or at least delayed.

Quote:

Quote: thought it was odd: I gave it for granted that it would have been unswappable. Does it make any sense at all using swappable memory for it Actually, it does make sense. RAM disk is potentially the biggest consumer of memory in the whole system (even a complex application like Blender currently uses some 100 MB of memory when I run it with medium complexity scenes... copy a large LHA to RAM disk, and it's bigger). Therefore, you would potentially have the largest consumer of memory NOT being swappable. We decided that this wasn't a good situation. Just imagine that you can't start a program because there's still some files in RAM disk that you copied there an hour ago and didn't touch since then. In addition, what is "RAM" ? If you run Quake and it runs out of memory on 4.1, it will start paging out memory to disk. So the *RAM* Quake uses is OK to swap, but the *RAM* ram disk uses isn't ?

Your reasoning makes sense, no denying about it. But it overlooks a basic premise which changes the perspective quite a lot. The point is that the RAM disk gives the user a (oh, well, the only) direct way of accessing physical memory.
When the user starts Blender, Quake, you name it, he is not requesting memory usage, but an application to run: memory gets used indirectly by the applications, so it's only normal that the system tries to cope with the applications requirements at its best, without directly involving the user in the process.
When instead the user chooses to use the RAM disk, he is intentionally and directly requesting a piece of that precious resource which is the physical memory. When he does that he has a reason to. With swapping, the OS may get in the way of the user (see my answer to thomas), which is certainly not desirable nor an Amiga way of thinking.
At the same time, when the user directly accesses the memory resource he (must be aware that he) is also taking the responsibility of using it adequately. He must know how to properly make use of it. That happens with HDs, CDs, USB keys and whatnot, so why has it to be any different with the RAM disk? Users must be taught how to use that resource and they surely can learn it - it is not difficult at all. So, as regards the risk of memory wastage due to leftovers or huge files in the RAM disk: that's the user's direct fault (notable exception: applications stuffing the RAM disk with their own internal junk, but luckily I can't think of a single application with such a behaviour). It is up to him to take care of his own dirt. The system should not try to do the job for him because it cannot tell whether by doing so it gets in the way of the user. At most, when it notices that the RAM disk is too big and the system has run out of physical memory, it could give out a warning.

Please note how before the VM introduction, users did know how to make use of the RAM disk: they certainly will not forget just because now there is VM. As for new users, well, they can learn just like the old ones did.
RAM disk is a wonderful thing AmigaOS sports with pride, let's not spoil it

Quote:

My tests indicated that copying a large file to ram: and then copying it from ram: to ram: under a different name was faster than the first copy, even though the second copy was causing massive pager activity.

Speed is not the only reason why a user chooses the RAM disk (see again my previous posts - I do not want to clutter the thread with the same things over and overs again).

Quote:

Sure, the pager isn't as fast as it could be yet, but we'll be updating it in the future.

Optimizations are always welcome


That said, to be a bit more constructive, here are a couple of suggestions:
1. add an option to let the user decide whether RAM disk is swappable;
2. keep RAM unswappable and add a brand-new device that implements the swappable RAM disk - call it "FAST disk", "TMP disk", "UFO disk" = "Usually Fast Operations disk" (horrible pun intended), ...

Finally, a question about RAM disk: does it use its own filesystem or does it use FFS? - I'm asking because I tried to comment out the FastFileSystem module in the kicklayout (because I do not use FFS), but then the OS won't start

saimo

_________________
RETREAM - retro dreams for Amiga, Commodore 64 and PC

@saimo

How about the T: assign which is usually pointing to RAM? Should people be forced to have it on a disk (I wouldn't want that) or should an application fail simply because the RAM-disk gets full (I wouldn't want that either).

You've got some points, but I still think the current solution is the best one.