OK, a potentially interesting thread was locked, presumably because of the Red vs Blue bitching that polluted it. The moderators kindly said it wasn't my fault, and I could start another thread on the same topic. So let's try to keep it clean this time, and try not to mention M*rphOS...

Rogue said Quote:

Quote: jahc wrote: If this plan goes ahead, does that mean current OS4 apps will run under a sandbox with the older kernel, and that the newer OS4.x apps will have enhanced features, and maybe a better form of memory protection? I wouldn't call it a sandbox, rather a virtual environment. The old system will run in its own address space as a separate task. It's not like a UAE or anything like that. It might or might not be able to become aware of the outside world, we still need to clear up the details.

He then tried to clarify it Quote:

FWIW, I am not saying that AmigaOS 4.0 is a dead end design. However, everybody with half a brain cell will agree that to evolve the current design, you need to do something about the way certain things are handled right now. Apple did the same thing. They invented the Classic environment for binary compatibility, Carbon for source code compatibility, and Cocoa for the new stuff. This is very similar to what we want to do: - A virtual environment to run classic OS 4 binaries in. - A compatibility library that will allow you to compile OS 4 source code to the new system, trying to stay as compatible as possible and to go with as little changes as possible, with possibilities to narrow down incompatibilities (like with Carbon where you could "close" system structures and use new accessor functions). - A new API for the "native" stuff. The virtual environment may still be self-hosted for resource-limited devices, and will be continued to be developed. If you really think that AmigaOS can be developed into a more modern system without doing this, you must clearly have a different idea about "modern" than I have. But memory protection is a must, and it simply is NOT POSSIBLE with the open structure approach that we have right now.

My answer to Rogue (even if he won't read it) is:

It was always clear that AmigaOS's current design makes many modern features impossible, and I have said so on many occasions, as have others. But it always appeared to me that OS4 was trying to get as close to those modern features, without breaking the compatibility with old programs. Off the top of my head, here's a list of features that kept compatibility without using a sand-box (or "virtual environment" if you prefer) :

* 68k emulation on a per task basis, so that 68k code can be used in drivers/filing-systems/etc & still maintain the responsiveness we expect from AmigaOS.
* Libraries now accessed using a new Interfaces system, so that old libraries can still be accessed in their old way, but they can also be changed (for modern needs) without breaking existing programs.
* Memory allocations are only protected if they are flagged as such.
* A super fast Slabs-based memory allocator, which was then used to emulate the old memory allocation systems (such as pools).
* A 32-bit virtual address space, so that memory fragmentation can be kept low (but this gets harder the closer you get to 4GB of real memory). The full benefit would only be obtained from a 64-bit address space, but this would break all existing programs (both 68k & PPC).
* Lots 'hacks' to make the new memory system & other stuff be more compatible with old 68k programs.
* No doubt lots of other stuff I forgot, or was not aware of.

And it strikes me as strange that after having gone to all this effort & complexity, that Hyperion now effectively says "Forget all that hard work we did trying to maintain compatibility without a sand-box to slow it down, because OS4 as you know it will now run in a 'virtual environment' anyway."

If you want an OS4 program to NOT run in the sand-box (and thus avoid the potential speed hit that Rogue says won't happen) then you must modify the source code (perhaps drastically if it is a complex program) & recompile it.

If sand-boxing has been planned all along for OS4, then it has been a very well kept secret, as far as I am concerned.

_________________

A related topic is: If OS4.5 (for want of a better name) will not run any OS3 or OS4.0 programs natively, because it will use a drastically different set of APIs, albiet in an "Amiga OS style", then why bother with OS4.5 at all? Why not go for one of the already existing "Amiga like" OSes, such as Haiku, Syllable, and maybe even Amiga Inc's OS5?

That's playing devils advocate, but it's also a damn good question, because we know so little about OS4.5 . Perhaps Rogue can answer that?

P.S. There isn't really a valid answer to this question anyway, since it depends on what "Amiga like" means to YOU personally. As the whole OS3 vs OS4 vs MOS vs AROS vs TekLib vs UAE thing shows, everyone has a different interpretation.

_________________

"OK, a potentially interesting thread was locked, presumably because of the Red vs Blue bitching that polluted it."

that you have punctually reopened ... where the point of locking threads then?

Let starts!

_________________

@Framiga

Quote:

that you have punctually reopened ... where the point of locking threads then?

Don't ask, pretty please?

But as ChrisH said it wasn't his fault, and the reasons are better to forget as soon as possible.

@all
Be nice and discuss on topic, will you?

Now, what comes to future design of OS4, well,
memory protection seems to be more or less a must for an OS of today, and AmigaOS 4 is no exception.
Backwards compability, the more the merrier, but something need to go if to make memory protection as good as possible, if this virtual environment fixes that then fine.

Last edited by tomazkid on 15-Aug-2007 at 07:16 PM.
Last edited by tomazkid on 15-Aug-2007 at 07:16 PM.

_________________
Site admins are people too..pooff!

@tomazkid

ok but now i PRETEND a popcorn smilie!

_________________

@ChrisH

Why would "OS4.5" have to have a drastically different API? The same underlying API could be used, minus all the direct structure access stuff. Removing direct access to the structure was talked about way back in the early nineties when the transition to PowerPC was first being discussed. I linked to a document on programming recommendations from back then here.

Contrary to what people think, message passing between programs doesn't need to involve copying data between their memory spaces. Message data could be placed in shared memory or the MMU could be used to tamporarily give another process the message's memory. This would mean that messages couldn't contain pointers to private data. In doing so, the same message passing interface would be preserved, but extra rules would exist regarding the content of the messages and memory allocation.

It would be nice to hear how big the API changes are going to be. IMHO, they could keep the Amiga OS4 API intact and just make the following changes (plus others of course, this is not an exhaustive list):
- Remove direct access to OS structures (the virtual environment would emulate these for old apps)
- Add accessor functions for the data that used to be obtained directly from the OS structures
- Remove Forbid()/Permit() (the virtual environment would provide these for old apps)
- Extend the memory allocation system to use full memory protection (apps in the virtual environment will locally have the current system)
- Change message passing to use the new memory system and share message memory where appropriate (old apps within their virtual environment would not be using the new memory system and so would still be able to pass messages the old way)
- Add multiprocessing capabilities (old apps won't be taking advantage of this) now that forbit()/permit() is gone

It would be necessary to provide a virtual environment for old apps as these changes would break compatibility. However, it wouldn't be a drastically different API.

Hans

_________________
Join the Kea Campus - upgrade your skills; support my work; enjoy the Amiga corner.
https://keasigmadelta.com/ - see more of my work

Here are my thoughts on what I believe Rogue meant (disclaimer: I program for a living, which means I sortof know what I'm talking about, but I'm not an OS4 dev or beta tester, which means I don't - here's the salt )

In the discussion below, "OS3.x" means "AmigaOS before OS4.0", "OS4.0" means "what we have now" and "OS4.x" means "what Rogue's talking about".

I believe that Rogue sees OS3.x programs running on OS4.x not as constrained as Chris imagines (hence Rogue's remark that it's not a sandbox or UAE), but more like a 16 bit DOS program running on WinXP or a 32 bit program running on x64's WOW64 environment. What does this mean? simple: the program runs just like other programs, it is a task that gets scheduled like other tasks and has some interaction with other tasks, but it is limited in that it doesn't have the freedom that it would in its native environment. The way I imagine this will happen is like this: a separate 4GB address space (not 4GB of actual memory, but a "walled off" set of addresses) will be dedicated to 3.x programs. These programs will behave as they do on OS4.0 - they are scheduled individually and share the same address space; however they will not be able to directly affect programs written using the 4.x APIs. Those new programs will each have their own address space and use different ways of interacting with the system (different message passing mechanism, system structures will be hidden, etc - there's your break in compatibility). The walled off area of the system will not use the same system libraries than the rest of the system, but a set of proxies to the "real" devices and libraries so that if they crash, only the walled off area will need to be "rebooted" - all your 3.x programs will die, but your 4.x programs, which includes OS4.x itself, will not. The proxy devices and libraries will make the 3.x programs feel like they're part of the rest of the system, so the experience should be fairly seamless. If implemented correctly, the overhead should be pretty small - the walling off is done by the MMU and the device and library proxies will probably consist of a transition to supervisor mode and a memory copy - on modern CPUs mode transitions are designed to be cheap. There may be a small "speed bump" when a 3.x program tries to communicate with a 4.x program because a memory copy will be involved, but with even a 500MHz CPU, copying < 64KB of memory (the maximum size of an AmigaOS message) is hardly going to be noticeable. Clever use of DMA may even do away with this.

This is the model used by the DOS box in Windows, and it works amazingly well there - I can run Duke 3D, an intensive hardware banging game, full screen or in a window, and if it hangs all I do is alt-enter to put the game in a window, and click the 'x' button on the window frame. The DOS box emulates display memory and audio in a similar way to the way OS4 emulates HAM in RTG screens and NallePuh emulates Paula, but that's it - it's nowhere near as thorough as say UAE or Bochs. I don't know MacOS that well, but I believe they use something similar to run System 9 and below programs in PPC MacOS X (Intel MacOS X obviously needs to use something more like UAE)

@CodeSmith

Nice explanation!

_________________
/Harry

[SOLD] µA1-C - 750GX 800MHz - 512MB - Antec Aria case

Avatar by HNL_DK!

I really can't understand anyone having a problem here. Look at the 68k emulation we have already - all Exec and other OS calls are passed through to the existing OS without interference. The only restriction on 68k stuff is that which applies to all native PPC stuff as well - you can't write over the Exec code areas. 68K emulated apps (and OS4 apps) can still read or write inner details of Exec structures and "hack" things. So could any malicious code that might appear.

Now we have decided to modernise the OS a bit more. The next step is to introduce some deliberate security for the OS by making inner details of Exec structures invisible. The reason? Well, the Semaphore problem is one example: in order to test a Semaphore, we have to Forbid() all task switching, examine the 16-bit counter in the Semaphore structure and then Permit() task switching again. We're trying to get rid of Forbid()s and Permit()s for many reasons, at least one of which is that we have to do it before we can even think about multiple CPUs. If ONLY we could test the semaphore with an atomic instruction that cannot be interrupted and (possibly) cause a task switch, we could remove the Forbid/Permit pair from the Semaphore handling.

Well, we do have such an atomic, uninterruptible instruction in the PPC, but it only works on longwords (32-bit). To change the Semaphore handling to get rid of the Forbid/Permit pair and use the atomic instruction instead, we need to change the internal counter from 16 bits to 32. But if we do that, the structure will get bigger and every program out there that is compiled for Semaphores with 16-bit counters will fail. Nearly every other OS structure has the same problem - we can't change them if their contents have been exposed to the world, because some one or some compiler will have reserved space for the old format. Hence we have introduced the Mutex, which is just a Semaphore with a 32-bit usage counter.

The new approach will probably be to forbid (arrghh, that word again) programs from reading or writing from/into OS structures directly. New OS calls will be provided so that an app can examine an inner field of a structure and/or modify it, but only the Exec will know exactly where the modified field lives. This means that if we want to change it, we don't have to recompile all the applications out there.

For existing or new native PPC apps it's easy - just replace a few existing references here and there with Exec calls that do the same thing, but in a controlled manner. Some OS4 apps might need some changes to make them more neighbourly. There aren't many, so now is the time to do it.

In the case of old 68k programs, the emulation will have to be changed so that Petunia, the built-in interpreter and 68k library stubs use the new Exec calls rather than direct access to system structures.

Now if people want to call that "putting a wrapper around the 68k app", "building a sandpit" or "stealing ideas from someone who did the same thing years ago", it's their preference. I seem to remember learning about encapsulation back in 1971-72. Doesn't matter what you call it, old 68k apps are going to run the same, native apps are going to run the same, the Workbench is going to look the same and the sky will stay up.

_________________
cheers
tony

Hyperion Support Forum: http://forum.hyperion-entertainment.biz/index.php

It's pretty unneccessary to start to talk about the next OS4.x version
in the middle of a ongoing court case. It might not be they that are
programming when the court case is over.

_________________
AmigaOS or MorphOS on x86 would sell orders of magnitude more than the current,
hardware-intensive solutions. And they'd go faster.-- D.Haynie

@tonyw

Quote:

Well, we do have such an atomic, uninterruptible instruction in the PPC, but it only works on longwords (32-bit). To change the Semaphore handling to get rid of the Forbid/Permit pair and use the atomic instruction instead, we need to change the internal counter from 16 bits to 32.

It also works on 16bit integers.

_________________
Amiga Developer
Amiga 500, Efika, Mac Mini and PowerBook

@ChrisH

I cant see how the OS can been seen as moving anywhere at the moment, I mean we don't even have the hardware to run it on and the availability of the OS at the minute is only available for AmigaOne owners.

This could be an interesting topic to quiz the owners of the OS when it comes out of court.

ESG

_________________
A500 + A501 OS1.3
A1200 OS3.1

@ChrisH

I don't understand you. There will be no progress without sacrifices. I think that Rogues idea is clear and wise: modern OS with AmigaOS 4.x compatibility. Do we need anything else? If Rogue says that current method of emulation isn't possible for future versions of OS, then he says truth. I don't think that you know AmigaOS better than Friedens.

@ExecSG

Quote:

I cant see how the OS can been seen as moving anywhere at the moment, I mean we don't even have the hardware to run it on and the availability of the OS at the minute is only available for AmigaOne owners.

Well, development is ongoing despite the lawsuit, so the OS is definately "moving" somewhere. It just remains unclear if the end-users will ever be able to make any use of these advancements when the dispute is settled (2008? 2009?). And... it also remains to be seen if there will be many who will still care at that time...

@Framiga
I raised the issue of why the last thread was locked, because some people might not realise why, and thus repeat the same mistake in this thread. (And as you may have gathered, I don't believe in not mentioning things that are important, even if they are unpleasant.)

@Hans who said Quote:

Contrary to what people think, message passing between programs doesn't need to involve copying data between their memory spaces

But more interestingly, although Bernie doesn't quite agree, is that memory protection does not REQUIRE a separate address space per process. It's quite possible to have a single "unified address space" used by the OS & all processes - only the protection of memory areas needs to change for each process. This does loose you the ability to do a few neat Unix-like hacks, such as Fork(), but we've gotten by fine without those so far.

@CodeSmith
Interesting proposal for how a seamless "virtual environment" might work: But if true it would mean that Hyperion will need to completely re-implement all the core OS3.x libraries, as a set of emulation "wrappers" for the new OS4.x libraries. That sounds like an awful lot of work, and one that will detract from work on the new OS4.x itself.

Surely it would be MUCH easier to simply run OS4.0 on a "virtual machine"? I think Rogue even said something to that effect, about rewriting the HAL as software layer over OS4.x. That way OS3.x (and OS4.0) stuff can use all their existing code, and then only maybe later would they add conduits to the real OS4.x, as & when time allowed (such as a virtual ethernet device that acted as a proxy to the real network/internet).

Last edited by ChrisH on 16-Aug-2007 at 10:15 AM.

_________________

@pavlor
I don't know why neither you nor Rogue nor others understands me: I did not say this was a bad idea, I simply said it was a completely unexpected change of direction (if you extrapolated the current OS4.0 developments). Like any idea it has good & bad points, but I was under the impression that Hyperion & Amiga Inc originally rejected the sand-boxing approach, at least until OS5.0 .

_________________

@tonyw who said Quote:

For existing or new native PPC apps it's easy - just replace a few existing references here and there with Exec calls that do the same thing, but in a controlled manner.

For less-complicated that may well be true, but I suspect that for larger & more complex apps such modifications are going to be quite large in number. Non-trivial even, which I think is why Rogue said they may add SOME compile-time emulation (redirection) of the old OS4.0 system.

_________________

@ChrisH

I can't understand it either. But it usually happens as soon
as some from the dev. team said anything. I draw the conlclusion
that you are not allowed to question anything thus upsetting the
coders which will result in them not posting anymore or worst
case stop developing.
And I guess some think that there are only a few
that know how to crunch code.

_________________
AmigaOS or MorphOS on x86 would sell orders of magnitude more than the current,
hardware-intensive solutions. And they'd go faster.-- D.Haynie

@ChrisH

Quote:

I don't know why neither you nor Rogue nor others understands me: I did not say this was a bad idea, I simply said it was a completely unexpected

Not unexpected at all, I explain the current behaviour in AmigaOS and short comings like this.

Currently when you think about adding some thing to a list structure you most disable multitasking in order to protect the list structure from being read/writen to by other tasks, this behaviour does force the OS in to a stop, it not SMP friendly, it’s not even multitasking friendly.

The idea is to only stop the task that tries to reads from the list structure, but currently no program check if is safe to read, because it is not possible to read under modification of list structure anyway.

There is also the issue about open structures that can be expanded because some silly program expect the structure stay unchanged, system architects are not so happy about this, and likes to hide it away and memory protect it, so no one tries to hack it.

_________________
http://lifeofliveforit.blogspot.no/
Facebook::LiveForIt Software for AmigaOS

@ChrisH

Quote:

ChrisH wrote: @pavlor I don't know why neither you nor Rogue nor others understands me: I did not say this was a bad idea, I simply said it was a completely unexpected change of direction (if you extrapolated the current OS4.0 developments). Like any idea it has good & bad points, but I was under the impression that Hyperion & Amiga Inc originally rejected the sand-boxing approach, at least until OS5.0 .

As I pointed out before:

- It is not a sandbox. The Mach kernel does the same, it can run multiple operating systems at once (for example OSF/1 and BSD 4.2. None of them runs in a sandbox. Each of them runs in their own virtual environment.

- This approach was part of my very first specification for AmigaOS (I think at that time I called it the "Chinese Wall Approach" because both parts of the systems would essentially be walled from each other).

So this was neither unexpected nor a change of direction. We didn't discuss future plans in the public because plainly, it's difficult to say how things will evolve, But the plan was there and is still there.

_________________
Seriously, if you want to contact me do not bother sending me a PM here. Write me a mail