@Raziel
me too

@Daedalus

Quote:

Daedalus wrote: @DWolfman Not by the looks of it - it seems to be a much deeper problem... I've just done a quick test copying a 3.8MB MP3 from my Amiga (1200/060/Mediator) to my Windows 7 machine. It took 3 minutes 32 to complete the transfer. Copying the same file from the Windows 7 machine took 20 seconds.

I just got around to look into this more deeply, and it seems that this is "normal" behaviour on account of how writing data works on the SMB file sharing level, as opposed to reading.

As I mentioned before, the SMB file sharing protocol as implemented by smbfs supports several different commands for reading and writing data from/to a file on the server.

The way it's set up, smbfs prefers to use the 'raw' read and write commands, because most of the time that's faster because the protocol overhead, compared to the 'not quite as raw' commands, is lower.

This holds true for the 'raw read' command, which tells the server to send back a portion of the file data in question in (as the name says) raw form with a minimum of packaging involved. There's a combined message type/length field which precedes the data, but that's it. You send one request, you receive the requested data (if available), and you're good to go.

Not so for the 'raw write' command... The protocol overhead is considerably higher. The client sends a message to the server, indicating where to store the data in the file, how much data there is, etc. This first message can contain payload data to write, but it doesn't have to. The server then responds with an acknowledgement for the write command and the client will then follow this up by sending more data, which will be acknowledged by the server. This is twice the effort of the 'raw read' command, and it can get even worse, because the server might just transmit "progress report" messages along the way.

Now this 'raw write' overhead may not sound like much, but because each request-response message pair has to be exchanged in sequence, and there is a delay between each message exchange, things will get slow by necessity. Considering that it happens on a 10 MBit/s Ethernet link, the impact will be felt.

Anyway, if you've read this far without getting thoroughly annoyed by the wicked details of this thing we call smbfs, rest assured that I'm working on a fix for the write speed issue

If you'd like to test the work in progress, feel free to contact me via PM.

@olsen

A feature request I guess... using smb as "default" network filesystem for Amiga is perhaps not a bad idea, allthough lighter variants exist, such as Amiga netfs, envoy and even ch_nfs, but one thing missing (I think) is support for Amiga filesystem metadata, such as protection flags and comment. WinUAE and FS-UAE works around this by using metadata files with suffix .uae (iirc), so how about (optionally, per mount) supporting these meta files? If you ever get around to writing an Amiga native smbd, I would also hope to see direct support for Amiga protection bits :)

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@Daedalus

I apologize, I misread your posting :)

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

Another update on the write speed issue: looks like smbfs has always (since before 2001) decoded the server protocol negotiation response information incorrectly. The server's "maximum buffer size" information was always decoded as 0, which had the effect of making smbfs switch to the default "safe" value of 1024.

This is problematic because it makes smbfs perform read and write operations using a maximum of 1024 bytes for all SMB file sharing protocol messages. The 'raw read' command was unaffected by this error, but the 'raw write' and the "normal" write operation were. This explains why the write operations were so very, very slow. The "maximum buffer size" is supposed to be 65535 bytes or more these days (some servers report more than twice that much).

I think I also just figured out why reading from a networked file system could be aborted before reading even a single byte, reporting an "object is in use" error.

Stay tuned... There could be another smbfs update next week.

@olsen

Wow, you're on fire! :) I did not find any way to make donations to you for your work so i just bought roadshow instead ;) keep up the good work.

@olsen

You are amazing!!!
Can't wait to try the new speedy smbfs.

Thank you very much, olsen

@olsen

I've no words....

Last edited by vulture on 17-Apr-2016 at 09:01 AM.

@kolla

Quote:

kolla wrote: @olsen A feature request I guess... using smb as "default" network filesystem for Amiga is perhaps not a bad idea, allthough lighter variants exist, such as Amiga netfs, envoy and even ch_nfs, but one thing missing (I think) is support for Amiga filesystem metadata, such as protection flags and comment. WinUAE and FS-UAE works around this by using metadata files with suffix .uae (iirc), so how about (optionally, per mount) supporting these meta files? If you ever get around to writing an Amiga native smbd, I would also hope to see direct support for Amiga protection bits :)

Hm... it may be possible to take care of the Amiga "metadata" aspects right within the file system client itself.

One could, for example, store the metadata for a file or drawer within a hidden file that bears the same name, except for a prefix tacked onto it (e.g. a drawer called "Tools" would have an associated "._Tools" file, or something like that). This is how Mac OS X approaches the same problem.

The file system client would then hide the fact that these files exist in the first place by filtering them out during directory scanning.

I think that it might be possible to implement such functionality with a minimum of effort, and the same drawbacks that exist for the Mac OS X solution. If you change the name of a file on the server side but neglect to rename the Amiga metadata file associated with it, the metadata information may be lost and just take up space.

@smf

Quote:

smf wrote: @olsen Wow, you're on fire! :) I did not find any way to make donations to you for your work so i just bought roadshow instead ;) keep up the good work.

Thank you! I hope it will be useful for you

And here's another update on the speed issue. After testing the fix for the maximum message size decoding error, I found that the impact is noticeable, but not as effective as I had hoped.

There is less overhead for writing because the file system no longer chops up the data into 1024 byte portions, sends them to server and then waits for the server to confirm that the data was received. Instead of sending/receiving 16 messages in total, 2-3 will suffice now.

However, it appears that writing is still noticeably slower than reading. From what I learned this may be because the server will not confirm to the smbfs client that the data has been written until the data has actually been written to disk and the disk cache buffers have been successfully flushed to disk.

While this procedure is (sort of) nice and reliable, even if a power failure should occur, it is also dog slow by design.

I did some more reading and found that you can deactivate this "don't confirm the write operation as being successful until the data has been written to the storage medium" behaviour, which is called "write-through mode" in the documentation. You can switch it off, activating whas is called "write-behind mode". This mode allows for the data to remain in a memory cache, to be written to storage when convenient for the server.

From the looks of it, it might be much faster than the alternative. Stay tuned, I'm currently tinkering with the code some more. Maybe this time it will actually work out and have a significant impact on write performance.

@olsen

Sorry, I haven't had a chance to do any testing in the past couple of days, but this does sound promising! Great work!

As for the metadata, I had actually started work on a custom copy command that did that, storing the extra data in a "dot-hidden" file and reading it back when copying the other direction. It worked very well and was a simple solution, though implementing it into a handler would be much better as that would allow it to be transparent for all functions (e.g. icon information), not just a custom Shell command.

_________________
RobTheNerd.com | InstallerGen | SMBMounter | Atoms-X

@olsen

Quote:

This explains why the write operations were so very, very slow. The "maximum buffer size" is supposed to be 65535 bytes or more these days (some servers report more than twice that much).

How does this relate to the MTU or even the amount of data per block requested to the TCP/IP stack?

I ask as years ago I was experimenting with some simple ARexx code that opened a HTTP file and loaded it off the server. I found the operation would go dog slow if I tried to read too much in at once or tried to read the whole file in one go. I then found out that using a buffer size of 4096 bytes was a happy medium and gave me best results. Even if a lot less than the file size.

This was a while ago so possibly I was experimenting with a PPP connection over dial up if that makes a difference.

@Hypex

Quote:

Hypex wrote: @olsen Quote: This explains why the write operations were so very, very slow. The "maximum buffer size" is supposed to be 65535 bytes or more these days (some servers report more than twice that much). How does this relate to the MTU or even the amount of data per block requested to the TCP/IP stack?

As far as I can tell from the documentation and what I see on the wire there is no relation whatsoever. How about that?

The SMB file sharing protocol sits on top of a session layer (that being SMB), which in turn sits on top of some transport layer which does not need to be TCP, actually.

This is why the SMB file sharing protocol is mostly oblivious to the fact that the underlying transport layer could be considerably more sophisticated than, say, IPX/SPX or whatever had crawled out of the primordial slime and ended up making a home in Windows 3.11 for workgroups as part of the bundle of networking stacks (I suppose it probably supported token ring networks, too, back then, in IBM's own protocol).

By the time IPX/SPX, etc. had been largely displaced by TCP/IP it was mostly too late to ditch those bits of the protocol which assumed that the transport layer didn't handle flow control, network congestion, segmentation, reassembly, etc. and all that fancy stuff which up to this point apparently nobody expected Windows users to find helpful.

However, some commands of the SMB file sharing protocol were introduced which did acknowledge that the service now ran exclusively over TCP and made use of that.

As far as I know the SMB service bumped against resource restrictions on the Windows platform which made it necessary to statically allocate memory buffers for each session. This is why the SMB setup involves a "maximum buffer size" negotiation as well as a "maximum raw read/write size" value. Windows NT is documented to have some hard-code defaults for the buffer size, depending upon how much RAM is available, for example.

But it turns out that even today's Windows SMB file sharing support is quite restricted. I just found out why write access was still so dog slow on Windows 7: it only allows for a buffer size of some 4K, which is more than it used to be when that value was decoded incorrectly by smbfs, but not much more. And it doesn't get better: in my tests I found that Windows 7 does not even allow the client to use raw read/write operations. It's designed to be slow, for some reason.

Testing with Samba 3.x, however, let the dogs off the leash: raw read/write is supported with a maximum raw read/write size of 65535 bytes, maximum buffer size set to 65535 bytes. It doesn't go any faster than that with the portion of the SMB file sharing protocol supported by the smbfs client code right now.

In so many words, Windows as an SMB file sharing server still leaves much to be desired as far as smbfs is concerned right now.

@olsen

Quote:

olsen wrote: Testing with Samba 3.x, however, let the dogs off the leash: raw read/write is supported with a maximum raw read/write size of 65535 bytes, maximum buffer size set to 65535 bytes. It doesn't go any faster than that with the portion of the SMB file sharing protocol supported by the smbfs client code right now.

So, since I don't care that much about machine-to-machine and do all my file sharing to shares on a central server with Samba 3.x.....

If you need someone to test it, I'm here.

Quote:

In so many words, Windows as an SMB file sharing server still leaves much to be desired as far as smbfs is concerned right now.

For some reason that does not surprise me.

_________________
This posting, in it's entirety, is the opinion and/or statement of the author and does not reflect the views and/or position of this site.

@olsen

Quote:

As far as I can tell from the documentation and what I see on the wire there is no relation whatsoever. How about that?

Well it is what it is. I don't know what was going on in my tests and I wasn't using SMB but a HTTP file loader programmed in ARexx. It was using the TCP: device. Mosy likely the Miami version. I'm sure I tried the Roadshow version a few years later. That may give a clue, being it was a DOS device, perhaps 4K was all it could handle.

In any case it's still a bit of a shocker what Windoes does to the SMB buffer size. Thanks for clueing us all in. And I expected Windoes to be faster given it would be considered a benchmark other platforms are compared against. Haha.

This reminds me of the MaxTransfer setting. Rather similar to the MTU. Well it almost works in the same exact way. Splitting data reads and writes into a set block size.

It wouldn't help outside of the Amiga world but I have been considering writing my own Amiga file sharing application. Been throwing a few ideas around my head for some while. I'd like some application where you install it with little setup and it sits in the background waiting to be used for file transfer. When a machine comes on it detects it over the network and auto mounts volumes on both systems. Either as seperate volumes or inside one volume. I would call it AmigaShare. Well just an idea or few.

@Hypex

Quote:

Hypex wrote: It wouldn't help outside of the Amiga world but I have been considering writing my own Amiga file sharing application. Been throwing a few ideas around my head for some while. I'd like some application where you install it with little setup and it sits in the background waiting to be used for file transfer. When a machine comes on it detects it over the network and auto mounts volumes on both systems. Either as seperate volumes or inside one volume. I would call it AmigaShare. Well just an idea or few.

NetFS Revised pretty much does that already. You can mount remote shares whenever you want (for example at boot), and if the remote machine isn't on you get standard "no disk in drive" for the mounted device and no icon on the desktop, but when you start the remote machine, icon appears on the desktop and volume name is assigned for it and it's accessible without any user activity. And same way if you turn remote machine off, icon and volume name disappears, and the device is left to "no disk in drive" state.

It's a cool Amiga native network filesystem and of course supports/preserves all Amiga's file properties etc. You can mount any devices and any number of them from remote Amigas, be it ram disk or real partitions etc.

Last edited by jPV on 22-Apr-2016 at 07:28 PM.
Last edited by jPV on 22-Apr-2016 at 07:27 PM.
Last edited by jPV on 22-Apr-2016 at 07:26 PM.

_________________
- The wiki based MorphOS Library - Your starting point for MorphOS
- Software made by jPV^RNO

@Hypex

Quote:

Hypex wrote: @olsen Quote: As far as I can tell from the documentation and what I see on the wire there is no relation whatsoever. How about that? Well it is what it is. I don't know what was going on in my tests and I wasn't using SMB but a HTTP file loader programmed in ARexx. It was using the TCP: device. Mosy likely the Miami version. I'm sure I tried the Roadshow version a few years later. That may give a clue, being it was a DOS device, perhaps 4K was all it could handle. In any case it's still a bit of a shocker what Windoes does to the SMB buffer size. Thanks for clueing us all in. And I expected Windoes to be faster given it would be considered a benchmark other platforms are compared against. Haha.

The Windows file sharing software is likely hampered by some 25 years of guesswork as to what the other end of the wire is trying to do. The SMB file sharing protocol and everything else connected to it were supposed to be proprietary technology, which, however, didn't keep 3rd parties from writing their own client and server software (there is Samba, and it's not the only 3rd party solution).

My best guess is that the commands used by smbfs are supported by legacy code in the Microsoft file server software. They keep working, but they are restricted to low performance operations. According to the documentation, several read/write command sets are designated as deprecated. I haven't checked, but I expect that the Microsoft file server provides significantly better performance if the more modern read/write commands are used. However, as I found out during my recent tests, Samba supports these legacy commands quite well and yields decent performance for smbfs.

Speaking of performance, I just ran a few new tests to get an idea of what the small changes I made can deliver. These tests exercised smbfs on AmigaOS 3.9 in WinUAE, which ran inside a "Windows 7" virtual machine on "VMware Fusion". I tested I/O performance between smbfs and "Windows 7" file sharing inside the same virtual machine, as well as between smbfs and Samba 3.0.25 running on a local server connected to my work machine via Gigabit Ethernet.

First things first: in this test smbfs was about exactly as fast when doing read/write operations with the local "Windows 7" file server as when doing the same with the Samba 3.0.25 server connected via Ethernet. By rights the "Windows 7" system should have been much, much faster since the data would have been copied from/to memory inside the virtual machine. That isn't the case, which means that Samba is significantly faster than the "Windows 7" file server because data had to leave the virtual machine, travel across the wire to the Samba server, get processed there and return over the wire into the virtual machine.

I also tested if write performance improves if smbfs is configured to use "raw write" commands instead of the normal "write" commands. The answer is no, because most of the time "raw write" requires two packets to be sent to the server, and that exchange causes a major performance hit. What works better is to let smbfs choose either the normal or the "raw" write command, depending upon how much data needs to be written. In so many words: write performance with Windows file server software is going to be poor for smbfs, probably until I can find a replacement set of read/write commands that make a difference. If you can, choose Samba.

Finally, I tested if it makes a difference to enable "write behind" mode for "raw write" commands. By default smbfs uses "write through" which requires that the Samba server (I only tested this with Samba; "Windows 7" does not offer "raw write" operations) concludes the entire write operation before reporting back to smbfs. Enabling "write behind" mode permits the file server to acknowledge the write operation immediately, carrying out the actual writing in the background. This is a bit risky because if a write error occurs, it will be reported by the server upon receipt of the next command, and by then the error could be out of context.

Anyway, in my tests I found that enabling "write behind" mode for "raw write" commands would yield an improvement of about 10%, which isn't much, but it might make a difference on 10 MBit/s Ethernet.

And that's all the news fit to print for this weekend

If you want to jump in and test-drive this smbfs version (1.109), please contact me via PM.

@olsen

Quote:

olsen wrote: @kolla Quote: kolla wrote: @olsen A feature request I guess... using smb as "default" network filesystem for Amiga is perhaps not a bad idea, allthough lighter variants exist, such as Amiga netfs, envoy and even ch_nfs, but one thing missing (I think) is support for Amiga filesystem metadata, such as protection flags and comment. WinUAE and FS-UAE works around this by using metadata files with suffix .uae (iirc), so how about (optionally, per mount) supporting these meta files? If you ever get around to writing an Amiga native smbd, I would also hope to see direct support for Amiga protection bits :) Hm... it may be possible to take care of the Amiga "metadata" aspects right within the file system client itself. One could, for example, store the metadata for a file or drawer within a hidden file that bears the same name, except for a prefix tacked onto it (e.g. a drawer called "Tools" would have an associated "._Tools" file, or something like that). This is how Mac OS X approaches the same problem. The file system client would then hide the fact that these files exist in the first place by filtering them out during directory scanning. I think that it might be possible to implement such functionality with a minimum of effort, and the same drawbacks that exist for the Mac OS X solution. If you change the name of a file on the server side but neglect to rename the Amiga metadata file associated with it, the metadata information may be lost and just take up space.

._Tools is not hidden on Windows, is it?

Anyways, I suggest discussing this with Toni Wilen of WinUAE and Frode Solheim of FS-UAE since they already do this, in my view it would make a lot of sense to standardize how filesystem handlers shall preserve amiga specific metadata on "alien" filesystems!!

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@kolla

Quote:

._Tools is not hidden on Windows, is it?

Windows files and folders can be marked as "hidden" using a file system layer attribute, and unless the file system explorer is configured to display hidden files, they will not show up in directory listings.

smbfs supports this "hidden" attribute, both when reading directories and through the attributes which can be set. Well, at least the protocols and data structures are available, though the Amiga side makes little use of them: you can tell smbfs to filter out hidden directory entries during directory scanning, and that's it for now.

Quote:

Anyways, I suggest discussing this with Toni Wilen of WinUAE and Frode Solheim of FS-UAE since they already do this, in my view it would make a lot of sense to standardize how filesystem handlers shall preserve amiga specific metadata on "alien" filesystems!!

It's an idea

I believe that WinUAE uses the NTFS "alternate data streams" feature to keep the Amiga-specific file properties (comment text, HSPA protection bits, plus the owner/group IDs and the three respective sets of RWED bits). This is a platform-specific solution for the Amiga metadata problem.

I'll have to check FS-UAE - it's possible that it has its own solution for the Amiga metadata problem, being a cross-platform application.