Your support is needed and is appreciated as Amigaworld.net is primarily dependent upon the support of its users.
|
|
|
|
Poster | Thread | NutsAboutAmiga
| |
interesting discussion about endianness. Posted on 4-Aug-2021 18:07:53
| | [ #1 ] |
| |
|
Elite Member |
Joined: 9-Jun-2004 Posts: 12795
From: Norway | | |
|
| | Status: Offline |
| | NutsAboutAmiga
| |
Re: interesting discussion about endianness. Posted on 4-Aug-2021 18:35:59
| | [ #2 ] |
| |
|
Elite Member |
Joined: 9-Jun-2004 Posts: 12795
From: Norway | | |
|
| @Tygre
Pritt much covered already, topic, but I add a few words.
AmigaOS / Amiga hardware is big-endian, Network cards as well. MSDOS / x86 / Window and PC Hardware is little-endian.
Most C/C++ programmers will have to deal with this one time or another.
It’s too late agree on a standard, you run into old files, or files from obscure operating systems. files made by old programs.
you run into issues if you use PC hardware on Amiga, or if use files made on PC on Amiga, the same is true if use Amiga files on PC. Or older MacOS7.x.x files.
lot of code deals with endies is poorly optimized, a lot of it uses masking & shifting, and you store it, so at least 4 or 5 instructions, it adds up quickly.
some CPU’s support reverse order, as well native order, not all CPU do, sometimes these instructions are removed, as often seen as unnecessary, this make hard to depend on this instructions, and so must write a work around for it anyway.
for 16bit fastest way is lookup tables, as far I’m concerned, dest16b = lookup16b[souce16b]
(its only 128 kb, large table)
lookup tables can be where fast way, to filter out data as well, instead of masking and shifting, multiplications etc. to convert from one value to another.
lookup tables are big help in converting from 8bit to 16bit or 32bit, or from 16bit to 32bit. 8bit is same on little and big-endian so not an issue.
32bit, is too big for lookup table, however if you shift rotating and masking, you are doing it the wrong way, you need access etch byte in the integer, you can use arrays, or use unions, I believe unions to be cleanest way, you might run into issues with alignment, but most cases it should work fine.
Don’t write general routines should be able to handle everything, write a routine for etch case instead, if speed is your problem. Anyway, problem is not writing new code, the problem is finding issues someone else’s code.
Last edited by NutsAboutAmiga on 04-Aug-2021 at 06:55 PM. Last edited by NutsAboutAmiga on 04-Aug-2021 at 06:49 PM. Last edited by NutsAboutAmiga on 04-Aug-2021 at 06:36 PM.
_________________ http://lifeofliveforit.blogspot.no/ Facebook::LiveForIt Software for AmigaOS |
| Status: Offline |
| | BigD
| |
Re: interesting discussion about endianness. Posted on 4-Aug-2021 22:52:27
| | [ #3 ] |
| |
|
Elite Member |
Joined: 11-Aug-2005 Posts: 7307
From: UK | | |
|
| @NutsAboutAmiga
This thread title is an oxymoron; there is no interesting discussion to be had about endians just as you cannot comment in an interesting way about people driving on the opposite side of the road in different countries! _________________ "Art challenges technology. Technology inspires the art." John Lasseter, Co-Founder of Pixar Animation Studios |
| Status: Offline |
| | kolla
| |
Re: interesting discussion about endianness. Posted on 5-Aug-2021 3:53:19
| | [ #4 ] |
| |
|
Elite Member |
Joined: 20-Aug-2003 Posts: 2859
From: Trondheim, Norway | | |
|
| The linked pointed to a 41 year old April first RFC, not meant to be taken seriously.
It mentions that... Quote:
In English, we write numbers in Big-Endians' left-to-right order. I believe that this is because we SAY numbers in the Big-Endians' order, and because we WRITE English in Left-to-right order
|
That may well be, but this hasn't always been the case. In German and Dutch for example, numbers are expressed Little-Endian, and it's not that long ago since it was also common in English, as one can see in bible translations and maybe 100 year old novels, and still is in various old rhymes and lyrics.
Norwegian implemented Big-Endian numbers quite recently, with the introduction of phones and phone centrals - phone numbers were cumbersome to read out Little-Endian and note down in the other end of a line, so a new Big-Endian system was developed. For example, 27 went from "syv-og-tyve" (seven-and-twenty) to "tjuesju" (twentyseven). The old Little-Endian system has not vanished yet, and the two systems are used in parallelle. If you go to a bingo, you may here the following "Tallet er førti-tre - jeg gjentar, tre-og-førr" ("The number is forty-three - I repeat, three-and-fourty" ("førti" is often shortened to just "førr" in the old system))Last edited by kolla on 05-Aug-2021 at 03:56 AM. Last edited by kolla on 05-Aug-2021 at 03:55 AM.
_________________ B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC |
| Status: Offline |
| | Varthall
| |
Re: interesting discussion about endianness. Posted on 5-Aug-2021 8:33:55
| | [ #5 ] |
| |
|
Super Member |
Joined: 17-Feb-2004 Posts: 1559
From: Up Rough | | |
|
| @kolla
Same in Slovenian, we always read numbers in little-endian order.
Varthall _________________ AmigaOne XE - AmigaOS 4.1 - Freescale 7457 1GHz - 1GB ram |
| Status: Offline |
| | tygre
| |
Re: interesting discussion about endianness. Posted on 5-Aug-2021 14:27:02
| | [ #6 ] |
| |
|
Regular Member |
Joined: 23-Mar-2011 Posts: 279
From: Montreal, QC, Canada | | |
|
| | Status: Offline |
| | IridiumFX
| |
Re: interesting discussion about endianness. Posted on 5-Aug-2021 15:18:35
| | [ #7 ] |
| |
|
Member |
Joined: 7-Apr-2017 Posts: 80
From: London, UK | | |
|
| @kolla
I remember from childhood that small numbers in German are read little endian, but larger numbers are read big-endian. I can't remember when the cut-off happens, maybe after 99
32 = zweiunddreißig 256 = zweihundertsechsundfünfzig 262144 = zweihundertzweiundsechzigtausendeinhundertvierundvierzig
zwei (2) is easy to spot
|
| Status: Offline |
| | TRIPOS
| |
Re: interesting discussion about endianness. Posted on 5-Aug-2021 20:12:32
| | [ #8 ] |
| |
|
Super Member |
Joined: 4-Apr-2014 Posts: 1204
From: Unknown | | |
|
| One little, two little, three little endians...
|
| Status: Offline |
| | matthey
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 0:00:32
| | [ #9 ] |
| |
|
Super Member |
Joined: 14-Mar-2007 Posts: 1968
From: Kansas | | |
|
| NutsAboutAmiga Quote:
lot of code deals with endies is poorly optimized, a lot of it uses masking & shifting, and you store it, so at least 4 or 5 instructions, it adds up quickly.
some CPU’s support reverse order, as well native order, not all CPU do, sometimes these instructions are removed, as often seen as unnecessary, this make hard to depend on this instructions, and so must write a work around for it anyway.
for 16bit fastest way is lookup tables, as far I’m concerned, dest16b = lookup16b[souce16b]
(its only 128 kb, large table)
lookup tables can be where fast way, to filter out data as well, instead of masking and shifting, multiplications etc. to convert from one value to another.
|
Using a table for endian conversion wastes DCache and memory. 32 bit endian conversion is only 3 instructions on the 68k but they are all dependent.
byterev: ror.w #8,d0 ; 1 cycle pOEP swap d0 ; 1 cycle pOEP ror.w #8,d0 ; 1 cycle pOEP === 3 cycles total, 3 cycles/conversion on modern core
If more than one conversion is necessary then interleave.
byterev_2times: ror.w #8,d0 ; 1 cycle pOEP ror.w #8,d1 ; 1 cycle sOEP swap d0 ; 1 cycle pOEP swap d1 ; 1 cycle sOEP ror.w #8,d0 ; 1 cycle pOEP ror.w #8,d1 ; 1 cycle sOEP === 3 cycles total, 1.5 cycles/conversion on modern core
Unfortunately, the SWAP instruction was *not* made superscalar on the 68060 even though throughput is 1 cycle and the result is 32 bit which can be forwarded.
moveq #16,d2 ; 1 cycle pOEP byterev_2times: ror.w #8,d0 ; 1 cycle pOEP ror.w #8,d1 ; 1 cycle sOEP ror.l d2,d0 ; 1 cycle pOEP ror.l d2,d1 ; 1 cycle sOEP ror.w #8,d0 ; 1 cycle pOEP ror.w #8,d1 ; 1 cycle sOEP
The MOVEQ can be placed before a loop doing conversions so it has little affect on performance.
pOEP is the primary integer pipeline sOEP is the secondary integer pipeline
kolla Quote:
The linked pointed to a 41 year old April first RFC, not meant to be taken seriously.
|
It's meant to be funny or at least sarcastic at times but the points are still valid.
kolla Quote:
That may well be, but this hasn't always been the case. In German and Dutch for example, numbers are expressed Little-Endian, and it's not that long ago since it was also common in English, as one can see in bible translations and maybe 100 year old novels, and still is in various old rhymes and lyrics.
|
That's true but more modern numbers using 0-9 symbols are nearly universally written from most significant to least significant digits, at least with left to right reading languages. It makes sense that a right to left reading language would be the opposite. The bigger magnitude digits are more significant so they come first. Perhaps the most useful is to be consistent which has become the universal standard even though it didn't use to be. There used to be mixed endian hardware too like VAX. At least computers use base 2 and humans base 10 fairly consistently now. Historically, there were base 60 and 20 numbering systems which are still found in modern use today (time, degrees, score).
Last edited by matthey on 06-Aug-2021 at 12:03 AM.
|
| Status: Offline |
| | kolla
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 6:04:44
| | [ #10 ] |
| |
|
Elite Member |
Joined: 20-Aug-2003 Posts: 2859
From: Trondheim, Norway | | |
|
| @matthey
And then there is the US date format which causes a lot of confusion, where 10/11/21 means eleventh of October and not tenth of November.
PS - oh, so you have started looking into VAX now :) Last edited by kolla on 06-Aug-2021 at 06:05 AM.
_________________ B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC |
| Status: Offline |
| | matthey
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 8:07:04
| | [ #11 ] |
| |
|
Super Member |
Joined: 14-Mar-2007 Posts: 1968
From: Kansas | | |
|
| kolla Quote:
And then there is the US date format which causes a lot of confusion, where 10/11/21 means eleventh of October and not tenth of November.
|
I don't like the default U.S. date format of month/day/year either. It is a ugly mixed format for computer use and sorting. The day/month/year format is better and more universal but I prefer year/month/day and using a 4 digit year as historical dates work better. Maybe the AD or BC should be added too like AD2021/8/6. Padding would make sorting and formatting easier so maybe AD2021/08/06 would be better. That would be politically incorrect though using big endian formatting and recognizing Christ. Cancel culture has tried to eliminate both big endian and all mention of Christ from the standard Gregorian calendar system which still dates from Christ despite name changes like CE and BCE.
Quote:
PS - oh, so you have started looking into VAX now :) |
I first looked at the VAX ISA when I was part of the Apollo team. The PDP-11 and VAX are probably the two ISAs that most influenced the 68k ISA. The little endian PDP-11 and VAX both use a mixed PDP-endian format for floating point stores (PDP-11 also for later extended sizes wider than 16 bits). Despite this, the IEEE 754 floating point single precision format was taken from this format (sign, 8 bit exponent, 23 bit mantissa with hidden bit). The VAX went too far at catering to the compiler instead of the hardware but it does give very compact code. The PDP-11 is more practical and had more influence on the 68k. The 68k is more like a super PDP-11 processor with 32 bit instruction sizes and registers added and double the GP registers.
Last edited by matthey on 06-Aug-2021 at 08:14 AM.
|
| Status: Offline |
| | Jupp3
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 10:13:47
| | [ #12 ] |
| |
|
Super Member |
Joined: 22-Feb-2007 Posts: 1225
From: Unknown | | |
|
| Since no-one else posted it yet, I'll do it.
Why is there such a thing as little endian format in the first place?
Take a 6502, for example (CPU in C64, NES, etc.), which is an 8-bit little endian CPU.
Then consider following command: LDA $1234, X
Which means, "Load into accumulator value from address $1234 + the value of the X register"
In hexadecimal it is $BD $34 $12 (command, followed by the lowest byte of the address, and lastly, the highest byte of the address)
So first the CPU reads the command from memory. Second comes the lowest byte. Then CPU can immediately add the value of X register to it, while loading the highest byte! - so it's kind of parallel processing.
(Obviously, if lowest byte overflows, the highest byte has to be incremented aswell, which adds 1 cycle to the execution time)
So little endian most definitely has simple & easy to understand efficiency-based reasoning behind it. Of course with more modern CPUs, this no longer makes a difference in execution time. Last edited by Jupp3 on 06-Aug-2021 at 10:14 AM.
|
| Status: Offline |
| | amigang
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 10:24:34
| | [ #13 ] |
| |
|
Elite Member |
Joined: 12-Jan-2005 Posts: 2018
From: Cheshire, England | | |
|
| On question I have, is I believe endian is the reason Odyssey Web browser development stalled, so how does Wayfarer get around it. Cheers _________________ AmigaNG, YouTube, LeaveReality Studio |
| Status: Offline |
| | kolla
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 11:13:47
| | [ #14 ] |
| |
|
Elite Member |
Joined: 20-Aug-2003 Posts: 2859
From: Trondheim, Norway | | |
|
| I suppose Christ was born one week too early. _________________ B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC |
| Status: Offline |
| | ppcamiga1
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 11:18:32
| | [ #15 ] |
| |
|
Cult Member |
Joined: 23-Aug-2015 Posts: 762
From: Unknown | | |
|
| If in the past aros devs and users where smart enough to support ppc we probably today will have good open source Amiga os reimplemenation and software for it. They were idiots and instead of cooperate waste time on holy war against ppc. No integration, no working software. After twenty five years still there is not working mui clone for aros. Most of Amiga software is big endian. Amiga will stay big endian forever.
Last edited by ppcamiga1 on 06-Aug-2021 at 11:19 AM.
|
| Status: Offline |
| | michalsc
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 13:16:21
| | [ #16 ] |
| |
|
AROS Core Developer |
Joined: 14-Jun-2005 Posts: 377
From: Germany | | |
|
| | Status: Offline |
| | NutsAboutAmiga
| |
Re: interesting discussion about endianness. Posted on 6-Aug-2021 15:45:11
| | [ #17 ] |
| |
|
Elite Member |
Joined: 9-Jun-2004 Posts: 12795
From: Norway | | |
|
| @amigang
Personally, I found it tiresome to work on mplayer, keep it update, and test different ffmpeg version and changes, there are changes every day, new bugs every day, bug fixes every day, to keep things up to date, it’s a day job only without pay. (Well got some cash but it was 1/3 what i need to doing as full time gig.), I doudt everyone be happy to keep paying for it... so no...
When talk about Odyssey Web browser you’re talking about something hugely more complex, contains many more modules, does not even compile under AmigaOS, so can’t compile and test easily, I know some developers have good cygwin / winuae, to setup good development Environment, you actually have to work on a different computer.
So, endianness is just one of many issues you need to work on, only developer who has the right motivation, and time for it, and is willing to offer their time, has successful in setting up cross development environment, and has skill to work on it, is willing to work on it, this narrows down list developers quickly in a tiny community. Maybe there are developers who might have the skill that do not try, because simply scared, many developers tried and fail, and don’t won’t try again, because when you fail you also waste a lot of time. So, it’s not without cost to just try. Even worst newer Odyssey Web browser also means you need newer gcc, newer libc, or newlib and other stuff, you also need to update, Odyssey Web browser is a large ecosystem.
Last edited by NutsAboutAmiga on 07-Aug-2021 at 08:41 AM. Last edited by NutsAboutAmiga on 06-Aug-2021 at 07:18 PM. Last edited by NutsAboutAmiga on 06-Aug-2021 at 04:18 PM. Last edited by NutsAboutAmiga on 06-Aug-2021 at 03:51 PM. Last edited by NutsAboutAmiga on 06-Aug-2021 at 03:47 PM.
_________________ http://lifeofliveforit.blogspot.no/ Facebook::LiveForIt Software for AmigaOS |
| Status: Offline |
| | NutsAboutAmiga
| |
Re: interesting discussion about endianness. Posted on 7-Aug-2021 8:53:01
| | [ #18 ] |
| |
|
Elite Member |
Joined: 9-Jun-2004 Posts: 12795
From: Norway | | |
|
| @BigD
Quote:
BigD wrote: @NutsAboutAmiga
This thread title is an oxymoron; there is no interesting discussion to be had about endians just as you cannot comment in an interesting way about people driving on the opposite side of the road in different countries! |
Traveling from one country to another can be confusing an disorientating, when need to change what side of the road you are driving on, many people traveling to England will choose to use Taxi instead of rental car, to overcome this problem. _________________ http://lifeofliveforit.blogspot.no/ Facebook::LiveForIt Software for AmigaOS |
| Status: Offline |
| | Hypex
| |
Re: interesting discussion about endianness. Posted on 9-Aug-2021 6:19:30
| | [ #19 ] |
| |
|
Elite Member |
Joined: 6-May-2007 Posts: 11180
From: Greensborough, Australia | | |
|
| @NutsAboutAmiga
Now you've got my attention. I'm going to add some excerpts from my article on "The worlds fastest endian" I wrote back in 2017 when I researched this. First hardware:
Computers: A lot of the early big computers such as minicomputers and mainframes made by IBM are big endian. Like the IBM Series/1 minicomputer. The IBM System/360, System/370 and ESA/390 mainframes. And the more recent IBM z/Architecture mainframe. The DEC PDP-10 also supported big endian.
Big CPUs: There is also the Microsoft Xbox 360 using Xenon processor, Nintendo GameCube, Wii, Wii U and Sony PlayStation 3 using the Cell CPU. All consoles using a CPU based on the Power architecture. And all big endian. Known current CPUs with big endian architecture are the (Motorola 68000 based) Freescale ColdFire, Xilinx Microblaze (FPGA), Hitachi SuperH, IBM z/Architecture and Atmel AVR32.
Bi-endian CPU: Other CPUs supporting big endian by being bi-endian now would be SPARC, Power Architecture and derivative PowerPC, which were originally big. ARM architecture was exclusively little-endian but became bi-endian with ARM 3. MIPS, HP PA-RISC, Hitachi SuperH SH-4 and Intel Itanium IA-64 are also bi-endian.
Big supported CPU: Intel x86 has had implicit big endian support with BSWAP instruction to swap bytes since 80486 series. And since Core series has specific big endian support with MOVEBE instruction to read and write memory data in big endian format.
Big data: Such as LZMA compression algorithm using it in some data types. The XDR External Data Representation Standard. And Java byte code has data encoded in big endian. For internet data structures, the data for IP addresses also has to appear in a certain endian, even though the dotted quad has referenced four bytes as separate units. This is called network order. And it's big endian. There is also BCD, binary coded decimal, as used in the 6502 series. Here decimal digits are stored in each nibble, so a bit different to byte based storage, yet the nibbles are stored as big endian.
Big formats: Adobe Photoshop IMG (GEM Raster) JPEG MacPaint SGI (Silicon Graphics) Sun Raster WPG (WordPerfect Graphics Metafile) (From the PC!)
Bi-endian formats: DXF (AutoCad) Microsoft RIFF (.WAV & .AVI) TIFF XWD (X Window Dump)
Looking for magic ID numbers (big endian) with portable code? It can be done! Most code I've seen isn't portable and looks like an x86 ASM programmer had a go at C. They search for an ID with letters reversed. I explain below why this not needed and how portable code can be written that is both cleaner and easy to understand.
I've seen code myself that assumes the hardware works in a certain way when reading memory. A simple case would be looking for an ID (such as a text ID) and comparing it with the reverse of what it really wants. At this point the code has gone below the high level it should be at and entered the low level. It is also unnecessary since simple macros can be created and do exist to deal with these types of values that are both readable and portable. Yet, they don't do that. And prefer to, it seems, write more confusing cryptic code that is tied to one particular hardware architecture. In lots of cases it would be a four character string ID so technically incorrect to treat as one quantity and it is just a cheat. But on the opposite side, if done on big endian, it works as intended and is readable in a WYSIWYG way. Still, in what ever method is used, I think the simple portable way would be best and a subset of endian problems wouldn't appear.
Example:
#define MakeID(a,b,c,d) ( (a<<24 | (b<<16 | (c<<8 | (d) )
LONG ID=MakeID('M','G','I','C');
if *id[]==ID // Such as...
That's from the Amiga world and it's bi-endian. It will put a four letter ID into the correct order and let the compiler figure it out without any need for any special conditional if's or def's. Just a simple macro. Might need updating for modern C. Sometimes I wonder if people can program with the code I've read.
Reading and writing in one specific endian, like little, is another matter. I've done this in the Stargus port and it was a lot of work for probably less in return. I hacked the code so every word read and write went through my own routine. Then later rewrote it as every new source version broke it and made it more forward portable but didn't end up releasing the code.
There are also endian built ins like gcc has. But they are flawed on PPC. PPC can read and write backwards like x86 but it can only do so with memory. So endian macro routines become useless as they are made to convert a variable. The variable is read in, then normal code needs to convert it, like 68K does. A swap on 68K is rol, swap, rol; but on PPC a swap can be a mess. Even though PPC has some rotating features. But, a proper working PPC macro needs to be given a direct address or base offset really, so the PPC does the operation directly with the code to do it. Ironic since endian issues are memory related at the core. Once inside the computer all data is equal.
https://blitterwolf.blogspot.com/2017/08/the-worlds-fastest-endian.htmlLast edited by Hypex on 09-Aug-2021 at 06:24 AM.
|
| Status: Offline |
| | ppcamiga1
| |
Re: interesting discussion about endianness. Posted on 9-Aug-2021 7:24:46
| | [ #20 ] |
| |
|
Cult Member |
Joined: 23-Aug-2015 Posts: 762
From: Unknown | | |
|
| @Hypex
These examples are simple cases that has nothing in common with endianess problem on Amiga Os. Amiga apps peek and poke into os and other apps. Amiga apps uses complicated data structures and expect data in big endian format. Amiga Os 1.x API was not designed to work on little endian. It will never be source and binary compatible on little endian.
|
| Status: Offline |
| |
|
|
|
[ home ][ about us ][ privacy ]
[ forums ][ classifieds ]
[ links ][ news archive ]
[ link to us ][ user account ]
|