Let us now discuss low-level technical questions that help to optimise the consumption of valuable resources such as memory and its access bandwidth, using the mainstream version of one of the highest-level languages, Python, as an example of how a careful selection of data types as well as their judicious organisation can bring concrete benefits in the execution of applications written in this language.

English: https://www.appuntidigitali.it/22159/not-always-big-is-better-the-importance-of-choosing-data-types-an-example-with-cpython/

Italian: https://www.appuntidigitali.it/22049/non-sempre-big-is-better-limportanza-della-scelta-dei-tipi-di-dati-un-esempio-con-cpython/

@cdimauro

Now do it using ARexx.

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@kolla: do what? AREXX is already quite optimized: written in assembly, and I doubt that it needs those kind of careful selections of data types & their locations.

Fun thought of the day:

Use ARexx stem variable notation to define data structures to be consumed in a manner directly analogous to JSON. Only you could call it ArSON.

_________________
Doing stupid things for fun...

@Karlos

Reminds me that I must build jq for os3.

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@cdimauro

Quote:

cdimauro wrote: @kolla: do what?

Write an in-depth article about it, its strengths and how to mitigate or work around its weaknesses and pitfalls.

Quote:

AREXX is already quite optimized: written in assembly, and I doubt that it needs those kind of careful selections of data types & their locations.

But how does it compare with other rexx interpreters?

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@kolla

Quote:

kolla wrote: @cdimauro Quote: cdimauro wrote: @kolla: do what? Write an in-depth article about it, its strengths and how to mitigate or work around its weaknesses and pitfalls.

I don't see any value on that.
Quote:

Quote: AREXX is already quite optimized: written in assembly, and I doubt that it needs those kind of careful selections of data types & their locations. But how does it compare with other rexx interpreters?

I don't recall how many times I've used AREXX, and I had/have even less interest on other languages like that.

If I've to spend some of my little free time, then Python has absolutely way more value compared to A/REXX.

Really, who cares about AREXX nowadays? You can even write an AREXX server in Python, if needed...

@cdimauro

I care about arexx, it’s the primary language on amiga to get stuff done. I used python on amiga a couple of decades ago, it was ok-ish, bloated for what it could offer. The state of Python on Amiga today makes it much less… desirable.

But maybe I’m pullng your leg, why on earth are you write about python on an Amiga forum …

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@cdimauro

OMG

Python is using string for everything and numbers written in strings are very big data types, far from small, fast or effective, that's why they introduced NumPy.

But you are telling us old news.

OneTimer1 Quote:

Python is using string for everything and numbers written in strings are very big data types, far from small, fast or effective, that's why they introduced NumPy.

ARexx does the same thing of storing all variable datatypes as strings. Universal variables stored as strings have advantages but so do variables using standard C datatypes. I've thought about ways to retrofit standard C datatypes to ARexx to have the advantages of both.

Last edited by matthey on 03-Jul-2024 at 12:24 AM.

@kolla

Do you know any good resource for learning arexx?

_________________
Indigo 3D Lounge, my second home.
The Illusion of Choice | Am*ga

pixie Quote:

Do you know any good resource for learning arexx?

There is good documentation for ARexx.

https://www.rexxla.org/rexxlang/history/mfc/rexxbook.html

I have the "Amiga Programmer's Guide to ARexx" and it is very good.

Amiga Programmer's Guide to ARexx -- Eric Giguere
Commodore-Amiga, Inc., 1991

The William Hawes manual is a classic too.

https://archive.org/details/ARexx_Users_Reference_Manual_1.0_1987_Hawes_William_S.

Last edited by matthey on 03-Jul-2024 at 03:28 AM.

For reference, https://wiki.amigaos.net/wiki/AmigaOS_Manual:_ARexx

A lot of the "power" of arexx is in the third party libraries and programs that have arexx support. The arexx support in Amiga OS itself could certainly have been better, for example Workbench can be controlled through arexx, but quite few obvious things are also missing (like getting path corresponding to selected files when they are "left out" on desktop, or when they are located in more than just active drawer window) and it’s also quite slow (adding and removing menus).

Last edited by kolla on 03-Jul-2024 at 04:50 AM.

_________________
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC

@kolla

Quote:

kolla wrote: @cdimauro I care about arexx, it’s the primary language on amiga to get stuff done.

That's fine, if you also like it.
Quote:

I used python on amiga a couple of decades ago, it was ok-ish, bloated for what it could offer.

What do you expect? It's not written in assembly and the current C implementation (especially the operators execution) has several things that might be further optimized.

Plus, it's way more general purpose of many other languages (e.g.: it has a lot to offer "batteries included", and this has a cost).
Quote:

The state of Python on Amiga today makes it much less… desirable.

Indeed. A system without Python nowadays is like one without a modern browser. Python is used everywhere (even on oses installers).
Quote:

But maybe I’m pullng your leg, why on earth are you write about python on an Amiga forum …

Why on Earth people don't know the forum where they are active since decades? Here is it:

"General Technology (No Console Threads)"

Clear now?

@OneTimer1

Quote:

OneTimer1 wrote: @cdimauro OMG Python is using string for everything and numbers written in strings are very big data types, far from small, fast or effective, that's why they introduced NumPy. But you are telling us old news.

Let me guess: you haven't read the article and you clearly have no idea on how data types are stored in Python, right?

Because actually the situation is the exact opposite of what you're telling: Python has a very rich set of DIFFERENT data types which can be used depending on the specific scenario.

Guess what: strings are used... where they are needed. For everything else there's a very high chance that there's a specific data type, which is obviously better optimized for the scope.

TLDR; Python do NOT use strings for everything and numbers are NOT written in strings. That's a completely misleading information that you're giving.

BTW, NumPy is used for VECTORS and MATRIXES of numbers (of different types: from small ints to double FPs). Which is a completely different thing compared to the numbers data type which are builtin in Python (ints, floats, complexes, fractions).


@matthey

Quote:

matthey wrote: OneTimer1 Quote: Python is using string for everything and numbers written in strings are very big data types, far from small, fast or effective, that's why they introduced NumPy. ARexx does the same thing of storing all variable datatypes as strings. Universal variables stored as strings have advantages but so do variables using standard C datatypes. I've thought about ways to retrofit standard C datatypes to ARexx to have the advantages of both.

Don't worry: it's absolutely NOT the case with Python. See above.

BTW, there's an array data type in Python, which stores ints and floats of any size in a much more efficient way. But it offers no operators besides story those datas. That's why NumPy is the preferred way.

@kolla

Quote:

kolla wrote: For reference, https://wiki.amigaos.net/wiki/AmigaOS_Manual:_ARexx A lot of the "power" of arexx is in the third party libraries and programs that have arexx support. The arexx support in Amiga OS itself could certainly have been better, for example Workbench can be controlled through arexx, but quite few obvious things are also missing (like getting path corresponding to selected files when they are "left out" on desktop, or when they are located in more than just active drawer window) and it’s also quite slow (adding and removing menus).

Using AREXX ports could be emulated with other languages (I don't recall now which Amiga Python implementation has some libraries for it).

However, the main problem with AREXX is that it's obsolete and unmaintained. And its source is unavailable (that's the reason why AmigaOS4 is still using the 68k one: it was impossible to port it and completely rewriting it from scratch is a huge task).

But actually it can be replaced. For the AREXX ports, see above.
For the language itself a parser can be written with one of the many tools which allow to quickly write parsers (I prefer ANTLR4 for this: it's easy, much more readable, and supports tons of different programming languages).

Easy-peasy: with a quick search I've found a REXX grammar written in ANTL4: https://github.com/antlr/grammars-v4/tree/master/rexx

Java and Python are supported as target languages, but any other can be easily added (there's very little boilerplate code to be written for enabling another language: see the Java or Python implementations).

That's a very good (and easy!) starting point for writing an AREXX interpreter.

/OT

@kolla

Quote:

kolla wrote: A lot of the "power" of arexx is in the third party libraries and programs that have arexx support.

Yep, if it wasn't for that it would be of no use to me.

The problem with interpreters like AREXX and Python is that while the code is reasonably compact for what it typically does, the interpreter itself has a large overhead. We use these interpreted languages for convenience, not efficiency.

AREXX is written in asm so the interpreter is reasonably compact at ~35k bytes, but I still found it onerous when trying to make an AmiTCP boot disk. Amiga python 2.4 is 832k, so it would use up the whole disk! When could efficient coding make up for that overhead? Practically never. If you have a program large and complex enough to justify the space taken up by the interpreter then the execution time will be biting, so you lose either way.

As for 'the importance of choosing data types", well duh.

I've done a lot of coding for small microcontrollers with as little as 512 words of program memory and 16 bytes of data memory. With these tiny chips you can't afford bloat. In one project I had to port the C code for a GAL programmer from Windows to Atmel AVR. The GAL configuration '.jed' (text) file is received over the serial port, and the configuration data stored in the MCU's RAM before being programmed into the GAL one 'fuse' bit at a time. Problem is the Windows program stored the fuse bits in an array using one byte per bit. The ATMega328 only has 2k of RAM which isn't enough to store the 5892 fuse settings. So I changed the code to store the fuse data as 8 bits per byte and then had plenty of RAM for the job.

"Well duh", you say. "Just use python, it's got bit arrays!" Except there's no way the python interpreter will fit into the ATMega328. Even the so-called 'micro' python that is described as a 'lean and efficient implementation' with only a 'small subset' of the Python standard library is much too big, requiring 256k ROM and 16k RAM (ATMega328 only has 32k ROM).

cdimauro Quote:

Using AREXX ports could be emulated with other languages (I don't recall now which Amiga Python implementation has some libraries for it).

The ARexx message ports are normal Amiga message ports. Any program can send messages to the public ARexx ports so other languages can too.

bhabbott Quote:

The problem with interpreters like AREXX and Python is that while the code is reasonably compact for what it typically does, the interpreter itself has a large overhead. We use these interpreted languages for convenience, not efficiency. AREXX is written in asm so the interpreter is reasonably compact at ~35k bytes, but I still found it onerous when trying to make an AmiTCP boot disk. Amiga python 2.4 is 832k, so it would use up the whole disk! When could efficient coding make up for that overhead? Practically never. If you have a program large and complex enough to justify the space taken up by the interpreter then the execution time will be biting, so you lose either way.

MicroPython has been gaining in popularity.

https://micropython.org/ Quote:

MicroPython is packed full of advanced features such as an interactive prompt, arbitrary precision integers, closures, list comprehension, generators, exception handling and more. Yet it is compact enough to fit and run within just 256k of code space and 16k of RAM.

The RP2040 SoC RPi Pico with 264 kiB SRAM uses it. Only a CLI text based interface is available. The Amiga 1000 had a GUI, preemptive multitasking OS and enough memory for ARexx with 256kiB of memory. It wasn't even optimized with 68020 code which would improve the code density and more is possible with ColdFire and other optimizations. The AmigaNone is a fat overpriced PPC desktop computer though. Only Amiga1 makes it impossible to proliferate the 68k Amiga.

Last edited by matthey on 04-Jul-2024 at 05:35 PM.

@bhabbott

Quote:

bhabbott wrote: AREXX is written in asm so the interpreter is reasonably compact at ~35k bytes, but I still found it onerous when trying to make an AmiTCP boot disk. Amiga python 2.4 is 832k, so it would use up the whole disk! When could efficient coding make up for that overhead? Practically never.

It's very simple: it depends on the time which developers have available. Since you retired, you've plenty of time which you can waste still writing assembly code.

However, for many other developers time is a valuable and a scarse resource. That's the reason why high(er)-level language are still so much used.
Quote:

If you have a program large and complex enough to justify the space taken up by the interpreter then the execution time will be biting, so you lose either way.

If the execution time is enough, then I don't see what's the problem.
Quote:

As for 'the importance of choosing data types", well duh. I've done a lot of coding for small microcontrollers with as little as 512 words of program memory and 16 bytes of data memory. With these tiny chips you can't afford bloat. In one project I had to port the C code for a GAL programmer from Windows to Atmel AVR. The GAL configuration '.jed' (text) file is received over the serial port, and the configuration data stored in the MCU's RAM before being programmed into the GAL one 'fuse' bit at a time. Problem is the Windows program stored the fuse bits in an array using one byte per bit. The ATMega328 only has 2k of RAM which isn't enough to store the 5892 fuse settings. So I changed the code to store the fuse data as 8 bits per byte and then had plenty of RAM for the job.

And? You've no idea what I was able to pack on an AVR microcontroller with just 28kB of RAM (included the bookloader) and 2.5kB of RAM.
Quote:

"Well duh", you say. "Just use python, it's got bit arrays!"

No, it hasn't bit arrays. However, ints can be used for this.
Quote:

Except there's no way the python interpreter will fit into the ATMega328. Even the so-called 'micro' python that is described as a 'lean and efficient implementation' with only a 'small subset' of the Python standard library is much too big, requiring 256k ROM and 16k RAM (ATMega328 only has 32k ROM).

http://www.tinypy.org

tinypy is a minimalist implementation of python in 64k of code