Spyke
linux·Linuxbygomp

Recommend me a scripting language

I've been looking around for a scripting language that:

  • has a cli interpreter
  • is a "general purpose" language (yes, awk is touring complete but no way I'm using that except for manipulating text)
  • allows to write in a functional style (ie. it has functions like map, fold, etc and allows to pass functions around as arguments)
  • has a small disk footprint
  • has decent documentation (doesn't need to be great: I can figure out most things, but I don't want to have to look at the interpter source code to do so)
  • has a simple/straightforward setup (ideally, it should be a single executable that I can just copy to a remote system, use to run a script and then delete)

Do you know of something that would fit the bill?


Here's a use case (the one I run into today, but this is a recurring thing for me).

For my homelab I need (well, want) to generate a luhn mod n check digit (it's for my provisioning scripts to generate synchting device ids from their certificates).

I couldn't find ready-made utilities for this and I might actually need might a variation of the "official" algorithm (IIUC syncthing had a bug in their initial implementation and decided to run with it).

I don't have python (or even bash) available in all my systems, and so my goto language for script is usually sh (yes, posix sh), which in all honestly is quite frustrating for manipulating data.

View original on lemmy.ml

Why aren't python and bash be available in all your systems? Which languages would be?

I would've recommended python, otherwise perl or Haskell (maybe Haskell's too big) or something, but now I'm worried that whatever reason makes python undoable also makes perl etc. undoable

94

I mean, it's a functional programming language and an incredibly good one. But it probably has a far too big footprint for their use-case...

8
gompreply
lemmy.ml

Why aren’t python and bash be available in all your systems?

Among others, I run stuff on alpine and openwrt.

I don't need to run these scripts everywhere (strictly speaking, I don't need the homlab at all), but I was wondering if there's something that I can adopt as a default goto solution without having to worry about how each system is packaged/configured.

As for python, I doubt the full version would fit in my router plus as said I don't want to deal with libraries/virtualenvs/... and (in the future) with which distro comes with python3 vs pyton4 (2 vs 3 was enough). Openwrt does have smaller python packages, but then I would be using different implementations on different systems: again something I'd rather not deal with.

As for perl, it would be small enough, but I find it a bit archaic/esoteric (prejudice, I know), plus again I don't want to deal with how every distro decides to package the different things (eg. openwrt has some 40+ packages for perl - if I were doing serious development that would be ok, but I don't want to worry about that for just some scripts).

9

Sounds like you want MicroPython. It’s definitely available on OpenWrt and AlpineLinux and has a very small footprint.

If you don’t like Python, have a look at Lua/luajit.

17

Is compiling scripts an option? Aka compiling them in C, C++, Rust, whatever for your router on another machine, and copying and executing those binaries on your router?

9

You might be interested in Raku. It is Perl6, or what used to be called Perl6, but it deviated too far away from the original perl and it ended up with a different team of developers than perl 5, so they forked it, changed the name and turned it into a new language.

1
Sharenireply
programming.dev

if there's something that I can adopt as a default goto solution without having to worry about how each system is packaged/configured.

Go is probably your best bet. Simple to use, and you can compile it so it runs everywhere

0

I found installing Go-sdk a total PiTA. It is okay as a developer environment. But bash + gnu utils + core utils seem much more sane to me.

Of course I mostly work with Linux systems and hardly ever have to deal with Scripting for Windoze.

3

luajit is small, fast(well, it can jit), and has a small but complete standard library and can do FFI pretty easily, should be ideal for most homelab usecase

ldd $(which luajit)                                                                                
        linux-vdso.so.1 (0x00007ffee9dc7000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb4db618000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb4db613000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb4db5f3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb4db3ca000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb4db799000)
30
lemmy.zip

Python is what you want. You can install it on just about any system.

Other than that maybe Lua but that will be hell.

21
Dingalingreply
lemmy.ml

Python is what you want. You can install it on just about any system.

Perl and bash are already there, no need to install anything.

-1
Dingalingreply
lemmy.ml

Try it now - go on. Type "perl" and tell me what you get.

And if you're so certain it's not used, try removing it and see how well your computer works afterwards.

9

If you just run perl it will sit waiting for input. Try perl --version

2

It isn't installed

I know that because I installed it as it was a dependency of Buildroot.

Edit: My bad I must of been thinking about a Perl library

2
neomachinoreply
lemmy.world

Is this the case? I don't feel like I've ever had to install Perl but I've had to install Python plenty of times and I use both pretty frequently on a daily basis. Not to mention a newer version, older version, 2.7.4 instead of 2.7.3.

1

Um, exactly the opposite on all the distros I use. All Enterprise Linux distros, Suse and Debian.

1
lemmy.world

What about Lua/Luajit?

In most scripting languages you have the interpreter binary and the (standard) libraries as separate files. But creating self-extracting executables, that clean up after themselves can easily be done by wrapping them in a shell script.

IMO, if low dependencies and small size is really important, you could also just write your script in a low level compiled language (C, Rust, Zig, ...), link it statically (e.g. with musl) and execute that.

19

I use Lua for this sort of thing. Not my favorite language, but it works well for it. Easy to build for any system in the last 20-30 years, and probably the next 20 too. The executable is small so you can just redistribute it or stick it in version control.

2

Perl is already installed on most linux machines and unless you start delving into module usage, you won't need to install anything else.

Python is more fashionable, but needs installing on the host and environments can get complicated. I don't think it scales as well as Perl, if that's a concern of yours.

16

That does seem to be just one, maybe two small files, and no dependencies. And a built in map() function.

4
vzq
lemmy.blahaj.zone

I can’t really think of anything that’s less frustrating than sh and ticks all your boxes. You can try TCL but it’s bound to be a shit show. It was painful to use two decades ago.

Perl is a step up in terms of developer comfort, but it’s at the same time too big and too awkward to use.

Maybe a statically linked Python?

13

I was thinking about recommending TCL as a joke. My favorite thing about it is it's "whimsicly typed."

2
Dingalingreply
lemmy.ml

Perl is a step up in terms of developer comfort, but it’s at the same time too big and too awkward to use.

How do you mean?

It's already on nearly every distro, so there's no core size unless you lean into modules. The scripts aren't exactly big either.

1
vzqreply
lemmy.blahaj.zone

He doesn’t have bash. I’m not sure I’ve seen a system this millennium with Perl but not bash.

0
Dingalingreply
lemmy.ml

Try it now - type perl. It's a dependency on a huge amount of core system tools.

1
programming.dev

OP is on OpenWRT (a router distro), and Alpine. Those distros don't come with very much by default, and perl is not a core dependency for any of their default tools. Neither is python.

Based on the way the cosmo project has statically linked builds of python, but not perl, I'm guessing it's more difficult to create a statically linked perl. This means that it's more difficult to put perl on a system where it isn't already there, and that system doesn't have a package manager*, than python or other options.

*or the the user doesn't want to use a package manager. OP said they just want to copy a binary around. Can you do that with perl?

2

OP is on OpenWRT

Fair point - I missed that, buried in the comments as it was.

In that scenario, you use what's available, I guess.

OP said they just want to copy a binary around. Can you do that with perl?

This is linux. Someone will have done it.

1

I don't have python (or even bash) available in all my systems, and so my goto language for script is usually sh (yes, posix sh), which in all honestly is quite frustrating for manipulating data.

Why are you making it hard on yourself? apt/dnf install a language to use and use it.

12
lemmy.world

Only 5 years ago, everybody would be singing and shouting "perl".

Nowadays it is python that has taken this position (even though Perl is still there and can do so much more).

12
Sharenireply
programming.dev

More like

20 years ago - perl

10 years ago - python

Nowadays - go

8
lemmy.ml

Go isn’t a scripting language, and it isn’t a system language either, despite what Wikipedia currently says. To be a system language, a language should support assembly language and shouldn’t require an embedded garbage collector. And if you’re going to make a compiled language anyway, why not make it capable of system work? Go is a platypus that’s popular with devops for some reason—probably Google’s clout in the industry.

2
lemmy.zip

The one thing Go does have going for is performance. It compiles and runs pretty fast. It isn't as fast as C but it is very close.

2
maniiireply
lemmy.world

Even after using CPAN ? I found Perl to be much more "manageable" than Python.

Python with Venv and Pip at least works as expected. Which makes it easier.

1

Python has a clean syntax and is popular. I can't say the same for Perl. Also I'm not sure if Perl is cross platform.

1
lemmy.ml

Why would you say that Perl “can do so much more” than Python? That assertion sounds indefensible.

1

Right, so you’ve got nothing to back it up. Sure, 1990s, let’s get you back to bed.

-5
mub
lemmy.ml

I honestly love Powershell, but haven't tried the Linux version yet. I only use Bash on linux but it has a load of odd quirks that make it unpleasant to use imo. Can't comment on anything else.

12

I use powershell for work as I need the m365 modules for work and its very flexible with decent module availability to plug in all sorts.

However it absolutely sucks for large data handling, anything over 10k rows is just horrendous, I typically work with a few million rows. You can make it work with using .Net to process it within your script but its something to be aware of. Being able to extend with .Net can be extremely useful.

1

another vote for Lua – lua5.4 is available for all 8 Alpine architectures, tiny installed size (120–200 kB) (and Alpine package only installs two files)

12
lemmy.world

Realistically whatever problems you see in python will be there for any other language. Python is the most ubiquitously available thing after bash for a reason.

Also you mentioned provisioning scripts, is that Ansible? If so python is already there, if you mean really just bash scripts I can tell you that does not scale well. Also if you already have some scriptsz what language are they on? Why not write the function there?

Also you're running syncthing on these machines, I don't think python is larger than that (but I might be wrong).

11

Also you mentioned provisioning scripts, is that Ansible? If so python is already there, if you mean really just bash scripts I can tell you that does not scale well. Also if you already have some scriptsz what language are they on? Why not write the function there??

Currently it's mostly nixos, plus a custom thing that generates preconfigured openwrt images that I then deploy manually. I have a mess of other vms and stuff, but I plan to phase out everything and migrate to nixos (except the openwrt stuff, since nixos doesn't run on mips).

I don't really need to run this specific synchthing-ID script except on my PC (I do the provisioning from there), but I have written scripts that run on my router (using busybox sh) and I was wondering if there is a "goto" scripting that I can use everywhere.

1

The smallest footprint for an actual scripting probably will be posix sh - since you already have it ready.

A slightly bigger footprint would be Python or Lua.

If you can drop your requirement for actual scripting and are willing to add a compile step, Go and it's ecosystem is pretty dang powerful and it's really easy to learn for small automation tasks.

Personally, with the requirement of not adding too much space for runtimes, I'd write it in go. You don't need a runtime, you can compile it to a really small zero dependency lib and you have clean and readable code that you can extend, test and maintain easily.

10
feddit.org

No one has mentioned PHP yet? Man, times really have changed.

8

I used it for scripting a decade after everyone else, but even I have to admit PHP is rarely the best choice now.

2

I've actually tried using PHP on OpenWRT and embedded before. It's not exactly lightweight, it's a memory and CPU hog. Keep in mind that the kind of machine that runs OpenWRT might only have 32 or even 16 MB of RAM to work with.

Also, PHP is not the first language that comes to mind when doing data processing and/or functional programming. You can but it doesn't lend itself well to it.

1

Yeah, it's really flexible when it comes to arrays and string processing. Super easy to work with, too.

1
lemmy.sdf.org

I still love bash. I'm able to accomplish quite a lot with it. I vote bash.

8
endoflinereply
lemmy.ca

Bash? On windows it's pretty much unusable even if you use cygwin. Python or Lua are the only reasonable suggestions here

3

Nah, gross. You need to set a bunch of global options to get sane behavior on errors.

Nushell is shaping up really really nicely, and it'll actually stop executing if something fails! Even if that happens in a pipe! And it's not super eager to convert between arrays and strings if you use the wrong cryptic rune.

1

Technically, you could bundle a Perl script with the interpreter on another system using pp and run the packed version on systems with no installed Perl, but at that point you might as well just use a compiled language.

7

Not quite a scripting language, but I highly recommend you check out cosmo for your usecase. Cosmopolitan, and/or Actually Portable Executable (APE for short) is a project to compile a single binary in such a way that is is extremely portable, and that single binary can be copied across multiple operating systems and it will still just run. It supports, windows, linux, mac, and a few BSD's.

https://cosmo.zip/pub/cosmos/bin/ — this is where you can download precompiled binaries of certain things using cosmo.

From my testing, the APE version of python works great, and is only 34 megabytes, + 12 kilobytes for the ape elf interpreter.

In addition to python, cosmopolitan also has precompiled binaries of:

  • Janet 2.5 MB
  • Berry 4.0 MB
  • Python 34 MB
  • Php 11 MB
  • Lua 2.1 MB
  • Bash 5.1 MB

And a few more, like tclsh, zsh, dash or emacs (53 MB), which I'm pretty sure can be used as an emacs lisp intepreter.

And it should be noted these may require the ape elf interpeter, which is 12 kilobytes, or the ape assimilate program, which is 476 kilobytes.

EDIT: It also looks like there is an APE version of perl, and the full executable is 24 MB.

EDIT again: I found even more APE/cosmo binaries:

6

I've looked into this a lot actually. There see many options. I'll highlight the pros and cons of each option.

Lua: extremely lightweight, but standard library is lacking, and doesn't include stuff like map or fold. But that would be easy to fix.

Python: thicc standard library, but is not lightweight by any means. There are modifications made to be more shell like, such as xonsh

Rash: based on scheme, very much functional but if you're not used to lisp style, might take a bit to get used to it. This is actually my favorite option. It has a cli interpreter, and really pleasant to use. Cons is... Well it's not very common

You can honestly use any language. Even most compiled languages have a way to run immediately.

6

If you can, just use Perl. Probably installed on your systems, even the ones without python.

6

So often the right answer, perl. It's a shame that it's so unfashionable these days.

3

Maybe something like Elvish or Nushell could be worth a look. They have a lot of similarities to classic shells like bash, but an improved syntax and more powerful features. Basically something in between bash and Python. Not sure about disk footprint or general availability/portability though

6

A scripting language written in Rust would certainly fulfill you requirement of only needing to copy one file since they are always statically linked and you can even statically compile against musl so it will work on any Linux system without needing a correct libc. Maybe check out rhai.

5

I don't know if it matches your desire for easy install of small disk space, but it might make up for it in other arenas - Ruby is my new-found love when making simple scripts. Being able to mostly emulate the shell integration that bash has by just using backticks to call a shell command is the killer feature in my book.

5

(Chicken) Scheme.

Schemes have one of the best and most interactive interpreters Is general purpose, allows functional, procedural and OO programming, small disk size and compilable to native executables, Throughout documented and supplemented by years of research, simple setup.

It also is CGI compatible, if necessary.

4

Neither of them seem to be a single file, and both seem to have several dependencies, at least that’s the case with the Homebrew versions.

1
lemmy.ml

JavaScript through Node.js, or TypeScript through Deno if you like typed languages. They both check all your boxes (just check the size of the executables to make sure that it's what you would consider "small footprint").

Both languages and runtimes are quite popular, so you will find any answers on StackOverflow.

They are both single-executable with little dependencies, and Deno can also compile your scripts to self-contained executables.

As a bonus, both support the vast and extensive NPM package repository where you can find all sort of libraries for even the most complex tasks.

And they work with your favourite IDE or editor, not just syntax highlighting, but also contextual suggestions.

4
gompreply
lemmy.ml

Installing node uses some 60MB (according to zypper on my current desktop). I'd rather have something small and possibly that consists of a single executable.

As a bonus, both support the vast and extensive NPM package repository

That's not necessarily a feature :) Package repos are great if you are a developer (I am one) working primarily with that language, but are frustrating if you just want to run things.

4

I thought so. Although almost nothing for modern standards, 60MB is not exactly tiny. Sorry about that.

On a different note, a repository is always a good thing imho. If you'd rather not have to worry about the dependency-pull step you can always include the dependencies with your sources, or just limit your code to using features included in the standard library.

1

As a Java developer, and someone who never learned Python or other scripting languages, Node is my go-to scripting language. I've only come around to it for that in the past year or two. But it's great.

2

Python.

Just remember to use pyenv for interpreter installation, version and environment management. It's pretty straightforward that way and you have predictability.

Don't ever manually fiddle with the system python and/or libraries or you'll break your system. You should just rely on the package manager for that.

4
lemmy.ml

Tried bash, Make, and awk/sed. All hit brick walls. Finally landed on pyinvoke. Two dependencies to install on any new machine. Never had problems. Also, easy to debug and modify as projects evolve.

4

If you are interested in tiny lisp like languages, this gitlab could be of interest to you.

Full disclaimer, I came across it a few years back as I am the maintainer of arkscript (which will get faster and better in the v4, so that data about it there is accurate as a baseline of « it should be at least this good but can be even better now »).

4
fedia.io

Perl or python for things likely to already be there. Maybe ruby or PHP if you must. I used to work in groovy a lot but I think it requires the JVM

4
atzanteolreply
sh.itjust.works

Perl requres the perl interpreter and python requires a python interpreter. Why is it bad that groovy also needs a vm?

-3
nyanreply
sh.itjust.works

Last I checked, the JVM was larger than the standard Perl and Python interpreters, and had a much worse startup time (which is bad for short scripts).

3
atzanteolreply
sh.itjust.works

Sorry - what potato from the '90s are you running software on these days?

-3

Dude, I've worked with all these interpreters. The JVM's startup is slow, although there's been some improvement since the version 1.1 that I started out with. The interpreter speed is quite good once it gets going, so it remains a good choice for long-running or interactive programs where the startup is insignificant as a fraction of total program run-time, but if you're running a script that takes only a fraction of a second to execute, the JVM's startup can lengthen the time by an order of magnitude or more. Horses for courses and all that—I wouldn't write a complex interactive GUI program in Perl, either.

1
Dingalingreply
lemmy.ml

Perl's core to most distros and will be there already. Python isn't and can be quite heavy - plus some of are are still smarting over the major version change breaking everything and the need for complicated environments.

3
atzanteolreply
sh.itjust.works

"heavy". It's 2024. We have gigabytes of storage on handheld phones. I don't buy it.

-2
Dingalingreply
lemmy.ml

I work at scale - deploying scripts to hundreds of linux machines and any package you install will be multiplied that many times on the backend storage. You don't get the luxury of installing anything that isn't essential.

1
atzanteolreply
sh.itjust.works

Ohhh, you work at scale do you? With... "hundreds" of Linux servers? And 50MB, which is basically a rounding error for most storage solutions, will break your bank? Sure Jan.

-1

Probably. But I'm not the one whining over like 5GB of data across "hundreds of systems" in 2024.

-1
lemmy.ml

I would go with Guile, because it is built-in to the Guix Package Manager which is a really good general-purpose package manager.

It ticks several of your boxes:

  • has a CLI interpreter
  • is a general purpose language, Scheme, amd compliant with revisions 5, 6, and 7 of the language standard
  • allows writing in a functional style (it is one of the original functional programming languages)
  • small disk footprint, but still large enough to be "batteries included"
  • decent documentation, especially if you use Emacs
  • simple setup: not so much, unless you are using Guix to begin with. The standard distribution ships with lots of pre-built bytecode files, you need an installer script to install everything.

It also has pretty good libraries for system maintenance and reporting:

4

Since you like guile, I would recommend you checkout rash (search "rash shell language" on Google. Sorry too lazy to link it).

It is based on racket, but made to be shell-like, and is very nice. I believe guile used to have a similar project that isn't maintained anymore.

2
gompreply
lemmy.ml

nim is great, but it is >200mb (plus AFAIK it is compiled... does it also have an interpreter?)

2

The part where it's compiled is what makes it have no dependencies to actually execute

2

Perl would be my candidate for more advanced text handling than what sh can do.

Never used Lua but I think it's fun.

If nothing else works, just learn C/Rust. There's plenty of that on Linux systems, I think you'll be able to manage. Yes, it doesn't meet a lot of your requirements.

3

Kotlin script is fantastic! I wish it would become more popular. Dependency support, cached complier output, etc. I really like it for non-trivial scripting since you didn't need a venv for dependencies.

OP is being ridiculous about space requirements. 60MB is a rounding error these days.

3

@gomp I like TypeScript.

I used Python for 15 years or so until they changed from v2 to v3. At that point I realised I couldn't understand my old code because it lacked types, so I got discouraged with that. So rather than learn v3 I stopped using it.

Perl is a disaster. sh is good for shell scripts but let's not stretch it.

TypeScript can use all the JS libraries and runs on node which is supported by all sorts of platforms. Yes there are a few holes in the type safety, so don't do that.

The internet is full of "how to do X in JS". You can read them and add the types you need.

3

It is possible to wrap something like python into a single file, which is extracted (using standard shell tools) into a tmpdir at runtime.

You might also consider languages that can compile to static binaries - something like nim (python like syntax), although you could also make use of nimscript. Imagine nimscript as your own extensible interpreter.

Similarly, golang has some extensible scripting languages like https://github.com/traefik/yaegi - go has the advantage of easy cross compiling if you need to support different machine architectures.

2

Bro seriously just slap pyenv + pyenv-virtualenv on your systems and you’re good to go. They’re absolutely trivial to install. Iirc the latter is not a thing in windows, but if you’re stuck on windows for some reason and doing any serious scripting, you should be using WSL anyways.

2
gompreply
lemmy.ml

I fear I am not enough reverse (or Polish, for that matter) :)

Anyway, I have great esteem for you (if you actually use forth and are not just trolling)

1

@gomp Well no I know someone who does forth, not me really. Perhaps forth is just too low-level for anything except hardware drivers and so.

1

Not sure how big node footprint is but would fit the bill. Would only recommend if you wanna go into web dev career in the future tho 🙃

1
lemmy.ml

perl might be on all your systems. It’s kind-of a legacy, but still actively developed. It’s not a great language: it looks like bash scripting on steroids. But if you just need to write some small scripts with a language more powerful than awk or bash, it does the job. If perl isn’t on all of your systems already, then I would choose a better scripting language.

1
Findmysecreply
infosec.pub

TBH I don't even use awk that much, even that is plenty powerful for my needs. Perl absolutely blows my mind with how needlessly complex I can make stuff with it

2

Everyone always dunkin' on Perl, but I can't even tell you how often it's been the best tool for the job. Like, at least 3

3

vlang might fit your request pretty nicely. It's a bit patchy in places but mainly stable and gets pretty frequent updates

1

Looking through the packages available for OpenWRT I would suggest Tcl, Lua, Erlang or Scheme (the latter is available through the Chicken interpreter). Try them out, see what you like.

1

My go to for most of what you mention is Go, but that’s obviously a compiled language and not for scripting. Or is it - What do you think about https://github.com/traefik/yaegi, which provides an interpreter and REPL for Go? It would let you use a performant and well documented language in a more portable scripting way, but not preclude you from generating statically linked binaries if and when that’s convenient.

-1

Powershell is the superior choice. But if you can't even have bash you probably won't have access.

-3