Hello everyone!

TL;DR: I want to propose a community-driven effort to research and improve 1-bit LLM models, for use within and outside the Horde. I think having access to such models would be very useful for the overall project, as you do not need a lot of compute to run "bigger" models if they're compressed well. Relevant paper.

We currently live in very interesting times regarding AI development. The big companies in the US seem to still be ahead, but groups from China, working on open-weights models, are making very impressive strides. However, the focus has been and still is on developing really big models. Most of the impressive models that keep coming out are bigger and bigger, leaving most people to pay for API tokens if they want something useful. Probably this is also incentivised, as Nvidia wants to make more money from sales to data centers.

I have been keeping an eye out for the AI Horde project, and I always wanted to help out, but never got around to setting it properly up due to my AMD GPU and running Windows. However, really cool project, and I wanted to congratulate everyone involved!

In my opinion, the goal of the AI Horde community also positions it as one of the very few that are capable of bringing forward some LLMs that can be "smart" on consumer-level hardware. That is, getting models that can handle long-horizon tasks well without paying providers for API access.

Since the first paper on 1.68-bit LLMs, there has been quite a few other suggestions thrown around (recent example). No one really uses these models much, as the quality is seriously degraded. Similarly, no one is really trying to improve these models further, as there is no incentive to do so when you can just pay someone 3$ per million token output from an existing open-weight model. For example, to the best of my knowledge, no one has tried to introduce latent reasoning (example) in this context, or specifically training/fine-tuning models at 1-bit levels.

So, to get to the point, would it make sense to get some community-driven research in this area? I believe that we could all pool together compute, good training data, ideas for fine-tuning / RL-training, etc. If it works out, we could have a method that makes existing larger models (say, up to 200B) available on a single 24GB GPU.

First thing I would try is to expand on the recent NanoQuant paper:

1. Wait for weights to be released.
2. If no weights come out, quantize a Qwen 3 32B, and try a more diverse dataset, with more tokens to see if fidelity can improve. I could get some access to GPUs for this myself. Another option for 1-bit models would be using other existing ones (e.g., Unsloth), but performance degradation is much bigger in those versions, from what I have seen. Furthermore, the compression of these models is not as efficient, and you would not be able to fully run a 70B parameter (as described in the paper), with only 8GB VRAM.
3. Get an LLM (or human volunteers) to determine behaviour on various tasks: find limitations, strengths, etc. Get some human preferences for RLHF, or use a bigger LLM to grade output quality. Preference here on logic tasks.
4. Perform fine-tuning of 1-bit model based on the gathered data, and deploy for use. Return to step 2 after a while.

Fine-tuning the 1-bit model might get a bit hairy, as the binary operation are not differentiable. We also wouldn't be able to up-cast back to F32 for regular training, as this would completely invalidate the consumer-driven access to these models. Simplest idea would be to train a LoRA head, or do some stochastic-driven training (e.g., flipping bits). However, the latter would probably be very unstable, and not work out, and LoRA might be the only option. I'm not a mathematician, so I am open to suggestions here :)

Past the initial prototype, I would consider the following stuff that's already implemented in other quants:

• Try to re-calibrate the LayerNorm / RMSNorm parameters based on the original model's activations.
• Perform some regular KL-divergence distillation
• Other than using an LLM as a judge for RL, perhaps one could also fine-tune using a semantic similarity metric while aligning with the output of the original model. This could ensure that the intent is the same, even if the style differs.
• Depending on token complexity, look into reducing compression just for difficult tokens and compressing further for easy ones (à la FlexQuant)
• Mixture-of-Experts-style quantization, where we increase compression for experts that are not important, and reduce it for higher-frequency ones.

Curious what everyone thinks!

Some great news regarding the development of computer chips in the EU. However, the total of €240 million allocated to the project is not as much as I would say we need to invest in this area. Let's see how things change in the next few years!

Previously used link: https://archive.ph/ICJZZ

Link to petition

Until now, the EU has allowed a majority of countries to rely on American big tech companies for communication and storage of sensitive data. For example, many universities across Europe rely on Google or Microsoft for email services, research data storage, and department communication. Similarly, many of them write their research using Microsoft Word, which could be used by these big companies to train their own AI models.

A majority of regular citizens rely on Meta for instant messaging apps (WhatsApp), Facebook, Instagram, but also on X (formerly Twitter), and TikTok. None of these apps are properly regulated even with EU's efforts, leaving people unshielded to other states' attempts at polarization. There is also the problem of mass profiling of users, which is used to supply targeted advertisements and sometimes influence public opinion on certain topics (cough Musk tweaking the Twitter algorithm to promote AfD cough).

The article that I supplied focuses mainly on the aspect of maintaining data privacy when our data is harvested by outside entities. However, this is, in my opinion, a horrible approach. We need to move everything ASAP to open source alternatives, and preferably EU based ones. Some attempts at this have been previously made in Germany, which should give hope to other countries in the EU.

The cost of moving away from Google/Microsoft tech stacks will be a drop in the bucket compared to the wealth that these companies extract from EU. Similarly, offering alternatives to social media like Friendica, Mastodon, Pixelfed, Lemmy, and perhaps PeerTube, would be a huge win against disinformation and propaganda from other countries. We should also push for instant messaging platforms like SimpleX that do not rely on Google's proprietary Push Notification services, and perhaps deGoogled Android devices.

If the recent events are not a catalyst to push everyone away from US software in the EU, I do not know what else will. Do you think that this would be possible at all?

I have never donated money in my life before, but what happened yesterday really upset me. Ended up sending some money this morning. I know that my small donation won't contribute to much, but I am trying to help :D

I hope this post doesn't break any rules!

I recently discovered that Redox OS got a new release earlier this month. I'm quite surprised how far they managed to get, given that only a handful of people are working on this project (compared to the Linux kernel).

Now, I'm curious what it would take to get bigger players to focus on this project. Given the recent Linux + Rust drama, it would surprise me if the backers of Rust for Linux would not give this project some attention.

Hello everyone! I've been playing around with Wayland for a bit and was hoping to start learning some more about it. For example, I would be interested in making a lock screen, similar to Swaylock, as a toy project.

What GUI toolkit would you use to develop apps on Wayland? I've added a little poll below with some of the popular choices I've seen thrown around. Feel free to add your own suggestions and maybe leave a comment as to why you'd use that!

Link to poll

Hi! I am trying to automate my install process by creating a json file that can be used by archinstall (example). One of the example shows how you can run custom commands to get paru (yay, but written in Rust):

"custom-commands": [
        "cd /home/devel; git clone https://aur.archlinux.org/paru.git",
        "chown -R devel:devel /home/devel/paru",
    ]

However, their example doesn't provide any further information about installing packages with paru. I would like to install some stuff just for my user.

My idea was the following:

• using archinstall, install everything according to the config
• disregard the "custom-commands" option in the config and create a separate custom script
• get all the users from the system and allow user to choose which one to chroot as
• run all commands as the chosen user ( e.g., install Rust with curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh )

I need to install a few packages that are not in the official repository, as well as moving my dotfiles in /home/user/.config and making sure everything is accessible by that user. If there are any better approaches to this, I would be glad to hear them!

An example of the script I am planning to use after running archinstall:

::: spoiler spoiler

#!/bin/bash

# Find all users on the system
for user in $(ls /home); do
    if [ "$user" != "lost+found" ]; then
        users+=($user)
    fi
done

# If there is more than one user, ask which user to install for
if [ ${#users[@]} -gt 1 ]; then
    echo "Multiple users found on system. Please select a user to install for:"
    select user in "${users[@]}"; do
        if [[ " ${users[@]} " =~ " ${user} " ]]; then
            break
        else
            echo "Invalid selection"
        fi
    done
else
    user=${users[0]}
fi

echo "Installing for user $user"

# chroot as the user
arch-chroot -u $user /mnt/archinstall # This only opens bash, but I am working on it :D 
cd /home/$user

# Install paru
git clone https://aur.archlinux.org/paru.git
cd paru
makepkg -si

# Install stuff with paru
paru -S tlrc --noconfirm

:::

Server performance is not very good with so many mods, and I have been looking into ways to fix this. One of the latest comments on the ATM8 page on CurseForge is from XZot1K, and says the following:

After lots of testing I resolved most of my issues by installing the following mods to the server (Ensure to install the correct versions, as of writing this the version is latest of each for 1.19.2):

https://www.curseforge.com/minecraft/mc-mods/too-fast

https://www.curseforge.com/minecraft/mc-mods/smooth-chunk-save

https://www.curseforge.com/minecraft/mc-mods/chunk-sending-forge-fabric

https://www.curseforge.com/minecraft/mc-mods/packet-size-doubler

These mods will resolve larger packet disconnect issues, chunk lag, and irregular movement rubber banding.

In addition to these, for further improvement, set the tick rate to -1 in the server.properties file.

Paste the following into the bottom of your "user_jvm_args.txt" (change the 6GB and 256m to your liking --- Xms must be less than Xmx):

-Xmx6G -Xms256m -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=32M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true

Please note that while these additional mods do work on the client the major improvement comes from the server-side.

I've already used those jvm arguments, but I didn't look for performance mods before. Now, after fiddling a bit around with them, the server feels much snappier (and I don't have to install anything client side)! I'm hosting on Azure, with a Standard D2s v3 (2 vcpus, 8 GiB memory) VM, and when I would do a /home from a far away place it would take a few seconds to load. Now, it's almost instantaneous! Thanks XZot1K! :)

The server also used to crash whenever multiple people entered the Nether, but I haven't been able to test this yet with the new configuration.

If you have any tips to improve performance, please share them here :)

As the title implies, should I do it? I love Arch so far, and I can fix most issues that pop out. However, I sometimes wish to start fresh without too much hassle, but I get a feeling NixOS isn't as mature as Arch.

Have any of you used both, and if so, what do you miss from Arch? What are you grateful for in NixOS?