> Starting in the first week of 2024, the FreeBSD boot process suddenly got about 3x slower. I started bisecting commits, and tracked it down to... a commit which increased the root disk size from 5 GB to 6 GB. Why? Well, I reached out to some of my friends at Amazon, and it turned out that the answer was somewhere between "magic" and "you really don't want to know"; but the important part for me was that increasing the root disk size to 8 GB restored performance to earlier levels.
(I'm currently using 1GB snapshots, because my actual disk image is a tiny fraction of that size. But if bumping that to 2GB or 4GB would make it faster, that's a small price to pay.)
Do you have any other wisdom regarding mysterious reasons for fast or slow booting? EC2's boot process is deeply opaque, and any insight at all is better than nothing.
Less puppet/chef
https://aws.amazon.com/blogs/aws/amazon_s3/
I do not know if this has anything to do with the cliff that you saw.
But yes, I built a lot of AMIs. And launched new EC2 instances for each of them -- it wasn't just a matter of rebooting since the first time an AMI launches there's different behaviour (both from FreeBSD, e.g. growing the root disk, and from EC2, e.g. disk caching).
It's also now a first-class supported cross-compilation target, including when linking libc, so you can do stuff like `zig cc -o hello hello.c -target riscv64-freebsd`.
And then of course if you have any C/C++ dependencies, you can fetch and build them with the zig build system, so it should be possible to easily cross-compile even quite complex projects for FreeBSD now.
Hopefully that helps more projects decide to add FreeBSD support and respective testing to their CI!
you can find the project laptop here https://github.com/FreeBSDFoundation/proj-laptop
Don’t know how he manages all of this + Tarsnap.
To be fair, some of the time I spent on this came away from Tarsnap. But less than you might imagine.
When it comes to drywall, always hire a professional. Learn from other's mistakes... it's not as easy as you think and it won't turn out well.
The other day I had the opportunity to get a 10% discount on a fridge if I could pay the whole thing in one payment. If I didn't have the money I wouldn't get the discount, so in a way being poor means everything is more expensive.
All poor people know this in their bones because they face this every day of their lives.
Regardless, it's quite unfortunate to see Colin's nits picked in this manner, dredging up some mistake from almost 15 years ago (which he handled as responsibly as could be expected), given all of the work he's done on FreeBSD and for giving the world scrypt.
Ironically that bug happened because of scrypt. Creating scrypt led me to refactor Tarsnap's crypto code, which is when the bug slipped in.
Amazon isn't even on FreeBSD sponsors [1]. And Google only sponsored $9K last year. Apple isn't there. Edit: And Credit to Microsoft being at least on the list! And forgot to mention Meta / Facebook missing from it as well.
I would have expect them to sponsor FreeBSD and OpenBSD annually by default given they use and continue to benefits the work out of both.
[1] https://freebsdfoundation.org/our-donors/donors/?donationYea...
First, it presents the snapshot of donations within a given year to the Foundation. The history of donations is not represented by definition.
Second, it does not present contributed development. Those are typically summarily available on the release notes of each release [1].
Another gift of polarized politics I guess.
I never thought about this before but is it possible that there was an employee who donated, and the $500-$999 was just the portion that Apple gave as the employee charity match? It would make a lot of sense, but it does make the donation look kind of hilarious.
It's a small price to pay and it stops me having to install less stable operating systems.
I favored FreeBSD until around 2021 when computers with different CPU mixed together started to become common. I first bought a RockPro64 with 2 big and 4 little cores and then an Intel Alder lake. As far as I understand FreeBSD scheduler to this day don't know how to properly play with these so it brings the system to the lowest denominator of the slower cores.
I would love to know who is using FreeBSD in EC2.
I'm genuinely actually curious. FreeBSD exists in kind of a shadow realm for me where I've never been quite able to pin down the soul that keeps it chugging, but I know it exists somewhere in there.
ZFS was not leveraged that much but it saved our beacon once when a table in the production database was accidentally dropped and we could instantly rollback to the previous zfs snapshot (there was a tiny bit of data loss as a result but this did not matter too much for this application - uptime was more important). ZFS was also used for backups I believe.
A few times I used dtrace in production to troubleshoot.
When we introduced Linux to our fleet of FreeBSD servers, every team picked a different distro organically so it was a bit of a zoo. With FreeBSD on the server you only have the one variant.
I still use and like both, but I must say I really like that FreeBSD is a kernel+OS integrated together.
It really makes sense to think of different Linux distros as different operating systems. At the very least, the ones from different "families".
There are a lot of differences between Debian and RHEL. Suse, Alpine, Void, or Chimera Linux are completely different again. In some ways, they are almost as different from each other as FreeBSD is from them.
Compared to that "zoo", using FreeBSD everywhere is far more cohesive. But if you use RHEL, Alma, Rocky, and even Fedora, things are still going to feel pretty consistent. Or Debian, LMDE, and Kali. I am not advocating an ecosystem.
My experience with FreeBSD is that it provides a nice balance of the concerns OpenBSD and NetBSD specifically address. Historically, FreeBSD prioritized Intel CPU's (where NetBSD had greater portability) and had solid security (where OpenBSD had more of a focus on it).
The FreeBSD ZFS support really is a game changer. I believe Nvidia only recently has had native FreeBSD drivers - for a long time FreeBSD's kernel Linux support was required.
> I'm genuinely actually curious. FreeBSD exists in kind of a shadow realm for me where I've never been quite able to pin down the soul that keeps it chugging, but I know it exists somewhere in there.
Again, for me, FreeBSD has proven to be a nice blend of the features other BSD's provide as well as being incredibly stable on the h/w platforms I tend to use.
When I first started using FreeBSD, in 2004, Nvidia had native FreeBSD drivers for all their boards.
All of the BSDs tend to have a lot less churn, for better and worse; so IMHO, they make a nicer platform to integrate on.
Could you do that work with Linux? Probably --- but nobody who does is talking about it as much.
This kind of high throughput service has been a FreeBSD niche since forever too. Walnut Creek CDROM, Inc ran what was reportedly the world's busiest ftp site, ftp.cdrom.com on FreeBSD in the early days of the internet.
Yahoo ran on FreeBSD (I worked there 2004-2011) WhatsApp ran on FreeBSD (I worked there 2011-2019) Both were leaving FreeBSD when I left, but sadly, I didn't leave to work somewhere else with FreeBSD :p
FreeBSD has better ZFS support than Linux, because it doesn't have the licensing issues.
Its software catalog is also much bigger. It's a viable modern desktop daily driver and I can't say that for the other two.
As to why not Linux? I don't want Linux. It's too bogged down by corporate interests.
This is a rather funny statement because at various points, high level execs at Apple Computer (and on another occasion Sun Micro) invited Linus Torvalds out to lunch and pitched teaming up together to take on Microsoft. Linus turned them down.
Then a little bit later Jordan Hubbard announces FreeBSD would be the UNIX layer of OS X.
A few companies do. Skype and Netflix did but hardly use it now (at least Skype left it, not sure about Netflix but I never hear about it from bsd devs). Ix systems and netgate do but they're tiny.. No, it's not influenced in a trivial way and certainly not by apple.
This is a huge difference to Linux where the vast majority of kernel commits come from big tech and have nothing to do with things end users care about. Also there's nothing in the FreeBSD world like the Linux Foundation which is basically a corporate lobby group.
I understand the former. But with how Apple operates, it's really hard to believe they'd pull downstream from something they don't have some kind of soft power over. They do still pull downstream AFAIK? Maybe that's changed?
>Ix systems
I did some reading and saw a FreeBSD contributor ended up going to Apple until 2013 before he founded this company. https://www.ixsystems.com/clients/ Apple is listed here. Six degrees of separation and all, but probably not a coincidence. Nothing wrong with that, business is a social structure. This is how they work. We make and keep friends, even if only professionally. Backchannels are where real deals are made. But this to me is not nothing. No corporate influence means there's a lot of nice things you don't get. You just can't afford the manpower. It looks more like 9 Front than a BSD that has some serious billion-dollar problems under its belt.
That sounds harsh, not a judgement. Just very deep skepticism of the assertion of no influence. I'm realizing there's not a lot that can be done to sway that intentionally.
> This is a huge difference to Linux
This I'm well aware of. I just like having a perspective across the fence. These days they're starting to get a little too aggressive for my tastes. FreeBSD seems fine in comparison.
Apple doesn't merge often. They basically haven't merged kernel tcp since 2002. When I started using OSX in 2011, they hadn't merged userland for several years, and when I stopped in 2019, they had only merged once.
They famously stopped picking up bash when upstream changed the license, and most of the FreeBSD userland doesn't change that frequently, so most things you wouldn't notice a difference. cal(1) started highlighting the current day at some point, tar probably grew new compresion arguments, etc.
Apple certainly was a major contributor/driving force/etc of LLVM for a while, not sure if they still are? And LLVM was adopted by FreeBSD, so maybe that's where this idea is coming from?
Partially, but after seeing the Jordan Hubbard connection, there's a lot of layers to this. May have reinforced my biases, but it's definitely non-trivial according to my hippie-tier anarchist baseline. Oops. Worst case scenario of answering your own question.
But your reply does give me actually contradicting evidence. It wouldn't surprise me that distance has grown to the point of total atrophy, given the general trajectory Apple has been on since 2012 or so. This is why I ask these questions, because the people on the ground give the most informative answers.
As Ptahhotep advises circa ~2300BCE:
> Fine words are more sought after than greenstone, but can be found with the women at the grindstone.
The definition of 'trivial' would come into play yes. I would only consider it non-trivial if a commercial party can (and does) influence the direction of development. I don't think Apple does so. Even Netflix. In the Linux world there's billions of investment and many contributors are directly employed by big business. The waters are much murkier there.
Again, I'm not saying it's a bad thing. It's just not something I want which is one of the reasons I picked FreeBSD. Other reasons were the great ports collection, the division between OS and apps (you can have rolling apps but a stable OS), the traditionalism (only change things if it's really needed) and the single main flavour of the OS which makes support much easier. Also the excellent documentation.
Yes that is the flipside. But I don't mind that. If you choose your hardware carefully it works fine.
Note that this is not too different from using Windows or Mac. Your hardware is also chosen carefully to work with those, just not by you but by the vendor. With FreeBSD you're more involved with the nuts & bolts and this is exactly what I want. I don't want my OS to be a black box I don't understand.
Unless I misunderstood you, Netflix delivers all video content via FreeBSD and contributes code back and money to the foundation.
If they still use it they are more of a user than an influencer in its development.
Apple has no influence over the FreeBSD project.
> I know the kernels are different and obviously only part of the userspace is the same, but is FreeBSD actually far enough away from Apple to say it's not bogged down by corporate interests?
Yes.
OS-X (now macOS) is based on XNU[0], which itself has roots in the Mach[1] microkernel. The Unix user-space programs distributed with OS-X/macOS are those found in FreeBSD distributions AFAIK. This is also conformant with FreeBSD licenses for same.
So there is no "soft power" Apple has over FreeBSD. And FreeBSD is not "Apple's shadow in FOSS".
> I don't imagine it's the same as Linux at all, but it exists in a non-trivial way, no?
No. It does not.
EDIT: Just in case you'd like to verify any of the above yourself, see here[2].
0 - https://en.wikipedia.org/wiki/XNU
The only thing I've ever heard from FreeBSD-land, not paying attention to users, but the maintainers and the tools. Apple comes up. In the same manner that RedHat and others come up for Linux. How to explain? It's an abstract pattern. Transparent, understandable.
I mentioned somewhere about the connection through ix systems. And honestly to project, if I was a maintainer of something used between Netflix and Apple, I'd prioritize Apple. Apple has outlived IBM. If you know your history, you know how serious that is. If you've got authority over something as large as FreeBSD? Yeah, you don't ignore that kind of actual power especially when it's personal. Like I say, all based on guesses. But some things are hard to mistake.
As far as "power" is concerned... speaking as release engineer, I don't give special treatment to anyone; nor have I even been asked to. If anyone has a special relationship it's Netflix but if anything that's the opposite way around: "Can you please throw 10% of all Internet traffic at this TCP stack patch and let us know if anything breaks" is a thing. They're incredibly helpful with Q/A.
I'm sure if Apple wants something it would be considered but there would be a strong validation of "what's in it for us" on the freebsd side. There's also some pretty bad experiences with corporate influence and this is reviewed a lot more independently since the netgate wireguard disaster. https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice...
Unlike in the Linux world where RedHat and canonical are so embedded due to most of the devs working for them that there will be a lot less questions. And not just those two, also companies like Huawei are heavy kernel contributors.
I'm not saying it's bad to have such commercial influence. But it's not what I want for the OS I run.
As to why I use it over the various systemd free linux distros? Well, there's a couple things. First lot of those distros, like Artix linux say, actually have smaller communities than FreeBSD(I'm guesstimating based on the activity level in their irc channels). The Linux community might be much, much larger than the FreeBSD community, but it's also extremely fragmented.
triggerwarning, hyperbole incoming. Don't bother correcting me, it's a polemic, not a scientific paper
Secondly, for someone like me, who's been using various unix like OSes for two decades, FreeBSD is just a nice, batteries included, well integrated system. Things like jails, Dtrace, ZFS, Bhyve, pf etc. All being in the base install means they're just better integrated with the kernel, and eachother. Most of those things exist for linux, or have equivalents, but they're not all part of the same project. Obviously Dtrace and ZFS originated in Solaris, but they've been made first-class citizens. There's a harmony to FreeBSD that Linux distros lack. Documentation is also very good, all accessible via manpages(no GNU INFO...). And, as I mentioned briefly before. It doesn't have a lot of the cruft that's been added to linux distros over the years(though some of it is available in ports if you want it). In FreeBSD, my experience is actually useful. Things I remember how to do from 5 years ago, 10 years ago, 15 years ago, still work. If I'm on some modern, plug and play linux distro, I have no idea what's going on under the hood any more. All I know is it's not what was going on 5 years ago, which isn't what was going on 10 years ago, which isn't what was going on 15 years ago. The amount of pointless churn going on in the linux space is ridiculous. When I started using linux, what I loved about it was that it was transparent. I could change anything. The system was easy to understand. Yes, it was janky, but it was understandable jank, whereas Windows was janky in an opaque way. 20 years later, Linux is still janky, but nothing is understandable, at least not to my greybeard brain. Systemd takes over a new daemon every distro upgrade. DNS resolving now involves 4 different daemons with 15 different configuration files, there's two display protocols, both broken in different ways, /etc is full of long files written in strange, alien languages, and every file has its own bespoke language. There seems to be 54 different ways to make any change to your system, and all of them are somehow unsatisfactory in a unique way. I just can't, anymore. Enough already.
The mains issues with Linux is it’s just the kernel, and anything is developed in their corner without taking account of the rest. Also, I tend to think the Linux folk in general seem to want to reinvent the wheel every 6 months, where FreeBSD and BSD in general have tendency to make things better from previous work in comparison
But for me the fact Linux is just the kernel doesn’t make the previous criticisms invalid. The first concerning the development of the different components in sort of echo chamber where no one seem to communicate with each other is directly taken from the Linux Kernel philosophy, the maintainer have expressed in multiple time they don’t care what happen outside of the kernel, in contrast with FreeBSD developers for example
The second point is more towards distribution I admit
Linus has a pretty firm hand on the tiller of Linux evolution. I counter "don't care what happen outside of the kernel" with his many, many public "never, ever break userland" rants. And many kernel devs and maintainers are employees of companies like Intel, Red Hat, Google, IBM, and AMD that absolutely care about coordinating kernel dev with the bigger picture.
Something like 250 devs contribute to FreeBSD each year. For just the Linux kernel, the number is closer to 5000. There are just way more people working on way more stuff. It is not a surprise to see a more significant halo of chaos around Linux. Coordinating the Linux kernel is herding cats and, even when everybody eventually lines up, there are going to be periods where it seems like everybody is talking past each other.
And while the Linux kernel does have a "release early, release often" mantra, it also touts "trust but verify" and has a strong meritocracy and hierarchy. So I am not sure "no one seem to communicate with each other" is fair. Not just anybody can drop whatever they want into Linux. We also need to remember that shipping the Linux kernel is not the same as shipping a Linux distro (operating system). Actual Linux distros bring kernel versions in according to the philosophy of the distro. Many are very stable and conservative. Others are a whole lot less so (but that is users choice).
I hear this a lot when people talk about FreeBSD but I am not sure about it.
A LOT of the core Linux ecosystem comes from Red Hat developers for example. If I look at RHEL as an operating system, they have a definite vision for the OS, they take a long-term view, and they invest in development to get it there. My guess is that Red Hat alone employs more devs than work on FreeBSD.
Red Hat contributes heavily to the kernel, the core C library (glibc), the userland (GNU utils), the system supervisor (systemd), the compiler (GCC), the desktop environment (GNOME), the GUI framework (Wayland now, Mesa, etc), the sound system (pipewire), the hypervisor system (KVM, libvirt), and the container system (podman and Flatpak). Red Hat heavily influences the direction of all this stuff with a common vision and they work to implement it as a cohesive expression in their distro. This is a broader swath of what makes the operating system than FreeBSD considers its scope and it is all built to work together.
If you use RHEL, you know it is very stable (static). When Red Hat makes changes, they tell you about them years in advance.
I honestly do not think you can say that FreeBSD is more cohesively developed or better documented than RHEL. FreeBSD arguably has less control over key aspects of the OS than Red Hat does.
I am not advocating for Red Hat here by the way. I am not even a RHEL user. I use Chimera Linux which rejects quite a lot of the Red Hat vision including SystemD and pretty much the whole GNU system (userland, glibc, gcc).
My point is that Red Hat is truly a maker of their own destiny and their distro reflects their vision. They want to move to SystemD. They introduced DRM and KMS instead of the traditional Xorg driver model. They want to move to Wayland. They have heavily embraced the OCI container model. It is all part of their vision and design.
Pragmatically, FreeBSD has to create tools like Linuxulator. FreeBSD is adding support for OCI containers. FreeBSD is adding Wayland support and, as popular desktop environments abandon X11, may have to move to Wayland as the preferred display server. Even the FreeBSD utils have added many options over the years to be compatible with the userland that Red Hat developed. Was 'ls --color=auto' a FreeBSD design? In other words, the Red Hat agenda drives the evolution of FreeBSD (but not much the other way around).
So sure, FreeBSD is more stable and cohesive than the universe of Linux distros. But even BSD has fragmentation. GhostBSD is close to FreeBSD but not quite and would be more different if they had more devs. DragonFly BSD certainly has its own agenda (and again, is held back more by bandwidth than solidarity). The free-for-all in the Linux world is an expression of its size and collective innovation. But how much of this you want as a user is up to you. As many have said, you don't use "Linux", you use a Linux distro.
Again, my main distro is Chimera Linux. The whole point of the name is that it pulls together things never designed to work together (including the FreeBSD userland on Linux). And yet, the Chimera Linux dev team has a very strong vision of what they want their OS to look like and they work very hard to build that into a cohesive implementation. This includes keeping the system and the code small and understandable. It is a goal that you can sanely build the entire system from the ground up. That is why Chimera uses a BSD userland and does not use SystemD. But while they want to keep things simple, they also want "modern" features.
They choose components that fit their vision. Where changes are required, they make them. Where they deem good options not to exist, they invent them (eg. Turnstile, cports). As a user, I get that "solid, cohesive, well-designed, intentional, and heavily curated" experience that FreeBSD users talk about. More to the comment above, Chimera reeks of "looking to preserve tradition while striving to make things better". Of course, it is also still a niche distro with a tiny community (at this point). As somebody said above, FreeBSD may be a better choice for this and other reasons. But Chimera Linux is still Linux and that has its advantages. The box I am typing on uses bcachefs and Distrobox. For me, it is perfect.
Anyway, I apologies for the length. When you talk about FreeBSD vs "Linux", you really have to choose a specific Linux distro for the comparison to be meaningful. Depending on which one you pick, the statements made by @MrArthegnor may or may not hold. At least, that is my view.
You can choose a Linux distro that reflects your own preferences in terms of pace of innovation. Sure Arch has 100 package updates a day and 30 ways to do everything. However, RHEL (or its compatibles) is not that way. You can go 10 years without changing your config files. Precisely because there are so many distros with so many different curated experiences, you can find a Linux distro that matches your own preferences.
And yet all Linux distros give you the hardware support and things like the OCI ecosystem that only the Linux kernel can provide.
Given the above, I wonder sometimes why you would choose FreeBSD over a Linux distro. But your statement that FreeBSD has more users than many Linux distros is a good one. It is also true that, while distros like Arch or Debian have more software in their repos than FreeBSD, the FreeBSD ports collection has a much larger selection than most distro repos. So, overall, FreeBSD does achieve a nice balance. So, that makes sense to me.
I don't say this to besmirch FreeBSD, FWIW. I think it's very important that Linux is not the only game in town.
Current practice is to put a meta tag with your encoding, use a Unicode BOM, or less favorably, send the charset attribute in the Content-type header.