#Rust #Cargo #ecosystem #Debian #unstable not #stable #btrfs #Ad: laptops & more @Amazon: services.exact... You can support my work at: / renerebe github.com/spo... exactcode.com t2sde.org rene.rebe.de Terminal font: Comic Code ;-)
God I was arguing against too many, unnecessary, micro- dependencies for forever and everyone always called me crazy. But even without rust it's getting so hard to build anything because everything depends on everything. And without even using Rust I knew that cargo is bad because I saw what npm does and I used go build, pip and other stuff that by passes system package manager and just breaks things.
Because people simply make less stuff themselves as they don't get how it even works and security is not there concern or safety in general. Which yeah is not expected to be deliverd but if you need hundreds of micro deps to make anything work as the language is such a pain that people write allocators that are index based into vec to not have the BC fail there build.
@@platin2148 they are making too much themselves instead of contributing to a larger library. It made more sense in JS because in the semi-old post jQuery days that was one of the trendy ways to minimize the size of your package - to import microdependencies for everything you need instead of bundling some jQuery replacement. But then they got the ability of selective imports yet the culture remained. The alternative to microdepencies isn't nothing, it's macrodependencies. Like, instead of importing leftPad, you import lodash
I like the idea of Rust more than the reality of Rust. I wish more direct competitors existed that offered the same type of memory safety, but without the drawbacks.
I think memory safety is overrated. You have decent memory safety using garbage collection. The costs of GC are actually not really that bad. Where these costs are not acceptable, you can achieve most of what Rust does by using partial support, as in what Zig provides. Not perfect but I think it's easily good enough. The more I learn about Rust, the more I perceive it as harmful. Not because it's a bad concept or implementation, simply because it is not a general enough concept to be used where Rust is thriving. What I like about Zig is that it does not impose abstractions, it uses the same interfaces as C and at the same time provides many benefits. It's not a high level language, there are certainly better high level languages than Rust. Languages that are not as painful to learn, difficult to understand and pervasive in their opinionatedness.
This is not a Rust problem in particular. What I'm detecting here is an impedance mismatch between the way an OS like Debian expects to deliver its code and how a language like Rust expects code to be delivered. The same problem exists when things like node.js and Python get packaged in the OS. Invariably the version of the language and its libs I want to use is something totally different to that used in the base OS.
What makes Rust special is that it introduces the same problems interpreted languages have even though it's compiled. Other compiled languages have that to some extend with shared libraries, but these tend to be better maintained than random cargo dependencies. I definitely prefer the approach of Zig (though it's still in its infancy) favoring builds from source and self contained binaries, at least as a popular default.
@@michaelutech4786 Whilst it is true that library/module packaging and distribution systems have historically been the province of interpreted languages I don't see that is because they were interpreted languages. One could imagine a module system like NPM for C for example. Why not? Just for some reason nobody ever thought to do that. In fact I would argue that when you use an OS like Debian and "apt get install libwhatever" then apt is exactly that kind of package system, except it is tied to the OS rather than the language. I don't know why you state it as a problem. If you spend a lot of time writing some code that I might find useful what is wrong with having a package manager that makes it easy for you to make your code available? And vice versa. Of course we cannot make such comparisons because popular compiled languages like C and the rest don't have such a package system,. Which is a huge problem in itself. I know nothing of Zig but if building from source is what you like then that is exactly what Rust's Cargo does.
@@Heater-v1.0.0 I would love to have this discussion in person. I have no problem at all with micro dependencies or dependency managers as such. I love to use them while developing. My problem is that these mostly only work well during initial development. The more a product evolves and matures, the more these transform from an asset to a liability. The reason for this is that most often, dependencies are changing more than the actual product. You have to keep updating them and account for breaking changes just to keep your product technically viable. You also have to account to the possibility of malicious code being introduced in all dependencies, which becomes more important and more difficult with the number of dependencies you are using, and this is transitive. This is expensive because you had to do that long after your code settled and no longer requires a lot of changes. This is different in traditional compiled languages, because there you most often find bigger libraries bundling features. They tend to have more stable release cycles, respect semantic versioning but also follow release policies, consist of bigger project teams. They suffer from the same problems as micro dependencies, but their problems do not scale as fast. The reason why they tend to bundle features in libraries is because they had no dependency manager. And there we are back again at the effect, that these managers make development easier, but create a debt in the overall life cycle of products. If Rust is building static binaries, then I don't understand why Debian would be packaging dependencies at all. Then it would be entirely up to the upstream developer to define which dependencies are used and save and the maintainer would not need to relax dependencies at all. I'm not using Rust myself and assumed that there must be a good reason to package dependencies if Debian folks do that.
So the core problem seems that the rust build system expects to vendor their libraries while debian tries to repackage every single micro-dependency which is a lot of work and doesn't work well. But why repackage rust "crates" anyway? Rust always statically links so why repackage the crates? They are only needed for building Rust packages and most will fetch them with cargo anyways? While I don't like Rust's micropackagages and cringe terminology either, it's hard to see how this is Rust's fault
@@MoreReneRebeThen use rustc and a makefile??? You are expecting cargo to be something which it isn’t. There’s ways to vendor packages and dynamically link, but it’s obviously not the stable or preferred way the way Rust is.
Yes bro just go through the entire dependency tree that has hundreds of dependencies and vendor all of them one by one, what a great use of everyones time!
From my experience with rust, there was a program I wrote which necessitated some cargo project that I could use that didn't require me to write boiler plate to translate read memory from a running process to stuff that users can actually read live. Unfortunately it wasn't thread safe and was just abandonware, so I just copied the source code into my project, took only what I wanted(only the code necessary for reads), and then just modified it to make it thread safe. I'm sure for some of these projects that have been stuck in 0.X.x update land you can just cut them up and take what you need in a similar fashion. Though they've probably shot themselves in the foot by adding too many of these dependencies in the first place, now. Devs should probably realize it's time to take on the headache of going through each dependency and figuring out which one is essentially abandonware to copy and absorb into the codebase! Maybe when the dependency maintainer releases a "1.0" or "completed" update and the dependency project actually requires the whole dependency is when they can consider seriously using it as a dependency.
I think the only way is to maintain Rust in GNU/Linux, is by not maintaining Rust as a package in distributions. rustup seems to the only way to install binary Rust compiler and Rust related toolchains. It's sad to see that most Rust people aren't even aware of this issue (none of them seems have any plan to addressing this?), and only from a maintainers perspective. The number of dependencies will just increase over time, so it's a good decision to drop this early on. Nice stream, thanks for the update Rene!
Building rustc is totally not the problem and rustup solves nothing but would cause even more integration issues. Building all the packages with their dependencies long term and stable is.
@sofiaknyazeva Honestly, GNU/Linux needs to overall move away from the model where software packages are maintained by distributions into the one where all software is provided by the upstream. The first one unfortunately creates soooo many problems.
Rust is gonna ship a new ABI named crabi, which should make it easier to build shared libraries. Also, yeah, there's a problem with micro dependencies, one way to solve it (when crabi is stabilized) is to select biggest and/or most widely used dependencies and make them special by compiling them into a shared lib, whilst leaving all other dependencies for static compilation. I am not as experienced in making Linux from scratch, so I can't know if this solves all the problems, so correct me if I'm wrong
Only valid for cases where there is no runtime input required which is at least in my code quite rare. And for those cases i actually do have tests + use slice_t types when passing anything that isn't a simple pod but arrays of them.
@@RustIsWinning I dunno what projects you are talking about i didn't refer to any. I referred to runtime can't be fixed at compile time if you think so you better learn how computers work. Also better to no shit talk to much as the things you made probably are not used anywhere, while the stuff i made is working and secure in GM and Siemens devices running..
What is the problem with Rust syntax? It seems to me that people say that because signatures can get complex sometimes, but maybe that's just because the language explicits constraints which in other languages would be implicit, which is arguably not a bad thing.
The signatures aren't just a little complex, they're crazy and the concept of lifetimes with those little single-quote symbols are so bizarre and out of whack, Rust feels like a demo for something that should be much more mature and stable than Rust, meaning that it has some good ideas but it's implementation sucks, reeks of esoteric college OCaml hipsterdom and needs to be improved upon with something that's more akin to C with memory safety.
@@liquidsnake6879 Ohh boo hoo! I dont understand how languages work and everything has to be like C!! Cry more please. Go design a language that solves temporal memory safety at compile time. C is outdated. Get over it boomer.
I honestly agree with you wholeheartedly that having a centralized dependency ecosystem with little leeway, even if it's just static bindings, poses lots of dangers and risks, especially if the checksums are hard to verify manually. Maybe in the future they can implement a cargo-bare setting in the toml that disables external dependencies. I did want to say a few things in favor of the language, not necessarily the ecosystem. 1. If you're compiling multiple projects with it, cargo benefits greatly from sccache. 2. This goes without saying, but Rust technically does not need any external dependencies. The dependency inclusion falls on the project maintainer and Rust stdlib is included when running with an empty Cargo.toml. Most dependencies are just bindings to their static library calls unless there is a cargo features that statically links a vendored version (similar to how the openssl crate does it, e.g if you try compiling a project with openssl as a cargo dependency without libssl actually compiled
Zig doesn't have the same problem because Zig doesn't have libraries, which is a bigger problem. If you want to learn a language you can be productive with, learn one that's at least 10+ years old and popular. Not popular on youtube or twitter, but popular in the real world. It can be fun to learn boutique new languages to entertain yourself, but it HAS to be laid on a strong foundation of a tool that actually works that you are productive with If you're asking this about Zig, get better at C++. You'll know you're ready for Zig when you'll be completely disinterested in what some influencer says about it because you know your own gripes and preferences better than anyone else
The disturbing thing about Rust is that Cargo easily brings in hundreds of dependencies for a typical project. There is no way to analyze that huge attack surface. JavaScript/TS and Python have the exact same issue.
The problem isn't in the repository, it's in the culture and expectations. Rust historically was connected to JS and hence inherited a lot of JS mindsets including microdependencies. Meanwhile, Maven predates NPM by a lot yet Java doesn't have the same problem. Java packages tend to be well maintained and medium to large sized, with core external helper libraries to work with files, documents, network, concurrency, etc often being supported by large companies or foundations like Apache or Google. This isn't right or wrong situation. JS style allows the community to develop at breakneck speed independently of any corporate influence, Java style allows the project to be maintained for years and decades while often being dated and dependent on what some company or a group of obsessed enthusiasts think you can have
@@NJ-wb1cz Rust has no historic connection to JS. It's not about package repository, but do the same with Odin (by adding package repository and a package manager), and you'll get the same result.
@@sofiaknyazeva developed at Mozilla and used in Firefox, JS culture migrated to Rust to a large extent but not to others. Compare that to Go, Java, Flutter, Kotlin, etc that have package managers but don't have NPM's problems of one function packages and hundreds or thousands of dependencies per project. The problem is not in package managers and repositories, tools don't mind control people using those tools
@@sofiaknyazeva meanwhile, the creator of Odin 8 months ago - "I agree having 1 de facto (not de jure) package manager is the best option. Making it official (de jure) does reduce the chances of de facto package managers from arising". He's not making your argument and doesn't think that package manager shouldn't exist. And there are multiple other other languages with oackage managers that don't have the problems of NPM and Rust
@MoreReneRebe this video was quite disturbing I have been using Rust for about a year and am new to programming... your insights on the ecosystem have me reconsidering doing any project in Rust because of the backdoor micro-dependencies... I think I will my learning continue with vanilla C and Assembler
I'm not a programer but have experience managing parts of a divers fleet of hard/software/OS (perhaps some times administrating is a better description) products, and have been following opensource and proprietary software development. First be aware of the pro's and con's for decisions and features what is being used. An important part is that rust makes many decisions most languages don't dare to take. While complicating things it does give you insights in a proper strategy of software design. So continuing does not only allow you to not trow away your progress but will also help you understanding design decisions for you future projects. Perhaps it would be a good idea to brush up a older perhaps more basic language to widen your knowledge and perhaps apply things you have already learnt to do in rust. While many languages are different many concepts and perspectives are the same and is the more impotent and future prove part. Only you know what you need to learn, but don't be upset there are positive points in learning rust compared to others. But knowing a other language does allow you to put things in perspective and recognize what is important and what is less important.
C is great if you like writing the same code over and over again for different types, or use void pointers to implement polymorphism. Use C++ if you prefer to not repeat yourself. And that's before even mentioning resource management.
It's not that the language itself is bad, just that people depend too much on external dependencies and can't help themselves. This is a problem for every language that has a rich package ecosystem with easy access to said packages.
@@thundertastic896 Conversely: It's not that the C language is bad, just that people keep writing the same thing over and over again rather than using external dependencies. This is a problem for every language that does not have a package ecosystem.
^^ having a background in mechanical engineering the name Rust is not the best. From this video i'm really curious what type of system and language would be best suited for OS/Kernel dev? Memory safety at a language level sounds really good. Dependencies or Libraries can be great if you isolate pieces that will change together, or ideally wont change. The idea of a checksum isn't too bad either. As it checks if you got the right dependency when building. But you got to be able to change the code and update the checksum if necessary. Checksums could also help with only downloading packages once. Designing such a thing sounds like a similar complicated system engineering tasks than orchestrating all the engineers you need for making a car or similar.
Well the entire language seems to be made for transitioning people that are JS devs to more system level. Creates even more trash that one has to cleanup. Would be great if we could just realize that there isn't enough people and slow a bit down with adding even more layers for people that are less competent to work with.
@platin2148 Rust is static and strongly typed, while JS is dynamic and weakly typed.. They are very different, and I doubt Rust was created by JS devs.
@@MrWorshipMe I think you simply don‘t get it. There are more js devs migrating then system devs as such the lang develops in that region. Its getting to be the bridge
@@MrWorshipMe I didn't say JS developers, I said web developers. Many of the original core team members were originally Ruby/RoR devs. And it's well known npm was the inspiration for Cargo. That has been talked about openly by core team members. I'm not a gnostic in possession of secret knowledge.
I don't know why but this microdependency hell does not seem to happen in go. Do not get me wrong there are dependencies but i feel like its is because go developers like integrating things into the standard library instead. That being said I would not add golang into the kernel. You don't want garbage collection into the kernel.
Because Go is run by a corporation that manages it. Rust is run by a community. This influences what kinds of expectations developers have of their environment, and consequently what kind of expectations they try to fulfill. Like, you won't find a React clone written in COBOL compiled into WASM. Why? Because that's not the expectation and culture of COBOL programmers. No one prevents anyone from making it, people just don't want to do it because they don't think that way.
Very nice video … die Kritiken sind mehr als berechtigt. Die Rust Leute werden aber leider nicht mehr aus diesem mess heraus kommen. Was meinst Du. Liebe Grüsse aus Zürich.
Übrigens auch erstaunlich wie sehr Fan Boys selbst offensichtliche Probleme bestreiten. vor kurzen mal einen Reddit Artikel abgesetzt mit hohem reach. am Ende stand 50:50 up and down votes …. Und es wahr schon fast eine formal logische Beweiskette, die ich dort gebracht habe.
Thanks! I glaube die Rust Erdinder lieben ihr Ecosystem so sehr, dass Sie es wohl eher nicht großartig ändern werden. Da kommt man ggf. nur raus kommen wenn ein Großteil der Entwickler Cates.io meidet und Rust Bibliotheken ohne Cargo entwickelt. Wenn ich mich richtig erinnere hat Rust aber 10 Jahre später immer noch nicht shared object support, so dass man nichtmal seine libSuper3Drust nicht einmal als .so verteilen könnte, .... selbst wenn man wollte. Dass alle Abhängigkeiten und built Artefakts bei jedem Paket neu gebaut werden, wird aber zu einen noch viel größerem Skalierung Problem. Sehen wir mal an in 5 Jahren ist in Firefox und alle Abhängigkeiten alles Rust. Dann würde z.B. beim Übersetzen von Firefox quasi das gesamte System neu gebaut und statisch gelinkt, also von einer dann hypothetischen libc-rust, libjpeg-in-rust, libWayland-in-rust, etc. da ja derzeit die Installation von Bibliotheken, vor alles als .so nicht einmal vorgesehen geschweige denn unterstützt wird, ... Build Time 3-4h anstatt 10m auf aktuellen Workstations, ... ??!?!?
@@MoreReneRebe ich stimme sofern zu dass viele microdependencies einfach ein project cluttern und nervig zum maintainen machen, aber an sich ist das cargo ecosystem sehr nice, und dynamic linking ist möglich wenn mans braucht, und das wird nur neugebaut wenn sich die compile optionen (optimization etc) ändern oder der cache gelöscht wird, weil halt nunmal alle dependencies von source compiled werden standardmäßig was ich gut finde, dewegen macht es auch sinn dass man die nicht universell installieren kann (jedenfalls prebuilt, der source is obviously in nen global cache). aber kannst natürlich auch ohne probleme statisch/dynamisch c libraries etc linken
The touble with memory safety seems more like a feature than a dis-functionality, if rust forces memory save compiling and a OS administrator/programer chooses to change some things (use central OS provided binary's or library's) it wil complicate things like forcing rust compile decisions requireing specific configuration settings to vendor bind them. I believe rust should have(i'm no expert does it have, is it feasible?) settings/features to enable those requirements in OS level library's. I suspect this wil require more from both the application developers and the OS maintainers te make sure this can and will be enabled. But in general I do believe Rene has a point in that the OS should be able to apply the for them required code maturity and if posible optimize the system by centralizing the library's to save memory what is multiplied by the nr of aplications using it and managing code maturity and security by there OS standards. But I do suspect this will require more decisions that will both complicate things and help design better code quality, but that would be subjective.
@@MoreReneRebe Oke I don't know enough about the subject, so I presume I am indeed making a mistake here. I believe my mistake is underestimating the difficulty of a rust program to use any library outside the application itself. Because Rene was complaining about changing some code and having Rust complain about it. Together with a reaction "rust build system expects to vendor their libraries" made me think Rust requires specific constrains on what libary's it can interact with. I was thinking about something like linking specific library's and signing allowed code. So considering your remark probably I'm wrong or possibly very wrong. But despite that I hope it is something some languages or frameworks are working at or thinking about.
Yay DLL hell for maintainers and anyone who wants to do anything with any rust codebase! Yes, I remember you making your point many years ago... was it in here or was it in phoronix? who knows!?
Its rust so it basically oxidized away. Cargo sinply makes no sense for rust and people are way to reliant on it. I do wonder when the Kernel modules will be abounded that are written in it as its a pain to maintain.
@@gregandark8571 Nothing. Asking for a NPM like system will create tons of crap. People simply don't get how licensing and also how security and how code works. But i suggest just watching one of the later primeagen vids or reading a blog from bvisness for why it's bad in general.
@@theevilcottonball That is basically the only plus this system has. And that is also only because it has to be compiled in the first place. God forbid the abi will be ever stable and one can push binaries this will let hell lose.
I don't see the relationship of having a build system and it being a compiled language. ODIN for example is a compiled language and the conpiler figures out how to build it based on its module system. Also scripting languages can have build systems/bundlers to. And there a build systems dor JIT compiled languages too. Even if the ABI gets stabilized, I don't think people will download binary libraries en masse. Swift got a stable ABI, is it binary chaos over there? I don't know.
Maybe the Cargo registry need to crowdsource audit by the community. Maybe it should allow auditors to be paid too. Though, I am unaware if there are any existing system that uses this model.
Yes, obviously. It's also very common to split projects into multiple crates, which are not always usefull outside of the project. I am kind of dumbfounded that anyone could be surprised that copying all this shit over to Debian is a viable idea. I think it's more accurate to say that packaging in Debian & co. is broken, so noone is trying to make the rust ecosystem fit into that broken mess.
I write Ada and wear a dog suit on the weekends. I'm a little out of the loop on this Rust stuff*, but I have zero idea what "solid engineering meritocratic excellence" means in the context of software. Most of what we do in software development isn't anywhere near engineering: software being infinitely malleable just doesn't lend itself well to it. In a lot of respects, what we do is much more an art than a science. (This goes double for anything grounded in mathematics like formal verification à la SPARK: proof is inherently a creative endeavor.) *: I don't usually have a need for heap safety when Ada lets me avoid the heap entirely, but rustc has also gone a little too far in the weeds with ABI bikeshedding that makes redistributable precompiled libraries problematic. There are some features from Rust's ML underpinnings that I wish I could have, but meh.
This is because Rust compiler adds a lot of "extra" code, alongside the main code-base. That includes, overflow/underflow checks (it's disabled in release mode), function polymorphism (it's a language feature, just like in C++), panic messages, additional undefined behavior checks (I don't think most of the std does this, but third-party libraries can), debug symbols, and etc. Usually the debug symbols are the main cause of large binaries and object files, as generated debug lists from the Rust compiler are significantly verbose. Besides, Rust libraries/programs tends to have many other small libraries that they rely on, and many(?) small libraries rely on other small libraries, and this usually ends up, "a program has 1 dependency, and that dependency has 10 different other dependencies".
that debian packager did exactly the thing Kent told him not to do because it would cause problems, and when that happened and he broke the build, he didn't even attempt to debug it, so debian users just stopped getting updates. And when Kent asked him what he was going to do and if he was going to revisit the packaging policy, the packager said a flat "no". That's when Kent had to tell people not to use Debian, since the impact of this was that people were stuck on a buggy version of tools that wasn't passing mount options correctly (it was long fixed in the upstream) and weren't able to access their filesystems when they needed to mount in degraded mode. Debian dev really messed up, and him stepping down is the best possible outcome. Someone else will come along to maintain the debian package and they WILL take a more responsible approach. If you're putting yourself in the critical path of a critical system package, you need to be bringing your best possible. If you're not able to handle that responsibility, it's better to let someone else do it than screw it up (imho).
@@MoreReneRebe yeah, it’s just that debian’s (just like fedora’s) packaging system is faulty. We need to use more modern distros in order to run bcachefs or we’ll have to build bcachefs-tools ourselves.
@@MoreReneRebe I’ve watched the video and all I’ve heard is whining of a stuck-in-the-past dynamic linker who’s afraid of modern development practices .
Linuxes and especially Debian is the shittiest shit OSes, impossible to install so it just works, something doesn't work everytime I do. Those guys can't create stable OS, yet they think they can teach others.
@@MoreReneRebe it’s only an educated guess, but you’re not dissing anyone else (apart from some nameless Intel engineers, but pretty sure they don’t care) and the kiddies who just do it for the lulz neither have the patience nor interest to sit through your highly technical videos, I believe. Also, you have a crowd here, but you’re not that famous outside Linux development, much juicier targets out there.
I'm glad to hear someone else comment on Rust's readability. I sometimes feel like I'm taking crazy pills. I find Rust's syntax much less readable than most other languages
@@MoreReneRebe If you get some time, could you please post the link that. I'd be really interested to watch it. I've tried to find what video you're talking about but can't seem to find it.
Package systems are not toxic. It is a handy tool for majority of non-hardcore users to make their life easier. It is the content which becomes toxic. I always try to use near-to-zero-dependency packages. If there is a popular package that has 300 dependencies, well, it is mainly creator's fault, but also user's fault that they put so much attention and hype into house of cards. It is also about obsession of some ppl that need to have everything organized on atomic level.
package systems outside of the OS have proven to be problematic again and again for all the reasons outlined in the video. It is not my fault to focus on the 300+ micro dependencies, I just need to patch them daily again and again for all the non mainstream architectures to build and work on arm, mips, powerpc, spare, or the latest riscv. The help me with nothing and only make my work orders of magnitudes harder and more time wasting.
@@MoreReneRebe I'm not a low-level programmer. I encounter a problem with npm or pip a few times a year at most so..nothing unusual most of the time. I understand that this has a multiplier effect for you, due to the fact that you are working with so many architectures at once. Sometimes you sound like an avenger who is trying to fix all the multiverses and is on the verge of madness. Considering your love for old staff, I still don't understand that you haven't joined some BSD distro, ideally focusing just on 1-2 archs.
As a high level programmer I have never seen anyone use a os package instead of npm, ruby gems etc and for good reason. Microdependencies are certainly a problem and it makes me happy when people call them out. But for us normies the ability to run the latest or a specific version of a library is a big deal - like a specific git revision, a fork, or a private package - you get all these for free without tying yourself to a linux distribution. Actually it might even work crossplatform.
