Log4J & JNDI Exploit: Why So Bad? - Computerphile 

Do you think this vulnerability was designed and placed into Java by an actor, or do you think this vulnerability was simply an accident?

@@dsmyify I think log4j is open source and many jdk implementations are also open source, so I don't think someone would put this for malicious reason. Maybe it was just an accident. However what might be interesting is if this vulnerability is not something that was found just recently, but was known to bad actors since 2013 and they have been cleverly getting benefits out of this without letting the world know about this.

@@dsmyify hanlon's razor makes me believe it was probably an oversight or mistake rather than a concious exploit

@@dsmyify This was almost guaranteed a case of "Wouldn't it be cool if this library could do X? I recently did something that would have benefited from this existing". And then it got implemented and activated as being on by default without thinking "Does this need to exist?", which is especially difficult in open source where there's not as much coordination.

could be another eternalblue situation, maybe it was discovered a long time ago and was only leaked now

remember the times when excel didn't ask for confirmation to run a script?

@@tyfyh622 yes it would run it without it, I still remember macro viruses

I... I want a pdf that can make popcorn

ActiveX was da bomb.

Actually, the issue was the extension of the PDF format with the JavaScript abomination that had access to your file system. This is made worse if you used the win9x/ winNT/XP default that any user you addded had administrative rights from the start... If you want interactivity, just create a different file format, so that you can clearly tell them apart and avoid the interactive one.

It's a bad and unintended behavior of a piece of software. That's a bug, just a complex one. Just because it's caused by a combination of a bunch of intended features doesn't negate the fact that it's unintended behavior. If this was the *intended* result, we should call it a back door.

@@fwiffo I personally prefer to call it an exploit. Exploit -> unintended application of intended functionality (every module is doing what it's supposed to) Bug -> unintended functionality (at least one module isn't doing what it's supposed to) modules being functions, classes, etc EDIT: An exploit is a design mistake, a bug is an implementation mistake.

@@fwiffo A bug would be unintentional. This is just incompetent software design.

I would agree with you up until the "send and run arbitrary java objects," that is inexcusable. It's perfectly fine and valid to send POJOs that the other system has loaded in the JVM, but it would have been better to stick with XML or JSON or some such. Running any kind of code that somebody (¿over the internet!?) has handed you is absolutely horrific.

The semantics don't really matter.

Just saying a philosophy made in the absolute infancy of software and microprocessor computing doesn't necessarily apply to modern day. The vulnerability doesn't exist because it had the functionality, it existed because of extreme and utter negligence of the developers working on it, the vulnerability existed for months and wasn't fixed.

@@ChristopherGray00 modern day software engineering is a horrible mess. you must be really a fool to appreciate it (just like taking a jab from a corrupt industry and obeying the government).

@@Amine-gz7gq It sounds like you are really out of touch with software engineering in general, using modern software while not knowing it. Linux and BSD unix has been exploding in features, stability and security in the last 3 years, the amount of work on the open source software world is more than it has ever been in history, saying "modern day software engineering is a horrible mess" is very ignorant.

@@ChristopherGray00 OK enjoy your life.

The word you are looking for is 'tenet' (a belief), not 'tenant' (a rentee).

Not many that are used to any great extent really, this is a case of extreme negligence.

@@ChristopherGray00 That we know of

Fewer because in closed source and especially paid apps someone is actually responsible for the code instead of the open source mess where nobody is responsible and nobody is getting paid to maintain high quality.

@@Nors2Ka I guess you're not familiar with Windows, Flash, and so on...

@@Nors2Ka You're very wrong about open source software. Closed source has way more bugs and exploits.

every bug is a design flaw

@@bimjean1053 false, there's easy mistakes and there's extreme negligence, log4j's situation was the latter, it had the functionally to remotely execute code, the vulnerability existed for months, and it was not fixed in that time.

@@bimjean1053 Form my understanding every part of this vulnerability is working as intended.

@@pupip55 Yep. Every individual piece is doing exactly what it was meant to. Someone just linked them together in a way that was short-sighted from the security standpoint.

Well, since everyone here seems to be an "expert" I humbly request you dedicate your valuable time to supporting this Apache project and other free services.

Right, we're so adverse to tradeoffs we'd ideally prefer not to have to think about them, let alone impose them consciously and deliberately. For example, in typical corporate settings developers don't get to discuss architecture choices and tradeoffs until the existing architecture is decided inadequate - again, most often ,not by the developers. At this point, a hectic modernization effort begins - but you cannot modernize stuff all at once, so you get bits and pieces patched in - leaving you with the worse of both architectures.

Every ABC should include XYZ, because every alternative to ABC also does. "Your framework does not include a toaster? Ha! I will rather use one that does."

Agreed but also keep in mind too that programmers and whole teams are under the constant pressure to meet deadlines and production levels. ANY software that offers a boost in productivity, and becomes popular, because if it, becomes a target for hackers and malicious actors of all levels. FOSS products and their libraries can be accessed and updated by almost anyone and some are actively pursued to achieve malicious ends. Even without malicious intentions bloating of components is something that is likely to continue and create these situations.

I'm just stunned that anyone even thought that was a good idea to begin with. To quote Emperor's New Groove: "Why do we even have that lever?"

@@davidmorton8170 The bad idea is doing anything to user input before sanitising it. If you want to go down the path of why have the logger do that, then why bother with a logger at all? Just make your own. create the strings in your own code and send it to the appropriate place to log it, without using any library.

@@jeffreyblack666 It turns out this is a bad way to do security. There have been hundreds of exploits written against code that looks like this. For example: Sanitizer #1 looks at a string and sees that it has no HTML tags in it and is HTML-safe. Sanitizer #2 sees it has invalid unicode in it and normalizes it all to valid unicode. Now the string is considered "safe" and printed into an HTML file, but it turns out that Sanitizer #2 allowed a malicious user to sneak some HTML into the input in a way that undid Sanitizer #1's job. The consensus best way is to try and represent the user input as faithfully as possible and do the sanitizing as *late* as possible. For example, use an HTML template language that sanitizes strings automatically when it includes them, or a JSON library that automatically encodes unicode with escape codes when it serializes it. Strings are *never* safe unless a programmer controlled them the whole way. A log library with a structured API that treats strings as unsafe user data by default is the safest way to log data. Concatenating strings in your own code is the least-safe way.

@@jeffreyblack666 Yup. Add a timestamp, then to stdout with you. It's someone elses problem now. _Every_ feature described is something I don't want from a logger. Variable interpolation? No, I'll just build the string myself. Recursive interpolation? Wut. Why? I want logging to be _fast_ not add potentialy _everything_ to it. Global variables. No. Never even once. Network requests as a result of logging? You're joking right? I'm not saying I wouldn't have added anything like that; but in retrospect they all seem insane. It's a logger; what should it do? Log; faster is better. It shouldn't be an entire application taking a life of its own.

@@dtkedtyjrtyj There are features I need from a logger more than just writing to stdout with a timestamp: per-category logging level configured at run-time; simultaneous writing to standard output and log file; automated archiving and purging of daily logs.

To be fair, one can also turn of JNDI lookups in config file for log4j quite easily to limit the issue too.

Does this vulnerability relate to log4j-core only or all log4j installs of the versions in question?

@@stewartdahamman From the Log4j homepage: Apache Log4j2 versions 2.0-alpha1 through 2.16.0 did not protect from uncontrolled recursion from self-referential lookups.

@@EwanMarshall Thank you for this 👍

There's something wrong with java culture when you have a widely used facade package just for logging. People aren't joking about the AbstractSingletonAspectInstanceProxyFactoryBean

Same here. I am not even a cyber guy, but a developer/integrator that answers to their scans.

Stop inventing languages in languages. I'm looking at you, format strings.

But the whole point of OOP is adding new features via inheritance. And don't forget running agents on other servers...

@@RonJohn63 : Neither of those should be treated as the purpose of OOP, it should be treated as just a way to use encapsulation to create programming entities that act in coherent, self-consistent ways. As the most famous of the multiple Unix principles says- Do one thing, and do it well. Don't try to produce honey with your composite structure just because it inherits from the "hexagonal grid" type that gets used to implement honey comb.

I mean sure. Yes. But at the same time this is very much a non answer. This is like saying, "we don't need to work thru the local and global economical, political and social transformations needed to bring about the change of the world. All we need to do is. Be nice. To other people.

@@izayus11 That analogy doesn't make any sense. @Jerome Thiel's advice isn't going to prevent *all* security vulnerabilities of course, but they would be a lot fewer and a lot less serious if developers would take that to heart.

You can with an Oreilly subscription. Mike runs some courses there.

@@drgr33nUK 🤩🤩🤩 my hero!!!

I want to super like this comment

Nice guys yes but aren’t they a Java shop ?

based on possibly unfiltered user data:)

Not true unfortunately. Apache credited the disclosure to Alibaba's cloud security team.

@@TheRavenMad Whoever else ALSO discovered it, it WAS clearly discovered by some Minecraft players.

On the anarchy minecraft server 2b2t one person exploited it and did bad stuff

@@JCBOOMog fitmc fan?

@@siddharthkhamithkar5920 I've watched him ye

Lol

Remember the day when "left-pad" library almost broke the internet?

It's not as much because of bad developers as it is because of a bad standard library though. JavaScript lacks most features required to write code, which is why they are being imported.

As they mentioned in the video, Log4J is already doing the composition of the libraries under the covers. So there’d be no performance impact by exposing it. Quite the opposite. Most developers would NOT compose it and performance would be faster.

It isn't the expansion that is the issue, but the recursion of it. Expanding is fine, but do not expand the data that returns.

The first problem was that it was configured by default to do lookups in messages. They disabled that by default and later made it impossible. The second problem was that, after doing a round of lookups in your template (which is a trusted string the programmer put in) it did additional lookups in the result, which now contains untrusted data.

Exactly, surprised it doesn't escape strings

@@vikramkrishnan6414 You mean we need prepared statements for logging? :D Well, ... could be a solution ... or we just decide that loggers only log given strings. Hm .... :)

True, just cut the complexity

Exactly. Turn stuff "off" by default or not include it in the first place. Have people activate whatever features they want to use, so they invest at least a little bit into learning what the feature is doing in the first place.

That's the reason java open source libraries are piled with vulnerabilities

Maybe such advanced or fancy features which the majority of library users won't need should have been switched off by default and would require those few who covetted to make use of them to have a little extra effort by explicitly enabling some sort of macro switch and having to recompile it altogether?

@@ThisNameIsBanned Sure, why not go all out. Disable every single method and property and so on, requiring some kind of config option to turn it on. That way they need to find the config option, and turn it on to know what it does. That way it can be nice and secure.

Hah don't hold your breath :) Sean

Bingo

unix philosophy violation

FreeDOS exists, it is the FOSS replacement for MS-DOS, is still actively maintained and has it's uses to this day.

I don't agree with his sentiment at all. Maintained software is "past its sell-by date" when it is no longer useful to anyone. DOS is useful for a variety of things - hence why FreeDOS has an active user community. It offers something that few other operating systems do, in that you have full access to the hardware. Does he not realize how much COBOL there is in the wild? Or how many engineering firms still use Fortran?

If people want to maintain obsolete operating systems as a hobby, good for them. It's their free time, who are we to tell them what to do with it?

@@jeffspaulding9834 I don't agree with the sentiment either. No one is doing anything wrong by maintaining software, however old, that people still use for things. On the other hand, using software that isn't maintained is probably not the best of all possible ideas.

Yeah, mentioning DOS here is a bit like blowing how air here. There simply is no attack angle here like it applies to JNDI and Log4j in DOS. Any DOS. Even having a 3rd party web browser like in the case of FreeDOS doesn't change that...

@@jonathanguthrie9368 Yeah, unmaintained software is always a bit of a minefield, and it's only going to get worse. Here's an example: I use a lot of 90s era software for working with industrial equipment. And it's fine, for the most part - although some of it won't run on 64 bit Windows and has to run in a VM. None of it listens to the network by default (unless you spin up a simulator, and even then I'm behind the corporate firewall or air-gapped). Today's software is often written as Windows services with a thin shell program to interact with the service. As that software becomes unmaintained (long before the hardware for it gets replaced), it'll still be installed and running in the background on the machines of every technician and system integrator that still needs to work on that hardware. Ticking time bombs - can we say Stuxnet 2.0?

I mean, this is basically everyone. Understanding every nuance of every nested dependency in a modern application is an impossibly high bar.

Especially with the looming deadlines over our heads

Yeh that's a pretty throw away line that's not very useful

hm do I also have to understand the runtime env and the operating system?

Yup, this kinda thing seems like it's essentially inevitable. A lot of managers effectively incentivize devs to put out hasty, sloppy code... so of course that's exactly what they do (because why wouldn't they?). This is what happens when you let non-tech people (specifically: business majors -_-) manage tech people ;)

Its also the reason App development has rocketed so fast and the digital world has become our world we depend on so fast. Cyber Security has been the afterthought.

Ah yes my favorite security by obscurity

It was safe BECAUSE it was opensource, imagine if the sourcecode wasn't public and you had to wait for a company to fix it for you. It might have been ok, but usually thousands of heads are better than a few with extreme deadlines and a maybe not optimal work environment

"I can't imagine anyone solving a problem except under penalty of being fired."

A worm that patches the vulnerability that it entered through... That sounds genius... If true.

I think you are referring to 'Cybereason Log4Shell Vaccine' use the exploit to patch the system.

This was attempted for the code red exploit (with some unintended consequences). Also, some 2b2t hackers were doing this to patch log4j vulnerable Minecraft clients.

Today jfrog found a log4j-like vulerability in the java based H2 database console.

Oh man! I'm so glad I left that world. I already disliked that employer enough. I couldn't imagine having to patch pyspark on GCP.

Software Engineer: feature

I assure you, those devs don't like it one bit, but when management is presented with the bill (a.k.a. effort estimate) nobody actually wants to pay it to be done in a safer way...

@@bonononchev634 You must associate with sane devs. I got out of Linux distro development when Linux desktop software started developing circuitous and deep dependency chains. It was also pretty obvious that Linux package managers made the wrong things easy. They make it very easy for devs to use libraries while hardly helping system builders to check the quality of those libraries at all. After my experiences with Linux, I almost got depressed when I learned package managers were appearing for programming languages. Ever since then, I've been thankful I don't have to work with any of this.

@@bonononchev634 I wish this was true, but unfortunately I have met many the React/Angular dev that sees absolutely no issue with pulling down a wad of spaghetti on the order of hundreds of megs. A Lovecraftian horror, knotted into Lolth's web in the form of node.js code.

I'm really, really hoping that we can get computers to review code before we mess things up too badly. I'm not sure how possible that is, or if it could ever be a full replacement for a human (surely eventually, but how soon?), but it would definitely help. I feel like most managers who haven't written much code themselves don't quite understand how important it is (and unfortunately, they're also the least likely to listen to that input in the first place), so it'd be a lot easier to get them to pay for an AI to do it rather than "paying people to sit around staring at an already finished product" (an actual quote from an especially bad manager I had).

@@idontwantahandlethough I hope so too. Everyone likes to quote the halting problem to say "it can't be done," but I like to point to the fact that the halting problem applies to computers without finite limits. ;) I'm not smart enough to go deeper than that, though. There is ongoing research around finite computers being able to understand themselves in ways unbounded Turing machines can't, but the last I heard was it's all blue-sky mathematical work. The Agda programming language is part of it.

That's an insane design pattern.

@@fwiffo No it’s not. The purpose is that you can configure an application server and code will look up key features (e.g. database connections, ejbs, message servers, service handlers, etc.) at runtime. It makes code highly portable so that you don’t have to recompile your artifact for each environment. (Which IS insanity.) The real WTF is that a logging library is doing JNDI lookups based on log messages. Recursively. 🤦‍♂️🤦‍♂️🤦‍♂️

A really breaindead design, especially without whitelisted servers.

@@devilaverage6718 yeah, and if your network design includes allowing all your production servers/containers to hit JNDI requests externally there's probably no stopping tons of other vulnerabilities as well

@@thewiirocks It is insane. You're trusting another computer to give you arbitrary code and assuming it's safe to execute. And by default, it assumes every computer in the world gives you safe code. That's fundamentally broken. Maybe if it works off of a whitelist it's OK, but it is wildly unsafe by default.

Not built in... It's a library.

Keep in mind logging is like breathing - it happens everywhere at all times.. so every bit of extra code you whack into the critical path - you create more work. Log4j (like any software) is juggling competing non functional requirements - performance, simplicity, security, extensibility etc etc.. along with functional requirements. It does raise the idea that logging should have different categories of log message - trusted/internal only stuff and messages that may have some part that comes from the outside world (because rarely do you log a single value like that - log4j is quite often a String that is stuck together via various things, which is a prime case of injection risk.. Anyhow - this has been an important realisation for everyone to think more carefully about logging as a vector for attack.. or look more closely at what log4j settings should be disabled to lock it down more. Ironically one of the best tools for detecting and analysing security attacks is the logs! So it is a critical capability. I also look on the bright side: this will mean many enterprise systems get an upgrade!

@@--Nath-- The moment they decided to start evaluating ${} in the data to be logged, they added more to be added to the "critical path" To prevent injection attacks, or manipulation of the data being logged, the data coming from users needs to be sanitized, which is likely going to be on par, or less than, with evaluating the nested ${} in the first place.

@@bakslashr in just about every enterprise system out there or framework that shields from directly calling log4j - that needs to happen further up the call.. by the time log4j gets it it's too late (you've got valid substitution value stuff mixed in with a string with bits of user input shoved in). The log4j calling JNDI bit is the real problem though - that feature needs to be gone. Substitution values are kinda key to how logging works. Anyhow - I think this situation, while a massive a shitstorm does mean people will be thinking more critically about the matter of logging.

@@--Nath-- I don't think this is quite correct? Log4j is executing as code a string that the developer though was going to be treated as data. Its gonna be extremely difficult for a dev to sanitize this from their log inputs if they don't even know the functionality is there?

@@kxjx developers have *assumed* which is the problem.. Everyone uses log4j by default almost but without really looking at the functionality.. (because hey, it's just logging.. right?) and why this has been so widespread a problem because it is so common/humdrum an activity that no one paid much attention to it. Logging message/communication payloads are one common problem I try to educate people about.. but this one extends to any field that gets logged.. so log4j-injection is now something everyone is (hopefully) going to be aware of..

"Secure by default" as it is known. That has been Oracle's policy for many years.

‘features should be "off" by default’ - Yes, but OTOH this can just make the problems more obscure and even less likely to be found. If it's merely off by default it'll be all too easy to turn it on based on some tutorial and then forget about it. A better takeaway is that such problematic features should just _not be implemented_ at all. It should never be possible that a logging library executes arbitrary code based on string inputs.

Log4j itself is off by default. You have to install it as an external dependency. The fact that so many people are using Log4j proves how little people pay attention to security considerations of even adding external dependencies, let alone just changing a config.

Features being "off by default" just leads to loads of confusion with people trying something and not knowing why it doesn't work until they find whatever config option they need to change to enable it. What you are suggesting basically amounts to having no functionality unless you explicitly enable it. Imagine if there was a config option for every possible function.

Like the Vulkan API... But I agree. Things that aren't necessarily useful in majority of cases should be off. Sane defaults over simplicity for some things.

Does the default one allow logging to DB?

@@maagiapall It doesn't have out of the box support for logging into a database. If you want to log into database using the default one you'll need to write a Handler object to be used for the logger.

I remember reading (although I never saw it in practice) that it is used to save logs on a separate dedicated log server or something like that.

Log4j also provided the capability to do remote logging to an external server instead of all on local storage This is entirely for convenience, it's not as bad as it sounds

Well the JVM is a hard thing to beat!