IBM System x3650 M2 Restoration: Part 1 

As a former mechanic, the dirt remind me the kind of dirt that you can find in automobile workshop, a mix of dust (from brake pad etc) and old grease/oil sludge.

This is a server from "CDK" (formerly ADP). DMS = Dealer Management System. They are often found at car dealerships in North America. Past few years they have moved 'serverless' now relying on SD-WAN. I encounter these all the time. As far as the IBM 3650m2 - it was a solid machine. I still have one running at my office, we will decommission it. The IP address on the front of the server is likely not reachable via the Internet. CDK (ADP) used a Cisco WAN to extend that IP address out to the dealership. Controls printers and communication to/from the CDK data center.

My first thought with this is smoke damage. While this server obviously wasn’t burning, I wonder if something near it caught fire and had to be extinguished. That might explain the minor moisture damage and soot (maybe?) inside. Enjoying the restoration and looking forward to part 2.

100% car dealership server for ADP's dealer management service (DMS). That exhaust dust gets everywhere and sticks. They spun off the DMS under their subsidiary, CDK.

My hands started drying up uncomfortably while I was watching you clean the thing with IPA and contact cleaner. Wear gloves my man! Awesome job, regardless of the hand-drying horrors haha :)

Ahhh this is perfect love this older server hardware as a 23 year old in a data center

i still have one of these on service for a customer, the CISCO MCS 7800 / IBM X3650 M2, the cisco label one, what a realible machine

That server looks like it was though a fire. The grime is soot and smoke residue and the water damage is from sprinklers. That would be my guess at least.

Neat. ADP sold a lot of these to Dealerships. They provided their software to service and body shop, estimate software for certified dealers. Incredible estimating software.

The fact you didn't wear gloves really proves there is a thin line between madness and bravery

Hey, me again... So what you have here is a server that ran a DMS, dealer management system. I am not sure what the product was called when ADP owned it, but nowadays the division was sold off to a company called CDK Global, and the new iteration of the software is called Drive. I haven't watched the video yet but it will be running a heavily customized version of CentOS (or maybe RHEL) I believe. This was absolutely located in a dealership of some kind, I am intimately familiar with the software. The software manages the entire dealership from service department, to part sales and inventory, HR and payroll...all text menu (green screen) driven. There is almost certainly data on those drives. I actually work for a dealership (as IT) and one of the first things we did during my employment was migrate away from this CDK drive into a platform called Excede. The dealership had migrated away from autonet years prior in favor of SD-WAN. Autonet was really just T1. I still have a working instance of drive running if you wanted a closer look, but its not very exciting stuff.

This is the server I use for my home NAS, the main difference being its branded by Cisco. Happy to see you restoring it, it's a nice piece of hardware.

I appreciate you courting old-age cancer for our entertainment today. Your workshop is now a superfund candidate.

"I dunno what that is" at 6:20 It's an upgrade key, usually either for the better RAID modes or the IPMI. Without it, the associated module will not be as fully functional, or maybe not work at all. Think of it as a hardware DLC.

ADP had a sewerage pipe burst at a data center back in 2011 in Ohio. Caused an outage if I remember correctly.

The excitement I am feeling, unmatched, new clabretro video, let’s go

AHHH, I love this! When I was working, (part time) when I was 12, the company was using X3650 M4's for their file servers, and even Active Directory :P on Server 2012 R2, in 2018... It's a small company but janky... They got a Synology iSCSI with only 1 PSU running, which was like damn... Great video btw. Awesome job.

Never have I felt smarter than watching you take the tab off the LPD, thinking 'you should cut the ramp off the latch and put the tab back on as a puller' then seeing you do exactly that haha

For that gunk I probably would've chanced an "Adrian's Digital Basement" style wash with dish soap, water, and a very soft bristled brush for that mainboard, then an IPA dunk for drying. Yikes! Either way, nice cleanup job, that's a big board with a lot of things to go wrong.I think the theories of a datacenter fire are likely true, either that or a long stay in a stack in a humid environment with a lot of other crap in the air (my first thought was also that that grime looks a lot like auto shop/garage grime). Very happy to have this channel recommended to me by the YT algo (it gets it right every once in a while!). Excellent mix of expertise, curiosities, and that liitle bit of jank that every good homelab should have.

The dirt in it makes sense to be related to a dealership, when I saw DMS at first I thought Document Management System.

I had been feeling like this machine had spent some time as a homelab in a basement or garage from the looks of the residue in there, and I feel like an auto plant (or even a larger dealership) tracks with that, like you're seeing a decade of buildup from oil changes and radiator flushes.

Loving the content. As a former techie that now rarely gets to touch the hardware part of the environment, this really is the stuff.

Looks like there was a datacenter fire, and the rust on top was from sprinklers.

Man, I love enterprise gear! Everything is so thought through and modular.

That looks like 1000 people smoked a pack of smokes near that server. ACK. Great video

Man, that server was grungy! I've pulled a few servers apart that had a little bit of that filth with close to 10 years of service, but nothing like that! Wow!

thats the younger brother to systems we still have in daily prod, yes the support contract is expensive to cover them. we have x3550s I have in my rack although i rarely power them on to noisy and slow compared to my other old junk, also have 3 dell pweredge6600? quad xeon cpus (still 32bit) and a few p3 era 1u servers. preiously had some old proliant and a p90 prosignia. my favorite server i have ever had though was an ibm netfinity pentium pro200 with 4GB ram for its age it was a decently quick machine

perfect timing for my food

In 2015, I got 3 of this machine and install Proxmox VE 5 (I guess) at that time, to create a Proxmox Cluster. It worked pretty well till recently.

I love the industrial design of servers

Great video and channel! The m4 series boxes are great. I have a x3550 m4 (the 1U version of this box) and it’s running great to this day.

When cleaning old boards, I like submerging in hot soapy water and letting it soak for a while, and then scrubbing with a toothbrush. Then I give it a bath in alcohol while hitting it with a toothbrush again. For rust, I sand what I can off and then dump some vinegar on it and clean up again with alcohol. After it dries for a day or so, it's good to go. I'm not sure how that would fair with that black gunk though.

6:14 That is the feature enable hardware key for the IMM advanced license. Without that, you don't get access to some features in the management system, I think including virtual media support. As for the BRD error, I had that with my 3650m3..... I spent days working on it until finding out that one of the risers just wasn't all the way into the slot. It damn near SNAPS down into the slot. When you think you have it in, then *lean* on it more. Well. You have part two up, so I'll have to see if you figured that out by then. :)

Recently got one of the 3650 M3 versions, interesting seeing the differences and similarities, mine did have a hyper visor key in that USB slot on the rider so that is what it’s for

A fun new video from the handsome Clabretro guy! Seeing the grime in that system makes me glad I don’t work at a dealership. Imagine breathing that every day. Now I know what that dealership smell is. Regardless, thanks for sharing. I’m curious if you’ll be able to resurrect it. Maybe this system failed while in service and they moved it to some leaky storage area before disposing of it. Your research showed what looked like the name of a Chrysler dealership. Could ADP hunt them down for not destroying the drives? Hmm. Suspense, drama, intrigue!

Careful with that brake cleaner, it eats rubber / plastics. I believe the ABS sensor safe versions are better for this but I would still avoid it near electronics.

yay a new video

I've had quite a few IBM systems over the years, and had a few of them blow transistors on the power rail. So that would be my guess of what happened. Check the board LEDs to see how far the power gets and do a smell test.

We have these servers (and their predecessor x3650 (without an „M“ designation) and successor Generations M3 - M5) at work. I used to service them but apart from swapping Hard Disks, Power Supplies, RAM, CPUs, PCIe Cards and the occasional SAS controller I never got to taking one apart this far. Thanks for taking us on this journey! In terms of diagnostics you might check the MM (Management Module) of this beast. It‘s IBM‘s way of Lights Out Management and if you manage to access it it might provide more diagnostic output than the LEDs. It should have a Web Interface (terribly slow and not exceedingly informative (you may want to check the event log, though)) but also telnet / ssh access which lets you execute diagnostic commands. As a firefighter I‘d say that gunk is very probably smoke particles of a major fire incident nearby and if water was used to extinguish it could explain the water damage mainly on the outside, too.

Great video

At least it looks way better than my Sun X4500, that was in good condition, when ordered, but then the delivery guy let it fell from a pallet , even cpu socket was ripped off

Such a dog to work on, you have my sympathies mate!

Next time if you encounter a such dirty motherboard, just put in the dishwasher. Before that remove the batteries and drain any leftover electricity from the board. Dont use any detergent, just water. Without any electrcity in the board it wont corrode. Der8auer use this method after his sub-zero OC sessions to remove vaseline from his boards and cards.

I used to deploy these systems... My main gripe was always the absolutely ridiculous BIOS boot time

You ever thought about getting one of them electric cleaners? Would be good for the smaller boards and things. They really clean things good. They probably have ones for the bigger board but I am sure they are very expensive. Or put it in your dish washer (I hear people do that with motherboards). Also I hear BW100 is great for cleaning electrical stuff. It's non conductive and drys like IPA if not faster.

I used to own a X3650 M5 and that steady power light blink is normal...if the server is "not ready to power on" it blinks faster at you lol...BTW the IMM messages you saw are the IBM version of Hp ilo Dell Idrac etc. IMM needs to be happy otherwise the fans go full tilt like they did in the video. if you plug it in and wait about 3-5 minutes or so for IMM to fully boot it shouldn't blast off like that-that would be something else to check try to connect to the IMM webUI it think it defaults to DHCP...

20:22 - the connector is there for reasons same as the USB one (and this is likely SAS not SATA) - internal tape drives usually come with SAS (LTO and stuff like that) or USB interface (for DAT tape drives).

This thing probably was in the shop area of the car dealership (probably mounted on a rack that hung from the ceiling) so this is malaise era Chrysler dust

Got 2 R410’s one was a little dirty the other one was something like this machine but without the rust.

This has more than a slight resemblance to the Lenovo RD220 which is "the server" where I work. Turns out it wasn't being used for much so I shut it down this year and nobody was affected. As part of a Wi-Fi upgrade I then installed a same-generation HP DL320 G6 with Proxmox to run the UniFi controller and an OpnSense instance. These old servers run the exact same software as new servers so I don't consider them retro. If they supported software (or hardware gizmos) not usable on modern systems then I would call them retro. Actually then I'd call it vintage but whatever. It's nice to see stuff come back from being junk no matter what you call it.

ADP. Dealer Management System. It is car dealership server managed by ADP Dealer Services. ADP DMS would be the "IT department" or supplement support for the ADP hardware like in my case for the in house IT I used to manage the worlds largest dealership deployment back in the day. (98-03) Had cool toys like Robot WORM drive storage. Vertical conveyor belt/elevator storage for car parts. (like a giant Carvana vending machine) WRQ Reflection custom programming.. Y2K... I met Ralph, the HMFIC of ADPDS at the time. It was fun times until they gave the IT Director job to a buffoon con-man car salesman.. Back stabbing industry. Almost every conceivable function of a car dealership/service is ran from a server like that. We used a dual CPU DEC Alpha and some Unix to run over 1.2k VT200 terminals for all the individual dealerships we managed. From sales/accounting to the technicians looking up parts in the service department. Multiple locations.

the systems are always cool but i do agree that that is a bio hazard! Great video!

Evaporust would work great on that big metal sheet.

if you want a way to protect the steel after rust and cleaning has taken the plating off, and don't want to bother with painting, get a wax-based anticorrosive coating (like boeshield). most of that class of products have matte finishes when dry that won't look too weird, beside the visual texture of the stripped zinc plating.

6:15 That is an xclarity license key and may be the only part worth anything from that machine. Good luck!

I have one of these minus the ADP branding in working condition heading for the scrap heap in 3 weeks time. If there happens to be any parts you need I'm sure I can arrange shipping.

I've pulled stuff from a sewer before, but this server was just dirty. What does an ADP datacenter look like? :) Great videos by the way, I'm hooked!

Since this server is the nastiest I've EVER seen. I didn't catch on on vid...but check CPU the Xeon itself and CPU socket 1 for gime-ness lol

Brakes, Tires and Lubes. Each having their own effect on air quality in an environment such as a mechanics shop. Used oil even when it is cold has significant off gassing of everything from micro particles of wear parts metals, oil additives to unburnt gasoline. anti-freeze and coolants evaporate. brake cleaner rapidly evaporates let alone the break pad dust. tire rubber new and used off gasses constantly, and the anti sun deteriorate chemical has its own special kind of funk. exhaust soot. heck even rust dust. and to top it all off the normal skin, hair, and dust from outside. If I can recall there was a university study of mechanics shops and the effect on health it has for the mechanics. it can take off 10 to 15 years of life working a mechanic job. Even a lube tech, tire tech, even service advisors are affected. fresh air is mandatory.

It looks like it was in a data center fire and the damage on the case looks like it got hit by water instead of halon like most data enters use

Maybe it suffered from "basement/warehouse damage" high humidity enviroment, atleast I'd think that real waterdamage would look worse? Also for next time cleaning a dirty machine: Nitril gloves, the ones for carmechanics that offer extra grip, don't wanna risk a cut from sharp metal with all that grime on your hands.

i just use a hammer each time my server bricks it self surprisingly it boots right after.

YIKES!!!! Looks like it was used in an auto shop and was stored in the back room under the engine cores and used oil tanks! That's alot of filth and schmutz!

I would have tried to JB weld the button for the light path module

Typical IBM. When they work, they almost bullet proof. But when they break, it's ugly. Beautiful machine.👍

Yup, 13 years ago feels like yesterday. When I had hair.

I have a slightly older 3650 that complains about the raid battery being dead and wont boot past that even with raid disabled. It is aggravating as a battery is more expensive than the thing is worth. .

Yea ADP and the Okidata branded ADP printers too. If anybody wants to go down a costly rabbit hole. The upside was all the equipment, software, etc etc. was already per-configured to plug in and mostly go. But, if you was just to purchase the hardware separate in some cases. It was $$$ less. If they would give you that option. I don't know about ADP but most of those type of systems started out on Mini computers and moved into x86 systems due to cost saving. Well for the company mfg. the software / hardware. Not really towards the end customer.

This could well have been the result of a building fire. Some fire suppressions can leave residue dunno though obviously

Much to surprised, this was quite older.

When doing restoration like that it is good to use vinegar and baking soda

Is it possible this server was in the vicinity of a fire, and maybe got splashed with water? Or has someone just run it in their garage for a few years?

I bet if you would turn it on before cleaning it would work flawlessly :)

👍👍

Bling-Bling. That was so non-IBM-like thing to see. :) Funny

what a beaut of a server! Is using contact cleaner safe on CPU sockets? Those things are so fragile, anything will bend a pin. Regardless, nice to see IBM including those handy little diagnostic lights, it's just cool to have for what ever reason even when the system works.

what is the blue part next to the battery?

These ADP servers used to run a software called DMS which was programmed in I think Kerridge KCML which was a terrible programming language.

Have you removed the CPU and cleaned beneath it? All that washing gunk around probably washed some into the LGA socket... something might be shorting out under there.

Other than the LPD providing a code, doesn't this model have a management port and interface? The default userID and password can prob be reset.

Insurance Loss Adjuster here.. This machine has been in a fire and then sat for a month or so in a place with no or partial roof.

Water damage and a black soot like dust?…..perhaps it was in a building that had a fire ?

This server has seen some shit.

I start looking for part two and then I noticed that this video is just one day ago. I mistakenly overlooked and thought it was one year ago. Crap!

Looks like it was running at a dealership. Their server closet probably was just set it in the back with the mechanics where theres soot and brake dust all over. Probably a carcinogen.

I got one HP proliant dl 360p gen 8 and 2 Proliant dl380P gen 8 but rgey say that IBM is more easy to work whiy i dont know but i will get one some day...What do people do whit there computers/servers????....where have this server been ? in a factory?...brrrr....do people know abiut taking care of there servers? clean them etc...they work longer if they take care of them..By the way do you know about ILO? My 360 does not find the ILO and the IP adress and i have try every thing but stil and i am not a technican so i dont know what to do or search...

It must of been in some sort of Fire

IBM has always over engineered their hardware and it is always built like a tank. Can’t say that about most server hardware these days.

Hi @clabretro, i have a clean and running x3650 M3 System, if you are interrested. The only downside is the shipping from germany^^"

that server might be toast with a error code of e7 not shown anywhere while there is a possibility of voltage regulator has failed

usually they are dusty this is just gross.

looks like fire damage.

shocked it only had a single CPU and 2 1gb dimms. ddr 3 or 4 at least? :p

9:39 not the connectors😩

My guess would be smoke/melted plastic particulates that get sucked up by fans during a fire. Horrible stuff.

almost looks like its been in a fire

look like it was in a fire

take contact,i have system x experience...