What a shame Sangoma has fallen so far in the last 5 years. I worked with them quite a bit in the past, but before the core of the team moved to Clearly. When someone tells you who they are, you should listen. No CVE issued, deemed a 'minor' issue, nuked the bug bounty program, refused to pay out, and sparse details about what was patched (and if it was even successful). Add in the hardcoded 'login' password bypass for authentication because they can't be arsed to implement authentication on devices THEY MAKE... what an absolute mess.
As someone who works in Digital Forensics and Incident Response, this is an awesome video! I applaud your transparency and the way you explained this attack and your response process. Good communication is the most critical part of response to any incident, and, for what it’s worth, as someone who does this day in and out, you did great here. Keep it up!
Thank you for not only outlining the problem, but offering a solution. You have demonstrated in your videos that you have a lot of contacts in the VoIP and open source community that you could coordinate this project and not let it die.
I would love to see a video from you showing how you use and setup Grafana and Zabbix to do this type of monitoring. Thanks for the detailed video of how this hack went down.
Good Work Chris! This was well presented. I actually found that exploit video back in September and had looked into that. I had locked my edge firewall down so tight that I was having trouble with my Trunking Service. We know who they are because you love them. So I had to take a different approach. It didn't take much but let me know if you want some info on how we stopped the SIP attack that happened with us almost immediately. If you're interested I'll try and send you an email about it. I have been trying to spread as much information about it as I can and even provided some instructions to our trunking service for suggestions to their customers on how to better secure their PBX internet connections. Thanks for removing my previous comment....after posting it I realized that there might have been a little too much information in there. Keep up the hard work! The information you provide is worth gold.
Sangoma has been having issues with their Fax Stations. This issue causes the fax line to silently hangup and it acting like it was it was making the faxes successful but very much wasn’t. We got with Sangoma a few times with many examples and they did not believe us. We had all of medical clients effected. it took our owner threatening them with dropping them for fax and then they acknowledged they knew their was an issue.
You have to understand that the faxing through Fax Station is far different than traditional copper lines. It's never been secure enough to compete with traditional faxing, and honestly it made me scratch my head as to why anyone would want it. All these issues that people have had with Fax Station is never going to go away. The technology is just bad in general, and to be honest it's always better to have a traditional dedicated (copper) fax line just for that one job. Ever hear of a traditional fax being hacked? Yeah me neither.
I would love a video on how you are monitoring the servers. I'm a big Grafana and Zabbix fan so would like to see what you are monitoring, including what you monitored to catch the hacks.
Yea - it's pretty interesting, and it relates to some custom stuff we had built. The FreePBX firewall (which uses iptables at its core), when disabled does not stop the iptables service, so you can't just monitor iptables status - that's not accurate. But, there is a huge difference in the number of iptables chains/rules when the FreePBX firewall is on or off - so we do a count of those iptables rules, and if it drops below a certain threshold, that creates an error condition in Zabbix/Grafana. There's a bit more to it than that, but that's essentially what we're doing.
How did you think to add that monitoring? That's definitely useful now that we know about the exploit, but I never would have thought of it beforehand. I would love to see a "best practices" video where you talk about the system-level signals that you monitor.
Chris Thank you for being transparent in this. It opens an eye on how Sangoma is treating Freepbx. I would love if you would do a video or two around the two monitoring software you use to monitor Freepbx and other items. Cheers Richard
Since FreePBX is actually competitive to their other products and services that create revenue for them, what Sangoma is doing makes perfect sense; they are slowly but surely killing off the competition...
Good video, TY. It's clear that unless something changes, your only option is to walk away from the product. Sangoma will of course say that you can just move to a paid product, but all the rest of us know, that most of the users utilising the free version, are not "cheap" enterprises/businesses, but rather small entities with no or little money, and few options, so they can not do that.
How are you gonna leave us on this cliffhanger for 14 more hours!? All jokes aside, hope everything went okay. I manage a FreePBX server so I’m a little nervous now to find out more.
Been on FreePBX since 2018; 2 installs, 1 being v14 and the second being v15. Can't believe it's already been 4 years since we had a proper full release and just makes me nervous about the future of my company's phone system. We came from a 1997 NEC system, so even if we keep this one going for 30 years, we'll live, but jeez.... Have you done any looking into the new UniFi Talk service/system? I wonder if that's any good...
I have long wanted to setup a PBX system, I have watched all your videos on it and am pretty convinced FreePBX is the way to go I love open source software and now I might finally have a reason to set one up. I really don't want to see FreePBX die... I guess it can always be forked
You are absolutely right!!! After more than 20 PBXact ot FreePBX projects We at my company realize that sangona don't want frewPBX to grow up!!! Sad, but there are other projects growing fast!!!
The whole removal of the bug bounty confuses me. Wouldn't you want to know of vulnerabilities in your software before a bad actor finds it and either abuses it or sells it on?
It is like offering a reward to someone who finds your puppy, then when they bring the puppy to you, you don't want to pay....you better not lose your puppy again!
Why do these systems have any exposure to the public internet at all? Is there not a better way to maintain/manage these systems? Allow-listing IP addresses is a step in the right direction but I would think zero exposure is a better solution entirely.
Agreed - which is why so few of our customers were actually affected. We prefer ZERO open access to the outside world (perhaps just locked down for the SIP trunk) - there are many ways to do this including FreePBX's built-in VPN. But there are just some cases where customers have to have ports open - such as when they have a large work from home user base who are on dynamic IP addresses. In those cases, the Responsive Firewall is really the only line of defense.
With internet exposure, you should lock down the pubic facing services in Linux with either systemd security primitives or containers, this will protect the host and limit the damage even if the service is exploited..
@@CrosstalkSolutionsI can’t see a reason why they would not just use a service like DuckDns so the IP could freely change but still be updated. I don’t miss working with Freepbx. Also limiting the attack ability with services like Unifi’s by geo-blocking all countries excluding the US limiting an attackers entry point to data center servers and private residential and mobile IP addresses. That is what I do at least and just VPN outside if I need to access something from another country. The amount of alerts I get from port scanning alone from Russia, China, India, Africa etc could easily crash a notepad log file. I love the video btw and have been watching for years. Possibly one of the largest reasons I stuck my toes in the MSP world. I would love to see a collaboration with you and @rossmanngroup on getting Freepbx actually open sourced to the fullest potential. I have been thinking about picking up some of Unifi’s phones just to mess around with a buddy, have you had any good experience with them?
Figuring out the chronology of a breach is a real challenge sometimes. Sometimes impossible tbh. And once people start to think they'll never figure it out, they no longer do their best work. But without knowing the chronology you can't always identify the hole you need to plug
I have to address the elephant in the room: is it wise anymore, to have sangoma based systems then? Or would it be better to switch to something else. I mean the “bounty policy” of sangoma seems to be messed up. That researcher deserves his reward and pay out for that
I note you say you only allow connections from authorised IP addresses (on the LAN?) but that this was not sufficient to prevent access. Any idea how this happened? Im wondering if an associated issue with some small business routers cooperates with FreePBX here as the firewall on some routers allow any incoming traffic on port 5600 to cross. You can see this in iptables rules if you have an affected router.
Good you guys did not suffer much with that cyber attack but something i dont understand you guys have the pbx machines at least hardened and behind a firewall before this correct ?
if i understand this correctly, this hack happened to some PBXs in the cloud, correct? now the question is, are local PBXs also vulnerable? i mean, on my local LAN it would be much harder to do this hack than if the PBX is somewhere out on the internet, right? the way sangoma handled this raises loads of big red flags. what PBX software would you recommend now, after this hack?
It seems to me based on the hacker presentation like they would have to have access to the actual php page in order to do this. If a spoofed phone or actor could get onto that network and call the code where the php webserver is hosted then they could execute the exploit. I guess if a user was compromised who had access to the webserver or could spoof an IP on the network then this would also be an issue. (I'm doing a lot of guess work here so take all this with a grain of salt).
Well....they were. If you trust Sangoma and their fix for it! The vector Chris stated was through a phone app on FreePBX that hosted services on the network to phones. My guess is it was usually used to Provision Sangoma phones. The password for those phones (unprovisioned) was simply "Login" so all you had to do was find a MAC address (NIC Serial Number) that was actually registered with FreePBX and you were in with a few lines of code in php added/modified. All security on the FreePBX server could be bypassed because of this. The best thing to do is restrict only the ports you need on the internet (SIP port, and TLS) and block everything else. And on those ports exposed to the internet you should have IPS, SIP blacklists, and GEOIP filtering guarding them. In my opinion you should never just "trust" another companies product anyway. To directly answer your question. Yes, It COULD be harder. It all depends on how you have your LAN setup as well. If a local device on your LAN could be compromised then no this would be easy. This requires a fair amount of networking knowledge to secure. In general though you are probably fine unless you have things on your LAN hosting things out on the internet. Just be sure you are NOT publicly hosting the provisioning part of FreePBX on the internet and work on Securing those ports as mentioned above and you SHOULD be fine.
Can you do a video on how you configured zabbix to monitor the firewall and your ssh keys? I been looking for a way to do this but have been unable to find anything.
MAC address aren't really relevant once you cross a router. The phone must be passing their MAC address as a parameter in API calls...about as verifiable as the user agent.
My question is. Are the freepbx servers on prem, cloud hosted or crosstalk hosted? If they are on prem, then what host was used to connect to the freepbx servers? Then that host is most likely still compromised! And they are not in the clear! Plus how was that host compromised? Your clients need to have a full IR performed, if it has not already been done. My thoughts anyways! Hope you'll are able to completely remedy the intrusion. It does not seem that the freepbx servers were the point of entry into the network. G'Day
Im not sure i fully understand. I have my pbx behind a firewall with the only incoming port 5060 open from our sip trunk server. Was i still vulnerable?
I figure this would happened at one point or another, Freepbx has so many modules available , they have too much development going on with little extensive testing for security vulnerabilities.
I'm still curious how this hack occurred and targeted networks with the FreePBX server. In order to get the MAC address of a phone registered on the system wouldn't the hacker need LAN access?
Well yeah, except one other problem......as stated in the video by the guy that found the exploit. Some of the code is NOT opensource. This means it's not a complete opensource project. I'm not even sure it would work without that closed code. BUT!....if someone picked it up and just hired developers to look at it and take care of what's needed they might be able to just....make an alternative to Sangoma's version of it.
@@JonGeorge-j6h okey as you say: You will have to figure that out and see if it is possible to work around that. I though it was as opensource as PFSense for example and that has been forked.
Sounded like there is some closed source content, so perhaps not the simplest task… The bigger issue with forking this, is you have to spread the word to those that are using FreePBX, so they migrate away. Much easier to take over the project & provide new releases, as then existing & future users benefit from a properly maintained product.
Isn’t this exactly what IncrediblePBX did? Seemingly based on FreePBX but a lot of the commercial modules are stripped out but with other security minded features added in.
Why don’t the original developers release a better version of something similar to FPBX. Giving this project to independent developers in effect would be direct competition to their cloud based systems and servers.
It will be a great thing for sangoma to do the needful immediately and I'm sure there's turn of support that will be gotten if it's let go to open source community
Chris, I hope that one day in the near future clearlyip get the freepbx project ownership back. Also they need to get it on a red hat distro like Rocky Linux
The original creator still owns his code for FreePBX. Even the code he wrote for Sangoma. He's just really not wanting to walk on Sangoma as the came to an agreement a few years ago over the signing key for FreePBX. It's just easier not to kick a hornets nest. Hopefully Sangoma would consider giving it back to him or selling it back to him.
why has FreePBX dropped all development?!??! and lettting things like other PHP vulnerabilities and such.... makes me wonder how many other zero day exploits are active in the software??? I am going to watch that DEFCON vid next... I hadnt seen that one.... I love the Defcon confrence sessions... There is so much cool stuff people are able to do and figure out.
sounded like a $$$ issue with maintenance and they want you on the paid version... though if they get a bad reputation I could see it hurting their paid part.
A true ethical hacker would not have publicly disclosed a vulnerability as serious as that until it had been confirmed it had been fully patched. What he did there was unnecessarily put your customers and other in the position of being targeted by the malicious hackers with a known working exploit.
@@davepusey and if they still not update? There are researchers, that had problems, because they reported a vuln, and they didn't publish it. Iirc more than a year later, they were raided by police, because the vulnerability was used, even if they didn't publish anything, and didn't use it.
Please do a video on how they got into your system, even with all the firewalls. what was configure wrongly? Did your firewall also have a zero-day issue?
This makes me wonder...Is development work still being done on Asterisk, the underlying engine of FreePBX? I mean, I see releases happening, latest one was July 2023, but how "motivated" are they to "improve" it? I assume Asterisk is the underlying engine of their cloud and paid products, which would tell me at least that they are motivated. Just curios (I got my feet wet in this field hacking asterisk files many years ago....freepbx has been a god send to make configuration much easier and faster).
Asterisk is the engine for all of their commercial products so it is the one thing that does get focus. Note that focus is centric to what has a proper business case for any of their commercial solutions.
That's really messed up that they changed their bug-bounty program after he found and notified them about a problem with their software. Just going to make them a target...
It would be better to use an SBC (Session Boarder Controller). SIP proxies seem to cause more problems than they solve. You might also look into DPI/IPS (Deep Packet Inspection/Intrusion Prevention System) systems on firewalls. They look at network traffic and block or notify bad stuff based on rules you setup.
Why was the bug bounty program pulled? Your reasoning for ClearIP for assistance instead of Sangoma is probably a clear indication of why they are scrapping their Bug Bounty program.
Passkeys are great toys for people who don't use Linux or Chromebooks. If you thought carrying around a Yubikey was bad, now you can be tethered to your smartphone. Sangoma? FreePBX? Stewardship. I would work for your venture. [If you knew me you'd understand that's the highest compliment.]
Easier said than done - in order to fork, you'd have to remove any reference to Sangoma, FreePBX, etc. and also unwind any development related to their commercial modules. Plus, you'd lose the branding.
@@CrosstalkSolutions This is true Chris, but have you possibly considered asking the community to help you do this? Vates did this with Xen Server and created XCP-ng which had phenomenal financial support from the community. Would you be willing to take this on if the community was behind you on it?
This is what happens when money becomes more important then the product, you ultimately kill the goose that laid the golden egg. Greed destroys so many companies these days. They need to be aware of potential reputation damage, look at Unity as an example. If they allow free pbx to decline under their own label people may stop dealing with them full stop. Better separate the two now or add more development to it.
Open source, open standards, publishers that give out the source code for free but yet want to make money for their work...this hot mess makes you appreciate them 1A2 key systems huh?
This is one area I do not see anything else close in the open source community to replace the open source asterisk and the only real feature complete version of asterisk is from FreePBX. Seems to work good but it looks like a dying project.
In my opinion they were acting that way back when the creator of FreePBX left Sangoma for similar reasons. Even worse is how they treat people on their forums looking for support from the community.
