Coleslaw Posted July 6, 2018 Share Posted July 6, 2018 Hey everyone, Over the last week, we've been investigating an intermittent but extremely noticeable site delay that reached a tipping point on June 26, causing massive outages. On initial inspection, the server wasn't anywhere near capacity and all logs reported that everything was fine, when it clearly wasn't. After a couple general fix attempts (a full reboot of the server, cleanup of old files), we were left with the task of removing individual services to see if we could isolate the cause of the outages. As some of you have certainly noticed, features have gone missing or been unavailable (avatar uploading, hot-linking, ArrowChat) as we tested the entire system from top to bottom. We're in the process of reenabling those services now. Currently, we suspect the culprit was a malware file uploaded to our Elastic File System (EFS) bucket, camouflaged as a JPG file. It overloaded the asset server which snowballed through the rest of our ecosystem and caused the server to randomly stall out, waiting for the EFS. We have removed the file and restructured our server so that we no longer rely on EFS; all assets are now stored on the main file server to withstand load at scale (our file server and database system are still separate for security purposes). Additionally, we've installed fresh versions of IP.Board and Drupal and additional malware detection software to ensure we're in top order. As with any malware, we recommend that users update their passwords. Even though this file did not have access to our database, we cannot be sure what its purpose was and recommend all users update regardless. Thank you for your patience, especially now during Pride. While it's nostalgic to have the server crash when we're receiving an influx of new visitors, we'd much rather be back up and running. We'll be monitoring closely over the next few weeks and will update with any additional info we uncover. Link to post Share on other sites
Homer Posted July 6, 2018 Share Posted July 6, 2018 Thank you, Cole. You rock! Link to post Share on other sites
SkyWorld Posted July 6, 2018 Share Posted July 6, 2018 Woo! Thank you so much, Cole! Link to post Share on other sites
LeChat Posted July 6, 2018 Share Posted July 6, 2018 Thanks for the update! I'm glad it was figured out and resolved; I even tried asking a relative who knows a bit about computers whether they had an idea of what the problem was, but they said it was above their skill level, computer science stuff, which they didn't major in. Link to post Share on other sites
Nanoic Posted July 6, 2018 Share Posted July 6, 2018 I have noticed since yesterday AVEN began operating normally again. Congrats to the team for fixing the various problems. I changed my password. Was this jpg containing the malware uploaded accidentally by a user on the forum? Link to post Share on other sites
michaeld Posted July 6, 2018 Share Posted July 6, 2018 I'm so glad this is now fixed!! Link to post Share on other sites
Snao Cone Posted July 6, 2018 Share Posted July 6, 2018 Thank you so much Cole, for the fix and the explanation! Link to post Share on other sites
Tanwen Posted July 6, 2018 Share Posted July 6, 2018 Thanks Coles - awesome as always Link to post Share on other sites
kelico Posted July 6, 2018 Share Posted July 6, 2018 Thank you, Coles!!!!!! You and the tech team are the best!! Link to post Share on other sites
FaerieFate Posted July 6, 2018 Share Posted July 6, 2018 6 hours ago, Nanoic said: I have noticed since yesterday AVEN began operating normally again. Congrats to the team for fixing the various problems. I changed my password. Was this jpg containing the malware uploaded accidentally by a user on the forum? As a follow up, I'd like to know what webmasters have done to prevent further files from being uploaded and disguised as jpgs. And I'd like to know where it was uploaded. Posts and signatures don't upload pictures. So the only places where AVEN is uploading is profile pics and images webmasters use, from my knowledge. So did a user upload this as a profile picture? Because in that case admods or BOD need to get involved, or did someone use injection? Because then we may not be able to trace it back to someone. Thanks for fixing it, @Coleslaw. As always, you webmasters are the best. I just want to confirm that it can't happen again. Link to post Share on other sites
AndrewT Posted July 6, 2018 Share Posted July 6, 2018 changing password would be good if it was actually a password i remembered in the first place..... a two factor sign in would be good to protect accounts (although it's not like any payment details are stored) Link to post Share on other sites
Coleslaw Posted July 6, 2018 Author Share Posted July 6, 2018 2 hours ago, FaerieFate said: As a follow up, I'd like to know what webmasters have done to prevent further files from being uploaded and disguised as jpgs. And I'd like to know where it was uploaded. Posts and signatures don't upload pictures. So the only places where AVEN is uploading is profile pics and images webmasters use, from my knowledge. So did a user upload this as a profile picture? Because in that case admods or BOD need to get involved, or did someone use injection? Because then we may not be able to trace it back to someone. Thanks for fixing it, @Coleslaw. As always, you webmasters are the best. I just want to confirm that it can't happen again. The file in question was uploaded as an avatar; there is no way to trace it back to a particular account because it was not being used (uploaded and then replaced), thus there's no associated database reference to it. Our software checks for filetypes so if something says its a JPG, the software allows it. The only way to completely eliminate this risk would be to disallow avatars entirely; we already severely limit their size and don't allow off-site linkages. Additionally, in moving our assets off of EFS (a completely separate AWS system) onto our EC2 file server, we now have everything running under a malware scanner that quarantines suspect files. The risk of these kinds of attacks are low; the uploads folder (where avatars end up) is the only publicly writeable folder in the entire forum software system. The issue for us was the load issues it caused the asset server, which caused our server to stall out. Link to post Share on other sites
Goonie Posted July 6, 2018 Share Posted July 6, 2018 Wahoo! Excellent detective work. It's never the easy option. Link to post Share on other sites
FaerieFate Posted July 6, 2018 Share Posted July 6, 2018 2 hours ago, Coleslaw said: The file in question was uploaded as an avatar; there is no way to trace it back to a particular account because it was not being used (uploaded and then replaced), thus there's no associated database reference to it. Our software checks for filetypes so if something says its a JPG, the software allows it. The only way to completely eliminate this risk would be to disallow avatars entirely; we already severely limit their size and don't allow off-site linkages. Additionally, in moving our assets off of EFS (a completely separate AWS system) onto our EC2 file server, we now have everything running under a malware scanner that quarantines suspect files. The risk of these kinds of attacks are low; the uploads folder (where avatars end up) is the only publicly writeable folder in the entire forum software system. The issue for us was the load issues it caused the asset server, which caused our server to stall out. Cool. I figured you had it under control. But the programmer side of me had to make sure vulnerabilities were limited. Dunno what the file was, but some people do intentionally try to crash websites as a "hack" though this is an entirely unusual way to do that. But I'm glad we have such smart webmasters! Since admods recently added groups, I was worried groups was overloading the servers or something! Link to post Share on other sites
FaerieFate Posted July 6, 2018 Share Posted July 6, 2018 3 hours ago, AndrewT said: changing password would be good if it was actually a password i remembered in the first place..... a two factor sign in would be good to protect accounts (although it's not like any payment details are stored) The concern with passwords is people use the same password for multiple accounts. So if you used a password for this account and an account on another site that account would be vulnerable. There's more complex vulnerabilities as well, but I'm not gonna go into that as... well we don't want a "how to try to hack sensitive accounts" tutorial out there. Link to post Share on other sites
Lia Posted July 7, 2018 Share Posted July 7, 2018 3 hours ago, FaerieFate said: we don't want a "how to try to hack sensitive accounts" tutorial out there. You ruin all the fun Link to post Share on other sites
FaerieFate Posted July 7, 2018 Share Posted July 7, 2018 3 minutes ago, .Lia said: You ruin all the fun I know sorry, but I don't want AVEN getting in trouble XD Link to post Share on other sites
Guest Posted July 7, 2018 Share Posted July 7, 2018 Hooray!! \(^O^)/ Link to post Share on other sites
Tercy Posted July 7, 2018 Share Posted July 7, 2018 My intuition is that if you need to have a malware scanner check uploaded JPEG (or any other image format) files, you're either missing something much easier or doing something very wrong (e.g. a questionable server configuration). If everything is programmed and configured properly, image files should be no threat. Link to post Share on other sites
Clumsy Fairy Posted July 8, 2018 Share Posted July 8, 2018 I am puzzled by the whole thing too.. How on earth is an image executed? Surely they would have 660 permissions, and the web software just points to an image, it doesn't try and run it in any way. If you are saying that the invision software runs images in any way, then that REALLY needs to be reported. Link to post Share on other sites
Recommended Posts
Archived
This topic is now archived and is closed to further replies.