Cracking the Interweb
I work on campus at BYU. Well, one of my jobs is on campus at BYU - I do the other one (and potential other ones) from various locations to include on campus and at home (basically wherever I can find a high-speed internet connection).
My job on campus is supposed to be a webdeveloper position. Essentially, I come to work and help in the creation and display of websites. I don't mind telling you all that I felt qualified to make changes (I still do), but I didn't feel a whole lot when it came to creating pages from scratch. I don't see myself as possessing the visually creative gene. Some people do and can create dynamically appealing websites; I, on the other hand, can create functional websites with (what I consider) very little visual appeal. S'why this site (and others) remain so... BLAH!!!!
I do like knowing something about CSS though.
I also like the fact that my boss will push different non-design related things my way - and he is patient in allowing me to do what I can (and am comfortable) do within the group. What I can do: create new pages from old templates and research issues and provide (possible) solutions (often implemented).
One problem that presented its ugly head (last month) was the main site being cracked. Yes, I am tempted to post the site I work for on here (and no it is not the main BYU website), but I won't. The crack on the site was pretty benign, though a lot disturbing, which inserted a lot (read thousands) of invisible links into the index.html page. What this does is create a positive improvement for the receiving site (Google, MSN,
Move forward in time. Yesterday, when one of the coworkers got into the office, the site was down. Specifically, it had gone down around 7:15 a.m. Mountain Time and had stayed down until 8:30 a.m. Mountain Time. At which point the coworker along with our webhost discovered that the Apache file had its permissions changed to block anyone from reading it. Permissions are shifted and our site is back up and alive. Enter me. I start to look into the crack. To do this I needed to come up with a hypothesis about what caused the downtime and permissions change. Since I have a little bit of experience with computers and the internet (I am amazing on paper) I knew that the Apache file was not something a regular joe could get at and manipulate; therefore, I postulated that someone or something internally had to have access to it and lock it up. Before I could test that, though, I had to test under what conditions (and through what programs) the file would have its permissions set wrong. Once I established the permissions that I needed to replicate I proceeded to launch and run a variety of applications that had access top that particular file. Pretty much, you have to have account Root access to get to the file which means there are (about) five people who have access to it (right now). Well, six. Anyway, as I stated I started with a hypothesis and proceeded to test against it. The problem I (almost immediately) came up with was that I couldn't replicate the problem. Nothing I did would cause the permissions to even hickup let alone change and, as a result, I decided to look into other aspects of the site. One of the initial changes I made to the administration of the site (upon employment) was logging access by visitors. What this does is create insanely large access logs on the server in the terms of tens of thousands of lines that, when cracked, you have to go through. Knowing when the crack ended allowed me to do a couple of things: first, to look at various web analytics systems the school uses and pays for to see how website traffic was affected by this crack and when that affect took place; and second, moving backward through the log files from when I knew the crack ended (there are error codes that indicate whether a person was successful in connecting to the site or not). This allowed me to establish a) time the site went down (to the second); and b) time site came live. More, I could look at who had connected to the site (and how, from where, and with what) and see what they were doing just before (and trying to do after) the site went down. Since I knew that the file had been altered at 3:30 p.m. on Friday, and I knew (from having done it) that the site was accessible until 5:00 p.m. on Friday, and the analytics software indicated a lower-than-normal visitor trending over the weekend (possibly due to Easter), but did have visitors with an average time-on-site of five minutes-ish, I could establish when the crack to place and how long it lasted. At that point, I changed my hypothesis (since I could not change permissions without intentionally accessing the file) and decided that there was a security hole. Since I could see who accessed, where the access took place, and what was accessed I decided to try and search for the Apache file - a file that shouldn't EVER show up in the log files. I found the file. At that point I copied the string used to access the file into a browser, pre-pending the string with the appropriate domain URL, and brought up a screen (black background, lime green text) that allowed me to not only access but control any file on the server. This is bad news. At that point I took the first three octets and searched the logs for them, wanting to see from what IP's the user was trying to access the site. It ended up being four different IP addresses. I recorded the IP addresses and the proceeded to search the internet for information on the interface screen that was in front of me. In my head I wondered if the screen was a part of PHP that I'd never heard of (not out of line since I am not a PHP guru), but didn't know whether it was authorized PHP and a part of the PHP5 release we are using or something malicious. What I came across, rather quickly, was that the script was malicious and often found within TXT files that are then accessed through an includes script within PHP. I then proceeded to search the TXT files on the server to see if anything jumped out at me as the culprit for the includes, and discovered that all TXT files were valid and legitimate files. I then went to the .php page that the cracker accessed through to see if I could find an includes file and dropped down (intentionally) to line 183 to see what would've failed there once other aspects of the site had become unavailable (again, intentionally done since I'd broken an aspect of the site to discover that part of the scripting broke at line 183). At that point I noticed identical script to what I'd found when I was doing my searching for malicious intent. I immediately cut out the script and copied it into the database file used for tracking the crack. At that point (about the end of my shift) I decided that the site was (more or less) secure and could go home; even though I'd broken a search function within the software that had been compromised. I knew that I needed to find out when the script had been inserted and I needed to check against earlier log files (February) for access logs; but had already changed the "edited on" date of the file. The last thing I did was to make sure the users IP addresses were blocked from our server (with the intent of running a search against new log file with the first three octets over the coming days/weeks). So, I left the files broken last night and didn't pursue more research until this morning. This morning I established that the file had been cracked on November 29th in the afternoon and the malicious code (most likely) inserted at that time. My guess is that this was a part of the includes crack I'd stopped in February (which also, incidentally, stopped the insertion of invisible links) and after establishing a potential insertion date went through the February log files (oldest we have) to search for running of malicious script and known IP strings. Neither came up in February, which allows me to believe that the February crack attempts were definitely caused by running includes scripts in the address bar. After I finished searching the logs, I proceeded to locate and verify the validity of all .php files that could've had the malicious code inserted. Once I established that all files were clear of the code, and then copied a clean version of the .php file (in question) into the cracked directory of the server, restoring search capabilities; I could then focus my attention on known vulnerabilities and upgrades to the software we are using. With all of that accomplished and a new version of the software to install on all sites using it, as well as needing to go into all of the databases and assure that we have a universal administrative username and password for our team, I believe that this crack attempt has been (for the moment) thwarted. This does not mean that I believe our site is universally crack proof and need to verify the inclusion of .ini files and Apache mods into all of the (potentially) vulnerable folders, but for the moment I believe we have successfully removed this one threat from our pool of potential crackers. Truth told, even though the process was very much me and a computer screen, and felt a lot like a 90's movie where a cracker is actively being thwarted by a geek at a 24-hour manned terminal (knowing that most internet security is NOT manned actively), the outcome and process was boring to watch and probably equally boring to read about - even if I do feel rather proud of the discovery and how the crack was able to be stopped, thwarted, and the cracker blocked. John Hattaway | smokingpen | Alicia Grey | Clockwork Princess | Cassandra West Real Heroes Fly