I’ve been finding bugs in software since 1999 or so, I’ve reported over 150 vulnerabilities in that time, ranging from format string vulnerabilities to XSS. I also started developing my own web server in C around 1994. I did this to learn more about programming and not lose the skills I picked up during my studies in computer science at the University of Southern Maine. The web server has been tweaked, patched and modified off and on over the years as I started using it to serve traffic to my own website in 2002 after my site was hacked while running Dokuwiki.
My site only served static content as the HTTPd server I developed was much too simple to provide dynamic content. It only served .html and .txt file types, and it only accepted alphanumeric characters in the request URL. In 2010 Brian Martin (@attritionorg), one of the Infosec communities top vulnerability and bug curators mentioned that my site should be searchable, that he’d had trouble finding vulnerabilities I had published while he was building his database.
That was the fire under my ass that I needed to organize all of my old inconsistent text files and inject them into a MySQL table. I also had to add dynamic code execution to my web server, I took the simplest approach, I’d accept only a basic GET method and php-cgi execution of PHP scripts. I did some research and added basic cgi execution functionality.
In April of 2013, I transferred into Akamai’s then CSIRT where I had the opportunity to put my website behind Akamai’s massive CDN and enlist some of our security products to help my site. This shook out many bugs in the server as I wasn’t exactly RFC 2616 compliant.
Time went on, I ran into bugs when fixing the code or adding features. I eventually ran my code through Valgrind, fixed various null pointer dereferences, memory corruption issues and leaks. I mentioned my Valgrind adventures to my friend Julien a.k.a Vladz, he is a seasoned C developer who works as a penetration tester and participates routinely in CTFs. We went back and forth on how to generate a core dump after I would occasionally see a segmentation fault in my dmesg logs. A segmentation fault is when a program reads or writes outside of its assigned memory segment. Also for those of you who aren’t familiar with the dmesg command, it’s a utility in Linux to examine your kernel ring buffer, where the kernel posts the system’s console logs.
We exchanged a flurry of emails and tweets and I decided to email him a copy of my HTTPd server code I had named Bunyip. This was the first time anyone other than myself had seen the crap ball of C I had been tempering over the years. Not only that Julien was a really good C programmer, much better than myself.
After a few days, I contacted Julien and found that he compiled my HTTPd server but didn’t really have time to examine it. He said he’d have some time while commuting on the train that morning. That night I saw a message from Vladz on my twitter account, he was able to reproduce the segmentation faults I had been seeing with a simple GET request. It started to dawn on me, thinking back to my code I realized there was no mechanism for handling mangled requests. I had always expected the requests to come in as [GET or HEAD] [path][?][query string] and [protocol] like this:
GET /index.php?v=213 HTTP/1.1
What if someone sent the string malformed like:
GET ? HTTP/1.1
This would cause my parser to copy a NULL string into a buffer generating the null pointer dereference and segfault.
[Thu Mar 2 08:32:58 2017] httpd[19155]: segfault at 0 ip b7602e22 sp bff7cd10 error 4 in libc-2.19.so[b7576000+1a7000]
Next Julien moves on to looking for security holes in my code. I’m nervous, I had been focused on getting things working and less focused on security, but I knew I had a secret weapon. My attack checking code. I had written a small function that would only allow alphanumeric characters in the GET request, and not allow any “..” or “./” to thwart path traversal attacks.
Julien returns saying he has found some buffer overflows and a path traversal attack. I’m now shocked and embarrassed as to where I went wrong. Then I remembered disabling the attack checking code when testing my server with Akamai’s CDN and Kona WAF. I knew I was screwed with that code being disabled. I feared Julien would never speak to me again after this. That wasn’t the worst of it, in my code, I had been checking that the executable bit was set on files for execution. If that bit was set then the execve() function was called on the file requested. This coupled with the path traversal meant Julien was able to get remote command execution on my server via my HTTPd process. He could execute commands as www-data the user the HTTPd server dropped its root priveledges to. Julien sent me the exploits he developed for the path traversal and remote code execution.
He also sent me the output from the fuzzer he used to find the buffer overflows. I examined my code with the information he sent, fixed the buffer overflows, reinstated the attack checking code while making improvements to it. I’ve made many changes to the code, running it through Valgrind and auditing all of the functions both system and my own to ensure I’m checking return values correctly. I no longer check the executable bit but instead check the file extension, also the code now executes cleanly through Valgrind and I can build it with compiler optimizations set (-O2), this would have caused a crash before, due to the memory corruption issues I had. I’ve been running the code on my production web server for a few weeks now with no issues what so ever.
Vladz, or Julien rather speaks to me less now. It seems he is always busy washing his hair when I’m looking to chat.
Conclusion
It’s easy to lose sight of all the important things in a programming project when you’re focused on getting a specific task completed. In my situation, I was working on stability and functionality and lost the hacker mindset along the way. How can I break this? How can this be abused? If I were looking to get this to behave differently how would I do it? If I would have taken a step back and looked at my code from a hacker’s point of view like I had when I started writing it, I think I could have prevented most of Julien’s attacks.
If you’re developing software try to take a step back and examine your code from an adversaries perspective.
