Is there a Cuckoo in your control system?

Notice: if you haven’t read The Cuckoo’s Egg by Cliff Stoll, STOP, go read the book (or at least watch the NOVA special) then come back.
You can get a signed hardcover copy from the man himself for $20.  https://www.kleinbottle.com/ExtrasAndMisc.htm

---

tl;dr – Defend your ICS against intruders: You need at least one person solely dedicated to hunting evil, Network Security Monitoring, and an ICS honeypot.

---

If you have read the book, I have some simple questions for you:

• Is there an intruder in your control system network, like a cuckoo laying an egg in another bird’s nest?
• Are you sure?
• How do you know?
• Is the intruder altering the system in any way?
• Can you (as the control system or IT engineer) prove it in court?

There haven’t been many publicly known control system security related incidents or malware.  Is this because there really aren’t that many, or is it because we aren’t looking?  We can certainly say there won’t be any less attacks or malware that targets control systems.  Just last week, F-Secure reported the Havex Remote Access Trojan (RAT) that targets systems using OPC protocol.  ICS-CERT has chimed in.  Yesterday, Symantec reported a group called Dragonfly is targeting the Energy Sector with a RAT as well as PLC software malware.

So, how would you know that malicious actors like Dragonfly are in your control network or not?  Cliff’s story from 25+ years ago offers some great insights on defending a computer network, forensics, and incident response. We can surely apply these techniques to control systems today.  I’ve selected a few key things from Cliff’s book that will help you defend your control system.

1.  Most importantly…you need someone like Cliff Stoll on your security team.

Cliff was an astronomer and also a self-proclaimed “KlugeMeister”  (some history on the term if you’ve never heard it), but learned how to be an expert in computer security.  He truly cared (he even slept at his desk to wait on the hacker), had a questioning and curious attitude , was self-driven, was willing to learn and improvise, and was just as persistent as the hacker (even to his boss and the three-lettered Government entities).  You can have all of the best security technology available, but if you don’t have someone that cares like Cliff (if they are a SCADA security expert or not), then there’s no point.

2.  Monitor logs and system accounts.

One of the simplest ways that Cliff found evidence of an intruder was looking at who was logging into the Lawrence Berkeley Labs Unix and VMS systems.  He noticed a $0.75 accounting difference in the billing and found an account that had no billing address.  Then he noticed a user, Sventek (who was out of the country), was logging into the LBL system and then hopping onto the Arpanet and Milnet to other computer systems.  He also noticed that the hacker used default and well-known system accounts.  In control systems equipment, there are also default and well-known accounts that can’t be changed…as well as hardcoded backdoor accounts put there by the vendor.  Monitoring logs and system accounts is crucial to knowing your ICS network.

3.  Use an IDS and/or SIEM.

Cliff invented one of the first computer intrusion detection systems.  At first, he made his computer terminal beep any time anyone logged in, but it was too noisy.  Eventually Cliff made a better one that was kluged together from a tapped serial line, a logic analyzer, some code, a dialer, and a pocket pager.  This detection system wasn’t pretty, but it reliably alerted Cliff when the hacker used the “sventek” account so he could record it in his tracking log.  Modern IDS and SIEMs have many more features than his original design, but the principle is the same.  Know your systems and create alerts when something needs to be tracked or if something is out of the ordinary.  This is being done on most modern enterprise IT systems, but most ICS networks do not use an IDS or SIEM…yet.  What if you could push your ICS logs up to the corporate SIEM for a true picture of your entire network?

4.  Use full-packet capture, even on serial networks.

When Cliff and his coworkers at Lawrence Berkeley Lab discovered the intruder was coming into their Unix system over one of 50 dial-up lines, they decided to set up 50 PCs with printers to print out all traffic.  It was a silent way of capturing all of the hacker’s activity.  The hacker had no idea his actions were all being captured.  When Cliff got an alert from his IDS beeper, he would go look at the print out and see what the hacker was doing…often in real time (albeit 2400 baud).  Today, we have sophisticated and high-bandwidth packet capture systems to record and store multiple terabytes of traffic.  Today’s enterprise systems use this same technique to show exactly how an intrusion or any other type of event really happened.  As far as I am aware, very very few ICS networks utilize packet capture technology.   So please consider using this, even on serial-only networks (except you can use terminal servers instead of PCs and printers).

5.  Document document document!

Cliff’s story was the first fully-documented case of a hacker intruding on a computer network.  He kept a daily log of what he did, what the hacker did, along with timestamps and other data.  This data allowed him to determine many things about who this hacker was and what his motives were, not to mention what he actually did.    He was able to share this documentation to prove to the authorities what was happening and eventually led to the hacker’s arrest, trial, and conviction.  Documenting ICS networks and malicious activity allow the creation of indicators of compromise (IOCs) and share them with the industry, ISACs, and ICS-CERT.

6.  Use Honeypots.

The hacker would often quickly leave, which left Cliff with little time to have the telecom companies trace the call.  Cliff’s girlfriend Martha came up with the idea to create a honeypot with fake documents that might attract the hacker to stay long enough for a trace to be completed.  Sure enough, the hacker took the bait.  Using a honeypot in a production system in your ICS is a great way to detect activity with a low rate of false positives.  Here are 5 reasons to use honeypots.  You can make a file-based honeypot like Cliff did, or you can make a decoy PLC or HMI.  There’s a free ICS honeypot called Conpot if you can’t make a custom one of your own.

There are many more items I’d like to put here, but these are enough to get started.   The book doesn’t really cover a Disaster Recovery Plan, Forensics Plan, Testing Backups/Redundancy, Red Team/Blue Team, and other modern defense tactics.  Perhaps I’ll talk about those next time.  Stay tuned for more thoughts on ICS/SCADA defense.

NOW GO FIND THE CUCKOO!!  I bet there are some out there now, going undetected, and laying eggs in YOUR control system.

@chrissistrunk

Special thanks to Cliff Stoll for inspiring me and countless others in computer security.

---

For further reading, check out:

Richard Bejtlich’s “Cooking the Cuckoo’s Egg”, especially the last few slides on lessons learned from Cliff’s book.  Also read Richard’s book, The Practice of Network Security Monitoring.
Doug Burks and his team have put together Security Onion (which Richard covers extensively in his book).  Security Onion has tools for IDS, SIEM, and full-packet capture, and it is being used in a few ICS environments.