Friday, December 18, 2020

Lessons from the Solarwinds Orion Cyber Attack

On December 13, 2020, notifications were sent out from the Solarwinds CEO, Kevin Thompson alerting the estimated 300,000 customers of their product, ORION, that they were just made aware of a "a highly sophisticated, manual supply chain attack on SolarWinds® Orion® Platform software builds for versions 2019.4 through 2020.2.1."

It appears, and the details are still unfolding, that the malicious code was seemingly injected into patches and/or software updates sent out to approximately 33,000 customers, of which an estimated 18,000 were running the affected versions.  But Solarwinds was only alerted, some 5 days after cyber security experts FIREYE announced they suffered an intrusion from which some 300 of their tools were stolen. 


The full extent of the attack and how it was carried out will have further ramifications on the Cyber Security industry for years to come.  Not only did this attack infiltrate Fortune 500 based companies, but it also affected government organizations, used to protect our security, both domestically and internationally.  The Department of Homeland Security and their branch of Cybersecurity and Infrastructure Security Agency (CISA) were affected.

Type of Attack Matters

Traditionally there have been a few types of main attacks that occur.  Outside-to-Inside attacks (i.e. penetration testing, denial-of-service) or Inside-to-Outside attacks (i.e. social engineering, ransomware, spyware/malware).  But this sort of attack was definitely an inside attack, but artfully crafted in a very unique way.  I don't mean to glamorize the attack at all, but the level and sophistication of the attack is one that we haven't seen in years.  

Access was granted at some level for the malicious code to be introduced into the software development life cycle (SDLC), with zero checks or balances in that development cycle, so the code could be introduced into the software releases (2019 and 2020 versions).  It has all the earmarks of a social engineered attack, but so much more.  

What it calls out is that every company could have potential liabilities within their software development process where malicious code could be injected into what was thought to be a secure part of the development cycle. 

For years, we have put software and hardware in place to protect our perimeter.  We have installed intrusion detection systems (IDS) and intrusion prevention systems (IPS) for internal threats. Paid and supported for a security industry to ensure our laptops, desktops, servers and systems were protected by the best software money could buy.  But we forgot to look at what is now the most vulnerable and exposed systems in our companies.  The software development process.  

This attack will give rise to new technologies and protections, and I am sure, new industry standards and regulations to help bolster our defenses against such future attacks.

What can be done?

Better Communication

Certainly, we need much better communication channels.  While the age of privacy is important, and with the strict privacy laws of California and EMEA (GDPR) as examples, we have to balance the privacy and anonymization of data with that of an effective response to cyber terrorism.  When we talk about cyber attacks, more information could mean the difference between stopping a threat quickly before it has a chance to take off.  Think of the multi-agencies involved with domestic terrorist threats - the breaking down of the walls between agencies is critical to stop a threat from foreign terrorists.

Handling of cyber attacks are not much different than how we respond and react to terrorist threats to our cities and people.  

Better Quality Control and Reviews

While the details of this attack are still being uncovered, we do know that malicious code was introduced into the Solarwinds Orion software process.  In an article from KREBSONSECURITY.COM, a user reported on twitter of an open GITHUB Repo.

If this is the method from which the malicious code was introduced to the system, then what is critical is that the quality control and testing process failed Solarwinds.  In this day and age of rapid (Agile) software development and automation, we forget that there still should be strict code reviews.  I am not suggesting that Solarwinds needed to perform a line-by-line review of the code for each and every patch, but using AI and machine learning, it might be necessary to run code review through software to verify the veracity and integrity of the software from patch to patch and version to version.  

If we can automate the development process to create the patches, surely we can automate the code review process in a way that is efficient and looks for all the DIFFS (differences between two files) and analyzes those details for any malicious code or call-outs. 

On top of that - proper patch testing in a very controlled and segregated environment with massive auditing and logging facilities to see what the software is doing might also be necessary.

What Can Be Done Now?

Stay vigilant.  Analyze your current exposure points and don't take any part of your processes for granted, from development, to the quote-to-cash process, to the most minor systems in use on your network.

A good cyber security program might have uncovered this issue.  It might find exposures at your company that you can mitigate.  Don't be comforted by the level of your SDLC automation or sophisticated.  In fact, I would say the opposite.  With the level of sophistication and automation multiplied by the amount of hands touching that process, will increase the exposures and level of scrutiny required to ensure the integrity of the process.

IT cannot do it all.  A good cyber-security process cannot protect everything.  The statement, 'it takes a village', comes to mind.  Internally, this becomes a multi-departmental process to uncover possible exposures.  Tearing down department silos and exposing all the warts and ugliness of the SDLC process are critical to mitigate these sorts of issues in the future. 

Technology will catch up, quickly, but until then, it will be the work of the company to ensure this sort of attack never happens again.

Sunday, September 27, 2020

Covid and The Great Re-Negotiation Strategy

The Great Three

 There are events in our lives that are watershed moments.  Snapshots in time that usher in either great change or awareness.  I have lived through several of these and have coined these, The Great Three.  

The first was my marriage to my spouse.  From our first date in '91, to this date, she is my everything and my life has never been the same. Every day is and adventure and sharing my life with my best friend has been so good for my soul.  She is my biggest fan and it shows every day in her actions and words.

The second was the birth of my children.  I used to say it was the birth of 'a' child, but all three of my children are so important to me in so many ways.  But the demarcation between pre-children and post-children is very stark and life is never the same.  I knew there was love between a child and a parent, but was unclear until the moment my son came into this world, the love and bond between a parent and their child.  

The third was the death of a parent.  I have only lost one, so far (knock on wood), but that is also a very important if not tragic or sad milestone in ones life.  I lost my father some 16 years ago this December.  Not a day goes by that I don't think of him in some way or fashion.  Thankfully I still have my mother and my in-laws.  

What about COVID-19?

But what does this have to do with Covid and the title of this article, you ask?  Well - until 2020, and I think it is safe to say that the year 2020 has been one dumpster fire after another - I have never considered the impact of a global pandemic.  Now, 'The Great Three' has become 'The Great Four', because we will never be the same, again.

Sure, as an IT Executive, I have had to plan for disasters of all sorts from fire, flood, electrical storms, hurricanes, tornadoes and even the more common West Coast killer, Earthquakes.   But, on paper, the idea of a pandemic was very far fetched and difficult to comprehend the complete scope of what that sort of disaster would look like, in today's economy, society, and business world.  Sure, we had historical references and if you asked me to name a global pandemic, I would have had to go back to the 1918 Pandemic.  Yes, we more recently had the H1N1 flu pandemic, but that came and went like a whisper.  It didn't have any affect on us, in general.  

Now comes 2020 and COVID-19 is weaving its way through the American fabric and our businesses are forced to adjust accordingly with new work-from-home policies, shelter-in-place orders from the state, and figuring out new ways to make money, attend school, get healthcare!  It has been quite an interesting 6 months, not the mention the horrific death toll some Americans have had to pay and will continue to pay.  

But the crisis has started me looking at our contracts in very unique ways.  The contracts we have with our suppliers and vendors.   For the past 9 months, I have been tasked with running my corporate facilities department (as well as my regular IT management duties) and I have found one thing to be abundantly clear: There is no empathy in business.  

Empathetic Entities

What do I mean by that statement?  Since March, 2020, my company has been working-from-home.  Prior to the WFH/SIP orders, my company had 5 main locations and several other smaller offices around the world.  As the spring days gave way to the summer months, our company as well as so many others have sought a little relief from unceasing lease payments on buildings we were not occupying.  Landlords, while having bills and mortgage payments, themselves, refused to discuss any sort of relief.  And this was across the board.  You could threaten them with non-payment and they would invoke clauses within their agreements or threaten to send you to collections.  We could seek suspension of payments under local Santa Clara County ordinances, but this would only delay the inevitable. 

The Great Pandemic Clause

But it occurred to me... all our contracts (that I have seen in my past 12 years at this company) don't contain any verbiage/legalese to protect us, the end-user, in case of a Pandemic.  Sure, there might be clauses around acts of God or disasters, but a Pandemic clause is going to be a must on all future contracts.    

Why?   Because it will stipulate the relief parameters in advance of any pandemic.  God willing, there won't be another event for 100 years, but wouldn't it be great to have a 50% reduction in rent as relief, in case we are mandated to shelter-in-place?  By no fault of ours, we cannot occupy a space, but have to keep paying the full rate.  That isn't fair.  Quite honestly, it isn't fair for anyone, the landlord or the tenant.  

Any long term, 3 year or greater lease arrangement for a building or office equipment, no matter what it is - if a future pandemic may cause a loss of use of that item, then you should include some sort of clause to protect your monetary investments.  Why pay 100% for copiers that nobody is there to use or print on.  Why pay 100% for office space that nobody is allowed to visit.  

Now, I am not talking about a company that has decided to Work-From-Home as a new policy.  I am talking about being forced to work-from-home because of a shelter-in-place order.  When 95% of most businesses are not essential, you have very little recourse to save some money - that is, until now.  

Hindsight is 20/20, I know, but hopefully we won't have another pandemic of this magnitude for another 100 years.   Food for thought, anyway. 

Friday, June 19, 2020

The Human Side of Technology - People Come First

We are definitely living through some very strange times.  Our county has been under a 'shelter-in-place' order for 3 months and now extending into month 4.  We are starting to see some relaxation of the very strict guidelines, but our COVID-19 numbers have also been manageable/workable, by our first responders and healthcare workers.  We have been lucky here.   But there is another toll that is starting to take root and instead of talking about technology and the health of our networks, I thought it would be good to step back, don my sociology hat for a bit and talk about the health of our workers and employees - not just the physical health, but the mental health and well being.

Setting the Foundation

Let me start off by telling you a fairly brief story.  Back in 1999-2002, I was employed by a tech startup called eHealthInsurance.  It was everything in a technology startup, I wanted.  A fresh approach to health insurance, a dynamic management team and the company was eager for me as I was eager for it.  I was young.  I was hungry.  I did wonderful things for that company.    In 2001, I spoke with my manager and told him I was expecting a new baby and that I wanted to transfer to our Folsom location and work out the Sacramento area.  I got approved to do that.  Sold my house in San Jose - moved in with my in-laws, purchased a home in Roseville and the plan was to stay in San Jose until my daughter was born in 2002.  Life has a way of throwing these wicked curve balls at you.  First, 9/11 happened.  As an ex-Firefighter/EMT with Saratoga Fire District, I was devastated.  There is a special bond and trust firefighters go through.  That was hard. 

Then the dot com bubble burst and we entered a bit of a recession at the end of 2001, beginning of 2002.  It wasn't until the beginning of 2002 that the VP of engineering who oversaw my IT department, caught wind of my impending move to the Sacramento area.  Without mincing words, he rescinded the entire deal.  Revoked the approval for me to transfer to Sacramento.  Mind you, this was 2 months before my daughter was expected and I had been living with my in-laws for about a month.  I had sold my home in San Jose.  Moved in with my in-laws and was two months short of my daughters birth and three months short of moving to Roseville.  Talk about a shock to our system.  What were we going to do?

The Last Meeting

I spent the next month or two working my butt off and trying to keep the lines of communication open between me and eHealthInsurance.  By the VP of Engineering (I won't mention names - he knows who he is) wasn't having any of it.  To add insult to injury, the company had a telecommute policy in place that extended a work-from-home day to any employee who lived more than 50 miles from the office.  At our last meeting, I was summarily told that my position would not be transferring to Folsom. Furthermore, I would be required to work from Sunnyvale, 5 days a week and, even if I did move to Folsom/Roseville area (some 120 miles North), I would not be granted the current telecommute policy or be allowed to work even out of the Folsom office on that 5th day.  I was dejected and sad.  But I knew after 2 months of negotiations and me trying to do the honorable and right thing, that this was it.  I walked out of the office, to my desk, grabbed the three pre-typed letters (my just in case, letters) and walked my resignation back to HR, my boss and the VP of Engineering.  My time at eHealth Insurance was over.  There is a lot more to my story afterwards, but that is best saved for another time. 

My point with the preceding was to help point out that as managers we have a responsibility for the health and well-being of our employees, before that of their jobs.  Our most critical resource and assets are the people, themselves.  This VP lost sight of that.  Business was business and there would be no middle ground. He didn't care about my circumstances or the crumbling economy around us, or my new baby girl, or the very fact that my manager, the Director of IT, authorized this change 6 months prior. 

Understanding Now

Fast forward to now.  Now, I am a Vice President of IT.  I manage people and have responsibilities and I answer to people still above my pay grade.  But that doesn't negate the compassion or empathy I must have to effectively manage my people with their myriad of issues, personal and professional.

I spoke with one of my employees today and they need to talk with me about their mental well being and feelings of depression since the world was halted by COVID-19.  I immediately sympathized and told them they weren't alone in their feelings.  There have been mornings that I haven't wanted to crawl out of bed.  It is like being in a state of limbo, caught between some sort of permanent vacation and work. 

 lines between personal lives and professional lives have been grayed out completely to the point of stretching work days into 10-12 hours interspersed with being cooped up in your home with 4 other people who aren't co-workers.  Some days, I don't take showers.  Some days, I don't wear pants (not if I have a video call!), but you get the idea.  I have had two haircuts now, from my wife.  She has done a wonderful job, but she is a Marcom Coordinator for another tech company - she isn't a hairstylist. 

So I listened and we talked for a bit.  It doesn't matter what the employee was seeking from me.  It mattered that I listened, was receptive and more importantly, with compassion and understanding, I was accommodating during a very difficult economic and health crisis.

This employees health and well-being, was more important to me than their job.  I understood what they were saying.  The conversation ended very well. 


What was interesting is that as I was reflecting on this meeting, it occurred to me that I was the VP I needed to be, to the person, in every way that I needed my VP to be to me, back in 2002!   In 26 years of working in Information Technology, my goal has always been to take the best of the best managers I have had and couple it with my passion for I.T. and be the best version of a boss that I could be.  And subconsciously, I proved it with this person, recently.

I was a better version than that VP of Engineering from eHealth Insurance in 2002.  My conscious is clear and I now feel a strong moral victory. 

It is important that as managers and supervisors of people, we take the time out to be understanding and live with compassion and empathy.   Yes, business is business, but that is not how I run my organization or business. 

We are living in very strange times.  COVID-19, forced shelter-in-place orders, the recent protests for #BLM movement and the new economic recession we are in are taking a mental toll on a lot of people.  Our expectations for a separation of work life and personal life have slowly been stripped away over the past 100 days.   Depression and mental confusion will seep in.  Maintaining focus on projects and tasks becomes more and more difficult.  It is our jobs to do what we can, to take care of the people in our charges, be them engineers, or firefighters, or nurses, or clerks.   The mental well-being and health of my employees, comes before any job or task they need to perform - always.  This should be how we all operate, everyday.