Friday, December 18, 2020

Lessons from the Solarwinds Orion Cyber Attack

On December 13, 2020, notifications were sent out from the Solarwinds CEO, Kevin Thompson alerting the estimated 300,000 customers of their product, ORION, that they were just made aware of a "a highly sophisticated, manual supply chain attack on SolarWinds® Orion® Platform software builds for versions 2019.4 through 2020.2.1."

It appears, and the details are still unfolding, that the malicious code was seemingly injected into patches and/or software updates sent out to approximately 33,000 customers, of which an estimated 18,000 were running the affected versions.  But Solarwinds was only alerted, some 5 days after cyber security experts FIREYE announced they suffered an intrusion from which some 300 of their tools were stolen. 


The full extent of the attack and how it was carried out will have further ramifications on the Cyber Security industry for years to come.  Not only did this attack infiltrate Fortune 500 based companies, but it also affected government organizations, used to protect our security, both domestically and internationally.  The Department of Homeland Security and their branch of Cybersecurity and Infrastructure Security Agency (CISA) were affected.

Type of Attack Matters

Traditionally there have been a few types of main attacks that occur.  Outside-to-Inside attacks (i.e. penetration testing, denial-of-service) or Inside-to-Outside attacks (i.e. social engineering, ransomware, spyware/malware).  But this sort of attack was definitely an inside attack, but artfully crafted in a very unique way.  I don't mean to glamorize the attack at all, but the level and sophistication of the attack is one that we haven't seen in years.  

Access was granted at some level for the malicious code to be introduced into the software development life cycle (SDLC), with zero checks or balances in that development cycle, so the code could be introduced into the software releases (2019 and 2020 versions).  It has all the earmarks of a social engineered attack, but so much more.  

What it calls out is that every company could have potential liabilities within their software development process where malicious code could be injected into what was thought to be a secure part of the development cycle. 

For years, we have put software and hardware in place to protect our perimeter.  We have installed intrusion detection systems (IDS) and intrusion prevention systems (IPS) for internal threats. Paid and supported for a security industry to ensure our laptops, desktops, servers and systems were protected by the best software money could buy.  But we forgot to look at what is now the most vulnerable and exposed systems in our companies.  The software development process.  

This attack will give rise to new technologies and protections, and I am sure, new industry standards and regulations to help bolster our defenses against such future attacks.

What can be done?

Better Communication

Certainly, we need much better communication channels.  While the age of privacy is important, and with the strict privacy laws of California and EMEA (GDPR) as examples, we have to balance the privacy and anonymization of data with that of an effective response to cyber terrorism.  When we talk about cyber attacks, more information could mean the difference between stopping a threat quickly before it has a chance to take off.  Think of the multi-agencies involved with domestic terrorist threats - the breaking down of the walls between agencies is critical to stop a threat from foreign terrorists.

Handling of cyber attacks are not much different than how we respond and react to terrorist threats to our cities and people.  

Better Quality Control and Reviews

While the details of this attack are still being uncovered, we do know that malicious code was introduced into the Solarwinds Orion software process.  In an article from KREBSONSECURITY.COM, a user reported on twitter of an open GITHUB Repo.

If this is the method from which the malicious code was introduced to the system, then what is critical is that the quality control and testing process failed Solarwinds.  In this day and age of rapid (Agile) software development and automation, we forget that there still should be strict code reviews.  I am not suggesting that Solarwinds needed to perform a line-by-line review of the code for each and every patch, but using AI and machine learning, it might be necessary to run code review through software to verify the veracity and integrity of the software from patch to patch and version to version.  

If we can automate the development process to create the patches, surely we can automate the code review process in a way that is efficient and looks for all the DIFFS (differences between two files) and analyzes those details for any malicious code or call-outs. 

On top of that - proper patch testing in a very controlled and segregated environment with massive auditing and logging facilities to see what the software is doing might also be necessary.

What Can Be Done Now?

Stay vigilant.  Analyze your current exposure points and don't take any part of your processes for granted, from development, to the quote-to-cash process, to the most minor systems in use on your network.

A good cyber security program might have uncovered this issue.  It might find exposures at your company that you can mitigate.  Don't be comforted by the level of your SDLC automation or sophisticated.  In fact, I would say the opposite.  With the level of sophistication and automation multiplied by the amount of hands touching that process, will increase the exposures and level of scrutiny required to ensure the integrity of the process.

IT cannot do it all.  A good cyber-security process cannot protect everything.  The statement, 'it takes a village', comes to mind.  Internally, this becomes a multi-departmental process to uncover possible exposures.  Tearing down department silos and exposing all the warts and ugliness of the SDLC process are critical to mitigate these sorts of issues in the future. 

Technology will catch up, quickly, but until then, it will be the work of the company to ensure this sort of attack never happens again.

Sunday, September 27, 2020

Covid and The Great Re-Negotiation Strategy

The Great Three

 There are events in our lives that are watershed moments.  Snapshots in time that usher in either great change or awareness.  I have lived through several of these and have coined these, The Great Three.  

The first was my marriage to my spouse.  From our first date in '91, to this date, she is my everything and my life has never been the same. Every day is and adventure and sharing my life with my best friend has been so good for my soul.  She is my biggest fan and it shows every day in her actions and words.

The second was the birth of my children.  I used to say it was the birth of 'a' child, but all three of my children are so important to me in so many ways.  But the demarcation between pre-children and post-children is very stark and life is never the same.  I knew there was love between a child and a parent, but was unclear until the moment my son came into this world, the love and bond between a parent and their child.  

The third was the death of a parent.  I have only lost one, so far (knock on wood), but that is also a very important if not tragic or sad milestone in ones life.  I lost my father some 16 years ago this December.  Not a day goes by that I don't think of him in some way or fashion.  Thankfully I still have my mother and my in-laws.  

What about COVID-19?

But what does this have to do with Covid and the title of this article, you ask?  Well - until 2020, and I think it is safe to say that the year 2020 has been one dumpster fire after another - I have never considered the impact of a global pandemic.  Now, 'The Great Three' has become 'The Great Four', because we will never be the same, again.

Sure, as an IT Executive, I have had to plan for disasters of all sorts from fire, flood, electrical storms, hurricanes, tornadoes and even the more common West Coast killer, Earthquakes.   But, on paper, the idea of a pandemic was very far fetched and difficult to comprehend the complete scope of what that sort of disaster would look like, in today's economy, society, and business world.  Sure, we had historical references and if you asked me to name a global pandemic, I would have had to go back to the 1918 Pandemic.  Yes, we more recently had the H1N1 flu pandemic, but that came and went like a whisper.  It didn't have any affect on us, in general.  

Now comes 2020 and COVID-19 is weaving its way through the American fabric and our businesses are forced to adjust accordingly with new work-from-home policies, shelter-in-place orders from the state, and figuring out new ways to make money, attend school, get healthcare!  It has been quite an interesting 6 months, not the mention the horrific death toll some Americans have had to pay and will continue to pay.  

But the crisis has started me looking at our contracts in very unique ways.  The contracts we have with our suppliers and vendors.   For the past 9 months, I have been tasked with running my corporate facilities department (as well as my regular IT management duties) and I have found one thing to be abundantly clear: There is no empathy in business.  

Empathetic Entities

What do I mean by that statement?  Since March, 2020, my company has been working-from-home.  Prior to the WFH/SIP orders, my company had 5 main locations and several other smaller offices around the world.  As the spring days gave way to the summer months, our company as well as so many others have sought a little relief from unceasing lease payments on buildings we were not occupying.  Landlords, while having bills and mortgage payments, themselves, refused to discuss any sort of relief.  And this was across the board.  You could threaten them with non-payment and they would invoke clauses within their agreements or threaten to send you to collections.  We could seek suspension of payments under local Santa Clara County ordinances, but this would only delay the inevitable. 

The Great Pandemic Clause

But it occurred to me... all our contracts (that I have seen in my past 12 years at this company) don't contain any verbiage/legalese to protect us, the end-user, in case of a Pandemic.  Sure, there might be clauses around acts of God or disasters, but a Pandemic clause is going to be a must on all future contracts.    

Why?   Because it will stipulate the relief parameters in advance of any pandemic.  God willing, there won't be another event for 100 years, but wouldn't it be great to have a 50% reduction in rent as relief, in case we are mandated to shelter-in-place?  By no fault of ours, we cannot occupy a space, but have to keep paying the full rate.  That isn't fair.  Quite honestly, it isn't fair for anyone, the landlord or the tenant.  

Any long term, 3 year or greater lease arrangement for a building or office equipment, no matter what it is - if a future pandemic may cause a loss of use of that item, then you should include some sort of clause to protect your monetary investments.  Why pay 100% for copiers that nobody is there to use or print on.  Why pay 100% for office space that nobody is allowed to visit.  

Now, I am not talking about a company that has decided to Work-From-Home as a new policy.  I am talking about being forced to work-from-home because of a shelter-in-place order.  When 95% of most businesses are not essential, you have very little recourse to save some money - that is, until now.  

Hindsight is 20/20, I know, but hopefully we won't have another pandemic of this magnitude for another 100 years.   Food for thought, anyway. 

Friday, June 19, 2020

The Human Side of Technology - People Come First

We are definitely living through some very strange times.  Our county has been under a 'shelter-in-place' order for 3 months and now extending into month 4.  We are starting to see some relaxation of the very strict guidelines, but our COVID-19 numbers have also been manageable/workable, by our first responders and healthcare workers.  We have been lucky here.   But there is another toll that is starting to take root and instead of talking about technology and the health of our networks, I thought it would be good to step back, don my sociology hat for a bit and talk about the health of our workers and employees - not just the physical health, but the mental health and well being.

Setting the Foundation

Let me start off by telling you a fairly brief story.  Back in 1999-2002, I was employed by a tech startup called eHealthInsurance.  It was everything in a technology startup, I wanted.  A fresh approach to health insurance, a dynamic management team and the company was eager for me as I was eager for it.  I was young.  I was hungry.  I did wonderful things for that company.    In 2001, I spoke with my manager and told him I was expecting a new baby and that I wanted to transfer to our Folsom location and work out the Sacramento area.  I got approved to do that.  Sold my house in San Jose - moved in with my in-laws, purchased a home in Roseville and the plan was to stay in San Jose until my daughter was born in 2002.  Life has a way of throwing these wicked curve balls at you.  First, 9/11 happened.  As an ex-Firefighter/EMT with Saratoga Fire District, I was devastated.  There is a special bond and trust firefighters go through.  That was hard. 

Then the dot com bubble burst and we entered a bit of a recession at the end of 2001, beginning of 2002.  It wasn't until the beginning of 2002 that the VP of engineering who oversaw my IT department, caught wind of my impending move to the Sacramento area.  Without mincing words, he rescinded the entire deal.  Revoked the approval for me to transfer to Sacramento.  Mind you, this was 2 months before my daughter was expected and I had been living with my in-laws for about a month.  I had sold my home in San Jose.  Moved in with my in-laws and was two months short of my daughters birth and three months short of moving to Roseville.  Talk about a shock to our system.  What were we going to do?

The Last Meeting

I spent the next month or two working my butt off and trying to keep the lines of communication open between me and eHealthInsurance.  By the VP of Engineering (I won't mention names - he knows who he is) wasn't having any of it.  To add insult to injury, the company had a telecommute policy in place that extended a work-from-home day to any employee who lived more than 50 miles from the office.  At our last meeting, I was summarily told that my position would not be transferring to Folsom. Furthermore, I would be required to work from Sunnyvale, 5 days a week and, even if I did move to Folsom/Roseville area (some 120 miles North), I would not be granted the current telecommute policy or be allowed to work even out of the Folsom office on that 5th day.  I was dejected and sad.  But I knew after 2 months of negotiations and me trying to do the honorable and right thing, that this was it.  I walked out of the office, to my desk, grabbed the three pre-typed letters (my just in case, letters) and walked my resignation back to HR, my boss and the VP of Engineering.  My time at eHealth Insurance was over.  There is a lot more to my story afterwards, but that is best saved for another time. 

My point with the preceding was to help point out that as managers we have a responsibility for the health and well-being of our employees, before that of their jobs.  Our most critical resource and assets are the people, themselves.  This VP lost sight of that.  Business was business and there would be no middle ground. He didn't care about my circumstances or the crumbling economy around us, or my new baby girl, or the very fact that my manager, the Director of IT, authorized this change 6 months prior. 

Understanding Now

Fast forward to now.  Now, I am a Vice President of IT.  I manage people and have responsibilities and I answer to people still above my pay grade.  But that doesn't negate the compassion or empathy I must have to effectively manage my people with their myriad of issues, personal and professional.

I spoke with one of my employees today and they need to talk with me about their mental well being and feelings of depression since the world was halted by COVID-19.  I immediately sympathized and told them they weren't alone in their feelings.  There have been mornings that I haven't wanted to crawl out of bed.  It is like being in a state of limbo, caught between some sort of permanent vacation and work. 

 lines between personal lives and professional lives have been grayed out completely to the point of stretching work days into 10-12 hours interspersed with being cooped up in your home with 4 other people who aren't co-workers.  Some days, I don't take showers.  Some days, I don't wear pants (not if I have a video call!), but you get the idea.  I have had two haircuts now, from my wife.  She has done a wonderful job, but she is a Marcom Coordinator for another tech company - she isn't a hairstylist. 

So I listened and we talked for a bit.  It doesn't matter what the employee was seeking from me.  It mattered that I listened, was receptive and more importantly, with compassion and understanding, I was accommodating during a very difficult economic and health crisis.

This employees health and well-being, was more important to me than their job.  I understood what they were saying.  The conversation ended very well. 


What was interesting is that as I was reflecting on this meeting, it occurred to me that I was the VP I needed to be, to the person, in every way that I needed my VP to be to me, back in 2002!   In 26 years of working in Information Technology, my goal has always been to take the best of the best managers I have had and couple it with my passion for I.T. and be the best version of a boss that I could be.  And subconsciously, I proved it with this person, recently.

I was a better version than that VP of Engineering from eHealth Insurance in 2002.  My conscious is clear and I now feel a strong moral victory. 

It is important that as managers and supervisors of people, we take the time out to be understanding and live with compassion and empathy.   Yes, business is business, but that is not how I run my organization or business. 

We are living in very strange times.  COVID-19, forced shelter-in-place orders, the recent protests for #BLM movement and the new economic recession we are in are taking a mental toll on a lot of people.  Our expectations for a separation of work life and personal life have slowly been stripped away over the past 100 days.   Depression and mental confusion will seep in.  Maintaining focus on projects and tasks becomes more and more difficult.  It is our jobs to do what we can, to take care of the people in our charges, be them engineers, or firefighters, or nurses, or clerks.   The mental well-being and health of my employees, comes before any job or task they need to perform - always.  This should be how we all operate, everyday.

Monday, April 20, 2020

Supply Chains and Infrastructure Limits

Supply Chains and Infrastructure Limits

It starts with a cough - congestion - restriction of vital pathways that are critical for our survival.  No.  I am not talking about COVID-19 and our bodies, but the analogy of that virus and the impact the pandemic is having for our nation are very similar. 

Beginning Stages

January 6, the CDC issued a travel alert from Wuhan. While China was busy battling the virus overseas, we were having 1 or 2 cases here in the United States.  January 31 - US government started issuing a travel warning from China.  I moved my home office from Comcast Business to Xfinity Residential in February, thinking the increased bandwidth and lower cost would be good for us.  

In early March, I had to procure sanitizers for the office (hand and wipes) and started to secure those through various Amazon channels, but found that the dates for shipping those items were 30+ days out.  These items and toilet paper became a very hot, very sought after commodity that we normally take for granted. We still had sanitizer but not in the qty we wanted.  This was our canary in the coal mine for the supply chain. The first time in about a 10-15 years that I noticed we couldn't purchase something that was a normal commodity.  First, it disappeared from Amazon, and next our local markets. The run was on!

Then mid March, we had an incident at our office in San Jose, that required us to work from home prior to the Bay Area issuing its Shelter In Place (SIP) order and well before the State of California did the same, but relating to the Covid-19 pandemic we are in the midst of now.  My kids, two at Leland High School and one at De Anza Jr. College, were also told to stay home starting on March 16, 2020 with the rest of the Bay Area.  While our supply chain was starting to have issues, we weren't aware until all of Silicon Valley was working remotely that our Network Infrastructure was about to feel the congestion of 10 million people and kids working from their homes or going on streaming binges.  Kids were supposed to be restricted not only from school, but from one another.  No parties, no get togethers at the local Safeway parking lot (SPL - it is a thing).  With no outlet for our children and no remote schooling plan prepared by our school district, they turned to Netflix, YouTube, Hulu and other streaming services.  Couple this with people using their ZOOM, GoTo and WebEx to be productive for work?  And we have the perfect storm of stress testing our entire network infrastructure.

My move to Xfinity residential back in February seemed to be a cost effective move that my family loved.  Now, with every local family jumping on their Xfinity links, the entire system bogged down to a stutter and shuffle.  I quickly had to reshuffle my chess pieces on my board to allow me better ways to attack the problem.  I had to get XFinity Business back in the house as soon as possible. I simply could not work from 10AM to 4-5PM every day.  The contract wasn't stellar, but it proved useful.  I got 100 Mbit dedicated link with a LTE backup modem and battery backup device for $120 month for the first year.  The kicker was, it would only take 2 weeks to install and configure. 

Meantime, my wife and I were using our hotspots to remain productive during business hours.  Sometimes that worked good - and other times, not. 

As people were working through their lives of being at home and online, more and more people became reliant upon technologies like ZOOM to stay connected.  But the rapid influx of people onto ZOOM sessions and the 'ease-of-use' of Zoom, soon showed us the fragility of the platform.  The general lack of security as a standard, allowed many people to perform ZOOM bombs and jump into public meetings, playing all sorts of things like pornography sounds or displaying graphic images, all to simply be disruptive. 

Things are much better. Zoom has better security, our home internet is also very stable now, but still experiences outages every once in a while.  What this means is that we should now start looking at our general infrastructure and what load capacity it can sustain.  Companies have a responsibility to ensure that their systems have the capacity to expand to peak usage times.  Nobody thought that having 50-70 million school kids online at the same time would ever happen and all in video conferences.   

The silver lining for 2020 is that it is causing a shift in the thinking of businesses.  Altering our perceptions about what was acceptable and what we should be building out in the future.  We don't know if this pandemic will end in 2020 or 2021?  We hope it will end soon, but we really need to keep planning for the future.  Preparing for the worst.  Protecting our Infrastructure and Supply Chain!   Food for thought.

Wednesday, April 8, 2020

Cloud Strategies and Authentication Methodologies

Like any forward thinking IT Department, we have been vigilant in trying to find more and more efficient methods of connecting our users with their data/resources.   Do more with less resources is the mantra that no IT person ever wants to hear, but it is chanted by every executive, either breathlessly behind closed doors, or touted as the norm.  While I have never been a stranger to this, working in start-ups for the past 25 years, it has become more about how efficiently I can run a department by offloading repetitive tasks to automation while maintaining the integrity and security of a system. 

Clouds are Brewing

A lot of companies 'doing more with less' have adopted cloud strategies where only on-premise systems used to live.  At first - 10 years ago - I was against moving things to someone else to host/serve.  Why would I want to give up control of a system or service - the hardware, the software, the management and configuration, not to mention the customization.  

My first foray into cloud services was Online Exchange - this pre-dates Office 365 by about 3 years. At first, the thought of me allowing my Sys Admin skills in managing a mail system going stagnant was horrifying.  But then the ease of managing a system as complex as Exchange, yet offloading all the technical bluster to a 3rd party was exactly what I needed at this new job I was starting.  

Gone were the worries about disaster recovery or day-to-day maintenance that Exchange systems love to produce.  If there was a problem, I engaged that 3rd party to find the solution - they became my 1st level of technical support for issues.   I could learn to like this.  It was providing me more freedom to grow into other areas and focus on the company and not the tools.

Fast forward 10 years and now a lot of companies are adopting a cloud strategy or culture (cloud culture).  It allows the offloading of otherwise critical systems that would require multiple administrators to manage/run.  My exchange system today consumes over 3.3 TB with 375 active e-mail accounts.  It services a global community of sales, services and support folks and remains very accessible and has an up-time around 99.99%.  To run that same system on premise, would take a lot of hardware, dedicated storage area network, with replication to a hot site some place, not to mention a full time admin if not two. 

How Many Passwords?

Now we come to the crux of the article - and that is the disparate nature of cloud systems/services.  This new urban sprawled compute environment has one glaring issue.  Each system usually has its own set of logins - ways to authenticate and/or password systems specific to that resource.  That is until companies like OneLogin or OKTA came around to help extend a single authentication schema that could be used across multiple different platforms, including Atlassian, O365, Salesforce and online ERP systems like Netsuite or Dynamics 365.  Again - looking at doing more for our company but with less manpower to do it with.  Instead of several different logins I have to remember and onboarding processes, I can now relay upon a single provider to assist with authentication and provisioning of these different systems. 

Virtana uses OKTA.  They are a very robust provider of authentication services, ranging not only from the actual SAML Authentication process, but multi-factor authentication, automated provisioning/deprovisioning services that ensure the mitigation of user errors.

Fast forward to today and we have deployed this single solution (OKTA) in over 20 different services my team provides for our company.  Couple this with employment of multi-factor authentication with rotating keys or SMS verification and we have a fairly robust and stable authentication mechanism.

Now when we are tasked with bringing online a new system (SurveyMonkey or Monday), we first look for the SAML integration points and how we can continue to leverage our single sign on system.  

Monitor My Cloud!

Recently, Virtana acquired a company called Metricly.  This company has a wonderful framework for monitoring Cloud systems - like AWS instances.  This became the third pillar in our triad of performance monitoring solutions, called Cloud Wisdom.  Now - this isn't an infomercial or advertisement.  It is a product that truly delivers what it says - an analysis of your infrastructure and an uncovering of the abstract nature of what cloud service providers currently offer.  The best part of this platform is the Cost Optimization model (from an OPEX point of view).  It is one thing to say AWS cost us $50,000 this month - but it is another thing to see that breakdown and understand that by shifting and moving resources around, we could save the company some dollars!  That is extra money that we could use to innovate more or invest in infrastructure or people!  I might be biased - then again, I also just might be experienced!   Check out CloudWisdom here: