Supply Chain and Infrastructure Limits

 

It often starts with congestion: a cough, a blockage, or a restriction in a pathway we depend on to function. I am not talking about COVID-19 in the human body. I am talking about the effect the pandemic had on the systems that keep our communities and businesses running.

In many ways, the same idea applied. As demand increased and normal patterns changed, supply chains and network infrastructure became congested. Things we had always assumed would be available suddenly were not.

The early warning signs

In January 2020, the CDC issued a travel alert related to Wuhan, China. At that point, the United States had only a small number of known cases. By the end of January, the U.S. government had begun issuing travel restrictions related to China.

In February, I moved my home internet service from Comcast Business to Xfinity Residential. At the time, it seemed like a practical decision. The service offered more bandwidth at a lower cost, and it worked well for my family. 

Then, in early March, as head of IT and the Facilities, I needed to purchase hand sanitizer and disinfecting wipes for the office - to replenish our stock. I searched through Amazon and other suppliers, only to find that delivery dates were more than 30 days away. Soon, sanitizer, wipes, and toilet paper became difficult to find almost everywhere.

These were ordinary items we had always taken for granted. We still had some supplies, but not in the quantities we wanted. For me, that was the first real warning sign that the supply chain was under strain. It was the first time in many years that a common item had simply disappeared from online retailers and local stores.

A sudden shift to working from home

By mid March, an incident at our San Jose office required us to begin working from home before the Bay Area issued its shelter in place order. Soon after, California followed with statewide restrictions.

My children were also sent home from school. Two attended Leland High School, and one attended De Anza College. Beginning March 16, 2020, they joined millions of other students across the Bay Area in trying to learn and connect from home.

At the same time, Silicon Valley and much of the region moved to remote work. We did not immediately realize how much pressure this would place on our network infrastructure.

Millions of people were suddenly working from home, attending online classes, streaming video, and trying to stay connected. Children who could no longer see friends in person turned to Netflix, YouTube, Hulu, gaming, and social media. Adults relied on Zoom, GoToMeeting, Webex, and other platforms to keep their jobs moving.

All of it created a massive and unplanned stress test for the internet.

When home internet became a business problem

My move to residential internet had seemed like a good decision in February. But once nearly every household in the area was relying on home connections throughout the day, the service became slow and unreliable.

I had to rethink the setup quickly.

I brought Xfinity Business back into the house because I could no longer depend on a connection that slowed down during the middle of the workday. The contract was not perfect, but it gave me a dedicated 100 Mbps connection, an LTE backup modem, and battery backup for about $120 per month during the first year. The installation and configuration took about two weeks.

In the meantime, my wife and I relied on mobile hotspots to keep working. Sometimes they worked well. Other times, they did not.

The experience made it clear how dependent we had become on systems that were never designed for everyone to rely on them at once.

The growth and growing pains of video meetings

As people adjusted to working and learning from home, platforms such as Zoom became essential. They gave families, schools, and businesses a way to stay connected when in person meetings were not possible.

But the rapid growth also exposed weaknesses. Zoom had been designed for ease of use, and some meetings lacked basic security settings. This made it easier for disruptive people to enter public meetings, sometimes sharing offensive sounds or graphic images.

The platform improved over time, adding stronger security options and better guidance for meeting hosts. Still, the early days were a reminder that rapid growth can reveal gaps in security, capacity, and planning.

Planning for the next disruption

Things are more stable now. Zoom is more secure, and my home internet is generally reliable, even though outages still happen from time to time.

The larger lesson is that we need to look more closely at the capacity of the systems we depend on. Businesses, service providers, schools, and governments all have a responsibility to consider what happens when demand reaches unexpected levels.

Few people imagined that tens of millions of students would need to learn online at the same time, often through video calls. Few expected entire workforces to depend on home internet connections overnight. Yet that is exactly what happened.

One positive outcome of 2020 was that it forced us to rethink what is acceptable and what we need to build for the future. We may not know exactly when a crisis will end, but we can prepare better for what comes next.

That means protecting our supply chains, strengthening our infrastructure, and planning for capacity before we need it.

Cloud Strategies and Authentication Methodologies

 

Making the Cloud Work for IT

Like most forward thinking IT teams, we are always looking for better ways to connect people with the data, tools, and services they need.

“Do more with less” is a phrase no IT leader loves to hear, but it is a reality in many organizations. After more than 25 years working in startups, I have learned that efficiency is not simply about reducing headcount or spending less. It is about using automation to handle repetitive work while protecting the reliability, security, and integrity of the systems the business depends on.

Moving toward the cloud

Many organizations have adopted cloud services for systems that once had to be managed on site. Ten years ago, I was hesitant about that idea.

Why hand over control of a system to someone else? Hosting a service internally meant control over the hardware, software, configuration, management, and customization. Giving that up felt risky.

My first real experience with cloud services was hosted Exchange, before Office 365 became widely available. As a systems administrator, the idea of no longer managing an email platform myself was uncomfortable at first. I worried that my skills would become less relevant.

Instead, it gave me room to focus on more important work.

Exchange is a complex system to operate well. By using a hosted service, I no longer had to spend as much time on day to day maintenance, disaster recovery planning, and troubleshooting every technical issue. When a problem occurred, the provider became the first level of support.

That freed me and my team to focus more on the company’s needs, rather than spending all of our time maintaining the underlying tools.

The value of cloud services

Today, cloud strategy is common across many organizations. Cloud services can take critical systems that might otherwise require several administrators and make them easier to manage at scale.

For example, our Exchange environment supports roughly 375 active email accounts and uses more than 3.3 TB of storage. It serves a global community of sales, services, and support employees, with availability close to 99.99 percent.

Running that same environment on site would require substantial hardware, dedicated storage, replication to a secondary location, backup and recovery systems, and likely one or more full time administrators.

Cloud services do not eliminate responsibility, but they can reduce the operational burden and allow IT teams to focus their time where it matters most.

Too many passwords

The move to cloud services also creates a challenge. Each platform often comes with its own login, password, and user management process.

As companies adopt more applications, this quickly becomes difficult for employees and IT teams alike. People have more passwords to remember. New hires need accounts in multiple systems. When someone leaves, IT needs to make sure access is removed everywhere.

Single sign on platforms such as Okta and OneLogin help solve this problem. They allow companies to use one identity system across multiple services, including Atlassian, Microsoft 365, Salesforce, NetSuite, Dynamics 365, and many others.

At Virtana, we use Okta to support authentication, multi factor authentication, and automated user provisioning and deprovisioning. This reduces the likelihood of manual errors and makes it easier to manage access consistently.

We have deployed this approach across more than 20 services used by the company. When we evaluate a new platform, such as SurveyMonkey or Monday.com, one of our first questions is whether it supports SAML and can integrate with our single sign on environment.

The goal is simple: make access easier for employees while keeping company systems secure.

Monitoring the cloud

As cloud environments grow, visibility becomes just as important as access management.

Virtana’s acquisition of Metricly added cloud monitoring capabilities for services such as AWS. The platform became part of our broader CloudWisdom offering and helps organizations understand how their cloud infrastructure is performing.

One of the most valuable capabilities is cost optimization. It is one thing to know that AWS cost $50,000 in a given month. It is much more useful to understand what drove that cost and where changes could reduce spending without affecting performance.

Those savings can be reinvested in innovation, infrastructure, or people.

Cloud services are not a replacement for thoughtful IT leadership. They are a way to spend less time maintaining routine systems and more time helping the business move forward.

Nest and Your Peace of Mind

 Here's #10:

---

Nest and Your Peace of Mind

Nest has been taking a lot of heat lately over reports of their camera systems being hacked. Before we accept that framing at face value, it is worth understanding what actually happened and where the real problem lies.

Here are the facts. Yes, unauthorized people gained access to accounts belonging to two families, and the results were frightening. One family received a false nuclear alert through their home system. Another had a stranger speaking through the camera in their child's room. Nobody should ever experience either of those things. As a parent, the thought of hearing an unknown voice coming from my child's bedroom triggers every protective instinct I have.

But here is the part that tends to get lost in the headlines: Nest was not breached. The accounts were compromised because the owners were reusing passwords that had already been exposed in breaches at other websites. The attackers did not break into Nest. They walked in through a door the users had left unlocked, using credentials that were already floating around on the dark web.

That distinction matters, and it is important to say it clearly without blaming the victims. We have all become far too comfortable in our digital lives. Most people have credentials on dozens of websites and apps, accessed across phones, tablets, computers, and smart devices. Keeping track of unique, strong passwords for all of them feels impossible, and so people fall back on the same email and password combination across multiple sites. That is an understandable habit. It is also a genuinely dangerous one.

Here is the reality: over a three to four year period, there is a very good chance that at least one site you use has been compromised and your credentials exposed. Once that happens, those credentials get tested against other services automatically. If you use the same password on Nest that you used on a site that was breached two years ago, someone will eventually find their way in.

So what can you actually do?

First, stop reusing passwords. Every account should have a unique password. I know that sounds overwhelming, but a good password manager makes it manageable. You only need to remember one strong master password, and the manager handles the rest.

Second, turn on two-factor authentication everywhere it is available, and especially on anything connected to your home. Nest supports it. Use it. Even if someone has your password, they cannot get in without the second verification step.

Third, check whether your credentials have already been exposed. The website haveibeenpwned.com lets you enter your email address and see which known breaches included your information. It is free, it is reputable, and the results are often sobering.

Companies like Nest share some responsibility here too. Making security features more prominent and strongly encouraging two-factor authentication during setup would go a long way. Balancing ease of use with genuine security is not easy, but when the product involves cameras inside someone's home, that balance needs to lean harder toward protection.

Your home should be a place where you feel safe. A few simple habits can go a long way toward making sure it stays that way.

Responsible Corporations: My Privacy Does Matter

Yesterday was a big day in the Apple world, and not in a good way.

Around 4 PM on January 28th, MacRumors, my go-to source for Apple news, began reporting on a serious flaw discovered in the Group FaceTime feature of the latest iOS update. The bug was straightforward and alarming: if you initiated a FaceTime call with someone and then added yourself back into the call as a third participant, an audio bridge would open on the other end, whether or not the person you called ever accepted. In plain terms, you could listen in on someone without their knowledge or consent.

By 11 PM, local television news was running the story. That same night, Apple disabled Group FaceTime on their servers entirely, shutting down the vulnerability before it could spread further.

I went to bed feeling reasonably reassured. Apple has spent years positioning itself as a company that genuinely champions user privacy. It is woven into their marketing, their product announcements, and their public messaging. I had just upgraded a Mac, and one of the first things Mojave presented me with was a notice about privacy and Apple's commitment to protecting it. Their entire privacy page exists as a public declaration of values. I believed them. I still want to.

Then I woke up the next morning.

MacRumors had posted a thread showing that a teenager had discovered this exact bug and reported it to Apple on January 21st, a full seven days before it became public. The timestamps on those posts cannot be faked. Apple had been made aware of a significant privacy vulnerability affecting millions of users, and nothing changed until mainstream media picked up the story.

That is the part that concerns me most.

I am not suggesting Apple acted maliciously. What I am suggesting is that somewhere in the chain between a teenager's bug report and a company-wide response, something failed. Either the report did not reach the right people, or it did and the urgency was not recognized, or, in the worst case scenario, someone knew and was quietly working toward a backend fix before the story got out. Any of those outcomes points to a process problem that a company of Apple's size and stated values should not have.

Technology companies, especially those that have built their brand on privacy and trust, have a responsibility that goes beyond good marketing. When a flaw is reported, the response cannot depend on whether a journalist picks up the story first. Users deserve to know when their security is at risk. Transparency, even when it is uncomfortable, is what actually builds trust over time.

Apple has done remarkable things. I remain a believer in a lot of what they stand for. But this was a stumble, and it deserves to be called one clearly.

Put people above the bottom line. Always.

Windows 10 Fall 2018 Update: A LTSC Usability Problem Microsoft Missed

Let me be clear upfront: the Windows 10 Fall 2018 update, version 1809, did not delete my user files and did not touch my activation keys. Those horror stories made the rounds and understandably scared a lot of IT administrators. That is not what this post is about.

This is about a quieter, more frustrating problem. A usability issue that suggests Microsoft's engineers lost track of which version of Windows they were actually building for.

At work, we run the Long Term Servicing Channel build of Windows 10, now called LTSC, previously called LTSB. The Long Term Servicing Channel exists specifically for environments that need maximum stability and minimal disruption. It is the right choice for a lot of businesses, and we made that choice deliberately. The tradeoff has always been clear: you give up Microsoft Edge and the Microsoft Store in exchange for a leaner, more predictable operating system.

Losing Edge was never a hardship. Losing the Store created some friction. A few users wanted access to the Windows Subsystem for Linux, which inexplicably requires the Microsoft Store to download and install. Others ran into printer driver issues, specifically with HP hardware, that also required the Store to resolve. Those are legitimate complaints worth a separate conversation with Microsoft.

But the 1809 update introduced something new and genuinely irritating. The updated interface now includes a Themes section, which has never appeared in an LTSC build before. Scattered throughout the updated UI are hooks and prompts pointing directly to the Microsoft Store, a component that is not installed and was never meant to be part of this build.

Click on certain interface elements and you are told you need a Store package to proceed. Try to use the updated Snipping Tool, which now prompts you to upgrade to Snip and Sketch, and you are directed to download an app that only exists in the Store. The Store that is not there. The Store that was intentionally left out.

This is what makes it frustrating. These are not edge cases or obscure settings. These are visible, everyday parts of the interface. In the previous 2016 LTSB build, none of these prompts existed. Someone made a decision during the 1809 development cycle to update the interface without accounting for the fact that LTSC users would be staring at buttons and options that lead nowhere, for the next three years.

The ask here is simple. Microsoft, when you build for LTSC, please take the time to remove or hide the elements that depend on components your LTSC users deliberately do not have. Do not leave people staring at a prompt they can never fulfill. That is not a small thing when you are managing a fleet of business machines and your users are asking why something that looks like it should work simply does not.

Do better with the details, Microsoft. Your enterprise customers are counting on it.

Proper Documentation REQUIRED!

I have been in IT since 1989. Twenty-nine years in the field, twenty-five of them as paid professional engagements. My first computer was a Macintosh SE with dual floppy drives, no hard drive, and 1MB of RAM. I say that not to date myself, but to establish that I have seen a lot of what works and a lot of what does not.

Recently, I took over full responsibility for the IT organization at Virtual Instruments. My background has always been rooted in infrastructure: servers, routers, switches, endpoints. Taking on the application side of the business was a new challenge, and what I found when I got there was instructive in the worst possible way.

From Order to Chaos, and Back Again

I joined Virtual Instruments in 2009. On paper, I am employee number 76, and I am one of a small handful of people who have been with the company from its earliest days to now. During that time, the applications division was always a separate world from infrastructure.

For a while, it was run well. A Director of Applications named Anne was exceptional at her job. She had the skills, the discipline, and the professionalism to run a tight operation. You always knew where things stood. When she left in 2015, that position was not backfilled, and the vacuum it created had consequences that lasted years.

From late 2015 through early 2018, the applications team existed largely in firefighting mode. Too many tasks, not enough oversight, and no consistent framework to work within. A lot changed during that period. Almost none of it was documented.

When I took over in 2018, my first priority was to bring stability and industry best practices to a department that had been operating in the dark. That meant a significant amount of reverse engineering.

We use an ERP system that has no native integration with our CRM or warehouse platform. To bridge that gap, we rely on middleware, specifically Dell Boomi, to move orders between systems. What I found when I started digging into the existing Boomi workflows ranged from straightforward and logical to genuinely difficult to interpret. Some processes had multiple forked paths that seemed to meander without clear purpose. Staring at someone else's undocumented code and asking yourself what they were trying to accomplish is a frustrating and time-consuming way to run an IT operation.

It does not have to be this way.

Documentation Is Not Bureaucracy. It Is Respect for the Next Person.

I have lived by one principle throughout my career: there is no acceptable reason for an IT professional to skip proper documentation. None. Not timeline pressure, not team size, not the assumption that you will always be there to explain it yourself.

The framework I keep coming back to is ITIL. If you are not familiar with it, here is the short version. ITIL 3.0 is an industry standard framework that structures IT service management into five stages:

• Service Strategy
• Service Design
• Service Transition
• Service Operations
• Continual Service Improvement

Service Strategy aligns IT activities with the core needs of the business. Service Design creates and adapts services to support those needs. Service Transition moves systems through change management, testing, and knowledge transfer. Service Operation manages day-to-day events and incidents. Continual Service Improvement identifies opportunities to do things better over time.

Working within a framework like ITIL or COBIT makes undocumented work structurally difficult. The discipline is built in. When my predecessor operated without any framework, critical context was lost the moment a project closed. No notes. No decision rationale. No institutional knowledge. Just systems running on logic nobody could fully explain anymore.

Good documentation paired with sound architectural decisions creates systems that are manageable. The absence of both creates systems that are fragile and expensive to maintain.

The Tools That Made a Difference

Choosing the right framework is only part of the answer. You also need tools that support the work without getting in the way. Here is what we put in place:

Atlassian Confluence became our documentation hub. Of all the wiki-style platforms I have evaluated, Confluence is the most intuitive and flexible. The tool has to disappear into the work. If people are fighting the interface, they are not documenting.

A proper helpdesk and ticketing system is non-negotiable. Every IT team, regardless of size, needs a way to track incidents and requests. Metrics depend on it. Accountability depends on it.

GitHub gave our application team a place to store and version their code. Infrastructure teams may not use it daily, but for any team writing or managing scripts and integrations, it is indispensable.

Monday.com helped bring visibility to complex, multi-phase projects with a lot of moving parts and stakeholders who needed to stay informed without being in every meeting.

Slack improved day-to-day communication and collaboration across the team in ways that email simply cannot match.

Where We Are Now

Today, the applications team is visible, accessible, and effective. We are doing more with a lean, skilled group, supplementing with contractors when we have genuine technical gaps to fill. We are methodically working through systems that were previously black boxes, exposing what is inside, documenting what we find, and building a foundation that the next person can actually work from.

The lesson is simple. No IT team is too small to operate with discipline and documentation. A team of two people benefits from a framework. A team of five people needs one. The investment in doing it right the first time pays back every single time someone has to touch that system after you.

Document your work. Every time. No exceptions.

Apple Announcement Day - Fall 2018

I am not going to run through predictions or spec leaks. There are plenty of sites better positioned for that. What I do want to talk about is something that deserves more direct attention: the death of the sub-$1,000 smartphone.

The iPhone has been around for just over ten years. In that time, the price has climbed steadily while the subsidized carrier model that once softened that climb has quietly disappeared. Today you essentially have two realistic options: buy the phone outright, or spread the same full retail cost across 12 or 24 monthly installments. Either way, you are paying the same amount. The installment plan just makes it easier not to notice.

I bought an iPhone X outright last year. $1,200 for the 256GB model, before tax, plus whatever time I spent waiting in line. Apple's cost to manufacture that phone was approximately $400. That is a $600 margin on a single device.

Here is how the numbers have looked across every base iPhone model over the years:

Model	Date	Build Cost	Retail Price
iPhone	Jun 2007	$220	$499
iPhone 3G	Jul 2008	$174	$199
iPhone 3GS	Jun 2009	$179	$199
iPhone 4	Jun 2010	$187	$199
iPhone 4S	Oct 2011	$188	$199
iPhone 5	Sep 2012	$194	$199
iPhone 5C	Sep 2013	$173	$99
iPhone 5S	Sep 2013	$198	$199
iPhone 6	Sep 2014	$211	$199
iPhone 6S	Sep 2015	$211	$649
iPhone 6S+	Sep 2015	$236	$749
iPhone SE	Mar 2016	$160	$399
iPhone 7	Sep 2016	$224	$649
iPhone 7+	Sep 2016	$277	$769
iPhone 8	Sep 2017	$247	$699
iPhone 8+	Sep 2017	$295	$799
iPhone X	Sep 2017	$370	$999
iPhone XS	Sep 2018	—	$999
iPhone XS Max	Sep 2018	$370	$1,099
iPhone XR	Sep 2018	—	$749

The pattern is clear. Carrier subsidies began disappearing around the iPhone 6 era, a transition that started in 2013 and was essentially complete by the time the iPhone 7 launched. Retail prices jumped immediately and have not looked back.

The XS, XS Max, and XR were all announced today. The era of the sub-$1,000 iPhone is over.

There are alternatives worth considering. OnePlus continues to produce excellent Android hardware at prices that make Apple's lineup look difficult to justify on value alone. But if you are deeply embedded in the Apple ecosystem and have no interest in leaving it, you are going to keep paying. The ecosystem lock-in is real, and Apple knows it.

The Apple Watch tells a similar story. I bought a Series 3 with cellular last September for $429. Today Apple announced the Series 4, and the base model without cellular starts at $429. Add cellular and you are at $529. That is a $100 increase for the same feature I was already paying for.

To refresh both my phone and my watch in 2018 would cost me somewhere around $1,800.

Here is where I landed. The Series 4 watch is worth the upgrade. The EKG capability alone justifies it for me. The phone is a different calculation. The XS is an "S" year, which historically means refinement rather than reinvention. I am going to wait for the reviews, specifically around camera performance and the practical value of the upgrades, before I decide whether the cost is worth it this cycle.

Apple makes extraordinary products. But at some point the value conversation has to happen honestly, and that point is now.