The Wall Street Journal had an article that I reviewed this morning where they interviewed multiple CEOs to ask them what was more important - strategy or execution. A CEO's focus should be first on strategy and then on execution. You can't have one without another. However, strategy needs to come before execution so that you can actually move in the right direction.
Last year we spent a great deal of time working as an executive team focusing on how to cut costs and increase revenue. Prioritization. Measurements through metrics. Innovation. All of these elements allowed us to push through some very difficult times with great success. As we ended the year, we were already watching and getting ahead of the economy. We started 2009 with a plan to greatly increase revenue even in a down economy. It seems that our hard work last year is already paying dividends this year. Revenue may be even higher than expected and we are still finding optimization elements where we are increasing capacity while lowering costs. Definitely a good mix. We are quite aware that we could easily lose more customers as everyone listens to the news and buckles down to ride out the recession. Obviously, this would put a damper on any positive aspects.
In the meantime, it was great to see Spirit with positive feedback on the local Charlotte news channel where we were mentioned as only 1 of 2 companies hiring this year. If you follow this link and click on the video on the right, you can see how the businesses in Charlotte are viewing the opportunities that 2009 has to offer.
http://www.wbtv.com/global/story.asp?s=9754142
(The question for another blog is: Are the news agencies to blame for the economy by constantly telling everyone how bad the economy is?)
Friday, January 30, 2009
Saturday, January 24, 2009
Conficker or Downadup, large anti-virus downloads can take your Internet connection down
We have seen multiple instances lately where customers have called and complained that their Internet connection was going down at various periods of the day. The customers were absolutely sure that something was wrong with their circuit and called our NOC to ask for troubleshooting assistance. Interestingly enough, we have traced the failures back to a single issue with each of these customers.
1. The first key we found was that the customer thought their internet connection was going down for short periods of time on a regular basis. It never seemed to be for more than a few minutes.
2. Once we proved that the circuit was not going down but was receiving a large increase in the volume of bandwidth, the customer wanted to know why they were receiving a denial of service attack. The source of the increased bandwidth did come from a small set of IP addresses.
3. Working with several customers, we were able to positively identify that the source IP addresses were a cluster of servers from Akamai on our network. We were also able to identify that the receiptient of the increased bandwidth, in each case, was a PC, server, or set of PCs downloading virus definition updates. This was accomplished by having the local IT department actually verify that the target IP was in fact a PC downloading the virus definitions/engine.
One particular customer who had a 20mb Ethernet connection was actually receiving 27mb of bandwidth during the downloads. I hope that this post can help other network providers or enterprise customers to pinpoint these short term specific issues.
1. The first key we found was that the customer thought their internet connection was going down for short periods of time on a regular basis. It never seemed to be for more than a few minutes.
2. Once we proved that the circuit was not going down but was receiving a large increase in the volume of bandwidth, the customer wanted to know why they were receiving a denial of service attack. The source of the increased bandwidth did come from a small set of IP addresses.
3. Working with several customers, we were able to positively identify that the source IP addresses were a cluster of servers from Akamai on our network. We were also able to identify that the receiptient of the increased bandwidth, in each case, was a PC, server, or set of PCs downloading virus definition updates. This was accomplished by having the local IT department actually verify that the target IP was in fact a PC downloading the virus definitions/engine.
One particular customer who had a 20mb Ethernet connection was actually receiving 27mb of bandwidth during the downloads. I hope that this post can help other network providers or enterprise customers to pinpoint these short term specific issues.
Wednesday, January 21, 2009
Internet integrity, or should I say, fragility
How reliable is the Internet?
Our team is working on metrics for last year and it gave me a chance to reflect on what works well and what doesn't. As I looked at the statistics, I was quite pleased and definitely proud of our team. We had several hardware issues that were difficult to push through as we worked with the hardware vendors. The goal of any carrier is to maintain a core network that meets or exceeds 99.999% reliability. The reality of that number is less than 6 minutes of total downtime in a year out of potential 525,600 minutes in a year. I actually have held our team accountable to 5 minutes and 18 seconds per year. We were able to meet 99.999% (around 3 minutes total downtime) on our Internet core and 100% on our other networks. Does this mean that we didn't have any customers go down? Unfortunately, no. On the core network nodes that all customers cross, we hit our numbers. On direct links, we did have customers go down. As copper plants (DSL, T1s, DS3s) continue to age, the number of failures will continue to grow over time. :-(
So how does this relate to fragility on the Internet core? That question takes us to the basic infrastructure of how an enterprise or carrier network is setup. Most large networks are setup using OSPF and BGP. There are multiple ways for network engineers to configure the network and I rarely find one engineer that agrees with another. (Can you say "standards"!) To understand how this is setup using two protocols, you need to understand how BGP determines a best path. Imagine you are headed to Washington D.C. to watch the presidential inauguration. When you get to DC, you ask for directions. Here are your two sets of directions:
Person 1: To get to the party, go down this street and you should get there in about 30 minutes.
Person 2: To get to the party, take a right on Johnson Street and go 2.3 miles. Then turn right on Constitution Avenue, go 1.4 miles and merge onto X street. Go .7 miles down X street....
Which set of directions would you want to follow? The Internet is the same way. Routers will look at their neighbor and choose the more specific path. I watched another carrier in early 2001 take a large percentage of my carrier Internet traffic to Europe by mis-configuring their BGP tables. They announced a more specific route for large portions of our routing table. It took a while, but we finally got them to notice what they had done. On the other end, I've watched customers mis-configure their own routers and try to suck down as much traffic as they could. We could handle the traffic easily, but they could not. We configure our customers with very specific elements to prevent these types of mis-configurations. I can't, however, stop another large carrier from fat-fingering a configuration. When they do, we work with other carriers to blackhole their routing announcements to minimize any issues.
The Internet is amazing in its resiliency to handle traffic the way it does. When you are configuring BGP on your network, make sure that you understand that your configuration does have the potential to impact those around you.
Our team is working on metrics for last year and it gave me a chance to reflect on what works well and what doesn't. As I looked at the statistics, I was quite pleased and definitely proud of our team. We had several hardware issues that were difficult to push through as we worked with the hardware vendors. The goal of any carrier is to maintain a core network that meets or exceeds 99.999% reliability. The reality of that number is less than 6 minutes of total downtime in a year out of potential 525,600 minutes in a year. I actually have held our team accountable to 5 minutes and 18 seconds per year. We were able to meet 99.999% (around 3 minutes total downtime) on our Internet core and 100% on our other networks. Does this mean that we didn't have any customers go down? Unfortunately, no. On the core network nodes that all customers cross, we hit our numbers. On direct links, we did have customers go down. As copper plants (DSL, T1s, DS3s) continue to age, the number of failures will continue to grow over time. :-(
So how does this relate to fragility on the Internet core? That question takes us to the basic infrastructure of how an enterprise or carrier network is setup. Most large networks are setup using OSPF and BGP. There are multiple ways for network engineers to configure the network and I rarely find one engineer that agrees with another. (Can you say "standards"!) To understand how this is setup using two protocols, you need to understand how BGP determines a best path. Imagine you are headed to Washington D.C. to watch the presidential inauguration. When you get to DC, you ask for directions. Here are your two sets of directions:
Person 1: To get to the party, go down this street and you should get there in about 30 minutes.
Person 2: To get to the party, take a right on Johnson Street and go 2.3 miles. Then turn right on Constitution Avenue, go 1.4 miles and merge onto X street. Go .7 miles down X street....
Which set of directions would you want to follow? The Internet is the same way. Routers will look at their neighbor and choose the more specific path. I watched another carrier in early 2001 take a large percentage of my carrier Internet traffic to Europe by mis-configuring their BGP tables. They announced a more specific route for large portions of our routing table. It took a while, but we finally got them to notice what they had done. On the other end, I've watched customers mis-configure their own routers and try to suck down as much traffic as they could. We could handle the traffic easily, but they could not. We configure our customers with very specific elements to prevent these types of mis-configurations. I can't, however, stop another large carrier from fat-fingering a configuration. When they do, we work with other carriers to blackhole their routing announcements to minimize any issues.
The Internet is amazing in its resiliency to handle traffic the way it does. When you are configuring BGP on your network, make sure that you understand that your configuration does have the potential to impact those around you.
Wednesday, January 14, 2009
Virtual Desktop Infrastructure
I listened last night to a presentation on Virtual Desktop Infrastructures (VDI). Interesting thoughts from Chris Horton from Yorel (www.yorel.com). He discussed the history of moving from "green screens" to the current options of how and when to deploy virtual desktops. I'm not sure that I want to go back to mainframe types of services, but are the new solutions going to be the same? Chris reviewed the details and options, vendor agnostic, and covered what would work best. There doesn't appear to be any Capex savings, only Opex savings. I don't see a large rush for people moving to virtual desktops. In a market where everyone is working to cut costs in both Capex and Opex, there needs to be a difinitive ROI and IRR in order to justify the expense or cost cut.
If anyone has good feedback on how they have implemented virtual desktops in the new VDI environment, I would love to hear what works and does not work.
If anyone has good feedback on how they have implemented virtual desktops in the new VDI environment, I would love to hear what works and does not work.
The Future of the Communications World - Serving the Customer
The Future of the Communications World - Serving the Customer.
How I wish that I had the crystal ball that would show us the best path to follow. I talk with various vendors on a regular basis who promise that they are the best and their solution will take us into the next century. Today's announcement that Nortel is filing for bankruptcy started me think, again, along the paths of how best to serve the customer. The customer is different depending on what company you work for. Do you sell to enterprise customers? Do you sell to residential customers? Do you sell to government agencies? Do you care? Many companies spend a lot of time trying to define who the customer is. The definition changes with new products/services, the economy, and, well, the new marketing person.
Who is the customer?
The basic definition usually comes down to two specific points. There is an Internal customer and an external customer. The internal customer is usually your employees in your department or another department within your company. The external customer is the customer you have sold a product or service to outside your company. There is nothing relevational regarding this definition. So why is it so hard for many companies to stay focused? If you look at companies such as Nortel and the others that have filed for bankruptcy, it is usually poor decisions made throughout a company.
What does the customer want?
Many people in technology positions look first at the technology and then how the technology can serve the customer. This works well in some cases, but not in many. The customer is not looking for a technology. Not really. The customer is looking for a solution that will solve the job they want done. It may be a particular technology or it may be a solution combining multiple technologies. Each customer has a job that they want done. Be it washing clothes, sending email, paying bills, moving from one place to another, communications, teaching, etc. The customer just wants to get the job done. If you follow this basic premise, then you understand where to start to find the right technology or solution for your internal or external customer. Price plays a role in the purchase of a solution for a job, but not always. If it always played a role, Apple would not have sold millions of iPhones or iPods. People are willing to pay more if you have the only solution to get a job done. If you are just another vendor, well, get ready for a volume game in order to make money on a commodity product.
How do you choose which way to serve the customer?
This one is easier. The primary focus of any company that wants to stay in business is to build a relationship with a customer. Yes, you can build a relationship with a customer even if you are selling a widget. Apple has a huge following from their customer base. They spent a great deal of time trying to understand the job that the customer needed done. Then they built a product/widget that met those needs. By listening to the customer, they built a relationship that goes deep and strong. Nortel, ironic enough, used to do the same thing. Nortel spent a great deal of time building products that the customer needed. First, it was a product that no one else had and they could charge a premium for the product(s). Then they built a relationship by setting up regular and constant visits/communication with the customer. They were good about bringing customers into the lab and showing them how their product. They did multiple testing scenarios and really took care of the customer.
Why does Apple succeed and why is Nortel failing?
Look back at the history of what Apple has achieved and where they have failed. Apple has had plenty of successes and failures. Many people attribute the success of Apple only to Steve Jobs. Jobs is a driving force in their success, but he is not the only one. He has the vision, determination, and focus, but other people do the work. Apple went downhill when Jobs left and grew to dominance when he returned. Compare Nortel. They have been a leader for a long, long time in several areas. In particular, we have used their phone switch DMS gear and their SONET gear. The systems were just well built and ran. They had their share of problems. But between a system that worked and the relationship with the customer, they built a strong company. The problem from my viewpoint is that Nortel missed the change in the winds. They produced phone systems where a new feature (i.e. Caller ID) would cost a carrier $50k +. They had SONET gear that was about as redundant as you could get. The winds changed, the customer changed their need for the job, and new products/solutions/services were needed. Customers wanted solutions where they could get 50 features on a phone instead of just 3 for the same price. Hence, VoIP became stable and took off. Customers wanted flexibility in networks and wanted to be able to grow. Ethernet moved from a LAN technology to a WAN technology with scalability. Convergence arrived. VoIP and Ethernet changed the landscape.
I'm not saying that VoIP and Ethernet caused the problem for Nortel. I'm saying that the customer's needs changed. They wanted something different and "better". Nortel was not able to move in the right direction. Customers bought more of the other solutions. Nortel hung on to current products. Customers bought more of other solutions. Nortel stayed firm. Customers moved and Nortel tried change. But it was too late. The new innovators came in with new solutions to solve the needs of the customer and simply took the customers away from Nortel. Now their isn't enough money from current customers to research and deliver new products. Too little too late.
VoIP and Ethernet were not, initially, the best techonologies. Ethernet still does not provide a customer the level of redundancy that SONET does. But everyone wants the flexibility of Ethernet and will accept a lower level of performance - for now. Ethernet, however, will either provide the same level of redundancy over time or it will be replaced. (As a comparison, we achived 99.999% uptime on a core MPLS/VPLS network for 2008.)
Conclusion
Apple has succeeded because they listened to the customer and are currently delivering a product that meets the need for the job. They have the hammer to pound the nail. It won't last though. Plenty of competitors are on their tails. Nortel missed the boat and failed to respond to the basic principle. Meet the needs of the customer for the job they want done when they want it done.
Good Reading.
If you are interested in getting more details along these lines with some good data, take a look at these two books to help undestand the customer focus and the internal disciple needed to be successful.
- The Innovator's Dilemma: The Revolutionary Book that Will Change the Way You Do Business - by Clayton M. Christensen (Author)
- Good to Great: Why Some Companies Make the Leap... and Others Don't - by Jim Collins (Author)
** All views are my own. Feedback always welcome. :)
How I wish that I had the crystal ball that would show us the best path to follow. I talk with various vendors on a regular basis who promise that they are the best and their solution will take us into the next century. Today's announcement that Nortel is filing for bankruptcy started me think, again, along the paths of how best to serve the customer. The customer is different depending on what company you work for. Do you sell to enterprise customers? Do you sell to residential customers? Do you sell to government agencies? Do you care? Many companies spend a lot of time trying to define who the customer is. The definition changes with new products/services, the economy, and, well, the new marketing person.
Who is the customer?
The basic definition usually comes down to two specific points. There is an Internal customer and an external customer. The internal customer is usually your employees in your department or another department within your company. The external customer is the customer you have sold a product or service to outside your company. There is nothing relevational regarding this definition. So why is it so hard for many companies to stay focused? If you look at companies such as Nortel and the others that have filed for bankruptcy, it is usually poor decisions made throughout a company.
What does the customer want?
Many people in technology positions look first at the technology and then how the technology can serve the customer. This works well in some cases, but not in many. The customer is not looking for a technology. Not really. The customer is looking for a solution that will solve the job they want done. It may be a particular technology or it may be a solution combining multiple technologies. Each customer has a job that they want done. Be it washing clothes, sending email, paying bills, moving from one place to another, communications, teaching, etc. The customer just wants to get the job done. If you follow this basic premise, then you understand where to start to find the right technology or solution for your internal or external customer. Price plays a role in the purchase of a solution for a job, but not always. If it always played a role, Apple would not have sold millions of iPhones or iPods. People are willing to pay more if you have the only solution to get a job done. If you are just another vendor, well, get ready for a volume game in order to make money on a commodity product.
How do you choose which way to serve the customer?
This one is easier. The primary focus of any company that wants to stay in business is to build a relationship with a customer. Yes, you can build a relationship with a customer even if you are selling a widget. Apple has a huge following from their customer base. They spent a great deal of time trying to understand the job that the customer needed done. Then they built a product/widget that met those needs. By listening to the customer, they built a relationship that goes deep and strong. Nortel, ironic enough, used to do the same thing. Nortel spent a great deal of time building products that the customer needed. First, it was a product that no one else had and they could charge a premium for the product(s). Then they built a relationship by setting up regular and constant visits/communication with the customer. They were good about bringing customers into the lab and showing them how their product. They did multiple testing scenarios and really took care of the customer.
Why does Apple succeed and why is Nortel failing?
Look back at the history of what Apple has achieved and where they have failed. Apple has had plenty of successes and failures. Many people attribute the success of Apple only to Steve Jobs. Jobs is a driving force in their success, but he is not the only one. He has the vision, determination, and focus, but other people do the work. Apple went downhill when Jobs left and grew to dominance when he returned. Compare Nortel. They have been a leader for a long, long time in several areas. In particular, we have used their phone switch DMS gear and their SONET gear. The systems were just well built and ran. They had their share of problems. But between a system that worked and the relationship with the customer, they built a strong company. The problem from my viewpoint is that Nortel missed the change in the winds. They produced phone systems where a new feature (i.e. Caller ID) would cost a carrier $50k +. They had SONET gear that was about as redundant as you could get. The winds changed, the customer changed their need for the job, and new products/solutions/services were needed. Customers wanted solutions where they could get 50 features on a phone instead of just 3 for the same price. Hence, VoIP became stable and took off. Customers wanted flexibility in networks and wanted to be able to grow. Ethernet moved from a LAN technology to a WAN technology with scalability. Convergence arrived. VoIP and Ethernet changed the landscape.
I'm not saying that VoIP and Ethernet caused the problem for Nortel. I'm saying that the customer's needs changed. They wanted something different and "better". Nortel was not able to move in the right direction. Customers bought more of the other solutions. Nortel hung on to current products. Customers bought more of other solutions. Nortel stayed firm. Customers moved and Nortel tried change. But it was too late. The new innovators came in with new solutions to solve the needs of the customer and simply took the customers away from Nortel. Now their isn't enough money from current customers to research and deliver new products. Too little too late.
VoIP and Ethernet were not, initially, the best techonologies. Ethernet still does not provide a customer the level of redundancy that SONET does. But everyone wants the flexibility of Ethernet and will accept a lower level of performance - for now. Ethernet, however, will either provide the same level of redundancy over time or it will be replaced. (As a comparison, we achived 99.999% uptime on a core MPLS/VPLS network for 2008.)
Conclusion
Apple has succeeded because they listened to the customer and are currently delivering a product that meets the need for the job. They have the hammer to pound the nail. It won't last though. Plenty of competitors are on their tails. Nortel missed the boat and failed to respond to the basic principle. Meet the needs of the customer for the job they want done when they want it done.
Good Reading.
If you are interested in getting more details along these lines with some good data, take a look at these two books to help undestand the customer focus and the internal disciple needed to be successful.
- The Innovator's Dilemma: The Revolutionary Book that Will Change the Way You Do Business - by Clayton M. Christensen (Author)
- Good to Great: Why Some Companies Make the Leap... and Others Don't - by Jim Collins (Author)
** All views are my own. Feedback always welcome. :)
Subscribe to:
Posts (Atom)