Brief Description of the Netflix Company

NETFLIX

Background of the company prior to Cloud Migration

Netflix was co-founded in 1997 by Reed Hastings who previously started a software company and by Mark Randolph who was the software executive in the company. In 1998, they launched their website where people can buy their film(DVD’S) on what they are choosing. But, when the Amazon started selling DVD’s Netflix stopped selling DVD’s and started to rent them and later in the year 1999, they offered the subscription service where people can rent unlimited DVD’s. In 2000, they stopped the single rental model completely and started focusing on the subscription services by making the films available exclusively by partnering with Warner Brothers and Columbia pictures and also established distribution centers across America to increase the speed of delivery and provide the customers overnight service option.

Business Drivers/Goals behind cloud migration (E.g. TCO, Time to market)

Migrating to the Cloud helps to increase overall operational agility of a company. This helps a lot while responding to market needs more quickly when compared to the traditional on-premises model. A high-level study found that companies will be able to reduce their total cost of ownership (TCO) by an average of around 60 percent over a five-year life cycle by moving to AWS cloud services. Data centers hosting, Infrastructure costs and IT Staff costs reduced significantly. Unplanned downtime, unexpected network changes were cut down. These were the business drivers that inspired Netflix to move to cloud. Also, taking advantage of AWS capabilities and scalability, focus on reinvention will help in innovation and transform business by speeding time to market.

Technology Drivers/Goals (E.g. Improved Availability by X, scalability, performance, Security)

Availability:

There was a time in 2008, when Netflix had its data center and running its own relational databases. Netflix data center was down and has undergone a major database corruption because of which the company were unable to ship the DVD’s for 3 days. For any company to improve its business they need to overcome bad customer experience, that is when they started thinking of having a reliable database platform that reduces critical issues especially downtime. Migrating to cloud provides high availability (99.99% for example).

Scalability:

Before moving to cloud, Netflix team spent a lot time and effort in increasing the processors, disk space etc, to meet timely public demands. After they migrated to cloud, it became easy for them to upscale and downscale the resources for IT requirements. Simply Netflix need not worry about the architectural limits. Netflix was fast growing, they had to handle ever increasing volumes of data in their data centers and were not sure with the increase in volumes of data is it the best way to go with data centers. That is when they have planned to migrate to cloud which will allow them to handle lot of virtual servers and petabytes of data storage within less time.

Performance:

Netflix has started the Online streaming in the year 2007, with the large number of customers using the online streaming resource the traffic has increased and they were not able to provide customers with better performance during the peak streaming hours. So, migrating to cloud they can use EC2 instances to improve the performance during peak hours of streaming. AWS Cloud is pay as you use model we can leverage the EC2 instances based on the traffic. Customer facing any issues with streaming won’t benefit their business. Performance is one of the major driving factor for their business growth.

Security:

For any industry security is one of the major factor. Having all the data in the data center in one place makes it unsecure as all operations route via data center, moving to the amazon cloud helps them to not worry about the patching and the low-level infrastructure as everything is taken care by the cloud provider which makes it more secure.

Migration strategy used for cloud migration

Netflix followed the strategy of “Migrate Features One at a Time (roman riding)”. So, what is roman riding? Let me explain it with an example in general terms:

• Roman Riding is the name of a Circus stunt, where rider starts off with one horse and after that parellely he rides on both horse and later while finishing the stunt he ends up on one horse. How can we relate to Netflix migration?
• In this strategy, they had the own infrastructure and AWS infrastructure running side by side slowly they migrated the applications to AWS infrastructure (Front end applications) first with backend still in the data center. Slowly they replicated the backend data using home grown framework known as IR (Item Replication) into AWS S3 and SimpleDB. The main functionality of IR is to keep the data in sync with the Data Center and the S3 and SimpleDB.
• So basically, they have refactored the monolithic application into hundreds of micro services and they have migrated them one at a time starting with Front end applications. Slowly once the billing and customer stuff is migrated into the cloud they have retired the traditional data centers.

Organizational Change

Cloud architecture is completely handled by the AWS (COE), Netflix doesn’t have any IT staff working on cloud. Capacity of cloud is more than 10 times bigger than the Data center, Netflix data center staff were reduced and became flat. Netflix even moved few of their IT staff to working on the coding parts of the applications. As the Data center necessity is overcome there is less need of the System Admins, Database Admins, Storage and Network admins (Traditional IT roles are becoming extinct). Instead of long transitions they have organized cloud camps to provide transition to developers which is simple and short and ready to work model. New transition includes some of the features below:

1.  Orientation for one day.
2.  Use Netflix PaaS platform to design simple “Hello World”.
3. Security Integration and monitoring.
4. Read or writes to Cassandra.

Netflix follows flat organization structure for better job satisfaction and better communication among the lower levels of management.

Cultural Change

Employee is given a full end-end responsibility of the application(DevOps), a simple diagram which explains everything.

DevOps is nothing but development and operations team combined into one team. Where the associates will completely own the application, they are building. (For Example, in the below image you are working for Netflix and you developed some application in devops model you will the one who is responsible for testing, releasing that to customers and support that application if any issues, in case of any issues re-plan the design to fix them and do all the process again to make it available to customers).

Netflix before migrating into cloud they have followed the Waterfall approach to build their applications, test and deploy them to make it available to customers. Having migrated to cloud they have adopted to Agile methodologies and followed continuous integration and continuous deployment patterns to make the services available to customers at faster rate. Earlier there was a large team responsible for the developing the Monolithic application but now micro services architecture is adopted as whole monolithic application is divided into sub applications and large team is also divided into small team and each team is completely responsible for the applications they develop.

Migration Journey

Below is the simple diagram that explains the migration journey of Netflix. I felt that pictorial representation will be helpful in better understanding before going into the description part of the migration journey.

Netflix has started its online streaming services in 2007. In 2008, due to major hardware failure company was not able to ship the DVD’s for 3 days which was a major setback.

Online streaming business was growing faster, most of the people prefer using streaming services compared to DVD rentals and the traffic coming into data centers via online streaming is quite high and they were in situation to establish more data centers to handle it. With the high usage of online streaming, management were down to two choices:

1. Recruit world class datacenter operations build teams and guess how much capacity they would need, and built it before it was needed – lot of upfront $$ investment.
2. Use the Elastic compute services of AWS and spend $$ on video content and developers.

Keeping the above problems in mind, Netflix management came up with 5 goals (Availability, Scalability, Performance, Agility and Security). After long research, they came up with the solution to migrate to AWS Public cloud.

The journey of cloud migration in Netflix started with migrating some application which weren’t really customer facing. They didn’t have enough space in the datacenters for encoding backlogs. They moved to EC2 to show that capacity existed on demand. The next thing they moved is Quality of Service(QoS) logging into S3 for unlimited storage space and log analysis to Elastic Map Reduce(EMR) service. They decided not to build anymore datacenters and moved front workloads to AWS before 2010. They migrated web pages and API clients to AWS and had backend in the datacenters. Slowly, they moved customer data from datacenters to AWS and were able to backup their data at lower costs. In the last step, they migrated corporate IT and billing to cloud and closed their data centers.

Where are they now in their journey?

In 2008, they started their journey. It took almost 7years to completely migrated to cloud, in early January 2016 Netflix officially announced they have completely migrated to cloud and shut down the last remaining data center used by streaming service. We can say that Netflix is a fully migrated cloud industry.

Migration journey into cloud was not easy they have faced lot of difficulties and it took almost 7 years to completely migrate everything. Few lessons learnt by Netflix during their migration:

Few wrong choices initially, NOSQL before Cassandra, API platform prior to Zuul.

All data center tools don’t cope up too well with cloud. Initially data center IT staff faced lot of challenges in writing the code and learning the new stuff.

If migration journey had started a year or two later would have been more stable and mature and discussion to get it done rite would have been much easier. When their migration started they were almost the starting company to adopt to this migration and didn’t knew much about it and it outcomes.

They even came to know not everything is perfect and system can fail even in cloud. In 2012 during Christmas eve company suffered a streaming failure when it was hosted on one AWS region and it went down. So, they have built Simon Army and Chaos Monkey tools to test the up-front disaster in advance and make sure their application is not getting effected by any type of disasters.

Netflix has rolled its services to 190 countries and has almost 130 million subscribers around the globe. This growth would be almost impossible if they had been still using data centers as fast racking of the servers would have been a problem.

Netflix had a problem in past on their products availability, migrating to cloud they were able to achieve 99.999% uptime for their resources.

Netflix had 10 times the customers they had 6 years ago and 100 times the online traffic using their resources which is a trade mark sign for any industry.

Cost is one of the major outcome of the cloud migration, before migration they had to upfront invest on the data centers without knowing the outcome. But, now in cloud it provides the flexibility to pay only for the resources we can use. Infrastructure is all done in cloud so cost reduction in maintaining data centers and racking servers and IT staff to maintain them.

In my view, the approach followed by Netflix is the best one and even if I was to embark on that journey I would go with the same approach. Please find my justification to that below:

Rehosting (Lift and Shift): Netflix could have followed this simple and very easy approach but by doing lift and shit they will just transfer the existing issues or problems into cloud and it would not help their business outcome. Shifting the existing application to cloud and later making changes into it as they desire could be easy but doing that they would have bought the bad architecture and bad practices along with it. So, I feel this approach is not suitable for the company who has its customers all over the world.

Platforming (Lift and Reshape): Just by migrating the servers and Database to the AWS wouldn’t solve the problems of availability, performance and reduce latency in case of Netflix. As Netflix application is Monolithic, there are chances that there can be issues in large application and cause problems to the customers if any piece of code doesn’t work properly. In Monolithic application, everything is inter-related even if something doesn’t work it effects the whole process. So, I feel this strategy might not be suitable for Netflix.

Repurchasing: They have purchased the Virtual machines of AWS to run their operations. As AWS is pay as you use model it helped them save cost.

Refactoring/Re-architecting: This is the strategy followed by Netflix and even I would like to use the same. As Netflix had large Monolithic application it is very difficult for them to manage it so using this strategy they have re-designed the architecture but building hundreds of micro services and each team is given complete ownership of the application they develop which is easy to maintain. They followed the Devops approach with each team take complete end to end ownership of their application. Even in case of any issues it is very easy to track them and solve them with less SLA compared to huge monolithic application.

Retire: Having migrated to cloud, in my view there is no point in maintain the traditional data centers anymore and they are too costly as well. I would have used this strategy to get rid of the data centers after the cloud migration is completed successfully. As there is no point in having data centers going forward.

Retain: Having migrated to cloud successfully and business has business had tremendous growth after cloud migration I feel that everything is in place in new system to support their business and they need not worry about retaining any of the old stuff they had earlier.          

Cite this page