Wix Media Service Fortifies Its Disaster Recovery Cluster on Google Cloud Platform
Download 25.78 Kb.
|
- Navigate this page:
- “ Web Creation Made Simple”
- Google Cloud Storage We used Cloud Storage to store our static media files. Google Compute Engine
- Datastore Consistency
Wix Media Service Fortifies Its Disaster Recovery Cluster on Google Cloud Platform Editor’s Note: This article is written by guest author Eugene Olshenbaum. Eugene is the Head of Media Services at Wix, a cloud-based web development platform that makes it easy for everyone to create beautiful websites. About Wix“Web Creation Made Simple”
Wix provides technology that makes it simple for everyone to create a stunning, professional, and functional web presence. It is a cloud-based web development platform that allows users to create HTML5 websites and mobile sites through the use of a powerful online editor. Wix was founded in 2006, has 36 million registered users from all around the world, and is growing rapidly. OverviewThis technical case study describes how Wix replicated their services, which are running on their managed data centers, to Google Cloud Platform to increase availability and improve disaster recovery. Taking advantage of the features provided by App Engine, Compute Engine, and Cloud Storage, Wix was able to complete the migration in a very short time. This case study also includes key lessons learned during the migration process. The ChallengeAs a website building and serving platform, we want to provide close to 100% uptime for data serving and protect our user data against loss. We originally ran our service in one managed hosting environment. To improve data disaster recovery, we added a second one, running both services in active/active mode. Later, we added a third data center to run our services in 3x active/active mode. However, we learned that maintaining three cross-data center replicas was much more complex than managing two, especially with the data centers owned by different ISPs for ISP redundancy. One of the challenges in 3x active/active mode was database replication. To replicate across three data centers, we had to configure our MySQL in a ring topology. The ring would break when one data center went down for a long time or failed completely. To address this, instead of implementing 3x active/active mode with our current infrastructure, we decided to run in 2x active/active mode, with the third replica running on an entirely different technology platform. The third replica also added protection against data poisoning—when a faulty piece of code unintentionally corrupts data and remains undetected for some time.
Solving the ChallengeWe decided to build a full replica of our service on Google Cloud Platform and port all system components to work efficiently on Google Cloud. This replica would be capable of working as a primary server without dependencies to the Wix servers in our managed data centers. We chose Google Cloud Platform for the following reasons: Ease of Management
Scalability
Speed of Development
Our system uses the following Cloud Platform components: App EngineWe built the following two applications on App Engine:
Google Cloud StorageWe used Cloud Storage to store our static media files. Google Compute EngineWe do all content manipulations, such as image cropping or resizing, on the fly. Although App Engine provides an Image API for image manipulation, we require image processing capabilities such as noise reduction, sharpening, and image filters, which are not provided by the API. We have highly optimized code written in C for this purpose. We developed a set of servers on Compute Engine that handles image manipulation using our code. To optimize performance, we use a high-memory instance type and perform all the I/O on the RAM disk. This is the only component that requires attention to its scalability and health management. Figure 1 shows the high-level architecture of our serving system on Google Cloud Platform:  Figure 1: High-level architecture of the Wix media serving system
Next, we had to choose a database. Our existing application was built on top of MySQL. MySQL has scaling issues and lacks features we require, such as full text search, arbitrary attributes data structure (image files have completely different sets of attributes than audio or PDF files), and paging based on multiple criteria. Instead of continuing development with MySQL, we decided to use the App Engine Datastore since it fits our fuzzy data models better and could scale with our growth. Since we wanted to keep the Wix servers in our managed data center and Google Cloud running at the same time, we had to develop bidirectional synchronization between the systems at the application level.
Replication ProcessTo synchronize data between the Wix managed hosting and Google Cloud Platform systems, we had to build a replication supervisor. In our first attempt, we replicated the metadata and static files synchronously as they were uploaded to our current system (Figure 2).  Figure 2: First version of the replication supervisor
However, replication often failed with a DeadlineExceeded error. App Engine Frontend Instances are designed to respond to a request within 60 seconds. Our initial design included too many operations in a single request, and their execution depended on various external factors such as file size and network jitter. We learned that, in general, App Engine Frontend handlers should be as small as possible and involve at most one or two Datastore requests. If a handler is more complex, it should be split and chained into smaller asynchronous blocks. Task Queue was the perfect match for our revised design. Tasks are executed asynchronously and, instead of 60 seconds, have a 10-minute deadline. Since the time it takes to copy media files varies greatly from request to request, we used task queues to make file replication an asynchronous process. After the metadata is replicated and saved in the App Engine Datastore, our application adds a task to the task queue. The task is then pushed to another handler that handles the copying. If copying is successful, a new task is added to the task queue to notify the initiator that the transaction is complete. If copying fails, the system retries, until it completes the task successfully. Figure 3 shows the revised process.  Figure 3: Final design of the replication supervisor
DatastoreConsistencyThe App Engine Datastore is designed for scalable web applications. It manages scaling, availability, and replication across multiple data centers automatically. When data is replicated, there is a delay from the time a Datastore write is committed until the change becomes visible in all data centers. If an application has to wait for replication to complete (and data to be consistent across all data centers) before reading the data, its throughput can be affected.
A good example is a blog application. A reader’s own comment to a blog post should be strongly consistent; that is, they should see their comment after posting it. On the other hand, comments posted by other users can be eventually consistent; that is, the reader just needs to see them eventually. To balance data throughput and consistency, developers can choose when eventual data consistency is sufficient, and when they require strong consistency for their application. Coming from using MySQL, this was a concept that we had to grasp.
Entity GroupsData objects in the App Engine Datastore are called entities. Entities can be arranged in parent-child relationships to form entity groups. An entity group is a group of tightly linked entities to achieve strong consistency and transactionality. Strong consistency is achieved by limiting a query to a single entity group. For our application, we need to keep track of users and the media they upload for their websites. Querying the user data has to be strongly consistent. So, we created an entity group for each user and their media information. MemcacheDatastore limits the frequency of writes to an entity group in order to provide strong consistency. Since our application has a high write rate, we ran into concurrent modification exceptions. To reduce write contention, we decided to use memcache to manage user data updates. Changes are stored in memcache and pushed to the Datastore once a minute using deferred tasks. We understood that we might lose data if the memcache was flushed, but our application is designed specifically with that in mind, and no user data or files are actually lost. ConclusionBy using Google Cloud Platform, we were able to rewrite our Java-MySQL application to a Python App Engine application that was ready for integration tests in just two weeks. App Engine provides a rich set of APIs and managed services that freed us from the chore of writing scalability and fault tolerance-related code. We were able to solve our business problem quickly and seamlessly. We have started serving our production media traffic from Compute Engine as a primary server and performance is impressively good and stable. Directory: files files -> Answer True False 2 points Question 2 files -> Integer programming and game theory files -> 4 Integer Programming files -> Bpa vehicle Window Repair Scenario #1 task: Procure vehicle window relacement. Objective files -> Pop Warner History files -> North Carolina Inclusion Initiative Mapping Where Children with ieps are Being Served Purpose files -> Fall 2013 Spring 2014 Program Data: Standard 1 Exhibit 4d files -> Hanban – asia society confucius classrooms network 2010 request for proposal files -> Northern England’s set-jetting locations Download 25.78 Kb. Share with your friends:
|