Wiera: Policy-Driven Multi-Tiered Geo-Distributed Cloud Storage Instances

Many cloud providers have multiple data centers (DCs) in multiple locations and they offer a range of storage services, each of these offers different tradeoffs between cost, performance, and durability. Application developers typically need to write code for a data placement policy in multiple DCs and multiple storage tiers based on their desired goals. However, writing code for diverse data placement policies is not an easy task because (of)

• Different applications emphasize different factors e.g., consistency model, SLA, degree of fault tolerance and so on to yield policies
• Complexities from multiple storage tiers across cloud providers
• Dynamics from cloud environment and applications

Wiera is a policy driven key-value storage system for geo-distributed cloud environment built upon our storage system called Tiera. A Tiera instance is used as a local Instance to handle multiple storage tiers within a DC and a Wiera instance logically contains multiple local instances on multiple DCs. Wiera enables an application to specify flexible storage instances that comprise of different kinds of storage in different DCs (e.g., Amazon S3 in US East, Windows Azure storage in US West, Edge Storage etc.) and data placement policy achieve the desired tradeoff between cost, durability, and performance. Wiera also enables an application to handle dynamics from cloud infrastructure and applications by changing data placement policy at run-time.

Architecture

• Wiera User Interface (WUI)

The Wiera User Interface (WUI) provides an API to applications to manage Wiera instances. The API allows applications to: launch multiple local instances as part of a Wiera instance with a global policy specification, stop instances, and get the list of currently running instances.

• Global Policy Manager (GPM)

Global Policy Manager (GPM) creates a new policy for a Wiera instance. It stores metadata for the policy and executes a Local Instance Manager (LIM) to manage the local instances which belong to the Wiera instance.

• Local Server Manager (LSM)

Local Server Manager (TSM) manages local servers at different locations, which spawn and remove local instances based on application requests. For instance, if the application calls startInstances through WUI to start local instances at Region 1 and Region 2, LSM will direct the local servers in Region 1 and Region 2 to each spawn a new local instance.

• Data Placement Manager (TripS)

Data Placement Manager (TripS) finds the optimized data placement (DC locations and storage tiers) with given inputs e.g., applications' goals, cost information, workload information and latency information to relax application developers' burdens to find optimized data placement in a multi-cloud environment. Please visit TripS project page for more detail.

Goals

We are designing our Wiera framework with several goals in mind:

• Build an integrated geo-distributed cloud storage system that runs both within and across data-centers owned by different cloud providers
• Provide an easy way to specify policies for multi-tiers accross different cloud providers' DCs
• Provide a first-class support for handling network and application dynamics within the storage policies to achieve user metrics (e.g., reduced cost, latency, and so on)
• Provide a flexibility that allows unmodified applications to furhter reap benefits by replacing data/storage policies externalized at run-time

Related Projects

• Tiera: Towards Flexible Multi-Tiered Cloud Storage Instances


• TripS: Automated Multi-tiered Data Placement in a Geo-distributed Cloud Environment