OpenStack Folsom Design Summit

Nova is too monolithic. It's difficult to develop external components without affecting the core. Nova-core reviewers are expected to be specialists in everything. We need to split Nova into implicit (same project) or explicit parts (separate projects). How can this be accomplished ? What are the no-brainer candidates ? Where are the pain points ? What can we plan to do over the Folsom timeframe (Session lead is Thierry Carrez)

Nova has an RPC mechanism used to communicate between services. There is no way to detect potential incompatibilities between services due to version skew. I would like to discuss approaches to make these interfaces more robust so that failures due to interfaces not matcing up can be explicit and obvious. (Session lead is Russell Bryant)

As per recent discussion in the mailing list - discus how to expose to the scheduler which group of host support a particular abstract subset of flavors/images/capabilities Host Aggregates may provide the base for this - but it is currently tied to hypervisor tyep capabilities (Session lead is Phil Day)

The DMTF is working with a number of Cloud vendors and consumers to develop an IaaS API called CIMI (Cloud Infrastructure Management Interface). This sessions is focused on designing an implementation as an optional alternative to the existing OpenStack and EC2 APIs. A brief overview of the specification will be given with a mapping of CIMI concepts and APIs to existing OpenStack APIs. Discussions will be centered on where the implementation would exist, on the ideal architecture for invoking APIs across the entire spectrum of OpenStack projects, and on addressing incompatibilities between the spec and existing OpenStack APIs. (Session lead is Doug Davis)

This is a collection of related ideas to improve the management of Nova Services: i) Add a text field to capture the reason a service is disabled. Disabling a specific services is normally a manual operation, carried out for some specific operational reason, such as an issue on a specific server, or to reserve some servers as part of capacity management. We’ve found that adding a reason field which can be set when the service is disabled and displayed as part of the service status makes it much easier to manage this aspect of the system. The messages are normally short and refer to, for example, issues captured in JIRA. ii) In addition to updating the services table with a timestamp and report count, services should report the number of threads currently running in their greenpool – as this provides an indication of how busy the service is. This information could be used, for example, by schedulers. iii) Extend the services table to include a status field that is set by the service to “starting/started/stopping/stopped” at the appropriate points in its lifecycle. This helps distinguish between services that have stopped as part of a software update from those that have failed for some reason. iv) Service shutdown currently kills any threads which are in progress. Service shutdown should try to stop the service in a clean state by stopping reading new messages from the queue and waiting for the running thread count to reduce to 0. If the threads don’t stop within a configurable timeout window, then the service should force a stop anyway. (Session lead is Phil Day)

We would like to manage a number of items which are defined per tenant in an alternative central system. The first step is to make the following parts of the system pluggable drivers: - ssh keys - quotas - security group definitions. The default driver would keep this behaviour in the Nova database as per the current implementation, but this would allow alternative drivers to be developed. It would be good to get feedback on this idea and identify any other similar areas. (Session lead is Phil Day)

During the Diablo cycle work was done to make nova-compute work for ARM processors. We need to finish the work that we started to make ARM a first class supported architecture for folsom. (Session lead is Chuck Short)

As discussed https://lists.launchpad.net/openstack/msg07581.html , config-drive is less OS agnostic than we would like it to be. I think that a discussion would lead to a more general purpose mechanism for communicating data from platform to guest without a network. (Session lead is Scott Moser)

There needs to be a standard around how network information get injected into images. Some possibilities are the following: 1. Usage of https://fedorahosted.org/netcf/ (or its client library) inside of nova when injecting into the image 2. Usage of https://fedorahosted.org/netcf/ inside a VM (nova would write the agnostic format into the VM) and having software inside the VM on firstboot interpret that format (cloud-init??) 3. Using the current ubuntu network format that is already written, but change where it is written to an agreed upon location and perform #2 (the ubuntu format would then be the "agnostic" format) 4. Do nothing and cry. (Session lead is Joshua Harlow)

At the Essex design summit I presented an idea to limit the operations on a Instance based on the combination of vm_state and task_state. We tried to make the model quite restrictive to eliminate as many race conditions as possible (for example rebuilds occurring in the middle of a rebuild). However having run with this model for some time we have found it necessary to allow more operations that we at first envisaged to cope with Instances that get “stuck” in a particular state (for example is a service is restarted whilst a rebuild is in progress the Instances can get stuck in “Rebuilding-Spawing”). Only allowing delete in this case (which is the operation which always leads to a deterministic end state no matter what it’s paired with) isn’t a great user experience, but allowing other operations (such as a further rebuild) creates more potential race conditions that have to be handled in the code. I therefore propose moving to a model which can block operations on a combination of vm_state, task_state, and updated_at – leading to three possible rules: i) Operations which are always allowed ii) Operations which are always blocked iii) Operations which are allowed only if now()- instance[updated_at] is greater than some thresholds (which can be different for each operation) So for example, for an Instance which has the state Active-None Start, Unpause, Resume, UnRescue would always be blocked Other operations would always be allowed Whereas for an instance which has the state Active-Rebooting: Start, Unpause, Resume, and Unrescue would always be blocked Terminate, Reboot, and Rebuild would only be allowed if the now()-instance[updated_at] is greater than 3 minutes I believe this consideration of “how long since the last time the state changed” comes very close to the symanatic of limiting each instance to only one concurrent operation whilst still being safe against failed operations and without having to make major changes to the overall concurrency model. I would also like to explore with the community the degree to which these rules should be configurable vs hard coded (for example some might see a need to be able to configure to always allow operations such as delete which have a billing impact). (Session lead is Phil Day)

Need for an orchestration layer/ service has been visited and talked about during Essex summit (http://etherpad.openstack.org/orchestration). However, Orchestration couldn't be materialized during Essex. This session is to revisit the need, identify the gaps, brainstorm and review design options and come up with a plan to implement the same. Summary from previous Summit: Nova needs to be able to monitor and manage long-running transactions. For example, when creating 1,000 servers, it may be necessary to create a build plan, retry failed builds, refresh the plan, and possibly even roll back the entire transaction if it cannot be completed rapidly enough. To do this, an orchestration service is proposed that can monitor workflows around Nova events and transactions. (Session lead is Sriram Subramanian)