LHC Tunnel

LHC Tunnel

Friday 2 March 2018

Expiry of VMs in the CERN cloud

The CERN cloud resources are used for a variety of purposes from running compute intensive workloads to long running services. The cloud also provides personal projects for each user who is registered to the service. This allows a small quota (5 VMs, 10 cores) where the user can have resources dedicated for their use such as boxes for testing. A typical case would be for the CERN IT Tools training where personal projects are used as sandboxes for trying out tools such as Puppet.

Personal projects have a number of differences compares to other projects in the cloud
  • No non-standard flavors
  • No additional quota can be requested
  • Should not be used for production services
  • VMs are deleted automatically when the person stops being a CERN user
With the number of cloud users increasing to over 3,000, there is a corresponding growth in the number of cores used by personal projects, growing by 1,200 cores in the past year. For cases like training users, there is often the case that the VMs are created and the user then does not remember to delete the resources so they consume cores which could be used for compute capacity to analyse the data from the LHC.



One possible approach would be to reduce the quota further. However, tests such as setting up a Kubernetes cluster with OpenStack Magnum often need several VMs to perform the different roles so this would limit the usefulness of personal projects. The usage of the full quota is also rare.



VM Expiration

Based on a previous service which offered resources on demand (called CVI based on Microsoft SCVMM), the approach was taken to expire personal virtual machines.
  • Users can create virtual machines up to the limit of their quota
  • Personal VMs are marked with an expiry date
  • Prior to their expiry, the user is sent several mails to inform them their VM will expire soon and how to extend it if it is still useful.
  • On expiry, the virtual machine is locked and shutdown. This helps to catch cases where people have forgotten to prolong their VMs.
  • One week later, the virtual machine is deleted, freeing up the resources.

Implementation

We use Mistral to automate several OpenStack tasks in the cloud (such as regular snapshots and project creation/deletion). This has the benefit of a clean audit log to show what steps worked/failed along with clear input/output states supporting retries and an authenticated cloud cron for scheduling.

Our OpenStack projects have some properties set when they are created. This is used to indicate additional information like the accounting codes to be charged for the usage. There are properties for indicating if the type of project such as personal and if the expiration workflow should apply. Mistral YAQL code can then select resources where expiration applies.

task(retrieve_all_projects).result.select(dict(id => $.id, name => $.name, enabled => $.enabled, type => $.get('type','none'),expire => $.get('expire','off'))).where($.type='personal').where($.enabled).where($.expire='on')

The expire_at parameter is stored as a VM property. This makes it visible for automation such as CLIs through the openstack client show server CLI.

There are several parts to the process
  • A cron trigger'd workflow which
    • Machines in error state or currently building are ignored
    • A newly created machine which does not have an expiry date set has the expiration date set according to the grace period
    • Sees if any machines are entering close to their expiry time and sends a mail to the owner
    • Checks for invalid settings of the expire_at property (such as people setting it a long way in the future or deleting the property) and restores a reasonable value if this is detected
    • If a machine has reached it's expiry date, it's locked and shutdown
    • If a machine has past it's date by the grace period, it's deleted
  • A workflow, launched by Horizon or from the CLI
    • Retrieves the expire_at value and extends it by the prolongation period
The user notification is done using a set of mail templates and a dedicated workflow (https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/blob/master/workflows/send_mail_template.yaml). This allows templates such as instance reminders to have details about the resources included, such as the example from the mail template.

The Virtual Machine {instance} from the project {project_name} in the Cloud Infrastructure Service will expire on {expire_date}.

A couple of changes to Mistral will be submitted upstream
  • Support for HTML mail bodies which allows us to have a nicer looking e-mail for notification with links included
  • Support for BCC/CC on the mail so that the OpenStack cloud administrator e-mail can also be kept on copy when there are notifications
A few minor changes to Horizon were also done (currently local patches)
  • Display expire_at value on the instance details page
  • Add a 'prolong' action so that instances can be prolonged via the web by using the properties editor to set the date of the expiry (defaulting to the current date with the expiry time). This launches the workflow for prolonging the instance.

Author

Jose Castro Leon from the CERN cloud team

References



69 comments:

  1. Montedo é o ministro da economia brasileiro e os nossos carros já estão com o IPVA SP pagos e o Licenciamento anual em dia. E o nosso Coluna do Flamengo ? dssd

    ReplyDelete
  2. Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us.

    Check out : Hadoop training in Chennai
    big data analytics training and placement
    big data training in chennai chennai tamilnadu
    big data workshop in chennai

    ReplyDelete
  3. More great information! Thanks Andrew! Definitely taking your recommendations.
    open stack course

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. IEEE final year projects on machine learning In case you will succeed, you have to begin building machine learning projects in the near future.

    Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


    Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

    ReplyDelete
  6. This is most informative and also this post most user friendly and super navigation to all posts. Thank you so much for giving this information to me.microsoft azure training in bangalore







    ReplyDelete
  7. If you are guilty of letting your bread machine collect dust when you aren't making bread, its time to change things up a bit. The benefits of a bread machine aren't limited to making fresh baked bread - they include excellent recipes for everything from jams, pasta doughs, cakes to quick breads and pound cakes. Bread is just the tip of the iceberg. In this article, you'll learn a few of the many dishes you can prepare in a bread maker that aren't bread. dumpling maker

    ReplyDelete
  8. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our Data Science course Mumbai
    data science course mumbai | https://www.excelr.com/data-science-course-training-in-mumbai

    ReplyDelete
  9. This article is worthy of recognition and comment. I found this material attention-grabbing and engrossing. SAP training in Kolkata
    Best SAP training in Kolkata
    SAP training institute in Kolkata

    ReplyDelete
  10. I had to read this three times because I wanted to be sure on some of your points. I agree on almost everything here, and I am impressed with how well you wrote this article.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata
    Best SEO company in kolkata
    Top SEO company in kolkata
    Top SEO services in kolkata
    SEO services in India
    SEO copmany in India

    ReplyDelete
  11. Reading this article gave me many things to think about. You have some quality information here that any reader would enjoy. I share many of your views in this article.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata
    SAP training institute Kolkata

    ReplyDelete
  12. Nice information. I felt that the process is similar but the mode of work has been changed.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata

    ReplyDelete
  13. Other substance online can't match the work you have put around here. Your knowledge regarding this matter has persuaded me regarding a large number of the focuses you have communicated. This is incredible one of a kind composition.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata
    Best SEO company in kolkata
    Top SEO company in kolkata
    Top SEO services in kolkata
    SEO services in India
    SEO copmany in India

    ReplyDelete
  14. The candidates must have a good command of mathematics and statistics to comprehend huge figures. artificial intelligence course in hyderabad

    ReplyDelete
  15. Very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing, best data science courses in Hyderabad

    ReplyDelete
  16. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work. data science training in Hyderabad

    ReplyDelete
  17. It's so great to know you are a writer that cares about the information you provide. This is smartly done and well-written in my opinion.


    IT managed services provider in UK

    ReplyDelete
  18. I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site.
    data science training in Hyderabad

    ReplyDelete
  19. Through this post, I know that your good knowledge in playing with all the pieces was very helpful. I notify that this is the first place where I find issues I've been searching for. You have a clever yet attractive way of writing.
    data analytics courses

    ReplyDelete
  20. "Very good article with very useful information. Visit our websitedata science training in Hyderabad
    "

    ReplyDelete
  21. Informative information, Thanks.
    Buy Abortion Pill Online to end early pregnancy at home.

    ReplyDelete
  22. I recently found a lot of useful information on your website, especially this blog page. Among the lots of comments on your articles. Thanks for sharing.
    data scientist training and placement

    ReplyDelete
  23. I simply couldn’t go away your web site before suggesting that I actually loved the standard info a person provide for your guests? Is gonna be again steadily to inspect new posts.
    토토사이트
    토토

    ReplyDelete
  24. you have a great blog here! would you like to make some invite posts on my blog?
    경마
    온라인경마

    ReplyDelete
  25. Thanks a bunch for sharing this with all people you actually recognise what you’re speaking approximately! Bookmarked.
    토토사이트
    토토

    ReplyDelete
  26. Wonderful blog. I delighted in perusing your articles. This is genuinely an incredible perused for me. I have bookmarked it and I am anticipating perusing new articles. Keep doing awesome!
    data analytics courses in hyderabad with placements

    ReplyDelete
  27. Nice blog and impressive. Useful information for many people. Keep sharing more blogs with us. Thank you.
    Data Science Course in Hyderabad

    ReplyDelete
  28. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. data analytics course in surat

    ReplyDelete
  29. I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site.data science training in jabalpur

    ReplyDelete
  30. This CERN Cloud approach for VM expiry is clever and efficient. Mistral's automation of the process reduces resource waste and encourages appropriate usage. The email notifications and processes keep people informed. Excellent initiative!
    Data Analytics Courses in India

    ReplyDelete
  31. This VM expiry implementation at CERN is a clever method for properly managing resource usage while preserving user comfort and system integrity. A well-rounded strategy is demonstrated by the mix of automated workflows, user alerts, and resource recovery. Congratulations to the team for their hard work!
    Data Analytics Courses in India

    ReplyDelete
  32. I think this is a very well-designed solution that addresses the problem of unused VMs consuming resources. The use of a cron-triggered workflow ensures that the process is automated and that VMs are not forgotten about. The use of mail templates makes it easy to send clear and concise notifications to users. Thank you.
    Data Analytics Courses in Nashik

    ReplyDelete
  33. This article outlines a practical approach to managing personal virtual machines in the CERN cloud, addressing resource consumption and expiration effectively. The automation using Mistral is commendable for ensuring resource efficiency. Great insights!
    Is iim skills fake?

    ReplyDelete
  34. It's wonderful to know that you are a writer who genuinely cares about the knowledge you offer. I think this is wonderfully written and smartly done.
    Data Analytics Courses in Agra

    ReplyDelete
  35. Hi,
    The author's detailed explanation of managing VM expiration in the CERN cloud using OpenStack and Mistral is immensely informative and practical. Sharing this knowledge is greatly appreciated.
    Data Analytics Courses In Dubai

    ReplyDelete
  36. Thank you so much for posting this detailed post on the topic of the Expiry of VMs in the CERN cloud. I was having some trouble with this and now it is cleared after reading your blog. Keep posting more.
    Visit - Data Analytics Courses In Bangalore

    ReplyDelete
  37. good blog
    Data Analytics Courses In Vadodara

    ReplyDelete
  38. The insights provided regarding the expiry of VMs in the CERN cloud shed light on the dynamic nature of cloud-based resources and the challenges associated with resource management.
    Digital Marketing Courses in Hamburg

    ReplyDelete
  39. I appreciate your sharing of this information. The next occasion I encounter a blog, I have a positive outlook that it will offer a more captivating and contemplative experience.
    data Analytics courses in leeds

    ReplyDelete
  40. I appreciate you for sharing this information. When I encounter another blog in the future, I am hopeful that it will offer a more captivating and intellectually stimulating experience
    data Analytics courses in leeds

    ReplyDelete
  41. I appreciate you sharing this information. When I encounter another blog in the future, I look forward to a more captivating and stimulating experience.
    data Analytics courses in leeds

    ReplyDelete
  42. Thank you to the CERN team for their dedication to maintaining a reliable and efficient cloud infrastructure."
    Digital Marketing Courses in Italy

    ReplyDelete
  43. Thanks for sharing valuable and informative post on Expiry of VMs in the CERN cloud.
    data analyst courses in limerick

    ReplyDelete
  44. "It's crucial for CERN to manage VM expiry effectively in their cloud infrastructure. This ensures optimal resource utilization and cost efficiency. Implementing automated processes for VM lifecycle management could be a game-changer. Looking forward to seeing how they address this challenge!"
    Best Data analytics courses in India

    ReplyDelete
  45. Thank you for sharing in depth knowledge and insights on Expiry of VMs in the CERN cloud.
    Adwords marketing

    ReplyDelete
  46. Smart approach to manage CERN cloud resources! The VM expiration system ensures efficiency and helps users stay informed. Thanks for sharing!

    Investment Banking Industry

    ReplyDelete
  47. Thank you for sharing information regarding Expiry of VMs in the CERN cloud.
    Investment banking training Programs

    ReplyDelete
  48. BuddyRoute's website expertly combines educational reviews, engaging features, and the most latest the best rental car in fannet rakyatnesia industry news to give you the impression that you're travelling with an informed buddy.

    ReplyDelete
  49. The concise explanations and to-the-point examples make the content easy to grasp. Thanks for sharing this informative blog with us. Great blog. keep posting.
    Data analytics framework

    ReplyDelete
  50. Thanks for this awesome blog post. I have been struggling with some of the concepts for very long and you made understand them a walk in the park. Cheers.
    Investment banking analyst jobs

    ReplyDelete