LHC Tunnel

LHC Tunnel

Friday, 2 March 2018

Expiry of VMs in the CERN cloud

The CERN cloud resources are used for a variety of purposes from running compute intensive workloads to long running services. The cloud also provides personal projects for each user who is registered to the service. This allows a small quota (5 VMs, 10 cores) where the user can have resources dedicated for their use such as boxes for testing. A typical case would be for the CERN IT Tools training where personal projects are used as sandboxes for trying out tools such as Puppet.

Personal projects have a number of differences compares to other projects in the cloud
  • No non-standard flavors
  • No additional quota can be requested
  • Should not be used for production services
  • VMs are deleted automatically when the person stops being a CERN user
With the number of cloud users increasing to over 3,000, there is a corresponding growth in the number of cores used by personal projects, growing by 1,200 cores in the past year. For cases like training users, there is often the case that the VMs are created and the user then does not remember to delete the resources so they consume cores which could be used for compute capacity to analyse the data from the LHC.



One possible approach would be to reduce the quota further. However, tests such as setting up a Kubernetes cluster with OpenStack Magnum often need several VMs to perform the different roles so this would limit the usefulness of personal projects. The usage of the full quota is also rare.



VM Expiration

Based on a previous service which offered resources on demand (called CVI based on Microsoft SCVMM), the approach was taken to expire personal virtual machines.
  • Users can create virtual machines up to the limit of their quota
  • Personal VMs are marked with an expiry date
  • Prior to their expiry, the user is sent several mails to inform them their VM will expire soon and how to extend it if it is still useful.
  • On expiry, the virtual machine is locked and shutdown. This helps to catch cases where people have forgotten to prolong their VMs.
  • One week later, the virtual machine is deleted, freeing up the resources.

Implementation

We use Mistral to automate several OpenStack tasks in the cloud (such as regular snapshots and project creation/deletion). This has the benefit of a clean audit log to show what steps worked/failed along with clear input/output states supporting retries and an authenticated cloud cron for scheduling.

Our OpenStack projects have some properties set when they are created. This is used to indicate additional information like the accounting codes to be charged for the usage. There are properties for indicating if the type of project such as personal and if the expiration workflow should apply. Mistral YAQL code can then select resources where expiration applies.

task(retrieve_all_projects).result.select(dict(id => $.id, name => $.name, enabled => $.enabled, type => $.get('type','none'),expire => $.get('expire','off'))).where($.type='personal').where($.enabled).where($.expire='on')

The expire_at parameter is stored as a VM property. This makes it visible for automation such as CLIs through the openstack client show server CLI.

There are several parts to the process
  • A cron trigger'd workflow which
    • Machines in error state or currently building are ignored
    • A newly created machine which does not have an expiry date set has the expiration date set according to the grace period
    • Sees if any machines are entering close to their expiry time and sends a mail to the owner
    • Checks for invalid settings of the expire_at property (such as people setting it a long way in the future or deleting the property) and restores a reasonable value if this is detected
    • If a machine has reached it's expiry date, it's locked and shutdown
    • If a machine has past it's date by the grace period, it's deleted
  • A workflow, launched by Horizon or from the CLI
    • Retrieves the expire_at value and extends it by the prolongation period
The user notification is done using a set of mail templates and a dedicated workflow (https://gitlab.cern.ch/cloud-infrastructure/mistral-workflows/blob/master/workflows/send_mail_template.yaml). This allows templates such as instance reminders to have details about the resources included, such as the example from the mail template.

The Virtual Machine {instance} from the project {project_name} in the Cloud Infrastructure Service will expire on {expire_date}.

A couple of changes to Mistral will be submitted upstream
  • Support for HTML mail bodies which allows us to have a nicer looking e-mail for notification with links included
  • Support for BCC/CC on the mail so that the OpenStack cloud administrator e-mail can also be kept on copy when there are notifications
A few minor changes to Horizon were also done (currently local patches)
  • Display expire_at value on the instance details page
  • Add a 'prolong' action so that instances can be prolonged via the web by using the properties editor to set the date of the expiry (defaulting to the current date with the expiry time). This launches the workflow for prolonging the instance.

Author

Jose Castro Leon from the CERN cloud team

References



58 comments:

  1. Montedo é o ministro da economia brasileiro e os nossos carros já estão com o IPVA SP pagos e o Licenciamento anual em dia. E o nosso Coluna do Flamengo ? dssd

    ReplyDelete
  2. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information regarding Microsoft Azure which is latest and newest,

    Regards,
    Ramya

    Azure Training in Chennai
    Azure Training Center in Chennai
    Best Azure Training in Chennai
    Azure Devops Training in Chenna
    Azure Training Institute in Chennai
    Azure Training in Chennai OMR
    Azure Training in Chennai Velachery
    Azure Online Training
    Azure Training in Chennai Credo Systemz
    DevOps Training in Chennai Credo Systemz

    ReplyDelete
  3. Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us.

    Check out : Hadoop training in Chennai
    big data analytics training and placement
    big data training in chennai chennai tamilnadu
    big data workshop in chennai

    ReplyDelete
  4. More great information! Thanks Andrew! Definitely taking your recommendations.
    open stack course

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. Very informative blog i found this very help full to get information.
    industrial suppliers in pakistan

    ReplyDelete


  7. Soma pill is very effective as a painkiller that helps us to get effective relief from pain. This cannot cure pain. Yet when it is taken with proper rest, it can offer you effective relief from pain.
    This painkiller can offer you relief from any kind of pain. But Soma 350 mg is best in treating acute pain. Acute pain is a type of short-term pain which is sharp in nature. Buy Soma 350 mg online to get relief from your acute pain.

    https://globalonlinepills.com/product/soma-350-mg/


    Buy Soma 350 mg
    Soma Pill
    Buy Soma 350 mg online



    Buy Soma 350 mg online
    Soma Pill
    Buy Soma 350 mg

    ReplyDelete
  8. The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. IEEE final year projects on machine learning In case you will succeed, you have to begin building machine learning projects in the near future.

    Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


    Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

    ReplyDelete
  9. This is most informative and also this post most user friendly and super navigation to all posts. Thank you so much for giving this information to me.microsoft azure training in bangalore







    ReplyDelete
  10. If you are guilty of letting your bread machine collect dust when you aren't making bread, its time to change things up a bit. The benefits of a bread machine aren't limited to making fresh baked bread - they include excellent recipes for everything from jams, pasta doughs, cakes to quick breads and pound cakes. Bread is just the tip of the iceberg. In this article, you'll learn a few of the many dishes you can prepare in a bread maker that aren't bread. dumpling maker

    ReplyDelete
  11. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    data analytics courses

    data science interview questions

    business analytics course

    data science course in mumbai

    ReplyDelete
  12. This is a nice article you shared great information I have read it thanks for giving such a wonderful Blog for the reader.
    Openstack Training
    Openstack Training Online
    Openstack Training in Hyderabad

    ReplyDelete
  13. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    best servicenow online training

    ReplyDelete
  14. Awesome blog. I enjoyed reading your articles. This is truly a great read for me.


    data analytics course

    data science course

    business analytics course

    ReplyDelete
  15. I have read your blog its very attractive and impressive. Very systematic indeed! Excellent work!
    Data Science Course Training in Bangalore

    ReplyDelete
  16. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions
    Data Science Course

    ReplyDelete
  17. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science training

    ReplyDelete
  18. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our data analytics courses mumbai
    data analytics course mumbai | https://www.excelr.com/data-analytics-certification-training-course-in-mumbai

    ReplyDelete
  19. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

    data science course

    ReplyDelete
  20. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our Data Science course Mumbai
    data science course mumbai | https://www.excelr.com/data-science-course-training-in-mumbai

    ReplyDelete
  21. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    data science course

    ReplyDelete
  22. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
    data analytics course in Bangalore

    ReplyDelete
  23. This article is worthy of recognition and comment. I found this material attention-grabbing and engrossing. SAP training in Kolkata
    Best SAP training in Kolkata
    SAP training institute in Kolkata

    ReplyDelete
  24. It's very nice to find out other writers share like minds on some content. This is the case with your article. I really enjoyed this.
    SAP training in Mumbai
    Best SAP training in Mumbai
    SAP training institute Mumbai

    ReplyDelete
  25. I had to read this three times because I wanted to be sure on some of your points. I agree on almost everything here, and I am impressed with how well you wrote this article.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata
    Best SEO company in kolkata
    Top SEO company in kolkata
    Top SEO services in kolkata
    SEO services in India
    SEO copmany in India

    ReplyDelete
  26. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance

    ReplyDelete
  27. Reading this article gave me many things to think about. You have some quality information here that any reader would enjoy. I share many of your views in this article.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata
    SAP training institute Kolkata

    ReplyDelete
  28. Nice information. I felt that the process is similar but the mode of work has been changed.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata

    ReplyDelete
  29. Other substance online can't match the work you have put around here. Your knowledge regarding this matter has persuaded me regarding a large number of the focuses you have communicated. This is incredible one of a kind composition.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata
    Best SEO company in kolkata
    Top SEO company in kolkata
    Top SEO services in kolkata
    SEO services in India
    SEO copmany in India

    ReplyDelete
  30. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  31. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  32. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  33. The candidates must have a good command of mathematics and statistics to comprehend huge figures. artificial intelligence course in hyderabad

    ReplyDelete
  34. Thanks for sharing nice information. I highlyrecommend you data science training Hyderabad

    ReplyDelete