LHC Tunnel

LHC Tunnel

Thursday, 21 April 2016

Containers and the CERN cloud

In recent years, different groups at CERN started looking at using containers for different purposes, covering infrastructure services but also end user applications. These efforts have been mostly done independently, resulting in a lot of repeated work especially for the parts which are CERN specific: integration with the identity service, networking and storage systems. In many cases, the projects could not complete before reaching a usable state, as some of these tasks require significant expertise and time to be done right. Alternatively, they found different solutions to the same problem which led to further complexity for the supporting infrastructure services. However, the use cases were real, and a lot of knowledge had been built on the available tools and their capabilities.

Based on this, we started a project with the following goals:
  • integrate containers into the CERN OpenStack cloud, building on top of already available tools such as resource lifecycle, quotas, identity and authorization
  • stay container orchestration agnostic, allowing users to select any of the most common solutions (Docker Swarm, Kubernetes, Mesos)
  • allow fast cluster deployment and rebuild
We had done a prototype using the Nova LXC driver in the past but the long term support was not clear and we wanted access to the native container functions using the standard tools.

Looking for other possibilities, OpenStack Magnum seemed to be offering a lot of what we needed, and we decided to try it out. At around the same time we were also heading to the OpenStack Tokyo summit, which was a great opportunity to follow the Magnum sessions and learn more of what it provides.

Magnum relies heavily on Heat for the orchestration part of the container clusters - called bays. Bays are instantiated based on pre-defined bay models, which set how the master and the other nodes should look like (flavor, image, etc) and which container orchestration engine (COE) should be used - among other possible configuration options. Current choices include Docker Swarm, Kubernetes and Mesos. The Magnum homepage gives a lot more details.

At the beginning of November 2015, we started investigating the Magnum project in depth. At that time the project was functional but some of its requirements posed problems in our deployment:
  • Dependency on OpenStack Neutron, something we had not yet deployed (we have nova-network since we started the cloud in 2012). Luckily we were working on it in parallel, and we got a functional control plane just in time. And as we use Nova Cells, we could enable Neutron in a dedicated cell where we would also enable Magnum, reusing the rest of the production infrastructure
  • Requirement on Neutron LBaaS, which we don't have. This is something we plan to try, but it is not obvious how to implement this currently due to the way the CERN network is structured. We made some changes to the Heat templates to remove this requirement
The other pre-requisite projects, such as Keystone, Glance and Heat were already in production in the CERN cloud.

But no real show stoppers and very quickly we got a prototype deployment. For a more detailed evaluation we initially chose 3 internal projects that cover the most common use cases:
  • GitLab CI, a continuous integration service we use internally - it has integration with Docker, making it a perfect example of how to use a Docker Swarm cluster as a drop in replacement for a local Docker daemon
  • Infrastructure services, namely one of the critical services for the data movement between the multiple sites of the LHC Computing Grid (WLCG) - for a nice example of scaling a service by scaling its individual components
  • Jupyter Notebooks - a growing trend for end user analysis in different scientific communities, providing a browser based interactive session running in a remote container
In addition, we are also working with the European Union Horizon 2020 project Indigo Datacloud which is developing an open source data and computing platform targeted at scientific communities, deployable on multiple hardware and provisioned over hybrid, private or public, e-infrastructures. Using Magnum, we can provide the test resources for this project to the partners.

For our users and resource managers, there are significant advantages of the Magnum approach:
  • Native tools - anything that works with Docker will work talking to a Docker Swarm COE or kubectl with a Kubernetes cluster. This allows smaller physics sites to provide native Docker or Kubernetes while the larger sites provide Containers-as-a-Service on-demand. User applications written to work with Docker or Kubernetes can be used without modification against the provisioned resources.
  • Container engine agnostic - with our user community, there is a strong need for flexibility to allow different avenues to be explored. Magnum allows the IT department to offer Kubernetes, Docker Swarm and Mesos at a low cost within the same service at an affordable load for the support team. The users can then prototype different application approaches and select the best combination for them. Enforcing a central IT service decision on the end user community is never easy, especially where there are diverse user requirements being covered within a central cloud.
  • Accounting, quota and permissions remains within the existing framework. Thus, whether resources are used for containers or VMs is a choice for the project user. Capacity planning can be done by cores/RAM rather than segmentation of resources for container or VM resources. Access controls follow the existing admin/member structures for projects. 
  • Elasticity - within the quota limits, containers can scale, with new bays as needed within the quota. This allows the resources to be allocated where there is a user need (and as importantly, shrunk when things are quiet)
  • Repairs - failures in the infrastructure (software or hardware) are looked after by the cloud support team. For the user, the workloads can be scheduled elsewhere. For the hardware repair teams, the operations can be performed in a consistent fashion in bulk rather than on a one-by-one basis. Infrastructure monitoring procedures are the same for VMs and containers.
  • The operating system support teams can provide reference images and follow up issues with the upstream providers. They can be confident that the image is based on supported configurations rather than ad-hoc builds. Rebuilding base images with the appropriate security patches can sometimes be delayed, raising the risk of incidents.
By the end of March, we had the use cases covered, and the few hick-ups covered in blueprints or patches upstream, and had contributed for the missing bits in puppet and documentation. And with a service running on our production resources and thanks to keystone endpoint filtering, we could increase service usage by enabling it for individual projects. Today we have around 15 different projects using Magnum as a pilot service and the number keeps growing.

In just a few months, we got Magnum up and running and it has proved to be a significant addition to the OpenStack cloud. Which makes us excited about what is coming next, including:
  • Integration with Cinder - ready upstream, and we'll be trying it very soon
  • Magnum benchmarks in Rally - we rely on Rally to make sure our cloud is performing as expected
  • Further integration with our local storage systems such as CVMFS and EOS - relying on the ability to add site specific configurations to the bay templates
  • Integration with Barbican - the recommended way to handle the required TLS certificates to talk to the native APIs of the orchestration engines, and the only option today to get Magnum in HA (though that's about to change)
  • Integration with Horizon - this will help as we expand the service into production to communities who are used to using the web interfaces
If you're interested in more details on the available container orchestration technologies or our usage of OpenStack Magnum, or simply want to see some fancy demos, check our recent presentation at a CERN Technical Forum.

Acknowledgments

  • Mathieu Velten for his work on testing and adapting Magnum at CERN and contributions to Indigo DataCloud
  • Bertrand Noel for all his time spent researching existing container technologies
  • Spyros Trigazis, a fellow in the CERN OpenLab collaboration with Rackspace, for all his work upstream both for features and documentation improvements
  • Jarek Polok for the CERN docker repository
  • The OpenStack Magnum team for their support and collaboration
  • All CERN users that helped us debug and set the service requirements

References


45 comments:

  1. Really you have done a great job, There are may person searching about that topic. now they will easily find your post
    Docker Training in Hyderabad
    Kubernetes Training in Hyderabad
    Docker and Kubernetes Training
    Docker and Kubernetes Online Training

    ReplyDelete
    Replies
    1. Great Article Cloud Computing Projects

      Networking Projects

      Final Year Projects for CSE

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. Hi,
    Certified data center professional

    A certified data center expert must understand the design life cycle of the data center and the various stages included. It is your responsibility to follow the levels of redundancy for the design and configuration of the data center.

    For More: Certified data center professional

    ReplyDelete
  3. I work as a marketing specialist and staff author at Externetworks which is a pioneer in Managed Technology Services with over 17+ years of experience in providing end-to-end solutions featuring design, deployment and 24*7 support to top IT companies. We offer world-class managed services for businesses to stay agile & profitable. Our services include 24/7 Network Monitoring, Uptime maintenance, NOC Support, IT Helpdesk services.

    Read more at: Career of a NOC Technician

    ReplyDelete
  4. Nice post,  very useful blogs with very useful information, thank you for sharing this post  noc monitoring services 

    ReplyDelete
  5. Our experience with students enables us to determine the fastest and most effective teaching methods that, Allah willing, make them achieve the best results in memorization of the Holy Quran within the shortest possible period of time misinterpretation of the quran

    ReplyDelete
  6. This article is deserving of acknowledgment and remark. I discovered this material eye catching and charming. This is very much scripted and profoundly useful. These perspectives offer to me. This is the way genuine composing is finished. Much thanks to you.


    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata
    Best SEO company in kolkata
    Top SEO company in kolkata
    Top SEO services in kolkata
    SEO services in India
    SEO copmany in India

    ReplyDelete
  7. To be honest I found very helpful information your blog thanks for providing us such blog IPL Betting

    ReplyDelete
  8. To be honest I found very helpful information your blog thanks for providing us such blog Dream11 IPL 2020: DC vs RCB Preview

    ReplyDelete
  9. This is really best article for new internet users , You are a very skilled blogger and content writing. I’ve joined your feed and look forward to seeking more of your excellent and so helpful post. Also, I’ve shared your website in my Facebook! Thank you. assignments help australia -
    nursing assignment help -
    university assignment help australia

    ReplyDelete
  10. My friend suggested I might like this article website. He was saying right that you blog very informative and match our study. I agree with her. Your content is very helpful for me; I am also writing on educational topics, I really like thanks so much.


















    ReplyDelete
  11. I have voiced some of the articles info on your website now and, I really like your blogging style and many ideas. I added it to my list of favorite blogging website and will be back soon. Thank you my dear friend for sharing many useful thoughts.
    myob assignment help -
    assignment writers -
    buy assignment online

    ReplyDelete
  12. Is it accurate to say that you are feeling stuck under a monster heap of your confounded and extensive programming assignments, which appear hard to wrap up? Indeed, fret no more. At ABC, we can help you with probably the most capable programming assignment specialists who can help you with your assignments at the most moderate and pocket-accommodating rates. programming assignment help , matlab assignment help

    ReplyDelete
  13. To make your writing more successful, you should go investigate the idea of the PEEL section. It assists you with giving the correct arrangement to your writing and keep on the right track. Critically, PEEL Paragraph is useful when you are dealing with your article. In article writing, you need to introduce your contentions to accentuate a subject.

    ReplyDelete
  14. Accounting assignment help allows you to connect with experts and get effective assistance for your project submission. Don’t forget to check out all the features before buying the services of online assignment help for your accounting papers.

    ReplyDelete
  15. I am always left astounded at the level of dedication and hard work you put in every situation. May you reach every height of success!
    tamilprint 2021
    soap2day 2021
    9xrockers 2021
    klwap2021
    extramovies 2021
    skymovieshd 2021
    300mbmovies 2021
    123movierulz 2021

    ReplyDelete
  16. Assignment help from GoAssignmentHelp is surely a thing you don't want to miss out on. AssignmentHelp offers you the law assignment help in Australia. GoAssignmentHelp has a team of highly experienced writers programming assignment help they have already set a benchmark with their work all around the world. Our team consists of many Ph.D. assignment experts who will help you with expert java assignment help assistance in every way possible.

    ReplyDelete
  17. excellent post, very informative. I wonder why the other experts of this sector don’t notice this. You should continue your writing. I am confident, you have a huge readers’ base already! 토토

    ReplyDelete
  18. Wonderful post however , I was wanting to know if you could write a litte more on this subject? I’d be very grateful if you could elaborate a little bit further. Bless you! 경마

    ReplyDelete
  19. It is in point of fact a nice and helpful piece of info. I’m happy that you shared this helpful info with us. Please stay us up to date like this. Thank you for sharing. 카지노사이트

    ReplyDelete
  20. I gotta favorite this site it seems extremely helpful very helpful 토토

    ReplyDelete
  21. Hello to all, it’s genuinely a pleasant for me to pay a quick visit this website, it includes valuable Information. 야설

    ReplyDelete
  22. I finally found great post here. 카지노사이트 Thanks for the information. Please keep sharing more articles.

    ReplyDelete
  23. Excellent Blog! I would like to thank you 바카라사이트for the efforts you have made in writing this post.

    ReplyDelete
  24. Thank you for sharing this information. I read your blog and I can't stop my self to read your full blog. 토토사이트 Again Thanks and Best of luck to your next Blog in future.

    ReplyDelete
  25. Hard to ignore such an amazing article like this. You really amazed me with your writing talent. 카지노사이트Thank you for sharing again.

    ReplyDelete
  26. Thanks for writing this informative content, keep sharing.

    The right sales tracker software will help you identify the root cause behind restricted sales. The right steps will allow sales staff to easily analyze customers' actions. This means that sales success is largely dependent on the sales process.

    ReplyDelete
  27. The law assignment helper regards Taxation Law Assignment Help to be an inclusion of illegal action which doesn’t interest the criminal liberty. The assignments are generally contract or civil liabilities. These assignments offer the understanding of rules, regulations, rights of citizens, and conceptual legal remedies.

    ReplyDelete
  28. Opt for our Assignment Help to finish your assignments and submit them within the deadline. No need to take the stress of insufficient knowledge and lack of skills when you have an online assignment writing expert for writing your assignment in Qatar.

    ReplyDelete
  29. Best of luck with your future as your regular reader of your blog I am giving you the appraisal that if you get any ups down you can reach me out at any time. 바카라사이트

    ReplyDelete
  30. Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with some pics to drive the message home a little bit, but other than that, this is magnificent blog. A great read. I’ll certainly be back. 바카라사이트

    ReplyDelete
  31. I like what you guys are up also. Such clever work and reporting! Carry on the excellent works guys I?|ve incorporated you guys to my blogroll. I think it will improve the value of my website. 바카라사이트

    ReplyDelete
  32. It’s always a pleasure to read your magnificent articles on this site. You are among the top writers of this generation, and there’s nothing you can do that will change my opinion on that. My friends will soon realize how good you are. 토토사이트

    ReplyDelete
  33. Assignment writing is a significant difficult undertaking for generally every one of the students. What makes it significantly more upsetting is the impact it has on the grades and the moving toward cutoff times. In case you are ending up under these burdening conditions and are wildly looking for some assignment help to back out the entire cycle for you, then, at that point ABC assignment help is your all inclusive resource. matlab assignment help

    ReplyDelete
  34. Wow! Such an amazing article I really really loving your each blogs which your sharing on this website. It's so good and so awesome, Trust me guys, you archive so big goals after shar like this post on internet. I hope that you continue to do your work like this in the future also. Thank you and Wish you very good luck for your many new postings. Australia Assignment Help - best assignment help in brisbane - best assignment help in perth - Online Assignment Help

    ReplyDelete
  35. The information shared is really helpful and so useful to get to know about various suggestions and the information shared is rich in content. Please keep us up to date like this work in future also. Thanks for sharing your knowledge and, I'm glad I came into this article because it provides a lot of important information. Keep it guys. You doing so well, I know that :) nursing assignment help - english assignment help - case study help

    ReplyDelete
  36. Thanks for sharing this article. I also work in the same area and provide Assignment Help to students for various subjects. Most of my students are from parts of the UK, USA, India, and Singapore. Our team of professors is from reputed Colleges. Visit our website to know our areas of expertise and let us know if we can be of help.

    ReplyDelete
  37. How to get a cash app refund if a transaction is sent to the wrong person?
    How to get a Cash App Refund if a transaction is sent to the wrong person? Like many other cash app users if you also want the answer of this question then you have knocked on the right door. You will know here how to get a refund on the cash app if the money is sent to the wrong person, you just have to click on this link and after that, it’s just a matter of seconds.

    ReplyDelete
  38. Supply chain management assignment:: This subject spotlights on dealing with the progression of labor and products from natural substances into end results. Assuming you think that it is precarious to deal with the subjects of this subject, then, at that point, take inventory network the executives assignment composing administrations in Australia and free yourself from composing stresses. my assignment help

    ReplyDelete