LHC Tunnel

LHC Tunnel

Friday, 17 June 2016

Scaling Magnum and Kubernetes: 2 million requests per second

Two months ago, we described in this blog post how we deployed OpenStack Magnum in the CERN cloud. It is available as a pre-production service and we're steadily moving towards full production mode as a standard part of the CERN IT service offerings to give Containers-as-a-Service.

As part of this effort, we've started testing the upgrade procedures, the latest being to the final Mitaka release. If you're here to see some fancy load tests, keep reading below, but some interesting details on the upgrade:
  • We build our own RPMs to include a few patches from post-Mitaka upstream (the most important being the trustee user to support lifecycle operations on the bays) and some CERN customizations (removal of neutron LBaaS and floating ips which we don't yet have, adding the CERN Certificate Authority, ...). Check here for the patches and build procedure
  • We build our Fedora Atomic 23 image to get more recent versions of docker and kubernetes (1.10 and 1.2 respectively), plus support for an internal distributed filesystem called CVMFS. We do use the upstream disk-imagebuilder procedure with a few additional elements available here
While discussing how we could further test the service, we thought of this kubernetes blog post, achieving 1 million requests per second against a service running on a kubernetes cluster. We thought we could probably do the same. Requirements included:
  • kubernetes 1.2, which our recent upgrade offered
  • available resources to deploy the cluster, and luckily we were installing a new batch of a few hundred physical nodes which could be used for a day or two
So along with the upgrade, Bertrand and Mathieu got to work to test this setup and we quickly got it up and running.

Quick summary of the setup:
  • 1 kubernetes bay
  • 1 master node, 16 cores (not really needed but why not)
  • 200 minions, 4 cores each
In total there are 800 cores, which matches the cluster used in the original test. How did our test go?

We ended up trying a bit more and doubled the number to 2 million requests per second :)

We learned a few things on the way:
  • set Heat's max_resources_per_stack to something big. Magnum stacks create a lot of these, and with bays of hundreds of nodes the value gets high enough that unlimited (-1) is tempting and we have it like that now. It leaves the option for people to deploy a stack with so many resources that Heat could break, so we'll investigate what the best value is
  • while creating and deleting many large bays, Heat shows errors like 'TimeoutError: QueuePool limit of size ... overflow ... reached' which we've seen in the past for other OpenStack services. We'll contribute the patch to fix it upstream if not there yet
  • latency values get high even before the 1 million barrier, we'll check further the demo code and our setup (using local disk, in this case SSDs instead of the default volume attachment in Magnum should help)
  • Heat timeout and retrial configuration values need to be tuned to manage very large stacks. We're still not sure what are the best values, but will update the post once we have them
  • Magnum shows 'Too many files opened' errors, we also have a fix to contribute for this one
  • Nova, Cinder (bay nodes use a volume), Keystone and all other OpenStack services scaled beautifully, our cloud usually has a rate of ~150 VMs created and deleted per hour, here's the plot for the test period, we eventually tried bays up to 1000 nodes

And what's next? 
  • Larger bays: at the end of these tests we deployed a few bigger bays with 300, 500 and 1000 nodes. And in just a couple weeks there will be a new batch of physical nodes arriving, so we plan to upgrade Heat to Mitaka and build on the recent upstream work (by Spyros together with Ton and Winnie from IBM) adding Magnum scenarios to Rally to run additional scale tests and see where it breaks
  • Bay lifecycle: we stopped at launching a large number of requests in a bay, next we would like to perform bay operations (update of number of nodes, node replacement) and see which issues (if any) we find in Magnum
  • New features: lots of upstream work going on, so we'll do regular Magnum upgrades (cinder support, improved bay monitoring, support for some additional internal systems at CERN)
And there's also Swarm and Mesos, we plan on testing those soon as well. And kubernetes updated their test, so stay tuned...


  • Bertrand Noel, Mathieu Velten and Spyros Trigazis from CERN IT, for the work upstream and integrating Magnum at CERN, and on getting these demos running
  • Rackspace for their support within the CERN Openlab on running containers at scale
  • Indigo Datacloud building a platform as a service for e-science in Europe
  • Kubernetes for an awesome tool and the nice demo
  • All in the CERN OpenStack Cloud team, for a great service (especially Davide Michelino and Belmiro Moreira for all the work integrating Neutron at CERN)
  • The upstream Magnum team, for building what is now looking like a great service, and we look forward for what's coming next (bay drivers, bare metal support, and much more)
  • Tim, Arne and Jan for letting us use the new hardware for a few days


  1. Replies
    1. Great Article Cloud Computing Projects

      Networking Projects

      Final Year Projects for CSE

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

  2. Thanks for sharing the valuable content , its such a very informational article.
    click here to know more about Kubernetes.

  3. Your blog is in a convincing manner, thanks for sharing such an information with lots of your effort and time Kubernetes online training
    Kubernetes online course
    Kubernetes training

  4. I needed to peruse this multiple times since I needed to make certain on a portion of your focuses. I concur with nearly everything here, and I am intrigued with how well you composed this article.

    SEO services in kolkata
    Best SEO services in kolkata
    SEO company in kolkata
    Best SEO company in kolkata
    Top SEO company in kolkata
    Top SEO services in kolkata
    SEO services in India
    SEO copmany in India

  5. Your blog is in a convincing manner, thanks for sharing such an information with lots of your effort and time
    Kubernetes online training
    Kubernetes online course

  6. روش کاشت مو SUT در مقایسه با تکنیک های دیگر از قبیل FIT و FUT روش جدیدتر و موفق تری است که تمامی مراحل آن، از برداشت تا کاشت فولیکول‌ های مو، توسط دستگاه های اتوماتیک صورت می‌گیرد. از آنجایی که در کاشت مو SUT تمامی مراحل توسط دستگاه انجام می گیرد، میزان تخریب ریشه مو، درد و خونریزی کم است.


  7. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.I want to share about Mulesoft training .

  8. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    best mulesoft online training

  9. https://tourism-blog.jimdosite.com/

  10. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website blockchain online training

  11. toptan iç giyim tercih etmenizin sebebi kaliteyi ucuza satın alabilmektir. Ürünler yine orjinaldir ve size sorun yaşatmaz. Yine de bilinen tekstil markalarını tercih etmelisiniz.

    Digitürk başvuru güncel adresine hoşgeldiniz. Hemen başvuru yaparsanız anında kurulum yapmaktayız.

    tutku iç giyim Türkiye'nin önde gelen iç giyim markalarından birisi olmasının yanı sıra en çok satan markalardan birisidir. Ürünleri hem çok kalitelidir hem de pamuk kullanımı daha fazladır.

    nbb sütyen hem kaliteli hem de uygun fiyatlı sütyenler üretmektedir. Sütyene ek olarak sütyen takımı ve jartiyer gibi ürünleri de mevcuttur. Özellikle Avrupa ve Orta Doğu'da çokça tercih edilmektedir.

    yeni inci sütyen kaliteyi ucuz olarak sizlere ulaştırmaktadır. Çok çeşitli sütyen varyantları mevcuttur. iç giyime damga vuran markalardan biridir ve genellikle Avrupa'da ismi sıklıkla duyulur.

    iç giyim ürünlerine her zaman dikkat etmemiz gerekmektedir. Üretimde kullanılan malzemelerin kullanım oranları, kumaşın esnekliği, çekmezlik testi gibi birçok unsuru aynı anda değerlendirerek seçim yapmalıyız.

    iç giyim bayanların erkeklere göre daha dikkatli oldukları bir alandır. Erkeklere göre daha özenli ve daha seçici davranırlar. Biliyorlar ki iç giyimde kullandıkları şeyler kafalarındaki ve ruhlarındaki özellikleri dışa vururlar.


  12. برخي عيب‌هاي جاروبرقي، تنها به کمي دانش و تجربه نياز دارد تا تعمير جاروبرقي در منزل انجام گيرد
    و نيازي به مراجعه به نمايندگي نباشد. اما خرابي جاروبرقي، هميشه به دست خود شما قابل حل نيست و در بسياري موارد، به وجود يک نيروي متخصص نياز است
    . بنابراين در صورت در دست داشتن ضمانت نامه، به نمايندگي تعمير جاروبرقي مراجعه کنيد و در غير اين صورت از دانش يک تعميرکار بهره ببريد.
    البته براي مراجعه به نمايندگي، نياز قطعي به ضمانت نامه نيست و خدمات نمايندگي‌ها شامل حال هرگونه مراجعه‌کننده‌اي مي‌شوند
    تعميرکار جاروبرقي
    تعمير جاروبرقي در منزل

  13. 카지노사이트 value the article post.Thanks Again. Really Great.

  14. 온라인카지노 Just wanna input on few general things, The website style is
    perfect, the articles is very fantastic :D.

  15. After going over a few of the blog posts on your web page,
    I honestly like your way of writing a blog. 토토 I book-marked it to my bookmark webpage list and will be checking back soon. Take a look at my
    web site too and let me know how you feel.

  16. 토토 I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…

  17. I have to convey my respect for your kindness for all those that require guidance on this one field. Your special commitment to passing the solution up and down has been incredibly functional and has continually empowered most people just like me to achieve their dreams. Your amazing insightful information entails much to me and especially to my peers.
    đăng ký bay từ mỹ về việt nam

    vé máy bay từ úc về việt nam bao nhiêu
    Lịch bay từ Hàn Quốc về Việt Nam hôm nay

    Đặt vé máy bay Bamboo tu Nhat Ban ve Viet Nam

    săn vé may bay giá rẻ tu Dai Loan ve Viet Nam

    mua vé máy bay từ canada về việt nam

  18. Thanks for sharing this marvelous post. I m very pleased to read this article.

  19. I will recommend your website to everyone. You have a very good gloss. Write more high-quality articles. I support you.

  20. Thanks for such a valuable post. I am waiting for your next post, I have enjoyed a lot reading this post keep it up.

  21. Your blog is really nice. Its sound really good

  22. Great job for publishing such a beneficial web site

  23. Thank you so much for sharing this Informative article with us. I really love the way of your presentation. Please keep sharing more


  24. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own Blog Engine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it 청마담

  25. I envy your piece of work, appreciate it for all the good content.

  26. There is perceptibly a lot to realize about this. I believe you made various nice points in features also.

  27. I simply discovered this web journal and have high trusts in it to proceed. Keep up the considerable work, its elusive great ones. I have added to my top choices. Much obliged to You.

  28. While looking for articles on these topics, I came across this article on the site here. As I read your article, I felt like an expert in this field. I have several articles on these topics posted on my site. Could you please visit my homepage? 토토사이트모음

  29. What a nice post! I'm so happy to read this. baccarat What you wrote was very helpful to me. Thank you. Actually, I run a site similar to you. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.

  30. What a post I've been looking for! I'm very happy to finally read this post. 토토사이트 Thank you very much. Can I refer to your post on my website? Your post touched me a lot and helped me a lot. If you have any questions, please visit my site and read what kind of posts I am posting. I am sure it will be interesting.

  31. Looking at this article, I miss the time when I didn't wear a mask. 오공슬롯 Hopefully this corona will end soon. My blog is a blog that mainly posts pictures of daily life before Corona and landscapes at that time. If you want to remember that time again, please visit us.

  32. .You showed me what true friendship looks like
    .I have never thought that I will ever be able to find a friend in my adult life. But you truly saved me
    .I cannot thank you enough for everything you do for me, mate. دانلود آهنگ سینا درخشنده آدم برفی

  33. I like the helpful info you provide in your articles. I’ll bookmark your blog and check again here frequently. I’m quite sure I’ll learn plenty of new stuff right here! Good luck for the next. 스포츠토토

  34. Pretty useful article. I merely stumbled upon your internet site and wanted to say that I’ve very favored learning your weblog posts. Any signifies I’ll be subscribing with your feed and I hope you publish once additional soon. 메이저사이트

  35. That's a great article! The neatly organized content is good to see. Can I quote a blog and write it on my blog? My blog has a variety of communities including these articles. Would you like to visit me later? 메이저안전놀이터

  36. As the Internet develops further in the future, I think we need to collect materials that people might be interested in. Among the data to be collected, your 메가슬롯 will also be included.

  37. I finally found what I was looking for! I'm so happy. 사설토토사이트 Your article is what I've been looking for for a long time. I'm happy to find you like this. Could you visit my website if you have time? I'm sure you'll find a post of interest that you'll find interesting.

  38. Pretty! This has been an extremely wonderful post. Thank you for providing this



  39. İnstagram takipçi satın al! İnstagram takipçi sitesi ile takipçi satın al sende sosyal medyada fenomen olmaya bir adım at. Sende hemen instagram takipçi satın almak istiyorsan tıkla:

    1- takipçi satın al

    2- takipçi satın al

    3- takipçi satın al

  40. Very goood

  41. It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. 먹튀검증사이트 But by chance looking at your post solved my problem! I will leave my blog, so when would you like to visit it?!


  42. I am very happy to read this article. Thanks for giving us Amazing info. Fantastic post.
    Thanks For Sharing such an informative article, Im taking your feed also, Thanks.key ccleaner 5.21