Sydney Superuser Awards Nominee: Tencent TStack Team

Voting is now closed. Stay tuned to find out who wins!

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

The Tencent TStack Team is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination.

The Tencent TStack team is comprised of 76 members who developed the OpenStack-based TStack private cloud platform. It consists of four sub-teams:

Product design: Responsible for requirement analysis and interaction design
Technical architecture: Responsible for solution design and technology research
Product development: Responsible for feature design and implementation
Operations support: Responsible for deployment, monitoring and troubleshooting.

We built private cloud to provide services for internal IT environments and testing environments (e.g. QQ, WeChat).
We also provide complete hybrid cloud services for government departments and enterprises in China.

How has Openstack transformed the organization’s business?

The OpenStack-based private cloud platform cuts server costs by 30% and O&M costs by 55%, and saves RMB100 million+ each year for Tencent. It shortens resource delivery from two weeks to 0.5 hours and supports the development teams (such as QQ, WeChat, and Game) of the services that generate tens of billions of revenues for Tencent. It optimizes global resource scheduling. For example, the deployment duration of the global mail system has been cut from 10 days to 1 day. It is an important cloud computing platform for Tencent to achieve its “Internet plus” strategy. It is deployed in multiple provinces in China, including Sichuan government cloud, Guangdong government cloud, Xiamen government cloud, and Yunnan government cloud. It serves more than 100 million users.

How has the organization participated in or contributed to the OpenStack community?

Tencent actively participates in community activities, such as OSCAR, OpenStack Days China. As a participant and sponsor, Tencent shared experience in using OpenStack. Currently, the TStack team is preparing to participate in OpenStack Sydney Submit. We compiled the OpenStack Usage White Paper which we shared with our customers. We created a WeChat official account to share our experience with OpenStack users, and submitted bugs and blueprints to the community. We cooperated with Intel to apply the latest features (such as RDT, DPDK, SPDK, FPGA, and Clear Container) to accelerate the production. Tencent plans to join the OpenStack Foundation. Tencent is one of the members of the CNCF, Linux Foundation, and MariaDB Foundation.

What open source technologies does the organization use in its IT environment?

Technology stacks used in the TStack team are built with open source tools, including OpenStack, KVM, Centos, Ironic, HAProxy, keepalived, Docker, Clear Container, Kubernetes, Rabbitmq, Mairadb, Nginx, Ansible, Jenkins, Git, ELK, Zabbix, Grafana, Influxdb, Tempest, and Rally. These tools are used in development, testing, and CI/CD.

What is the scale of the OpenStack deployment?

TStack has 6,000+ nodes, including 2,000 ironic nodes. It manages more than 12,000 VMs, provides 80,000+ cores, 360+ TB mem, and 20+ PB disks. It covers 14 clusters in 7 data centers in 4 regions. 1,000+ nodes managed in a single region. It carries 300+ online services, including OA, WeChat gateway, email, and ERP. For example, the email system on the TStack provides 24/7 services for 40,000 employees. It provides CI/CD environment for 20,000 developers and testing environment for large applications (such as QQ and WeChat), and supports concurrent access of more than 100 million users. Tencent promoted TStack to Chinese government and enterprise. It signed cooperation agreements with 15 provinces and 50 cities, for example, TStack-based Xiamen government cloud in the BRICS Summit.

What kind of operational challenges have you overcome during your experience with OpenStack?

Many different types of VMs, such as XEN and KVM, are deployed in Tencent. It is a huge challenge to manage VMs seamlessly on the OpenStack platform. The TStack team developed a set of tools to manage VMs on existing heterogeneous virtualization platforms without interrupting services. Earlier versions of OpenStack had bottlenecks, including message queues and keystones in large-scale deployment. After large-scale optimization, a single region can now support more than 1,000 computing nodes. In large-scale deployment, the VxLAN performance is a bottleneck. Tencent developed an OpenStack-based technology that is compatible with SDN hardware and software of multiple vendors to accelerate the VxLAN. The adaptive compression technology is used to cut the VM migration duration by 50 percent.

How is this team innovating with OpenStack?

Strict real-time rate limit to control CPU usage time and ensure VM QoS. VM scheduling policies are customized based on service tag, with VMs dispatched to different hosts to ensure high availability. The online resize function resizes VMs without interruption. The adaptive compression technology is used to cut VM migration duration by 50%. Cooperated with Intel to provide Clear Container on TStack for better security. The Neutron-based SDN controller supports heterogeneous networks and performs unified management on legacy networks, SDNs and NFVs. Integration of Tencent cloud security technologies with OpenStack to provide complete cloud security services. Implemented unified management of OpenStack-based private cloud, public cloud, and VMware.

How many Certified OpenStack Administrators (COAs) are on your team?

One team member is COA-certified, 28 have received COA training and nine will take the COA examination in October.

Cover image: Shenzen’s Tencent building, courtesy Tencent.