Tag Archives: Starcraft

AI is beating almost all of mankind at Starcraft

A new algorithm called AlphaStar is beating all but the very best human players at Starcraft. This is not only a remarkable achievement in itself, but it could teach AIs how to solve complex problems in other applications.

A typical Protoss-Zerg combat. Credits: DeepMind.

The foray of AIs in strategy games is not exactly a new thing. Google’s ‘Alpha’ class of AIs, in particular, has taken the world by storm with their prowess. They’re revolutionizing chess and Go — once thought to be insurmountable for an algorithm. Researchers have also set their eyes on other games (DOTA and Poker for instance), with promising but limited results. The sheer complexity of the game, mixed with the fact that you don’t have all the information available to you (as opposed to Go and chess, where you see the entire board freely), raised serious challenges for AIs.

But fret not — our algorithm friends are slowly overcoming them. A new Alpha AI, aptly called AlphaStar, has now reached a remarkable level of prowess, ranking in the top 98.5% of all Starcraft II players.

Starcraft is one of the most popular computer strategy games of all time. Its sequel, Starcraft II, features a very similar scenario. The players choose one of three races: the technologically advanced humans, the Protoss (masters of psionic energy), or the Zerg (quickly-evolving biological monsters). They then mine resources, build structures, an army, and try to destroy the opponent(s).

There are multiple viable strategies in Starcraft, and there’s no simple way to overcome your opponent. The infamous ‘fog of war’ also hides your opponent’s movements, so you also have to be prepared for whatever they are doing.

AlphaStar managed to reach Grandmaster Tier — a category reserved for only the best Starcraft players.

Credits: Deep Mind.

Having an AI that is this good at such a complex game would have been unimaginable a decade ago. The progress is so remarkable that one of the researchers at DeepMind, the company training and running these AIs called it a ‘defining moment’ in his career.

“This is a dream come true,” said Oriol Vinyals, lead, AlphaStar project, DeepMind. “I was a pretty serious StarCraft player 20 years ago, and I’ve long been fascinated by the complexity of the game. AlphaStar achieved Grandmaster level solely with a neural network and general-purpose learning algorithms – which was unimaginable 10 years ago when I was researching StarCraft AI using rules-based systems.

AlphaStar advances our understanding of AI in several key ways: multi-agent training in a competitive league can lead to great performance in highly complex environments, and imitation learning alone can achieve better results than we’d previously supposed.

I’m excited to begin exploring ways we can apply these techniques to real-world challenges, such as helping improve the robustness of AI systems. I’m incredibly proud of the team for all their hard work to get us to this point. This has been the defining moment of my career so far.”

The AI didn’t play with ‘AI cheats’ — it had to face the same constraints as human players:

  • it could only see the map through a camera as a human would;
  • it had to play through a server, not directly;
  • it had an built-in reaction time;
  • it had to select a race and play with it.

Even with all these, the AI did remarkably well.

Every single combat has multiple aspects of strategy involved. Credits: DeepMind.

At every given moment, a Starcraft player (or algorithm) has to choose from up to 10^26 possible actions, all of which have potentially significant consequences. Therefore, researchers took a different approach than with Go or chess. In these ancient games, the AIs learned by playing millions and millions of games, practicing and learning alone. In the Starcraft algorithm, however, some initial information had to be input into the framework.

This is called imitation learning — the AI was basically taught how to play the game. By doing this and combining it with neural network architectures, the AI was already better than most players. With more supervised learning, it was able to surpass all but the very best players in the world. This enabled it to learn from existing strategies, but also develop its own ideas.

“StarCraft has been a grand challenge for AI researchers for over 15 years, so it’s hugely exciting to see this work recognised in Nature. These impressive results mark an important step forward in our mission to create intelligent systems that will accelerate scientific discovery,” said Demis Hassabis, co-founder and CEO, DeepMind.

Professional Starcraft players were also impressed and thrilled to see the AI play out its game. As is the case with previous iterations of Alpha AIs, the algorithm came up with new and innovative tactics.

“AlphaStar is an intriguing and unorthodox player – one with the reflexes and speed of the best pros but strategies and a style that are entirely its own,” said Diego “Kelazhur” Schwimer, professional StarCraft II player for Panda Global. “The way AlphaStar was trained, with agents competing against each other in a league, has resulted in gameplay that’s unimaginably unusual; it really makes you question how much of StarCraft’s diverse possibilities pro players have really explored. Though some of AlphaStar’s strategies may at first seem strange, I can’t help but wonder if combining all the different play styles it demonstrated could actually be the best way to play the game.”

It’s an impressive milestone. It’s also one that could get us to think whether teaching AIs how to beat us in strategy war games is a good idea or not. But for now, at least, there’s no need to worry. AIs are very limited in their scope. They can get very good, but strictly at the task they are trained to do — they have no way of applying what they’ve learned in the computer game setting to a real-life war scenario, for instance.

Instead, this application could help researchers learn how to design better AIs for dealing with simple real-world scenarios, like maneuvering a robotic arm or operating efficient heating for smart homes.

The research was published in Nature.

Now computers are also beating us at Starcraft

When the recent World Chess Championship match took place, it didn’t decide the best player on Earth — only the best human player on Earth. For years, computers have been beating us at chess and the recent Artificial Intelligence (AI) developments have only solidified their dominance. Go, the ancient strategy game which is immensely more complex than chess, was also mastered by an AI.

Now, AIs have their eyes set on our favorite strategy games — and they’re doing an excellent job.

Image via Deep Mind.

Google’s Deep Mind has the ambition to solve some of the world’s most challenging problems, but along the way, researchers are training it with board- and computer games alike.

In addition to obvious differences in gameplay, there’s another fundamental difference between games like chess and Starcraft II: vision. In chess, you have full information on what’s happening on the board, whereas in Starcraft, you only see you units and a small area around them — the rest is hidden by the “fog of war”. This type of uncertainty has been difficult to deal with for AIs, which have continuously struggled to grapple with the fog of war. StarCraft has therefore emerged as a “grand challenge” for AI research, being one of the most difficult games to master.

After months of training, DeepMind released AlphaStar — the cousin of AlphaZero and AlphaGo, which played chess and Go respectively. AlphaStar was trained directly from raw game data by supervised learning and reinforcement learning. In other words, it learned from the best humans. In contrast, the recent version of AlphaZero went through an unsupervised learning process, playing countless games against itself. It got better after each iteration and developed its own unique style, which led to spectacular games. However, this was not possible in Starcraft.

Even so, AlphaStar’s success was remarkable. After a training game, it faced two professional players and defeated them convincingly.

“In a series of test matches held on 19 December, AlphaStar decisively beat Team Liquid’s Grzegorz “MaNa” Komincz, one of the world’s strongest professional StarCraft players, 5-0, following a successful benchmark match against his team-mate Dario “TLO” Wünsch. The matches took place under professional match conditions on a competitive ladder map and without any game restrictions,” DeepMind writes.

However, this happened when AlphaStar was given complete reign over what it was allowed to do. It shined in the “micro” aspects, controlling its units with stunning accuracy and precision, making correct decisions in a split second — which is ultimately what you’d expect from an AI. In the grand scheme of things, MaNa did great strategically, but he just couldn’t overpower his opponent.

Things change substantially, however, when AlphaStar was made a bit more “human”.

A visualization of the AlphaStar agent during game two of the match against MaNa. This shows the game from the agent’s point of view: the raw observation input to the neural network, the neural network’s internal activations, some of the considered actions the agent can take such as where to click and what to build, and the predicted outcome. Image credits: Deep Mind.

In an additional game streamed on Twitch, AlphaStar was hobbled in some ways (like only being allowed to “see” by moving the focus of the in-game camera and not being allowed to make more clicks than a human would), which most commentators agreed was “fair.” Although it still did very well, MaNa ultimately managed to defeat the AI, scoring mankind’s only win so far.

However, to be fair, some of this result might be owed to the element of surprise. AlphaStar was very familiar with the style of human play, whereas humans weren’t really sure what to expect. This type of issue seems very similar to what happened in Dota 2, a game which shares many similarities with Starcraft. When humans first played against the algorithm, they were defeated handily and were surprised by the strategies employed by the AI. When they returned knowing what to expect, they did a much better job and were able to beat the AI.

Another aspect worth mentioning is that although AlphaStar faced professional opponents, they weren’t the best of the best — so given a fair playground, mankind still probably keeps the crown — but only barely.

Starcraft is a type of rock-paper-scissors game, where there’s no ideal strategy: everything is good against something and weak against something else. The Deep Mind researchers created a league where AIs duked it out between one another, akin to human matchmaking play. New competitors were dynamically added to the league, by branching from existing competitors.

Estimate of the Match Making Rating (MMR) — an approximate measure of a player’s skill — for competitors in the AlphaStar league, throughout training, in comparison to Blizzard’s online leagues. Image credits: Deep Mind. All the competitors developed new strategies and learned from one another, taking advantage of Starcrafts huge strategic potential. For instance, the first iteration attempted “cheesy” and very risky strategies, such as a quick rush with Photon Cannons or Dark Templars. These strategies were discarded as the AI progressed, leading to employ other, more complex strategies which focused on economic domination.

This was clearly visible in the type of units they developed.

As training progressed, AlphaStar built different units and chose varying tech trees. Image credits: Deep Mind.

Of course, beating humans at Starcraft can be a goal in and of itself, but Deep Mind has something much loftier in mind. They want to use Starcraft as a stepping stone towards addressing real-life complex issues such as climate change and language understanding.

“While StarCraft is just a game, albeit a complex one, we think that the techniques behind AlphaStar could be useful in solving other problems. For example, its neural network architecture is capable of modelling very long sequences of likely actions – with games often lasting up to an hour with tens of thousands of moves – based on imperfect information. Each frame of StarCraft is used as one step of input, with the neural network predicting the expected sequence of actions for the rest of the game after every frame. The fundamental problem of making complex predictions over very long sequences of data appears in many real world challenges, such as weather prediction, climate modelling, language understanding and more. We’re very excited about the potential to make significant advances in these domains using learnings and developments from the AlphaStar project.”

The Deep Mind team also said that Starcraft is a game which emphasizes many of the problems AIs have traditionally struggled with, and overcoming them could also pave the way for overcoming concrete issues in AI design.

However, a key problem is still unsolved: when an AI is pushed outside of its “comfort zone”, it collapses. This makes these algorithms surprisingly brittle — a kind of a “glass cannon” for solving specific issues. It’s important to develop robust algorithms capable of adapting to different types of situations, and it’s exactly here that playing Starcraft can make a substantial difference.

“Achieving the highest levels of StarCraft play represents a major breakthrough in one of the most complex video games ever created. We believe that these advances, alongside other recent progress in projects such as AlphaZero and AlphaFold, represent a step forward in our mission to create intelligent systems that will one day help us unlock novel solutions to some of the world’s most important and fundamental scientific problems.”

A full technical description of this work is being prepared for publication in a peer-reviewed journal, Deep Mind concludes.

Massive analysis of gamers’ habits reveals how to best reach excellence in any skill

Scientists are learning tips and tricks for reaching excellence in what most people would call an unlikely source — gamers. A team at Brown University has analyzed countless hours of competitive play to see which practices work best when trying to improve a skill.

Image credits Gerd Altmann.

It’s easy to dismiss gamers as slack-offs, but what most people who don’t partake in video gaming fail to understand is how competitive and skill-centric some game communities can be. Sure, there’s a lot of stuff out there that’s designed purely as a time-and-money sink — a title about crushing candies comes to mind. But in the kind of games that attract competitive players, the ones you’ll see in eSport competitions, you live or die by your skill. Through their very nature, by pitting player against player, these games rear their community with a single overarching goal — to git gud.

Mastering a game isn’t much different from mastering anything else. Mostly, it comes down to practice. So when you think about it, studying gamers, a demographic in which individuals continually order themselves after levels of excellence/skill at a common task, can yield some valuable insight into which practice patterns work best.

Which is exactly what a team led by a Brown University computer scientist has done. They’ve analyzed data gathered from thousands of online matches of Halo: Reach and StarCraft 2.

“The great thing about game data is that it’s naturalistic, there’s a ton of it, and it’s really well measured,” said Jeff Huang, a computer science professor at Brown and lead author of the study.

“It gives us the opportunity to measure patterns for a long period of time over a lot of people in a way that you can’t really do in a lab.”

The results obtained from the first showed how different patterns of play affected skill improvement rates, while the latter showed how highly successful players’ unique and consistent “rituals” play a key part in their success.

There were several reasons for choosing these games: they’re both hugely popular. They have built-in ranking systems which sort players according to their success, providing the team with a solid estimation of individual skill levels. They’re also both highly competitive games, but play very differently from one another.

From zero to hero

Halo: Reach is a first-person shooter, a genre of games in which players take on the role of a single character and have to use a plethora of weapons to battle others. FPS games generally rely on motor skills (e.g. hand-eye coordination), reaction and decision speed, as well as individual tactical choices. One of the most popular game modes is Team Slayer, where players are placed on opposing teams which compete to get the most kills by the end of the 10-15 minute game time.

To make sure the game is fair for everyone, the game uses a metric system called TrueSkill — whose ratings are constantly updated with a player’s performance following each match — to put the together teams of roughly equal ability. TrueSkill gave Huang and his colleagues the means to see how playing habits influence gamers’ skill acquisition. They looked at data mined from all the online Halo matches played since the game’s release — totaling a staggering seven months of continuous game-play.

People who played the most matches every week (more than 64) had the largest overall increase in skill over time. But simply playing a lot isn’t the most efficient way to improve your skill, the team reports. Over the first 200 matches played, people who played 4-8 matches a week showed the most improvement on a per-match basis. They were followed by those who played 8-16 matches every week.

“What this suggests is that if you want to improve the most efficiently, it’s not about playing the most matches per week,” Huang said.

“You actually want to space out your activity a little bit and not play so intensively.”

The team also looked at the effect breaks had on a player’s skill. Players who took short breaks — i.e. one or two days — showed some decrease in skill in the first match following the downtime, but no decrease in their second one. Longer term breaks, however, had more pronounced effects on their efficiency. The effects of a 30-day break, for example, lasted for around 10 matches. So the lesson here is moderation — don’t overdo it on either the practice or the rest.

Tap it like it’s (a) hot (key)

The second study focused on the real-time strategy game Starcraft 2. Like other RTS games, it puts players in control of an entire army. They have to secure resources and manage an economy, build up bases and infrastructure, train their forces, and direct them in battle — often taking place on multiple fronts, with hundreds of units at a time. It’s an entirely different game from Halo, promoting management skills, large-scale tactical choices, sustained attention, and strategical thinking.

By comparing the in-game habits of elite players to those of average or lower skill, the team found one major difference: heavy use of hotkeys. These are customizable keyboard shortcuts which enable complex commands to be issued much more quickly. Less skilled players usually gave orders using the mouse. Elite players universally prefer hotkeys, the team found.

Image credits: Josef Glatz.

This gives them a huge advantage. For instance, the task of finding a free worker and selecting it with the mouse, clicking on the build icon, selecting the right building, and finally placing it in the desired location — which can take 4-5 seconds — can be done in under a second with hotkeys. Just press F1, B, B, click, and you have a barracks under construction. Skilled players can issue up to 200 different orders a minute in typical matches with hotkeys, dwarfing anything possible through the mouse alone.

But that’s pretty common knowledge in the StarCraft player-base. Hotkeying is the one skill all my friends say has improved their game the most — and we’re nowhere near an ‘elite’ level of play. What top players do differently is that they form unique hotkeying habits and stick with them. These habits are so distinctive and consistent, in fact, that the team was able to identify specific players with over 90% accuracy just by looking at how they hotkey. These habits are almost like a second nature, the researchers say, enabling players to issue commands efficiently when pressure is on.

They also seem to “warm up” this skill in every match. The study shows that even in the very early stages of a game when there’s almost nothing going on, these players will rapidly scroll through their hotkey habits issuing dummy commands to whatever units they have available.

“They’re getting their minds and bodies into the routines that they’ll need when they’re at peak performance later in the game,” Huang said.

“They’re getting themselves warmed up.”

Mastering mastery

Huang hopes their research will help people can improve their performance in other areas of life. Professions which require specialists to pay attention to lots of different elements at once could benefit from a “warming up” similar to that in StarCraft.

“Air traffic controllers come to mind,” he said.

“Maybe when someone first gets in the seat, they should take a few moments and re-enact what they do until they can get warmed up and in the zone.”

The results of the Halo study confirm previous cognitive research, he adds, which suggests that moderate activity coupled with short breaks can improve learning efficiency.

“People have seen this for other things, like studying.”

“Cramming is generally regarded as less efficient than doing smaller bits of studying throughout the semester. I think we’re seeing something similar here in our study.”

Taken together, the results show that the best way to become good at something is to “practice consistently, stay warm,” Huang concludes.

The full paper “Master Maker: Understanding Gaming Skill Through Practice and Habit From Gameplay Behavior” has been published in the journal Topics in Cognitive Science.