 |
||||
|
Go Bet Results | |
||
|
||||
|
||||
|
Go Bet Results | |
||
|
||||
Move listings for most of the matches (sgf, pdf) are also available.
|
|
Will (black) lost to Scott's program (white), 154-212.5
(We had the program play itself at the end, and black wound up with an extra unexpected corner that Will would not have attempted. The final game score in self-play was 203.5 to 163, but Scott estimated the extra corner as worth 9 points.)
Much to Don and Will's shock, Scott reported that he had been working on his program (even though it was already winning easily), and had "programmed a lot" since the previous game. It was "very tricky" to put the different modules together. The program was now predicting the value of game states, rather than just the best move from each state.
The program is doing Monte Carlo tree search. It adds one additional node at every search (while keeping previous searches in memory, in a cache). At each leaf node, it calls the neural net to get the value of that position.
It operates like AlphaGo, with all Monte Carlo tree search, and no alpha-beta search. The program is considerably better than the previous one from six months ago. Scott has played it himself, but he waited until the end of the game with Will to report on his own understanding of his program.
Will played defensively, trying not to lose groups. The game was less "bloody", and with no important tactical fights, and only a single "ko". Despite this, it was a convincing victory for white.
Once we finished, Scott reported: this program is much stronger than Scott. (Who, himself, is significantly stronger as a human player than Will is.) Scott played against his own program. He beat it once with a 6 stone handicap, and never with only a 5 stone handicap. But that was a version of the program that was only given 1 second to make moves! In this match, the program got 30 seconds to make moves.
Scott also had the 30-second version of the program play against the 1-second version. The 30-second version beat the 1-second version despite a 9-stone handicap. (And, of course, the 1-second version beat Scott with a 5-stone handicap, and Scott easily beats Will. This match was Will playing against the 30-second version with no handicap.)
The program doesn't think at all on the other person's time. It would be a simple 2-line change to have it search while waiting for its opponent's move. Presumably, that would make it much stronger still.
|
|
Will (black) lost to Scott's program (white), 160-206.5
Scott hasn't changed his program since 2022, but has visions and grand plans for what he wants to implement in the future.
Will, first photo: "I think I'm playing kind of even."
Will, second photo: "I've lost all hope."
Will conceded early. Black had different disconnected groups, while white was all connected.
(Scott let the program play itself to finish the game, in order to get an approximate "score". But the game was stopped so early, that the "playout" by the program wasn't particularly realistic or meaningful.)
|
|
Will (black) lost to Scott's program (white), 142-224.5
Scott fielded the same program as the previous match. He experimented the day before with a small change, a few lines of code. They didn't work, he backed them out and returned to the previous program.
Scott was trying to make the program "more cautious" about how it evaluates whether groups are alive or dead. Scott thinks he really should just scrap that whole module, and rewrite it as Monte Carlo search.
Halfway through the game, Will said: "I think the game is pretty even." There was an early fight in the corner; Will thought that his stones still lived after that fight.
Near the end, there was a tense fight for Will's "dragon" in the center. But then the program started moving in the corner instead, which gave Will two "free" moves in the center. He needed both those moves to make two eyes and live, and thought the program made a mistake to give them to him. But he was happy to take the gift.
But it turned out, apparently, that the program allowed Will to save his his center — but only at the cost of losing that entire other corner. Will resigned. (We let the program play out both side for the final score.)
Note: Narinder "paid ahead" by mailing $40, which covered the previous loss, this loss, and potentially two more upcoming.
|
|
Will (black) lost to Scott's program (white), 18.5-63.
(Will's score was 13, plus the 5.5 komi = 18.5.)
Scott submitted a legal program for the bet! He replaced his (illegal) neural net module from the previous version, with a new neural net that he trained himself, based on 100,000 professional games. He also fixed various bugs in the search algorithm — including fixing the last bug an hour before the match!
That last bug was around multithreading. When checking some groups, the algorithm doesn't need to recompute a stone if it was already analyzed in another computation. It was supposed to copy the result of the previous computation, but instead it just concluded that it "knew nothing". Scott isn't sure how that (bugged) version of the program ever worked.
Scott played a couple games against the new algorithm himself, to test whether it worked. He wouldn't tell us how those games went.
The old program, in the neural net code, used the Cafe framework. That framework is no longer supported on the newest MacBooks. So the old program used older hardware as well. As long as Scott was replacing the neural net code, he also upgraded to a more modern framework, and thus was able to use a much newer (and faster) MacBook. The framework does deep learning, computes derivatives, and implements gradient descent.
The game started with random choice of color, and Will drew white.
In the previous 2 games, Will was aggressive, and got into tactical skirmishes early — which he lost every time. This time Will played much more conservatively. He made a strong effort to not attack early, but just defend his stones. There was one minor skirmish early, but Will escaped and survived.
The computer settings were not terrible different from the previous versions, but the program was now running on much faster hardware, so it was playing the game much faster.
This was the most competitive game we have experienced in this bet. In the early years, Will was much better than the program, so beat it easily without trying. In the last year, the program was "illegal", but also clearly much better than Will. But this program was competitive with Will, and the outcome was in doubt until the very end.
In the mid-game, Will reported feeling that he was "not doing too badly." Doing clearly better than the previous two games. "I think I'm still in the game." About 2/3 of the way through the game, Will said: "I seem to be surviving the tactical battles in a way I haven't been." As the endgame phase began, Will said, "Oh! Now things are looking pretty good for me! At least promising."
Near the end, Will made one small mistake. He thought he had two eyes in the upper left corner, but actually only had one, and wound up losing all of his stones. That was the game. He still had a chance at the end, because the program made a few mistakes too. Scott's count, near the end, was that the program (black) was up by 5 points (including the 5.5 komi, so ahead by 10 stones).
It was a game that could have gone either way.
Scott finally reported that he had played against the program a couple of times, with similar outcomes. Scott played more aggressively than Will ... but lost both times. (Scott is better at Go than Will.) Scott thinks Will's strategy of playing conservatively is a good idea.
At the end, the program won 63-13 stones (plus komi = 63-18.5). There was no "self-play": the game was close enough, that both players continued until Will passed (and then eventually Scott chose to pass on the program's behalf).
|
|
Will (black) lost to Scott's program (white), 141-225.5.
Scott's program was unchanged (and thus technically illegal). However, he had spent a few hours starting to work on training his own model.
Scott's program played white, Will played black.
Will: "The question is whether I can keep my groups and not be humiliated, or whether I'll lose my groups and be humiliated. I'm going to lose either way, but the only question is whether I'll be humiliated or not."
Will failed to save 1/3 of his groups. He conceded.
Scott's program self-played to finish the game. Final score: white 225.5 to black 141. (This is the first time Scott's program played as white.)
|
|
Will (white) lost to Scott's program (black), 15-53.
Scott's program had no change since the last match, so was still (knowingly) illegal. Thus Scott "lost" this round of the bet, despite his program beating Will.
2 minutes into the game, on move 5, Will said: "Can I take that back? I forgot how to play Go."
In the middle, Will said: "This is my last chance. I need 2 moves in a row. I need to trick it." Alas, the program saw the danger, and did the correct blocking move.
End game: It looked like black (Scott's program) filled up all liberties in one large section with only one eye ... but it managed to escape and make a second eye.
After: Will said Scott's program "beat me soundly". (We neglected to complete the financial transaction, so Scott owes Will and Don $10 each next time.)
|
|
Will resigned! (Although Scott's program was knowingly illegal, so he lost the bet.)
Scott didn't make the program any "smarter", but he made it much faster. He got a 4x or 5x speedup via multi-threading, and 2x speedup from a faster laptop. Other than that, he copied the previous configuration: 20/20/10/10/5/5/2/2/1.
Scott acknowledged ahead of time that his program was not (yet) legal, because he copied a section from a go program he found online. Hence he "forfeited" the match ahead of time, but played his program against Will anyway.
Halfway through the game, Will said: "I can't think of a good move to make ... which probably means the program is killing me."
Later, from Will: "I have a strategy to kill all his stones on this one side ... as long as he doesn't make this one key move."
At one point, Will was in atari, but both Will and Scott's program ignored it.
Finally the program captured a big chunk of Will's stones, that he didn't realize were in danger. Scott himself saw the danger and/or opportunity for 5 moves in a row, and kept waiting for either Will or his program to see it. Will thought the stones were alive, and the program thought the stones were dead, so neither made the move.
At the end, Will resigned! Scott's program won its first game against Will (albeit with code that doesn't qualify under the terms of this bet).
|
|
Will (white) beat Scott's program (black) by 210 1/2 to 156.
It was a wet & rainy afternoon. Will presented Scott with a gift (see first photo): the book "Deep Learning and the Game of Go".
Scott's program hasn't changed (much) in last 6 months. It is still "not legal", in the sense that he uses a (very old) neural net for evaluating board positions that was trained by someone else.
Scott slightly tuned some parameters: It is now less conservative about reusing cached values. (When a move at one corner, does that invalidate conclusions at a far corner?) This allowed a wider search tree, from 20/20/5/5/1/1/1/1 to 20/20/10/10/5/5/2/2/1.
Play-by-play game notes:
There were multiple points (3+) where the program had a winning move to win the whole game. However, it didn't see the right move each time, and so Will just barely escaped each time. There's a good chance, in a future game, he wouldn't be so lucky.
|
|
Will (white) beat Scott's program (black) by 223 1/2 to 143.
Before the game began, Scott reported: "My whole approach for the last 20 years has been wrong. It's trying to 'prove' stones alive. I've been trying to use a 'neat' solution to a 'fuzzy' problem. It's fundamentally the wrong approach."
Since the last match, Scott added improvements to the "classic search" portion of his program. He restricted the number of moves at each depth (so it searches deeper, but less broad). It uses heuristics to order the search to the "more likely" moves.
Scott is "not unhappy" with the evaluation of whether a group is alive or dead. But the blending of all the different information is "not very good".
The program did very well in the first corner. Especially with the A1 move. It read out the position before Will could. Will tried hard, but lost the first corner anyway. Because of that, Will (for the first time in 20 years!) thought he was going to lose the game.
The game was "very bloody". Many prisoners on both sides (which is rare in Go).
|
|
Will (white) beat Scott's program (black) by 197.5-169 (with the 5.5 komi).
Scott hadn't changed his program since the last match. Will played a game against it anyway.
The game was very competitive, and even in the middle, it was unclear which side had the advantage. Whoever won the center, was probably going to connect to each of the corners and thus dominate the endgame. In the middle, Will succeeded with a couple of critical moves in the middle of the board, as Scott's program made a few moves elsewhere, at the critical time. In this game, Scott's program didn't give away any "bonehead" moves (as had been usual in the past -- even in the previous match, at least a couple of times). But Will was able to secure the center when he most needed to, and thus won the game.
Scott paid Will and Don $10 each.
|
|
Will beat Scott's program 195.5-171.
Scott's program was "still not legal", as admitted by Scott, because it used a neural net from the internet rather than Scott training his own. Scott combined his search with the neural net, by multiplying the probabilities. The neural net has much better positional awareness (e.g. opening moves, the opening book). The search "score" is the value of the expected point gain + 2x the threat value. Every 3 points in expected difference equals double the probability.
Scott thought his program "might" beat Will, but he hadn't played against it yet himself, so he wasn't sure.
During the game, the program made at least one move that wasn't the top choice of either system (neither search, nor neural net). It was #5 in one, and #6 in the other. But the two systems disagreed strongly about the other potential moves, so when combined it became the #1 move choice. There was also another move, that was ranked #1 by the search system, and ranked 0% by the neural net (so it was rejected). There was also a "compromise" wasted move (on a group already dead), that was not a top choice of either system (but again became a top choice when combined).
This program was a huge step up in Go competence from Scott's previous versions. By far the best entry Scott has ever offered during this bet. About halfway through the game, we agreed to allow the program to "self play" (both sides) to the end. The final score was 195.5 for white (Will) to 171 for black (Scott) with 5.5 komi.
Will won, so Scott paid Will & Don $10 each. The game was surprisingly close, though!
|
|
Scott's program was unchanged. He played a human game against Will instead (no komi). In a mild surprise, Will actually won the game, 44-26.
Dusty wasn't available for the running bet. Instead, just for fun, Joe (who used to be a sprinter in high school) ran a race against Will. They headed in opposite directions on the horseshoe circle. Joe completed the circle in 25 seconds, by which time Will was only slightly past the halfway mark.
Also, Will paid Joe $1000, as a resolution of the side bet that the world champion Go player in 2024 would be human. With AlphaGo's success in 2016, Will conceded that the world champion is likely already a computer, and certainly would be in 2024.
Will beat Scott's program 215.5-151
Dusty inaugurated the first race in the 10-year running bet against Will. We found an approximately circular path outside the back of Tressider. They ran the circle simultaneously, but in opposite directions. Will finished in 39.10s, Dusty in 56.25s (with an initial fall). Will wins the first race.
Scott, following the lead from Dusty's computer Go victory the previous time, integrated the neural net, and was halfway to implementing the Monte Carlo search technique. His Mac doesn't have a CPU that works with CAFFE, so his workaround plan isn't yet clear. The program he fielded ran just the neural network, but -- as it was just a copy of the Go-playing neural network found on the internet (a 12-layer convolutional NN) -- the submission wasn't a legal entry into the bet.
Nonetheless, Will played Scott's program as an exhibition. The program was actually dominating the early to mid-game. (Perhaps not a surprise, as it was doing pattern matching on expert-level games, and the early board were excellent matches to real games.) It was also very fast, returning the next recommended move essentially instantaneously. However, after the mid-game, the program made about 3-4 moves that were tactical errors, and wound up losing large chunks of territory and pieces that even an average player should have been able to keep alive. At the end, Will (white) won, 215.5 to 151.
We also clarified that, under the terms of the bet, it was not legal for Scott to directly copy the 1-page NN structure code from the internet (as Dusty had done last time), nor was it legal to use the CAFFE neural net library. Legitimate programming submissions by Scott must be written completely by Scott, in a programming language no more capable than the typical general purpose languages (Java, C++, Lisp, etc.). Libraries are allowed for non-core work like graphics and file I/O, but the not for the reasoning algorithms of the program.
|
|
Will beat Scott's program 262-99. Dusty's program won, but was ruled invalid. Don and Will bought out the bet with Dusty.
Scott: No change to program. Scott conceded, then played vs. Will with a 9 stone handicap. Will (white) won, 262-99.
Dusty: Was 1 hour late, "working on the program". He used a "deep neural network" to suggest moves to a "Monte Carlo player". The network was trained on master games. Using the GPU, the NN gets 50 games /sec instead of 8000 /sec. (50milli to ask GPU NN.) Just the NN gives 44% match to grandmaster moves (but Dusty's gives 20% match). Monte Carlo now evaluates 5 moves (suggested by NN) instead of 300. Scott: "Best I've seen one of our programs do." Used CAFFE machine library, + 1-page NN model. Was this an invalid entry into the Go match? Likely, because of use of library code from other programmers. Cornell: Teaching Deep Convoluted Networks To Play Go. Monte Carlo uses NN to pick 5 moves at every level, so branching factor drops to 5 (vs. 300).
Dusty's program beat Will! But the computer entry was ruled invalid, due to use of other programmer's libraries. Nonetheless, we all agree that, with considerable effort, Dusty would be able to reprogram those libraries from scratch. And the program was sufficiently better than Will, that it seemed clear that Dusty would win future matches from then on. As a consequence, Don and Will bought Dusty out of the remainder of his bet for $800 (= repayment of the $700 Dusty had lost to date, plus an additional $100).
Running bet (vs. Dusty) begins: Joe Mathes took the final $5000 chunk.
Narinder was not present at the 5/17/2015 match.
|
|
Will won both games against the two computer players: beat Scott 318-43, and beat Dusty by 8 (+/- 42).
Will played as White in both games (receiving the usual 5.5 komi against Scott's program); the computers, as Black, went first (as usual). Dusty's program started with a 2-stone handicap.
Scott: Reimplemented group analysis (static analysis of what groups of stones are connected). Been working on it "reasonably consistently for the last month. Significant progress." Cleaned up the code, so that it is easier to work on in the future. Will won, 318-43; Scott's program secured one edge.
Dusty: Essentially the same program as last time (but with slightly better opening book knowledge). The program stopped in the midgame. The estimate was that white won by 8 (+/- 42).
Joe: On 3/31/2014, Joe exercised his right to "opt out" of his perpetual bet. The buyout provision technically required $1000 in losses first, plus a final match (including the stakes of the final match) before a final $100 termination fee. Joe was allowed to opt out for a total of $100.
Joe and Narinder were not present at the 5/18/2014 match.
Will won all 3 games against the 3 computer players. Will beat Scott's program 187 to 18, and defeated both Dusty's and Joe's programs.
Will played as White in all three games (receiving the usual 5.5 komi against Scott's program); the computers, as Black, went first (as usual). Dusty's and Joe's programs both started with a 2-stone handicap.
Scott: "Part of the reason I haven't made much progress with my program is: I decided Go was too easy. I decided to focus on the Turing Test." In the midgame, Scott's program passed. Will immediately passed as well, which meant the game was over. Will won, 187-18.
Dusty: Big improvements (or at least changes) to the opening book. Better at local pattern recognition.
Joe: Joe's program played an opening, and then forfeit. (It wasn't a serious program.)
|
|
Will won all 3 games against the 3 computer players. Will beat Scott's program 315 to 46, defeated Dusty's program, and beat Joe's program by whole board to 1 point.
Will played as White in all three games (receiving the usual 5.5 komi against Scott's program); the computers, as Black, went first (as usual). Dusty's and Joe's programs both started with a 2-stone handicap.
Scott: "Sadly, my program is no better. But not for lack of trying. I did build a simple stochastic player. Unfortunately, it was worse than my previous program." Scott's program passed, halfway through the game.
Dusty: "Got the pattern player working. It plays noticeably better on 9x9 boards. Sped up the code 10x." Played opening book moves (very strong) through the first handful of moves.
Joe: "Makes a move in a few microseconds, and then sleeps for 30-200 seconds, to pretend that it was thinking." The move choice was by "copying a move from the most similar master level game it could find", where similiar is "defined badly but for quick lookup". In the mid-game, from Will: "This is like popping bubble wrap." Joe's program captured a stone, so he got one point; Will got the whole board.
|
|
Will won all 3 games against the 3 computer players. Scott's program "passed" halfway through the game, and lost with zero territory. Dusty's program finished with a little territory, Will won that game 101 to 35. Joe got a working game player, but didn't play a game.
Scott was moving to a new house, so the standard date for the match was postponed by a couple of weeks until the Sunday of Thanksgiving weekend. Scott hosted at his new house.
Scott: "Between moving to a new house and getting a new job (@ Google), I haven't made any changes."
Scott's program "passed" halfway through the game, believing that all black stones were dead. Scott conceded, with zero territory.
Question to Dusty: "Your program had some bugs last time. Have you fixed the bugs?" Dusty: "I made a lot of changes. It's a whole new program." Dusty's program was not running on the Amazon servers this time, but instead his new MacBook (16GB RAM, retina display). Dusty added "open book knowledge trained off downloaded games" (both whole board and section matching), and moved from a 1 CPU implementation, to multicore.
Dusty implemented his "pattern player" (vs. a random player), but it was too slow. (20K games/sec vs. 300 games/sec.) Thought he was done at midnight, but after adding training data the program ran too slow, so he actually finished at 8:30am.
Dusty thought it was playing "connect-12", although it had been "trained on connect-4". His own evaluation was that it was playing "decent moves ... against a random opponent".
Result: Will won, but Dusty's program got some territory. (Don lost a $1 side bet to Scott, on whether Dusty's program would have any territory at all by the end of the game.) Will wins, about 101 to 35.
Joe had nothing, when the day started. He began programming, as the other two games started. Halfway through, he announced: "I downloaded a Go GUI. You can see the board, but it won't make a move." By the end of the match, Joe successfully implemented a random player, hooked up to the GUI.
Will defeats Scott, white 316 to black 65 (Chinese scoring). Also Will defeats Dusty.
Scott's program changed very little since the previous time. He tried to make it more conservative in the estimation of life & death. He also left Yahoo two weeks ago. Will began the game with a beer. Outcome (Chinese scoring): White (Will) 316, black (Scott) 65.
Dusty, for the first time, stayed up the entire 24 hours before. He improved the engine, and ported it to Amazon EC2. For the match, he purchased time on 100 servers, for a 100x parallel implementation. He also changed the evaluation: now using minimax for 2ply, instead of just the top move (before random search).
Will beat Scott's program, by 192 points. Dusty's program lost (resigned) to Will.
Scott was running on new hardware, a Mac (requiring reimplementation of the graphics), about 10x faster. He made a "small change" to incorporate learning, in the evaluation function.
Scott (in the middle of the match): "It makes me wonder whether, with a few tweaks, my proram could play a lot better."
A week ago, Dusty's program beat him on a 9x9. Dusty's program picks a move, then plays a sample of random games (with random moves) until the end. In contrast, Scott's program does a few ply of lookahead, and then evaluates the material strength of the resulting intermediate position. (Dusty's program does no intermediate evaluation.)
Will vs. Dusty's program: "It was kind of close, in a way: twice, I thought I saw ways that the bot could hook its big dragon to the live group in the lower left, or claim an eye and have a fighting chance of creating another eye in the upper left corner. But, it basically squandered those opportunities..." In the end, "black makes no life."
Dusty resigned, with no final score, but remarked: "I think it will always be hard to judge relative strength of my bot based on score, because it prefers to play bad moves that 'might' win rather than good moves that don't change the fact that it's losing, but prevent it from losing by 'less'."
Later, Will "demolished" Dusty's program on a 9x9. Dusty: "I realized from some corner play (and the bot's estimations of its chances) that I think there is a bug which is causing the bot to incorrectly assume the opponent is playing mostly randomly after a horizon of about 1.5 ply with the parameters it was fielding today. So, maybe with that fixed it will be better about not letting live groups die."
Scott's program lost to Will. (But it beat Dusty's program. Dusty's program lost to both Scott's program and also to Will. Narinder was unavailable, traveling.) Also, Will lost to Don (twice) in submission grappling ($1/match, 100:1 odds).
Scott attempted many improvements. His program appeared to play better. Scott credited the improvements to:
Dusty changed his program considerably, but it was unclear if it was any better. He slept not at all the previous night. He "finished the pattern player", but "didn't have time to generate the pattern database" (so that code was not used). Dusty was (jokingly) accused of "programming in real time", as it was noticed, during the match, that his Github source code repository was "last updated a minute ago". Halfway through the match, the consensus from the humans was that it seemed to be playing more randomly, and worse. An incredulous comment: "You were up all night ... and you made it worse?" Midway through both games, Dusty forfeited to both Scott's program and to Will. (The games were not completed.)
Separately, Will suggested a new bet with Don: physical submission grappling until one participant gives up. Will is about 175 lbs; Don about 145 lbs. However, Don has trained in martial arts for more than 20 years and holds a black belt in Brazilian Jiu-Jitsu, while Will only has some brief instruction in karate. Will agreed to risk $1, on 100:1 odds, that he could defeat Don. He attempted twice, and lost both matches. First match: Don feels out the takedown, pulls guard, gets full guard, transitions to omoplata, Will powers out by straightening his arm, Don (now on top) transitions to a rear naked choke, Will taps. Second match: Don remains standing, lots of back-and-forth arm pummeling for position, Don eventually gets double underhooks/bearhug, more jockeying for balance, finally Don gets an outside leg trip takedown (like Royce vs. Gordeau in UFC 1), and winds up straight into mount position on the ground; Will taps from position (no submission).
|
Videos courtesy Dusty Leary |
|
| Match 1 video (31M) | Match 2 video (32M) | |
A few days after the grappling match, Will said that he had spent some time thinking about what new strategy he might use against Don next time. But then he stopped himself, suddenly realizing that "six months thinking about strategies" vs. a BJJ black belt (who has trained submission grappling for a decade, under a teaching lineage that has about a century of experience) is sort of like "Scott writing a world championship Go program without reading a single paper on the the state of the art".
Scott's program was "slightly tweaked". Forfeited match against Will. Instead, played a match against Dusty's program. Dusty's program won, 67 to 50.5 (= 45 + 5.5 komi).
Dusty's program (black) played Scott's program (white) with 5.5 komi. Scott
"tried to write" his "next-generation life & death" algorithm, and wound up
instead with a new appreciation for his old algorithm.
Dusty got one hour of sleep the previous night (10am-11am). Dusty's program
used Monte Carlo methods. "It's never actually played a real game."
The games:
The end:
Scott forfeited without his program playing a game. (No change in six months.)
Scott spent a lot of time "thinking", but no time "coding". As the program hadn't changed, he forfeited the match. Meanwhile, Will has been training regularly with a professional Go tutor, and believes himself to be about 10 kyu. Scott only plays at our biannual Go challenges. Will said he would "easily" beat Scott in a straight up game. Will wound up losing to Scott, 44 1/2 (39 + 5 1/2 komi) to 53.
Ben (Will's brother) won easily. Scott's program had improved slightly, especially in the early game. Narinder was out of town, in India.
Jill (Will's girlfriend) wins by a score of 151 to 73.
Opening |
A week before the match, Scott admitted that
As of right now, my program has not changed at all since last time, though I have some vague hope of implementing a new learning idea tomorrow or Wednesday.
Early game |
After some scheduling chaos, this match took place on the Saturday after the officially scheduled date.
Will offered up his girlfriend Jill as the designated human champion. Jill had never played the game before a few months ago, and had only played about twenty games on a 9x9 board as practice. This was the very first game of her life on a 19x19 board. Scott was thrilled to offer his computer the best chance at a victory since the bet began five years earlier. (The game itself wound up being one of the most even and interesting in the history of the bet.)
Jill (a yoga instructor -- notice seated position in first photo) was extremely nervous all day, but bravely stepped up to the plate anyway, playing black.
Late game |
As the early game developed, Jill began to lose about 1/3 of the board to Scott's computer, not contesting one corner and edge at all. However, in the rest of it, she did a good job containing and counterattacking.
Final victory! Humans win. |
In the late game, Jill had lost the 1/3 of the board. The rest of the board was a tactical battle, where Jill generally had at least a slight advantage in almost every local area. She would have to basically win each of the battles, in order to win the game.
And ... she pulled it off! There was a minor scare at the end, where the computer barely managed to make (space for) two eyes at the bottom of the board, and a small formation survived. But other than that, Scott's program kept the 1/3 of the board that it won early, but Jill won everything else. At the position shown in the final photo, the computer passed. And Jill was thrilled to pass as well, thus ending the game and her long nervous nightmare. But with great success! Final score was Jill 151, to Scott's program 73.
Ben (Will's brother) wins 238 to 128.5. (Also, Will beats Scott by surprise, by 20 points.)
Will was supposed to be the human champion, but at the match offered his brother Ben instead. Ben beat Scott's program by 238 (black) to 128.5 (white). (As usual, the end stages of the game were completed by the computer playing both sides.)
Overheard during the match: Scott saying
What I've discovered after all these years of working on it, is that machine learning is pretty simple.
At the same time, Will challenged Scott to a human vs. human game. They've played many times together in the past years, and it's always been clear that Scott is much better. So, in these games, Will always starts with a handicap advantage of a handful of stones.
But in this game, Will offered to play Scott even, with no handicap. Scott readily agreed, and thought it would be amusing. And of course, he was distracted watching his computer play (and lose) against Ben.
As the game with Will progressed from the opening to the midgame, Scott discovered that things weren't going quite as easily as he expected. Areas he thought were secure, wound up being threatened. Areas he thought he could attack, somehow escaped at the last minute.
Confused, Scott really began to concentrate in the midgame, and tried his best in the endgame. Alas, he wound up losing to Will, straight up, by 38 to 57, for a margin of 19 stones. Scott was flabbergasted, as Will had never come close to beating him with only a couple stone handicap, much less with zero handicap.
Only then did Will admit that, for the previous six months (since the last Go match), he had secretly hired a professional Go tutor (a Romanian 4 dan coach), and had been explicitly studying the game. Apparently, after a decade or two of Scott being much better at Go than Will, a few months of dedicated training allowed Will to catch up. (At least, to an unprepared Scott, who hadn't been playing regularly himself.)
As a result, Will was able to catch Scott by surprise in a head-to-head no-handicap game, and won by almost 20 points.
Will won, by a score of (approximately) 252 1/2 to 114. Scott claimed to have only made a "few small changes" in his program, but Will judged it to be an order of magnitude better than any previous version.
Scott implemented an entirely new program ("this time with search"). Will agreed it was much improved. Scott's laptop lost power right before the end of the game, so he forfeited.
Since Scott's programs have been so pathetic compared to Will's skill, we have been reduced (for entertainment purposes) to making subgoals. Our usual one was wondering whether Scott's program would have any territory at all by the end of the game.
As this match began, Will said that the usual no-territory subgoal was no longer interesting enough to him. He wanted to try a different handicap. He decided that he would make every other move "blindfolded". This isn't quite random, as he could place a stone in the rough area that he wished. Still, it made the game, and even the strategy, quite interesting. For example, many obvious typical Go situations (such as whether a group of stones is alive or dead) assume that you can make the correct countermove any time your opponent makes a move. It's quite a bit harder if your every-other move might not go exactly where you wish.
About halfway through the game, it appeared that Will was winning. But it was close. Will was also running out of time, so we all decided to abandon Will's odd blindfold strategy. He played another 10-20 moves just in the regular manner, with the objective of getting sufficient strategic advantage that we could just have the computer play itself quickly to finish the game.
When the game was slightly more than 1/2 done, Will stated that he was satisfied that he could win, even if the computer played both sides from then on. For speed reasons, we also decided to drastically limit the amount of searching Scott's program would do at each step. Naturally this meant it would make far more mistakes, but we figured it would make mistakes on both sides so it should cancel out.
Much to our surprise, as the endgame evolved, the outcome appeared to be in doubt. Many areas that Will thought he had shored up sufficiently to be sure they would remain in his hands at the end, wound up not being secured by the computer in the way Will imagined. For a short time, we were concerned that the humans might actually lose the match for the first time ever.
Then, as we were all watching the endgame quickly unfold on Scott's computer screen, Will's brother Ben said "is your battery low?". Indeed, a notification message had shown up on Scott's computer screen. He had plugged the laptop in to an electrical outlet, but it appears that the outlet was not actually providing electricity. Scott quickly attempted to save the game in progress, but in the process of pressing some keys to do so, somehow managed to simply quit the entire application without saving. The laptop still had a tiny bit of power remaining, but now the Go program was no longer running, and the game state was lost forever.
Scott conceded the match.
Scott worked hard (for two weeks) on his program, but the program he actually fielded was clearly worse than those in previous matches. He forfeited the main game to Will without playing. He lost to Narinder by 45.5 points (16 to 56).
In the previous match, Scott's program had worked "bottom up" to evaluate the board. He decided that this was a theoretical error, as the search space grew too large too fast. He spent (the last) two weeks working hard to implement a top-down search instead, only to find (the day before that match) that it, too, suffered from the same too-large search space problem.
Scott forfeited the match against Will, since he would otherwise have submitted the same program as last time, and Will already crushed that one.
There was an additional one-time side bet of Narinder ($10 Will) vs. Scott's program ($10 Scott) in a separate match. Scott intended to use his latest working version, but the source was not in a working state on the last day. (Scott claims that, with just one more day, he could have gotten it working.) Scott then intended to use the most recent version from six months ago against Narinder, but was unable to find that executable on his laptop. (Apparently, it had been deleted.) The most recent executable he could locate was dated 2004. This program actually played an interesting match against Narinder, who had just learned Go for the first time and spent a total of about 15 hours in the previous two weeks studying the game.
Narinder (playing white) won, by a score of 56 to 16. Including the komi of 5.5 points, Narinder won by 45.5. A very respectable, although not quite dominating, victory to the novice human.
Scott's program lost with zero territory. However, it was agreed that its play had improved greatly since the previous match. A "small bug" resulted in often placing stones in atari; without the bug, Will guessed that it might actually have won some small territory.
Scott didn't change his program in the previous six months, and forfeited without playing the match. Narinder was travelling in India and didn't make it.
Scott didn't change his program in the previous six months, and forfeited without playing the match.
Will (white) won the entire board. Scott ended with no territory, and a handful of prisoners.
Will (white) won the entire board. Scott ended with no territory, and a single prisoner.
Scott forfeited.
Don ~+200, Scott ~-200. Scott ended with a small group of about 10 stones still alive. Original email report (reproduced below).
From: Don Geddis
The long awaited, six-months in the making, man vs. machine Go match finally took place yesterday:
"Scott Roy"
-------------------------------------
Don Geddis vs. Scott's Go Program
Sunday June 8th, 11:30am
Stanford Coffee House
-------------------------------------
It was probably my first and only game on a full 19x19 board. I didn't realize how long those games can take! Hours later, we were finally done.
How did it go? Scott's amazing program achieved the following remarkable feats:
Scott was a little rushed at the end of the game, so we didn't finish the whole thing and come up with an exact score. However, we can estimate the approximate score. 19x19 is a little under 400 points. Near the end of the game, there were roughly the same number of white and black stones. At the end, on behalf of his program, Scott conceded all the stones except the lone group with two eyes.
So, at the end, Don had perhaps 250-350 points of territory, plus ~200 prisoners from the program. Scott's program had two points of territory, perhaps 10 stones still on the board, and maybe 5 or so prisoners of his own. Making the score, roughly, positive a couple hundred for Don, to negative a couple hundred for Scott's program.
Humans win again!
A quick postscript: Scott, ever the confident one, proposed re-upping the same bet for another six months. I was a bit reluctant, having actually been (needlessly!) worried the first time, I thought I had done my fair share to hold up the human side of things. Conveniently, Will Harvey (a much better player than I) offered to take my place. As Scott debated whether to accept this more challenging bet, Will offered him 100-1 odds. Attempting to push Scott over the edge, I joined Will in the stakes.
Scott accepted. The Great Scott Roy Go Computer Challege, #2, is now set for six months away. On Sunday, December 7, 2003 (or within a week of that date), Scott's latest "new and improved" Go program will play a single challenge match against Will Harvey. If Will wins, Scott owes me and Will each $10. If Scott's program wins, we each owe Scott $1,000.
For those considering some side bets: it may interest you to know that in the first Great Go Challenge, Scott's program spent the first five months and three weeks "sucking". Then, in the last few days, it improved by "an order of magnitude". The final version was completed at 2am on Sunday morning, for the Sunday 11:30am contest. Those of you with experience in delivering robust software may wish to take this into account when estimating the likelihood that Scott's program six months from now might defeat Will Harvey.
| webmaster@geddis.org | For encrypted email, use my PGP Public Key. | Last updated 11/10/24 |
