I am a pretty mediocre chess player, but at the start of 2019 I was playing a lot of online Blitz. My opening repertoire was pretty slim and I felt myself repeating the same mistakes. I wanted to try out some new openings, but looking at existing opening databases, I found a few areas were lacking:
• Data from lower-rated players like me
• A clear visualization
• A tree showing transpositions
• Low/high Elo variations highlighted
I decided to play around with processing some PGNs of chess games myself and see what I could make. And after talking about the project with a friend who is less interested in chess, but more interested in web design, we decided to collaborate.
Initially, we were going to process a few thousand games, mostly for personal interest. But pretty soon we came across the whopping 800 million game (as of 2019) Lichess database. This was perfect as it contained a wide range of player abilities and time controls, and had such a large number of games that even after being filtered, the sample size would be large enough to draw conclusions from.
After some work, we were able to process all chess states with at least ~800 games (out of the 800 million game database). Using this threshold allowed us to transform the more than 1TB of Lichess PGN's into a much more manageable 60GB of chess state data.
However, 60GB is still completely unreasonable to ask someone to download when they open a webpage! This made it necessary for us to build a server that when requested can return the most relevant information from a particular chess state. This lets us show you an opening in much greater depth while sending you MBs of data instead of GBs of data.
A server capable of searching through 60GB of data, and quickly sending you the most relevant results is by far the most expensive part of this project. To make it work we need supporters to help pay for the servers and allow us to keep developing the site, if you are interested check it out here.
