Standalone server development update

September has begun, and we start getting natural questions like: “When will the open-source server go live that was promised in August?” So in this post we’d like to explain the situation that’s going on now.

Currently we have actually three parallel development pipelines for Screeps:

  • Features of the game itself and its gameplay
  • Optimizing the official server
  • Development of the standalone hosted server

When our second Indiegogo campaign was launched, the plan was roughly this: make a market system as a major feature, and then delve completely into the development of the standalone server and a Steam client for it. That’s why we set August as an approximate milestone date. However, after the campaign’s end and some consideration we realized that the Steam launch proper and the standalone server release are two vastly different things that should be better done one at a time rather than simultaneously. A higher traffic from Steam may require some adjustments in the scalability of the game servers, so we’d be having hard time watching this process and at the same time taking care of the open-source project that would appear along with it. We decided to first launch the Steam client to attract users to the main official world, debug it, and then move to the standalone server without any hassles.

Our experience has shown it was a right decision. When Steam gave us hundreds and hundreds of new players, it revealed a number of issues in our server architecture that prevented capacity scaling for running your scripts, and we put a lot of time and efforts to debug and optimize those bottlenecks. You might remember how the tick duration soared from 2.5 to 5 seconds, then fell back, then grew again, and then stabilized at 2.7-3 seconds. We are content with the current rate, but achieving it required a lot of work within the past two months.

Due to this reason, unfortunately, the standalone server won’t be launched within the coming weeks. We are sorry to realize that we missed some deadline, but please be assured this was not due to our laziness, but we decided to rearrange the work, and the history proved it was a right decision.

And there is another reason for delaying too. After the Indiegogo campaign was over, we spent some time on designing the concept of a standalone server, and we realized we couldn’t just take the current server code, open-source it, and rest on our laurels. For the project to gain traction as standalone as we see it, the following design concepts need to be implemented:

  • Zero dependencies. The official world works on a traditional server stack of many applications: nginx + nodejs + mongodb + redis. And it won’t work for the standalone server. The user should be able to just download an executable file via Steam, launch it, and get a working server at his IP address without any extra programs to install. We use Electron packaging to achieve this, but it also requires a separate database layer based on the in-memory database inside the application itself.
  • NPM-based application. Though many users will opt for the easy downloading and running the server as a simple desktop application, we have to support a more advanced installation as an NPM module as well. Being a JavaScript application, the Screeps server organically fits into the npm packets ecosystem, but the current game server architecture (which was initially developed as a set of layers with loose coupling) should be divided into multiple modules to publish them in the npm separately. This will allow, for example, replacing the database layer module with your own one to use the desired server-side stack.
  • Fully moddable architecture. To change the majority of server behavior aspects should require simple redefining of needed fragments through a system of external hooks and extensions rather than modifying the server code. We also need a possibility to add custom game entities even with their own visuals.
  • Single-player bot players. The single-player mode (which makes some people long for the standalone server) will be pretty boring unless there is somebody else but the player in the game world. But the development of a full-fledged AI that can start from scratch in the game world and seriously challenge the human player is a very interesting and complicated task. We want the players themselves to find the solution to it, since that’s exactly the main fun of Screeps. Any player could write a script for his or her standalone AI and submit it into the common repository where other players of the single-player mode can install it. When provided with its own room in the worls, such a standalone AI has to be able to analyze the situation, develop its colony, and bring some life to the game world where it exists. The repository will introduce a ranking system for AI performance, user comments to them, new version updates, etc.
  • Steam Workshop support. The two aforementioned options are perfect in terms of excellent mechanics of Steam Workshop. The system of downloading and installing any custom files provided by the Workshop can be utilized to automatically install and update custom mods, game object types, bot players, and entire complete worlds. This is a huge task, but it’s vital for the Screeps ecosystem.

Implementing some of these things requires the overhauling of the current server code of Screeps since they were not accounted for 2 years ago when the game was first designed. But we are convinced that short of these, the standalone server won’t be the type of product we really want to share with you.

In terms of deadlines, we don't want to be overly optimistic at this point making promises we won’t be able to meet. Let’s keep October in mind. If we get a working version earlier, we’ll definitely show it. As was promised, our Indiegogo campaign backers will be the first to get their hands on it, while the public availability will begin a month later.

Have more questions? Submit a request

Comments

  • Avatar
    Dissi

    Good write-up for the standalone server.

    Clear communication is one of the features what makes this game so amazing.

    I'm glad you analyzed and prevented a major catastrophe first by addressing scalability first. 

     

    I think we can wait a bit more for the standalone server :)

  • TooAngel

    Agree with Dissi, I like the open and transparent communication and good analysis. This improved a lot of the last year. Also the prioritization make IMHO sense.

     

    Regarding the standalone server, I already had the concerns on the last post about it, I'm not so sure if a complete package brings so much value compared to the prolonged time it takes to be open source. I don't fear setting up the nginx + nodejs + mongodb + redis stack / have it anyway running. I guess this fits for many others. I saw the first approaches setting up screeps as standalone from the simulator code, which shows me: 'We want to get our hands dirty' and hopefully we are able to help you to improve the game.

    TL;DR: Having something which I'm able to start and good documented dependencies: `git clone`, provide the credentials for these services, route the http traffic to this port and start this script, is more than enough.

    Don't fear if it is not perfect, it will never be. Anyway, just my 3 cents. Good work.

  • akuukis

    +1 Dissi for clear communication.

    +1 TooAngel, neither do I fear nginx+nodejs+mongodb+redis stack. IMHO those who like to "get our hands dirty" don't fear the stack or will set up some stack in the end anyway. And so you can get feedback from community on it before releasing truly standalone server.

     

    Would you also split your own features into a set of mods so they can be independently turned on/off? (i.e. if supercreeps is not very compatible with my own mod or if I just don't like it, then I just edit config file to disable it)

  • Atavus

    Thank you for taking the time to put this together.

    I do not believe anyone is doubting the dedication and efforts of the dev team. 

    Take your time to do things right. Just maintain active communication with the users to keep them informed.

  • DibiZibi

    There is one thing that I'm curious about...
    How powerful the server has to be to run the game on a very small scale?
    Let's say I want to setup a small world, maybe 25 rooms, to play against 1 or 2 people, and host it on my own personal PC (core i5 and 8GB or ram for example)

    How many ticks per second would I be able to achieve?

    Edited by DibiZibi
  • coteyr

    While I'm not really happy with the current tick rate, I am glad that you decided to spend time on stability rather then a standalone server. The standalone server will be a nice feature, and I am really looking forward to it, but I'd am much happier with a 3 sec tick then a 20 or 30 second tick. Now if you could just get that tick down to 1 second :P

  • ags131

    As TooAngel and akuukis stated, I too wouldn't mind having to setup a full stack.
    I've successfully rewritten /core and gotten simulator code mostly functional on my local server, although very buggy and missing a couple things it is functional with a local node + mongo + redis stack. even running it all from a single process its fairly quick and can handle 100 rooms smoothly.
    I would love to be able to poke at the full normal non-minified source and see what i could do with it, after digging through the minified code, even undocumented, the full normal source would be a goldmine. :D 

  • Avatar
    Artem from Screeps

    We are not going to release the existing database layer source code due to security reasons, not because it is complicated to set up. Revealing technical details of official server implementation might introduce a way to exploit it in some way.

  • coteyr

    That's just bad. 

    Security through obscurity is not security. You should be able to maintain your security even with the source code released. 

    You should also stop saying your going to open source the server. It's misleading, if you can't opensource all of it, then either stop staying you will, or  be very specific about what you are going to opensource. 

    Don't get me wrong, screeps is a fun game, and I'm not going to quite because of this decision, but as a big fan of opensource software, I don't like "Were going to opensource, well except this one important part". It takes away from the community. 

  • tedivm

    > We are not going to release the existing database layer source code due to security reasons, not because it is complicated to set up. Revealing technical details of official server implementation might introduce a way to exploit it in some way.

     

    This is the first time I've every really been upset at you or this game. You should have been clear before you took our money in the campaign that you were not actually going to open source the game like expected, but were instead going to open source an inferior version and force us to rewrite the database layer ourselves. 

  • Avatar
    Artem from Screeps

    We never told that we are going to release exactly this existing server-side codebase to open source. It just doesn't make any sense, it's optimized for very specific setup, we  cannot guarantee its security and reliability in different environments, and thus cannot maintain it as an open source project. What we have promised (and what we needed funding for) is to develop a new server based on the same game engine which will be fully open sourced and designed in an easily maintainable way.  Here is a quotation from the Indiegogo campaign:

    "To achieve this, we need to rework the existing engine in such a way that it can be launched outside of our server environment."

    It will be the same game with the same game client and persistent world. The engine itself will even use the same github repo along with the official server. And this is 95% of the current codebase. The database layer is really thin actually, I suppose it can be rewritten in a few days using any desired technologies. Moreover, one user already has done it, using only the obfuscated client-side code, so I doesn't see any real difficulties here. 

    Edited by Artem from Screeps
  • tedivm

    Will you be replacing the custom version you have here with the open source version you are building?

  • coteyr

    >  Will you be replacing the custom version you have here with the open source version you are building?

    That's the important question to me too. 

     

    To be clear, I don't care if we get today's whatever as an opensource project only that we get, and get to keep, the same software that actually runs the live, production, servers.

    Edited by coteyr
  • Avatar
    Artem from Screeps

    Well, I'm sorry if I was unclear, but you still don't seem to get the point.

    Look, we have layered loosely coupled architecture. There will be two (at least) open source codebases, two github repos, two npm modules. One (game engine) is about how the game works, its scripting API, how world objects are processed, how player code is executed using the node VM. The second (database layer) is literally a small set of methods which are responsible to make strictly defined database requests and also contains some specific optimizations and caching. The first codebase is shared between the official server, the official simulaton and the local server. The second one is different in every case. The game itself will be exactly the same in all cases, but how it connects to our server cluster will obviously differ a lot from how it connects to an in-memory JS database (this one for example) on your usual laptop, or even to a desktop-grade MongoDB installed as a Windows service. Database layer modules are easily pluggable as long as they follow the interface, and many unofficial modules may appear in the future. But the game engine is what the game really is, it will be officially maintained, and you can be sure that exactly the same game engine is used on the official server.

  • ags131

    Base on what I've seen from the sim code and what Artem has told us, effectively the part not being opensources is purely the /core part of the engine, which is almost entirely DB layer, which means, once released, my server should easily be ported over just by pointing the OSS engine code at my core implementation and not using the minified sim code. At that point I should even be able to simplify my code as I would no longer have to redirect all core requires to my code.
    And If I get what Artem is saying correctly, they will be releasing a core implementation, its just going to be a much more local version and not a distributed mongo/redis setup.
    @artem: I'm definitely still looking forward to it, and thanks for linking that db, I've never heard of it before, I may have to give it a try. :)

  • Avatar
    Artem from Screeps

    And If I get what Artem is saying correctly, they will be releasing a core implementation, its just going to be a much more local version and not a distributed mongo/redis setup.

    That’s correct. It will make use of LokiJS most likely, though we’re exploring the alternatives as well.

  • DoctorZuber

     

    Are you planning on keeping both servers current to the same features as you develop?

    It seems like you might be committing yourself to a dangerous amount of additional work if this is the case. As much as possible, you really do want them both to be as same as possible to lighten your own work load.

     

     

     

  • DoctorZuber

    I like that you are contemplating making a single player AI, that feature alone excites me quite a lot. But do yourself a huge favor, and don't.

    I don't mean don't include an AI. I mean don't write it. Use our AI instead. Instead of trying to make a challenging AI for players to face in the game world, make a Turing test that can identify our own code which is already a usable AI, and then just use that.

    Just take care with privacy, since leaking our own code to other players could be quite disruptive to the game itself.

  • Avatar
    Artem from Screeps

    Are you planning on keeping both servers current to the same features as you develop?

    Gameplay engine will be in one single codebase shared by both projects.

    I don’t mean don’t include an AI. I mean don’t write it. Use our AI instead.

    Apparently you have misunderstood something. This is our plan exactly. It is explained in the “Single-player bot players” paragraph:

    We want the players themselves to find the solution to it, since that’s exactly the main fun of Screeps. Any player could write a script for his or her standalone AI and submit it into the common repository where other players of the single-player mode can install it.

    Edited by Artem from Screeps
Powered by Zendesk