Archive:Update system

Introduction to the problem

The Mana World plans to be an constantly expanding game, with new features, maps and other game media being added all the time. This constant addition requires some way to distribute new material. The Mana World up to this point has been using a monthly release cycle to update the game, but this distribution method suffers in the following ways:

• Data redundancy - This is a major concern for those who have slower connections to the Internet. For each update, all existing data is also downloaded.
• Not frequent enough - Players want new material as fast as they can get it. Players like a fast updated, dynamically changing game world in which they can immerse themselves - any stagnation can lead to player loss.
• Complicated - Making a new client release involves a complete ritual of providing the necessary binaries and updating many websites. For content updates or tuning this process can be skipped.
• Breaking compatibility - Each time new content is added, the game will break for anybody using an older client. When content updates are done separately, very often a forced client update can be avoided.
• Single universe - Distributing and updating the data with the client assumes the client is only ever used with some single official server. Dynamically downloading and updating the data would allow the client to be used on unofficial servers.

The Mana World wishes to address these issues by using an built-in update system which will automatically download the latest necessary game data depending on the server you login to.

• The redundancy is taken care of by only downloading updated files.
• More frequent updates become possible by avoiding the client release cycle and reducing the complexity surrounding an update.
• Compatibility is greatly enhanced by ensuring the data is up to date at all times.
• By allowing the server to specify which data is to be downloaded, one client can be used on different servers featuring different worlds.

Update process

Updates come in compressed archives (.zip files). They are easy to create and allow individual files to be read directly from them (we use PhysFS for this). Initially, the client will retrieve an ordered list of archives that it should load, for example:

tmw-base-1.0.zip
tmw-base-patch-1.1.zip
tmw-tonori-region-1.0.zip

The client will check whether all the listed files are present in its data directory, and attempt to download any missing ones. Once downloaded, it will put each archive in the virtual file system provided by PhysFS. The archives will be searched bottom to top so that any file will be loaded from the last archive that provided it. This allows us to patch big data archives with a small archive that contains just the modified files.

Any files present in the data directory that are not in the list can be considered to be old, and could be cleaned up by the user with the click of a button.

This ordered list could contain protocol information for file retrieval. Such a list could look like:

ftp://themanaworld.org/updates/tmw-maps-tonori-1.3.zip
http://themanaworld.org/updates/tmw-gfx-items-1.1.zip

I'm not sure about sending full URLs yet, as long as we're using http/ftp I think it'd be better to set up one or two mirrors and optionally allow the user to choose where to download from. --Bjørn

Updates will be downloaded to a directory writable by the user. On Windows it is assumed the user has write access to ./updates while on UNIX-like systems ~/.tmw/updates is used.

I think updates should be downloaded to subdirectories depending on the server, and that as little default data should come with the client as possible. An example directory used to store the server data would be ~/.tmw/data/testing.themanaworld.org. --Bjørn 20:39, 24 December 2005 (CET)

Add-on to update protocol :

A way to avoid unuseful updates and out of sync data files when getting the latest tmw version could be made this way : We should think about naming the updates accordingly to what the user will be able to read. Maybe a list box should list the available updates and their versions (after downloading a list file.) <-- Using that file, we could enable/disable updates if necessary by removing/adding them to the list.

ftp://themanaworld.org/updates/tmw-maps-tonori_0.0.18.1_1.3.zip
http://themanaworld.org/updates/tmw-gfx-items_0.0.20_1.1.zip

We could use that format : updateName_tmwVersion_updateVersion.zip As seen above, the files should have the maximum version of tmw client in their names, so that if the user have the 0.0.21 version, these updates are not downloaded. Moreover, if they have been previoulsy downloaded, the client should take care about deleting these files as they aren't needed anymore, and all that, before initializing PhysFS. To list an update, the tmw version of it should be strictly higher of the current tmw version of the user. The _tmwVersion_ should then be read as 'for clients with the version less than XX.' So, if you want an update available till 0.0.19, you can name it 0.0.20.

I think the client version should be completely irrelevant to the updates downloaded. In the long run the goal should be to release a bare client which can connect to any server, and will download different data files depending on the server it connects to. Using a new client release to do data updates should already belong to the past, but sure I understand our update system is still rather broken. However, the part above this add-on should already cover the necessary improvements. I think this add-on only adds unnecessary filename parsing and complexity. --Bjørn 22:02, 8 March 2006 (CET)

If there is to be a second version of the update, a simple increase to the updateVersion is to be done. Again, the tmw client should then download the new file, and then delete the old one before physFS initialization.

The update window and news window :

As the users really like to KNOW what is downloaded to their computer, (and that something for the game is downloaded btw), the news window should be one thing, and then an update button should appear if there are updates that are to downloaded, and are not already. Clicking on it would then show another window (that one listing the available updates.) and with three buttons : - Close - update (if there is a selection in the list) - Update all (If there are still updates.) And a scroll bar to show the progresses made.

I know some will say that system is quite Debian like and they will be right, but isn't the debian update system one of the best ;) I hope that way, the unused files will be deleted, and the user will keep uptodate data files. Comments will be very appreciated :) -- Bertram

Update distribution

Updates must be distributed fast and effectively, providing a decent throughput even when the update server is under a high load. The Mana World development team has decided that the BitTorrent protocol will be utilised for update distribution. BitTorrent is a distributed peer-to-peer protocol which utilises the bandwidth of all users. For more information about BitTorrent, see the BitTorrent Introduction.

To easily utilize to the BitTorrent protocol, one of the following libraries can be used:

• libbt - C
• libtorrent - C++
• libTorrent - C++

Yeah I know we already choosed for BitTorrent, anyway we shouldn't discard HTTP/FTP both because some people don't want to be forced to share their bandwidth or if they're having problems. Someone also asked for the possibility to stop sharing patches (in case updater is built-in). A noticeable example is WoW updater which both allows for bitTorrent or HTTP downloads. And the patch sharing is limited to the time you download the patch (until you don't press "Play game!" I guess). A deeper evaluation of band usage and lag problems should be provided. --ElvenProgrammer 02:08, 4 Jul 2005 (PDT)

Because both libbt and libtorrent still seem immature and have annoying dependencies, I would suggest to hold off from BitTorrent for now and use libcurl to retrieve the updated data files through HTTP or FTP. --Bjørn 12:06, 4 Jul 2005 (PDT)

I agree, libcurl is at least used by libbt so if we're going to use it, it will be required anyway. I'd like also to say that libcurl it's really easy and could help us to create the HTTP/FTP updater in no time.--ElvenProgrammer 12:55, 4 Jul 2005 (PDT)

I've added libTorrent which seems like an interesting choice because it depends on just libsigc++ and libcurl. I still think we're fine with just libcurl for now though, the BitTorrent distribution would be an idea for later. --Bjørn 01:43, 20 Sep 2005 (CEST)

"Q: Is anyone got it work under windows? (maybe any plan on porting?) A: Haven't heard of anyone doing so and i don't really care about the win32 platform and its lack of posix support. Unless someone pays me it'll propably not happen. ;)"

Just to let you notice windows is not supported. --ElvenProgrammer 08:39, 20 Sep 2005 (CEST)

Questions

What about updating the binary?

Updating the binary is a very operating system dependent operation. On Windows this will likely need to be done using a separate executable or some re-launch trick and on Linux it is not reasonable to assume anything about the versions of installed libraries. On both Linux and Mac the rights of the current user will likely also be insufficient to update the software.

It is much more important to simply decouple client releases and data updates, so that client updates occur infrequently and can be handled in a convenient way on each type of operating system. This will also allow for a better compatibility, so that the game may be played with older clients until an update for a specific operating system is available. On Linux for example, using the existing package management system of the distribution is prefered, where it would be nice if we could even fit into a lengthy release cycle of for example Ubuntu.

What if we versioned all the datafiles separately?

Too tedious and also unnecessary. When using archives, all the client needs is the list of files it should have, a way to retrieve them and the order in which they should be loaded. A file should either be loaded or it should not be loaded, and in the latter it could be removed instead. It may be better to leave the cleanup to the user though, and just provide a one-click solution.