From The Mana World
(Added some responses)
m (removing category)
 
(20 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== Update system ==
:''This article is about the technical aspects of the updating system. See [[how to release an update]] for the steps currently required to release an update.''


The Mana World plans to be an constantly expanding game, with new features, maps and other game media being added all the time. This constant addition requires some way to distribute new material. The Mana World up to this point has been using a monthly release cycle to update the game, but this distribution method suffers in the following ways:
== Introduction ==


* Data redundancy - This is a major concern for those who have slower connections to the Internet.
The Mana World is a constantly expanding game, with new features, maps and other game media being added all the time. This constant addition requires some way to distribute new material. Our update system has been designed with the following requirements:
* Releases aren't frequent enough - Players want new material as fast as they can get it. Players like a fast updated, dynamically changing game world in which they can immerse themselves - any stagnation can lead to player loss.
* Bad releases hard to update - Once a package has been released, soon after its release some major problems may be found, which may require immediate attention. The old package must then be removed and replaced with a new fixed package, by this time many people would have downloaded it and would probably not know about the new updated release. This is a major problem in itself.


Regarding the last topic, we discussed it a bit, and thought that first of all, we need a version checking, so that everyone is informed about a new release. In windows we were evaluating the possibility of directly replacing the binary, while this won't be possible in GNU/linux. I guess we will continue with monthly releases, and will discuss in this article only static data updates. --[[User:ElvenProgrammer|ElvenProgrammer]] 02:08, 4 Jul 2005 (PDT)
* '''Low data redundancy''' - Files can be updated individually while not causing re-downloading of other data.
* '''Easy and independent of release cycle''' - Allowing updates to be released fast and frequently.
* '''Compatible''' - Updates can also keep older clients up to date, allowing them to keep playing without needing to update. Provided that any of the new data doesn't need new client features.
* '''Multiple worlds''' - The updates that the client downloads are server dependent, so different worlds can be run on different servers. ''(Not implemented yet, 0.1.0 feature)''


The Mana World wishes to address this by using an in-built update system which will automatically download the latest game media. By doing this, the second and third points noted above are eliminated. This update system should provide the following features:
== Update process ==


* Smart updating - the updater should be able to find files which need to be updated and update them, without trying to update files which are already up to date.
Updates come in compressed archives (.zip files). They are easy to create and allow individual files to be read directly from them (we use [http://icculus.org/physfs/ PhysFS] for this). Initially, the client will retrieve an ordered list of archives that it should load, for example:
* Cumulative patches (Optional)
* Media updates - should be able to fetch new and updated game media.
* Executable updates - should be able to update the binary executables appropriately.


The option of using a built-in updater is meaningful if using bitTorrent protocol (even if not necessary). If using HTTP/FTP, it could be valuable using an external application to avoid increasing the number of dependencies/size of executable in the client. About executable updates, see comment above. --[[User:ElvenProgrammer|ElvenProgrammer]] 02:08, 4 Jul 2005 (PDT)
<pre>
tmw-base-1.0.zip
tmw-base-patch-1.1.zip
tmw-tonori-region-1.0.zip
</pre>


== Update format ==
The client will check whether all the listed files are present in its data directory, and attempt to download any missing ones. Once downloaded, it will put each archive in the virtual file system provided by PhysFS. The archives will be searched bottom to top so that any file will be loaded from the last archive that provided it. This allows us to patch big data archives with a small archive that contains just the modified files.


The patch format must be easy to use and flexible, and should support the required features mentioned above.
Any files present in the data directory that are not in the list can be considered to be old, and could be cleaned up by the user. Updates will be downloaded to a directory writable by the user. On Windows it is assumed the user has write access to <code>./updates</code> while on UNIX-like systems <code>~/.tmw/updates</code> is used.


* The format will be added here.
:''I think updates should be downloaded to subdirectories depending on the server, and that as little default data should come with the client as possible. An example directory used to store the server data would be <code>~/.tmw/data/testing.themanaworld.org</code>.'' --[[User:Bjørn|Bjørn]] 20:39, 24 December 2005 (CET)


The patches will be released using compressed archives. Probably using bzip compression is a wise choice to reduce downloads size. I suggest that every client keeps a local copy of the version of each file:
:''About the update window and news window: As the users really like to KNOW what is downloaded to their computer, (and that something for the game is downloaded btw), the news window should be one thing, and then an update button should appear if there are updates that are to be downloaded, and are not downloaded already.<br>Clicking on it would then show another window (that one listing the available updates) and with three buttons: Close, Update (if there is a selection in the list) and Update all (if there are still updates). And a scroll bar to show the progresses made. --[[User:Bertram|Bertram]]
''/data/items.xml 0.12''
''/data/music/Faith.ogg 0.3''
and every patch file will have a list of the changes, let's say changes.txt:


::''It would be a good idea not to start downloading immediately, but I think the news and update windows should remain integrated.'' --[[User:Bjørn|Bjørn]] 23:14, 18 September 2006 (CEST)


== Update packages based on data structure ==
The <code>data</code> directory present in Subversion is no longer entirely included in releases. The following listing describes which parts are only released via dynamic updates, and to which subgroup they belong.


<pre>
<pre>
/data/items.xml U 0.13    means it need to be updated
equipment.xml                  database
/data/music/desert.ogg R    the file should be removed, no longer used
items.xml                       database
/data/graphics/avatars/hammerbearcaricature.png A 0.1    new file, needs to be added to local list
monsters.xml                    database
graphics/images/minimap_*.png  minimaps
graphics/images/ambient/*.png  ambient
graphics/items/                items
graphics/particles/*            particles
graphics/sprites/*              sprites
maps/                          maps
sfx/                            sounds
</pre>
</pre>
Using those infos, the previous data packages could be easily merged with the patch.
Note: this is a quick suggestion, we could think of a lot of better ways. --[[User:ElvenProgrammer|ElvenProgrammer]] 02:08, 4 Jul 2005 (PDT)


----
The naming convention is <code><subgroup>-<revision>.zip</code> for a package containing all the data at a certain Subversion revision, and <code>update-<start>-<end>.zip</code> for a package containing the material that changed or got added between the start revision and the end revision.
''This could be changed to:''
 
== Future update distribution ==
 
Updates must be distributed fast and effectively, providing a decent throughput even when the update server is under a high load. The Mana World development team has decided that the [http://www.bittorrent.com BitTorrent] protocol will be utilised for update distribution. BitTorrent is a distributed peer-to-peer protocol which utilises the bandwidth of all users. For more information about [http://www.bittorrent.com BitTorrent], see the [http://www.bittorrent.com/introduction.html BitTorrent Introduction].
 
To easily utilize to the BitTorrent protocol, one of the following libraries can be used:
 
* [http://libbt.sourceforge.net/ libbt] - C
* [http://libtorrent.sourceforge.net/ libtorrent] - C++
* [http://libtorrent.rakshasa.no/ libTorrent] - C++
 
Yeah I know we already chosen for BitTorrent, anyway we shouldn't discard HTTP/FTP both because some people don't want to be forced to share their bandwidth or if they're having problems. Someone also asked for the possibility to stop sharing patches (in case updater is built-in). A noticeable example is WoW updater which both allows for bitTorrent or HTTP downloads. And the patch sharing is limited to the time you download the patch (until you don't press "Play game!" I guess). A deeper evaluation of band usage and lag problems should be provided. --[[User:ElvenProgrammer|ElvenProgrammer]] 02:08, 4 Jul 2005 (PDT)
 
Because both libbt and libtorrent still seem immature and have annoying dependencies, I would suggest to hold off from BitTorrent for now and use [http://curl.haxx.se/libcurl/ libcurl] to retrieve the updated data files through HTTP or FTP. --[[User:Bjørn|Bjørn]] 12:06, 4 Jul 2005 (PDT)
 
:I agree, libcurl is at least used by libbt so if we're going to use it, it will be required anyway. I'd like also to say that libcurl it's really easy and could help us to create the HTTP/FTP updater in no time.--[[User:ElvenProgrammer|ElvenProgrammer]] 12:55, 4 Jul 2005 (PDT)


Updates come in compressed archives, probably .zip files as they are easy to create and allow individual files to be read directly from them (this is already being abstracted using PhysFS, so there is no need to inflate the archives after downloading). Initially, the client will retrieve an ordered list of archives that is should load, for example:
::I've added libTorrent which seems like an interesting choice because it depends on just [http://libsigc.sourceforge.net/ libsigc++] and [http://curl.haxx.se/libcurl/ libcurl]. I still think we're fine with just libcurl for now though, the BitTorrent distribution would be an idea for later. --[[User:Bjørn|Bjørn]] 01:43, 20 Sep 2005 (CEST)


<pre>
::"Q: Is anyone got it work under windows? (maybe any plan on porting?) A: Haven't heard of anyone doing so and i don't really care about the win32 platform and its lack of posix support. Unless someone pays me it'll propably not happen. ;)"
tmw-base-1.0.zip
::Just to let you notice windows is not supported. --[[User:ElvenProgrammer|ElvenProgrammer]] 08:39, 20 Sep 2005 (CEST)
tmw-base-patch-1.1.zip
tmw-tonori-region-1.0.zip
</pre>


The client will check whether all the listed zip files are present in its data directory, and attempt to download any missing ones (see update distribution below). Once downloaded, it will put each archive in the virtual file system provided by PhysFS. Once a data file is loaded, the archives are search bottom to top so that the file will be loaded from the last archive that provided it. This allows us to patch big data archives with a small archive that contains just the modified files.
== Questions ==


Correct me if I'm wrong, but this won't keep all the old data? Or maybe old packages will be overwritten by new ones? If not just think that having 3 different versions of a song would lead for example to not required 6-9 MBs. --[[User:ElvenProgrammer|ElvenProgrammer]] 12:55, 4 Jul 2005 (PDT)
=== What about updating the binary? ===
:Any files present in the data directory that are not in the list can be considered to be old, and could be cleaned up by the user with the click of a button, for example. --[[User:Bjørn|Bjørn]] 09:02, 7 Jul 2005 (PDT)


This ordered list could should contain protocol information for file retrieval and information concerning the platform which the patch can be used. Such a list could look like:
Updating the binary is a very operating system dependent operation. On Windows this will likely need to be done using a separate executable or some re-launch trick and on Linux it is not reasonable to assume anything about the versions of installed libraries. On both Linux and Mac the rights of the current user will likely also be insufficient to update the software.
<pre>
win32 http://themanaworld.org/updates/tmw-base-binary-win32-1.1.zip
osx http://themanaworld.org/updates/tmw-base-binary-osx-1.1.zip
all ftp://themanaworld.org/updates/tmw-maps-tonori-1.3.zip
all http://themanaworld.org/updates/tmw-gfx-items-1.1.zip
</pre>


It is also suggested that the updates can be used in the configuration director (~/.tmw) on UNIX based platforms, as root priveleges would be required to install the updates correctly otherwise.
It is much more important to simply decouple client releases and data updates, so that client updates occur infrequently and can be handled in a convenient way on each type of operating system. This will also allow for a better compatibility, so that the game may be played with older clients until an update for a specific operating system is available. On Linux for example, using the existing package management system of the distribution is prefered, where it would be nice if we could even fit into a lengthy release cycle of for example [http://www.ubuntulinux.org/ Ubuntu].
[[User:Nym|Nym]] 23:56, 4 Jul 2005 (PDT)
:I think updating the binary should stay out of the scope of the data updating system, as it is very OS specific. The data updating system can work consistently acros all platforms, so there is no need to indicate platform in the file list. I'm not sure about sending full URLs yet, as long as we're using http/ftp I think it'd be better to set up one or two mirrors and optionally allow the user to choose where to download from.


== Update distribution ==
=== What if we versioned all the datafiles separately? ===


Updates must be distributed fast and effectively, providing a decent throughput even when the update server is under a high load. The Mana World development team has decided that the [http://www.bittorrent.com Bit Torrent] protocol will be utilised for update distribution. BitTorrent is a distributed peer-to-peer protocol which utilises the bandwidth of all users. For more information about [http://www.bittorrent.com Bit Torrent], see the [http://www.bittorrent.com/introduction.html Bit Torrent Introduction].
Too tedious and also unnecessary. When using archives, all the client needs is the list of files it should have, a way to retrieve them and the order in which they should be loaded. A file should either be loaded or it should not be loaded, and in the latter it could be removed instead. It may be better to leave the cleanup to the user though, and just provide a one-click solution.


To easily utilize to the Bit Torrent protocol, one of the following libraries can be used:
=== What about downloading updates dependent on the client version? ===
* [http://libbt.sourceforge.net/ libbt] - C
* [http://libtorrent.sourceforge.net/ libtorrent] - C++


Yeah I know we already choosed for bitTorrent, anyway we shouldn't discard HTTP/FTP both because some people don't want to be forced to share their bandwidth or if they're having problems. Someone also asked for the possibility to stop sharing patches (in case updater is built-in). A noticeable example is WoW updater which both allows for bitTorrent or HTTP downloads. And the patch sharing is limited to the time you download the patch (until you don't press "Play game!" I guess). A deeper evaluation of band usage and lag problems should be provided.--[[User:ElvenProgrammer|ElvenProgrammer]] 02:08, 4 Jul 2005 (PDT)
A server will only assume a minimum client version (which will be checked), and ideally the game data that is downloaded is independent of this. The client version also shouldn't be used to determine whether certain updates are outdated, since contrary to what we're doing now, the idea is to stop distributing most of the game data with the client in the future.


Because both libbt and libtorrent still seem immature and have annoying dependencies, I would suggest to hold off from BitTorrent for now and use [http://curl.haxx.se/ libcurl] to retrieve the updated data files through HTTP or FTP. --[[User:Bjørn|Bjørn]] 12:06, 4 Jul 2005 (PDT)
=== What about including the protocol in the list of required updates? ===


:I agree, libcurl is at least used by libbt so if we're going to use it, it will be required anyway. I'd like also to say that libcurl it's really easy and could help us to create the HTTP/FTP updater in no time.--[[User:ElvenProgrammer|ElvenProgrammer]] 12:55, 4 Jul 2005 (PDT)
It is preferred that the protocol used to transfer the files is defined elsewhere, and it could even be a client option (for example mirror selection, or choosing downloading by torrent).

Latest revision as of 07:04, 11 March 2015

This article is about the technical aspects of the updating system. See how to release an update for the steps currently required to release an update.

Introduction

The Mana World is a constantly expanding game, with new features, maps and other game media being added all the time. This constant addition requires some way to distribute new material. Our update system has been designed with the following requirements:

  • Low data redundancy - Files can be updated individually while not causing re-downloading of other data.
  • Easy and independent of release cycle - Allowing updates to be released fast and frequently.
  • Compatible - Updates can also keep older clients up to date, allowing them to keep playing without needing to update. Provided that any of the new data doesn't need new client features.
  • Multiple worlds - The updates that the client downloads are server dependent, so different worlds can be run on different servers. (Not implemented yet, 0.1.0 feature)

Update process

Updates come in compressed archives (.zip files). They are easy to create and allow individual files to be read directly from them (we use PhysFS for this). Initially, the client will retrieve an ordered list of archives that it should load, for example:

tmw-base-1.0.zip
tmw-base-patch-1.1.zip
tmw-tonori-region-1.0.zip

The client will check whether all the listed files are present in its data directory, and attempt to download any missing ones. Once downloaded, it will put each archive in the virtual file system provided by PhysFS. The archives will be searched bottom to top so that any file will be loaded from the last archive that provided it. This allows us to patch big data archives with a small archive that contains just the modified files.

Any files present in the data directory that are not in the list can be considered to be old, and could be cleaned up by the user. Updates will be downloaded to a directory writable by the user. On Windows it is assumed the user has write access to ./updates while on UNIX-like systems ~/.tmw/updates is used.

I think updates should be downloaded to subdirectories depending on the server, and that as little default data should come with the client as possible. An example directory used to store the server data would be ~/.tmw/data/testing.themanaworld.org. --Bjørn 20:39, 24 December 2005 (CET)
About the update window and news window: As the users really like to KNOW what is downloaded to their computer, (and that something for the game is downloaded btw), the news window should be one thing, and then an update button should appear if there are updates that are to be downloaded, and are not downloaded already.
Clicking on it would then show another window (that one listing the available updates) and with three buttons: Close, Update (if there is a selection in the list) and Update all (if there are still updates). And a scroll bar to show the progresses made. --Bertram
It would be a good idea not to start downloading immediately, but I think the news and update windows should remain integrated. --Bjørn 23:14, 18 September 2006 (CEST)

Update packages based on data structure

The data directory present in Subversion is no longer entirely included in releases. The following listing describes which parts are only released via dynamic updates, and to which subgroup they belong.

equipment.xml                   database
items.xml                       database
monsters.xml                    database
graphics/images/minimap_*.png   minimaps
graphics/images/ambient/*.png   ambient
graphics/items/                 items
graphics/particles/*            particles
graphics/sprites/*              sprites
maps/                           maps
sfx/                            sounds

The naming convention is <subgroup>-<revision>.zip for a package containing all the data at a certain Subversion revision, and update-<start>-<end>.zip for a package containing the material that changed or got added between the start revision and the end revision.

Future update distribution

Updates must be distributed fast and effectively, providing a decent throughput even when the update server is under a high load. The Mana World development team has decided that the BitTorrent protocol will be utilised for update distribution. BitTorrent is a distributed peer-to-peer protocol which utilises the bandwidth of all users. For more information about BitTorrent, see the BitTorrent Introduction.

To easily utilize to the BitTorrent protocol, one of the following libraries can be used:

Yeah I know we already chosen for BitTorrent, anyway we shouldn't discard HTTP/FTP both because some people don't want to be forced to share their bandwidth or if they're having problems. Someone also asked for the possibility to stop sharing patches (in case updater is built-in). A noticeable example is WoW updater which both allows for bitTorrent or HTTP downloads. And the patch sharing is limited to the time you download the patch (until you don't press "Play game!" I guess). A deeper evaluation of band usage and lag problems should be provided. --ElvenProgrammer 02:08, 4 Jul 2005 (PDT)

Because both libbt and libtorrent still seem immature and have annoying dependencies, I would suggest to hold off from BitTorrent for now and use libcurl to retrieve the updated data files through HTTP or FTP. --Bjørn 12:06, 4 Jul 2005 (PDT)

I agree, libcurl is at least used by libbt so if we're going to use it, it will be required anyway. I'd like also to say that libcurl it's really easy and could help us to create the HTTP/FTP updater in no time.--ElvenProgrammer 12:55, 4 Jul 2005 (PDT)
I've added libTorrent which seems like an interesting choice because it depends on just libsigc++ and libcurl. I still think we're fine with just libcurl for now though, the BitTorrent distribution would be an idea for later. --Bjørn 01:43, 20 Sep 2005 (CEST)
"Q: Is anyone got it work under windows? (maybe any plan on porting?) A: Haven't heard of anyone doing so and i don't really care about the win32 platform and its lack of posix support. Unless someone pays me it'll propably not happen. ;)"
Just to let you notice windows is not supported. --ElvenProgrammer 08:39, 20 Sep 2005 (CEST)

Questions

What about updating the binary?

Updating the binary is a very operating system dependent operation. On Windows this will likely need to be done using a separate executable or some re-launch trick and on Linux it is not reasonable to assume anything about the versions of installed libraries. On both Linux and Mac the rights of the current user will likely also be insufficient to update the software.

It is much more important to simply decouple client releases and data updates, so that client updates occur infrequently and can be handled in a convenient way on each type of operating system. This will also allow for a better compatibility, so that the game may be played with older clients until an update for a specific operating system is available. On Linux for example, using the existing package management system of the distribution is prefered, where it would be nice if we could even fit into a lengthy release cycle of for example Ubuntu.

What if we versioned all the datafiles separately?

Too tedious and also unnecessary. When using archives, all the client needs is the list of files it should have, a way to retrieve them and the order in which they should be loaded. A file should either be loaded or it should not be loaded, and in the latter it could be removed instead. It may be better to leave the cleanup to the user though, and just provide a one-click solution.

What about downloading updates dependent on the client version?

A server will only assume a minimum client version (which will be checked), and ideally the game data that is downloaded is independent of this. The client version also shouldn't be used to determine whether certain updates are outdated, since contrary to what we're doing now, the idea is to stop distributing most of the game data with the client in the future.

What about including the protocol in the list of required updates?

It is preferred that the protocol used to transfer the files is defined elsewhere, and it could even be a client option (for example mirror selection, or choosing downloading by torrent).