From The Mana World
Archives
Archives of outdated information from this talk page are here:
Problems
itemdbtowiki does not take enough arguments
In the future, itemdbtowiki should be modified to take some extra arguments from the command line, such as 'image_path', 'checkimagesonwiki', and perhaps the username and password.
[FIXED] Cannot run script
Output:
Item-ID -1: Cannot convert integer attribute type to an integer. Value: 'hairsprite'
.
.
.
Item-ID 1213: Cannot convert integer attribute type to an integer. Value: 'generic'
Traceback (most recent call last):
File "./itemdbtowiki.py", line 531, in <module>
citems = parsexmlitems(f);
File "./itemdbtowiki.py", line 252, in parsexmlitems
items[curitem['id']] = curitem
KeyError: 'id'
I have tried to understand the script so I can solve it myself, but I do not really understand the logic of what the code is trying to do or the logic of python’s error message. It seems I need to learn more scripting and python as well. ✎ Kess☽ 10:48, 7 November 2008 (CET)
- I have solved this part, just needed to take the time to understand the script and update/fix the relevant parts. I will update the item pages in a few days time when I have ironed out some ‘small’ issues like dyed items and rarity. At that time I think I’ll make another archive of this talk page as well. ✎ Kess☽ 19:50, 10 November 2008 (CET)
- I do not have this time currently. The latest state of the script is available here: User talk:Kess/Sandbox, with output as shown here: User:Kess/Sandbox. Feel free to use it or skip it. I might get back to this next year. Depends on where my priorities will lie. ✎ Kess☽ 15:26, 18 November 2008 (CET)
How to create this page
I have completed the new new script to recreate this page, please leave me a note if you find any problems. It is now capable of uploading new images to the wiki! --Hype0 02:42, 3 September 2007 (CEST)
You can run itemdbtowiki to generate the wiki page. The usage is:
itemdbtowiki <item_db.txt> <items.xml> > wikipage.txt
If run without any arguments, it'll look for item_db.txt and items.xml in the current directory.
To upload new images to the wiki automatically, 'checkimagesonwiki' in the file to 1 and set a corresponding valid 'wikiuser' and 'wikipassword' as well as set the 'image_path' to be correct (these are all parameters at the start of the script.)
Warning: if you choose to upload images, authentication will be done in plaintext, which is exactly what happens if you log in over http:// anyway... just thought you might like a reminder.
Python-Tools
itemdbtowiki
#!/usr/bin/python
#Licensed under GNU General Public License
import sys, os, popen2;
import xml.parsers.expat
debug=0
checkimagesonwiki=0
#wikiuser="Hype0"
#wikipassword="password"
image_path="/usr/home/dex/src/themanaworld/tmw/trunk/data/graphics/items"
health_titles = ["Image","Name","ID","HP Bonus","SP Bonus",
"Price<BR />BUY/Sell","Weight","Description"]
status_titles = ["Image","Name","ID","Spell name","Parameter 1","Parameter 2",
"Price<BR />BUY/Sell","Weight","Description"]
weapon_titles = [" Image","Name","ID","Damage<BR />(Range)",
"Price<BR />BUY/Sell","Weight","Description"]
armor_titles = ["Image", "Name", "ID", "Defense",
"Price<BR />BUY/Sell", "Weight","Description"]
other_titles = ["Image","Name","ID","Price<BR />BUY/Sell",
"Weight","Description "]
imageurls = [
# The index represents the imageID from the items.xml of the client.
"", # 0
"http://img52.imageshack.us/img52/3072/423xc.png", # 42
"http://img187.imageshack.us/img187/2963/434dc.png", # 43
"http://img125.imageshack.us/img125/4331/442fm.png", # 44
"http://img260.imageshack.us/img260/6627/456rj.png", # 45
"http://img211.imageshack.us/img211/9089/469fy.png", # 46
"http://img219.imageshack.us/img219/1188/479yz.png", # 47
"http://img232.imageshack.us/img232/3517/489oa.png", # 48
"http://img291.imageshack.us/img291/8643/491eg.png", # 49
"http://img157.imageshack.us/img157/1425/502hg.png", # 50
"http://img293.imageshack.us/img293/4524/514dn.png", # 51
"http://img42.imageshack.us/img42/7271/523hc.png", # 52
"http://img187.imageshack.us/img187/5241/535fk.png", # 53
"http://img125.imageshack.us/img125/8093/543yr.png", # 54
"http://img260.imageshack.us/img260/1315/554bt.png", # 55
"http://img211.imageshack.us/img211/9687/564wu.png", # 56
"http://img219.imageshack.us/img219/6842/571wz.png", # 57
"http://img232.imageshack.us/img232/9236/582ue.png", # 58
"http://img291.imageshack.us/img291/435/598ni.png", # 59
"http://img157.imageshack.us/img157/1323/605pt.png", # 60
"http://img293.imageshack.us/img293/1371/618fu.png", # 61
"http://img42.imageshack.us/img42/69/625mj.png", # 62
"http://img187.imageshack.us/img187/6240/637ju.png", # 63
"http://img275.imageshack.us/img275/9440/648zq.png", # 64
]
imagesused = {}
class whatever: pass
log = []
# parseitems(file)
## Returns list with items from eathena item_db file.
def saveint(string, altval = 0):
a = 0
try:
a = int(string)
except:
a = altval
return a
def parsescript(s):
# Assumes that there's only one call of each method, otherwise it would need to know
# how to combine those function calls. In practice, the latter call would prevail.
script = {}
scriptentry = ""
parentry = ""
mode = 0
for a in s:
if mode == 0: # looking for method
if a.isalpha():
mode = 1
scriptentry += a
elif a == '}': mode = 9
elif mode == 1: # reading method name
if a in " ;}":
if a == " ": mode = 2
elif a == ";": mode = 1
elif a == "}": mode = 9
parentry = ""
script[scriptentry] = []
else: scriptentry += a
elif mode == 2: #looking for param
if a == " ": pass
elif a == ";":
mode = 0
scriptentry = ""
else:
mode = 3
parentry = a
elif mode == 3: #reading param
if (a == " ") or (a == ",") or (a == ";"):
script[scriptentry].append(parentry)
parentry = ""
if (a == ';'):
mode = 0
scriptentry = ""
else: mode = 2
else:
parentry += a
elif mode == 9: #finished
pass
# Convert all possible parameters to integers
for i in script.keys():
for j in range(len(script[i])):
try:
script[i][j] = int(script[i][j])
except:
#print script[i][j]
pass
return script
def parseitems(file):
objects = []
for line in file:
s = line[0:line.find('//')].strip().replace('\t','')
if s:
#Replace commas inbetween {} with |, so we can use split
mode = 0
sout = ""
for a in s:
if mode == 0: #Out of {}
if a == '{': mode = 1
sout += a
elif mode == 1: #Inside {}
if a == ',': sout += '|'
else:
sout += a
if a == '}': mode = 0
values = sout.split(',')
if line[0] == '#':
if debug:
log.append("FOUND COMMENT LINE: %s" % str(values))
continue
if (len(values) != 20):
log.append("item_db: Warning, item-line with ID %s has %d values instead of 19" % (values[0], len(values)))
if debug:
log.append(" line was %s" % str(values))
while (len(values) < 20):
values.append('')
while (len(values) > 20):
values.pop()
o = whatever()
o.id = int(values[0])
o.name = values[1]
o.jname = values[2]
o.type = saveint(values[3])
o.price = saveint(values[4])
o.sell = saveint(values[5])
o.weight = saveint(values[6])
o.atk = saveint(values[7])
o.defense = saveint(values[8])
o.range = saveint(values[9])
o.mbonus = saveint(values[10])
o.slot = saveint(values[11],-1)
o.job = saveint(values[12],-1)
o.gender = saveint(values[13],-1)
o.loc = saveint(values[14],-1)
o.wlv = saveint(values[15])
o.elv = saveint(values[16])
o.view = saveint(values[17],-1)
o.usescript = parsescript(values[18].replace('|',','))
o.equipscript = parsescript(values[19].replace('|',','))
objects.append(o)
return objects
# parsexmlitems(file)
## Creates a dictionary containing the values of a client items.xml
## Yeah, there are XML parsers in the standard python libraries, but they're too object
## oriented and thus don't fit the style of this program.
def parsexmlitems(file):
items = {}
pre = "<item "
term = "/>"
attrs = ["id", "image", "art", "name", "description", "type", "weight", "slot"]
intattrs = ["id", "art", "type", "weight", "slot"]
s = file.read()
index = 0
debug = 0
while s[index:].find(pre) >= 0:
index += s[index:].find(pre) + len(pre)
curitem = {}
termstart = index + s[index:].find(term) + len(term)
for attr in attrs:
found = s[index:].find(attr+'="')
if found >= 0:
start = index + found + len(attr+'="')
end= start + s[start:].find('"')
else:
start = termstart
if (start < termstart):
curitem[attr] = s[start:end]
for a in intattrs:
try:
if curitem.has_key(a): curitem[a] = int(curitem[a])
except:
log.append("Item-ID %s: Cannot convert integer attribute %s to an integer. Value: '%s'" % (curitem["id"], a, curitem[a]))
items[curitem['id']] = curitem
return items
# addclientinformation(items, citems)
## Entends the item data with the data collected from the client items.xml. Adding imageurls,
## client-name and -description
def addclientinformation(items,citems):
global imageurls
global imagesused
for i in items:
if citems.has_key(i.id):
i.imagename=citems[i.id]["image"]
url=i.imagename[0].upper() + i.imagename[1:]
i.imgurl = "[[Image:"+ url + "]]"
imagesused[ citems[i.id]["image"]] = 1
i.description = citems[i.id]["description"]
i.clientname = citems[i.id]["name"]
else:
i.imgurl = ''
i.description = ''
i.clientname = ''
# gettypedir (items)
## Returns sorted lists of items by itemtype
def gettypedir(items):
items.sort(key=lambda x: x.price+x.sell)
typedir = whatever()
typedir.healthy = []
typedir.status = []
typedir.inspiring = []
typedir.weapons = []
typedir.combos = []
typedir.armor = []
typedir.other = []
for item in items:
if (item.imgurl.strip() or item.clientname.strip()):
#if item.id == 537: log.append('"%s", "%s"' % (item.imgurl, item.name)
if (item.atk > 0):
if item.defense == 0: typedir.weapons.append(item)
else: typedir.combos.append(item)
elif (item.defense > 0): typedir.armor.append(item)
elif item.usescript.has_key("itemheal"):
if item.usescript["itemheal"][0] > item.usescript["itemheal"][1]:
typedir.healthy.append(item)
else:
typedir.inspiring.append(item)
elif item.usescript.has_key("sc_start"):
typedir.status.append(item)
else: typedir.other.append(item)
typedir.weapons.sort(key=lambda x: x.atk)
typedir.armor.sort(key=lambda x: x.defense)
typedir.combos.sort(key=lambda x: x.defense+x.atk)
typedir.healthy.sort(key=lambda x: int(x.usescript["itemheal"][0]))
typedir.inspiring.sort(key=lambda x: int(x.usescript["itemheal"][1]))
#typedir.other.sort(key=lambda x: x.price+x.sell)
typedir.status.sort(key=lambda x: x.price+x.sell)
return typedir
# printlog()
## Prints the global variable log to stdout
def printlog():
global log
if len(log) > 0:
sys.stdout.write('\n---------------------------------------\n')
for line in log:
sys.stdout.write(line+'\n')
# print<>items(items, title)
## Creates the table in wikicode, depending on what kind of item is being printed
def getmoneystring(buy, sell):
return '| align="right" | %d GP<br>%d gp\n' % (buy,sell)
def getidstring(id):
return '| align="center" | [%d]\n' % id
def printhealitems(items,title):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
# Print labels
for title in health_titles:
sys.stdout.write('! style="background:#efdead;" | %s\n' % title)
for i in items:
sys.stdout.write('|-\n')
sys.stdout.write('| align="center" | %s\n' % i.imgurl)
#sys.stdout.write('| %s\n' % i.jname.replace('_',' '))
sys.stdout.write('| %s\n' % i.clientname)
sys.stdout.write( getidstring(i.id) )
sys.stdout.write('| align="center" | %d\n' % i.usescript["itemheal"][0])
sys.stdout.write('| align="center" | %d\n' % i.usescript["itemheal"][1])
sys.stdout.write( getmoneystring(i.price,i.sell) )
sys.stdout.write('| align="center!" | %d\n' % i.weight)
sys.stdout.write('| %s\n' % i.description)
sys.stdout.write('|}\n\n')
def printstatusitems(items,title):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
# Print labels
for title in status_titles:
sys.stdout.write('! style="background:#efdead;" | %s\n' % title)
for i in items:
sys.stdout.write('|-\n')
sys.stdout.write('| align="center" | %s\n' % i.imgurl)
#sys.stdout.write('| %s\n' % i.jname.replace('_',' '))
sys.stdout.write('| %s\n' % i.clientname)
sys.stdout.write( getidstring(i.id) )
sys.stdout.write('| align="center" | %s\n' % i.usescript["sc_start"][0])
sys.stdout.write('| align="center" | %s\n' % i.usescript["sc_start"][1])
sys.stdout.write('| align="center" | %s\n' % i.usescript["sc_start"][2])
sys.stdout.write( getmoneystring(i.price,i.sell) )
sys.stdout.write('| align="center!" | %d\n' % i.weight)
sys.stdout.write('| %s\n' % i.description)
sys.stdout.write('|}\n\n')
def printweaponitems(items, title):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
# Print labels
for title in weapon_titles:
sys.stdout.write('! style="background:#efdead;" | %s\n' % title)
for i in items:
sys.stdout.write('|-\n')
sys.stdout.write('| align="center" | %s\n' % i.imgurl)
#sys.stdout.write('| %s\n' % i.jname.replace('_',' '))
sys.stdout.write('| %s\n' % i.clientname)
sys.stdout.write( getidstring(i.id) )
sys.stdout.write('| align="center" | %d (%d)\n' % (i.atk,i.range))
sys.stdout.write( getmoneystring(i.price,i.sell) )
sys.stdout.write('| align="center!" | %d\n' % i.weight)
sys.stdout.write('| %s\n' % i.description)
sys.stdout.write('|}\n\n')
def printarmoritems(items, title):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
sys.stdout.write('! style="background:#efdead;" | Image\n')
sys.stdout.write('! style="background:#efdead;" | Name\n')
sys.stdout.write('! style="background:#efdead;" | ID\n')
sys.stdout.write('! style="background:#efdead;" | Defense\n')
sys.stdout.write('! style="background:#efdead;" | Price<br>BUY/Sell\n')
sys.stdout.write('! style="background:#efdead;" | Description\n')
for i in items:
sys.stdout.write('|-\n')
sys.stdout.write('| align="center" | %s\n' % i.imgurl)
#sys.stdout.write('| %s\n' % i.jname.replace('_',' '))
sys.stdout.write('| %s\n' % i.clientname)
sys.stdout.write( getidstring(i.id) )
sys.stdout.write('| align="center" | %d\n' % i.defense)
sys.stdout.write( getmoneystring(i.price,i.sell) )
sys.stdout.write('| %s\n' % i.description)
sys.stdout.write('|}\n\n')
def printcomboitems(items, title):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
sys.stdout.write('! style="background:#efdead;" | Image\n')
sys.stdout.write('! style="background:#efdead;" | Name\n')
sys.stdout.write('! style="background:#efdead;" | ID\n')
sys.stdout.write('! style="background:#efdead;" | Damage<br>(Range)\n')
sys.stdout.write('! style="background:#efdead;" | Defense\n')
sys.stdout.write('! style="background:#efdead;" | Price<br>BUY/Sell\n')
sys.stdout.write('! style="background:#efdead;" | Description\n')
for i in items:
sys.stdout.write('|-\n')
sys.stdout.write('| align="center" | %s\n' % i.imgurl)
#sys.stdout.write('| %s\n' % i.jname.replace('_',' '))
sys.stdout.write('| %s\n' % i.clientname)
sys.stdout.write( getidstring(i.id) )
sys.stdout.write('| align="center" | %d (%d)\n' % (i.atk,i.range))
sys.stdout.write('| align="center" | %d\n' % i.defense)
sys.stdout.write( getmoneystring(i.price,i.sell) )
sys.stdout.write('| %s\n' % i.description)
sys.stdout.write('|}\n\n')
def getpropertystring(item):
s = ""
s += "Weight: %d, " % item.weight
s += "Slot: %d, " % item.slot
s += "Job: %d, " % item.job
s += "Gender: %d, " % item.gender
s += "Loc: %d, " % item.loc
s += "wLV: %d, " % item.wlv
s += "eLV: %d, " % item.wlv
s += "View: %d " % item.view
return s
def printotheritems(items, title):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
sys.stdout.write('! style="background:#efdead;" | Image\n')
sys.stdout.write('! style="background:#efdead;" | Name\n')
sys.stdout.write('! style="background:#efdead;" | ID\n')
#sys.stdout.write('! style="background:#efdead;" | Type\n')
#sys.stdout.write('! style="background:#efdead;" | Properties\n')
sys.stdout.write('! style="background:#efdead;" | Price<br>BUY/Sell\n')
sys.stdout.write('! style="background:#efdead;" | Description\n')
for i in items:
sys.stdout.write('|-\n')
sys.stdout.write('| align="center" | %s\n' % i.imgurl)
#sys.stdout.write('| %s\n' % i.jname.replace('_',' '))
sys.stdout.write('| %s\n' % i.clientname)
sys.stdout.write( getidstring(i.id) )
#sys.stdout.write('| align="center" | %d\n' % i.type)
#sys.stdout.write('| align="center" | %s\n' % getpropertystring(i))
sys.stdout.write( getmoneystring(i.price,i.sell) )
sys.stdout.write('| %s\n' % i.description)
sys.stdout.write('|}\n\n')
def printunuseditems(title):
global imageurls
global imagesused
ids = []
for i in range(1,len(imageurls)):
if not imagesused.has_key(i):
ids.append(i)
if len(ids):
sys.stdout.write('==%s==\n' % title)
sys.stdout.write('{| border="1" cellspacing="0" cellpadding="5" width="100%" align="center"\n')
sys.stdout.write('| ')
for i in ids:
sys.stdout.write(imageurls[i] + ' ')
sys.stdout.write('\n|}\n\n')
#####################################################################
# MAIN
#####################################################################
try:
if (len(sys.argv) == 1):
item_db = "item_db.txt"
item_xml = "items.xml"
elif (len(sys.argv) == 3):
item_db = sys.argv[1]
item_xml = sys.argv[2]
else:
item_db = ''
item_xml = ''
sys.stdout.write("Wrong number of arguments\n")
if item_db and (not os.path.isfile(item_db)):
sys.stdout.write("File does not exist: %s\n" % item_db)
item_db = ''
if item_xml and (not os.path.isfile(item_xml)):
sys.stdout.write("File does not exist: %s\n" % item_xml)
item_db = ''
if not (item_db and item_xml):
sys.stdout.write("\nUSAGE:\n")
sys.stdout.write("dbtowiki without any arguments will use item_db.txt and items.xml in the current directory.\n")
sys.stdout.write("to specify custom files, call: dbtowiki <item_db> <item_xml>\n")
exit(-1);
else:
log.append("Item-list [item_db] = %s" % item_db)
log.append("Item-list [item_xml] = %s" % item_xml)
f = open(item_db)
items = parseitems(f);
f = open(item_xml)
citems = parsexmlitems(f);
addclientinformation(items, citems)
typedir = gettypedir(items)
if len(typedir.healthy) > 0: printhealitems(typedir.healthy, "Health")
if len(typedir.status) > 0: printstatusitems(typedir.status, "Status")
if len(typedir.inspiring) > 0: printhealitems(typedir.inspiring, "Mana")
if len(typedir.weapons) > 0: printweaponitems(typedir.weapons, "Weapons")
if len(typedir.armor) > 0: printarmoritems(typedir.armor, "Armor")
if len(typedir.combos) > 0: printcomboitems(typedir.combos, "Combos")
if len(typedir.other) > 0: printotheritems(typedir.other, "Other")
if checkimagesonwiki:
sys.stdout.write("Checking for images on wiki: \n")
images = []
for item in typedir.healthy:
images.append( (item.imagename, item.description))
for item in typedir.inspiring:
images.append( (item.imagename, item.description))
for item in typedir.weapons:
images.append( (item.imagename, item.description))
for item in typedir.armor:
images.append( (item.imagename, item.description))
for item in typedir.combos:
images.append( (item.imagename, item.description))
for item in typedir.other:
images.append( (item.imagename, item.description))
need_to_upload=[]
for imagename, description in images:
fixedimagename = imagename[0].upper() + imagename[1:]
curlstring='curl "http://wiki.themanaworld.org/index.php/' \
'Image:%s"' % fixedimagename
if debug:
sys.stdout.write("CHECKING URL: %s\n" % curlstring)
p=popen2.Popen4(curlstring)
# this could hang if there is a lot of output!
returncode = p.wait() # wait for the process to finish
if debug:
sys.stdout.write("Process finished with %d\n" % returncode)
page = p.fromchild.read()
if debug >= 3:
sys.stdout.write("Read: %s\n" % page)
if page.find("No file by this name exists") != -1:
need_to_upload.append((imagename, description))
if debug:
sys.stdout.write("NEED TO UPLOAD: %s\n" %
str(need_to_upload[-1]))
if len(need_to_upload):
curlstring = 'curl -d "wpName=%s&wpPassword=%s&wpLoginattempt=Log+in" -c cookie-jar "http://wiki.themanaworld.org/index.php?title=Special:Userlogin&action=submitlogin&type=login"'% (wikiuser, wikipassword)
sys.stdout.write("Logging in as %s, your password can be seen "
"in `ps ax`, also this is using http://... "
"so be warned!\n" % wikiuser)
p=popen2.Popen4(curlstring)
# this could hang if there is a lot of output!
returncode = p.wait() # wait for the process to finish
data = p.fromchild.read()
if debug >= 1:
sys.stdout.write("Read: %s\n" % data)
if data.find("Login error") != -1:
sys.stdout.write("Login failed\n")
if debug:
sys.stdout.write("Process finished with %d\n" % returncode)
sys.stdout.write("Now it is time to upload: %s" %
str(need_to_upload))
for imagename, description in need_to_upload:
if debug:
sys.stdout.write("Uploading image: %s\n" % imagename)
curlstring='curl -b cookie-jar -F "wpUploadFile=@%s/%s" -F "filename=%s" -F "wpDestFile=%s" -F "wpUploadDescription=%s" -F "wpUpload=Upload file" "http://wiki.themanaworld.org/index.php?title=Special:Upload"' % (image_path, imagename, imagename, imagename, description)
p=popen2.Popen4(curlstring)
# this could hang if there is a lot of output!
returncode = p.wait() # wait for the process to finish
data = p.fromchild.read()
if debug >= 1:
sys.stdout.write("Read: %s\n" % data)
if debug:
sys.stdout.write("Process finished with %d\n" %
returncode)
sys.stdout.write("Uploaded: %s, %s\n" % (imagename,
description))
printunuseditems("Still unknown")
sys.stdout.write("\n\n")
finally:
printlog()
