[Dev Thread] Enhanced Thief Fan Missions Archive (Last update: 2017/12/24) - by EnYB0La
FenPhoenix on 15/12/2017 at 08:30
Sure, could do that. It'd be something a bit more of a fun challenge than "does this folder exist". :P
EnYB0La on 15/12/2017 at 15:03
Quote Posted by FenPhoenix
Started work today. Getting the basics set up took a while. Maybe I should have something to eat. In celebration. Anyway, everything is quick-n-dirty right now and it doesn't pull much data at the moment, but here's the git repo. I'll be updating this regularly.
(
https://github.com/FenPhoenix/FMInfoGen)
Edit:For clarification on some things:
I notice the "Books and scrolls",
is that to note that a mission has readables? I'm not sure I've ever seen one that didn't. Also, does "Background music" mean actual music only, or any custom ambience, or? And what's "Goal description"?
Those YAML keys...
Code:
details:
characteristics:
hascustommotions: false
hascustomcreatures: false
hascustomobjects: false
hascustomsounds: false
hascustomtextures: false
hascustomscripts: false
hascustomgamesys: false
hasbgmusic: false
hasmovies: false
hasmap: false
hasautomap: false
hasbooksandscrolls: false
hasgoaldescription: false
...are based on the DarkLoader content checker:
Inline Image:
https://image.ibb.co/fy7Qem/Dark_Loader_v4_3_2_quests_available.jpgAs you can see in this image (below) (
main.pas from DarkLoader source code, line 1118), is that the app searches for available files in specific folders:
(
https://image.ibb.co/f2x0em/main_pas_Source_Visual_Studio_Code.jpg)
Inline Image:
https://preview.ibb.co/niB45R/main_pas_Source_Visual_Studio_Code.jpgBut, yeah, this checker should be more complex and complete. Do you need help or more details on how we should resolve this issue?
FenPhoenix on 15/12/2017 at 19:16
Oh, whoops, I didn't realize (and/or remember) that DarkLoader had all those fields. Guess I never used them. :(
I looked at the DL code to try to find out how it was detecting Thief Gold FMs from Thief 2 FMs (which is broken with NewDark anyway) but I didn't think to look for anything else. I'm not familiar with Object Pascal or whatever it is, but I'll have another look. Thanks for pointing that out to me.
I was wondering exactly what the difference between the "obj" and "mesh" folders were. It looks like the code is saying mesh is "creatures" while obj is just "objects". I'm not a Dromeder, so am I right in thinking then that "mesh" is AI and "obj" is non-AI objects?
I'm also thinking, some of this stuff exists already on fan mission sites. Stuff like author, version (on thiefmissions), initial release date / last update date (also thiefmissions), forum links, and which game it's for. That last one would be extremely helpful to get from a fan mission site, because otherwise I'd have to detect it like DL, and that's broken with NewDark as mentioned. But I guess if it's reliable then it would work for OldDark missions which are the bulk of releases. But I could also just parse the HTML for Taffers Paradise for example, and grab it from there (as a one-time thing to get all current mission entries generated).
EDIT: Or I could just detect T1/T2 by folder, because if you've got a mission you probably have it sorted that way already...
I'm looking through the code, and it looks like this section that's detecting game type?
Code:
missionnames.sort;
ttype:=darkGameThief;
if missionnames.count<>0 then
begin
ttype:=darkini.readinteger(section,'type',0);
if not (ttype in [darkGameThief,darkGameThief2,darkGameT2x]) or Refresh then
begin
ii:=DBSearch(zip.ExtractFileToStream(missionnames[0]), ['SKYOBJVAR','MAPPARAM']);
if ii=1 then
begin
{Thief2 or T2x?}
ttype:=darkGameThief2;
for ii := 0 to (zip.Count-1) do
if AnsiPos('t2x\', AnsiLowercaseFilename(zip.DirEntry[ii]^.Filename))=1 then
begin
ttype:=darkGameT2x;
break;
end;
end else if ii=2 then
begin
ttype := darkGameSS2;
end else
ttype := darkGameThief;
darkini.writeinteger(section,'type',ttype);
end;
end;
selmistype:=ttype;
if ttype<>darkGameSS2 then
begin
x:=1;
for ii:=0 to (missionnames.count-1) do
if (ansipos('miss',AnsiLowercaseFilename(ExtractFilename(missionnames[ii])))=1) then
begin
missionnums[x]:=StrToIntDef(RemSubStr('miss',AnsiLowercaseFilename(ExtractBasename(missionnames[ii]))),0);
if missionnums[x]<>0 then
begin
Inc(x);
if x>99 then break;
end;
end;
end;
It's actually looking inside a .mis file for key strings, "SKYOBJVAR" and "MAPPARAM". Interesting. Since this misdetects NewDark TG missions as T2, maybe there's another reliable way, but I don't know what it would be.
Of course, the readme can also be parsed for info, because a lot of the time (but not all the time) FMs have a standard sort of structure in the readme. Preferably we would check something more concrete, but it's an option if all else fails. We may also be able to get the description / briefing from the readme if it's in roughly the standard format that a lot of readmes are.
Anyway, I'll study the DL code some more and get back to work...
Edit:
Just mentioning out of interest, it looks like "Background music" is being detected by looking for a "song" or "songs" folder, and .snc files. Can't say I've ever seen that either. :confused:
Edit again:
I just searched through the entire collection of T1/TG FMs (as downloaded from here (
http://ladyjo1.free.fr/thief/spip.php?rubrique96) here) and didn't find a single
song folder,
songs folder, or .snc file. Haven't got the T2 missions extracted yet but it's looking like this value may not be relevant for Thief. Maybe it's a System Shock 2 thing?
Edit:
I'm thinking some of these values definitely must be for SS2, like I've never seen a "cutscenes" folder either, etc. Maybe that's what "goal descriptions" are for, too. I mean I know it's checking for a goals.str which Thief has, but Thief
must have that file as far as I know, or else there would be no objectives to complete, yeah? I don't know SS2 very well but maybe it uses that file too but it's optional...?
And yeah, DL was made before NewDark, so some of these detections are indeed not complete any more, for example detecting only .pcx files for textures, whereas I know that nowadays you can use .png and .dds at least.
Edit: I'm also thinking we needn't necessarily be so specific for certain folders. For movies and snd, for instance, detecting if those folders exist and contain files would surely be enough (how likely is there to be a movie folder with files that aren't movies in it?), and then it would cover any future new formats that end up being allowed in there. Otherwise I detect a list of possible file extensions, which seems a bit much for movies and snd folders at least.
Yandros on 15/12/2017 at 23:28
1. Yes, /mesh is for AI and /obj is for objects.
2. The music thing is looking for custom songs, which I believe are T2 only and also are very very rarely used by FM authors. I would just nix all the logic around that and remove that checkbox from the UI entirely. Many missions do have background music, but it's just normal snd files. However, you can't rely on the presence of a /snd folder to mean there is custom ambients, either, since ALL custom sounds go in there.
FenPhoenix on 15/12/2017 at 23:53
Thanks for the help!
So far, I've got:
-Title (tries to get it from titles.str, else it tries to get it from the readme)
-Author name (tries to get it from the readme)
-Version (tries to get it from the readme)
-Whether the mission has:
-Custom motions, movies, custom textures, custom objects, custom creatures, custom gamesys, custom scripts (detects either .osm files in base, or a "scripts" folder in base), a map, and an automap.
I'm currently only parsing .txt readmes, not .rtf ones, and currently treating every FM as a single mission even if it's a campaign. I'll get that stuff in later.
Regarding automaps, DL is checking for *ra.bin files in intrface/miss* folders, so that's what I did as well. Is it always and/or still true that automaps have *ra.bin filenames?
Yandros on 16/12/2017 at 01:50
I think so, yes, they require specially-named .bin files in the mission's folder inside intrface. Someone more knowledgeable on the subject like R Soul can confirm.
john9818a on 16/12/2017 at 18:22
I don't know if this is relevant to the discussion, but T1/TG use the Gen script whereas T2 uses both the Gen and Convict scripts.
EnYB0La on 16/12/2017 at 20:08
Reading FenPhoenix and Yandros... I was thinking if this part is really necessary?
Inline Image:
https://image.ibb.co/fukHc6/ETFMA_20171216_1629_ai.jpgShould we really incorporate this kind of details? Is this relevant to the user/gamer?
Quote Posted by FenPhoenix
I'm currently only parsing .txt readmes, not .rtf ones.
Are we sure there is a README file in each FM Zip? How do you retrieve a value from it? Are you searching for a specific string?
By the way, nice work Fen :)
FenPhoenix on 16/12/2017 at 22:10
I'd say right off the bat to get rid of "Goal description", "Books and scrolls" (because both of those will almost certainly be present in any case), and "Background music" (see (
http://www.ttlg.com/forums/showthread.php?t=147890&p=2379121&viewfull=1#post2379121) Yandros' post).
As for the rest, for me personally, I'd say "Custom gamesys" doesn't tell me anything. I only vaguely know what a gamesys is, and I don't know what specific implications a custom one would have. If we know of any specific implications that matter to an average player, maybe we could extract those out and put those in the list instead; otherwise, maybe get rid of that one too. If a custom gamesys would make a mission visibly "non-stock" and if we want to denote that, then maybe we can use it.
On that topic, I think there may be some argument for keeping custom scripts, textures, sounds, objects, creatures, and motions if we want to denote a mission being "stock-resource" or not (and I don't know if we do, but some people seem to like that maybe?).
Map, automap, and movies seem like reasonable information to me. Not absolutely necessary perhaps, but sort of nice to know just to give you a taste of what the playing experience might be like.
Quote Posted by EnYB0La
Are we sure there is a README file in each FM Zip? How do you retrieve a value from it? Are you searching for a specific string?
By the way, nice work Fen :)
Thanks!
And there may not be a readme. And if there is, it may not be formatted such that it can feasibly be parsed for info. But often times it will be, for example a lot of readmes have a section like this:
Code:
* Playing Information *
Game : Thief: The Dark Project
Mission Title : Lady Whitman's Disease
File Name : miss15.mis
Difficulty Settings : Yes
Equipment Store : No
Map : Yes
Auto Map : No
New Graphics : No
New Sounds : No
Multi-Language Support : Yes
Briefing : No
Length : n/a
Size : n/a
Difficulty Level Info : (Normal, Hard and Expert)
* Construction *
Base : No
Build Time : Two months
So I can look for a line like "Title (some amount of spaces) : (some value)". But I check all other places first, before going for the readme. Parsing the readme is just if I can't find info anywhere else, and if I can't parse the readme then oh well, we just have to put that info in manually.
Incidentally, out of the 378 TDP missions I've tested this with so far, currently only 14 are failing the title detect (and mostly because they're either French versions, or they're campaigns and have multiple mission titles and the code doesn't deal with that yet), and 64 are failing the author detect (and I haven't even got to work tuning the author check yet).
No doubt a few missions are going to have to have certain fields set manually, but the goal is minimize that work.
Quote Posted by john9818a
I don't know if this is relevant to the discussion, but T1/TG use the Gen script whereas T2 uses both the Gen and Convict scripts.
Hmm! I just did a search inside the .mis files from four different missions:
The Death of Garrett (Thief 1, OldDark)
Endless Rain (Thief Gold, NewDark)
Broken Triad (Thief 2, OldDark)
Death's Cold Embrace (Mission 1) (Thief 2, NewDark)
All of them contained the string "convict" but only the Thief 2 .mis files contained the string "gen". So what you said, but the other way around. I'm gonna do some quick scanning to see if this is consistent. Thanks for the info!
EDIT: Hmm, nope, not consistent. A bunch of these T1/G files contain "gen". Meanwhile I can try to get this info from the readme, that should cover a majority of missions anyway.
