I got issues fixed to where I can (and R Soul can) create a working mission selector as an alternative to the fmsel.dll currently in use. These issues were:
1. Could not pass mission name back to the game so it would just run default missions. The fix for this was changing the data types for strings that need to be modified to IntPtr and then using Marshal.Copy to write the mission name to sName.
2. launching a common dialog (such as openfiledialog, folderbrowserdialog) or trying to enable Drag & Drop functionality would cause the process to lock and I had to use task manager to stop THIEF.EXE. After much researching and experimenting, I found this was due to the code running in a multi-threaded apartment state. The fix for this was to create a new STA thread and launch the UI in that. Now common dialogs work and Drag & Drop works.
So here's the new, working code for your coding pleasure (whoever wants to take a stab at creating an alternative fmsel.dll. I'll get it up on GitHub in the next day or so and post the link.
Code:
Imports System.Runtime.InteropServices
Imports RGiesecke.DllExport
Imports System.Text.Encoding
Imports System.IO
Public Module fmsel
Private FMSelData As sFMSelectorData
Public Enum eFMSelReturn
kSelFMRet_OK = 0 ' run selected FM 'data->sName' (0-len string to run without an FM)
kSelFMRet_Cancel = -1 ' cancel FM selection And start game As-Is (no FM Or If defined In cam_mod.ini use that)
kSelFMRet_ExitGame = 1 ' abort And quit game
End Enum
<StructLayout(LayoutKind.Sequential, Pack:=4, CharSet:=CharSet.Ansi)>
Public Structure sFMSelectorData
Public nStructSize As Integer
Public sGameVersion As String
' supplied initial FM root path (the FM Selector may change this)
Public sRootPath As IntPtr
Public nMaxRootLen As Integer
' buffer to copy the selected FM name
Public sName As IntPtr
Public nMaxNameLen As Integer
' set to non-zero when selector Is invoked after game exit (if requested during game start)
Public bExitedGame As Integer
' FM selector should set this to non-zero if it wants to be invoked after game exits (only done for FMs)
Public bRunAfterGame As Integer
' optional list of paths to exclude from mod_path/uber_mod_path in + separated format And Like the config
' vars, Or if "*" all Mod paths are excluded (leave buffer empty for no excludes)
' the specified exclude paths work as if they had a "*\" wildcard prefix
Public sModExcludePaths As IntPtr
Public nMaxModExcludeLen As Integer
Public sLanguage As IntPtr
Public nLanguageLen As Integer
Public bForceLanguage As Integer
End Structure
<DllExport(CallingConvention:=CallingConvention.Cdecl, ExportName:="SelectFM")>
Public Function SelectFM(ByRef data As sFMSelectorData) As Int32
FMSelData = data
' Our form must be opened in a separate thread set to Single-Threaded Apartment state.
' this is necessary for common dialogs and DragDrop operations to function.
Dim thd As New Threading.Thread(Sub()
With New frmMain
.ShowDialog()
End With
End Sub)
thd.SetApartmentState(Threading.ApartmentState.STA)
thd.Start()
thd.Join() 'blocks the calling thread until thd exits.
'assume user chose "AThiefsHoliday" from the UI
Dim sName As Byte() = ASCII.GetBytes("AThiefsHoliday") 'folder name of mission to play
Marshal.Copy(sName, 0, FMSelData.sName, sName.Length)
Return FMSelReturnValue
End Function