Fileformat question

[StakFallT]

Need some advice on a fileformat. My project at the moment is a world builder but because I'm constantly adding more and more stuff to it, the import and export of world files can't keep up and stay insynch with the features. The file format is just plain xml (fortunately) , with tab indendations, etc...But either way, without an constantly updated import/export feature this leads to lots of tedious reloading of meshes, their shaders, any bump textures etc. So I figured I'd implement something on a smaller scale, object templates. But here's the problem with that though, materials and textures should also be templatable.

So, my question: when the mesh loads from a template, should the mesh file contain the full information for each texture and material, or should it point to another, external, template material/texture file? If I keep the data about the material's color values and texture's size etc inside the mesh file, it could get big, if I leave them in their own file then I just created a situation where one template requires another template... Any thoughts?

[Raine]

I think you should be templating as much as you can. A template mesh contains a template texture and so on. You might also want to be able to hard-code stuff, though, either for testing purposes or just to make things faster. So, in my opinion the xml file could either contain a ref to another xml file or the informations about the contained object (be it a texture, a particle, what nots).

[StakFallT]

Somehow I had a feeling that would be the general consensus But that's ok, just means I was on the right thought track. I'm thinking each group is prepended with <group_[groupnumber]> and then <texture> then IsTemplate = True/False.. I think I can reuse some of my custom xml parser that I have already in the code (for exporting and importing worlds) so I should have a headstart on it in a way... Thanks

[Raine]

You're welcome.. let us see some example xml, might be food for thought for a future reader... I'm curious as well

[StakFallT]

Ok here's what I've kinda fleshed out, it's somewhat raw: A little tweaking during filling it out, but now I think I'm down the to details-tweaking stuff. But here it is:

Material Template (Material1.tma) - Template MAterial (.tma)

Code:

MaterialName=Test
Ambient.R=190
Ambient.G=130
Ambient.B=0
Ambient.A=255
Diffuse.R=190
Diffuse.G=130
Diffuse.B=0
Diffuse.A=255
Specular.R=190
Specular.G=130
Specular.B=0
specular.A=255
Emissive.R=190
Emissive.G=130
Emissive.B=0
Emissive.A=255
Power=1
Opacity=

Mesh Template (Mesh1.tme) - Template MEsh (.tme)

Code:

<Mesh>
File=\media\Objects\Mesh1.tvm
MeshName=Mesh1
<Position>
PositionIsSet=False
</Position>
<Rotation>
RotationIsSet=True
Rotation.X=0
Rotation.Y=90
Rotation.Z=0
</Rotation>
<Scale>
ScaleIsSet=True
Scale.X=20
Scale.Y=20
Scale.Z=20
</Scale>
<Group_1>
<Material>
MaterialIsTemplate=True
MaterialFile=\Media\Templats\Materials\Material1.tma
</Material>
</Group_1>
<Group_99>
<Material>
MaterialIsTemplate=False
MaterialName=Test
Ambient.R=190
Ambient.G=130
Ambient.B=0
Ambient.A=255
Diffuse.R=190
Diffuse.G=130
Diffuse.B=0
Diffuse.A=255
Specular.R=190
Specular.G=130
Specular.B=0
specular.A=255
Emissive.R=190
Emissive.G=130
Emissive.B=0
Emissive.A=255
Power=1
Opacity=
</Material>
<Texture>
<Normal>
TextureIsTemplate=False
TextureFile=\Media\Textures
TextureName=TestTexture
Width=
Height=
Colorkey=
</Normal>
<bump>
...
</bump>
</Texture>
</Group>
</Mesh>

Mesh name -shouldn't- conflict, at least for me, because everytime I load them into the Mesh object array, the element index gets post-pended onto the end of the mesh name giving it as unique a name as need be. And because it's read in a loop, more or less it's just checking the tags, while it has s single variable that keeps track of what group it's on. So if it's hits <group> it knows, it's on a new group, if it's something like <texture> it knows it's a texture for the group it's already on, this way I can avoid having to place in empty blocks, like alpha texture blocks if alpha textures aren't even used for that group. It'll just flow to the next line and take the appropriate action.

I think it's damn close but I'm fearful and reluctant to implementing it because I know how these things go (just like my importing worlds and exporting worlds feature) things need to change down the line and you wind up having to rip into code to get something to work and before you know it, you got something that works but you have no idea why it's working because you have no visual in your head as to what really is actually going on... So I think much much much more thought is needed by me before I implement it, but it should give anyone a good idea

-- StakFallT

[StakFallT]

Short bit of background about my newer, but related, fileformat problem: I've been working on actually implementing the XML template stuff into my editor since my last post. I tried it by hand once, and I was up to 112 lines of code just to import a mesh template. And even then it was nowhere's even remotely close to being finished. I had the foundation laid out (Parsing a trimmed line of file to determine where the <'s were in the line of text, and it filled a texture parameter array. But that was about it. 112 lines of code roughly for that. More or less I was writing my own xml parser. Can you say "ouch"? lol. I've done my own scripting language (with nested IF's and variables and "special" variables so I have no desire to do something like that again lol. So I looked around, and found Chilkat has a nice freeware xml component. I've looked in the past over some xml stuff and it was extremely overcomplicated, things like treenodes, etc.. WTF is all I gotta say. Chilkat's seemed a bit more straightforward than what I've seen previously so I'm -trying- to use it but even that is still somewhat complicated.

The problem I'm having --
Ok here's my code in delphi, so far:

Code:

Function ExportMeshTemplate(MeshFile: String): Integer;
var
    GroupNumber: Integer;
    TempTexture: TV_Texture;
    LookupIndex: Integer;
    ContinueLookingUp: Boolean;
begin
    MainRenderWindow.xml.Tag := 'MESH';
    GroupNumber := 1;
    while GroupNumber < 201 do
        begin
        MainRenderWindow.xNode0 := MainRenderWindow.xml.NewChild('GROUP_' + IntToStr(GroupNumber),'');
        MainRenderWindow.xNode1 := MainRenderWindow.xNode0.NewChild('TEXTURE','');
        If (MainRenderWindow.MeshData[MainRenderWindow.MeshSelected].Textures.TextureName[GroupNumber] <> '') then
            begin
            LookupIndex := 1;
            ContinueLookingUp := true;
            while ContinueLookingUp = true do
                begin
                if UpperCase(MainRenderWindow.MeshData[MainRenderWindow.MeshSelected].Textures.TextureName[GroupNumber]) = UpperCase(MainRenderWindow.TextureData[LookupIndex].TextureName) then
                    begin
                    MainRenderWindow.xNode2 := MainRenderWindow.xNode1.NewChild('NORMAL', '');

                    MainRenderWindow.xNode2.AddAttribute('TextureIsTemplate','False');
                    MainRenderWindow.xNode2.AddAttribute('TextureFile','\Media\Textures\' + MainRenderWindow.TextureData[LookupIndex].FileName);
                    MainRenderWindow.xNode2.AddAttribute('Width',MainRenderWindow.TextureData[LookupIndex].Width);
                    MainRenderWindow.xNode2.AddAttribute('Height',MainRenderWindow.TextureData[LookupIndex].Height);
                    MainRenderWindow.xNode2.AddAttribute('ColorKey',MainRenderWindow.TextureData[LookupIndex].ColorKey);
                    MainRenderWindow.xNode2.AddAttribute('FilterAndStretch',MainRenderWindow.TextureData[LookupIndex].FilterAndStretch);

                    ContinueLookingUp := False;
                end
                else
                    begin
                    LookupIndex := LookupIndex + 1;
                    If LookupIndex > MaxTextures Then
                        ContinueLookingUp := False;
                end;
            end;
        end
        else if (MainRenderWindow.MeshData[MainRenderWindow.MeshSelected].BumpTextures.BumpTextureName[GroupNumber] <> '') then
            begin
            LookupIndex := 1;
            ContinueLookingUp := true;
            while ContinueLookingUp = true do
                begin
                if UpperCase(MainRenderWindow.MeshData[MainRenderWindow.MeshSelected].BumpTextures.BumpTextureName[GroupNumber]) = UpperCase(MainRenderWindow.BumpTextureData[LookupIndex].TextureName) then
                    begin
                    MainRenderWindow.xNode2 := MainRenderWindow.xNode1.NewChild('BUMP', '');

                    MainRenderWindow.xNode2.AddAttribute('TextureIsTemplate','False');
                    MainRenderWindow.xNode2.AddAttribute('TextureFile','\Media\Textures\' + MainRenderWindow.BumpTextureData[LookupIndex].FileName);
                    MainRenderWindow.xNode2.AddAttribute('Width',MainRenderWindow.BumpTextureData[LookupIndex].Width);
                    MainRenderWindow.xNode2.AddAttribute('Height',MainRenderWindow.BumpTextureData[LookupIndex].Height);
                    MainRenderWindow.xNode2.AddAttribute('ColorKey',MainRenderWindow.BumpTextureData[LookupIndex].ColorKey);
                    if MainRenderWindow.BumpTextureData[LookupIndex].FilterAndStretch = true then
                        MainRenderWindow.xNode2.AddAttribute('FilterAndStretch','True')
                    else
                        MainRenderWindow.xNode2.AddAttribute('FilterAndStretch','False');

                    ContinueLookingUp := False;
                end
                else
                    begin
                    LookupIndex := LookupIndex + 1;
                    If LookupIndex > MaxTextures Then
                        ContinueLookingUp := False;
                end;
            end;
        end;
        MainRenderWindow.xNode1 := MainRenderWindow.xNode0.NewChild('MATERIAL','');
        If (MainRenderWindow.MeshData[MainRenderWindow.MeshSelected].Materials.MaterialName[GroupNumber] <> '') then
            begin
            LookupIndex := 1;
            ContinueLookingUp := true;
            while ContinueLookingUp = true do
                begin
                if UpperCase(MainRenderWindow.MeshData[MainRenderWindow.MeshSelected].Materials.MaterialName[GroupNumber]) = UpperCase(MainRenderWindow.MaterialData[LookupIndex].MaterialName) then
                    begin
                    MainRenderWindow.xNode2.Content := 'MaterialIsTemplate=False';
                    MainRenderWindow.xNode2.Content := MainRenderWindow.xNode2.Content + 'Power=' + MainRenderWindow.MaterialData[LookupIndex].Power;
                    MainRenderWindow.xNode2.Content := MainRenderWindow.xNode2.Content + 'Opacity=' + MainRenderWindow.MaterialData[LookupIndex].Opacity;
                   
                    MainRenderWindow.xNode2 := MainRenderWindow.xNode1.NewChild('Ambient', '');
                        MainRenderWindow.xNode2.AddAttribute('R',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].AmbientColor.R));
                        MainRenderWindow.xNode2.AddAttribute('G',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].AmbientColor.G));
                        MainRenderWindow.xNode2.AddAttribute('B',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].AmbientColor.B));
                        MainRenderWindow.xNode2.AddAttribute('A',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].AmbientColor.A));
                    MainRenderWindow.xNode2 := MainRenderWindow.xNode1.NewChild('Diffuse', '');
                        MainRenderWindow.xNode2.AddAttribute('R',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].DiffuseColor.R));
                        MainRenderWindow.xNode2.AddAttribute('G',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].DiffuseColor.G));
                        MainRenderWindow.xNode2.AddAttribute('B',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].DiffuseColor.B));
                        MainRenderWindow.xNode2.AddAttribute('A',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].DiffuseColor.A));
                    MainRenderWindow.xNode2 := MainRenderWindow.xNode1.NewChild('Specular', '');
                        MainRenderWindow.xNode2.AddAttribute('R',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].SpecularColor.R));
                        MainRenderWindow.xNode2.AddAttribute('G',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].SpecularColor.G));
                        MainRenderWindow.xNode2.AddAttribute('B',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].SpecularColor.B));
                        MainRenderWindow.xNode2.AddAttribute('A',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].SpecularColor.A));
                    MainRenderWindow.xNode2 := MainRenderWindow.xNode1.NewChild('Emissive', '');
                        MainRenderWindow.xNode2.AddAttribute('R',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].EmissiveColor.R));
                        MainRenderWindow.xNode2.AddAttribute('G',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].EmissiveColor.G));
                        MainRenderWindow.xNode2.AddAttribute('B',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].EmissiveColor.B));
                        MainRenderWindow.xNode2.AddAttribute('A',FloatToStr(MainRenderWindow.MaterialData[LookupIndex].EmissiveColor.A));

                    //MainRenderWindow.xNode2.AddAttribute('TextureFile','\Media\Textures\' + MainRenderWindow.TextureData[LookupIndex].FileName);
                    //MainRenderWindow.xNode2.AddAttribute('Width',MainRenderWindow.TextureData[LookupIndex].Width);
                    //MainRenderWindow.xNode2.AddAttribute('Height',MainRenderWindow.TextureData[LookupIndex].Height);
                    //MainRenderWindow.xNode2.AddAttribute('ColorKey',MainRenderWindow.TextureData[LookupIndex].ColorKey);
                    //MainRenderWindow.xNode2.AddAttribute('FilterAndStretch',MainRenderWindow.TextureData[LookupIndex].FilterAndStretch);

                    ContinueLookingUp := False;
                end
                else
                    begin
                    LookupIndex := LookupIndex + 1;
                    If LookupIndex > MaxMaterials Then
                        ContinueLookingUp := False;
                end;
            end;
        end;
        GroupNumber := GroupNumber + 1;
    end;
    MainRenderWindow.xml.SaveXml('Mesh2.tme');
end;

Here's what I get as a template file:

Code:

<?xml version="1.0" encoding="utf-8" ?>

<MESH>
    <GROUP_1>
        <TEXTURE>
            <NORMAL TextureIsTemplate="False" TextureFile="\Media\Textures\C:\Program Files\Borland\Delphi7\Projects\Council of Eight Game Client Revised (v2)\Media\Objects\Grass2.bmp" Width="Grass2.bmp" Height="756" ColorKey="" FilterAndStretch="False">MaterialIsTemplate=FalsePower=1Opacity=0</NORMAL>
        </TEXTURE>
        <MATERIAL>
            <Ambient R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
            <Diffuse R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
            <Specular R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
            <Emissive R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
        </MATERIAL>
    </GROUP_1>
    <GROUP_2>
        <TEXTURE>
            <NORMAL TextureIsTemplate="False" TextureFile="\Media\Textures\C:\Program Files\Borland\Delphi7\Projects\Council of Eight Game Client Revised (v2)\Media\Objects\Grass2.bmp" Width="Grass2.bmp" Height="756" ColorKey="" FilterAndStretch="False">MaterialIsTemplate=FalsePower=1Opacity=0</NORMAL>
        </TEXTURE>
        <MATERIAL>
            <Ambient R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
            <Diffuse R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
            <Specular R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
            <Emissive R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
        </MATERIAL>
    </GROUP_2>
    <GROUP_3>

...yadda,yadda,yadda..  -- Fairly technical term lol

The problem:
I want something along the lines of:

Code:

<Mesh>
File=\media\Objects\Mesh1.tvm
MeshName=Mesh1
LoadMaterials=False
LoadTextures=True
LoadShaders=True
MeshCastsShadows=True
AdditiveShadow=True
<Position>
...
<Texture>
<Normal>
TextureIsTemplate=False
TextureFile=\Media\Textures
TextureName=TestTexture
Width=
Height=
Colorkey=
FilterAndStretch=False
</Normal>
<bump>
...
</bump>
</Texture>

See how the parameters are not enclosed in <>'s? The closest I can get is a thing called attributes, but they enclose all attributes in a single set of <> and even if I could get each attribute in it's own set of <>'s (Without the prefix and postfix tags) it's still pretty ugly. A site on the net that I found this component at had an example that when completed, output a file that looks like this:

Code:

//  <product>
//     <properties>
//         <price SellPrice="11.99"></price>
//         <price BasePrice="11.99"></price>
//     </properties>
// </product>

See how the attribute SellPrice and BasePrice are not only enclosed in the <> but also have the price xml-tag prefixed and postfixed to it? That's not what I want. I'm looking for as if it were just:

Code:

<product>
    <properties>
        SellPrice="11.99"
        BasePrice="11.99"
    </properties>
</product>

The component has a provision for adding "content" which seems to be what I'm looking for, but it seems to add it after it's finished the attributes, or it jumbles it around or something (Kinda hard to figure out it's pattern). So it would seem using the content method is sort of a hacked work-around. I'm wondering if there is a correct way of doing it, so I can add "content" anywhere...

-- StakFallT

[Zaknafein]

It sounds to me like you don't want to use XML. There are alternatives!

http://www.ikayzo.org/confluence/display/SDL/Home
http://www.yaml.org/
http://www.json.org/

I like SDL a lot and I'm using it as the readable file format for all my data in Fez. Problem is it only has a C# and Java implementation right now so for you in Delphi it might not be the best time usage to write your own parser...

Anyhow I don't recommend embedding all your stuff as text-nodes ("content"), you'll lose a lot of XML goodies (like XPATH, DOM, SAX...) if you do that since your content doesn't follow the usual tag structure. It's like putting every variable in your class as a String and parsing the strings to get what type they really are... you lose all the metadata.

So either look into how to use attributes properly with your component or just try another markup language.

Edit : Also, you may want to read on automatic serialization/deserialization, which take objects and convert them to and from XML or other markup languages. This is a HUGE time-saver when saving or reading content. There's a short article on the wiki for how that works in .NET languages, you'll need to look around for Delphi since I know nothing about it.
http://wiki.truevision3d.com/tutorialsarticlesandexamples/introduction_to_serialization

Edit 2 : <blah attr="2" /> is a valid XML. You can always replace the closing tag by a /> if the element doesn't have any child nodes or content.

[StakFallT]

Well I've seen WoW use an xml format I was looking to to do, and it looked pretty nice. But point taken (about the loosing the benefits)   And yeah if I can help it, I'd rather steer as far away as possible from xml since it can get -really- confusing. But... it seemed like the perfect fit for what I'm needing to do. The alternatives definately look nice   And the automatic serialization/deserialization three words: *Homer Simpson drool*. Very helpful info, thanks MUCH!

[StakFallT]

Sorry to dredge up an aging thread but a question about XML... I actually have it witting, and I'll get over not using "content" as I see the benefit of being about to extract out a single variable value via attributes. But when exporting I leave empty tags
for instance it might go something like this:

Code:

<GROUP_1>
   <TEXTURE>
       <NORMAL [attributes] />
   </TEXTURE>
   <MATERIAL>
         <Header MaterialIsTemplate="False" />
         <Ambient R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
         <Diffuse R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
         <Specular R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
         <Emissive R="0.745098054409027" G="0.509803950786591" B="0" A="1" />
         <Footer Power="1" Opacity="0" />
   </MATERIAL>
</GROUP>
...
<GROUP_98>
       <TEXTURE />
       <MATERIAL />
</GROUP_98>

In the above example, the normal tag with attributes is not used because it's simply not written to the file because that group doesn't have any normal textures. But this cannot work (As far as a valid xml file goes) right? Because if I'm correct, there has to be the same amount of child nodes deep throughout right? In other words, I would need to create empty childs just to duplicate the nest-level -- empty, but childs nonetheless.. It's because there's no way to check if the next read tag is not within the child, it's just expected that the same depth follows throughout the file right?

[serial]

The method I came up with is fairly simple and carried over in to more than just a file format.

The TV Engine is broken up in to classes...

engine
scene
landscape
material

It basically looks something like this in a file...

landscape.<landscapename>.create
landscape.<landscapename>.BlankTerrain.<chunkx>.<chunky>
landscape.<landscapename>.FromHeightMap.<textureFileName>.<chunkx>.<chunky>
landscape.<landscapename>.SetTextureOne.<TextureName>
landscape.<landscapename>.SetTextureTwo.<TextureName>
landscape.<landscapename>.SetTextureThree.<TextureName>
landscape.<landscapename>.SetTextureAlpha.<TextureName>


The neat thing about this is I can run it as a script from a file or a commandline.

mesh.<meshname>.setposition.x.y.z 

Would move the named mesh to that location.  But in addition to that I have a undo que that I can save the command to return the mesh to its old position.

So I would do something like...

undoque.add("mesh.<meshname>.setposition.oldx.oldy.oldz")
priorityque.add("mesh.<meshname>.setposition.newx.newy.newz")

I have a 3rd que that all of the save script files get read in to in a seperate thread. 
In my main loop it checks the priority que for commands first then the loaderque second.
I have a little bit of code that does load balancing of sorts if it sees the priority que getting full it will do two commands from the priority que and ignore the loader que until the priority que its lower. 

The neat thing is later on I can add a network que that will receive commands over TCP/IP and process them.  So you can have sort of a collabritave sort of thing going on.

Wouldn't say its the fast way to do things but it is effective.  Having multiple ques does make it very responsive.

[StakFallT]

Well from what I've seen in XML, at least the component I'm using, there isn't any way to check the name of the tag you're pointed to. Which would imply the tags have to be there whether they're empty or not. This way you can assume that 2 child's deep is -always- such_and_such tag, etc..

[Zaknafein]

Well-formed XML only has a handful of rules that you need to follow, but the XML language is very permissive... See : http://en.wikipedia.org/wiki/XML#Well-formed_and_valid_XML_documents

You can vary the number of children (nodes & attributes) between elements that have the same name throughout the document. You can also name two elements the same thing at the same level, so no need to suffix your groups by a number like you do now, unless that means something for you.

If you use (or plan to use) automatic serialization, there's usually a way of making variables in a class as "optional" upon serialization, so if they're empty, null or equal to the type's default value, they'll be omitted. And when deserializing, the serializer doesn't complain if the element is missing.

Edit : Also, a "header" sub-node to the <MATERIAL> element seems like a silly design choice. Why not put the attributes on the parent element?

[StakFallT]

Thanks for the info, good to know I won't have to really redesign my template files to contain lots of bloat (empty tags). Until I know more though about how to retrieve specific parts of the template file I think I'll leave the group numbers in, because saying in code "grab the 9th group element and apply the data to the 9th group of mesh_such_and_such" doesn't really work if the 9th group's data was omitted because it didn't contain anything, in which case I just applied group 10's stuff on group 9.. This can probably be worked out through some attribute or something but the chilkat xml component is SERIOUSLY lacking good examples. All they have are pretty useless examples on their site that barely help at all because most of them don't include the xml they're working with. Only one was the ISBN lookup, and that follows a set (non-varying) number of elements. Even a document that says what which method does is kinda useless as it doesn't demonstrate what they're definition of a child or sibling is.

XML seems to unfortunately contain alot of terminology that does not seem necessary. I'm looking though that link you posted to wiki and that appears (Haven't read through all of it yet) to answer alot of questions!

hmmm interesting thought about moving the header stuff to the parent element. For me the way I typically lay something like this out is I look at it as categories and work deeper until the data fits a category. For the material moving if it's a template to the material element would be huge. It makes ALOT of sense. I'm thinking though that the header node should stay for the mesh, only because placing the header stuff in the mesh element would be like saying all values in the mesh element have a connection to all elements below it, which isn't really the case. I know that's kinda minutia but I kinda feel like a design should follow the "why"  , and moving the mesh header into the mesh element would damage that. But the material header, that makes good sense. Thanks!