Large World Paging???

[analysis_junky]

I am not sure if this is the best place to post this, I apoligize in advance.

Basically, I am trying to make an MMORPG with a huge, scalable (ability to add more to it) world.

I know that TV has Paging, but I am trying to find the best way to use it.

1.  Does anyone know of an editor that supports this type of world?
I have been making my own editor but reinventing the wheel sucks!

2.  The idea is that every nth frame a parimeter check would take place, checking to see which Landscapes must be loaded. Each of the Landscapes would be stored on the filesystem with the game address as the file names. Example: Zone_Region_Town.tvl (I am unsure of the TV Landscape file extension)


So basically, am I on the right track? What tool would be best to split up the world?

Thanks in advance!

[newborn]

youre on the right track. there are 2 ways to tackle this:

1) the landscape is already split in "chunks" which can be enabled or disabled at run time.

2) you load / unload new landscapes at runtime, which will require you to create a landscape editor

[analysis_junky]

Thanks for your reply. I thought of load/unload Lanscapes, however I foresee issues with the various Landscapes aligning properly (height at the seams).

On the other hand, chunks are only 256x256 units and that poses a problem - I want to possibly have varied sizes of each section.

Also, what is the maximum number of Landscapes that should be loaded at any given time?

Say that I am right on square 'e', and to be able to see 360 degrees I would need to load every square in the below example.

---------
a |b |c |
---------
d |e |f |
---------
g |h |i |
---------

[Raine]

I think seams have to be fixed during editing, and while you can do that at runtime too, I personally wouldn't do that.

The number of landscapes to be loaded depends on the size of the landscapes and the far plane distance (that's what I mean for max view distance).

Edit: if you're just about to begin your journey, I for one would tackle the paging problem with a simple program. Try out various combinations of methods and look for the best result. You will be able to tell what you're going to need for your game without messing around with the game itself.

[analysis_junky]

I have actually built 2 partial game/terrain editors and stopped both times due to this exact design issue... I even took a few months off from development to come back with (hopefully) a fresh approach.

For the seams, I am going to do this during the design stage with an editor.

Once I defeat this, I will post the solution because I imagine there are others out there trying to do the same thing.

[Omnicrash]

I already have a solution for this in my game, and it works really well:

Basically I have a 3x3 grid of landscapes;
1 2 3
4 5 6
7 8 9

With 5 being the active one.

Now, when the player reaches one of the boundaries, say from 5 to 2,
it actually deletes the 7, 8 and 9 landscapes and MOVES all the rest one place down, and stores the current values of the landscapes again so you get:

7 8 9
1 2 3
4 5 6
(now the player is on the lower most of 2)

It does the loading process on a separate tread, so it is completely unnoticeable to the end user. One problem through: if the player passes a boundary, and immediately returns, I made the code just wait a bit to avoid stuff going wrong . The wait is actually very short and neglectable,  and it doesn't occur often, so this solution works really well.

To know which landscape is where, I made an array that stores the current position of the landscapes, and to get those positions again I made this simple function:

Code:

static int[,][] tert = new int[3, 3][]; // Here the landscape position gets stored in

public static TV_2DVECTOR landpos, newpos; // These are used by the loader, to check if there is a new position to be processed, since the loader thread can only start once and will continuously loop and check for new positions

// Dynamic world loader thread
static Thread Loader = new Thread(World.Land.TerrainLoader); // The actual loader

static bool loading = false; // Is it loading? If so and the player passes another boundary, this will be checked and the program will pause till the previous one is loaded

// This function is used whenever terrain is needed so it always gets the right one (1, 1 is always the middle, the one the player is at)
public static TVLandscape Terrain(int ChunkX, int ChunkY)
{
    return terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]];
}

As for seaming, I have perfectly seamed heightmaps, yet ingame they do seem to still have a gap. I am currently working on some code to counteract this, also in the separate loading thread to avoid ingame pauses.

[Omnicrash]

Ah wth, here, my full landscape code :

Code:

        #region LANDSCAPES
        ///////////////////////////////////////////////////////////////////////////
        ///////     LANDSCAPES       //////////////////////////////////////////////
        ///////////////////////////////////////////////////////////////////////////
        public struct Land // Landscapes
        {
            static TVLandscape[,] terrain = new TVLandscape[3, 3];

            static int[,][] tert = new int[3, 3][];

            public static TV_2DVECTOR landpos, newpos;

            // Dynamic world loader thread
            static Thread Loader = new Thread(World.Land.TerrainLoader);

            static bool loading = false;

            public static TVLandscape Terrain(int ChunkX, int ChunkY)
            {
                return terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]];
            }

            static void LoadChunk(int ChunkX, int ChunkY)
            {
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]] = Core.Scene.CreateLandscape("ls" + Convert.ToString(newpos.x) + Convert.ToString(newpos.y));
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].SetAffineLevel(CONST_TV_LANDSCAPE_AFFINE.TV_AFFINE_HIGH);
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].GenerateTerrain(@"World\" + Convert.ToString(newpos.x + (ChunkX - 1)) + Convert.ToString(newpos.y + (ChunkY - 1)) + @"\height.png", CONST_TV_LANDSCAPE_PRECISION.TV_PRECISION_HIGH, 10, 10, 2560 * ChunkX, 0, 2560 * ChunkY, true);
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].SetLightingMode(CONST_TV_LIGHTINGMODE.TV_LIGHTING_MANAGED);
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].SetMaterial(Core.Globals.GetMat("MatteTer"));
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].ExpandTexture(Core.TextureFactory.LoadTexture(@"World\" + Convert.ToString(newpos.x + (ChunkX - 1)) + Convert.ToString(newpos.y + (ChunkY - 1)) + @"\tex.dds", "lstex" + Convert.ToString(tert[ChunkX, ChunkY][0]) + Convert.ToString(tert[ChunkX, ChunkY][1])), 0, 0, 10, 10);
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].SetDetailTexture(Core.Globals.GetTex("lsdet"));
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].SetDetailTextureScale(20, 20);
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].SetCustomLightmap(Core.TextureFactory.LoadTexture(@"World\" + Convert.ToString(newpos.x + (ChunkX - 1)) + Convert.ToString(newpos.y + (ChunkY - 1)) + @"\light.dds", "lslight" + Convert.ToString(tert[ChunkX, ChunkY][0]) + Convert.ToString(tert[ChunkX, ChunkY][1])));
                terrain[tert[ChunkX, ChunkY][0], tert[ChunkX, ChunkY][1]].EnableLOD(true, 1000, CONST_TV_LANDSCAPE_PRECISION.TV_PRECISION_AVERAGE, 500);
            }

            public static void InitLandscapes(int LandX, int LandY)
            {
                // Initialize dynamic terrain loading thread
                Loader.Start();

                newpos.x = landpos.x = LandX;
                newpos.y = landpos.y = LandY;

                // Initialize terrain transformation matrix
                tert[0, 0] = new int[2] { 0, 0 };
                tert[0, 1] = new int[2] { 0, 1 };
                tert[0, 2] = new int[2] { 0, 2 };
                tert[1, 0] = new int[2] { 1, 0 };
                tert[1, 1] = new int[2] { 1, 1 };
                tert[1, 2] = new int[2] { 1, 2 };
                tert[2, 0] = new int[2] { 2, 0 };
                tert[2, 1] = new int[2] { 2, 1 };
                tert[2, 2] = new int[2] { 2, 2 };

                // Load materials
                InitMaterials();

                // Load textures
                InitTextures();

                // Setup landscapes
                for (int ly = 0; ly < 3; ly++)
                {
                    for (int lx = 0; lx < 3; lx++)
                    {
                        LoadChunk(lx, ly);
                    }
                }

                // Seam landscapes together
                Seam(1, 1, RelativeLoc.Up);
                Seam(1, 1, RelativeLoc.Down);
                Seam(1, 1, RelativeLoc.Left);
                Seam(1, 1, RelativeLoc.Right);

                Seam(1, 0, RelativeLoc.Left);
                Seam(1, 0, RelativeLoc.Right);

                Seam(1, 2, RelativeLoc.Left);
                Seam(1, 2, RelativeLoc.Right);

                Seam(0, 1, RelativeLoc.Up);
                Seam(0, 1, RelativeLoc.Down);

                Seam(2, 1, RelativeLoc.Up);
                Seam(2, 1, RelativeLoc.Down);
            }

            static void InitTextures()
            {
                // Load textures
                Core.TextureFactory.LoadTexture(@"World\Textures\detail.dds", "lsdet");
            }

            static void InitMaterials()
            {
                int matId;

                matId = Core.MaterialFactory.CreateMaterial("MatteTer");
                Core.MaterialFactory.SetAmbient(matId, 1, 1, 1, 1);
                Core.MaterialFactory.SetDiffuse(matId, 1, 1, 1, 1);
                Core.MaterialFactory.SetSpecular(matId, 0, 0, 0, 1);
                Core.MaterialFactory.SetEmissive(matId, 0, 0, 0, 1);
            }

            public static void Render()
            {
                terrain[tert[1, 1][0], tert[1, 1][1]].Render();
                try
                {
                    for (int rx = 0; rx < 3; rx++)
                        for (int ry = 0; ry < 3; ry++)
                            if (!(rx == 1 && ry == 1) && (terrain[tert[rx, ry][0], tert[rx, ry][1]] != null))
                                terrain[tert[rx, ry][0], tert[rx, ry][1]].Render();
                }
                catch
                {
                    //Do nothing!!! Wheeeee..
                }
            }

            public static void TerrainLoader()
            {
                int loadX = new int();
                int loadY = new int();
                bool useY = new bool();

                while (Program.running)
                {
                    if (loading)
                    {
                        if (newpos.x < landpos.x)
                        {
                            loadX = 0;
                            useY = true;
                        }
                        else if (newpos.x > landpos.x)
                        {
                            loadX = 2;
                            useY = true;
                        }
                        else if (newpos.y < landpos.y)
                        {
                            loadY = 0;
                            useY = false;
                        }
                        else if (newpos.y > landpos.y)
                        {
                            loadY = 2;
                            useY = false;
                        }

                        for (int tl = 0; tl < 3; tl++)
                        {
                            if (useY)
                                loadY = tl;
                            else
                                loadX = tl;

                            LoadChunk(loadX, loadY);
                        }
                        landpos.x = newpos.x;
                        landpos.y = newpos.y;
                        loading = false;
                    }
                    Application.DoEvents();
                }
            }

            static void Destroy(int LandX, int LandY)
            {
                // Destroy its textures to avoid a memory leak
                Core.TextureFactory.DeleteTexture(Core.Globals.GetTex("lstex" + Convert.ToString(tert[LandX, LandY][0]) + Convert.ToString(tert[LandX, LandY][1])));
                Core.TextureFactory.DeleteTexture(Core.Globals.GetTex("lslight" + Convert.ToString(tert[LandX, LandY][0]) + Convert.ToString(tert[LandX, LandY][1])));

                terrain[tert[LandX, LandY][0], tert[LandX, LandY][1]].Destroy();
                terrain[tert[LandX, LandY][0], tert[LandX, LandY][1]] = null;
            }

            static void MoveTerrainX(int MoveX)
            {
                int[] transbuf = new int[2];

                if (MoveX < 0)
                {
                    for (int mt = 0; mt < 3; mt++)
                    {
                        Destroy(0, mt);
                        transbuf[0] = tert[0, mt][0];
                        transbuf[1] = tert[0, mt][1];
                        tert[0, mt][0] = tert[1, mt][0];
                        tert[0, mt][1] = tert[1, mt][1];
                        tert[1, mt][0] = tert[2, mt][0];
                        tert[1, mt][1] = tert[2, mt][1];
                        tert[2, mt][0] = transbuf[0];
                        tert[2, mt][1] = transbuf[1];
                    }
                }
                else if (MoveX > 0)
                {
                    for (int mt = 0; mt < 3; mt++)
                    {
                        Destroy(2, mt);
                        transbuf[0] = tert[2, mt][0];
                        transbuf[1] = tert[2, mt][1];
                        tert[2, mt][0] = tert[1, mt][0];
                        tert[2, mt][1] = tert[1, mt][1];
                        tert[1, mt][0] = tert[0, mt][0];
                        tert[1, mt][1] = tert[0, mt][1];
                        tert[0, mt][0] = transbuf[0];
                        tert[0, mt][1] = transbuf[1];
                    }
                }
                Reposition();
            }

            static void MoveTerrainY(int MoveY)
            {
                int[] transbuf = new int[2];

                if (MoveY < 0)
                {
                    for (int mt = 0; mt < 3; mt++)
                    {
                        Destroy(mt, 0);
                        transbuf[0] = tert[mt, 0][0];
                        transbuf[1] = tert[mt, 0][1];
                        tert[mt, 0][0] = tert[mt, 1][0];
                        tert[mt, 0][1] = tert[mt, 1][1];
                        tert[mt, 1][0] = tert[mt, 2][0];
                        tert[mt, 1][1] = tert[mt, 2][1];
                        tert[mt, 2][0] = transbuf[0];
                        tert[mt, 2][1] = transbuf[1];
                    }
                }
                else if (MoveY > 0)
                {
                    for (int mt = 0; mt < 3; mt++)
                    {
                        Destroy(mt, 2);
                        transbuf[0] = tert[mt, 2][0];
                        transbuf[1] = tert[mt, 2][1];
                        tert[mt, 2][0] = tert[mt, 1][0];
                        tert[mt, 2][1] = tert[mt, 1][1];
                        tert[mt, 1][0] = tert[mt, 0][0];
                        tert[mt, 1][1] = tert[mt, 0][1];
                        tert[mt, 0][0] = transbuf[0];
                        tert[mt, 0][1] = transbuf[1];
                    }
                }
                Reposition();
            }

            static void Reposition()
            {
                for (int repx = 0; repx < 3; repx++)
                {
                    for (int repy = 0; repy < 3; repy++)
                    {
                        if (terrain[tert[repx, repy][0], tert[repx, repy][1]] != null)
                            terrain[tert[repx, repy][0], tert[repx, repy][1]].SetPosition((repx * 2560), 0f, (repy * 2560));
                    }
                }
                // Load new terrain
                loading = true;
            }

            public static TV_3DVECTOR CheckBounds(TV_3DVECTOR PlayerPos)
            {
                if (PlayerPos.x < 2560)
                {
                    // Wait if there is still terrain loading
                    while (loading)
                        Application.DoEvents();
                    newpos.x = landpos.x - 1;
                    newpos.y = landpos.y;
                    PlayerPos.x = PlayerPos.x + 2560;
                    MoveTerrainX(1);
                    Meshes.MoveMeshesX(1);
                }
                else if (PlayerPos.x > 5120)
                {
                    // Wait if there is still terrain loading
                    while (loading)
                        Application.DoEvents();
                    newpos.x = landpos.x + 1;
                    newpos.y = landpos.y;
                    PlayerPos.x = PlayerPos.x - 2560;
                    MoveTerrainX(-1);
                    Meshes.MoveMeshesX(-1);
                }
                else if (PlayerPos.z < 2560)
                {
                    // Wait if there is still terrain loading
                    while (loading)
                        Application.DoEvents();
                    newpos.x = landpos.x;
                    newpos.y = landpos.y - 1;
                    PlayerPos.z = PlayerPos.z + 2560;
                    MoveTerrainY(1);
                    Meshes.MoveMeshesY(1);
                }
                else if (PlayerPos.z > 5120)
                {
                    // Wait if there is still terrain loading
                    while (loading)
                        Application.DoEvents();
                    newpos.x = landpos.x;
                    newpos.y = landpos.y + 1;
                    PlayerPos.z = PlayerPos.z - 2560;
                    MoveTerrainY(-1);
                    Meshes.MoveMeshesY(-1);
                }
                return PlayerPos;
            }

            public static void SetShader(TVShader NewShader)
            {
                for (int shadex = 0; shadex < 3; shadex++)
                {
                    for (int shadey = 0; shadey < 3; shadey++)
                    {
                        if (terrain[shadex, shadey] != null)
                            terrain[shadex, shadey].SetShader(NewShader);
                    }
                }
            }

            enum RelativeLoc
            {
                Up,
                Down,
                Left,
                Right
            }

            static void Seam(int TerX, int TerY, RelativeLoc Location)
            {
                float firstHeight = 0;
                float secondHeight = 0;
                float averageHeight = 0;
                int TerX2 = 0;
                int TerY2 = 0;
                bool UseX = false;
                int ModX = 0;
                int ModY = 0;

                switch (Location)
                {
                    case RelativeLoc.Left:
                        TerX2 = TerX - 1;
                        TerY2 = TerY;
                        UseX = false;
                        break;

                    case RelativeLoc.Right:
                        TerX2 = TerX + 1;
                        TerY2 = TerY;
                        UseX = false;
                        break;

                    case RelativeLoc.Up:
                        TerX2 = TerX;
                        TerY2 = TerY - 1;
                        UseX = true;
                        break;

                    case RelativeLoc.Down:
                        TerX2 = TerX;
                        TerY2 = TerY + 1;
                        UseX = true;
                        break;
                }

                for (int sI = 0; sI < 2560; sI++)
                {
                    if (UseX)
                    {
                        ModX = sI + (2560 * TerX);
                        ModY = 2560 * TerY;
                    }
                    else
                    {
                        ModX = 2560 * TerX;
                        ModY = sI + (2560 * TerY);
                    }
                    firstHeight = Terrain(TerX, TerY).GetHeight(ModX, ModY);
                    secondHeight = Terrain(TerX2, TerY2).GetHeight(ModX, ModY);
                   
                    averageHeight = (firstHeight + secondHeight) / 2;
                   
                    Terrain(TerX, TerY).SetHeight(ModX, ModY, averageHeight, true, false, false);
                    Terrain(TerX2, TerY2).SetHeight(ModX, ModY, averageHeight, true, false, false);
                }
            }
        }
        #endregion

The seaming code here still leaves a small gap through.

Also there might be some bits where 2560 & 5120 are hardcoded in: My landscapes are 10x10 so they are 2560 in size. Which means, the main (center) landscape always has its position 2560 - 5120.

And CHUNKS in the code do not refer to landscape chunks here, they refer to individual landscapes.

Enjoy!

[XXL]

Omnicrash.

which editor did you use for creating the landscape?

[serial]

I've been working on a terrain paging system myself.  Mainly paging terrain tiles from the harddrive.

I have been using terrains around 1024 to 2048.  Really just to keep the file sizes small.  I also made a custom terrain shader that saves about 240 fps over TVS splatting shader.  As well as allowing me to use 1 alpha texture instead of four individual textures.  Which lets me use a higher resolution alpha map at a much smaller file size.
So I only need to really load two files for a terrain tile.  The TVL file and the alphamap.

I limit a area to 10 differant terrain texture types(rock, grass, sand,  etc.) Basically I have themed areas like you would see in a typical MMO.  Which makes it easy to manage memory. 

Oh well bed time.  All I can say is it takes a lot of planning and time to get a terrain paging system to work.  Mine actually ended up intigrating loaders for textures and models to make it work even smoother.

[Omnicrash]

Omnicrash.

which editor did you use for creating the landscape?

I'm using the l3DT http://www.bundysoft.com/L3DT/ landscape generator to generate the heightmap, lightmap & texture map. I've made an in-game editor for everything else.

[analysis_junky]

Wow Omnicrash you really nailed it! One of the things I had been thinking about was reposition to keep the position numbers low and simple. It's nice to see how someone else has done it!

I had an idea however that could prevent the "player turning around" extra loading issue.
Basically there would be a buffer region that spans the initial loading point to a predefined distance.
As soon as the player gets inside of the region the terrain starts the new loading process BUT, the onload process would not start until they have reached the end of the region.
The idea is to seperate these 2 tasks.

That way if the player turned around while inside the buffer zone, nothing had been unloaded.

[Omnicrash]

To counter the turning around, I was going to do a load of the next terrains once the player passed a distance in the next one, but I figured, loading times aren't long, and it shouldn't recur often, so I didn't think it'd be worth the trouble.

Unless the player passes the corners of a terrain first, chances are he'll end up briefly in one and the next moment in another terrain.. But if in the end it really bothers me/players I might fix it.

Oh and I just thought of a rather easy way of fixing this, load queuing, just make a queue of everything that needs to be loaded and load it whenever the previous ones are unloaded.

[analysis_junky]

How big are your Landscape sections?

Currently, I am making a "mapper" tool where the designer can click an empty section to add a Landscape in that empty space. The idea is for us to be able to piece each section together and change the elevation (especially at the seams) to match the neighboring sections. I got this to work (not elevation yet) for 1x1 Landscape sections.

It throws my math off when I attempt to add a 2x3 Landscape or other non matching width/height pair.

Our scale is 1 unit = 1 meter

Each section is supposed to be a "town". The issue is that each town will be of different sizes...

[Omnicrash]

They are pretty big, 10x10 (so 2560x2560 ingame), its all in the code

[analysis_junky]

This solution "irons out" 96% of the seams between 2 landscapes on either of its 4 sides.

The 2 landscapes must be of the same size and square.

They must also be sharing a side (touching).

Landscape "a" is the "reference" landscape in that it doesn't change. Landscape "b" uses the referenece landscape as a guide to change its vertices to match those of the reference at their shared side.

Code:

private void FixTheSeams(ref TVLandscape a, ref TVLandscape b, float width)
        {
            //Find the shared (touching) sides
            TV_3DVECTOR aPos = a.GetPosition();
            TV_3DVECTOR bPos = b.GetPosition();

            float i = 0;

            //Is top
            if (bPos.x == aPos.x && bPos.z == aPos.z + width)
            {
                for (i = bPos.x; i < bPos.x + width; i++)
                {
                    b.SetHeight(i, bPos.z, a.GetHeight(i, bPos.z), false, true, false);
                }
            }
            //Is right
            if (bPos.x == aPos.x + width && bPos.z == aPos.z)
            {
                for (i = bPos.z; i < bPos.z + width; i++)
                {
                    b.SetHeight(bPos.x, i, a.GetHeight(bPos.x, i), false, true, false);
                }
            }

            //Is bottom
            if (bPos.x == aPos.x && bPos.z == aPos.z - width)
            {
                for (i = bPos.x; i < bPos.x + width; i++)
                {
                    b.SetHeight(i, bPos.z + width, a.GetHeight(i, bPos.z + width), false, true, false);
                }
            }

            //Is left
            if (bPos.x == aPos.x - width && bPos.z == aPos.z)
            {
                for (i = bPos.z; i < bPos.z + width; i++)
                {
                    b.SetHeight(bPos.x + width, i, a.GetHeight(bPos.x + width, i), false, true, false);
                }
            }

            b.FlushHeightChanges(true, true);
        }

[Raine]

TVLanscape is a class and it's already passed by reference, I think that ref token is redundant