How the palettes work
Hocus Pocus actually uses two separate palettes at a time: one being the lower 128 colors and the second being the upper 128 colors of 256-color VGA palette.
The "lower" palette is fixed through all the game and is stored in GAMEPAL.PAL file inside HOCUS.DAT. The "upper" palette is variable and is bound to a specific backdrop image. In other words, each backdrop image (BACKxx.PCX) has its own palette (BACKxx.PAL). The menu has also its own palette (MENUPAL.PAL).
All the original game tilesets and sprites use only colors within palette indexes 0 - 127. And all the backdrop images consist only of colors with indexes 128 - 255.
If you are making new graphics, the backdrop image may contain fixed colors from range 0 - 127 and for the rest colors (128 - 255) you can make your own palette. Just remember to save your upper palette into BACKxx.PAL as the game ignores the palette internally stored in PCX file itself.
You can use colors from range 128 - 255 in a tileset, but remember that the tileset must be bound to a specific backdrop image with the same upper palette. Because the upper palette depends on backdrop image which is loaded.
How graphics data are stored in memory
Unfortunately there is a data limit of how many tiles in the tileset and how many sprites can be used in a level. This is because the game uses a fixed-size memory area to store graphics. That area is shared by tiles, sprites, and backdrop image data.
Here is an example of how this memory area is used:
TTTTTTTTTTTTTTSSSSSSSSSSSSSSSSXXBBBBBBBBBBBBBBBBBBBB T = Tileset data S = Sprite data X = Free space B = Backdrop image data
The size of tileset data depends of how many tiles are used in tileset (see later how it works). Sprite data are stored immediately after tileset data. Some sprites are loaded always autimatically (the player, score tags etc.) and after that monster sprites are loaded according to which monsters are used in a level. Backdrop image data have fixed size and are always loaded at fixed position.
If you use too many monster types in a level, the pixel data of the "highest" sprites will load into memory area reserved for a backdrop image. Because backdrop image is loaded after sprites (in time order), the sprites will get overwritten with it. The same happens if there are too many tiles in tileset - the sprite data are pushed away to the area reserved for backdrop image and will get overwritten too.
Now how the number of tiles in tileset is determined. The 320x200 tileset image is not loaded into memory as a whole, but it is being stored tile by tile. After the last used tile follows a special "white tile" which is marking the end of tileset and telling the game to stop loading data. The empty part of tileset after this white tile is ignored. The white tile is actually fully made of pixels with palette index 255.
There is effective limit of 135 tiles and 3 simultaneous monster types in a level (that's reached in E1L5 and E1L6). In some levels there are more (even 4 or 5) monster types, but less tiles in the tileset. So this is always a trade-off.