Predictable File Management Reduces Stress

The city model has a lot of parts. The conventions for file naming and location suggested throughout this documentation are oriented toward relieving the model technician and curator and their colleagues from having to make a lot of decisions, and toward letting one enjoy a situation where one generally knows where to find things. Following a convention such as this provides the management with the confidence that the critical city model assets are not subject to predictable silent entropy.

citySchema Modular File System

The citySchema architecture makes use of a modular file-system that has evolved over time to provide a scheme for storing assets that does not grow more complicated as the number of assets or their spatial or temporal extent grows. The file system facilitates sharing and syncing of assets between in-house and off-site installations where parts of the city model may be worked on.

File Re-Use and Local Backup Strategy

Every agency or project will have its local file-systems that have their own arrangements for sharing, permissions, regular backups and restore capability. There are basically two classes of file security that ought to be considered.

The figure to the right illustrates the file system context of the model management workspace. The folders are prefixes with Bos3D because this architecture template us based on the city of Boston 3D model. Your organization may use another prefix. It is recommended that you do use a prefix for your folders since eventually, your city model may incorporate assets from other organizations, such as neighboring towns or campus in-holdings. To keep our text generic, we are going to pretend that our folders are prefixed with the Cty3d prefix.

The Cty3dMaster folder holds much of the prepared source material for the city model, including city-wide terran and groundplan information. These data-sets originate in GIS workflows described elsewhere. Bos3dMaster is where users and applications reference the currently active 3d model collection. The Bos3dArchive folder is used to preserve information that is not routinely used -- including the source files for 3D models that have been imported, previous versions of the development log, and selected archived snapshots of the 3D model collection.

The Bos3dDev folder contains workflow folders used for creating and maintaining components of the city model. Workflow Folders are usually self-contained packages of ArcGIS projects, tools, source data, scratch folders and folders for data that is produced in the workflow. The ModelMgt Workflow is such a folder. A deeper dive into this folder is provided below.

The Cty3dDev Folder

City model maintenance and development projects make use of source data from the Cty3d_Master folder. There may be other sources that different projects have in common that are not yet in the Master repository. Project workflows may be cloned and forked from time to time. We don't want to have to make duplicates of shared resources every time a workflow is cloned. This is one example illustrating the need for a modular system for organizing development work.

Because you may have more than one person or agency sharing development work, it is customary to for each of them to append an identifier name to their Development folder, e.g. Cty3dDev_pbc.

The sub-folders of the Cty3d_Dev folder are described below:

ArcDocs Folder: Holds ArcGIS projects and layer files that may be re-used by different projects and possibly uploaded to the Cty3d_master/ArcDocs folder.

Documentation Folder: The place where user documentation is stored.

ServicePack Folder: The way that changes get shared between a developer and the Cty3dMaster or Cty3dArchive folder is through service packs, that bundle new assets with installation instructions. Service packs are folders that incorporate a date string into the name e.g. Cty3d_SrcvicePack_20211221.

Sources Folder: Many workflows incorporate shared sources, such as terrain models or parcel dat6abases. Where possible the tools and documents in a workflow would reference these form the Cty3dMaster folder. In cases where the sources do not exist in the Master folder, they may be placed here and referred to with relative path references, as usual.

Tools Folder: Most tools are workflow-specific and stored within specific workflow folders. There are some tools that are useful in many workflows. These may be stored in the Cty3dDev_usr folder. In the cae of ArcGIS toolboxes or FME workbenches or Python classes, these would be referenced with relative pathnames from within the workflow folders where they are used. This makes it easy to keep one up-to-date version of these utilities.

Workflows Folder: Workflow folders contain documents, tools and sources that produce new derived data. For example, the SandBoxClipper workflow folder includes an arcGIS project and toolbox that provides the setting fro clipping out a rectangular extract form the city model which may then be exported to an editable 3D model by an FME workbench that is also included in the Tools folder within that workflow.

Workflow folders are designed to use relative path references so that they may be cloned and renamed.

Backup Strategy

All filesystems fail eventually. Often there is no warning. This is why professional data management organizations have a regular backup policy that is capable of detecting trouble and restoring data without loss. Backup policies and systems should not be taken for granted. Their existence should be confirmed with system admininstrators and tested regularly.

Bos3D_Archive and Bos3D_Master folders will be on a network file share which is accessible to all of the technical staff.

The Bos3D_Dev folder is likely to be stored on the workstation used by the model curator. The work in this folder can be intensively managed on a day-to-day basis. Mistakes and disk failure can be lead to weeks worth of work being erased. Therefore, this data should be backed up every night, and easily restored. Nightly backups on user workstations is probably not standard procedure in most organizations. But considering the potential cost of disk failure or accidents, these backup policies should be checked, and if they are not practical to arrange, then it is worthwhile to consider moving the 3D_Dev folder to a RAID or some other filesystem that is backed up daily.

Modular Structure of a WorkFlow Folder

The citySchema project has four primary workflows:

Long Name	Workspace Name
Building Model Management	ModelMgt
Terrain Assembly and Tiling	ModularTerrain
Groundplan Tiling	GroundplanTiler
Preparation of Custom Sandbox Models	SandboxClipper

It is helpful to think of each of these workflows as several patterns in common.

Workspace Patterns

• Each workspace serves as a sort of factory that processes Source data with Tools that produce Derived Data and Scratch Data.
• Sources unique to a project are contained within the local Sources folder. But other sources may be referenced from the Cty3dDev_xyz/Sources folder or from the Cty3DMaster folder. Each project uses relative path names to refer to sources. Choices about the storage location of projects should consider maintaining the integrity of path references in order to make workspaces forkable and re-useable.
• Working documents provide a context for applying the tools to the sources and savng re-useable project data -- in our case these project documents are stored in the ArcDocs folder. If you are using some other software like Rhino, you would keep those documents in your RhinoDocs folder.
• When a project has served its purpose, it can be retired by zipping it. And the same project can then be forked to re-use the latest tools, etc for the next application.