file_manager package

Subpackages

Submodules

file_manager.base_file_manager module

class file_manager.base_file_manager.BaseFileManager(scale: str, object_name: str, script_source_name: str = '', description: str = '')[source]

Bases: object

The core file manager class handles the creation of absolute paths across our project, imported in different file managers.

Functions:
generate_file_name_gdb (str, str) -> str:

Generates the absolute path for files that can be stored in geodatabases (gdb).

generate_file_name_general_files (str, str, str) -> str:

Generates the absolute path for files that cannot be stored in geodatabases, specifying its file type.

generate_file_name_lyrx (str, str) -> str:

Generates the absolute path for ArcGIS layer files (.lyrx) that cannot be stored in geodatabases.

local_root_directory

The local root directory for the project’s output.

Type:

str

project_root_directory

The root directory name of the project.

Type:

str

general_files_directory_name

The directory name for storing general files.

Type:

str

lyrx_directory_name

The directory name for storing ArcGIS layer files (.lyrx).

Type:

str

Parameters:

  • scale (str) – The scale of the geospatial data, used to differentiate directories and file names. Needs to be defined when imported.

  • object_name (str) – The name of the geospatial object, used to differentiate directories and file names. Needs to be defined when imported.

This class provides methods to generate standardized absolute paths for geodatabase files, general files, and layer files, using the project’s naming conventions and directory structure.

local_root_directory = 'C:\\path\\to\\folder\\you\\want\\your\\outputs\\in'
project_root_directory = 'ag_outputs'
general_files_directory_name = 'general_files'
lyrx_directory_name = 'lyrx_outputs'
property script_source_name
property description
validate_inputs(*args)[source]

Uses the validate_input method to validate the input.

generate_file_name_gdb(script_source_name: str, description: str) → GdbFilePath[source]

Generates a file path for geodatabase (.gdb) files. After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the file’s purpose or contents.

Returns:

The absolute path for the .gdb file.

Return type:

str

generate_file_name_general_files(script_source_name: str, description: str, file_type: str) → GeneralFilePath[source]

Generates a file path for general files (e.g., CSV, TXT). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the file’s purpose or contents.

  • file_type (str) – The filetype extension (without dot).

Returns:

The absolute path for the general file, including the file extension.

Return type:

str

generate_file_name_lyrx(script_source_name: str, description: str) → LyrxFilePath[source]

Generates a file path for ArcGIS layer files (.lyrx). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the layer file’s purpose or contents.

Returns:

The absolute path for the .lyrx file.

Return type:

str

generate_file_lyrx_directory_deprecated(script_source_name: str, description: str)[source]

Generates a file path for ArcGIS layer files (.lyrx). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the layer file’s purpose or contents.

Returns:

The absolute path for the .lyrx file.

Return type:

str

generate_final_outputs(file_name: str) → GdbFilePath[source]

Generates a file path for geodatabase (.gdb) files for the final output files. After validating the input.

Parameters:

file_name (str) – The name of the file name for the final output file.

Returns:

The absolute path for the .gdb file.

Return type:

str

generate_general_subdirectory(description: str) → SubdirectoryPath[source]

Generates a subdirectory path under the general files directory.

Parameters:

description (str) – A short descriptor for the subdirectory purpose.

Returns:

Absolute path of the subdirectory.

Return type:

str

file_manager.work_file_manager module

class file_manager.work_file_manager.WorkFileManager(config: WorkFileConfig)[source]

Bases: object

What:

This class handles the creation and deletion of work files used in other classes or processes. It is designed to make it easy to switch between writing to disk and in-memory, and to delete/stop deleting work files to better troubleshoot issues. This class is not intended to be used to create final outputs of logics or processes.

How:

The same instance of WorkFileManager can create and manage different structures containing files. for each call of the setup_work_file_paths is designed to take a single structure. Each file path generated by the WorkFileManager is tracked by the created_paths attribute so if you do not need stage the deletion of the files you can simply call the delete_created_files method without any parameters.

Parameters:

config (WorkFileConfig) – Configuration object containing work file options.

general_files_directory_name = 'general_files'
lyrx_directory_name = 'lyrx_outputs'
build_file_path(file_name: str, file_type: Literal['gdb'] = 'gdb', index: int | None = None) → GdbFilePath[source]
build_file_path(file_name: str, file_type: Literal['lyrx'], index: int | None = None) → LyrxFilePath
build_file_path(file_name: str, file_type: str, index: int | None = None) → GeneralFilePath

Generates a file path based on the file name, type, and an optional index.

Parameters:

  • file_name (str) – The name of the file.

  • file_type (str) – The type of file to generate the path for.

  • index (int, optional) – An optional index to append for uniqueness.

Returns:

A string representing the file path.

Return type:

str

generate_output(instance: object, name: str, iteration_index: int) → str[source]
What:

Generates a unique file path for a given base name and iteration index. Designed to allow users of WorkFileManager to generate indexed outputs in a loop.

Parameters:

  • instance (object) – The caller instance to update attributes on if needed.

  • name (str) – The base name of the work file.

  • iteration_index (int) – The current iteration index for uniqueness.

  • instance – The caller instance to update attributes on if needed.

Returns:

The generated file path.

Return type:

str

setup_work_file_paths(instance: object, file_structure: Any, keys_to_update: str = None, add_key: str = None, file_type: str = 'gdb', index: int = None) → Any[source]
What:

Generates file paths for supported structures and sets them as attributes on the instance. Currently tested and supported structures include: - str - list[str] - dict[str, str] - list[dict[str, str]]

Parameters:

  • instance (object) – The instance to set the file paths as attributes on.

  • file_structure (Any) – The input structure to process and return.

  • keys_to_update (str, optional) – Keys to update if file_structure is a dictionary.

  • add_key (str, optional) – An additional key to add to the dictionary.

  • file_type (str, optional) – The type of file path to generate. Defaults to “gdb”.

  • index (int, optional) – Index to ensure uniqueness in file names when processing lists of dicts.

Returns:

The same structure as file_structure, updated with generated file paths.

Return type:

Any

delete_created_files(delete_targets: Iterable[GdbFilePath | GeneralFilePath | InjectIO | LyrxFilePath | None] | None = None, exceptions: Iterable[GdbFilePath | GeneralFilePath | InjectIO | LyrxFilePath | None] | None = None, delete_files: bool | None = None) → None[source]

Deletes created paths. If delete_targets is None, deletes all tracked paths. ‘exceptions’ (if any) are excluded. If delete_files is None, uses not self.keep_files.

static list_contents(data: Any, title: str = 'Contents')[source]

Pretty prints the contents of a data structure (list, dict, or other serializable objects).

Parameters:

  • data (Any) – The data structure to print (list, dict, or other serializable objects).

  • title (str, optional) – A title to display before printing. Defaults to “Contents”.

static apply_to_structure(data, func, **key_map)[source]
What:

Applies a function to elements within a supported data structure. Designed to work with dictionaries, lists of dictionaries, and extensible for other structures.

How:

Maps specified keys in the data structure to the function’s parameters and applies the function to each valid element.

Parameters:

  • data (Union[dict, list[dict]]) – The data structure to process.

  • func (callable) – The function to apply. The keys in key_map should match the function parameters.

  • **key_map (str) – Mapping of function parameter names to keys in the data structure.

Raises:

  • TypeError – If the data type is unsupported.

  • KeyError – If a required key is missing in a dictionary.

static extract_key_by_alias(data: list[dict], unique_alias: str, key: str) → str[source]

Extracts the value of a specific key from the dictionary with the specified alias.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to extract the value from.

Returns:

The value associated with the key in the specified dictionary.

Return type:

str

Raises:

  • ValueError – If no dictionary with the specified alias is found.

  • KeyError – If the key does not exist in the identified dictionary.

static extract_key_all(data: list[dict], key: str) → list[str][source]

Extracts all values for a specific key across all dictionaries.

static set_key_by_alias(data: list[dict], unique_alias: str, key: str, new_value: str) → None[source]

Sets the value of a key in the dictionary with the specified alias. Adds the key if it does not exist.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to set or update.

  • new_value (str) – The value to set for the key.

Raises:

ValueError – If no dictionary with the specified alias is found.

class file_manager.work_file_manager.PartitionWorkFileManager(config: WorkFileConfig)[source]

Bases: WorkFileManager

Extension of WorkFileManager to support partition-aware file path generation. Intended for use inside PartitionIterator or similar partition-based logic.

generate_partition_path(object_name: str, tag: str | None = None, partition_id: int | None = None, suffix: str = '', extension: str = 'gdb') → str[source]

Constructs a consistent file path for work files. If a partition_id is provided, it becomes a partition-aware path; otherwise, it generates a general work path.

Parameters:

  • object_name (str) – Identifier for the object (e.g., layer).

  • tag (str, optional) – Tag (e.g., ‘input’, ‘context’, ‘copy’, etc.).

  • partition_id (int | None, optional) – Partition number. Defaults to None.

  • suffix (str, optional) – Extra string to differentiate logic. Defaults to “”.

  • extension (str) – File type or extension. Defaults to ‘gdb’.

Returns:

Constructed file path.

Return type:

str

Module contents

class file_manager.WorkFileManager(config: WorkFileConfig)[source]

Bases: object

What:

This class handles the creation and deletion of work files used in other classes or processes. It is designed to make it easy to switch between writing to disk and in-memory, and to delete/stop deleting work files to better troubleshoot issues. This class is not intended to be used to create final outputs of logics or processes.

How:

The same instance of WorkFileManager can create and manage different structures containing files. for each call of the setup_work_file_paths is designed to take a single structure. Each file path generated by the WorkFileManager is tracked by the created_paths attribute so if you do not need stage the deletion of the files you can simply call the delete_created_files method without any parameters.

Parameters:

config (WorkFileConfig) – Configuration object containing work file options.

static apply_to_structure(data, func, **key_map)[source]
What:

Applies a function to elements within a supported data structure. Designed to work with dictionaries, lists of dictionaries, and extensible for other structures.

How:

Maps specified keys in the data structure to the function’s parameters and applies the function to each valid element.

Parameters:

  • data (Union[dict, list[dict]]) – The data structure to process.

  • func (callable) – The function to apply. The keys in key_map should match the function parameters.

  • **key_map (str) – Mapping of function parameter names to keys in the data structure.

Raises:

  • TypeError – If the data type is unsupported.

  • KeyError – If a required key is missing in a dictionary.

build_file_path(file_name: str, file_type: str = 'gdb', index: int | None = None) → GdbFilePath | LyrxFilePath | GeneralFilePath[source]

Generates a file path based on the file name, type, and an optional index.

Parameters:

  • file_name (str) – The name of the file.

  • file_type (str) – The type of file to generate the path for.

  • index (int, optional) – An optional index to append for uniqueness.

Returns:

A string representing the file path.

Return type:

str

delete_created_files(delete_targets: Iterable[GdbFilePath | GeneralFilePath | InjectIO | LyrxFilePath | None] | None = None, exceptions: Iterable[GdbFilePath | GeneralFilePath | InjectIO | LyrxFilePath | None] | None = None, delete_files: bool | None = None) → None[source]

Deletes created paths. If delete_targets is None, deletes all tracked paths. ‘exceptions’ (if any) are excluded. If delete_files is None, uses not self.keep_files.

static extract_key_all(data: list[dict], key: str) → list[str][source]

Extracts all values for a specific key across all dictionaries.

static extract_key_by_alias(data: list[dict], unique_alias: str, key: str) → str[source]

Extracts the value of a specific key from the dictionary with the specified alias.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to extract the value from.

Returns:

The value associated with the key in the specified dictionary.

Return type:

str

Raises:

  • ValueError – If no dictionary with the specified alias is found.

  • KeyError – If the key does not exist in the identified dictionary.

general_files_directory_name = 'general_files'
generate_output(instance: object, name: str, iteration_index: int) → str[source]
What:

Generates a unique file path for a given base name and iteration index. Designed to allow users of WorkFileManager to generate indexed outputs in a loop.

Parameters:

  • instance (object) – The caller instance to update attributes on if needed.

  • name (str) – The base name of the work file.

  • iteration_index (int) – The current iteration index for uniqueness.

  • instance – The caller instance to update attributes on if needed.

Returns:

The generated file path.

Return type:

str

static list_contents(data: Any, title: str = 'Contents')[source]

Pretty prints the contents of a data structure (list, dict, or other serializable objects).

Parameters:

  • data (Any) – The data structure to print (list, dict, or other serializable objects).

  • title (str, optional) – A title to display before printing. Defaults to “Contents”.

lyrx_directory_name = 'lyrx_outputs'
static set_key_by_alias(data: list[dict], unique_alias: str, key: str, new_value: str) → None[source]

Sets the value of a key in the dictionary with the specified alias. Adds the key if it does not exist.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to set or update.

  • new_value (str) – The value to set for the key.

Raises:

ValueError – If no dictionary with the specified alias is found.

setup_work_file_paths(instance: object, file_structure: Any, keys_to_update: str = None, add_key: str = None, file_type: str = 'gdb', index: int = None) → Any[source]
What:

Generates file paths for supported structures and sets them as attributes on the instance. Currently tested and supported structures include: - str - list[str] - dict[str, str] - list[dict[str, str]]

Parameters:

  • instance (object) – The instance to set the file paths as attributes on.

  • file_structure (Any) – The input structure to process and return.

  • keys_to_update (str, optional) – Keys to update if file_structure is a dictionary.

  • add_key (str, optional) – An additional key to add to the dictionary.

  • file_type (str, optional) – The type of file path to generate. Defaults to “gdb”.

  • index (int, optional) – Index to ensure uniqueness in file names when processing lists of dicts.

Returns:

The same structure as file_structure, updated with generated file paths.

Return type:

Any

created_paths: set[str]