file_manager package

Subpackages

Submodules

file_manager.base_file_manager module

class file_manager.base_file_manager.BaseFileManager(scale, object_name, script_source_name='', description='')[source]

Bases: object

The core file manager class handles the creation of absolute paths across our project, imported in different file managers.

Functions:
generate_file_name_gdb (str, str) -> str:

Generates the absolute path for files that can be stored in geodatabases (gdb).

generate_file_name_general_files (str, str, str) -> str:

Generates the absolute path for files that cannot be stored in geodatabases, specifying its file type.

generate_file_name_lyrx (str, str) -> str:

Generates the absolute path for ArcGIS layer files (.lyrx) that cannot be stored in geodatabases.

local_root_directory

The local root directory for the project’s output.

Type:

str

project_root_directory

The root directory name of the project.

Type:

str

general_files_directory_name

The directory name for storing general files.

Type:

str

lyrx_directory_name

The directory name for storing ArcGIS layer files (.lyrx).

Type:

str

Parameters:

  • scale (str) – The scale of the geospatial data, used to differentiate directories and file names. Needs to be defined when imported.

  • object_name (str) – The name of the geospatial object, used to differentiate directories and file names. Needs to be defined when imported.

This class provides methods to generate standardized absolute paths for geodatabase files, general files, and layer files, using the project’s naming conventions and directory structure.

local_root_directory = 'C:\\path\\to\\folder\\you\\want\\your\\outputs\\in'
project_root_directory = 'ag_outputs'
general_files_directory_name = 'general_files'
lyrx_directory_name = 'lyrx_outputs'
property script_source_name
property description
validate_inputs(*args)[source]

Uses the validate_input method to validate the input.

generate_file_name_gdb(script_source_name: str, description: str)[source]

Generates a file path for geodatabase (.gdb) files. After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the file’s purpose or contents.

Returns:

The absolute path for the .gdb file.

Return type:

str

generate_file_name_general_files(script_source_name: str, description: str, file_type: str)[source]

Generates a file path for general files (e.g., CSV, TXT). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the file’s purpose or contents.

  • file_type (str) – The filetype extension (without dot).

Returns:

The absolute path for the general file, including the file extension.

Return type:

str

generate_file_name_general_directory(script_source_name: str, description: str, file_type: str)[source]

Generates a file path for general files (e.g., CSV, TXT). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the file’s purpose or contents.

  • file_type (str) – The filetype extension (without dot).

Returns:

The absolute path for the general file, including the file extension.

Return type:

str

generate_file_name_lyrx(script_source_name: str, description: str)[source]

Generates a file path for ArcGIS layer files (.lyrx). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the layer file’s purpose or contents.

Returns:

The absolute path for the .lyrx file.

Return type:

str

generate_file_lyrx_directory(script_source_name: str, description: str)[source]

Generates a file path for ArcGIS layer files (.lyrx). After validating the input.

Parameters:

  • script_source_name (str) – The name of the script or source generating the file.

  • description (str) – A brief description of the layer file’s purpose or contents.

Returns:

The absolute path for the .lyrx file.

Return type:

str

generate_final_outputs(file_name: str)[source]

Generates a file path for geodatabase (.gdb) files for the final output files. After validating the input.

Parameters:

file_name (str) – The name of the file name for the final output file.

Returns:

The absolute path for the .gdb file.

Return type:

str

file_manager.work_file_manager module

class file_manager.work_file_manager.WorkFileManager(unique_id: int, root_file: str = None, write_to_memory: bool = True, keep_files: bool = False)[source]

Bases: object

What:

This class handles the creation and deletion of work files used in other classes or processes. It is designed to make it easy to switch between writing to disk and in-memory, and to delete/stop deleting work files to better troubleshoot issues. This class is not intended to be used to create final outputs of logics or processes.

How:

The same instance of WorkFileManager can create and manage different structures containing files. for each call of the setup_work_file_paths is designed to take a single structure. Each file path generated by the WorkFileManager is tracked by the created_paths attribute so if you do not need stage the deletion of the files you can simply call the delete_created_files method without any parameters.

Parameters:

  • unique_id (int) – Used to generate unique file names, can be self value or an iterated number.

  • root_file (str) – The core file name used to generate unique file names.

  • write_to_memory (bool) – Defaults to True, write to memory if True, write to disk if False.

  • keep_files (bool) – Defaults to False, delete work files if True, keep work files if False.

general_files_directory_name = 'general_files'
lyrx_directory_name = 'lyrx_outputs'
generate_output(instance: object, name: str, iteration_index: int) → str[source]
What:

Generates a unique file path for a given base name and iteration index. Designed to allow users of WorkFileManager to generate indexed outputs in a loop.

Parameters:

  • instance (object) – The caller instance to update attributes on if needed.

  • name (str) – The base name of the work file.

  • iteration_index (int) – The current iteration index for uniqueness.

  • instance – The caller instance to update attributes on if needed.

Returns:

The generated file path.

Return type:

str

setup_work_file_paths(instance: object, file_structure: Any, keys_to_update: str = None, add_key: str = None, file_type: str = 'gdb', index: int = None) → Any[source]
What:

Generates file paths for supported structures and sets them as attributes on the instance. Currently tested and supported structures include: - str - list[str] - dict[str, str] - list[dict[str, str]]

Parameters:

  • instance (object) – The instance to set the file paths as attributes on.

  • file_structure (Any) – The input structure to process and return.

  • keys_to_update (str, optional) – Keys to update if file_structure is a dictionary.

  • add_key (str, optional) – An additional key to add to the dictionary.

  • file_type (str, optional) – The type of file path to generate. Defaults to “gdb”.

  • index (int, optional) – Index to ensure uniqueness in file names when processing lists of dicts.

Returns:

The same structure as file_structure, updated with generated file paths.

Return type:

Any

setup_dynamic_file_paths(base_name: str, count: int, file_type: str = 'gdb') → list[str][source]

Generates a list of file paths for a dynamic number of files based on a base name.

Parameters:

  • base_name (str) – The base name to use for generating file paths.

  • count (int) – The number of file paths to generate.

  • file_type (str, optional) – The file type for the generated paths. Defaults to “gdb”.

Returns:

A list of generated file paths.

Return type:

list[str]

delete_created_files(delete_targets: list[str] = None, exceptions: list[str] = None, delete_files: list[str] = None)[source]
What:

Deletes the created paths, defaults to deleting all created paths, but can target or exclude specific paths.

Parameters:

  • delete_targets (list[str], optional) – List of paths to delete. Defaults to None.

  • exceptions (list[str], optional) – List of paths to exclude from deletion. Defaults to None.

  • delete_files (bool, optional) – Whether to delete files. Defaults to None, which uses self.keep_files.

static list_contents(data: Any, title: str = 'Contents')[source]

Pretty prints the contents of a data structure (list, dict, or other serializable objects).

Parameters:

  • data (Any) – The data structure to print (list, dict, or other serializable objects).

  • title (str, optional) – A title to display before printing. Defaults to “Contents”.

static apply_to_structure(data, func, **key_map)[source]
What:

Applies a function to elements within a supported data structure. Designed to work with dictionaries, lists of dictionaries, and extensible for other structures.

How:

Maps specified keys in the data structure to the function’s parameters and applies the function to each valid element.

Parameters:

  • data (Union[dict, list[dict]]) – The data structure to process.

  • func (callable) – The function to apply. The keys in key_map should match the function parameters.

  • **key_map (str) – Mapping of function parameter names to keys in the data structure.

Raises:

  • TypeError – If the data type is unsupported.

  • KeyError – If a required key is missing in a dictionary.

static extract_key_by_alias(data: list[dict], unique_alias: str, key: str) → str[source]

Extracts the value of a specific key from the dictionary with the specified alias.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to extract the value from.

Returns:

The value associated with the key in the specified dictionary.

Return type:

str

Raises:

  • ValueError – If no dictionary with the specified alias is found.

  • KeyError – If the key does not exist in the identified dictionary.

static extract_key_all(data: list[dict], key: str) → list[str][source]

Extracts all values for a specific key across all dictionaries.

static set_key_by_alias(data: list[dict], unique_alias: str, key: str, new_value: str) → None[source]

Sets the value of a key in the dictionary with the specified alias. Adds the key if it does not exist.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to set or update.

  • new_value (str) – The value to set for the key.

Raises:

ValueError – If no dictionary with the specified alias is found.

Module contents

class file_manager.WorkFileManager(unique_id: int, root_file: str = None, write_to_memory: bool = True, keep_files: bool = False)[source]

Bases: object

What:

This class handles the creation and deletion of work files used in other classes or processes. It is designed to make it easy to switch between writing to disk and in-memory, and to delete/stop deleting work files to better troubleshoot issues. This class is not intended to be used to create final outputs of logics or processes.

How:

The same instance of WorkFileManager can create and manage different structures containing files. for each call of the setup_work_file_paths is designed to take a single structure. Each file path generated by the WorkFileManager is tracked by the created_paths attribute so if you do not need stage the deletion of the files you can simply call the delete_created_files method without any parameters.

Parameters:

  • unique_id (int) – Used to generate unique file names, can be self value or an iterated number.

  • root_file (str) – The core file name used to generate unique file names.

  • write_to_memory (bool) – Defaults to True, write to memory if True, write to disk if False.

  • keep_files (bool) – Defaults to False, delete work files if True, keep work files if False.

static apply_to_structure(data, func, **key_map)[source]
What:

Applies a function to elements within a supported data structure. Designed to work with dictionaries, lists of dictionaries, and extensible for other structures.

How:

Maps specified keys in the data structure to the function’s parameters and applies the function to each valid element.

Parameters:

  • data (Union[dict, list[dict]]) – The data structure to process.

  • func (callable) – The function to apply. The keys in key_map should match the function parameters.

  • **key_map (str) – Mapping of function parameter names to keys in the data structure.

Raises:

  • TypeError – If the data type is unsupported.

  • KeyError – If a required key is missing in a dictionary.

delete_created_files(delete_targets: list[str] = None, exceptions: list[str] = None, delete_files: list[str] = None)[source]
What:

Deletes the created paths, defaults to deleting all created paths, but can target or exclude specific paths.

Parameters:

  • delete_targets (list[str], optional) – List of paths to delete. Defaults to None.

  • exceptions (list[str], optional) – List of paths to exclude from deletion. Defaults to None.

  • delete_files (bool, optional) – Whether to delete files. Defaults to None, which uses self.keep_files.

static extract_key_all(data: list[dict], key: str) → list[str][source]

Extracts all values for a specific key across all dictionaries.

static extract_key_by_alias(data: list[dict], unique_alias: str, key: str) → str[source]

Extracts the value of a specific key from the dictionary with the specified alias.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to extract the value from.

Returns:

The value associated with the key in the specified dictionary.

Return type:

str

Raises:

  • ValueError – If no dictionary with the specified alias is found.

  • KeyError – If the key does not exist in the identified dictionary.

general_files_directory_name = 'general_files'
generate_output(instance: object, name: str, iteration_index: int) → str[source]
What:

Generates a unique file path for a given base name and iteration index. Designed to allow users of WorkFileManager to generate indexed outputs in a loop.

Parameters:

  • instance (object) – The caller instance to update attributes on if needed.

  • name (str) – The base name of the work file.

  • iteration_index (int) – The current iteration index for uniqueness.

  • instance – The caller instance to update attributes on if needed.

Returns:

The generated file path.

Return type:

str

static list_contents(data: Any, title: str = 'Contents')[source]

Pretty prints the contents of a data structure (list, dict, or other serializable objects).

Parameters:

  • data (Any) – The data structure to print (list, dict, or other serializable objects).

  • title (str, optional) – A title to display before printing. Defaults to “Contents”.

lyrx_directory_name = 'lyrx_outputs'
static set_key_by_alias(data: list[dict], unique_alias: str, key: str, new_value: str) → None[source]

Sets the value of a key in the dictionary with the specified alias. Adds the key if it does not exist.

Parameters:

  • data (list[dict]) – The input list of dictionaries.

  • unique_alias (str) – The alias identifying the target dictionary.

  • key (str) – The key to set or update.

  • new_value (str) – The value to set for the key.

Raises:

ValueError – If no dictionary with the specified alias is found.

setup_dynamic_file_paths(base_name: str, count: int, file_type: str = 'gdb') → list[str][source]

Generates a list of file paths for a dynamic number of files based on a base name.

Parameters:

  • base_name (str) – The base name to use for generating file paths.

  • count (int) – The number of file paths to generate.

  • file_type (str, optional) – The file type for the generated paths. Defaults to “gdb”.

Returns:

A list of generated file paths.

Return type:

list[str]

setup_work_file_paths(instance: object, file_structure: Any, keys_to_update: str = None, add_key: str = None, file_type: str = 'gdb', index: int = None) → Any[source]
What:

Generates file paths for supported structures and sets them as attributes on the instance. Currently tested and supported structures include: - str - list[str] - dict[str, str] - list[dict[str, str]]

Parameters:

  • instance (object) – The instance to set the file paths as attributes on.

  • file_structure (Any) – The input structure to process and return.

  • keys_to_update (str, optional) – Keys to update if file_structure is a dictionary.

  • add_key (str, optional) – An additional key to add to the dictionary.

  • file_type (str, optional) – The type of file path to generate. Defaults to “gdb”.

  • index (int, optional) – Index to ensure uniqueness in file names when processing lists of dicts.

Returns:

The same structure as file_structure, updated with generated file paths.

Return type:

Any