Module ploigos_step_runner.utils.file
Shared utils for dealing with files.
Functions
def base64_encode(file_path)-
Given a file_path, read and encode the contents in base64 Returns
Base64Contents- base64 encoded string of file contents
def copy_file(file_path, destination_uri)-
Copies a given file to a destination on the local filesystem.
Notes
Known source protocols * / - local file path * file:// - local file path
Parameters
file_path:str- Path to file to upload.
destination_uri:str- URI to copy file to.
Returns
str- The full path of the destination file.
Raises
ValueError- If file_path does not exist.
def create_parent_dir(file_path)-
Helper method to create parent folder of given file if it does not exist.
Parameters
file_path:str- Absolute name of a file to create the parent folders for
def decompress_file(file_uri)-
Decompresses a file; see
is_compressed()for additional information. This method assumes that only a single file will reside inside the compressed archive (currently a moot point, since the only supported algorithm at the moment, bz2, only allows single files to be compressed).Notes
Known compression types * bz2
Parameters
file_uri:path- Path to the file to decompress.
Returns
str- Path to the decompressed file.
def download_and_decompress_source_to_destination(source_uri, destination_dir)-
Given a source url using a known protocol, downloads the file to a given destination. The file is also decompressed if the compression method is known; see
is_compressed()anddecompress_file()methods for additional information.Notes
Known source protocols * file:// * http:// * https://
Known compression types * bz2
Parameters
source_uri:url- URL to a source file using a known protocol to download to destination folder and decompress if necessary.
destination_dir:path- Path to directory to download and decompress if necessary the source url to.
Returns
str- Path to the downloaded and decompressed (if needed) file from given source.
Raises
RuntimeError- If error downloading file.
ValueError- If source_uri does not start with file://|http://|https://
def download_source_to_destination(source_uri, destination_dir)-
Given a source url using a known protocol downloads the file to a given destination.
Notes
Known source protocols * file:// * http:// * https://
Parameters
source_uri:url- URL to a source file using a known protocol to download to destination folder and decompress if necessary.
destination_dir:path- Path to directory to download and decompress if necessary the source url to.
Returns
str- Path to the downloaded file from given source.
Raises
RuntimeError- If error downloading file.
ValueError- If source_uri does not start with file://|http://|https://
def get_file_extension(file_uri)-
Given a file path, return only the extension of the file
Parameters
file_uri:string- File name (including full path) to a file
Returns
str- The file extension (e.g., '.bz2' for '/tmp/data.xml.bz2').
def get_file_hash(file_path)-
Returns file hash of given file.
Returns
StepResult- Object containing the dictionary results of this step.
def is_compressed(file_uri)-
Given a file path, determine if it the file is compressed using an algorithm that this class recognizes and can handle; see
decompress_file()method for additional information.Notes
Known compression types * bz2
Parameters
file_uri:string- File to check whether or not is in a supported compression format
Returns
bool- True, if the file extension matches a support compression algorithm.
def is_local_file_path(uri)-
Determines whether or a not a given uri refers to a file on the local file system
Parameters
uri:string- File path to inspect
Returns
bool- True, if source-uri is a local file (e.g., file:///path/to/file.xml)
def is_remote_http_path(uri)-
Determines whether or a not a given uri refers to a file location accessible via the Hypertext Transfer Protocol
Parameters
uri:string- File path to inspect
Returns
bool- True, if this is an http file (e.g., https://example.org/example.zip)
def normalize_file_path(uri)-
Normalizes the file path, if necessary.
Parameters
uri:string- File path that may need to be normalized (e.g., removal of 'file://')
Returns
str- The source_uri, normalized
def parse_yaml_or_json_file(yaml_or_json_file)-
Parse YAML or JSON config files.
Parameters
yaml_or_json_files:string- List of paths to YAML or JSON files to load as a dictionary.
Returns
dict- Dictionary parsed from given YAML or JSON file.
Raises
ValueError- If the given file can not be parsed as YAML or JSON.
def upload_file(file_path, destination_uri, username=None, password=None)-
Uploads a given file to a given destination.
Notes
Known source protocols * / - local file path * file:// - local file path * http:// - remote upload * https:// - remote upload
Parameters
file_path:str- Path to file to upload.
destination_uri:str- URI to upload file to using known protocol.
username:str, optional- Optional user name to use when uploading the file to an http(s) destination.
password:str, optional- Optional password to use when upload the file to an http(s) destination.
Returns
str- If given destination is a local file path, then return the full path of the destination file. If given destination is a remote upload, return the response body as a string.
Raises
ValueError- If file_path does not exist. If destination_uri does not start with /|file://|http://|https://
RuntimeError- If error uploading file.
def upload_file_to_http_target(file_path, destination_uri, username=None, password=None)-
Uploads a given file to a remote destination, using http.
Notes
Known source protocols * http:// - remote upload * https:// - remote upload
Parameters
file_path:str- Path to file to upload.
destination_uri:str- URI to upload file to using http.
username:str, optional- Optional user name for http destination.
password:str, optional- Optional password for http destination.
Returns
str- The response body.
Raises
ValueError- If file_path does not exist. If destination_uri does not start with /|file://|http://|https://
RuntimeError- If error uploading file.