topostats.processing ==================== .. py:module:: topostats.processing .. autoapi-nested-parse:: Functions for processing data. .. !! processed by numpydoc !! Attributes ---------- .. autoapisummary:: topostats.processing.LOGGER Functions --------- .. autoapisummary:: topostats.processing.run_filters topostats.processing.run_grains topostats.processing.run_grainstats topostats.processing.run_disordered_tracing topostats.processing.run_nodestats topostats.processing.run_ordered_tracing topostats.processing.run_splining topostats.processing.run_curvature_stats topostats.processing.get_out_paths topostats.processing.process_scan topostats.processing.process_filters topostats.processing.process_grains topostats.processing.process_grainstats topostats.processing.check_run_steps topostats.processing.completion_message Module Contents --------------- .. py:data:: LOGGER .. py:function:: run_filters(unprocessed_image: numpy.typing.NDArray, pixel_to_nm_scaling: float, filename: str, filter_out_path: pathlib.Path, core_out_path: pathlib.Path, filter_config: dict, plotting_config: dict) -> numpy.typing.NDArray | None Filter and flatten an image. Optionally plots the results, returning the flattened image. :param unprocessed_image: Image to be flattened. :type unprocessed_image: npt.NDArray :param pixel_to_nm_scaling: Scaling factor for converting pixel length scales to nanometres. ie the number of pixels per nanometre. :type pixel_to_nm_scaling: float :param filename: File name for the image. :type filename: str :param filter_out_path: Output directory for step-by-step flattening plots. :type filter_out_path: Path :param core_out_path: General output directory for outputs such as the flattened image. :type core_out_path: Path :param filter_config: Dictionary of configuration for the Filters class to use when initialised. :type filter_config: dict :param plotting_config: Dictionary of configuration for plotting output images. :type plotting_config: dict :returns: Either a numpy array of the flattened image, or None if an error occurs or flattening is disabled in the configuration. :rtype: npt.NDArray | None .. !! processed by numpydoc !! .. py:function:: run_grains(image: numpy.typing.NDArray, pixel_to_nm_scaling: float, filename: str, grain_out_path: pathlib.Path, core_out_path: pathlib.Path, plotting_config: dict, grains_config: dict) -> topostats.grains.ImageGrainCrops Identify grains (molecules) and optionally plots the results. :param image: 2d numpy array image to find grains in. :type image: npt.NDArray :param pixel_to_nm_scaling: Scaling factor for converting pixel length scales to nanometres. I.e. the number of pixels per nanometre. :type pixel_to_nm_scaling: float :param filename: Name of file being processed (used in logging). :type filename: str :param grain_out_path: Output path for step-by-step grain finding plots. :type grain_out_path: Path :param core_out_path: General output directory for outputs such as the flattened image with grain masks overlaid. :type core_out_path: Path :param plotting_config: Dictionary of configuration for plotting images. :type plotting_config: dict :param grains_config: Dictionary of configuration for the Grains class to use when initialised. :type grains_config: dict :returns: Either None in the case of error or grain finding being disabled or a dictionary with keys of "above" and or "below" containing binary masks depicting where grains have been detected. :rtype: dict | None .. !! processed by numpydoc !! .. py:function:: run_grainstats(image_grain_crops: topostats.grains.ImageGrainCrops, filename: str, basename: pathlib.Path, grainstats_config: dict, plotting_config: dict, grain_out_path: pathlib.Path) -> tuple[pandas.DataFrame, dict[int:npt.NDArray], dict[int, topostats.grains.GrainCrop]] Calculate grain statistics for an image and optionally plots the results. :param image_grain_crops: ImageGrainCrops object containing the GrainCrops to calculate stats for. :type image_grain_crops: ImageGrainCrops :param filename: Name of the image. :type filename: str :param basename: Path to directory containing the image. :type basename: Path :param grainstats_config: Dictionary of configuration for the GrainStats class to be used when initialised. :type grainstats_config: dict :param plotting_config: Dictionary of configuration for plotting images. :type plotting_config: dict :param grain_out_path: Directory to save optional grain statistics visual information to. :type grain_out_path: Path :returns: **tuple[pd.DataFrame, dict[int** -- Tuple of a pandas DataFrame containing the statsistics for each grain, a dictionary of images and a dictionary of GrainCrop objects. :rtype: npt.NDArray], dict[int, GrainCrop]] .. !! processed by numpydoc !! .. py:function:: run_disordered_tracing(full_image: numpy.typing.NDArray, image_grain_crops: topostats.grains.ImageGrainCrops, pixel_to_nm_scaling: float, filename: str, basename: str, core_out_path: pathlib.Path, tracing_out_path: pathlib.Path, disordered_tracing_config: dict, plotting_config: dict, grainstats_df: pandas.DataFrame = None) -> tuple[dict | None, pandas.DataFrame, pandas.DataFrame] Skeletonise and prune grains, adding results to statistics data frames and optionally plot results. :param full_image: Image containing the grains to pass to the tracing function. :type full_image: npt.ndarray :param image_grain_crops: ImageGrainCrops object containing the GrainCrops to perform tracing on. :type image_grain_crops: ImageGrainCrops :param pixel_to_nm_scaling: Scaling factor for converting pixel length scales to nanometers, i.e. the number of pixesl per nanometres (nm). :type pixel_to_nm_scaling: float :param filename: Name of the image. :type filename: str :param basename: Path to directory containing the image. :type basename: Path :param core_out_path: Path to save the core disordered trace image to. :type core_out_path: Path :param tracing_out_path: Path to save the optional, diagnostic disordered trace images to. :type tracing_out_path: Path :param disordered_tracing_config: Dictionary configuration for obtaining a disordered trace representation of the grains. :type disordered_tracing_config: dict :param plotting_config: Dictionary configuration for plotting images. :type plotting_config: dict :param grainstats_df: The grain statistics dataframe to be added to. This optional argument defaults to `None` in which case an empty grainstats dataframe is created. :type grainstats_df: pd.DataFrame | None :returns: Dictionary of "grain_" keys and Nx2 coordinate arrays of the disordered grain trace. :rtype: dict .. !! processed by numpydoc !! .. py:function:: run_nodestats(image: numpy.typing.NDArray, disordered_tracing_data: dict, pixel_to_nm_scaling: float, filename: str, core_out_path: pathlib.Path, tracing_out_path: pathlib.Path, nodestats_config: dict, plotting_config: dict, grainstats_df: pandas.DataFrame = None) -> tuple[dict, pandas.DataFrame] Analyse crossing points in grains adding results to statistics data frames and optionally plot results. :param image: Image containing the DNA to pass to the tracing function. :type image: npt.ndarray :param disordered_tracing_data: Dictionary of skeletonised and pruned grain masks. Result from "run_disordered_tracing". :type disordered_tracing_data: dict :param pixel_to_nm_scaling: Scaling factor for converting pixel length scales to nanometers, i.e. the number of pixels per nanometres (nm). :type pixel_to_nm_scaling: float :param filename: Name of the image. :type filename: str :param core_out_path: Path to save the core NodeStats image to. :type core_out_path: Path :param tracing_out_path: Path to save optional, diagnostic NodeStats images to. :type tracing_out_path: Path :param nodestats_config: Dictionary configuration for analysing the crossing points. :type nodestats_config: dict :param plotting_config: Dictionary configuration for plotting images. :type plotting_config: dict :param grainstats_df: The grain statistics dataframe to bee added to. This optional argument defaults to `None` in which case an empty grainstats dataframe is created. :type grainstats_df: pd.DataFrame | None :returns: A NodeStats analysis dictionary and grainstats metrics dataframe. :rtype: tuple[dict, pd.DataFrame] .. !! processed by numpydoc !! .. py:function:: run_ordered_tracing(image: numpy.typing.NDArray, disordered_tracing_data: dict, nodestats_data: dict, filename: str, basename: pathlib.Path, core_out_path: pathlib.Path, tracing_out_path: pathlib.Path, ordered_tracing_config: dict, plotting_config: dict, grainstats_df: pandas.DataFrame = None) -> tuple Order coordinates of traces, adding results to statistics data frames and optionally plot results. :param image: Image containing the DNA to pass to the tracing function. :type image: npt.ndarray :param disordered_tracing_data: Dictionary of skeletonised and pruned grain masks. Result from "run_disordered_tracing". :type disordered_tracing_data: dict :param nodestats_data: Dictionary of images and statistics from the NodeStats analysis. Result from "run_nodestats". :type nodestats_data: dict :param filename: Name of the image. :type filename: str :param basename: The path of the files' parent directory. :type basename: Path :param core_out_path: Path to save the core ordered tracing image to. :type core_out_path: Path :param tracing_out_path: Path to save optional, diagnostic ordered trace images to. :type tracing_out_path: Path :param ordered_tracing_config: Dictionary configuration for obtaining an ordered trace representation of the skeletons. :type ordered_tracing_config: dict :param plotting_config: Dictionary configuration for plotting images. :type plotting_config: dict :param grainstats_df: The grain statistics dataframe to be added to. This optional argument defaults to `None` in which case an empty grainstats dataframe is created. :type grainstats_df: pd.DataFrame | None :returns: A NodeStats analysis dictionary and grainstats metrics dataframe. :rtype: tuple[dict, pd.DataFrame] .. !! processed by numpydoc !! .. py:function:: run_splining(image: numpy.typing.NDArray, ordered_tracing_data: dict, pixel_to_nm_scaling: float, filename: str, core_out_path: pathlib.Path, splining_config: dict, plotting_config: dict, grainstats_df: pandas.DataFrame = None, molstats_df: pandas.DataFrame = None) -> tuple Smooth the ordered trace coordinates, adding results to statistics data frames and optionally plot results. :param image: Image containing the DNA to pass to the tracing function. :type image: npt.NDArray :param ordered_tracing_data: Dictionary of ordered coordinates. Result from "run_ordered_tracing". :type ordered_tracing_data: dict :param pixel_to_nm_scaling: Scaling factor for converting pixel length scales to nanometers, i.e. the number of pixels per nanometres (nm). :type pixel_to_nm_scaling: float :param filename: Name of the image. :type filename: str :param core_out_path: Path to save the core ordered tracing image to. :type core_out_path: Path :param splining_config: Dictionary configuration for obtaining an ordered trace representation of the skeletons. :type splining_config: dict :param plotting_config: Dictionary configuration for plotting images. :type plotting_config: dict :param grainstats_df: The grain statistics dataframe to be added to. This optional argument defaults to `None` in which case an empty grainstats dataframe is created. :type grainstats_df: pd.DataFrame | None :param molstats_df: The molecule statistics dataframe to be added to. This optional argument defaults to `None` in which case an empty grainstats dataframe is created. :type molstats_df: pd.DataFrame | None :returns: A smooth curve analysis dictionary and grainstats metrics dataframe. :rtype: tuple[dict, pd.DataFrame] .. !! processed by numpydoc !! .. py:function:: run_curvature_stats(image: numpy.ndarray, cropped_image_data: dict, grain_trace_data: dict, pixel_to_nm_scaling: float, filename: str, core_out_path: pathlib.Path, tracing_out_path: pathlib.Path, curvature_config: dict, plotting_config: dict) -> dict | None Calculate curvature statistics for the traced DNA molecules. Currently only works on simple traces, not branched traces. :param image: AFM image, for plotting purposes. :type image: np.ndarray :param cropped_image_data: Dictionary containing cropped images. :type cropped_image_data: dict :param grain_trace_data: Dictionary of grain trace data. :type grain_trace_data: dict :param pixel_to_nm_scaling: Scaling factor for converting pixel length scales to nanometres. ie the number of pixels per nanometre. :type pixel_to_nm_scaling: float :param filename: Name of the image. :type filename: str :param core_out_path: Path to save the core curvature image to. :type core_out_path: Path :param tracing_out_path: Path to save the optional, diagnostic curvature images to. :type tracing_out_path: Path :param curvature_config: Dictionary of configuration for running the curvature stats. :type curvature_config: dict :param plotting_config: Dictionary of configuration for plotting images. :type plotting_config: dict :returns: Dictionary containing curvature statistics. :rtype: dict .. !! processed by numpydoc !! .. py:function:: get_out_paths(image_path: pathlib.Path, base_dir: pathlib.Path, output_dir: pathlib.Path, filename: str, plotting_config: dict, grain_dirs: bool = True) Determine components of output paths for a given image and plotting config. :param image_path: Path of the image being processed. :type image_path: Path :param base_dir: Path of the data folder. :type base_dir: Path :param output_dir: Base output directory for output data. :type output_dir: Path :param filename: Name of the image being processed. :type filename: str :param plotting_config: Dictionary of configuration for plotting images. :type plotting_config: dict :param grain_dirs: Whether to create the ``grains`` and ``dnatracing`` sub-directories, by default this is ``True`` but it should be set to ``False`` when running just ``Filters``. :type grain_dirs: bool :returns: Core output path for general file outputs, filter output path for flattening related files and grain output path for grain finding related files. :rtype: tuple .. !! processed by numpydoc !! .. py:function:: process_scan(topostats_object: dict, base_dir: str | pathlib.Path, filter_config: dict, grains_config: dict, grainstats_config: dict, disordered_tracing_config: dict, nodestats_config: dict, ordered_tracing_config: dict, splining_config: dict, curvature_config: dict, plotting_config: dict, output_dir: str | pathlib.Path = 'output') -> tuple[dict, pandas.DataFrame, dict] Process a single image, filtering, finding grains and calculating their statistics. :param topostats_object: A dictionary with keys 'image', 'img_path' and 'pixel_to_nm_scaling' containing a file or frames' image, it's path and it's pixel to namometre scaling value. :type topostats_object: dict[str, Union[npt.NDArray, Path, float]] :param base_dir: Directory to recursively search for files, if not specified the current directory is scanned. :type base_dir: str | Path :param filter_config: Dictionary of configuration options for running the Filter stage. :type filter_config: dict :param grains_config: Dictionary of configuration options for running the Grain detection stage. :type grains_config: dict :param grainstats_config: Dictionary of configuration options for running the Grain Statistics stage. :type grainstats_config: dict :param disordered_tracing_config: Dictionary configuration for obtaining a disordered trace representation of the grains. :type disordered_tracing_config: dict :param nodestats_config: Dictionary of configuration options for running the NodeStats stage. :type nodestats_config: dict :param ordered_tracing_config: Dictionary configuration for obtaining an ordered trace representation of the skeletons. :type ordered_tracing_config: dict :param splining_config: Dictionary of configuration options for running the splining stage. :type splining_config: dict :param curvature_config: Dictionary of configuration options for running the curvature stats stage. :type curvature_config: dict :param plotting_config: Dictionary of configuration options for plotting figures. :type plotting_config: dict :param output_dir: Directory to save output to, it will be created if it does not exist. If it already exists then it is possible that output will be over-written. :type output_dir: str | Path :returns: TopoStats dictionary object, DataFrame containing grain statistics and dna tracing statistics, and dictionary containing general image statistics. :rtype: tuple[dict, pd.DataFrame, dict] .. !! processed by numpydoc !! .. py:function:: process_filters(topostats_object: dict, base_dir: str | pathlib.Path, filter_config: dict, plotting_config: dict, output_dir: str | pathlib.Path = 'output') -> tuple[str, bool] Filter an image return the flattened images and save to ''.topostats''. Runs just the first key step of flattening images to remove noise, tilt and optionally scars saving to ''.topostats'' for subsequent processing and analyses. :param topostats_object: A dictionary with keys 'image', 'img_path' and 'pixel_to_nm_scaling' containing a file or frames' image, it's path and it's pixel to namometre scaling value. :type topostats_object: dict[str, Union[npt.NDArray, Path, float]] :param base_dir: Directory to recursively search for files, if not specified the current directory is scanned. :type base_dir: str | Path :param filter_config: Dictionary of configuration options for running the Filter stage. :type filter_config: dict :param plotting_config: Dictionary of configuration options for plotting figures. :type plotting_config: dict :param output_dir: Directory to save output to, it will be created if it does not exist. If it already exists then it is possible that output will be over-written. :type output_dir: str | Path :returns: A tuple of the image and a boolean indicating if the image was successfully processed. :rtype: tuple[str, bool] .. !! processed by numpydoc !! .. py:function:: process_grains(topostats_object: dict, base_dir: str | pathlib.Path, grains_config: dict, plotting_config: dict, output_dir: str | pathlib.Path = 'output') -> tuple[str, bool] Detect grains in flattened images and save to ''.topostats''. Runs grain detection on flattened images to identify grains and save data to ''.topostats'' for subsequent processing and analyses. :param topostats_object: A dictionary with keys 'image', 'img_path' and 'pixel_to_nm_scaling' containing a file or frames' image, it's path and it's pixel to namometre scaling value. :type topostats_object: dict[str, Union[npt.NDArray, Path, float]] :param base_dir: Directory to recursively search for files, if not specified the current directory is scanned. :type base_dir: str | Path :param grains_config: Dictionary of configuration options for running the Filter stage. :type grains_config: dict :param plotting_config: Dictionary of configuration options for plotting figures. :type plotting_config: dict :param output_dir: Directory to save output to, it will be created if it does not exist. If it already exists then it is possible that output will be over-written. :type output_dir: str | Path :returns: A tuple of the image and a boolean indicating if the image was successfully processed. :rtype: tuple[str, bool] .. !! processed by numpydoc !! .. py:function:: process_grainstats(topostats_object: dict, base_dir: str | pathlib.Path, grainstats_config: dict, plotting_config: dict, output_dir: str | pathlib.Path = 'output') -> tuple[str, bool] Calculate grain statistics in an image where grains have already been detected. Runs just the first key step of flattening images to remove noise, tilt and optionally scars saving to ''.topostats'' for subsequent processing and analyses. :param topostats_object: A dictionary with keys 'image', 'img_path' and 'pixel_to_nm_scaling' containing a file or frames' image, it's path and it's pixel to namometre scaling value. :type topostats_object: dict[str, Union[npt.NDArray, Path, float]] :param base_dir: Directory to recursively search for files, if not specified the current directory is scanned. :type base_dir: str | Path :param grainstats_config: Dictionary of configuration options for running the Filter stage. :type grainstats_config: dict :param plotting_config: Dictionary of configuration options for plotting figures. :type plotting_config: dict :param output_dir: Directory to save output to, it will be created if it does not exist. If it already exists then it is possible that output will be over-written. :type output_dir: str | Path :returns: A tuple of the image and a boolean indicating if the image was successfully processed. :rtype: tuple[str, pd.DataFrame] .. !! processed by numpydoc !! .. py:function:: check_run_steps(filter_run: bool, grains_run: bool, grainstats_run: bool, disordered_tracing_run: bool, nodestats_run: bool, ordered_tracing_run: bool, splining_run: bool) -> None Check options for running steps (Filter, Grain, Grainstats and DNA tracing) are logically consistent. This checks that earlier steps required are enabled. :param filter_run: Flag for running Filtering. :type filter_run: bool :param grains_run: Flag for running Grains. :type grains_run: bool :param grainstats_run: Flag for running GrainStats. :type grainstats_run: bool :param disordered_tracing_run: Flag for running Disordered Tracing. :type disordered_tracing_run: bool :param nodestats_run: Flag for running NodeStats. :type nodestats_run: bool :param ordered_tracing_run: Flag for running Ordered Tracing. :type ordered_tracing_run: bool :param splining_run: Flag for running DNA Tracing. :type splining_run: bool .. !! processed by numpydoc !! .. py:function:: completion_message(config: dict, img_files: list, summary_config: dict, images_processed: int) -> None Print a completion message summarising images processed. :param config: Configuration dictionary. :type config: dict :param img_files: List of found image paths. :type img_files: list :param summary_config: Configuration for plotting summary statistics. :type summary_config: dict :param images_processed: Pandas DataFrame of results. :type images_processed: int .. !! processed by numpydoc !!