ccpn.util package


ccpn.util.Anchor module

Anchor file, used as a starting point for functions that return the absolute location of the repository.

Needed to allow ObjectDomain (uses Python 2.1) to find the repository location

ccpn.util.AttrDict module

Module Documentation here

class ccpn.util.AttrDict.AttrDict(*args, **kwargs)[source]

Bases: dict

ccpn.util.Colour module

Module Documentation here

class ccpn.util.Colour.Colour(value)[source]

Bases: str

A class to make colour manipulation easier and more transparent.
Assumes that r, g, b values are 8-bit so between 0 and 255 and have optional a.
>>> c = Colour('magenta')
>>> c = Colour('#ff00ff')
>>> c = Colour((255, 0, 255))

Returns 4-tuple of (r, g, b, a) where each one is in range 0 to 255


Returns 4-tuple of (r, g, b, a) where each one is in range 0.0 to 1.0

ccpn.util.Colour.rgbToHex(r, g, b)[source]
ccpn.util.Colour.rgbaToHex(r, g, b, a=255)[source]

ccpn.util.Common module

Miscellaneous common utilities

ccpn.util.Common.convertStringToFileName(fileNameString, validChars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-.+', defaultChar='_')[source]
ccpn.util.Common.dictionaryProduct(dict1: dict, dict2: dict) → dict[source]

multiply input {a:x}, {b:y} to result {(a,b):x*y} dictionary

ccpn.util.Common.flattenIfNumpy(data, shape=None)[source]

If data are Numpy, convert to flat array If shape is given, check that numpy fits shape, and flat sequence fits total number of elements


Changes an input string to the one used for a component of file names.

ccpn.util.Common.getTimeStamp() → str[source]

Get iso-formtted timestamp

ccpn.util.Common.getUuid(programName, timeStamp=None)[source]

Get UUid following the NEF convention

ccpn.util.Common.incrementName(name: str) → str[source]

Add ‘_1’ to name or change suffix ‘_n’ to ‘_(n+1)

ccpn.util.Common.isClose(a: float, b: float, relTolerance: float = 1e-05, absTolerance=1e-08) → bool[source]

Are a and b identical within reasonable floating point tolerance? Uses sum of relative (relTolerance) and absolute (absTolerance) difference

Inspired by numpy.isclose()


split sequence code into (seqCode,seqInsertCode, offset) tuple

ccpn.util.Common.recursiveImport(dirname, modname=None, ignoreModules=None, force=False)[source]

recursively import all .py files (not starting with ‘__’ and not containing internal ‘.’ in their name) from directory dirname and all its subdirectories, provided they contain ‘’ Serves to check that files compile without error

modname is the module name (dot-separated) corresponding to the directory dirName. If modname is None, dirname must be on the pythonPath

Note that there are potential problems if the files we want are not the ones encountered first on the pythonPath

ccpn.util.Common.splitIntFromChars(value: str)[source]

convert a string with a leading integer optionally followed by characters into an (integer,string) tuple

ccpn.util.Common.stringToIdentifier(value: str) → str[source]

Convert string to identifier, replacing non-alphanumeric values by underscore

ccpn.util.Common.uniquify(sequence: → list[source]

Get list of unique elements in sequence, in order of first appearance

ccpn.util.Constants module

Constants used in the program core, including enumerations of allowed values

ccpn.util.Graph module

General graph handling code

ccpn.util.Graph.minimumStepPath(graph: dict, startNode, endNode=None) → Tuple[dict, dict][source]

Minimum-step-path by breadth-first traversal, inspired by Dijkstras algorithm Each edge has the same weight; among paths of the same length the function selects the first encountered. Breadth-first search guarantees that the paths with fewest steps are encountered first.

Input: graph is given in form {node:{node:edgeInfo}} nodes can be any object that can be used as a dictionary key

Output: (costDict, predecessorDict) tuple, where costDict is {node:Tuple[edgeInfo, ...]}, with the edgeInfo along the path from start to node predecessorDict is {node:predecessor} the predecessor of node in the shortest path from start

ccpn.util.Hdf5 module

Module Documentation here

ccpn.util.Hdf5.convertDataToHdf5(spectrum: ccpn.core.Spectrum.Spectrum, outputPath: str)[source]

Convert binary data of spectrum into HDF5 and store at outputPath File suffix is made to end with .hdf5

ccpn.util.LocalShutil module

Local copy of shutil - modified to ignore errors in copying permission bits with copystat WIndows-type file systems osmetimes cannot copy permissions, and teh error is not reliably caught on e.g. VMs

ccpn.util.LocalShutil.copytree(src, dst, symlinks=False, ignore=None, copy_function=<function copy>, ignore_dangling_symlinks=False)[source]

Recursively copy a directory tree.

The destination directory must not already exist. If exception(s) occur, an Error is raised with a list of reasons.

If the optional symlinks flag is true, symbolic links in the source tree result in symbolic links in the destination tree; if it is false, the contents of the files pointed to by symbolic links are copied. If the file pointed by the symlink doesn’t exist, an exception will be added in the list of errors raised in an Error exception at the end of the copy process.

You can set the optional ignore_dangling_symlinks flag to true if you want to silence this exception. Notice that this has no effect on platforms that don’t support os.symlink.

The optional ignore argument is a callable. If given, it is called with the src parameter, which is the directory being visited by copytree(), and names which is the list of src contents, as returned by os.listdir():

callable(src, names) -> ignored_names

Since copytree() is called recursively, the callable will be called once for each directory that is copied. It returns a list of names relative to the src directory that should not be copied.

The optional copy_function argument is a callable that will be used to copy each file. It will be called with the source path and the destination path as arguments. By default, copy2() is used, but any function that supports the same signature (like copy()) can be used.

ccpn.util.Logging module

CCPN logger handling

ccpn.util.Logging.createLogger(loggerName, project, stream=None, level=None, mode='a', removeOldLogsDays=7)[source]

Return a (unique) logger for this project and with given programName, if any. Puts log output into a log file but also optionally can have output go to another, specified, stream (e.g. a console)


ccpn.util.OrderedSet module

Ordered Set By Raymond Hettinger,

class ccpn.util.OrderedSet.OrderedSet(iterable=None)[source]



ccpn.util.Path module

Utilities for path handling

Includes extensions of sys.path functions and CCPN-specific functionality

ccpn.util.Path.checkFilePath(filePath, allowDir=True)[source]

Find lowest directory that contains all files in list NB does not normalise file names.

Input: a list of file names Output: lowest directory that contains all files. Does not end with a file


Similar to splitPath but with head being the top directory and tail being the rest.


Removes path whether file or directory, taking into account whether symbolic link.


Get absolute path to module (directory or file)


Returns the ‘top’ python directory, the one on the python path.


Returns the ‘top’ directory of the containing repository (ccpnv3).


The same as os.path.join but normalises the result.

ccpn.util.Path.normalisePath(path, makeAbsolute=None)[source]

Normalises the path, e.g. removes redundant .. and slashes and makes sure path uses ‘/’ rather than ‘’ as can happen on Windows.


The same as os.path.split but with normalisation taken into account.

ccpn.util.Path.suggestFileLocations(fileNames, startDir=None)[source]

From a list of files, return a common superdirectory and a list of relative file names. If any of the files do not exist, search for an alternative superdirectory that does contain the set of relative file names. Searches in either a superdirectory of the starting/current directory, or in a direct subdirectory.

Input: list of file names

Output: Superdirectory, list of relative file names. If no suitable location is found, superdirectory is returned as None


On Unix does nothing, on Windows replaces ‘/’ with ‘’/

ccpn.util.Phasing module

Module Documentation here

ccpn.util.Phasing.phaseComplexData(data: Sequence[complex], ph0: float = 0.0, ph1: float = 0.0, pivot: float = 1.0) → Sequence[complex][source]
ccpn.util.Phasing.phaseRealData(data: Sequence[float], ph0: float = 0.0, ph1: float = 0.0, pivot: float = 1.0) → Sequence[float][source]

ccpn.util.Register module

Module Documentation here

ccpn.util.Register.updateServer(registrationDict, version='3')[source]

ccpn.util.Sorting module


Sort key for strings.

If the entire string evaluates to a float, the result is (‘’, ‘(floatVal, stringVal), ‘’)

Otherwise returns _numericSplitString(key)

Example of sorting order: [‘’, ‘NaN’, ‘-1’, ‘-1A’, ‘0.0’, ‘1’, ‘2’, ‘15’, ‘3.2e12’, ‘Inf’, ‘Ahh’, ‘b’, ‘b2’, ‘b12’, ‘bb’, ‘ciao’]


Universal sort key, using stringSortKey for strings

ccpn.util.Sorting.universalSortKey(key, _stringOrderingHook=None, _orderedKeyHook=<function _orderedKey>, _unorderedKeyHook=<function _unorderedKey>)[source]

Sort mixed types by type, then value or id. Should work for all object types.

This function serves to sort mixed and unpredictable types, e.g. for sorting rows in a mixed-type (or any-type) table, for processing unordered input in semi-reproducible order, or for preliminary hacking about with mixed-type data. Dicts, lists, and tuples are sorted by content, recursively. Circular references are treated by waiting for the RunTimeError after infinite recursion, and then treating the objects as unorderable.

Sorting order of types is giving in the sortOrder list. Booleans are treated as a separate class. Otherwise all real numbers are compared by value, with NaN treated as small than -Infinity. Objects with types that are not treated explicitly (‘ordered’ and ‘unordered’) are sorted first by class name and class id. ‘Ordered’ objects are sorted by their internal comparison method, ‘unordered’ objects by __name__, length, and id (where each is defined).

The ordering keys for strings, orderable types (that support a __lt__ method), and unorderable types (that do not) can be modified by the optional hook parameters.

The function identifies objects whose __lt__ function does not return a Boolean (e.g. numpy.ndarray) as unorderable. set and frozenset are special-cased as unorderable. The function may give unstable sorting results for objects whose __lt__ function returns a Boolean but does not define an ordering (as for sets), but it will sort these correctly by class.

ccpn.util.SphinxExtensions module

Custom extensions to Sphinx documentation generator


Return a listener that will modify doc strings from their Python annotations. If what is a sequence of strings, only docstrings of a type in what will be processed.

In the first version it adds type and modifiability annotation to properties


Return a listener that will modify doc strings from their Python annotations. If what is a sequence of strings, only docstrings of a type in what will be processed.

In the first version it adds type and modifiability annotation to properties

ccpn.util.Tensor module

class ccpn.util.Tensor.Tensor(xx: float = 0.0, yy: float = 0.0, zz: float = 0.0, isotropic: float = 0.0, axial: float = 0.0, rhombic: float = 0.0, orientationMatrix=None)[source]

Bases: object

Rank 2 tensor value.

Unlike most other ccpn classes Tensor is NOT linked to a Project and does not have a pid


- float, immutable - axial component of tensor (along z axis)


- float, immutable - isotropic component of tensor


- <built-in function array>, immutable - 3,3 numpy array containing orientation matrix of tensor. NBNB TBD agree on a sign convention and modify documentation to suit


- float, immutable - rhombic component of tensor


- float, immutable - xx component of tensor


- float, immutable - yy component of tensor


- float, immutable - zz component of tensor

ccpn.util.Ticks module

Module Documentation here


ccpn.util.Time module

Simple wrapper to make time more managable

class ccpn.util.Time.Time[source]

Bases: float

Simple class to print time in ascii, represented as floats as in time.time()

static fromString(string)[source]

Make from a string, inverse of __str__[source]

ccpn.util.Undo module

General undo handle supporting undo/redo stack

class ccpn.util.Undo.Undo(maxWaypoints=20, maxOperations=10000, debug=False)[source]

Bases: collections.deque

Implementation of an undo and redo stack, with possibility of waypoints. A waypoint is the level at which an undo happens, and each of them could consist of multiple individual undo operations.

To create a waypoint use newWaypoint().


- None, immutable - Undo blocking. If true (non-zero) undo setting is blocked. Allows multiple external functions to set blocking without trampling each other

Modify with increaseBlocking/decreaseBlocking only

canRedo() → bool[source]

True if a redo operation can be performed

canUndo() → bool[source]

True if an undo operation can be performed?


Clear and reset undo object


Reduce level of blocking - when level reaches zero, undo is unblocked


Set one more level of blocking

newItem(undoMethod, redoMethod, undoArgs=None, undoKwargs=None, redoArgs=None, redoKwargs=None)[source]

Add item to the undo stack.


Start new waypoint


Redo one waypoint - or one operation if waypoints are not set.

For now errors are handled by printing a warning and clearing the undo object


Undo one operation - or one waypoint if waypoints are set

For now errors are handled by printing a warning and clearing the undo object


Does nothing - for special undo situations where only one direction must act

ccpn.util.Undo.resetUndo(memopsRoot, maxWaypoints=20, maxOperations=10000, debug: bool = False)[source]

Set or reset undo stack, using passed-in parameters. NB setting either parameter to 0 removes the undo stack.

Set obj values using obj2Value dictionary

ccpn.util.Url module

Utilities for Url handling

ccpn.util.Url.fetchUrl(url, data=None, headers=None, timeout=None)[source]
ccpn.util.Url.uploadFile(url, fileName, data=None)[source]

ccpn.util.jsonIo module

Local enhancements to json, adding support for reading and writing pandas.Series, pandas.DataFrame, pandas.Panel, numpy.ndarray, OrderedDict, and ccpnmodel.ccpncore.Tensor

ccpn.util.jsonIo.dump(obj: object, fp: str, indent: int = 2, **kw)[source]

Dump object to json file with extended object type support

ccpn.util.jsonIo.dumps(obj: object, indent: int = 2, **kw)[source]

Dump object to json string with extended object type support

ccpn.util.jsonIo.load(fp, **kw)[source]

Load json from file fp with extended object type support

ccpn.util.jsonIo.loads(s: str, **kw)[source]

Load json from string s with extended object type support

ccpn.util.recursive_import module

python-recursive-import-test [<options>] [<dir> ...]

Recursively find the files under the given directories and launch Python subprocess that attempt to import them individually. This is used to check initalization time problems and dependencies: a well-written program should not do much on module initialization. In other words, a good program allows its modules to be imported individually. This has important consequences in large codebases, and allows more flexibility.


Given a Python source filename, return a module name to import.

ccpn.util.recursive_import.find_all_pyfiles(topDirectory, ignoreDirs=(), ignoreFiles=())[source]

Find all the files ending with .py inside topDirectory


Find all the files ending with .py

ccpn.util.recursive_import.importAllPyfiles(topDirectory, ignoreDirs=(), ignoreFiles=())[source]

Test import all python files in a directory tree. topDirectory must be (intended to be) on the Python path


ccpn.util.updateHeader module