util
Utilities for FastEstimator.
DefaultKeyDict
¶
Bases: dict
Like collections.defaultdict but it passes the key argument to the default function.
This class is intentionally not @traceable.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
default |
Callable[[Any], Any]
|
A function which takes a key and returns a default value based on the key. |
required |
**kwargs |
Initial key/value pairs for the dictionary. |
{}
|
Source code in fastestimator\fastestimator\util\util.py
FEID
¶
An int wrapper class that can change how it's values are printed.
This class is intentionally not @traceable.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
val |
int
|
An integer id to be wrapped. |
required |
Source code in fastestimator\fastestimator\util\util.py
set_translation_dict
classmethod
¶
Provide a lookup table to be invoked during value printing.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
mapping |
Dict[int, Any]
|
A mapping of id: printable id. |
required |
Source code in fastestimator\fastestimator\util\util.py
Flag
¶
A mutable wrapper around a boolean.
This class is intentionally not @traceable.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
val |
bool
|
The initial value for the Flag. |
False
|
Source code in fastestimator\fastestimator\util\util.py
LogSplicer
¶
A class to send stdout information into a file before passing it along to the normal stdout.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
log_path |
str
|
The path/filename into which to append the current stdout. |
required |
Source code in fastestimator\fastestimator\util\util.py
NonContext
¶
Bases: object
A class which is used to make nothing unusual happen.
This class is intentionally not @traceable.
Source code in fastestimator\fastestimator\util\util.py
Suppressor
¶
Bases: object
A class which can be used to silence output of function calls.
This class is intentionally not @traceable.
Source code in fastestimator\fastestimator\util\util.py
Timer
¶
Bases: ContextDecorator
A class that can be used to time things.
This class is intentionally not @traceable.
x = lambda: list(map(lambda i: i + i/2, list(range(int(1e6)))))
with fe.util.Timer():
x() # Task took 0.1639 seconds
@fe.util.Timer("T2")
def func():
return x()
func() # T2 took 0.14819 seconds
Source code in fastestimator\fastestimator\util\util.py
draw
¶
get_batch_size
¶
Infer batch size from a batch dictionary. It will ignore all dictionary value with data type that doesn't have "shape" attribute.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data |
Dict[str, Any]
|
The batch dictionary. |
required |
Returns:
Type | Description |
---|---|
int
|
batch size. |
Source code in fastestimator\fastestimator\util\util.py
get_num_devices
¶
Determine the number of available GPUs.
Returns:
Type | Description |
---|---|
The number of available GPUs, or 1 if none are found. |
get_shape
¶
A function to find the shapes of an object or sequence of objects.
Lists or Tuples will assume that the zeroth dimension is ragged (shape==None). If entries in the list have mismatched ranks, then only the list dimension will be considered as part of the shape. If all ranks are equal, an attempt will be made to determine which of the interior dimensions are ragged.
x = fe.util.get_shape(np.ones((12,22,11))) # [12, 22, 11]
x = fe.util.get_shape([np.ones((12,22,11)), np.ones((18, 5))]) # [None]
x = fe.util.get_shape([np.ones((12,22,11)), np.ones((18, 5, 4))]) # [None, None, None, None]
x = fe.util.get_shape([np.ones((12,22,11)), np.ones((12, 22, 4))]) # [None, 12, 22, None]
x = fe.util.get_shape({"a": np.ones((12,22,11))}) # []
Parameters:
Name | Type | Description | Default |
---|---|---|---|
obj |
Any
|
Data to infer the shape of. |
required |
Returns:
Type | Description |
---|---|
List[Optional[int]]
|
A list representing the shape of the data. |
Source code in fastestimator\fastestimator\util\util.py
get_type
¶
A function to try and infer the types of data within containers.
x = fe.util.get_type(np.ones((10, 10), dtype='int32')) # "int32"
x = fe.util.get_type(tf.ones((10, 10), dtype='float16')) # "<dtype: 'float16'>"
x = fe.util.get_type(torch.ones((10, 10)).type(torch.float)) # "torch.float32"
x = fe.util.get_type([np.ones((10,10)) for i in range(4)]) # "List[float64]"
x = fe.util.get_type(27) # "int"
For container to look into its element's type, its type needs to be either list or tuple, and the return string will be List[...]. All container elements need to have the same data type becuase it will only check its first element.
x = fe.util.get_type({"a":1, "b":2}) # "dict"
x = fe.util.get_type([1, "a"]) # "List[int]"
x = fe.util.get_type([[[1]]]) # "List[List[List[int]]]"
Parameters:
Name | Type | Description | Default |
---|---|---|---|
obj |
Any
|
Data which may be wrapped in some kind of container. |
required |
Returns:
Type | Description |
---|---|
str
|
A string representation of the data type of the |
Source code in fastestimator\fastestimator\util\util.py
is_number
¶
Check if a given string can be converted into a number.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
arg |
str
|
A potentially numeric input string. |
required |
Returns:
Type | Description |
---|---|
bool
|
True iff |
Source code in fastestimator\fastestimator\util\util.py
pad_batch
¶
A function to pad a batch of data in-place by appending to the ends of the tensors. Tensor type needs to be numpy array otherwise would get ignored. (tf.Tensor and torch.Tensor will cause error)
data = [{"x": np.ones((2, 2)), "y": 8}, {"x": np.ones((3, 1)), "y": 4}]
fe.util.pad_batch(data, pad_value=0)
print(data) # [{'x': [[1., 1.], [1., 1.], [0., 0.]], 'y': 8}, {'x': [[1., 0.], [1., 0.], [1., 0.]]), 'y': 4}]
Parameters:
Name | Type | Description | Default |
---|---|---|---|
batch |
List[MutableMapping[str, np.ndarray]]
|
A list of data to be padded. |
required |
pad_value |
Union[float, int]
|
The value to pad with. |
required |
Raises:
Type | Description |
---|---|
AssertionError
|
If the data within the batch do not have matching rank, or have different keys |
Source code in fastestimator\fastestimator\util\util.py
pad_data
¶
Pad data
by appending pad_value
s along it's dimensions until the target_shape
is reached. All entris of
target_shape should be larger than the data.shape, and have the same rank.
x = np.ones((1,2))
x = fe.util.pad_data(x, target_shape=(3, 3), pad_value = -2) # [[1, 1, -2], [-2, -2, -2], [-2, -2, -2]]
x = fe.util.pad_data(x, target_shape=(3, 3, 3), pad_value = -2) # error
x = fe.util.pad_data(x, target_shape=(4, 1), pad_value = -2) # error
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data |
np.ndarray
|
The data to be padded. |
required |
target_shape |
Tuple[int, ...]
|
The desired shape for |
required |
pad_value |
Union[float, int]
|
The value to insert into |
required |
Returns:
Type | Description |
---|---|
np.ndarray
|
The |
Source code in fastestimator\fastestimator\util\util.py
parse_modes
¶
A function to determine which modes to run on based on a set of modes potentially containing blacklist values.
m = fe.util.parse_modes({"train"}) # {"train"}
m = fe.util.parse_modes({"!train"}) # {"eval", "test", "infer"}
m = fe.util.parse_modes({"train", "eval"}) # {"train", "eval"}
m = fe.util.parse_modes({"!train", "!infer"}) # {"eval", "test"}
Parameters:
Name | Type | Description | Default |
---|---|---|---|
modes |
Set[str]
|
The desired modes to run on (possibly containing blacklisted modes). |
required |
Returns:
Type | Description |
---|---|
Set[str]
|
The modes to run on (converted to a whitelist). |
Raises:
Type | Description |
---|---|
AssertionError
|
If invalid modes are detected, or if blacklisted modes and whitelisted modes are mixed. |
Source code in fastestimator\fastestimator\util\util.py
parse_string_to_python
¶
Convert a string into a python object.
x = fe.util.parse_string_to_python("5") # 5
x = fe.util.parse_string_to_python("[5, 4, 0.3]") # [5, 4, 0.3]
x = fe.util.parse_string_to_python("{'a':5, 'b':7}") # {'a':5, 'b':7}
Parameters:
Name | Type | Description | Default |
---|---|---|---|
val |
str
|
An input string. |
required |
Returns:
Type | Description |
---|---|
Any
|
A python object version of the input string. |
Source code in fastestimator\fastestimator\util\util.py
prettify_metric_name
¶
Add spaces to camel case words, then swap _ for space, and capitalize each word.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
metric |
str
|
A string to be formatted. |
required |
Returns:
Type | Description |
---|---|
str
|
The formatted version of 'metric'. |
Source code in fastestimator\fastestimator\util\util.py
show_image
¶
Plots a given image onto an axis. The repeated invocation of this function will cause figure plot overlap.
If im
is 2D and the length of second dimension are 4 or 5, it will be viewed as bounding box data (x0, y0, w, h,
boxes = np.array([[0, 0, 10, 20, "apple"],
[10, 20, 30, 50, "dog"],
[40, 70, 200, 200, "cat"],
[0, 0, 0, 0, "not_shown"],
[0, 0, -10, -20, "not_shown2"]])
img = np.zeros((150, 150))
fig, axis = plt.subplots(1, 1)
fe.util.show_image(img, fig=fig, axis=axis) # need to plot image first
fe.util.show_image(boxes, fig=fig, axis=axis)
Users can also directly plot text
Parameters:
Name | Type | Description | Default |
---|---|---|---|
axis |
plt.Axes
|
The matplotlib axis to plot on, or None for a new plot. |
None
|
fig |
plt.Figure
|
A reference to the figure to plot on, or None if new plot. |
None
|
im |
Union[np.ndarray, Tensor]
|
The image (width X height) / bounding box / text to display. |
required |
title |
Optional[str]
|
A title for the image. |
None
|
color_map |
str
|
Which colormap to use for greyscale images. |
'inferno'
|
stack_depth |
int
|
Multiple images can be drawn onto the same axis. When stack depth is greater than zero, the |
0
|
Returns:
Type | Description |
---|---|
Optional[plt.Figure]
|
plotted figure. It will be the same object as user have provided in the argument. |
Source code in fastestimator\fastestimator\util\util.py
591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 |
|
strip_prefix
¶
Remove the given prefix
from the target
if it is present there.
x = fe.util.strip_prefix("astring.json", "ast") # "ring.json"
x = fe.util.strip_prefix("astring.json", "asa") # "astring.json"
Parameters:
Name | Type | Description | Default |
---|---|---|---|
target |
Optional[str]
|
A string to be formatted. |
required |
prefix |
Optional[str]
|
A string to be removed from |
required |
Returns:
Type | Description |
---|---|
Optional[str]
|
The formatted version of |
Source code in fastestimator\fastestimator\util\util.py
strip_suffix
¶
Remove the given suffix
from the target
if it is present there.
x = fe.util.strip_suffix("astring.json", ".json") # "astring"
x = fe.util.strip_suffix("astring.json", ".yson") # "astring.json"
Parameters:
Name | Type | Description | Default |
---|---|---|---|
target |
Optional[str]
|
A string to be formatted. |
required |
suffix |
Optional[str]
|
A string to be removed from |
required |
Returns:
Type | Description |
---|---|
Optional[str]
|
The formatted version of |
Source code in fastestimator\fastestimator\util\util.py
to_list
¶
Convert data to a list. A single None value will be converted to the empty list.
x = fe.util.to_list(None) # []
x = fe.util.to_list([None]) # [None]
x = fe.util.to_list(7) # [7]
x = fe.util.to_list([7, 8]) # [7,8]
x = fe.util.to_list({7}) # [7]
x = fe.util.to_list((7)) # [7]
x = fe.util.to_list({'a': 7}) # [{'a': 7}]
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data |
Any
|
Input data, within or without a python container. |
required |
Returns:
Type | Description |
---|---|
List[Any]
|
The input |
Source code in fastestimator\fastestimator\util\util.py
to_number
¶
Convert an input value into a Numpy ndarray.
This method can be used with Python and Numpy data:
b = fe.backend.to_number(5) # 5 (type==np.ndarray)
b = fe.backend.to_number(4.0) # 4.0 (type==np.ndarray)
n = np.array([1, 2, 3])
b = fe.backend.to_number(n) # [1, 2, 3] (type==np.ndarray)
This method can be used with TensorFlow tensors:
This method can be used with PyTorch tensors:
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data |
Union[tf.Tensor, torch.Tensor, np.ndarray, int, float]
|
The value to be converted into a np.ndarray. |
required |
Returns:
Type | Description |
---|---|
np.ndarray
|
An ndarray corresponding to the given |
Source code in fastestimator\fastestimator\util\util.py
to_set
¶
Convert data to a set. A single None value will be converted to the empty set.
x = fe.util.to_set(None) # set()
x = fe.util.to_set([None]) # {None}
x = fe.util.to_set(7) # {7}
x = fe.util.to_set([7, 8]) # {7,8}
x = fe.util.to_set({7}) # {7}
x = fe.util.to_set((7)) # {7}
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data |
Any
|
Input data, within or without a python container. The |
required |
Returns:
Type | Description |
---|---|
Set[Any]
|
The input |