Adversarial Robustness with Error Correcting Codes (and Hinge Loss)¶

(Never use Softmax again)¶

[Paper] [Notebook] [TF Implementation] [Torch Implementation]

In this example we will show how using error correcting codes as a model output can drastically reduce model overfitting while simultaneously increasing model robustness against adversarial attacks. In other words, why you should never use a softmax layer again. This is slightly more complicated than the our other ECC apphub example, but it allows for more accurate final probability estimates (the FE Hadamard network layer results in probability smoothing which prevents the network from ever being 100% confident in a class choice). The usefulness of error correcting codes was first publicized by the US Army in a 2019 Neurips Paper. For background on adversarial attacks, and on the attack type we will be demonstrating here, check out our FGSM apphub example. Note that in this apphub we will not be training against adversarial samples, but only performing adversarial attacks during evaluation to see how different models fair against them.

Imports¶

In [1]:

import tempfile

from tensorflow.keras import Sequential, layers
from tensorflow.keras.layers import Concatenate, Conv2D, Dense, Flatten, Input, MaxPooling2D
from tensorflow.keras.models import Model

import fastestimator as fe
from fastestimator.architecture.tensorflow import LeNet
from fastestimator.dataset.data import cifair10
from fastestimator.op.numpyop.univariate import Hadamard, Normalize
from fastestimator.op.tensorop import UnHadamard
from fastestimator.op.tensorop.gradient import FGSM, Watch
from fastestimator.op.tensorop.loss import CrossEntropy, Hinge
from fastestimator.op.tensorop.model import ModelOp, UpdateOp
from fastestimator.summary.logs import visualize_logs
from fastestimator.trace.adapt import EarlyStopping
from fastestimator.trace.io import BestModelSaver
from fastestimator.trace.metric import Accuracy

import tempfile from tensorflow.keras import Sequential, layers from tensorflow.keras.layers import Concatenate, Conv2D, Dense, Flatten, Input, MaxPooling2D from tensorflow.keras.models import Model import fastestimator as fe from fastestimator.architecture.tensorflow import LeNet from fastestimator.dataset.data import cifair10 from fastestimator.op.numpyop.univariate import Hadamard, Normalize from fastestimator.op.tensorop import UnHadamard from fastestimator.op.tensorop.gradient import FGSM, Watch from fastestimator.op.tensorop.loss import CrossEntropy, Hinge from fastestimator.op.tensorop.model import ModelOp, UpdateOp from fastestimator.summary.logs import visualize_logs from fastestimator.trace.adapt import EarlyStopping from fastestimator.trace.io import BestModelSaver from fastestimator.trace.metric import Accuracy

In [2]:

parameters

# training parameters
epsilon=0.04  # The strength of the adversarial attack
epochs=60
batch_size=50
log_steps=500
train_steps_per_epoch=None
eval_steps_per_epoch=None
save_dir=tempfile.mkdtemp()

# training parameters epsilon=0.04 # The strength of the adversarial attack epochs=60 batch_size=50 log_steps=500 train_steps_per_epoch=None eval_steps_per_epoch=None save_dir=tempfile.mkdtemp()

Getting the Data¶

For these experiments we will be using the ciFAIR10 Dataset (like cifar10, but with duplicate testing data replaced)

In [3]:

train_data, eval_data = cifair10.load_data()
test_data = eval_data.split(0.5)
pipeline = fe.Pipeline(
    train_data=train_data,
    eval_data=eval_data,
    test_data=test_data,
    batch_size=batch_size,
    ops=[
        Normalize(inputs="x", outputs="x", mean=(0.4914, 0.4822, 0.4465), std=(0.2471, 0.2435, 0.2616)),
        Hadamard(inputs="y", outputs="y_code", n_classes=10)
        ])

train_data, eval_data = cifair10.load_data() test_data = eval_data.split(0.5) pipeline = fe.Pipeline( train_data=train_data, eval_data=eval_data, test_data=test_data, batch_size=batch_size, ops=[ Normalize(inputs="x", outputs="x", mean=(0.4914, 0.4822, 0.4465), std=(0.2471, 0.2435, 0.2616)), Hadamard(inputs="y", outputs="y_code", n_classes=10) ])

Defining Estimators¶

In this apphub we will be comparing a baseline model against two models using hinge loss to enable training with error correcting codes. The setting up the hinge loss models requires a few extra Ops along the way.

In [4]:

def get_baseline_estimator(model):
    network = fe.Network(ops=[
        Watch(inputs="x", mode=('eval', 'test')),
        ModelOp(model=model, inputs="x", outputs="y_pred"),
        CrossEntropy(inputs=("y_pred", "y"), outputs="base_ce"),
        UpdateOp(model=model, loss_name="base_ce"),
        FGSM(data="x", loss="base_ce", outputs="x_adverse", epsilon=epsilon, mode=('eval', 'test')),
        ModelOp(model=model, inputs="x_adverse", outputs="y_pred_adv", mode=('eval', 'test')),
        CrossEntropy(inputs=("y_pred_adv", "y"), outputs="adv_ce", mode=('eval', 'test'))
    ])
    traces = [
        Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"),
        Accuracy(true_key="y", pred_key="y_pred_adv", output_name="adversarial_accuracy"),
        BestModelSaver(model=model, save_dir=save_dir, metric="base_ce", save_best_mode="min", load_best_final=True),
        EarlyStopping(monitor="base_ce", patience=10)
    ]
    estimator = fe.Estimator(pipeline=pipeline,
                             network=network,
                             epochs=epochs,
                             traces=traces,
                             log_steps=log_steps,
                             train_steps_per_epoch=train_steps_per_epoch,
                             eval_steps_per_epoch=eval_steps_per_epoch,
                             monitor_names=["adv_ce"])
    return estimator

def get_baseline_estimator(model): network = fe.Network(ops=[ Watch(inputs="x", mode=('eval', 'test')), ModelOp(model=model, inputs="x", outputs="y_pred"), CrossEntropy(inputs=("y_pred", "y"), outputs="base_ce"), UpdateOp(model=model, loss_name="base_ce"), FGSM(data="x", loss="base_ce", outputs="x_adverse", epsilon=epsilon, mode=('eval', 'test')), ModelOp(model=model, inputs="x_adverse", outputs="y_pred_adv", mode=('eval', 'test')), CrossEntropy(inputs=("y_pred_adv", "y"), outputs="adv_ce", mode=('eval', 'test')) ]) traces = [ Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"), Accuracy(true_key="y", pred_key="y_pred_adv", output_name="adversarial_accuracy"), BestModelSaver(model=model, save_dir=save_dir, metric="base_ce", save_best_mode="min", load_best_final=True), EarlyStopping(monitor="base_ce", patience=10) ] estimator = fe.Estimator(pipeline=pipeline, network=network, epochs=epochs, traces=traces, log_steps=log_steps, train_steps_per_epoch=train_steps_per_epoch, eval_steps_per_epoch=eval_steps_per_epoch, monitor_names=["adv_ce"]) return estimator

In [5]:

def get_hinge_estimator(model):
    network = fe.Network(ops=[
        Watch(inputs="x", mode=('eval', 'test')),
        ModelOp(model=model, inputs="x", outputs="y_pred_code"),
        Hinge(inputs=("y_pred_code", "y_code"), outputs="base_hinge"),
        UpdateOp(model=model, loss_name="base_hinge"),
        UnHadamard(inputs="y_pred_code", outputs="y_pred", n_classes=10, mode=('eval', 'test')),
        # The adversarial attack:
        FGSM(data="x", loss="base_hinge", outputs="x_adverse_hinge", epsilon=epsilon, mode=('eval', 'test')),
        ModelOp(model=model, inputs="x_adverse_hinge", outputs="y_pred_adv_hinge_code", mode=('eval', 'test')),
        Hinge(inputs=("y_pred_adv_hinge_code", "y_code"), outputs="adv_hinge", mode=('eval', 'test')),
        UnHadamard(inputs="y_pred_adv_hinge_code", outputs="y_pred_adv_hinge", n_classes=10, mode=('eval', 'test')),
    ])
    traces = [
        Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"),
        Accuracy(true_key="y", pred_key="y_pred_adv_hinge", output_name="adversarial_accuracy"),
        BestModelSaver(model=model, save_dir=save_dir, metric="base_hinge", save_best_mode="min", load_best_final=True),
        EarlyStopping(monitor="base_hinge", patience=10)
    ]
    estimator = fe.Estimator(pipeline=pipeline,
                             network=network,
                             epochs=epochs,
                             traces=traces,
                             log_steps=log_steps,
                             train_steps_per_epoch=train_steps_per_epoch,
                             eval_steps_per_epoch=eval_steps_per_epoch,
                             monitor_names=["adv_hinge"])
    return estimator

def get_hinge_estimator(model): network = fe.Network(ops=[ Watch(inputs="x", mode=('eval', 'test')), ModelOp(model=model, inputs="x", outputs="y_pred_code"), Hinge(inputs=("y_pred_code", "y_code"), outputs="base_hinge"), UpdateOp(model=model, loss_name="base_hinge"), UnHadamard(inputs="y_pred_code", outputs="y_pred", n_classes=10, mode=('eval', 'test')), # The adversarial attack: FGSM(data="x", loss="base_hinge", outputs="x_adverse_hinge", epsilon=epsilon, mode=('eval', 'test')), ModelOp(model=model, inputs="x_adverse_hinge", outputs="y_pred_adv_hinge_code", mode=('eval', 'test')), Hinge(inputs=("y_pred_adv_hinge_code", "y_code"), outputs="adv_hinge", mode=('eval', 'test')), UnHadamard(inputs="y_pred_adv_hinge_code", outputs="y_pred_adv_hinge", n_classes=10, mode=('eval', 'test')), ]) traces = [ Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"), Accuracy(true_key="y", pred_key="y_pred_adv_hinge", output_name="adversarial_accuracy"), BestModelSaver(model=model, save_dir=save_dir, metric="base_hinge", save_best_mode="min", load_best_final=True), EarlyStopping(monitor="base_hinge", patience=10) ] estimator = fe.Estimator(pipeline=pipeline, network=network, epochs=epochs, traces=traces, log_steps=log_steps, train_steps_per_epoch=train_steps_per_epoch, eval_steps_per_epoch=eval_steps_per_epoch, monitor_names=["adv_hinge"]) return estimator

The Models¶

1 - A LeNet model with Softmax¶

In [6]:

softmax_model = fe.build(model_fn=lambda:LeNet(input_shape=(32, 32, 3)), optimizer_fn="adam", model_name='softmax')

softmax_model = fe.build(model_fn=lambda:LeNet(input_shape=(32, 32, 3)), optimizer_fn="adam", model_name='softmax')

Metal device set to: Apple M1 Max

2023-03-22 11:38:58.701436: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-03-22 11:38:58.701457: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)

2 - A LeNet model with Error Correcting Codes¶

In [7]:

def EccLeNet(input_shape=(32, 32, 3), code_length=16):
    model = Sequential()
    model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.Flatten())
    model.add(layers.Dense(64, activation='relu'))
    model.add(layers.Dense(code_length, activation='tanh'))  # Note that this is the only difference between this model and the FE LeNet implementation
    return model

def EccLeNet(input_shape=(32, 32, 3), code_length=16): model = Sequential() model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape)) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.Flatten()) model.add(layers.Dense(64, activation='relu')) model.add(layers.Dense(code_length, activation='tanh')) # Note that this is the only difference between this model and the FE LeNet implementation return model

In [8]:

ecc_model = fe.build(model_fn=EccLeNet, optimizer_fn="adam", model_name='ecc')

ecc_model = fe.build(model_fn=EccLeNet, optimizer_fn="adam", model_name='ecc')

3 - A LeNet model using ECC and multiple feature heads¶

While it is common practice to follow the feature extraction layers of convolution networks with several fully connected layers in order to perform classification, this can lead to the final logits being interdependent which can actually reduce the robustness of the network. One way around this is to divide your classification layers into multiple smaller independent units:

In [9]:

def HydraEccLeNet(input_shape=(32, 32, 3), code_length=16):
    inputs = Input(input_shape)
    conv1 = Conv2D(32, (3, 3), activation='relu')(inputs)
    pool1 = MaxPooling2D((2, 2))(conv1)
    conv2 = Conv2D(64, (3, 3), activation='relu')(pool1)
    pool2 = MaxPooling2D((2, 2))(conv2)
    conv3 = Conv2D(64, (3, 3), activation='relu')(pool2)
    flat = Flatten()(conv3)
    # Create multiple heads
    n_heads = 4
    heads = [Dense(16, activation='relu')(flat) for _ in range(n_heads)]
    heads2 = [Dense(code_length // n_heads, activation='tanh')(head) for head in heads]
    outputs = Concatenate()(heads2)
    return Model(inputs=inputs, outputs=outputs)

def HydraEccLeNet(input_shape=(32, 32, 3), code_length=16): inputs = Input(input_shape) conv1 = Conv2D(32, (3, 3), activation='relu')(inputs) pool1 = MaxPooling2D((2, 2))(conv1) conv2 = Conv2D(64, (3, 3), activation='relu')(pool1) pool2 = MaxPooling2D((2, 2))(conv2) conv3 = Conv2D(64, (3, 3), activation='relu')(pool2) flat = Flatten()(conv3) # Create multiple heads n_heads = 4 heads = [Dense(16, activation='relu')(flat) for _ in range(n_heads)] heads2 = [Dense(code_length // n_heads, activation='tanh')(head) for head in heads] outputs = Concatenate()(heads2) return Model(inputs=inputs, outputs=outputs)

In [10]:

hydra_model = fe.build(model_fn=HydraEccLeNet, optimizer_fn="adam", model_name='hydra_ecc')

hydra_model = fe.build(model_fn=HydraEccLeNet, optimizer_fn="adam", model_name='hydra_ecc')

The Experiments¶

Let's get Estimators for each of these models and compare them:

In [11]:

softmax_estimator = get_baseline_estimator(softmax_model)
ecc_estimator = get_hinge_estimator(ecc_model)
hydra_estimator = get_hinge_estimator(hydra_model)

softmax_estimator = get_baseline_estimator(softmax_model) ecc_estimator = get_hinge_estimator(ecc_model) hydra_estimator = get_hinge_estimator(hydra_model)

In [12]:

softmax_estimator.fit('Softmax')
softmax_results = softmax_estimator.test()

softmax_estimator.fit('Softmax') softmax_results = softmax_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Warn: The following key(s) are being pruned since they are unused outside of the Pipeline. To prevent this, you can declare the key(s) as inputs to Traces or TensorOps: y_code
FastEstimator-Start: step: 1; logging_interval: 500; num_device: 1;
WARNING:tensorflow:From /Users/skynet/.pyenv/versions/miniforge3-4.10.3-10/envs/femos38/lib/python3.8/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089
FastEstimator-Train: step: 1; base_ce: 2.2683728;
FastEstimator-Train: step: 500; base_ce: 1.3852535; steps/sec: 136.35;
FastEstimator-Train: step: 1000; base_ce: 1.263034; steps/sec: 144.46;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time(sec): 7.86;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 135.37;
Eval Progress: 66/100; steps/sec: 131.58;
Eval Progress: 100/100; steps/sec: 133.65;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_ce: 1.9911065; adversarial_accuracy: 0.2962; base_accuracy: 0.568; base_ce: 1.1979523; min_base_ce: 1.1979523; since_best_base_ce: 0;
FastEstimator-Train: step: 1500; base_ce: 1.1411203; steps/sec: 147.52;
FastEstimator-Train: step: 2000; base_ce: 1.098317; steps/sec: 157.97;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time(sec): 6.56;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 122.16;
Eval Progress: 66/100; steps/sec: 119.91;
Eval Progress: 100/100; steps/sec: 131.84;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_ce: 2.1250355; adversarial_accuracy: 0.2818; base_accuracy: 0.6378; base_ce: 1.0305675; min_base_ce: 1.0305675; since_best_base_ce: 0;
FastEstimator-Train: step: 2500; base_ce: 0.9761512; steps/sec: 142.73;
FastEstimator-Train: step: 3000; base_ce: 0.9483634; steps/sec: 156.99;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time(sec): 6.68;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 131.63;
Eval Progress: 66/100; steps/sec: 119.31;
Eval Progress: 100/100; steps/sec: 118.55;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_ce: 2.3528206; adversarial_accuracy: 0.273; base_accuracy: 0.6666; base_ce: 0.9486573; min_base_ce: 0.9486573; since_best_base_ce: 0;
FastEstimator-Train: step: 3500; base_ce: 0.92775565; steps/sec: 133.45;
FastEstimator-Train: step: 4000; base_ce: 0.56857014; steps/sec: 160.2;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time(sec): 6.87;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 118.59;
Eval Progress: 66/100; steps/sec: 118.71;
Eval Progress: 100/100; steps/sec: 118.13;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_ce: 2.5284605; adversarial_accuracy: 0.2542; base_accuracy: 0.6818; base_ce: 0.9189946; min_base_ce: 0.9189946; since_best_base_ce: 0;
FastEstimator-Train: step: 4500; base_ce: 0.7068735; steps/sec: 136.94;
FastEstimator-Train: step: 5000; base_ce: 0.73779446; steps/sec: 148.39;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time(sec): 7.02;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 113.56;
Eval Progress: 66/100; steps/sec: 120.9;
Eval Progress: 100/100; steps/sec: 131.89;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_ce: 2.6685538; adversarial_accuracy: 0.2508; base_accuracy: 0.7108; base_ce: 0.8376476; min_base_ce: 0.8376476; since_best_base_ce: 0;
FastEstimator-Train: step: 5500; base_ce: 0.6371431; steps/sec: 132.42;
FastEstimator-Train: step: 6000; base_ce: 0.46461108; steps/sec: 153.62;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time(sec): 7.02;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 126.53;
Eval Progress: 66/100; steps/sec: 119.25;
Eval Progress: 100/100; steps/sec: 122.45;
FastEstimator-Eval: step: 6000; epoch: 6; adv_ce: 2.9378023; adversarial_accuracy: 0.2368; base_accuracy: 0.708; base_ce: 0.8474487; min_base_ce: 0.8376476; since_best_base_ce: 1;
FastEstimator-Train: step: 6500; base_ce: 0.46799204; steps/sec: 133.58;
FastEstimator-Train: step: 7000; base_ce: 0.5437822; steps/sec: 140.84;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time(sec): 7.33;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 108.6;
Eval Progress: 66/100; steps/sec: 117.26;
Eval Progress: 100/100; steps/sec: 120.32;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_ce: 3.108251; adversarial_accuracy: 0.2416; base_accuracy: 0.7204; base_ce: 0.8334195; min_base_ce: 0.8334195; since_best_base_ce: 0;
FastEstimator-Train: step: 7500; base_ce: 0.43123123; steps/sec: 133.37;
FastEstimator-Train: step: 8000; base_ce: 0.3585026; steps/sec: 144.86;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time(sec): 7.17;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 127.4;
Eval Progress: 66/100; steps/sec: 121.69;
Eval Progress: 100/100; steps/sec: 122.4;
FastEstimator-Eval: step: 8000; epoch: 8; adv_ce: 3.373532; adversarial_accuracy: 0.2182; base_accuracy: 0.712; base_ce: 0.8518958; min_base_ce: 0.8334195; since_best_base_ce: 1;
FastEstimator-Train: step: 8500; base_ce: 0.27695072; steps/sec: 131.19;
FastEstimator-Train: step: 9000; base_ce: 0.633098; steps/sec: 133.57;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time(sec): 7.56;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 121.13;
Eval Progress: 66/100; steps/sec: 119.04;
Eval Progress: 100/100; steps/sec: 123.78;
FastEstimator-Eval: step: 9000; epoch: 9; adv_ce: 3.745503; adversarial_accuracy: 0.2044; base_accuracy: 0.7214; base_ce: 0.87055266; min_base_ce: 0.8334195; since_best_base_ce: 2;
FastEstimator-Train: step: 9500; base_ce: 0.98312986; steps/sec: 131.19;
FastEstimator-Train: step: 10000; base_ce: 0.6474985; steps/sec: 147.32;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time(sec): 7.19;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 124.32;
Eval Progress: 66/100; steps/sec: 131.03;
Eval Progress: 100/100; steps/sec: 114.09;
FastEstimator-Eval: step: 10000; epoch: 10; adv_ce: 4.1154675; adversarial_accuracy: 0.1826; base_accuracy: 0.7176; base_ce: 0.8841103; min_base_ce: 0.8334195; since_best_base_ce: 3;
FastEstimator-Train: step: 10500; base_ce: 0.47512043; steps/sec: 140.26;
FastEstimator-Train: step: 11000; base_ce: 0.58152944; steps/sec: 159.54;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time(sec): 6.69;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 117.27;
Eval Progress: 66/100; steps/sec: 133.66;
Eval Progress: 100/100; steps/sec: 128.22;
FastEstimator-Eval: step: 11000; epoch: 11; adv_ce: 4.478873; adversarial_accuracy: 0.1806; base_accuracy: 0.7132; base_ce: 0.9255649; min_base_ce: 0.8334195; since_best_base_ce: 4;
FastEstimator-Train: step: 11500; base_ce: 0.67085576; steps/sec: 141.1;
FastEstimator-Train: step: 12000; base_ce: 0.44638634; steps/sec: 158.58;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time(sec): 6.7;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 119.56;
Eval Progress: 66/100; steps/sec: 135.7;
Eval Progress: 100/100; steps/sec: 134.19;
FastEstimator-Eval: step: 12000; epoch: 12; adv_ce: 5.1792316; adversarial_accuracy: 0.1676; base_accuracy: 0.7184; base_ce: 0.973945; min_base_ce: 0.8334195; since_best_base_ce: 5;
FastEstimator-Train: step: 12500; base_ce: 0.47898817; steps/sec: 140.41;
FastEstimator-Train: step: 13000; base_ce: 0.49361518; steps/sec: 164.74;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time(sec): 6.6;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 108.3;
Eval Progress: 66/100; steps/sec: 121.47;
Eval Progress: 100/100; steps/sec: 120.78;
FastEstimator-Eval: step: 13000; epoch: 13; adv_ce: 5.278939; adversarial_accuracy: 0.1518; base_accuracy: 0.6994; base_ce: 1.0175647; min_base_ce: 0.8334195; since_best_base_ce: 6;
FastEstimator-Train: step: 13500; base_ce: 0.39300925; steps/sec: 140.67;
FastEstimator-Train: step: 14000; base_ce: 0.39745486; steps/sec: 158.44;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time(sec): 6.71;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 110.16;
Eval Progress: 66/100; steps/sec: 129.45;
Eval Progress: 100/100; steps/sec: 130.31;
FastEstimator-Eval: step: 14000; epoch: 14; adv_ce: 5.9825478; adversarial_accuracy: 0.1608; base_accuracy: 0.7144; base_ce: 1.0789033; min_base_ce: 0.8334195; since_best_base_ce: 7;
FastEstimator-Train: step: 14500; base_ce: 0.15753141; steps/sec: 136.8;
FastEstimator-Train: step: 15000; base_ce: 0.38502944; steps/sec: 156.91;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time(sec): 6.85;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 122.61;
Eval Progress: 66/100; steps/sec: 119.23;
Eval Progress: 100/100; steps/sec: 126.33;
FastEstimator-Eval: step: 15000; epoch: 15; adv_ce: 6.3035426; adversarial_accuracy: 0.1558; base_accuracy: 0.7166; base_ce: 1.0761362; min_base_ce: 0.8334195; since_best_base_ce: 8;
FastEstimator-Train: step: 15500; base_ce: 0.38979995; steps/sec: 141.01;
FastEstimator-Train: step: 16000; base_ce: 0.33493975; steps/sec: 162.2;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time(sec): 6.62;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 117.66;
Eval Progress: 66/100; steps/sec: 121.05;
Eval Progress: 100/100; steps/sec: 133.72;
FastEstimator-Eval: step: 16000; epoch: 16; adv_ce: 6.7911468; adversarial_accuracy: 0.1532; base_accuracy: 0.7088; base_ce: 1.140474; min_base_ce: 0.8334195; since_best_base_ce: 9;
FastEstimator-Train: step: 16500; base_ce: 0.25622618; steps/sec: 131.69;
FastEstimator-Train: step: 17000; base_ce: 0.44107574; steps/sec: 149.11;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time(sec): 7.15;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 118.12;
Eval Progress: 66/100; steps/sec: 123.88;
Eval Progress: 100/100; steps/sec: 123.47;
FastEstimator-EarlyStopping: 'base_ce' triggered an early stop. Its best value was 0.8334195017814636 at epoch 7
FastEstimator-Eval: step: 17000; epoch: 17; adv_ce: 7.4295278; adversarial_accuracy: 0.1398; base_accuracy: 0.7062; base_ce: 1.224549; min_base_ce: 0.8334195; since_best_base_ce: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/softmax_best_base_ce.h5
FastEstimator-Finish: step: 17000; softmax_lr: 0.001; total_time(sec): 140.39;
FastEstimator-Test: step: 17000; epoch: 17; adv_ce: 3.1372378; adversarial_accuracy: 0.2328; base_accuracy: 0.7114; base_ce: 0.8610914;

In [13]:

ecc_estimator.fit('ECC')
ecc_results = ecc_estimator.test()

ecc_estimator.fit('ECC') ecc_results = ecc_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Warn: The following key(s) are being pruned since they are unused outside of the Pipeline. To prevent this, you can declare the key(s) as inputs to Traces or TensorOps: y
FastEstimator-Start: step: 1; logging_interval: 500; num_device: 1;
FastEstimator-Train: step: 1; base_hinge: 1.002871;
FastEstimator-Train: step: 500; base_hinge: 0.72289234; steps/sec: 154.24;
FastEstimator-Train: step: 1000; base_hinge: 0.56725734; steps/sec: 154.25;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time(sec): 7.16;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 93.7;
Eval Progress: 66/100; steps/sec: 119.5;
Eval Progress: 100/100; steps/sec: 124.16;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_hinge: 0.7502558; adversarial_accuracy: 0.2852; base_accuracy: 0.438; base_hinge: 0.63009095; min_base_hinge: 0.63009095; since_best_base_hinge: 0;
FastEstimator-Train: step: 1500; base_hinge: 0.5216874; steps/sec: 134.1;
FastEstimator-Train: step: 2000; base_hinge: 0.44142443; steps/sec: 153.21;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time(sec): 6.98;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 94.9;
Eval Progress: 66/100; steps/sec: 116.89;
Eval Progress: 100/100; steps/sec: 119.88;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_hinge: 0.68161803; adversarial_accuracy: 0.337; base_accuracy: 0.543; base_hinge: 0.48797953; min_base_hinge: 0.48797953; since_best_base_hinge: 0;
FastEstimator-Train: step: 2500; base_hinge: 0.41664636; steps/sec: 128.61;
FastEstimator-Train: step: 3000; base_hinge: 0.45047393; steps/sec: 149.96;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time(sec): 7.23;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 97.55;
Eval Progress: 66/100; steps/sec: 117.57;
Eval Progress: 100/100; steps/sec: 117.23;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_hinge: 0.6701534; adversarial_accuracy: 0.3602; base_accuracy: 0.5978; base_hinge: 0.43356347; min_base_hinge: 0.43356347; since_best_base_hinge: 0;
FastEstimator-Train: step: 3500; base_hinge: 0.4599764; steps/sec: 121.98;
FastEstimator-Train: step: 4000; base_hinge: 0.5130554; steps/sec: 148.7;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time(sec): 7.47;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 88.99;
Eval Progress: 66/100; steps/sec: 117.31;
Eval Progress: 100/100; steps/sec: 121.19;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_hinge: 0.6625683; adversarial_accuracy: 0.3612; base_accuracy: 0.6168; base_hinge: 0.41697899; min_base_hinge: 0.41697899; since_best_base_hinge: 0;
FastEstimator-Train: step: 4500; base_hinge: 0.4074202; steps/sec: 113.32;
FastEstimator-Train: step: 5000; base_hinge: 0.4268818; steps/sec: 119.4;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time(sec): 8.63;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 92.68;
Eval Progress: 66/100; steps/sec: 114.78;
Eval Progress: 100/100; steps/sec: 111.21;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_hinge: 0.66037416; adversarial_accuracy: 0.3602; base_accuracy: 0.6302; base_hinge: 0.4005047; min_base_hinge: 0.4005047; since_best_base_hinge: 0;
FastEstimator-Train: step: 5500; base_hinge: 0.30908415; steps/sec: 117.47;
FastEstimator-Train: step: 6000; base_hinge: 0.41136062; steps/sec: 143.06;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time(sec): 7.73;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 85.75;
Eval Progress: 66/100; steps/sec: 104.26;
Eval Progress: 100/100; steps/sec: 110.28;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_hinge: 0.67121375; adversarial_accuracy: 0.3472; base_accuracy: 0.6522; base_hinge: 0.38113272; min_base_hinge: 0.38113272; since_best_base_hinge: 0;
FastEstimator-Train: step: 6500; base_hinge: 0.32569647; steps/sec: 122.86;
FastEstimator-Train: step: 7000; base_hinge: 0.42650512; steps/sec: 145.74;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time(sec): 7.49;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 88.98;
Eval Progress: 66/100; steps/sec: 113.54;
Eval Progress: 100/100; steps/sec: 109.73;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_hinge: 0.65554583; adversarial_accuracy: 0.3704; base_accuracy: 0.6786; base_hinge: 0.3541754; min_base_hinge: 0.3541754; since_best_base_hinge: 0;
FastEstimator-Train: step: 7500; base_hinge: 0.4188127; steps/sec: 118.74;
FastEstimator-Train: step: 8000; base_hinge: 0.26721835; steps/sec: 140.39;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time(sec): 7.77;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 96.23;
Eval Progress: 66/100; steps/sec: 119.28;
Eval Progress: 100/100; steps/sec: 121.34;
FastEstimator-Eval: step: 8000; epoch: 8; adv_hinge: 0.66013235; adversarial_accuracy: 0.3652; base_accuracy: 0.666; base_hinge: 0.36027008; min_base_hinge: 0.3541754; since_best_base_hinge: 1;
FastEstimator-Train: step: 8500; base_hinge: 0.30995122; steps/sec: 119.83;
FastEstimator-Train: step: 9000; base_hinge: 0.36237648; steps/sec: 139.2;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time(sec): 7.76;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 84.19;
Eval Progress: 66/100; steps/sec: 117.49;
Eval Progress: 100/100; steps/sec: 114.84;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 9000; epoch: 9; adv_hinge: 0.65348905; adversarial_accuracy: 0.3794; base_accuracy: 0.6842; base_hinge: 0.34559497; min_base_hinge: 0.34559497; since_best_base_hinge: 0;
FastEstimator-Train: step: 9500; base_hinge: 0.22155131; steps/sec: 127.83;
FastEstimator-Train: step: 10000; base_hinge: 0.28609377; steps/sec: 148.96;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time(sec): 7.26;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 90.98;
Eval Progress: 66/100; steps/sec: 123.78;
Eval Progress: 100/100; steps/sec: 124.04;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 10000; epoch: 10; adv_hinge: 0.63961726; adversarial_accuracy: 0.3858; base_accuracy: 0.6906; base_hinge: 0.32274678; min_base_hinge: 0.32274678; since_best_base_hinge: 0;
FastEstimator-Train: step: 10500; base_hinge: 0.43342048; steps/sec: 126.88;
FastEstimator-Train: step: 11000; base_hinge: 0.3079724; steps/sec: 153.98;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time(sec): 7.2;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 93.1;
Eval Progress: 66/100; steps/sec: 116.38;
Eval Progress: 100/100; steps/sec: 122.85;
FastEstimator-Eval: step: 11000; epoch: 11; adv_hinge: 0.65070283; adversarial_accuracy: 0.3754; base_accuracy: 0.6908; base_hinge: 0.32702965; min_base_hinge: 0.32274678; since_best_base_hinge: 1;
FastEstimator-Train: step: 11500; base_hinge: 0.35204497; steps/sec: 135.21;
FastEstimator-Train: step: 12000; base_hinge: 0.23414919; steps/sec: 157.26;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time(sec): 6.87;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 91.58;
Eval Progress: 66/100; steps/sec: 120.0;
Eval Progress: 100/100; steps/sec: 115.88;
FastEstimator-Eval: step: 12000; epoch: 12; adv_hinge: 0.6371187; adversarial_accuracy: 0.3906; base_accuracy: 0.6842; base_hinge: 0.32811126; min_base_hinge: 0.32274678; since_best_base_hinge: 2;
FastEstimator-Train: step: 12500; base_hinge: 0.3481172; steps/sec: 127.77;
FastEstimator-Train: step: 13000; base_hinge: 0.29147673; steps/sec: 152.37;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time(sec): 7.2;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 82.55;
Eval Progress: 66/100; steps/sec: 123.04;
Eval Progress: 100/100; steps/sec: 127.1;
FastEstimator-Eval: step: 13000; epoch: 13; adv_hinge: 0.63580626; adversarial_accuracy: 0.391; base_accuracy: 0.6826; base_hinge: 0.33037513; min_base_hinge: 0.32274678; since_best_base_hinge: 3;
FastEstimator-Train: step: 13500; base_hinge: 0.20024662; steps/sec: 119.91;
FastEstimator-Train: step: 14000; base_hinge: 0.20751736; steps/sec: 143.34;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time(sec): 7.67;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 79.93;
Eval Progress: 66/100; steps/sec: 122.05;
Eval Progress: 100/100; steps/sec: 118.2;
FastEstimator-Eval: step: 14000; epoch: 14; adv_hinge: 0.6609186; adversarial_accuracy: 0.3634; base_accuracy: 0.6894; base_hinge: 0.32796183; min_base_hinge: 0.32274678; since_best_base_hinge: 4;
FastEstimator-Train: step: 14500; base_hinge: 0.18016505; steps/sec: 123.18;
FastEstimator-Train: step: 15000; base_hinge: 0.27838; steps/sec: 152.15;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time(sec): 7.34;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 75.8;
Eval Progress: 66/100; steps/sec: 120.17;
Eval Progress: 100/100; steps/sec: 110.69;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 15000; epoch: 15; adv_hinge: 0.65110445; adversarial_accuracy: 0.376; base_accuracy: 0.704; base_hinge: 0.3105381; min_base_hinge: 0.3105381; since_best_base_hinge: 0;
FastEstimator-Train: step: 15500; base_hinge: 0.23774771; steps/sec: 118.47;
FastEstimator-Train: step: 16000; base_hinge: 0.1762178; steps/sec: 145.18;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time(sec): 7.67;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 78.48;
Eval Progress: 66/100; steps/sec: 98.42;
Eval Progress: 100/100; steps/sec: 105.77;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 16000; epoch: 16; adv_hinge: 0.6461493; adversarial_accuracy: 0.3766; base_accuracy: 0.7074; base_hinge: 0.30660486; min_base_hinge: 0.30660486; since_best_base_hinge: 0;
FastEstimator-Train: step: 16500; base_hinge: 0.21141994; steps/sec: 106.26;
FastEstimator-Train: step: 17000; base_hinge: 0.20739278; steps/sec: 137.39;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time(sec): 8.35;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 80.85;
Eval Progress: 66/100; steps/sec: 124.72;
Eval Progress: 100/100; steps/sec: 118.24;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 17000; epoch: 17; adv_hinge: 0.6533371; adversarial_accuracy: 0.3718; base_accuracy: 0.7124; base_hinge: 0.30504858; min_base_hinge: 0.30504858; since_best_base_hinge: 0;
FastEstimator-Train: step: 17500; base_hinge: 0.21114208; steps/sec: 119.61;
FastEstimator-Train: step: 18000; base_hinge: 0.23942472; steps/sec: 146.56;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time(sec): 7.58;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 70.27;
Eval Progress: 66/100; steps/sec: 118.14;
Eval Progress: 100/100; steps/sec: 119.6;
FastEstimator-Eval: step: 18000; epoch: 18; adv_hinge: 0.66355395; adversarial_accuracy: 0.3658; base_accuracy: 0.6932; base_hinge: 0.3256587; min_base_hinge: 0.30504858; since_best_base_hinge: 1;
FastEstimator-Train: step: 18500; base_hinge: 0.18134522; steps/sec: 118.93;
FastEstimator-Train: step: 19000; base_hinge: 0.15870592; steps/sec: 152.58;
FastEstimator-Train: step: 19000; epoch: 19; epoch_time(sec): 7.49;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 74.53;
Eval Progress: 66/100; steps/sec: 117.47;
Eval Progress: 100/100; steps/sec: 101.21;
FastEstimator-Eval: step: 19000; epoch: 19; adv_hinge: 0.6584557; adversarial_accuracy: 0.3666; base_accuracy: 0.7072; base_hinge: 0.30712467; min_base_hinge: 0.30504858; since_best_base_hinge: 2;
FastEstimator-Train: step: 19500; base_hinge: 0.13574408; steps/sec: 127.74;
FastEstimator-Train: step: 20000; base_hinge: 0.20718066; steps/sec: 150.91;
FastEstimator-Train: step: 20000; epoch: 20; epoch_time(sec): 7.22;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 71.35;
Eval Progress: 66/100; steps/sec: 117.44;
Eval Progress: 100/100; steps/sec: 120.15;
FastEstimator-Eval: step: 20000; epoch: 20; adv_hinge: 0.6691775; adversarial_accuracy: 0.3612; base_accuracy: 0.7098; base_hinge: 0.30683315; min_base_hinge: 0.30504858; since_best_base_hinge: 3;
FastEstimator-Train: step: 20500; base_hinge: 0.24493937; steps/sec: 122.09;
FastEstimator-Train: step: 21000; base_hinge: 0.23800884; steps/sec: 146.09;
FastEstimator-Train: step: 21000; epoch: 21; epoch_time(sec): 7.54;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.41;
Eval Progress: 66/100; steps/sec: 122.68;
Eval Progress: 100/100; steps/sec: 123.88;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 21000; epoch: 21; adv_hinge: 0.64913946; adversarial_accuracy: 0.3784; base_accuracy: 0.7108; base_hinge: 0.30218562; min_base_hinge: 0.30218562; since_best_base_hinge: 0;
FastEstimator-Train: step: 21500; base_hinge: 0.21396293; steps/sec: 118.23;
FastEstimator-Train: step: 22000; base_hinge: 0.24212593; steps/sec: 145.96;
FastEstimator-Train: step: 22000; epoch: 22; epoch_time(sec): 7.63;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 75.15;
Eval Progress: 66/100; steps/sec: 124.16;
Eval Progress: 100/100; steps/sec: 125.71;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 22000; epoch: 22; adv_hinge: 0.67573655; adversarial_accuracy: 0.3514; base_accuracy: 0.7102; base_hinge: 0.3012932; min_base_hinge: 0.3012932; since_best_base_hinge: 0;
FastEstimator-Train: step: 22500; base_hinge: 0.1826303; steps/sec: 121.78;
FastEstimator-Train: step: 23000; base_hinge: 0.13290118; steps/sec: 151.53;
FastEstimator-Train: step: 23000; epoch: 23; epoch_time(sec): 7.41;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 77.77;
Eval Progress: 66/100; steps/sec: 111.69;
Eval Progress: 100/100; steps/sec: 118.55;
FastEstimator-Eval: step: 23000; epoch: 23; adv_hinge: 0.67135483; adversarial_accuracy: 0.3614; base_accuracy: 0.7052; base_hinge: 0.3083685; min_base_hinge: 0.3012932; since_best_base_hinge: 1;
FastEstimator-Train: step: 23500; base_hinge: 0.18338877; steps/sec: 121.98;
FastEstimator-Train: step: 24000; base_hinge: 0.2141859; steps/sec: 153.08;
FastEstimator-Train: step: 24000; epoch: 24; epoch_time(sec): 7.36;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 77.79;
Eval Progress: 66/100; steps/sec: 120.18;
Eval Progress: 100/100; steps/sec: 126.18;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 24000; epoch: 24; adv_hinge: 0.6686717; adversarial_accuracy: 0.36; base_accuracy: 0.7198; base_hinge: 0.29314205; min_base_hinge: 0.29314205; since_best_base_hinge: 0;
FastEstimator-Train: step: 24500; base_hinge: 0.1325094; steps/sec: 121.1;
FastEstimator-Train: step: 25000; base_hinge: 0.2229343; steps/sec: 151.23;
FastEstimator-Train: step: 25000; epoch: 25; epoch_time(sec): 7.43;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 69.32;
Eval Progress: 66/100; steps/sec: 103.24;
Eval Progress: 100/100; steps/sec: 109.61;
FastEstimator-Eval: step: 25000; epoch: 25; adv_hinge: 0.65916747; adversarial_accuracy: 0.3696; base_accuracy: 0.7112; base_hinge: 0.30554205; min_base_hinge: 0.29314205; since_best_base_hinge: 1;
FastEstimator-Train: step: 25500; base_hinge: 0.1662754; steps/sec: 122.47;
FastEstimator-Train: step: 26000; base_hinge: 0.19041198; steps/sec: 145.21;
FastEstimator-Train: step: 26000; epoch: 26; epoch_time(sec): 7.53;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 72.0;
Eval Progress: 66/100; steps/sec: 122.62;
Eval Progress: 100/100; steps/sec: 114.06;
FastEstimator-Eval: step: 26000; epoch: 26; adv_hinge: 0.6587195; adversarial_accuracy: 0.37; base_accuracy: 0.7172; base_hinge: 0.2967423; min_base_hinge: 0.29314205; since_best_base_hinge: 2;
FastEstimator-Train: step: 26500; base_hinge: 0.13334934; steps/sec: 117.63;
FastEstimator-Train: step: 27000; base_hinge: 0.14624205; steps/sec: 139.45;
FastEstimator-Train: step: 27000; epoch: 27; epoch_time(sec): 7.84;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 78.22;
Eval Progress: 66/100; steps/sec: 121.65;
Eval Progress: 100/100; steps/sec: 114.53;
FastEstimator-Eval: step: 27000; epoch: 27; adv_hinge: 0.65940183; adversarial_accuracy: 0.3736; base_accuracy: 0.7134; base_hinge: 0.30316037; min_base_hinge: 0.29314205; since_best_base_hinge: 3;
FastEstimator-Train: step: 27500; base_hinge: 0.14235954; steps/sec: 115.35;
FastEstimator-Train: step: 28000; base_hinge: 0.1803187; steps/sec: 137.33;
FastEstimator-Train: step: 28000; epoch: 28; epoch_time(sec): 7.97;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.09;
Eval Progress: 66/100; steps/sec: 116.92;
Eval Progress: 100/100; steps/sec: 122.3;
FastEstimator-Eval: step: 28000; epoch: 28; adv_hinge: 0.69258124; adversarial_accuracy: 0.3386; base_accuracy: 0.7; base_hinge: 0.31553543; min_base_hinge: 0.29314205; since_best_base_hinge: 4;
FastEstimator-Train: step: 28500; base_hinge: 0.13245791; steps/sec: 120.74;
FastEstimator-Train: step: 29000; base_hinge: 0.26103622; steps/sec: 152.26;
FastEstimator-Train: step: 29000; epoch: 29; epoch_time(sec): 7.43;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 75.44;
Eval Progress: 66/100; steps/sec: 123.04;
Eval Progress: 100/100; steps/sec: 122.7;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 29000; epoch: 29; adv_hinge: 0.65981364; adversarial_accuracy: 0.3726; base_accuracy: 0.727; base_hinge: 0.29011464; min_base_hinge: 0.29011464; since_best_base_hinge: 0;
FastEstimator-Train: step: 29500; base_hinge: 0.18364395; steps/sec: 121.27;
FastEstimator-Train: step: 30000; base_hinge: 0.17344308; steps/sec: 146.39;
FastEstimator-Train: step: 30000; epoch: 30; epoch_time(sec): 7.53;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.58;
Eval Progress: 66/100; steps/sec: 121.82;
Eval Progress: 100/100; steps/sec: 122.94;
FastEstimator-Eval: step: 30000; epoch: 30; adv_hinge: 0.6824101; adversarial_accuracy: 0.3488; base_accuracy: 0.7096; base_hinge: 0.30300286; min_base_hinge: 0.29011464; since_best_base_hinge: 1;
FastEstimator-Train: step: 30500; base_hinge: 0.14005479; steps/sec: 118.18;
FastEstimator-Train: step: 31000; base_hinge: 0.18205415; steps/sec: 149.06;
FastEstimator-Train: step: 31000; epoch: 31; epoch_time(sec): 7.6;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.81;
Eval Progress: 66/100; steps/sec: 114.98;
Eval Progress: 100/100; steps/sec: 124.63;
FastEstimator-Eval: step: 31000; epoch: 31; adv_hinge: 0.6711335; adversarial_accuracy: 0.3612; base_accuracy: 0.711; base_hinge: 0.30420983; min_base_hinge: 0.29011464; since_best_base_hinge: 2;
FastEstimator-Train: step: 31500; base_hinge: 0.14259018; steps/sec: 120.81;
FastEstimator-Train: step: 32000; base_hinge: 0.11322089; steps/sec: 149.91;
FastEstimator-Train: step: 32000; epoch: 32; epoch_time(sec): 7.46;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 72.81;
Eval Progress: 66/100; steps/sec: 114.35;
Eval Progress: 100/100; steps/sec: 122.16;
FastEstimator-Eval: step: 32000; epoch: 32; adv_hinge: 0.67378473; adversarial_accuracy: 0.3582; base_accuracy: 0.7114; base_hinge: 0.30332875; min_base_hinge: 0.29011464; since_best_base_hinge: 3;
FastEstimator-Train: step: 32500; base_hinge: 0.11214094; steps/sec: 122.67;
FastEstimator-Train: step: 33000; base_hinge: 0.1267754; steps/sec: 149.69;
FastEstimator-Train: step: 33000; epoch: 33; epoch_time(sec): 7.41;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 77.8;
Eval Progress: 66/100; steps/sec: 108.11;
Eval Progress: 100/100; steps/sec: 107.83;
FastEstimator-Eval: step: 33000; epoch: 33; adv_hinge: 0.6758255; adversarial_accuracy: 0.3566; base_accuracy: 0.7202; base_hinge: 0.2957484; min_base_hinge: 0.29011464; since_best_base_hinge: 4;
FastEstimator-Train: step: 33500; base_hinge: 0.16455775; steps/sec: 120.7;
FastEstimator-Train: step: 34000; base_hinge: 0.10459029; steps/sec: 141.1;
FastEstimator-Train: step: 34000; epoch: 34; epoch_time(sec): 7.7;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.38;
Eval Progress: 66/100; steps/sec: 117.06;
Eval Progress: 100/100; steps/sec: 114.14;
FastEstimator-Eval: step: 34000; epoch: 34; adv_hinge: 0.6677006; adversarial_accuracy: 0.3658; base_accuracy: 0.7164; base_hinge: 0.299565; min_base_hinge: 0.29011464; since_best_base_hinge: 5;
FastEstimator-Train: step: 34500; base_hinge: 0.23067388; steps/sec: 119.58;
FastEstimator-Train: step: 35000; base_hinge: 0.20918787; steps/sec: 141.49;
FastEstimator-Train: step: 35000; epoch: 35; epoch_time(sec): 7.7;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 80.8;
Eval Progress: 66/100; steps/sec: 122.93;
Eval Progress: 100/100; steps/sec: 125.42;
FastEstimator-Eval: step: 35000; epoch: 35; adv_hinge: 0.6804726; adversarial_accuracy: 0.3538; base_accuracy: 0.7064; base_hinge: 0.30698383; min_base_hinge: 0.29011464; since_best_base_hinge: 6;
FastEstimator-Train: step: 35500; base_hinge: 0.17706497; steps/sec: 117.24;
FastEstimator-Train: step: 36000; base_hinge: 0.16568424; steps/sec: 145.52;
FastEstimator-Train: step: 36000; epoch: 36; epoch_time(sec): 7.72;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 77.54;
Eval Progress: 66/100; steps/sec: 124.56;
Eval Progress: 100/100; steps/sec: 124.74;
FastEstimator-Eval: step: 36000; epoch: 36; adv_hinge: 0.6806769; adversarial_accuracy: 0.351; base_accuracy: 0.7164; base_hinge: 0.29772627; min_base_hinge: 0.29011464; since_best_base_hinge: 7;
FastEstimator-Train: step: 36500; base_hinge: 0.25378636; steps/sec: 123.75;
FastEstimator-Train: step: 37000; base_hinge: 0.13252667; steps/sec: 152.8;
FastEstimator-Train: step: 37000; epoch: 37; epoch_time(sec): 7.31;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 78.51;
Eval Progress: 66/100; steps/sec: 127.18;
Eval Progress: 100/100; steps/sec: 126.07;
FastEstimator-Eval: step: 37000; epoch: 37; adv_hinge: 0.68055105; adversarial_accuracy: 0.3504; base_accuracy: 0.7078; base_hinge: 0.30485085; min_base_hinge: 0.29011464; since_best_base_hinge: 8;
FastEstimator-Train: step: 37500; base_hinge: 0.15213989; steps/sec: 120.86;
FastEstimator-Train: step: 38000; base_hinge: 0.12855332; steps/sec: 140.38;
FastEstimator-Train: step: 38000; epoch: 38; epoch_time(sec): 7.71;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 70.03;
Eval Progress: 66/100; steps/sec: 120.87;
Eval Progress: 100/100; steps/sec: 121.3;
FastEstimator-Eval: step: 38000; epoch: 38; adv_hinge: 0.6721651; adversarial_accuracy: 0.3578; base_accuracy: 0.7184; base_hinge: 0.2957858; min_base_hinge: 0.29011464; since_best_base_hinge: 9;
FastEstimator-Train: step: 38500; base_hinge: 0.18033153; steps/sec: 108.43;
FastEstimator-Train: step: 39000; base_hinge: 0.15155438; steps/sec: 143.18;
FastEstimator-Train: step: 39000; epoch: 39; epoch_time(sec): 8.1;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 74.85;
Eval Progress: 66/100; steps/sec: 125.11;
Eval Progress: 100/100; steps/sec: 117.82;
FastEstimator-EarlyStopping: 'base_hinge' triggered an early stop. Its best value was 0.2901146411895752 at epoch 29
FastEstimator-Eval: step: 39000; epoch: 39; adv_hinge: 0.7088697; adversarial_accuracy: 0.3278; base_accuracy: 0.6944; base_hinge: 0.3195179; min_base_hinge: 0.29011464; since_best_base_hinge: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/ecc_best_base_hinge.h5
FastEstimator-Finish: step: 39000; ecc_lr: 0.001; total_time(sec): 361.11;
FastEstimator-Test: step: 39000; epoch: 39; adv_hinge: 0.66619796; adversarial_accuracy: 0.363; base_accuracy: 0.7164; base_hinge: 0.2986943;

In [14]:

hydra_estimator.fit('Hydra')
hydra_results = hydra_estimator.test()

hydra_estimator.fit('Hydra') hydra_results = hydra_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; logging_interval: 500; num_device: 1;
FastEstimator-Train: step: 1; base_hinge: 0.9966292;
FastEstimator-Train: step: 500; base_hinge: 0.7640507; steps/sec: 88.76;
FastEstimator-Train: step: 1000; base_hinge: 0.678862; steps/sec: 95.95;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time(sec): 12.0;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 47.58;
Eval Progress: 66/100; steps/sec: 95.06;
Eval Progress: 100/100; steps/sec: 86.62;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_hinge: 0.7892957; adversarial_accuracy: 0.2276; base_accuracy: 0.3724; base_hinge: 0.68919855; min_base_hinge: 0.68919855; since_best_base_hinge: 0;
FastEstimator-Train: step: 1500; base_hinge: 0.72929424; steps/sec: 72.07;
FastEstimator-Train: step: 2000; base_hinge: 0.59415627; steps/sec: 89.09;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time(sec): 12.56;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 46.36;
Eval Progress: 66/100; steps/sec: 83.63;
Eval Progress: 100/100; steps/sec: 85.94;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_hinge: 0.7073509; adversarial_accuracy: 0.3144; base_accuracy: 0.4748; base_hinge: 0.5738656; min_base_hinge: 0.5738656; since_best_base_hinge: 0;
FastEstimator-Train: step: 2500; base_hinge: 0.6003836; steps/sec: 81.03;
FastEstimator-Train: step: 3000; base_hinge: 0.45942596; steps/sec: 95.35;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time(sec): 11.4;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.71;
Eval Progress: 66/100; steps/sec: 95.78;
Eval Progress: 100/100; steps/sec: 94.6;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_hinge: 0.67819905; adversarial_accuracy: 0.3482; base_accuracy: 0.5504; base_hinge: 0.4906526; min_base_hinge: 0.4906526; since_best_base_hinge: 0;
FastEstimator-Train: step: 3500; base_hinge: 0.5691358; steps/sec: 76.3;
FastEstimator-Train: step: 4000; base_hinge: 0.4292397; steps/sec: 97.81;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time(sec): 11.67;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 46.91;
Eval Progress: 66/100; steps/sec: 92.21;
Eval Progress: 100/100; steps/sec: 95.91;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_hinge: 0.6486282; adversarial_accuracy: 0.379; base_accuracy: 0.5894; base_hinge: 0.44183797; min_base_hinge: 0.44183797; since_best_base_hinge: 0;
FastEstimator-Train: step: 4500; base_hinge: 0.40855202; steps/sec: 70.04;
FastEstimator-Train: step: 5000; base_hinge: 0.42965376; steps/sec: 86.23;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time(sec): 12.95;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 44.04;
Eval Progress: 66/100; steps/sec: 91.82;
Eval Progress: 100/100; steps/sec: 87.96;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_hinge: 0.66325796; adversarial_accuracy: 0.3604; base_accuracy: 0.6144; base_hinge: 0.41609535; min_base_hinge: 0.41609535; since_best_base_hinge: 0;
FastEstimator-Train: step: 5500; base_hinge: 0.43601158; steps/sec: 70.72;
FastEstimator-Train: step: 6000; base_hinge: 0.37749556; steps/sec: 86.44;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time(sec): 12.86;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 44.93;
Eval Progress: 66/100; steps/sec: 95.02;
Eval Progress: 100/100; steps/sec: 94.26;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_hinge: 0.6520918; adversarial_accuracy: 0.365; base_accuracy: 0.6316; base_hinge: 0.40041855; min_base_hinge: 0.40041855; since_best_base_hinge: 0;
FastEstimator-Train: step: 6500; base_hinge: 0.35293883; steps/sec: 72.96;
FastEstimator-Train: step: 7000; base_hinge: 0.3652949; steps/sec: 84.47;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time(sec): 12.76;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.61;
Eval Progress: 66/100; steps/sec: 82.6;
Eval Progress: 100/100; steps/sec: 81.61;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_hinge: 0.6460684; adversarial_accuracy: 0.3804; base_accuracy: 0.6692; base_hinge: 0.36885917; min_base_hinge: 0.36885917; since_best_base_hinge: 0;
FastEstimator-Train: step: 7500; base_hinge: 0.28133556; steps/sec: 66.17;
FastEstimator-Train: step: 8000; base_hinge: 0.30187488; steps/sec: 85.54;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time(sec): 13.4;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.85;
Eval Progress: 66/100; steps/sec: 89.6;
Eval Progress: 100/100; steps/sec: 93.41;
FastEstimator-Eval: step: 8000; epoch: 8; adv_hinge: 0.65356743; adversarial_accuracy: 0.3694; base_accuracy: 0.6588; base_hinge: 0.37363842; min_base_hinge: 0.36885917; since_best_base_hinge: 1;
FastEstimator-Train: step: 8500; base_hinge: 0.24869174; steps/sec: 77.01;
FastEstimator-Train: step: 9000; base_hinge: 0.27774674; steps/sec: 99.1;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time(sec): 11.55;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.49;
Eval Progress: 66/100; steps/sec: 88.82;
Eval Progress: 100/100; steps/sec: 93.31;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 9000; epoch: 9; adv_hinge: 0.6346296; adversarial_accuracy: 0.3852; base_accuracy: 0.6842; base_hinge: 0.3505857; min_base_hinge: 0.3505857; since_best_base_hinge: 0;
FastEstimator-Train: step: 9500; base_hinge: 0.30322844; steps/sec: 75.97;
FastEstimator-Train: step: 10000; base_hinge: 0.33541998; steps/sec: 95.09;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time(sec): 11.85;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.52;
Eval Progress: 66/100; steps/sec: 89.96;
Eval Progress: 100/100; steps/sec: 87.1;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 10000; epoch: 10; adv_hinge: 0.62861943; adversarial_accuracy: 0.3906; base_accuracy: 0.6884; base_hinge: 0.34795314; min_base_hinge: 0.34795314; since_best_base_hinge: 0;
FastEstimator-Train: step: 10500; base_hinge: 0.39936805; steps/sec: 76.76;
FastEstimator-Train: step: 11000; base_hinge: 0.21335937; steps/sec: 99.12;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time(sec): 11.54;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.67;
Eval Progress: 66/100; steps/sec: 86.46;
Eval Progress: 100/100; steps/sec: 91.32;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 11000; epoch: 11; adv_hinge: 0.64452505; adversarial_accuracy: 0.3758; base_accuracy: 0.694; base_hinge: 0.3412743; min_base_hinge: 0.3412743; since_best_base_hinge: 0;
FastEstimator-Train: step: 11500; base_hinge: 0.28195524; steps/sec: 70.92;
FastEstimator-Train: step: 12000; base_hinge: 0.34409276; steps/sec: 90.56;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time(sec): 12.6;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.64;
Eval Progress: 66/100; steps/sec: 93.23;
Eval Progress: 100/100; steps/sec: 84.78;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 12000; epoch: 12; adv_hinge: 0.63986564; adversarial_accuracy: 0.381; base_accuracy: 0.6958; base_hinge: 0.3376504; min_base_hinge: 0.3376504; since_best_base_hinge: 0;
FastEstimator-Train: step: 12500; base_hinge: 0.26378664; steps/sec: 73.0;
FastEstimator-Train: step: 13000; base_hinge: 0.26071703; steps/sec: 93.65;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time(sec): 12.2;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 43.47;
Eval Progress: 66/100; steps/sec: 98.01;
Eval Progress: 100/100; steps/sec: 88.9;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 13000; epoch: 13; adv_hinge: 0.63786435; adversarial_accuracy: 0.3826; base_accuracy: 0.7072; base_hinge: 0.32933062; min_base_hinge: 0.32933062; since_best_base_hinge: 0;
FastEstimator-Train: step: 13500; base_hinge: 0.27945766; steps/sec: 74.54;
FastEstimator-Train: step: 14000; base_hinge: 0.23886347; steps/sec: 92.94;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time(sec): 12.05;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.71;
Eval Progress: 66/100; steps/sec: 96.92;
Eval Progress: 100/100; steps/sec: 93.97;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 14000; epoch: 14; adv_hinge: 0.64192384; adversarial_accuracy: 0.3696; base_accuracy: 0.7108; base_hinge: 0.32311305; min_base_hinge: 0.32311305; since_best_base_hinge: 0;
FastEstimator-Train: step: 14500; base_hinge: 0.24576215; steps/sec: 73.15;
FastEstimator-Train: step: 15000; base_hinge: 0.27633974; steps/sec: 92.15;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time(sec): 12.26;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.45;
Eval Progress: 66/100; steps/sec: 84.14;
Eval Progress: 100/100; steps/sec: 89.04;
FastEstimator-Eval: step: 15000; epoch: 15; adv_hinge: 0.6450048; adversarial_accuracy: 0.372; base_accuracy: 0.7064; base_hinge: 0.32941842; min_base_hinge: 0.32311305; since_best_base_hinge: 1;
FastEstimator-Train: step: 15500; base_hinge: 0.2711668; steps/sec: 76.72;
FastEstimator-Train: step: 16000; base_hinge: 0.28701916; steps/sec: 98.65;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time(sec): 11.58;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.84;
Eval Progress: 66/100; steps/sec: 97.91;
Eval Progress: 100/100; steps/sec: 96.29;
FastEstimator-Eval: step: 16000; epoch: 16; adv_hinge: 0.64609456; adversarial_accuracy: 0.3714; base_accuracy: 0.7156; base_hinge: 0.3243042; min_base_hinge: 0.32311305; since_best_base_hinge: 2;
FastEstimator-Train: step: 16500; base_hinge: 0.26900485; steps/sec: 74.6;
FastEstimator-Train: step: 17000; base_hinge: 0.23061709; steps/sec: 95.86;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time(sec): 11.94;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.1;
Eval Progress: 66/100; steps/sec: 90.02;
Eval Progress: 100/100; steps/sec: 92.33;
FastEstimator-Eval: step: 17000; epoch: 17; adv_hinge: 0.64359355; adversarial_accuracy: 0.3784; base_accuracy: 0.7072; base_hinge: 0.32444945; min_base_hinge: 0.32311305; since_best_base_hinge: 3;
FastEstimator-Train: step: 17500; base_hinge: 0.25468764; steps/sec: 66.95;
FastEstimator-Train: step: 18000; base_hinge: 0.26488352; steps/sec: 93.61;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time(sec): 12.8;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 37.85;
Eval Progress: 66/100; steps/sec: 91.58;
Eval Progress: 100/100; steps/sec: 91.14;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 18000; epoch: 18; adv_hinge: 0.6398344; adversarial_accuracy: 0.38; base_accuracy: 0.7162; base_hinge: 0.3198119; min_base_hinge: 0.3198119; since_best_base_hinge: 0;
FastEstimator-Train: step: 18500; base_hinge: 0.3105155; steps/sec: 70.95;
FastEstimator-Train: step: 19000; base_hinge: 0.20092575; steps/sec: 93.74;
FastEstimator-Train: step: 19000; epoch: 19; epoch_time(sec): 12.37;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.63;
Eval Progress: 66/100; steps/sec: 98.2;
Eval Progress: 100/100; steps/sec: 96.05;
FastEstimator-Eval: step: 19000; epoch: 19; adv_hinge: 0.6472323; adversarial_accuracy: 0.3758; base_accuracy: 0.709; base_hinge: 0.32192203; min_base_hinge: 0.3198119; since_best_base_hinge: 1;
FastEstimator-Train: step: 19500; base_hinge: 0.28648773; steps/sec: 77.29;
FastEstimator-Train: step: 20000; base_hinge: 0.24984874; steps/sec: 98.3;
FastEstimator-Train: step: 20000; epoch: 20; epoch_time(sec): 11.55;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.24;
Eval Progress: 66/100; steps/sec: 87.18;
Eval Progress: 100/100; steps/sec: 84.34;
FastEstimator-Eval: step: 20000; epoch: 20; adv_hinge: 0.64978516; adversarial_accuracy: 0.372; base_accuracy: 0.7094; base_hinge: 0.3221413; min_base_hinge: 0.3198119; since_best_base_hinge: 2;
FastEstimator-Train: step: 20500; base_hinge: 0.21497366; steps/sec: 68.47;
FastEstimator-Train: step: 21000; base_hinge: 0.25525752; steps/sec: 84.8;
FastEstimator-Train: step: 21000; epoch: 21; epoch_time(sec): 13.21;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.43;
Eval Progress: 66/100; steps/sec: 92.85;
Eval Progress: 100/100; steps/sec: 85.01;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 21000; epoch: 21; adv_hinge: 0.64389527; adversarial_accuracy: 0.3786; base_accuracy: 0.7232; base_hinge: 0.3089102; min_base_hinge: 0.3089102; since_best_base_hinge: 0;
FastEstimator-Train: step: 21500; base_hinge: 0.19983594; steps/sec: 70.82;
FastEstimator-Train: step: 22000; base_hinge: 0.22720769; steps/sec: 91.67;
FastEstimator-Train: step: 22000; epoch: 22; epoch_time(sec): 12.51;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.24;
Eval Progress: 66/100; steps/sec: 87.55;
Eval Progress: 100/100; steps/sec: 87.26;
FastEstimator-Eval: step: 22000; epoch: 22; adv_hinge: 0.65149325; adversarial_accuracy: 0.374; base_accuracy: 0.7184; base_hinge: 0.311008; min_base_hinge: 0.3089102; since_best_base_hinge: 1;
FastEstimator-Train: step: 22500; base_hinge: 0.19273132; steps/sec: 70.74;
FastEstimator-Train: step: 23000; base_hinge: 0.2419047; steps/sec: 95.4;
FastEstimator-Train: step: 23000; epoch: 23; epoch_time(sec): 12.31;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 43.09;
Eval Progress: 66/100; steps/sec: 88.46;
Eval Progress: 100/100; steps/sec: 88.61;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 23000; epoch: 23; adv_hinge: 0.6482588; adversarial_accuracy: 0.3766; base_accuracy: 0.7254; base_hinge: 0.3053934; min_base_hinge: 0.3053934; since_best_base_hinge: 0;
FastEstimator-Train: step: 23500; base_hinge: 0.13282509; steps/sec: 73.06;
FastEstimator-Train: step: 24000; base_hinge: 0.1860265; steps/sec: 90.72;
FastEstimator-Train: step: 24000; epoch: 24; epoch_time(sec): 12.35;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.21;
Eval Progress: 66/100; steps/sec: 83.85;
Eval Progress: 100/100; steps/sec: 84.4;
FastEstimator-Eval: step: 24000; epoch: 24; adv_hinge: 0.6556362; adversarial_accuracy: 0.3646; base_accuracy: 0.7216; base_hinge: 0.3093628; min_base_hinge: 0.3053934; since_best_base_hinge: 1;
FastEstimator-Train: step: 24500; base_hinge: 0.27858675; steps/sec: 68.85;
FastEstimator-Train: step: 25000; base_hinge: 0.1775538; steps/sec: 83.87;
FastEstimator-Train: step: 25000; epoch: 25; epoch_time(sec): 13.23;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.92;
Eval Progress: 66/100; steps/sec: 87.18;
Eval Progress: 100/100; steps/sec: 86.56;
FastEstimator-Eval: step: 25000; epoch: 25; adv_hinge: 0.6680432; adversarial_accuracy: 0.3564; base_accuracy: 0.7172; base_hinge: 0.30669445; min_base_hinge: 0.3053934; since_best_base_hinge: 2;
FastEstimator-Train: step: 25500; base_hinge: 0.15762655; steps/sec: 70.95;
FastEstimator-Train: step: 26000; base_hinge: 0.25477043; steps/sec: 86.83;
FastEstimator-Train: step: 26000; epoch: 26; epoch_time(sec): 12.81;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.06;
Eval Progress: 66/100; steps/sec: 80.25;
Eval Progress: 100/100; steps/sec: 86.65;
FastEstimator-Eval: step: 26000; epoch: 26; adv_hinge: 0.66914505; adversarial_accuracy: 0.3594; base_accuracy: 0.7186; base_hinge: 0.30710185; min_base_hinge: 0.3053934; since_best_base_hinge: 3;
FastEstimator-Train: step: 26500; base_hinge: 0.14826104; steps/sec: 71.06;
FastEstimator-Train: step: 27000; base_hinge: 0.32448024; steps/sec: 87.18;
FastEstimator-Train: step: 27000; epoch: 27; epoch_time(sec): 12.78;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 41.94;
Eval Progress: 66/100; steps/sec: 93.14;
Eval Progress: 100/100; steps/sec: 94.98;
FastEstimator-Eval: step: 27000; epoch: 27; adv_hinge: 0.6589374; adversarial_accuracy: 0.3638; base_accuracy: 0.72; base_hinge: 0.30622676; min_base_hinge: 0.3053934; since_best_base_hinge: 4;
FastEstimator-Train: step: 27500; base_hinge: 0.22037223; steps/sec: 68.0;
FastEstimator-Train: step: 28000; base_hinge: 0.2091624; steps/sec: 90.74;
FastEstimator-Train: step: 28000; epoch: 28; epoch_time(sec): 12.87;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 42.03;
Eval Progress: 66/100; steps/sec: 93.39;
Eval Progress: 100/100; steps/sec: 92.38;
FastEstimator-Eval: step: 28000; epoch: 28; adv_hinge: 0.6687555; adversarial_accuracy: 0.3536; base_accuracy: 0.72; base_hinge: 0.30867222; min_base_hinge: 0.3053934; since_best_base_hinge: 5;
FastEstimator-Train: step: 28500; base_hinge: 0.1658495; steps/sec: 66.64;
FastEstimator-Train: step: 29000; base_hinge: 0.20974766; steps/sec: 96.11;
FastEstimator-Train: step: 29000; epoch: 29; epoch_time(sec): 12.71;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.53;
Eval Progress: 66/100; steps/sec: 86.81;
Eval Progress: 100/100; steps/sec: 89.44;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 29000; epoch: 29; adv_hinge: 0.6557974; adversarial_accuracy: 0.3644; base_accuracy: 0.7238; base_hinge: 0.30286953; min_base_hinge: 0.30286953; since_best_base_hinge: 0;
FastEstimator-Train: step: 29500; base_hinge: 0.19858567; steps/sec: 72.71;
FastEstimator-Train: step: 30000; base_hinge: 0.16443518; steps/sec: 93.92;
FastEstimator-Train: step: 30000; epoch: 30; epoch_time(sec): 12.18;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 43.31;
Eval Progress: 66/100; steps/sec: 94.79;
Eval Progress: 100/100; steps/sec: 96.85;
FastEstimator-Eval: step: 30000; epoch: 30; adv_hinge: 0.6945098; adversarial_accuracy: 0.3294; base_accuracy: 0.7194; base_hinge: 0.30769223; min_base_hinge: 0.30286953; since_best_base_hinge: 1;
FastEstimator-Train: step: 30500; base_hinge: 0.25927573; steps/sec: 74.83;
FastEstimator-Train: step: 31000; base_hinge: 0.15841994; steps/sec: 95.38;
FastEstimator-Train: step: 31000; epoch: 31; epoch_time(sec): 11.92;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 43.63;
Eval Progress: 66/100; steps/sec: 93.88;
Eval Progress: 100/100; steps/sec: 95.78;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 31000; epoch: 31; adv_hinge: 0.6700045; adversarial_accuracy: 0.3576; base_accuracy: 0.7236; base_hinge: 0.300413; min_base_hinge: 0.300413; since_best_base_hinge: 0;
FastEstimator-Train: step: 31500; base_hinge: 0.15759675; steps/sec: 74.73;
FastEstimator-Train: step: 32000; base_hinge: 0.18248507; steps/sec: 94.03;
FastEstimator-Train: step: 32000; epoch: 32; epoch_time(sec): 12.01;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 43.97;
Eval Progress: 66/100; steps/sec: 97.59;
Eval Progress: 100/100; steps/sec: 96.51;
FastEstimator-Eval: step: 32000; epoch: 32; adv_hinge: 0.68011796; adversarial_accuracy: 0.3394; base_accuracy: 0.719; base_hinge: 0.30457467; min_base_hinge: 0.300413; since_best_base_hinge: 1;
FastEstimator-Train: step: 32500; base_hinge: 0.20276971; steps/sec: 73.23;
FastEstimator-Train: step: 33000; base_hinge: 0.20559607; steps/sec: 90.42;
FastEstimator-Train: step: 33000; epoch: 33; epoch_time(sec): 12.37;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 42.65;
Eval Progress: 66/100; steps/sec: 88.78;
Eval Progress: 100/100; steps/sec: 89.26;
FastEstimator-Eval: step: 33000; epoch: 33; adv_hinge: 0.66992426; adversarial_accuracy: 0.3572; base_accuracy: 0.725; base_hinge: 0.3011261; min_base_hinge: 0.300413; since_best_base_hinge: 2;
FastEstimator-Train: step: 33500; base_hinge: 0.1434363; steps/sec: 76.12;
FastEstimator-Train: step: 34000; base_hinge: 0.17375213; steps/sec: 95.94;
FastEstimator-Train: step: 34000; epoch: 34; epoch_time(sec): 11.79;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 44.27;
Eval Progress: 66/100; steps/sec: 94.42;
Eval Progress: 100/100; steps/sec: 95.7;
FastEstimator-Eval: step: 34000; epoch: 34; adv_hinge: 0.6889131; adversarial_accuracy: 0.3308; base_accuracy: 0.7228; base_hinge: 0.30551085; min_base_hinge: 0.300413; since_best_base_hinge: 3;
FastEstimator-Train: step: 34500; base_hinge: 0.18364738; steps/sec: 78.64;
FastEstimator-Train: step: 35000; base_hinge: 0.19179207; steps/sec: 99.44;
FastEstimator-Train: step: 35000; epoch: 35; epoch_time(sec): 11.37;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 42.28;
Eval Progress: 66/100; steps/sec: 93.18;
Eval Progress: 100/100; steps/sec: 95.3;
FastEstimator-Eval: step: 35000; epoch: 35; adv_hinge: 0.68269205; adversarial_accuracy: 0.3424; base_accuracy: 0.722; base_hinge: 0.30776957; min_base_hinge: 0.300413; since_best_base_hinge: 4;
FastEstimator-Train: step: 35500; base_hinge: 0.15180495; steps/sec: 77.82;
FastEstimator-Train: step: 36000; base_hinge: 0.16076851; steps/sec: 97.57;
FastEstimator-Train: step: 36000; epoch: 36; epoch_time(sec): 11.54;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 43.71;
Eval Progress: 66/100; steps/sec: 87.02;
Eval Progress: 100/100; steps/sec: 88.45;
FastEstimator-Eval: step: 36000; epoch: 36; adv_hinge: 0.6857431; adversarial_accuracy: 0.339; base_accuracy: 0.7216; base_hinge: 0.30552375; min_base_hinge: 0.300413; since_best_base_hinge: 5;
FastEstimator-Train: step: 36500; base_hinge: 0.18327242; steps/sec: 74.79;
FastEstimator-Train: step: 37000; base_hinge: 0.24709253; steps/sec: 97.1;
FastEstimator-Train: step: 37000; epoch: 37; epoch_time(sec): 11.85;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 44.51;
Eval Progress: 66/100; steps/sec: 88.79;
Eval Progress: 100/100; steps/sec: 89.24;
FastEstimator-Eval: step: 37000; epoch: 37; adv_hinge: 0.6924243; adversarial_accuracy: 0.3308; base_accuracy: 0.7244; base_hinge: 0.30295706; min_base_hinge: 0.300413; since_best_base_hinge: 6;
FastEstimator-Train: step: 37500; base_hinge: 0.1045022; steps/sec: 72.11;
FastEstimator-Train: step: 38000; base_hinge: 0.14163926; steps/sec: 92.39;
FastEstimator-Train: step: 38000; epoch: 38; epoch_time(sec): 12.35;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.78;
Eval Progress: 66/100; steps/sec: 85.08;
Eval Progress: 100/100; steps/sec: 89.05;
FastEstimator-Eval: step: 38000; epoch: 38; adv_hinge: 0.6906722; adversarial_accuracy: 0.334; base_accuracy: 0.7222; base_hinge: 0.3049931; min_base_hinge: 0.300413; since_best_base_hinge: 7;
FastEstimator-Train: step: 38500; base_hinge: 0.090151735; steps/sec: 76.75;
FastEstimator-Train: step: 39000; base_hinge: 0.21342143; steps/sec: 99.03;
FastEstimator-Train: step: 39000; epoch: 39; epoch_time(sec): 11.56;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.36;
Eval Progress: 66/100; steps/sec: 92.35;
Eval Progress: 100/100; steps/sec: 92.44;
FastEstimator-Eval: step: 39000; epoch: 39; adv_hinge: 0.68931735; adversarial_accuracy: 0.339; base_accuracy: 0.7232; base_hinge: 0.3034847; min_base_hinge: 0.300413; since_best_base_hinge: 8;
FastEstimator-Train: step: 39500; base_hinge: 0.13618317; steps/sec: 74.33;
FastEstimator-Train: step: 40000; base_hinge: 0.23481017; steps/sec: 91.42;
FastEstimator-Train: step: 40000; epoch: 40; epoch_time(sec): 12.22;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.41;
Eval Progress: 66/100; steps/sec: 93.64;
Eval Progress: 100/100; steps/sec: 89.91;
FastEstimator-Eval: step: 40000; epoch: 40; adv_hinge: 0.6819035; adversarial_accuracy: 0.3482; base_accuracy: 0.725; base_hinge: 0.30202734; min_base_hinge: 0.300413; since_best_base_hinge: 9;
FastEstimator-Train: step: 40500; base_hinge: 0.19524212; steps/sec: 72.12;
FastEstimator-Train: step: 41000; base_hinge: 0.17070587; steps/sec: 92.54;
FastEstimator-Train: step: 41000; epoch: 41; epoch_time(sec): 12.32;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 37.12;
Eval Progress: 66/100; steps/sec: 96.38;
Eval Progress: 100/100; steps/sec: 92.21;
FastEstimator-EarlyStopping: 'base_hinge' triggered an early stop. Its best value was 0.30041301250457764 at epoch 31
FastEstimator-Eval: step: 41000; epoch: 41; adv_hinge: 0.6889992; adversarial_accuracy: 0.3346; base_accuracy: 0.7168; base_hinge: 0.31496406; min_base_hinge: 0.300413; since_best_base_hinge: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpc9g166fj/hydra_ecc_best_base_hinge.h5
FastEstimator-Finish: step: 41000; hydra_ecc_lr: 0.001; total_time(sec): 614.35;
FastEstimator-Test: step: 41000; epoch: 41; adv_hinge: 0.68556327; adversarial_accuracy: 0.334; base_accuracy: 0.7176; base_hinge: 0.30918294;

Comparing the Results¶

In [15]:

logs = visualize_logs(experiments=[softmax_results, ecc_results, hydra_results], ignore_metrics={'ecc_lr', 'hydra_ecc_lr', 'softmax_lr', 'logging_interval', 'num_device', 'epoch_time', 'min_base_ce', 'adv_ce', 'total_time'})

logs = visualize_logs(experiments=[softmax_results, ecc_results, hydra_results], ignore_metrics={'ecc_lr', 'hydra_ecc_lr', 'softmax_lr', 'logging_interval', 'num_device', 'epoch_time', 'min_base_ce', 'adv_ce', 'total_time'})

As you can see, the conventional network using softmax to convert logits to class probabilities actually gets more and more vulnerable to adversarial attacks as training progresses. It also quickly overfits to the data, reaching an optimal performance around epoch 7. By switching the output layer of the model to generate an error correcting code and training with hinge loss, the model is able to train almost 6 times longer before reaching peak conventional accuracy. Moreover, the adversarial performance of the network continues to improve even after the main training runs out. This is significantly better performance than networks trained specifically to combat this attack, shown in the FGSM notebook. It can also be seen that there is no additional cost to training using ECC as opposed to softmax in terms of steps/sec. This is a big benefit over FGSM, where the training time for each step is doubled. With these benefits in mind, you may want to consider never using softmax again.