Adversarial Robustness with Error Correcting Codes (and Hinge Loss)¶

(Never use Softmax again)¶

In this example we will show how using error correcting codes as a model output can drastically reduce model overfitting while simultaneously increasing model robustness against adversarial attacks. In other words, why you should never use a softmax layer again. This is slightly more complicated than the our other ECC apphub example, but it allows for more accurate final probability estimates (the FE Hadamard network layer results in probability smoothing which prevents the network from ever being 100% confident in a class choice). The usefulness of error correcting codes was first publicized by the US Army in a 2019 Neurips Paper. For background on adversarial attacks, and on the attack type we will be demonstrating here, check out our FGSM apphub example. Note that in this apphub we will not be training against adversarial samples, but only performing adversarial attacks during evaluation to see how different models fair against them.

Imports¶

In [1]:

import tempfile

from tensorflow.keras import Sequential, layers
from tensorflow.keras.layers import Concatenate, Conv2D, Dense, Flatten, Input, MaxPooling2D
from tensorflow.keras.models import Model

import fastestimator as fe
from fastestimator.architecture.tensorflow import LeNet
from fastestimator.dataset.data import cifair10
from fastestimator.op.numpyop.univariate import Hadamard, Normalize
from fastestimator.op.tensorop import UnHadamard
from fastestimator.op.tensorop.gradient import FGSM, Watch
from fastestimator.op.tensorop.loss import CrossEntropy, Hinge
from fastestimator.op.tensorop.model import ModelOp, UpdateOp
from fastestimator.summary.logs import visualize_logs
from fastestimator.trace.adapt import EarlyStopping
from fastestimator.trace.io import BestModelSaver
from fastestimator.trace.metric import Accuracy

import tempfile from tensorflow.keras import Sequential, layers from tensorflow.keras.layers import Concatenate, Conv2D, Dense, Flatten, Input, MaxPooling2D from tensorflow.keras.models import Model import fastestimator as fe from fastestimator.architecture.tensorflow import LeNet from fastestimator.dataset.data import cifair10 from fastestimator.op.numpyop.univariate import Hadamard, Normalize from fastestimator.op.tensorop import UnHadamard from fastestimator.op.tensorop.gradient import FGSM, Watch from fastestimator.op.tensorop.loss import CrossEntropy, Hinge from fastestimator.op.tensorop.model import ModelOp, UpdateOp from fastestimator.summary.logs import visualize_logs from fastestimator.trace.adapt import EarlyStopping from fastestimator.trace.io import BestModelSaver from fastestimator.trace.metric import Accuracy

In [2]:

parameters

# training parameters
epsilon=0.04  # The strength of the adversarial attack
epochs=60
batch_size=50
log_steps=500
train_steps_per_epoch=None
eval_steps_per_epoch=None
save_dir=tempfile.mkdtemp()

# training parameters epsilon=0.04 # The strength of the adversarial attack epochs=60 batch_size=50 log_steps=500 train_steps_per_epoch=None eval_steps_per_epoch=None save_dir=tempfile.mkdtemp()

Getting the Data¶

For these experiments we will be using the ciFAIR10 Dataset (like cifar10, but with duplicate testing data replaced)

In [3]:

train_data, eval_data = cifair10.load_data()
test_data = eval_data.split(0.5)
pipeline = fe.Pipeline(
    train_data=train_data,
    eval_data=eval_data,
    test_data=test_data,
    batch_size=batch_size,
    ops=[
        Normalize(inputs="x", outputs="x", mean=(0.4914, 0.4822, 0.4465), std=(0.2471, 0.2435, 0.2616)),
        Hadamard(inputs="y", outputs="y_code", n_classes=10)
        ])

train_data, eval_data = cifair10.load_data() test_data = eval_data.split(0.5) pipeline = fe.Pipeline( train_data=train_data, eval_data=eval_data, test_data=test_data, batch_size=batch_size, ops=[ Normalize(inputs="x", outputs="x", mean=(0.4914, 0.4822, 0.4465), std=(0.2471, 0.2435, 0.2616)), Hadamard(inputs="y", outputs="y_code", n_classes=10) ])

Defining Estimators¶

In this apphub we will be comparing a baseline model against two models using hinge loss to enable training with error correcting codes. The setting up the hinge loss models requires a few extra Ops along the way.

In [4]:

def get_baseline_estimator(model):
    network = fe.Network(ops=[
        Watch(inputs="x", mode=('eval', 'test')),
        ModelOp(model=model, inputs="x", outputs="y_pred"),
        CrossEntropy(inputs=("y_pred", "y"), outputs="base_ce"),
        UpdateOp(model=model, loss_name="base_ce"),
        FGSM(data="x", loss="base_ce", outputs="x_adverse", epsilon=epsilon, mode=('eval', 'test')),
        ModelOp(model=model, inputs="x_adverse", outputs="y_pred_adv", mode=('eval', 'test')),
        CrossEntropy(inputs=("y_pred_adv", "y"), outputs="adv_ce", mode=('eval', 'test'))
    ])
    traces = [
        Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"),
        Accuracy(true_key="y", pred_key="y_pred_adv", output_name="adversarial_accuracy"),
        BestModelSaver(model=model, save_dir=save_dir, metric="base_ce", save_best_mode="min", load_best_final=True),
        EarlyStopping(monitor="base_ce", patience=10)
    ]
    estimator = fe.Estimator(pipeline=pipeline,
                             network=network,
                             epochs=epochs,
                             traces=traces,
                             log_steps=log_steps,
                             train_steps_per_epoch=train_steps_per_epoch,
                             eval_steps_per_epoch=eval_steps_per_epoch,
                             monitor_names=["adv_ce"])
    return estimator

def get_baseline_estimator(model): network = fe.Network(ops=[ Watch(inputs="x", mode=('eval', 'test')), ModelOp(model=model, inputs="x", outputs="y_pred"), CrossEntropy(inputs=("y_pred", "y"), outputs="base_ce"), UpdateOp(model=model, loss_name="base_ce"), FGSM(data="x", loss="base_ce", outputs="x_adverse", epsilon=epsilon, mode=('eval', 'test')), ModelOp(model=model, inputs="x_adverse", outputs="y_pred_adv", mode=('eval', 'test')), CrossEntropy(inputs=("y_pred_adv", "y"), outputs="adv_ce", mode=('eval', 'test')) ]) traces = [ Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"), Accuracy(true_key="y", pred_key="y_pred_adv", output_name="adversarial_accuracy"), BestModelSaver(model=model, save_dir=save_dir, metric="base_ce", save_best_mode="min", load_best_final=True), EarlyStopping(monitor="base_ce", patience=10) ] estimator = fe.Estimator(pipeline=pipeline, network=network, epochs=epochs, traces=traces, log_steps=log_steps, train_steps_per_epoch=train_steps_per_epoch, eval_steps_per_epoch=eval_steps_per_epoch, monitor_names=["adv_ce"]) return estimator

In [5]:

def get_hinge_estimator(model):
    network = fe.Network(ops=[
        Watch(inputs="x", mode=('eval', 'test')),
        ModelOp(model=model, inputs="x", outputs="y_pred_code"),
        Hinge(inputs=("y_pred_code", "y_code"), outputs="base_hinge"),
        UpdateOp(model=model, loss_name="base_hinge"),
        UnHadamard(inputs="y_pred_code", outputs="y_pred", n_classes=10, mode=('eval', 'test')),
        # The adversarial attack:
        FGSM(data="x", loss="base_hinge", outputs="x_adverse_hinge", epsilon=epsilon, mode=('eval', 'test')),
        ModelOp(model=model, inputs="x_adverse_hinge", outputs="y_pred_adv_hinge_code", mode=('eval', 'test')),
        Hinge(inputs=("y_pred_adv_hinge_code", "y_code"), outputs="adv_hinge", mode=('eval', 'test')),
        UnHadamard(inputs="y_pred_adv_hinge_code", outputs="y_pred_adv_hinge", n_classes=10, mode=('eval', 'test')),
    ])
    traces = [
        Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"),
        Accuracy(true_key="y", pred_key="y_pred_adv_hinge", output_name="adversarial_accuracy"),
        BestModelSaver(model=model, save_dir=save_dir, metric="base_hinge", save_best_mode="min", load_best_final=True),
        EarlyStopping(monitor="base_hinge", patience=10)
    ]
    estimator = fe.Estimator(pipeline=pipeline,
                             network=network,
                             epochs=epochs,
                             traces=traces,
                             log_steps=log_steps,
                             train_steps_per_epoch=train_steps_per_epoch,
                             eval_steps_per_epoch=eval_steps_per_epoch,
                             monitor_names=["adv_hinge"])
    return estimator

def get_hinge_estimator(model): network = fe.Network(ops=[ Watch(inputs="x", mode=('eval', 'test')), ModelOp(model=model, inputs="x", outputs="y_pred_code"), Hinge(inputs=("y_pred_code", "y_code"), outputs="base_hinge"), UpdateOp(model=model, loss_name="base_hinge"), UnHadamard(inputs="y_pred_code", outputs="y_pred", n_classes=10, mode=('eval', 'test')), # The adversarial attack: FGSM(data="x", loss="base_hinge", outputs="x_adverse_hinge", epsilon=epsilon, mode=('eval', 'test')), ModelOp(model=model, inputs="x_adverse_hinge", outputs="y_pred_adv_hinge_code", mode=('eval', 'test')), Hinge(inputs=("y_pred_adv_hinge_code", "y_code"), outputs="adv_hinge", mode=('eval', 'test')), UnHadamard(inputs="y_pred_adv_hinge_code", outputs="y_pred_adv_hinge", n_classes=10, mode=('eval', 'test')), ]) traces = [ Accuracy(true_key="y", pred_key="y_pred", output_name="base_accuracy"), Accuracy(true_key="y", pred_key="y_pred_adv_hinge", output_name="adversarial_accuracy"), BestModelSaver(model=model, save_dir=save_dir, metric="base_hinge", save_best_mode="min", load_best_final=True), EarlyStopping(monitor="base_hinge", patience=10) ] estimator = fe.Estimator(pipeline=pipeline, network=network, epochs=epochs, traces=traces, log_steps=log_steps, train_steps_per_epoch=train_steps_per_epoch, eval_steps_per_epoch=eval_steps_per_epoch, monitor_names=["adv_hinge"]) return estimator

The Models¶

1 - A LeNet model with Softmax¶

In [6]:

softmax_model = fe.build(model_fn=lambda:LeNet(input_shape=(32, 32, 3)), optimizer_fn="adam", model_name='softmax')

softmax_model = fe.build(model_fn=lambda:LeNet(input_shape=(32, 32, 3)), optimizer_fn="adam", model_name='softmax')

2022-04-13 17:05:52.944653: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-04-13 17:05:52.944780: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)

Metal device set to: Apple M1 Max

2 - A LeNet model with Error Correcting Codes¶

In [7]:

def EccLeNet(input_shape=(32, 32, 3), code_length=16):
    model = Sequential()
    model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.Flatten())
    model.add(layers.Dense(64, activation='relu'))
    model.add(layers.Dense(code_length, activation='tanh'))  # Note that this is the only difference between this model and the FE LeNet implementation
    return model

def EccLeNet(input_shape=(32, 32, 3), code_length=16): model = Sequential() model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape)) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.Flatten()) model.add(layers.Dense(64, activation='relu')) model.add(layers.Dense(code_length, activation='tanh')) # Note that this is the only difference between this model and the FE LeNet implementation return model

In [8]:

ecc_model = fe.build(model_fn=EccLeNet, optimizer_fn="adam", model_name='ecc')

ecc_model = fe.build(model_fn=EccLeNet, optimizer_fn="adam", model_name='ecc')

3 - A LeNet model using ECC and multiple feature heads¶

While it is common practice to follow the feature extraction layers of convolution networks with several fully connected layers in order to perform classification, this can lead to the final logits being interdependent which can actually reduce the robustness of the network. One way around this is to divide your classification layers into multiple smaller independent units:

In [9]:

def HydraEccLeNet(input_shape=(32, 32, 3), code_length=16):
    inputs = Input(input_shape)
    conv1 = Conv2D(32, (3, 3), activation='relu')(inputs)
    pool1 = MaxPooling2D((2, 2))(conv1)
    conv2 = Conv2D(64, (3, 3), activation='relu')(pool1)
    pool2 = MaxPooling2D((2, 2))(conv2)
    conv3 = Conv2D(64, (3, 3), activation='relu')(pool2)
    flat = Flatten()(conv3)
    # Create multiple heads
    n_heads = 4
    heads = [Dense(16, activation='relu')(flat) for _ in range(n_heads)]
    heads2 = [Dense(code_length // n_heads, activation='tanh')(head) for head in heads]
    outputs = Concatenate()(heads2)
    return Model(inputs=inputs, outputs=outputs)

def HydraEccLeNet(input_shape=(32, 32, 3), code_length=16): inputs = Input(input_shape) conv1 = Conv2D(32, (3, 3), activation='relu')(inputs) pool1 = MaxPooling2D((2, 2))(conv1) conv2 = Conv2D(64, (3, 3), activation='relu')(pool1) pool2 = MaxPooling2D((2, 2))(conv2) conv3 = Conv2D(64, (3, 3), activation='relu')(pool2) flat = Flatten()(conv3) # Create multiple heads n_heads = 4 heads = [Dense(16, activation='relu')(flat) for _ in range(n_heads)] heads2 = [Dense(code_length // n_heads, activation='tanh')(head) for head in heads] outputs = Concatenate()(heads2) return Model(inputs=inputs, outputs=outputs)

In [10]:

hydra_model = fe.build(model_fn=HydraEccLeNet, optimizer_fn="adam", model_name='hydra_ecc')

hydra_model = fe.build(model_fn=HydraEccLeNet, optimizer_fn="adam", model_name='hydra_ecc')

The Experiments¶

Let's get Estimators for each of these models and compare them:

In [11]:

softmax_estimator = get_baseline_estimator(softmax_model)
ecc_estimator = get_hinge_estimator(ecc_model)
hydra_estimator = get_hinge_estimator(hydra_model)

softmax_estimator = get_baseline_estimator(softmax_model) ecc_estimator = get_hinge_estimator(ecc_model) hydra_estimator = get_hinge_estimator(hydra_model)

2022-04-13 17:05:53.690181: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.

In [12]:

softmax_estimator.fit('Softmax')
softmax_results = softmax_estimator.test()

softmax_estimator.fit('Softmax') softmax_results = softmax_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Warn: the key 'y_code' is being pruned since it is unused outside of the Pipeline. To prevent this, you can declare the key as an input of a Trace or TensorOp.
FastEstimator-Start: step: 1; logging_interval: 500; num_device: 0;
FastEstimator-Train: step: 1; base_ce: 2.3505318;
FastEstimator-Train: step: 500; base_ce: 1.2900878; steps/sec: 169.3;
FastEstimator-Train: step: 1000; base_ce: 1.0868878; steps/sec: 171.39;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time: 7.54 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_ce: 1.9342302; adversarial_accuracy: 0.3144; base_accuracy: 0.5896; base_ce: 1.146428; min_base_ce: 1.146428; since_best_base_ce: 0;
FastEstimator-Train: step: 1500; base_ce: 1.1560345; steps/sec: 164.6;
FastEstimator-Train: step: 2000; base_ce: 1.0841315; steps/sec: 175.9;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time: 5.88 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_ce: 2.1664774; adversarial_accuracy: 0.301; base_accuracy: 0.65; base_ce: 1.0123901; min_base_ce: 1.0123901; since_best_base_ce: 0;
FastEstimator-Train: step: 2500; base_ce: 0.73313004; steps/sec: 157.13;
FastEstimator-Train: step: 3000; base_ce: 1.0141094; steps/sec: 176.9;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time: 6.0 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_ce: 2.3282566; adversarial_accuracy: 0.2952; base_accuracy: 0.6774; base_ce: 0.9235382; min_base_ce: 0.9235382; since_best_base_ce: 0;
FastEstimator-Train: step: 3500; base_ce: 0.5947217; steps/sec: 163.7;
FastEstimator-Train: step: 4000; base_ce: 1.0634463; steps/sec: 178.62;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time: 5.85 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_ce: 2.6093562; adversarial_accuracy: 0.2648; base_accuracy: 0.6952; base_ce: 0.9063715; min_base_ce: 0.9063715; since_best_base_ce: 0;
FastEstimator-Train: step: 4500; base_ce: 0.81916714; steps/sec: 160.13;
FastEstimator-Train: step: 5000; base_ce: 0.8680224; steps/sec: 178.95;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time: 5.92 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_ce: 2.778289; adversarial_accuracy: 0.2582; base_accuracy: 0.7124; base_ce: 0.8622628; min_base_ce: 0.8622628; since_best_base_ce: 0;
FastEstimator-Train: step: 5500; base_ce: 1.0295881; steps/sec: 160.73;
FastEstimator-Train: step: 6000; base_ce: 0.5846585; steps/sec: 184.45;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time: 5.82 sec;
FastEstimator-Eval: step: 6000; epoch: 6; adv_ce: 3.056959; adversarial_accuracy: 0.235; base_accuracy: 0.7088; base_ce: 0.86487746; min_base_ce: 0.8622628; since_best_base_ce: 1;
FastEstimator-Train: step: 6500; base_ce: 0.50238585; steps/sec: 150.58;
FastEstimator-Train: step: 7000; base_ce: 0.54035467; steps/sec: 161.76;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time: 6.42 sec;
FastEstimator-Eval: step: 7000; epoch: 7; adv_ce: 3.3898497; adversarial_accuracy: 0.2372; base_accuracy: 0.7162; base_ce: 0.88421905; min_base_ce: 0.8622628; since_best_base_ce: 2;
FastEstimator-Train: step: 7500; base_ce: 0.46544573; steps/sec: 145.52;
FastEstimator-Train: step: 8000; base_ce: 0.71973133; steps/sec: 175.91;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time: 6.27 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Eval: step: 8000; epoch: 8; adv_ce: 3.5737696; adversarial_accuracy: 0.2222; base_accuracy: 0.7214; base_ce: 0.8615385; min_base_ce: 0.8615385; since_best_base_ce: 0;
FastEstimator-Train: step: 8500; base_ce: 0.6100302; steps/sec: 157.41;
FastEstimator-Train: step: 9000; base_ce: 0.58645684; steps/sec: 184.26;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time: 5.89 sec;
FastEstimator-Eval: step: 9000; epoch: 9; adv_ce: 3.8579502; adversarial_accuracy: 0.2122; base_accuracy: 0.7154; base_ce: 0.90025634; min_base_ce: 0.8615385; since_best_base_ce: 1;
FastEstimator-Train: step: 9500; base_ce: 0.5049728; steps/sec: 164.96;
FastEstimator-Train: step: 10000; base_ce: 0.6656696; steps/sec: 179.83;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time: 5.81 sec;
FastEstimator-Eval: step: 10000; epoch: 10; adv_ce: 4.3775306; adversarial_accuracy: 0.1894; base_accuracy: 0.7056; base_ce: 0.9664465; min_base_ce: 0.8615385; since_best_base_ce: 2;
FastEstimator-Train: step: 10500; base_ce: 0.29843685; steps/sec: 168.0;
FastEstimator-Train: step: 11000; base_ce: 0.43359518; steps/sec: 191.24;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time: 5.59 sec;
FastEstimator-Eval: step: 11000; epoch: 11; adv_ce: 5.0255003; adversarial_accuracy: 0.1828; base_accuracy: 0.707; base_ce: 1.0204912; min_base_ce: 0.8615385; since_best_base_ce: 3;
FastEstimator-Train: step: 11500; base_ce: 0.41417968; steps/sec: 164.39;
FastEstimator-Train: step: 12000; base_ce: 0.42944288; steps/sec: 189.79;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time: 5.68 sec;
FastEstimator-Eval: step: 12000; epoch: 12; adv_ce: 5.359952; adversarial_accuracy: 0.1696; base_accuracy: 0.7042; base_ce: 1.0600693; min_base_ce: 0.8615385; since_best_base_ce: 4;
FastEstimator-Train: step: 12500; base_ce: 0.3950038; steps/sec: 153.69;
FastEstimator-Train: step: 13000; base_ce: 0.16283879; steps/sec: 179.13;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time: 6.05 sec;
FastEstimator-Eval: step: 13000; epoch: 13; adv_ce: 5.7364006; adversarial_accuracy: 0.1794; base_accuracy: 0.7082; base_ce: 1.1232861; min_base_ce: 0.8615385; since_best_base_ce: 5;
FastEstimator-Train: step: 13500; base_ce: 0.32468435; steps/sec: 162.28;
FastEstimator-Train: step: 14000; base_ce: 0.19765873; steps/sec: 184.49;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time: 5.79 sec;
FastEstimator-Eval: step: 14000; epoch: 14; adv_ce: 5.8575244; adversarial_accuracy: 0.1704; base_accuracy: 0.7028; base_ce: 1.1115676; min_base_ce: 0.8615385; since_best_base_ce: 6;
FastEstimator-Train: step: 14500; base_ce: 0.26641122; steps/sec: 162.69;
FastEstimator-Train: step: 15000; base_ce: 0.2560778; steps/sec: 178.57;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time: 5.87 sec;
FastEstimator-Eval: step: 15000; epoch: 15; adv_ce: 7.1690974; adversarial_accuracy: 0.1616; base_accuracy: 0.705; base_ce: 1.2743791; min_base_ce: 0.8615385; since_best_base_ce: 7;
FastEstimator-Train: step: 15500; base_ce: 0.28445697; steps/sec: 160.85;
FastEstimator-Train: step: 16000; base_ce: 0.27751252; steps/sec: 179.17;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time: 5.91 sec;
FastEstimator-Eval: step: 16000; epoch: 16; adv_ce: 7.266639; adversarial_accuracy: 0.1604; base_accuracy: 0.7054; base_ce: 1.2886978; min_base_ce: 0.8615385; since_best_base_ce: 8;
FastEstimator-Train: step: 16500; base_ce: 0.2318586; steps/sec: 155.11;
FastEstimator-Train: step: 17000; base_ce: 0.14783162; steps/sec: 184.27;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time: 5.93 sec;
FastEstimator-Eval: step: 17000; epoch: 17; adv_ce: 8.196975; adversarial_accuracy: 0.159; base_accuracy: 0.7066; base_ce: 1.3935325; min_base_ce: 0.8615385; since_best_base_ce: 9;
FastEstimator-Train: step: 17500; base_ce: 0.3715953; steps/sec: 166.94;
FastEstimator-Train: step: 18000; base_ce: 0.21967937; steps/sec: 181.36;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time: 5.75 sec;
FastEstimator-EarlyStopping: 'base_ce' triggered an early stop. Its best value was 0.8615385293960571 at epoch 8
FastEstimator-Eval: step: 18000; epoch: 18; adv_ce: 8.571767; adversarial_accuracy: 0.1602; base_accuracy: 0.702; base_ce: 1.4734117; min_base_ce: 0.8615385; since_best_base_ce: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/softmax_best_base_ce.h5
FastEstimator-Finish: step: 18000; softmax_lr: 0.001; total_time: 128.94 sec;
FastEstimator-Test: step: 18000; epoch: 18; adv_ce: 3.6057885; adversarial_accuracy: 0.2186; base_accuracy: 0.7206; base_ce: 0.855904;

In [13]:

ecc_estimator.fit('ECC')
ecc_results = ecc_estimator.test()

ecc_estimator.fit('ECC') ecc_results = ecc_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Warn: the key 'y' is being pruned since it is unused outside of the Pipeline. To prevent this, you can declare the key as an input of a Trace or TensorOp.
FastEstimator-Start: step: 1; logging_interval: 500; num_device: 0;
FastEstimator-Train: step: 1; base_hinge: 0.999596;
FastEstimator-Train: step: 500; base_hinge: 0.77081025; steps/sec: 171.05;
FastEstimator-Train: step: 1000; base_hinge: 0.62755454; steps/sec: 177.78;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time: 6.29 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_hinge: 0.7516223; adversarial_accuracy: 0.259; base_accuracy: 0.3914; base_hinge: 0.6413806; min_base_hinge: 0.6413806; since_best_base_hinge: 0;
FastEstimator-Train: step: 1500; base_hinge: 0.6501051; steps/sec: 145.76;
FastEstimator-Train: step: 2000; base_hinge: 0.47245127; steps/sec: 173.18;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time: 6.32 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_hinge: 0.7038234; adversarial_accuracy: 0.3176; base_accuracy: 0.5038; base_hinge: 0.5589157; min_base_hinge: 0.5589157; since_best_base_hinge: 0;
FastEstimator-Train: step: 2500; base_hinge: 0.5615465; steps/sec: 150.65;
FastEstimator-Train: step: 3000; base_hinge: 0.51083314; steps/sec: 174.93;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time: 6.17 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_hinge: 0.6786241; adversarial_accuracy: 0.3506; base_accuracy: 0.5512; base_hinge: 0.50700206; min_base_hinge: 0.50700206; since_best_base_hinge: 0;
FastEstimator-Train: step: 3500; base_hinge: 0.49202853; steps/sec: 153.9;
FastEstimator-Train: step: 4000; base_hinge: 0.41295394; steps/sec: 180.68;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time: 6.05 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_hinge: 0.68908304; adversarial_accuracy: 0.3428; base_accuracy: 0.5982; base_hinge: 0.4595022; min_base_hinge: 0.4595022; since_best_base_hinge: 0;
FastEstimator-Train: step: 4500; base_hinge: 0.34178427; steps/sec: 137.02;
FastEstimator-Train: step: 5000; base_hinge: 0.43433386; steps/sec: 161.73;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time: 6.71 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_hinge: 0.6604099; adversarial_accuracy: 0.3722; base_accuracy: 0.6326; base_hinge: 0.42740837; min_base_hinge: 0.42740837; since_best_base_hinge: 0;
FastEstimator-Train: step: 5500; base_hinge: 0.41189644; steps/sec: 144.49;
FastEstimator-Train: step: 6000; base_hinge: 0.4554962; steps/sec: 160.9;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time: 6.58 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_hinge: 0.6719003; adversarial_accuracy: 0.3536; base_accuracy: 0.6472; base_hinge: 0.40192205; min_base_hinge: 0.40192205; since_best_base_hinge: 0;
FastEstimator-Train: step: 6500; base_hinge: 0.32702583; steps/sec: 134.67;
FastEstimator-Train: step: 7000; base_hinge: 0.32336193; steps/sec: 176.44;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time: 6.53 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_hinge: 0.6758878; adversarial_accuracy: 0.351; base_accuracy: 0.656; base_hinge: 0.38305122; min_base_hinge: 0.38305122; since_best_base_hinge: 0;
FastEstimator-Train: step: 7500; base_hinge: 0.3770857; steps/sec: 146.85;
FastEstimator-Train: step: 8000; base_hinge: 0.36640465; steps/sec: 176.21;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time: 6.24 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 8000; epoch: 8; adv_hinge: 0.66526324; adversarial_accuracy: 0.36; base_accuracy: 0.6564; base_hinge: 0.37443084; min_base_hinge: 0.37443084; since_best_base_hinge: 0;
FastEstimator-Train: step: 8500; base_hinge: 0.4330097; steps/sec: 152.61;
FastEstimator-Train: step: 9000; base_hinge: 0.28264937; steps/sec: 179.83;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time: 6.05 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 9000; epoch: 9; adv_hinge: 0.6605204; adversarial_accuracy: 0.3632; base_accuracy: 0.6688; base_hinge: 0.36260226; min_base_hinge: 0.36260226; since_best_base_hinge: 0;
FastEstimator-Train: step: 9500; base_hinge: 0.25725213; steps/sec: 158.7;
FastEstimator-Train: step: 10000; base_hinge: 0.33918726; steps/sec: 187.6;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time: 5.82 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 10000; epoch: 10; adv_hinge: 0.6670266; adversarial_accuracy: 0.3644; base_accuracy: 0.6786; base_hinge: 0.3502937; min_base_hinge: 0.3502937; since_best_base_hinge: 0;
FastEstimator-Train: step: 10500; base_hinge: 0.24327007; steps/sec: 156.93;
FastEstimator-Train: step: 11000; base_hinge: 0.27944666; steps/sec: 188.6;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time: 5.84 sec;
FastEstimator-Eval: step: 11000; epoch: 11; adv_hinge: 0.6592089; adversarial_accuracy: 0.3662; base_accuracy: 0.6704; base_hinge: 0.35914975; min_base_hinge: 0.3502937; since_best_base_hinge: 1;
FastEstimator-Train: step: 11500; base_hinge: 0.32648894; steps/sec: 157.13;
FastEstimator-Train: step: 12000; base_hinge: 0.25949845; steps/sec: 181.27;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time: 5.93 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 12000; epoch: 12; adv_hinge: 0.651387; adversarial_accuracy: 0.3758; base_accuracy: 0.6894; base_hinge: 0.33977306; min_base_hinge: 0.33977306; since_best_base_hinge: 0;
FastEstimator-Train: step: 12500; base_hinge: 0.36375687; steps/sec: 149.42;
FastEstimator-Train: step: 13000; base_hinge: 0.19662806; steps/sec: 182.06;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time: 6.09 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 13000; epoch: 13; adv_hinge: 0.64508665; adversarial_accuracy: 0.3786; base_accuracy: 0.7002; base_hinge: 0.3291817; min_base_hinge: 0.3291817; since_best_base_hinge: 0;
FastEstimator-Train: step: 13500; base_hinge: 0.32477826; steps/sec: 142.68;
FastEstimator-Train: step: 14000; base_hinge: 0.22079967; steps/sec: 169.74;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time: 6.47 sec;
FastEstimator-Eval: step: 14000; epoch: 14; adv_hinge: 0.66262734; adversarial_accuracy: 0.3688; base_accuracy: 0.6934; base_hinge: 0.3377885; min_base_hinge: 0.3291817; since_best_base_hinge: 1;
FastEstimator-Train: step: 14500; base_hinge: 0.30110896; steps/sec: 145.77;
FastEstimator-Train: step: 15000; base_hinge: 0.19694538; steps/sec: 172.02;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time: 6.34 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 15000; epoch: 15; adv_hinge: 0.65070426; adversarial_accuracy: 0.3776; base_accuracy: 0.7036; base_hinge: 0.32673293; min_base_hinge: 0.32673293; since_best_base_hinge: 0;
FastEstimator-Train: step: 15500; base_hinge: 0.29805455; steps/sec: 142.52;
FastEstimator-Train: step: 16000; base_hinge: 0.20518483; steps/sec: 181.85;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time: 6.25 sec;
FastEstimator-Eval: step: 16000; epoch: 16; adv_hinge: 0.6414124; adversarial_accuracy: 0.3878; base_accuracy: 0.7002; base_hinge: 0.33122116; min_base_hinge: 0.32673293; since_best_base_hinge: 1;
FastEstimator-Train: step: 16500; base_hinge: 0.27236506; steps/sec: 149.75;
FastEstimator-Train: step: 17000; base_hinge: 0.42725775; steps/sec: 182.74;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time: 6.08 sec;
FastEstimator-Eval: step: 17000; epoch: 17; adv_hinge: 0.6625693; adversarial_accuracy: 0.3716; base_accuracy: 0.7006; base_hinge: 0.33163765; min_base_hinge: 0.32673293; since_best_base_hinge: 2;
FastEstimator-Train: step: 17500; base_hinge: 0.2649264; steps/sec: 154.25;
FastEstimator-Train: step: 18000; base_hinge: 0.16135535; steps/sec: 181.42;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time: 5.98 sec;
FastEstimator-Eval: step: 18000; epoch: 18; adv_hinge: 0.6661872; adversarial_accuracy: 0.3626; base_accuracy: 0.697; base_hinge: 0.33215272; min_base_hinge: 0.32673293; since_best_base_hinge: 3;
FastEstimator-Train: step: 18500; base_hinge: 0.22723289; steps/sec: 144.34;
FastEstimator-Train: step: 19000; base_hinge: 0.35645804; steps/sec: 176.94;
FastEstimator-Train: step: 19000; epoch: 19; epoch_time: 6.31 sec;
FastEstimator-Eval: step: 19000; epoch: 19; adv_hinge: 0.6585936; adversarial_accuracy: 0.3732; base_accuracy: 0.6946; base_hinge: 0.33224463; min_base_hinge: 0.32673293; since_best_base_hinge: 4;
FastEstimator-Train: step: 19500; base_hinge: 0.32812256; steps/sec: 151.88;
FastEstimator-Train: step: 20000; base_hinge: 0.28209794; steps/sec: 179.77;
FastEstimator-Train: step: 20000; epoch: 20; epoch_time: 6.05 sec;
FastEstimator-Eval: step: 20000; epoch: 20; adv_hinge: 0.68119395; adversarial_accuracy: 0.3558; base_accuracy: 0.6958; base_hinge: 0.33329198; min_base_hinge: 0.32673293; since_best_base_hinge: 5;
FastEstimator-Train: step: 20500; base_hinge: 0.083697535; steps/sec: 149.25;
FastEstimator-Train: step: 21000; base_hinge: 0.32213867; steps/sec: 183.06;
FastEstimator-Train: step: 21000; epoch: 21; epoch_time: 6.1 sec;
FastEstimator-Eval: step: 21000; epoch: 21; adv_hinge: 0.65694094; adversarial_accuracy: 0.3712; base_accuracy: 0.6992; base_hinge: 0.3307393; min_base_hinge: 0.32673293; since_best_base_hinge: 6;
FastEstimator-Train: step: 21500; base_hinge: 0.21509376; steps/sec: 145.06;
FastEstimator-Train: step: 22000; base_hinge: 0.19089276; steps/sec: 176.3;
FastEstimator-Train: step: 22000; epoch: 22; epoch_time: 6.28 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 22000; epoch: 22; adv_hinge: 0.65987563; adversarial_accuracy: 0.3686; base_accuracy: 0.7086; base_hinge: 0.3198288; min_base_hinge: 0.3198288; since_best_base_hinge: 0;
FastEstimator-Train: step: 22500; base_hinge: 0.27360404; steps/sec: 148.15;
FastEstimator-Train: step: 23000; base_hinge: 0.20834382; steps/sec: 181.64;
FastEstimator-Train: step: 23000; epoch: 23; epoch_time: 6.12 sec;
FastEstimator-Eval: step: 23000; epoch: 23; adv_hinge: 0.6662962; adversarial_accuracy: 0.3658; base_accuracy: 0.7042; base_hinge: 0.32622346; min_base_hinge: 0.3198288; since_best_base_hinge: 1;
FastEstimator-Train: step: 23500; base_hinge: 0.2153913; steps/sec: 155.29;
FastEstimator-Train: step: 24000; base_hinge: 0.23683456; steps/sec: 179.49;
FastEstimator-Train: step: 24000; epoch: 24; epoch_time: 6.0 sec;
FastEstimator-Eval: step: 24000; epoch: 24; adv_hinge: 0.6681502; adversarial_accuracy: 0.3654; base_accuracy: 0.703; base_hinge: 0.3277005; min_base_hinge: 0.3198288; since_best_base_hinge: 2;
FastEstimator-Train: step: 24500; base_hinge: 0.17390369; steps/sec: 146.98;
FastEstimator-Train: step: 25000; base_hinge: 0.18928257; steps/sec: 177.46;
FastEstimator-Train: step: 25000; epoch: 25; epoch_time: 6.24 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 25000; epoch: 25; adv_hinge: 0.6698162; adversarial_accuracy: 0.3616; base_accuracy: 0.715; base_hinge: 0.31426823; min_base_hinge: 0.31426823; since_best_base_hinge: 0;
FastEstimator-Train: step: 25500; base_hinge: 0.16106723; steps/sec: 145.95;
FastEstimator-Train: step: 26000; base_hinge: 0.19729549; steps/sec: 165.93;
FastEstimator-Train: step: 26000; epoch: 26; epoch_time: 6.45 sec;
FastEstimator-Eval: step: 26000; epoch: 26; adv_hinge: 0.67746514; adversarial_accuracy: 0.3552; base_accuracy: 0.7008; base_hinge: 0.3292477; min_base_hinge: 0.31426823; since_best_base_hinge: 1;
FastEstimator-Train: step: 26500; base_hinge: 0.19739527; steps/sec: 150.1;
FastEstimator-Train: step: 27000; base_hinge: 0.29701933; steps/sec: 168.93;
FastEstimator-Train: step: 27000; epoch: 27; epoch_time: 6.27 sec;
FastEstimator-Eval: step: 27000; epoch: 27; adv_hinge: 0.68560266; adversarial_accuracy: 0.3502; base_accuracy: 0.684; base_hinge: 0.3441289; min_base_hinge: 0.31426823; since_best_base_hinge: 2;
FastEstimator-Train: step: 27500; base_hinge: 0.2622476; steps/sec: 131.3;
FastEstimator-Train: step: 28000; base_hinge: 0.29236382; steps/sec: 164.56;
FastEstimator-Train: step: 28000; epoch: 28; epoch_time: 6.87 sec;
FastEstimator-Eval: step: 28000; epoch: 28; adv_hinge: 0.67048955; adversarial_accuracy: 0.361; base_accuracy: 0.705; base_hinge: 0.32279128; min_base_hinge: 0.31426823; since_best_base_hinge: 3;
FastEstimator-Train: step: 28500; base_hinge: 0.12868015; steps/sec: 136.05;
FastEstimator-Train: step: 29000; base_hinge: 0.16600317; steps/sec: 162.03;
FastEstimator-Train: step: 29000; epoch: 29; epoch_time: 6.75 sec;
FastEstimator-Eval: step: 29000; epoch: 29; adv_hinge: 0.6665552; adversarial_accuracy: 0.3682; base_accuracy: 0.7064; base_hinge: 0.32228085; min_base_hinge: 0.31426823; since_best_base_hinge: 4;
FastEstimator-Train: step: 29500; base_hinge: 0.2084124; steps/sec: 148.01;
FastEstimator-Train: step: 30000; base_hinge: 0.25564948; steps/sec: 173.39;
FastEstimator-Train: step: 30000; epoch: 30; epoch_time: 6.24 sec;
FastEstimator-Eval: step: 30000; epoch: 30; adv_hinge: 0.6741418; adversarial_accuracy: 0.3604; base_accuracy: 0.7048; base_hinge: 0.32388806; min_base_hinge: 0.31426823; since_best_base_hinge: 5;
FastEstimator-Train: step: 30500; base_hinge: 0.16594283; steps/sec: 138.24;
FastEstimator-Train: step: 31000; base_hinge: 0.17639668; steps/sec: 174.51;
FastEstimator-Train: step: 31000; epoch: 31; epoch_time: 6.5 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 31000; epoch: 31; adv_hinge: 0.67253065; adversarial_accuracy: 0.3598; base_accuracy: 0.7096; base_hinge: 0.3093056; min_base_hinge: 0.3093056; since_best_base_hinge: 0;
FastEstimator-Train: step: 31500; base_hinge: 0.22921798; steps/sec: 147.73;
FastEstimator-Train: step: 32000; base_hinge: 0.18706411; steps/sec: 184.5;
FastEstimator-Train: step: 32000; epoch: 32; epoch_time: 6.09 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 32000; epoch: 32; adv_hinge: 0.6728549; adversarial_accuracy: 0.356; base_accuracy: 0.7112; base_hinge: 0.30470937; min_base_hinge: 0.30470937; since_best_base_hinge: 0;
FastEstimator-Train: step: 32500; base_hinge: 0.2344648; steps/sec: 148.86;
FastEstimator-Train: step: 33000; base_hinge: 0.10746834; steps/sec: 181.12;
FastEstimator-Train: step: 33000; epoch: 33; epoch_time: 6.12 sec;
FastEstimator-Eval: step: 33000; epoch: 33; adv_hinge: 0.6760917; adversarial_accuracy: 0.3552; base_accuracy: 0.6868; base_hinge: 0.32750443; min_base_hinge: 0.30470937; since_best_base_hinge: 1;
FastEstimator-Train: step: 33500; base_hinge: 0.20909442; steps/sec: 146.21;
FastEstimator-Train: step: 34000; base_hinge: 0.15598711; steps/sec: 177.38;
FastEstimator-Train: step: 34000; epoch: 34; epoch_time: 6.23 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 34000; epoch: 34; adv_hinge: 0.67582107; adversarial_accuracy: 0.36; base_accuracy: 0.7152; base_hinge: 0.30120456; min_base_hinge: 0.30120456; since_best_base_hinge: 0;
FastEstimator-Train: step: 34500; base_hinge: 0.102830425; steps/sec: 149.09;
FastEstimator-Train: step: 35000; base_hinge: 0.16139644; steps/sec: 182.41;
FastEstimator-Train: step: 35000; epoch: 35; epoch_time: 6.1 sec;
FastEstimator-Eval: step: 35000; epoch: 35; adv_hinge: 0.67635447; adversarial_accuracy: 0.3556; base_accuracy: 0.7058; base_hinge: 0.3057574; min_base_hinge: 0.30120456; since_best_base_hinge: 1;
FastEstimator-Train: step: 35500; base_hinge: 0.10657302; steps/sec: 145.67;
FastEstimator-Train: step: 36000; base_hinge: 0.17006141; steps/sec: 174.59;
FastEstimator-Train: step: 36000; epoch: 36; epoch_time: 6.3 sec;
FastEstimator-Eval: step: 36000; epoch: 36; adv_hinge: 0.67683244; adversarial_accuracy: 0.3572; base_accuracy: 0.7102; base_hinge: 0.30349904; min_base_hinge: 0.30120456; since_best_base_hinge: 2;
FastEstimator-Train: step: 36500; base_hinge: 0.15623659; steps/sec: 145.94;
FastEstimator-Train: step: 37000; base_hinge: 0.23949586; steps/sec: 181.96;
FastEstimator-Train: step: 37000; epoch: 37; epoch_time: 6.17 sec;
FastEstimator-Eval: step: 37000; epoch: 37; adv_hinge: 0.66844594; adversarial_accuracy: 0.3638; base_accuracy: 0.6966; base_hinge: 0.31694946; min_base_hinge: 0.30120456; since_best_base_hinge: 3;
FastEstimator-Train: step: 37500; base_hinge: 0.14576639; steps/sec: 147.11;
FastEstimator-Train: step: 38000; base_hinge: 0.18066089; steps/sec: 183.07;
FastEstimator-Train: step: 38000; epoch: 38; epoch_time: 6.13 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Eval: step: 38000; epoch: 38; adv_hinge: 0.68135715; adversarial_accuracy: 0.3566; base_accuracy: 0.7182; base_hinge: 0.29901487; min_base_hinge: 0.29901487; since_best_base_hinge: 0;
FastEstimator-Train: step: 38500; base_hinge: 0.09862997; steps/sec: 144.95;
FastEstimator-Train: step: 39000; base_hinge: 0.23862968; steps/sec: 179.02;
FastEstimator-Train: step: 39000; epoch: 39; epoch_time: 6.24 sec;
FastEstimator-Eval: step: 39000; epoch: 39; adv_hinge: 0.6850794; adversarial_accuracy: 0.3468; base_accuracy: 0.695; base_hinge: 0.318525; min_base_hinge: 0.29901487; since_best_base_hinge: 1;
FastEstimator-Train: step: 39500; base_hinge: 0.24544053; steps/sec: 142.26;
FastEstimator-Train: step: 40000; base_hinge: 0.09076198; steps/sec: 176.09;
FastEstimator-Train: step: 40000; epoch: 40; epoch_time: 6.34 sec;
FastEstimator-Eval: step: 40000; epoch: 40; adv_hinge: 0.6828284; adversarial_accuracy: 0.351; base_accuracy: 0.7042; base_hinge: 0.30795157; min_base_hinge: 0.29901487; since_best_base_hinge: 2;
FastEstimator-Train: step: 40500; base_hinge: 0.124062; steps/sec: 143.74;
FastEstimator-Train: step: 41000; base_hinge: 0.21180153; steps/sec: 177.15;
FastEstimator-Train: step: 41000; epoch: 41; epoch_time: 6.31 sec;
FastEstimator-Eval: step: 41000; epoch: 41; adv_hinge: 0.69568163; adversarial_accuracy: 0.3372; base_accuracy: 0.7032; base_hinge: 0.3095613; min_base_hinge: 0.29901487; since_best_base_hinge: 3;
FastEstimator-Train: step: 41500; base_hinge: 0.116228275; steps/sec: 142.75;
FastEstimator-Train: step: 42000; base_hinge: 0.20703846; steps/sec: 172.43;
FastEstimator-Train: step: 42000; epoch: 42; epoch_time: 6.4 sec;
FastEstimator-Eval: step: 42000; epoch: 42; adv_hinge: 0.7017528; adversarial_accuracy: 0.3346; base_accuracy: 0.7094; base_hinge: 0.3047698; min_base_hinge: 0.29901487; since_best_base_hinge: 4;
FastEstimator-Train: step: 42500; base_hinge: 0.105107; steps/sec: 140.82;
FastEstimator-Train: step: 43000; base_hinge: 0.18722667; steps/sec: 174.07;
FastEstimator-Train: step: 43000; epoch: 43; epoch_time: 6.43 sec;
FastEstimator-Eval: step: 43000; epoch: 43; adv_hinge: 0.6941656; adversarial_accuracy: 0.34; base_accuracy: 0.7082; base_hinge: 0.30632982; min_base_hinge: 0.29901487; since_best_base_hinge: 5;
FastEstimator-Train: step: 43500; base_hinge: 0.15885285; steps/sec: 146.18;
FastEstimator-Train: step: 44000; base_hinge: 0.06321986; steps/sec: 185.0;
FastEstimator-Train: step: 44000; epoch: 44; epoch_time: 6.12 sec;
FastEstimator-Eval: step: 44000; epoch: 44; adv_hinge: 0.6966642; adversarial_accuracy: 0.3352; base_accuracy: 0.711; base_hinge: 0.30576453; min_base_hinge: 0.29901487; since_best_base_hinge: 6;
FastEstimator-Train: step: 44500; base_hinge: 0.12628566; steps/sec: 142.54;
FastEstimator-Train: step: 45000; base_hinge: 0.199184; steps/sec: 175.02;
FastEstimator-Train: step: 45000; epoch: 45; epoch_time: 6.37 sec;
FastEstimator-Eval: step: 45000; epoch: 45; adv_hinge: 0.6890245; adversarial_accuracy: 0.3488; base_accuracy: 0.711; base_hinge: 0.3026847; min_base_hinge: 0.29901487; since_best_base_hinge: 7;
FastEstimator-Train: step: 45500; base_hinge: 0.17868556; steps/sec: 140.97;
FastEstimator-Train: step: 46000; base_hinge: 0.12751797; steps/sec: 174.32;
FastEstimator-Train: step: 46000; epoch: 46; epoch_time: 6.42 sec;
FastEstimator-Eval: step: 46000; epoch: 46; adv_hinge: 0.6926506; adversarial_accuracy: 0.3408; base_accuracy: 0.7074; base_hinge: 0.30677107; min_base_hinge: 0.29901487; since_best_base_hinge: 8;
FastEstimator-Train: step: 46500; base_hinge: 0.17218891; steps/sec: 143.82;
FastEstimator-Train: step: 47000; base_hinge: 0.11857964; steps/sec: 183.44;
FastEstimator-Train: step: 47000; epoch: 47; epoch_time: 6.2 sec;
FastEstimator-Eval: step: 47000; epoch: 47; adv_hinge: 0.68311316; adversarial_accuracy: 0.3482; base_accuracy: 0.71; base_hinge: 0.3053293; min_base_hinge: 0.29901487; since_best_base_hinge: 9;
FastEstimator-Train: step: 47500; base_hinge: 0.14120819; steps/sec: 140.98;
FastEstimator-Train: step: 48000; base_hinge: 0.12556206; steps/sec: 174.2;
FastEstimator-Train: step: 48000; epoch: 48; epoch_time: 6.41 sec;
FastEstimator-EarlyStopping: 'base_hinge' triggered an early stop. Its best value was 0.2990148663520813 at epoch 38
FastEstimator-Eval: step: 48000; epoch: 48; adv_hinge: 0.69799834; adversarial_accuracy: 0.3362; base_accuracy: 0.7064; base_hinge: 0.3066258; min_base_hinge: 0.29901487; since_best_base_hinge: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/ecc_best_base_hinge.h5
FastEstimator-Finish: step: 48000; ecc_lr: 0.001; total_time: 369.69 sec;
FastEstimator-Test: step: 48000; epoch: 48; adv_hinge: 0.68220603; adversarial_accuracy: 0.3522; base_accuracy: 0.7166; base_hinge: 0.29723442;

In [14]:

hydra_estimator.fit('Hydra')
hydra_results = hydra_estimator.test()

hydra_estimator.fit('Hydra') hydra_results = hydra_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; logging_interval: 500; num_device: 0;
FastEstimator-Train: step: 1; base_hinge: 1.005537;
FastEstimator-Train: step: 500; base_hinge: 0.6636967; steps/sec: 120.64;
FastEstimator-Train: step: 1000; base_hinge: 0.6184193; steps/sec: 126.98;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time: 8.95 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_hinge: 0.76180106; adversarial_accuracy: 0.2508; base_accuracy: 0.3742; base_hinge: 0.6694584; min_base_hinge: 0.6694584; since_best_base_hinge: 0;
FastEstimator-Train: step: 1500; base_hinge: 0.60712904; steps/sec: 105.12;
FastEstimator-Train: step: 2000; base_hinge: 0.586785; steps/sec: 128.26;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time: 8.66 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_hinge: 0.7100552; adversarial_accuracy: 0.3074; base_accuracy: 0.4702; base_hinge: 0.5881349; min_base_hinge: 0.5881349; since_best_base_hinge: 0;
FastEstimator-Train: step: 2500; base_hinge: 0.5594202; steps/sec: 106.0;
FastEstimator-Train: step: 3000; base_hinge: 0.51175797; steps/sec: 130.09;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time: 8.56 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_hinge: 0.6757895; adversarial_accuracy: 0.3456; base_accuracy: 0.5496; base_hinge: 0.5054883; min_base_hinge: 0.5054883; since_best_base_hinge: 0;
FastEstimator-Train: step: 3500; base_hinge: 0.51829374; steps/sec: 104.11;
FastEstimator-Train: step: 4000; base_hinge: 0.534348; steps/sec: 126.79;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time: 8.75 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_hinge: 0.66557425; adversarial_accuracy: 0.3584; base_accuracy: 0.5726; base_hinge: 0.47270003; min_base_hinge: 0.47270003; since_best_base_hinge: 0;
FastEstimator-Train: step: 4500; base_hinge: 0.43916184; steps/sec: 102.37;
FastEstimator-Train: step: 5000; base_hinge: 0.44243538; steps/sec: 124.66;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time: 8.9 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_hinge: 0.65442306; adversarial_accuracy: 0.3744; base_accuracy: 0.6044; base_hinge: 0.4457506; min_base_hinge: 0.4457506; since_best_base_hinge: 0;
FastEstimator-Train: step: 5500; base_hinge: 0.40786383; steps/sec: 100.59;
FastEstimator-Train: step: 6000; base_hinge: 0.4591022; steps/sec: 124.97;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time: 8.97 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_hinge: 0.6492514; adversarial_accuracy: 0.3838; base_accuracy: 0.6262; base_hinge: 0.4214871; min_base_hinge: 0.4214871; since_best_base_hinge: 0;
FastEstimator-Train: step: 6500; base_hinge: 0.39158216; steps/sec: 96.31;
FastEstimator-Train: step: 7000; base_hinge: 0.30446494; steps/sec: 115.93;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time: 9.5 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_hinge: 0.6397081; adversarial_accuracy: 0.391; base_accuracy: 0.6558; base_hinge: 0.39896628; min_base_hinge: 0.39896628; since_best_base_hinge: 0;
FastEstimator-Train: step: 7500; base_hinge: 0.37816882; steps/sec: 100.91;
FastEstimator-Train: step: 8000; base_hinge: 0.42791244; steps/sec: 130.84;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time: 8.78 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 8000; epoch: 8; adv_hinge: 0.647455; adversarial_accuracy: 0.3816; base_accuracy: 0.6634; base_hinge: 0.39305332; min_base_hinge: 0.39305332; since_best_base_hinge: 0;
FastEstimator-Train: step: 8500; base_hinge: 0.4074018; steps/sec: 96.92;
FastEstimator-Train: step: 9000; base_hinge: 0.36726004; steps/sec: 121.02;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time: 9.29 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 9000; epoch: 9; adv_hinge: 0.64966685; adversarial_accuracy: 0.3758; base_accuracy: 0.6612; base_hinge: 0.39002314; min_base_hinge: 0.39002314; since_best_base_hinge: 0;
FastEstimator-Train: step: 9500; base_hinge: 0.38659644; steps/sec: 95.88;
FastEstimator-Train: step: 10000; base_hinge: 0.32991368; steps/sec: 118.87;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time: 9.42 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 10000; epoch: 10; adv_hinge: 0.6384187; adversarial_accuracy: 0.3874; base_accuracy: 0.684; base_hinge: 0.37144434; min_base_hinge: 0.37144434; since_best_base_hinge: 0;
FastEstimator-Train: step: 10500; base_hinge: 0.31555474; steps/sec: 102.34;
FastEstimator-Train: step: 11000; base_hinge: 0.3198191; steps/sec: 127.85;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time: 8.8 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 11000; epoch: 11; adv_hinge: 0.6411109; adversarial_accuracy: 0.3818; base_accuracy: 0.6904; base_hinge: 0.3641706; min_base_hinge: 0.3641706; since_best_base_hinge: 0;
FastEstimator-Train: step: 11500; base_hinge: 0.30043656; steps/sec: 97.15;
FastEstimator-Train: step: 12000; base_hinge: 0.31473994; steps/sec: 117.96;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time: 9.39 sec;
FastEstimator-Eval: step: 12000; epoch: 12; adv_hinge: 0.6478225; adversarial_accuracy: 0.3748; base_accuracy: 0.6806; base_hinge: 0.3647029; min_base_hinge: 0.3641706; since_best_base_hinge: 1;
FastEstimator-Train: step: 12500; base_hinge: 0.3006091; steps/sec: 98.24;
FastEstimator-Train: step: 13000; base_hinge: 0.37336874; steps/sec: 123.79;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time: 9.14 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 13000; epoch: 13; adv_hinge: 0.64424914; adversarial_accuracy: 0.3846; base_accuracy: 0.6882; base_hinge: 0.35658637; min_base_hinge: 0.35658637; since_best_base_hinge: 0;
FastEstimator-Train: step: 13500; base_hinge: 0.27503738; steps/sec: 104.17;
FastEstimator-Train: step: 14000; base_hinge: 0.31450596; steps/sec: 129.9;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time: 8.65 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 14000; epoch: 14; adv_hinge: 0.6362553; adversarial_accuracy: 0.3898; base_accuracy: 0.6954; base_hinge: 0.34577057; min_base_hinge: 0.34577057; since_best_base_hinge: 0;
FastEstimator-Train: step: 14500; base_hinge: 0.29565284; steps/sec: 100.6;
FastEstimator-Train: step: 15000; base_hinge: 0.35400718; steps/sec: 122.85;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time: 9.04 sec;
FastEstimator-Eval: step: 15000; epoch: 15; adv_hinge: 0.6411361; adversarial_accuracy: 0.385; base_accuracy: 0.6904; base_hinge: 0.34718743; min_base_hinge: 0.34577057; since_best_base_hinge: 1;
FastEstimator-Train: step: 15500; base_hinge: 0.35471046; steps/sec: 100.02;
FastEstimator-Train: step: 16000; base_hinge: 0.26543427; steps/sec: 124.82;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time: 8.98 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 16000; epoch: 16; adv_hinge: 0.6434831; adversarial_accuracy: 0.3834; base_accuracy: 0.6978; base_hinge: 0.34060454; min_base_hinge: 0.34060454; since_best_base_hinge: 0;
FastEstimator-Train: step: 16500; base_hinge: 0.21976732; steps/sec: 100.93;
FastEstimator-Train: step: 17000; base_hinge: 0.29889458; steps/sec: 122.76;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time: 9.05 sec;
FastEstimator-Eval: step: 17000; epoch: 17; adv_hinge: 0.656668; adversarial_accuracy: 0.3722; base_accuracy: 0.6982; base_hinge: 0.34171763; min_base_hinge: 0.34060454; since_best_base_hinge: 1;
FastEstimator-Train: step: 17500; base_hinge: 0.14615066; steps/sec: 101.85;
FastEstimator-Train: step: 18000; base_hinge: 0.26993346; steps/sec: 128.53;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time: 8.79 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 18000; epoch: 18; adv_hinge: 0.64573526; adversarial_accuracy: 0.3792; base_accuracy: 0.7016; base_hinge: 0.33208665; min_base_hinge: 0.33208665; since_best_base_hinge: 0;
FastEstimator-Train: step: 18500; base_hinge: 0.25417802; steps/sec: 103.1;
FastEstimator-Train: step: 19000; base_hinge: 0.2457754; steps/sec: 128.98;
FastEstimator-Train: step: 19000; epoch: 19; epoch_time: 8.74 sec;
FastEstimator-Eval: step: 19000; epoch: 19; adv_hinge: 0.66556203; adversarial_accuracy: 0.3622; base_accuracy: 0.7068; base_hinge: 0.33437443; min_base_hinge: 0.33208665; since_best_base_hinge: 1;
FastEstimator-Train: step: 19500; base_hinge: 0.14611092; steps/sec: 97.02;
FastEstimator-Train: step: 20000; base_hinge: 0.2760552; steps/sec: 130.41;
FastEstimator-Train: step: 20000; epoch: 20; epoch_time: 8.99 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 20000; epoch: 20; adv_hinge: 0.6541269; adversarial_accuracy: 0.3698; base_accuracy: 0.715; base_hinge: 0.32563835; min_base_hinge: 0.32563835; since_best_base_hinge: 0;
FastEstimator-Train: step: 20500; base_hinge: 0.14353013; steps/sec: 96.85;
FastEstimator-Train: step: 21000; base_hinge: 0.20877028; steps/sec: 120.18;
FastEstimator-Train: step: 21000; epoch: 21; epoch_time: 9.32 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 21000; epoch: 21; adv_hinge: 0.6526895; adversarial_accuracy: 0.3674; base_accuracy: 0.7182; base_hinge: 0.31998664; min_base_hinge: 0.31998664; since_best_base_hinge: 0;
FastEstimator-Train: step: 21500; base_hinge: 0.20144889; steps/sec: 101.66;
FastEstimator-Train: step: 22000; base_hinge: 0.28719658; steps/sec: 128.25;
FastEstimator-Train: step: 22000; epoch: 22; epoch_time: 8.84 sec;
FastEstimator-Eval: step: 22000; epoch: 22; adv_hinge: 0.6620615; adversarial_accuracy: 0.3674; base_accuracy: 0.7204; base_hinge: 0.32229984; min_base_hinge: 0.31998664; since_best_base_hinge: 1;
FastEstimator-Train: step: 22500; base_hinge: 0.16186333; steps/sec: 98.64;
FastEstimator-Train: step: 23000; base_hinge: 0.27358246; steps/sec: 122.3;
FastEstimator-Train: step: 23000; epoch: 23; epoch_time: 9.16 sec;
FastEstimator-Eval: step: 23000; epoch: 23; adv_hinge: 0.6603125; adversarial_accuracy: 0.3658; base_accuracy: 0.7124; base_hinge: 0.32517755; min_base_hinge: 0.31998664; since_best_base_hinge: 2;
FastEstimator-Train: step: 23500; base_hinge: 0.1928054; steps/sec: 93.02;
FastEstimator-Train: step: 24000; base_hinge: 0.22112942; steps/sec: 111.7;
FastEstimator-Train: step: 24000; epoch: 24; epoch_time: 9.83 sec;
FastEstimator-Eval: step: 24000; epoch: 24; adv_hinge: 0.6635312; adversarial_accuracy: 0.362; base_accuracy: 0.7162; base_hinge: 0.32222956; min_base_hinge: 0.31998664; since_best_base_hinge: 3;
FastEstimator-Train: step: 24500; base_hinge: 0.18369232; steps/sec: 93.88;
FastEstimator-Train: step: 25000; base_hinge: 0.3199598; steps/sec: 116.33;
FastEstimator-Train: step: 25000; epoch: 25; epoch_time: 9.62 sec;
FastEstimator-Eval: step: 25000; epoch: 25; adv_hinge: 0.6594223; adversarial_accuracy: 0.3656; base_accuracy: 0.7094; base_hinge: 0.32736686; min_base_hinge: 0.31998664; since_best_base_hinge: 4;
FastEstimator-Train: step: 25500; base_hinge: 0.27891782; steps/sec: 95.58;
FastEstimator-Train: step: 26000; base_hinge: 0.21763876; steps/sec: 121.42;
FastEstimator-Train: step: 26000; epoch: 26; epoch_time: 9.35 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 26000; epoch: 26; adv_hinge: 0.66260374; adversarial_accuracy: 0.3638; base_accuracy: 0.722; base_hinge: 0.31514966; min_base_hinge: 0.31514966; since_best_base_hinge: 0;
FastEstimator-Train: step: 26500; base_hinge: 0.21337055; steps/sec: 97.9;
FastEstimator-Train: step: 27000; base_hinge: 0.2819699; steps/sec: 124.37;
FastEstimator-Train: step: 27000; epoch: 27; epoch_time: 9.12 sec;
FastEstimator-Eval: step: 27000; epoch: 27; adv_hinge: 0.6798921; adversarial_accuracy: 0.3464; base_accuracy: 0.7214; base_hinge: 0.31680486; min_base_hinge: 0.31514966; since_best_base_hinge: 1;
FastEstimator-Train: step: 27500; base_hinge: 0.19231571; steps/sec: 105.91;
FastEstimator-Train: step: 28000; base_hinge: 0.28198642; steps/sec: 131.18;
FastEstimator-Train: step: 28000; epoch: 28; epoch_time: 8.53 sec;
FastEstimator-Eval: step: 28000; epoch: 28; adv_hinge: 0.66766024; adversarial_accuracy: 0.3614; base_accuracy: 0.7218; base_hinge: 0.31777543; min_base_hinge: 0.31514966; since_best_base_hinge: 2;
FastEstimator-Train: step: 28500; base_hinge: 0.19805568; steps/sec: 103.93;
FastEstimator-Train: step: 29000; base_hinge: 0.2087434; steps/sec: 127.45;
FastEstimator-Train: step: 29000; epoch: 29; epoch_time: 8.75 sec;
FastEstimator-Eval: step: 29000; epoch: 29; adv_hinge: 0.67013097; adversarial_accuracy: 0.359; base_accuracy: 0.7214; base_hinge: 0.31696904; min_base_hinge: 0.31514966; since_best_base_hinge: 3;
FastEstimator-Train: step: 29500; base_hinge: 0.19910683; steps/sec: 103.06;
FastEstimator-Train: step: 30000; base_hinge: 0.23703016; steps/sec: 128.09;
FastEstimator-Train: step: 30000; epoch: 30; epoch_time: 8.75 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 30000; epoch: 30; adv_hinge: 0.6716123; adversarial_accuracy: 0.3554; base_accuracy: 0.722; base_hinge: 0.31445774; min_base_hinge: 0.31445774; since_best_base_hinge: 0;
FastEstimator-Train: step: 30500; base_hinge: 0.25689024; steps/sec: 104.6;
FastEstimator-Train: step: 31000; base_hinge: 0.16964242; steps/sec: 130.99;
FastEstimator-Train: step: 31000; epoch: 31; epoch_time: 8.6 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 31000; epoch: 31; adv_hinge: 0.6674604; adversarial_accuracy: 0.3616; base_accuracy: 0.727; base_hinge: 0.3101713; min_base_hinge: 0.3101713; since_best_base_hinge: 0;
FastEstimator-Train: step: 31500; base_hinge: 0.24562128; steps/sec: 103.56;
FastEstimator-Train: step: 32000; base_hinge: 0.2353131; steps/sec: 128.88;
FastEstimator-Train: step: 32000; epoch: 32; epoch_time: 8.7 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 32000; epoch: 32; adv_hinge: 0.68069166; adversarial_accuracy: 0.3494; base_accuracy: 0.7338; base_hinge: 0.3074732; min_base_hinge: 0.3074732; since_best_base_hinge: 0;
FastEstimator-Train: step: 32500; base_hinge: 0.14915092; steps/sec: 100.17;
FastEstimator-Train: step: 33000; base_hinge: 0.22575116; steps/sec: 123.11;
FastEstimator-Train: step: 33000; epoch: 33; epoch_time: 9.05 sec;
FastEstimator-Eval: step: 33000; epoch: 33; adv_hinge: 0.68312544; adversarial_accuracy: 0.3436; base_accuracy: 0.7224; base_hinge: 0.310704; min_base_hinge: 0.3074732; since_best_base_hinge: 1;
FastEstimator-Train: step: 33500; base_hinge: 0.21910556; steps/sec: 105.79;
FastEstimator-Train: step: 34000; base_hinge: 0.20641476; steps/sec: 131.46;
FastEstimator-Train: step: 34000; epoch: 34; epoch_time: 8.53 sec;
FastEstimator-Eval: step: 34000; epoch: 34; adv_hinge: 0.6892244; adversarial_accuracy: 0.342; base_accuracy: 0.7234; base_hinge: 0.31533897; min_base_hinge: 0.3074732; since_best_base_hinge: 2;
FastEstimator-Train: step: 34500; base_hinge: 0.17468475; steps/sec: 101.75;
FastEstimator-Train: step: 35000; base_hinge: 0.19306578; steps/sec: 126.45;
FastEstimator-Train: step: 35000; epoch: 35; epoch_time: 8.86 sec;
FastEstimator-Eval: step: 35000; epoch: 35; adv_hinge: 0.6868973; adversarial_accuracy: 0.3454; base_accuracy: 0.7208; base_hinge: 0.31726614; min_base_hinge: 0.3074732; since_best_base_hinge: 3;
FastEstimator-Train: step: 35500; base_hinge: 0.20393735; steps/sec: 94.62;
FastEstimator-Train: step: 36000; base_hinge: 0.16484459; steps/sec: 119.51;
FastEstimator-Train: step: 36000; epoch: 36; epoch_time: 9.47 sec;
FastEstimator-Eval: step: 36000; epoch: 36; adv_hinge: 0.6915254; adversarial_accuracy: 0.3434; base_accuracy: 0.7142; base_hinge: 0.31838906; min_base_hinge: 0.3074732; since_best_base_hinge: 4;
FastEstimator-Train: step: 36500; base_hinge: 0.30909944; steps/sec: 97.77;
FastEstimator-Train: step: 37000; base_hinge: 0.23421745; steps/sec: 122.0;
FastEstimator-Train: step: 37000; epoch: 37; epoch_time: 9.22 sec;
FastEstimator-Eval: step: 37000; epoch: 37; adv_hinge: 0.6845604; adversarial_accuracy: 0.3424; base_accuracy: 0.7224; base_hinge: 0.31112215; min_base_hinge: 0.3074732; since_best_base_hinge: 5;
FastEstimator-Train: step: 37500; base_hinge: 0.16261695; steps/sec: 100.76;
FastEstimator-Train: step: 38000; base_hinge: 0.19233434; steps/sec: 129.8;
FastEstimator-Train: step: 38000; epoch: 38; epoch_time: 8.79 sec;
FastEstimator-Eval: step: 38000; epoch: 38; adv_hinge: 0.67739683; adversarial_accuracy: 0.3542; base_accuracy: 0.7242; base_hinge: 0.31000316; min_base_hinge: 0.3074732; since_best_base_hinge: 6;
FastEstimator-Train: step: 38500; base_hinge: 0.20121937; steps/sec: 93.57;
FastEstimator-Train: step: 39000; base_hinge: 0.13809688; steps/sec: 115.84;
FastEstimator-Train: step: 39000; epoch: 39; epoch_time: 9.68 sec;
FastEstimator-Eval: step: 39000; epoch: 39; adv_hinge: 0.6804919; adversarial_accuracy: 0.3528; base_accuracy: 0.7204; base_hinge: 0.3120458; min_base_hinge: 0.3074732; since_best_base_hinge: 7;
FastEstimator-Train: step: 39500; base_hinge: 0.18955024; steps/sec: 91.47;
FastEstimator-Train: step: 40000; base_hinge: 0.207903; steps/sec: 110.3;
FastEstimator-Train: step: 40000; epoch: 40; epoch_time: 10.0 sec;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Eval: step: 40000; epoch: 40; adv_hinge: 0.6855146; adversarial_accuracy: 0.3444; base_accuracy: 0.731; base_hinge: 0.30590805; min_base_hinge: 0.30590805; since_best_base_hinge: 0;
FastEstimator-Train: step: 40500; base_hinge: 0.083864; steps/sec: 99.87;
FastEstimator-Train: step: 41000; base_hinge: 0.21492122; steps/sec: 129.7;
FastEstimator-Train: step: 41000; epoch: 41; epoch_time: 8.87 sec;
FastEstimator-Eval: step: 41000; epoch: 41; adv_hinge: 0.67458624; adversarial_accuracy: 0.3572; base_accuracy: 0.7192; base_hinge: 0.31428564; min_base_hinge: 0.30590805; since_best_base_hinge: 1;
FastEstimator-Train: step: 41500; base_hinge: 0.11375672; steps/sec: 99.68;
FastEstimator-Train: step: 42000; base_hinge: 0.12876363; steps/sec: 129.34;
FastEstimator-Train: step: 42000; epoch: 42; epoch_time: 8.88 sec;
FastEstimator-Eval: step: 42000; epoch: 42; adv_hinge: 0.6846123; adversarial_accuracy: 0.3488; base_accuracy: 0.7218; base_hinge: 0.312879; min_base_hinge: 0.30590805; since_best_base_hinge: 2;
FastEstimator-Train: step: 42500; base_hinge: 0.23576143; steps/sec: 99.47;
FastEstimator-Train: step: 43000; base_hinge: 0.23091209; steps/sec: 128.26;
FastEstimator-Train: step: 43000; epoch: 43; epoch_time: 8.93 sec;
FastEstimator-Eval: step: 43000; epoch: 43; adv_hinge: 0.6828584; adversarial_accuracy: 0.3462; base_accuracy: 0.7256; base_hinge: 0.31104147; min_base_hinge: 0.30590805; since_best_base_hinge: 3;
FastEstimator-Train: step: 43500; base_hinge: 0.16894166; steps/sec: 99.16;
FastEstimator-Train: step: 44000; base_hinge: 0.11686382; steps/sec: 127.34;
FastEstimator-Train: step: 44000; epoch: 44; epoch_time: 8.97 sec;
FastEstimator-Eval: step: 44000; epoch: 44; adv_hinge: 0.68396837; adversarial_accuracy: 0.3502; base_accuracy: 0.7228; base_hinge: 0.31255093; min_base_hinge: 0.30590805; since_best_base_hinge: 4;
FastEstimator-Train: step: 44500; base_hinge: 0.21182784; steps/sec: 97.19;
FastEstimator-Train: step: 45000; base_hinge: 0.20819433; steps/sec: 125.21;
FastEstimator-Train: step: 45000; epoch: 45; epoch_time: 9.13 sec;
FastEstimator-Eval: step: 45000; epoch: 45; adv_hinge: 0.67581666; adversarial_accuracy: 0.3584; base_accuracy: 0.7186; base_hinge: 0.3161821; min_base_hinge: 0.30590805; since_best_base_hinge: 5;
FastEstimator-Train: step: 45500; base_hinge: 0.13874252; steps/sec: 98.17;
FastEstimator-Train: step: 46000; base_hinge: 0.11777561; steps/sec: 126.43;
FastEstimator-Train: step: 46000; epoch: 46; epoch_time: 9.05 sec;
FastEstimator-Eval: step: 46000; epoch: 46; adv_hinge: 0.6749865; adversarial_accuracy: 0.3574; base_accuracy: 0.7286; base_hinge: 0.3084245; min_base_hinge: 0.30590805; since_best_base_hinge: 6;
FastEstimator-Train: step: 46500; base_hinge: 0.12897556; steps/sec: 94.37;
FastEstimator-Train: step: 47000; base_hinge: 0.1589519; steps/sec: 120.69;
FastEstimator-Train: step: 47000; epoch: 47; epoch_time: 9.45 sec;
FastEstimator-Eval: step: 47000; epoch: 47; adv_hinge: 0.6834855; adversarial_accuracy: 0.3484; base_accuracy: 0.7268; base_hinge: 0.30969155; min_base_hinge: 0.30590805; since_best_base_hinge: 7;
FastEstimator-Train: step: 47500; base_hinge: 0.21197909; steps/sec: 99.77;
FastEstimator-Train: step: 48000; base_hinge: 0.26024407; steps/sec: 128.18;
FastEstimator-Train: step: 48000; epoch: 48; epoch_time: 8.91 sec;
FastEstimator-Eval: step: 48000; epoch: 48; adv_hinge: 0.66962576; adversarial_accuracy: 0.3646; base_accuracy: 0.7244; base_hinge: 0.3099848; min_base_hinge: 0.30590805; since_best_base_hinge: 8;
FastEstimator-Train: step: 48500; base_hinge: 0.15549628; steps/sec: 101.18;
FastEstimator-Train: step: 49000; base_hinge: 0.17724009; steps/sec: 133.42;
FastEstimator-Train: step: 49000; epoch: 49; epoch_time: 8.69 sec;
FastEstimator-Eval: step: 49000; epoch: 49; adv_hinge: 0.6587408; adversarial_accuracy: 0.3744; base_accuracy: 0.7304; base_hinge: 0.30658415; min_base_hinge: 0.30590805; since_best_base_hinge: 9;
FastEstimator-Train: step: 49500; base_hinge: 0.2888211; steps/sec: 95.17;
FastEstimator-Train: step: 50000; base_hinge: 0.1778168; steps/sec: 122.93;
FastEstimator-Train: step: 50000; epoch: 50; epoch_time: 9.32 sec;
FastEstimator-EarlyStopping: 'base_hinge' triggered an early stop. Its best value was 0.30590805411338806 at epoch 40
FastEstimator-Eval: step: 50000; epoch: 50; adv_hinge: 0.67139465; adversarial_accuracy: 0.3584; base_accuracy: 0.7264; base_hinge: 0.30914956; min_base_hinge: 0.30590805; since_best_base_hinge: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmp0u08o2n6/hydra_ecc_best_base_hinge.h5
FastEstimator-Finish: step: 50000; hydra_ecc_lr: 0.001; total_time: 549.55 sec;
FastEstimator-Test: step: 50000; epoch: 50; adv_hinge: 0.69422853; adversarial_accuracy: 0.3418; base_accuracy: 0.7168; base_hinge: 0.31933317;

Comparing the Results¶

In [15]:

logs = visualize_logs(experiments=[softmax_results, ecc_results, hydra_results], ignore_metrics={'ecc_lr', 'hydra_ecc_lr', 'softmax_lr', 'logging_interval', 'num_device', 'epoch_time', 'min_base_ce', 'adv_ce', 'total_time'})

logs = visualize_logs(experiments=[softmax_results, ecc_results, hydra_results], ignore_metrics={'ecc_lr', 'hydra_ecc_lr', 'softmax_lr', 'logging_interval', 'num_device', 'epoch_time', 'min_base_ce', 'adv_ce', 'total_time'})

As you can see, the conventional network using softmax to convert logits to class probabilities actually gets more and more vulnerable to adversarial attacks as training progresses. It also quickly overfits to the data, reaching an optimal performance around epoch 7. By switching the output layer of the model to generate an error correcting code and training with hinge loss, the model is able to train almost 6 times longer before reaching peak conventional accuracy. Moreover, the adversarial performance of the network continues to improve even after the main training runs out. This is significantly better performance than networks trained specifically to combat this attack, shown in the FGSM notebook. It can also be seen that there is no additional cost to training using ECC as opposed to softmax in terms of steps/sec. This is a big benefit over FGSM, where the training time for each step is doubled. With these benefits in mind, you may want to consider never using softmax again.