Adversarial Robustness with Error Correcting Codes¶

(You might never use Softmax again)¶

[Paper] [Notebook] [TF Implementation] [Torch Implementation]

In this example we will show how using error correcting codes to convert model logits to probabilities can drastically reduce model overfitting while simultaneously increasing model robustness against adversarial attacks. In other words, why you should never use a softmax layer again. This phenomena was first publicized by the US Army in a 2019 Neurips Paper. For background on adversarial attacks, and on the attack type we will be demonstrating here, check out our FGSM apphub example. Note that in this apphub we will not be training against adversarial samples, but only performing adversarial attacks during evaluation to see how different models fair against them.

Imports¶

In [1]:

import tempfile

from tensorflow.keras import Sequential, layers
from tensorflow.keras.layers import Conv2D, Dense, Flatten, Input, MaxPooling2D
from tensorflow.keras.models import Model

import fastestimator as fe
from fastestimator.architecture.tensorflow import LeNet
from fastestimator.dataset.data import cifair10
from fastestimator.layers.tensorflow import HadamardCode
from fastestimator.op.numpyop.univariate import Normalize
from fastestimator.op.tensorop.gradient import FGSM, Watch
from fastestimator.op.tensorop.loss import CrossEntropy
from fastestimator.op.tensorop.model import ModelOp, UpdateOp
from fastestimator.summary.logs import visualize_logs
from fastestimator.trace.adapt import EarlyStopping
from fastestimator.trace.io import BestModelSaver
from fastestimator.trace.metric import Accuracy

import tempfile from tensorflow.keras import Sequential, layers from tensorflow.keras.layers import Conv2D, Dense, Flatten, Input, MaxPooling2D from tensorflow.keras.models import Model import fastestimator as fe from fastestimator.architecture.tensorflow import LeNet from fastestimator.dataset.data import cifair10 from fastestimator.layers.tensorflow import HadamardCode from fastestimator.op.numpyop.univariate import Normalize from fastestimator.op.tensorop.gradient import FGSM, Watch from fastestimator.op.tensorop.loss import CrossEntropy from fastestimator.op.tensorop.model import ModelOp, UpdateOp from fastestimator.summary.logs import visualize_logs from fastestimator.trace.adapt import EarlyStopping from fastestimator.trace.io import BestModelSaver from fastestimator.trace.metric import Accuracy

In [2]:

parameters

# training parameters
epsilon=0.04  # The strength of the adversarial attack
epochs=60
batch_size=50
log_steps=500
train_steps_per_epoch=None
eval_steps_per_epoch=None
save_dir=tempfile.mkdtemp()

# training parameters epsilon=0.04 # The strength of the adversarial attack epochs=60 batch_size=50 log_steps=500 train_steps_per_epoch=None eval_steps_per_epoch=None save_dir=tempfile.mkdtemp()

Getting the Data¶

For these experiments we will be using the ciFAIR10 Dataset (like cifar10, but with duplicate testing data replaced)

In [3]:

train_data, eval_data = cifair10.load_data()
test_data = eval_data.split(0.5)
pipeline = fe.Pipeline(
    train_data=train_data,
    eval_data=eval_data,
    test_data=test_data,
    batch_size=batch_size,
    ops=[Normalize(inputs="x", outputs="x", mean=(0.4914, 0.4822, 0.4465), std=(0.2471, 0.2435, 0.2616))])

train_data, eval_data = cifair10.load_data() test_data = eval_data.split(0.5) pipeline = fe.Pipeline( train_data=train_data, eval_data=eval_data, test_data=test_data, batch_size=batch_size, ops=[Normalize(inputs="x", outputs="x", mean=(0.4914, 0.4822, 0.4465), std=(0.2471, 0.2435, 0.2616))])

Defining an Estimator¶

In this apphub we will be comparing three very similar models, all using the same training and evaluation routines. Hence a function to generate the estimators:

In [4]:

def get_estimator(model):
    network = fe.Network(ops=[
        Watch(inputs="x", mode=('eval', 'test')),
        ModelOp(model=model, inputs="x", outputs="y_pred"),
        CrossEntropy(inputs=("y_pred", "y"), outputs="base_ce"),
        UpdateOp(model=model, loss_name="base_ce"),
        FGSM(data="x", loss="base_ce", outputs="x_adverse", epsilon=epsilon, mode=('eval', 'test')),
        ModelOp(model=model, inputs="x_adverse", outputs="y_pred_adv", mode=('eval', 'test')),
        CrossEntropy(inputs=("y_pred_adv", "y"), outputs="adv_ce", mode=('eval', 'test'))
    ])
    traces = [
        Accuracy(true_key="y", pred_key="y_pred", output_name="base accuracy"),
        Accuracy(true_key="y", pred_key="y_pred_adv", output_name="adversarial accuracy"),
        BestModelSaver(model=model, save_dir=save_dir, metric="base_ce", save_best_mode="min", load_best_final=True),
        EarlyStopping(monitor="base_ce", patience=10)
    ]
    estimator = fe.Estimator(pipeline=pipeline,
                             network=network,
                             epochs=epochs,
                             traces=traces,
                             log_steps=log_steps,
                             train_steps_per_epoch=train_steps_per_epoch,
                             eval_steps_per_epoch=eval_steps_per_epoch,
                             monitor_names=["adv_ce"])
    return estimator

def get_estimator(model): network = fe.Network(ops=[ Watch(inputs="x", mode=('eval', 'test')), ModelOp(model=model, inputs="x", outputs="y_pred"), CrossEntropy(inputs=("y_pred", "y"), outputs="base_ce"), UpdateOp(model=model, loss_name="base_ce"), FGSM(data="x", loss="base_ce", outputs="x_adverse", epsilon=epsilon, mode=('eval', 'test')), ModelOp(model=model, inputs="x_adverse", outputs="y_pred_adv", mode=('eval', 'test')), CrossEntropy(inputs=("y_pred_adv", "y"), outputs="adv_ce", mode=('eval', 'test')) ]) traces = [ Accuracy(true_key="y", pred_key="y_pred", output_name="base accuracy"), Accuracy(true_key="y", pred_key="y_pred_adv", output_name="adversarial accuracy"), BestModelSaver(model=model, save_dir=save_dir, metric="base_ce", save_best_mode="min", load_best_final=True), EarlyStopping(monitor="base_ce", patience=10) ] estimator = fe.Estimator(pipeline=pipeline, network=network, epochs=epochs, traces=traces, log_steps=log_steps, train_steps_per_epoch=train_steps_per_epoch, eval_steps_per_epoch=eval_steps_per_epoch, monitor_names=["adv_ce"]) return estimator

The Models¶

1 - A LeNet model with Softmax¶

In [5]:

softmax_model = fe.build(model_fn=lambda:LeNet(input_shape=(32, 32, 3)), optimizer_fn="adam", model_name='softmax')

softmax_model = fe.build(model_fn=lambda:LeNet(input_shape=(32, 32, 3)), optimizer_fn="adam", model_name='softmax')

Metal device set to: Apple M1 Max

2023-03-22 11:04:20.608318: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-03-22 11:04:20.608337: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)

2 - A LeNet model with Error Correcting Codes¶

In [6]:

def EccLeNet(input_shape=(32, 32, 3), classes=10):
    model = Sequential()
    model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.Flatten())
    model.add(layers.Dense(64, activation='relu'))
    model.add(HadamardCode(classes))  # Note that this is the only difference between this model and the FE LeNet implementation
    return model

def EccLeNet(input_shape=(32, 32, 3), classes=10): model = Sequential() model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape)) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation='relu')) model.add(layers.Flatten()) model.add(layers.Dense(64, activation='relu')) model.add(HadamardCode(classes)) # Note that this is the only difference between this model and the FE LeNet implementation return model

In [7]:

ecc_model = fe.build(model_fn=EccLeNet, optimizer_fn="adam", model_name='ecc')

ecc_model = fe.build(model_fn=EccLeNet, optimizer_fn="adam", model_name='ecc')

3 - A LeNet model using ECC and multiple feature heads¶

While it is common practice to follow the feature extraction layers of convolution networks with several fully connected layers in order to perform classification, this can lead to the final logits being interdependent which can actually reduce the robustness of the network. One way around this is to divide your classification layers into multiple smaller independent units:

In [8]:

def HydraEccLeNet(input_shape=(32, 32, 3), classes=10):
    inputs = Input(input_shape)
    conv1 = Conv2D(32, (3, 3), activation='relu')(inputs)
    pool1 = MaxPooling2D((2, 2))(conv1)
    conv2 = Conv2D(64, (3, 3), activation='relu')(pool1)
    pool2 = MaxPooling2D((2, 2))(conv2)
    conv3 = Conv2D(64, (3, 3), activation='relu')(pool2)
    flat = Flatten()(conv3)
    # Create multiple heads
    n_heads = 4
    heads = [Dense(16, activation='relu')(flat) for _ in range(n_heads)]
    outputs = HadamardCode(classes)(heads)
    return Model(inputs=inputs, outputs=outputs)

def HydraEccLeNet(input_shape=(32, 32, 3), classes=10): inputs = Input(input_shape) conv1 = Conv2D(32, (3, 3), activation='relu')(inputs) pool1 = MaxPooling2D((2, 2))(conv1) conv2 = Conv2D(64, (3, 3), activation='relu')(pool1) pool2 = MaxPooling2D((2, 2))(conv2) conv3 = Conv2D(64, (3, 3), activation='relu')(pool2) flat = Flatten()(conv3) # Create multiple heads n_heads = 4 heads = [Dense(16, activation='relu')(flat) for _ in range(n_heads)] outputs = HadamardCode(classes)(heads) return Model(inputs=inputs, outputs=outputs)

In [9]:

hydra_model = fe.build(model_fn=HydraEccLeNet, optimizer_fn="adam", model_name='hydra_ecc')

hydra_model = fe.build(model_fn=HydraEccLeNet, optimizer_fn="adam", model_name='hydra_ecc')

The Experiments¶

Let's get Estimators for each of these models and compare them:

In [10]:

softmax_estimator = get_estimator(softmax_model)
ecc_estimator = get_estimator(ecc_model)
hydra_estimator = get_estimator(hydra_model)

softmax_estimator = get_estimator(softmax_model) ecc_estimator = get_estimator(ecc_model) hydra_estimator = get_estimator(hydra_model)

In [11]:

softmax_estimator.fit('Softmax')
softmax_results = softmax_estimator.test()

softmax_estimator.fit('Softmax') softmax_results = softmax_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; logging_interval: 500; num_device: 1;
WARNING:tensorflow:From /Users/skynet/.pyenv/versions/miniforge3-4.10.3-10/envs/femos38/lib/python3.8/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089
FastEstimator-Train: step: 1; base_ce: 2.3516564;
FastEstimator-Train: step: 500; base_ce: 1.5423635; steps/sec: 146.94;
FastEstimator-Train: step: 1000; base_ce: 1.1915147; steps/sec: 144.8;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time(sec): 7.6;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 134.24;
Eval Progress: 66/100; steps/sec: 134.05;
Eval Progress: 100/100; steps/sec: 131.98;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/softmax_best_base_ce.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_ce: 1.9812572; adversarial accuracy: 0.3012; base accuracy: 0.5928; base_ce: 1.1643765; min_base_ce: 1.1643765; since_best_base_ce: 0;
FastEstimator-Train: step: 1500; base_ce: 0.8733902; steps/sec: 141.21;
FastEstimator-Train: step: 2000; base_ce: 0.8287111; steps/sec: 168.07;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time(sec): 6.51;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 129.19;
Eval Progress: 66/100; steps/sec: 122.96;
Eval Progress: 100/100; steps/sec: 129.95;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/softmax_best_base_ce.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_ce: 2.2336564; adversarial accuracy: 0.2836; base accuracy: 0.6398; base_ce: 1.0206374; min_base_ce: 1.0206374; since_best_base_ce: 0;
FastEstimator-Train: step: 2500; base_ce: 1.0899659; steps/sec: 138.34;
FastEstimator-Train: step: 3000; base_ce: 0.82012707; steps/sec: 152.94;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time(sec): 6.89;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 117.43;
Eval Progress: 66/100; steps/sec: 121.88;
Eval Progress: 100/100; steps/sec: 114.99;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/softmax_best_base_ce.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_ce: 2.2293031; adversarial accuracy: 0.2946; base accuracy: 0.6822; base_ce: 0.89989495; min_base_ce: 0.89989495; since_best_base_ce: 0;
FastEstimator-Train: step: 3500; base_ce: 0.88229865; steps/sec: 138.06;
FastEstimator-Train: step: 4000; base_ce: 0.8454884; steps/sec: 163.37;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time(sec): 6.68;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 112.31;
Eval Progress: 66/100; steps/sec: 117.14;
Eval Progress: 100/100; steps/sec: 128.34;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/softmax_best_base_ce.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_ce: 2.4691896; adversarial accuracy: 0.2742; base accuracy: 0.694; base_ce: 0.8833518; min_base_ce: 0.8833518; since_best_base_ce: 0;
FastEstimator-Train: step: 4500; base_ce: 0.7957852; steps/sec: 137.42;
FastEstimator-Train: step: 5000; base_ce: 0.6769937; steps/sec: 149.43;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time(sec): 6.98;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 121.88;
Eval Progress: 66/100; steps/sec: 119.23;
Eval Progress: 100/100; steps/sec: 120.26;
FastEstimator-Eval: step: 5000; epoch: 5; adv_ce: 2.7209516; adversarial accuracy: 0.2482; base accuracy: 0.694; base_ce: 0.88381416; min_base_ce: 0.8833518; since_best_base_ce: 1;
FastEstimator-Train: step: 5500; base_ce: 0.53626794; steps/sec: 133.48;
FastEstimator-Train: step: 6000; base_ce: 0.49186733; steps/sec: 145.81;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time(sec): 7.18;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 127.83;
Eval Progress: 66/100; steps/sec: 132.57;
Eval Progress: 100/100; steps/sec: 133.8;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/softmax_best_base_ce.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_ce: 3.007286; adversarial accuracy: 0.2452; base accuracy: 0.7136; base_ce: 0.84367865; min_base_ce: 0.84367865; since_best_base_ce: 0;
FastEstimator-Train: step: 6500; base_ce: 0.7859265; steps/sec: 143.24;
FastEstimator-Train: step: 7000; base_ce: 0.58587563; steps/sec: 170.79;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time(sec): 6.42;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 123.72;
Eval Progress: 66/100; steps/sec: 111.93;
Eval Progress: 100/100; steps/sec: 120.45;
FastEstimator-Eval: step: 7000; epoch: 7; adv_ce: 3.1947508; adversarial accuracy: 0.2418; base accuracy: 0.714; base_ce: 0.845398; min_base_ce: 0.84367865; since_best_base_ce: 1;
FastEstimator-Train: step: 7500; base_ce: 0.55412424; steps/sec: 140.0;
FastEstimator-Train: step: 8000; base_ce: 0.451165; steps/sec: 153.18;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time(sec): 6.83;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 121.28;
Eval Progress: 66/100; steps/sec: 130.78;
Eval Progress: 100/100; steps/sec: 130.83;
FastEstimator-Eval: step: 8000; epoch: 8; adv_ce: 3.3819828; adversarial accuracy: 0.2188; base accuracy: 0.712; base_ce: 0.8487379; min_base_ce: 0.84367865; since_best_base_ce: 2;
FastEstimator-Train: step: 8500; base_ce: 0.27468503; steps/sec: 143.02;
FastEstimator-Train: step: 9000; base_ce: 0.57739204; steps/sec: 166.78;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time(sec): 6.51;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 116.69;
Eval Progress: 66/100; steps/sec: 130.99;
Eval Progress: 100/100; steps/sec: 130.77;
FastEstimator-Eval: step: 9000; epoch: 9; adv_ce: 3.7390258; adversarial accuracy: 0.2012; base accuracy: 0.7168; base_ce: 0.8678848; min_base_ce: 0.84367865; since_best_base_ce: 3;
FastEstimator-Train: step: 9500; base_ce: 0.17839043; steps/sec: 133.43;
FastEstimator-Train: step: 10000; base_ce: 0.2089923; steps/sec: 147.07;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time(sec): 7.14;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 122.19;
Eval Progress: 66/100; steps/sec: 129.09;
Eval Progress: 100/100; steps/sec: 122.89;
FastEstimator-Eval: step: 10000; epoch: 10; adv_ce: 4.2170534; adversarial accuracy: 0.2024; base accuracy: 0.7142; base_ce: 0.9080029; min_base_ce: 0.84367865; since_best_base_ce: 4;
FastEstimator-Train: step: 10500; base_ce: 0.40431014; steps/sec: 130.08;
FastEstimator-Train: step: 11000; base_ce: 0.35620907; steps/sec: 145.51;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time(sec): 7.28;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 126.47;
Eval Progress: 66/100; steps/sec: 129.52;
Eval Progress: 100/100; steps/sec: 122.61;
FastEstimator-Eval: step: 11000; epoch: 11; adv_ce: 4.59961; adversarial accuracy: 0.1826; base accuracy: 0.7064; base_ce: 0.959485; min_base_ce: 0.84367865; since_best_base_ce: 5;
FastEstimator-Train: step: 11500; base_ce: 0.45397577; steps/sec: 143.44;
FastEstimator-Train: step: 12000; base_ce: 0.49342728; steps/sec: 169.24;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time(sec): 6.44;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 114.56;
Eval Progress: 66/100; steps/sec: 123.65;
Eval Progress: 100/100; steps/sec: 126.46;
FastEstimator-Eval: step: 12000; epoch: 12; adv_ce: 5.3441854; adversarial accuracy: 0.179; base accuracy: 0.7098; base_ce: 1.0305326; min_base_ce: 0.84367865; since_best_base_ce: 6;
FastEstimator-Train: step: 12500; base_ce: 0.25620738; steps/sec: 141.14;
FastEstimator-Train: step: 13000; base_ce: 0.3342251; steps/sec: 154.05;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time(sec): 6.8;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 117.09;
Eval Progress: 66/100; steps/sec: 125.28;
Eval Progress: 100/100; steps/sec: 130.2;
FastEstimator-Eval: step: 13000; epoch: 13; adv_ce: 5.671794; adversarial accuracy: 0.177; base accuracy: 0.7138; base_ce: 1.0395858; min_base_ce: 0.84367865; since_best_base_ce: 7;
FastEstimator-Train: step: 13500; base_ce: 0.15029068; steps/sec: 139.61;
FastEstimator-Train: step: 14000; base_ce: 0.40083763; steps/sec: 160.46;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time(sec): 6.69;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 112.09;
Eval Progress: 66/100; steps/sec: 133.57;
Eval Progress: 100/100; steps/sec: 130.07;
FastEstimator-Eval: step: 14000; epoch: 14; adv_ce: 6.048907; adversarial accuracy: 0.161; base accuracy: 0.7088; base_ce: 1.0822965; min_base_ce: 0.84367865; since_best_base_ce: 8;
FastEstimator-Train: step: 14500; base_ce: 0.27216437; steps/sec: 140.58;
FastEstimator-Train: step: 15000; base_ce: 0.32581636; steps/sec: 163.1;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time(sec): 6.62;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 115.28;
Eval Progress: 66/100; steps/sec: 123.02;
Eval Progress: 100/100; steps/sec: 126.88;
FastEstimator-Eval: step: 15000; epoch: 15; adv_ce: 6.604896; adversarial accuracy: 0.1682; base accuracy: 0.7028; base_ce: 1.1822042; min_base_ce: 0.84367865; since_best_base_ce: 9;
FastEstimator-Train: step: 15500; base_ce: 0.2126595; steps/sec: 139.96;
FastEstimator-Train: step: 16000; base_ce: 0.11821444; steps/sec: 157.3;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time(sec): 6.76;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 113.68;
Eval Progress: 66/100; steps/sec: 122.68;
Eval Progress: 100/100; steps/sec: 132.03;
FastEstimator-EarlyStopping: 'base_ce' triggered an early stop. Its best value was 0.8436786532402039 at epoch 6
FastEstimator-Eval: step: 16000; epoch: 16; adv_ce: 7.193444; adversarial accuracy: 0.1502; base accuracy: 0.7048; base_ce: 1.2549533; min_base_ce: 0.84367865; since_best_base_ce: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/softmax_best_base_ce.h5
FastEstimator-Finish: step: 16000; softmax_lr: 0.001; total_time(sec): 129.56;
FastEstimator-Test: step: 16000; epoch: 16; adv_ce: 3.020646; adversarial accuracy: 0.2434; base accuracy: 0.7222; base_ce: 0.84307134;

In [12]:

ecc_estimator.fit('ECC')
ecc_results = ecc_estimator.test()

ecc_estimator.fit('ECC') ecc_results = ecc_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; logging_interval: 500; num_device: 1;
FastEstimator-Train: step: 1; base_ce: 2.3042867;
FastEstimator-Train: step: 500; base_ce: 1.8377846; steps/sec: 124.87;
FastEstimator-Train: step: 1000; base_ce: 1.9707724; steps/sec: 131.01;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time(sec): 8.49;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 82.7;
Eval Progress: 66/100; steps/sec: 98.98;
Eval Progress: 100/100; steps/sec: 101.73;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_ce: 2.182454; adversarial accuracy: 0.301; base accuracy: 0.4734; base_ce: 1.7416534; min_base_ce: 1.7416534; since_best_base_ce: 0;
FastEstimator-Train: step: 1500; base_ce: 1.8481773; steps/sec: 112.18;
FastEstimator-Train: step: 2000; base_ce: 1.3395404; steps/sec: 131.11;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time(sec): 8.27;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 84.06;
Eval Progress: 66/100; steps/sec: 99.5;
Eval Progress: 100/100; steps/sec: 100.09;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_ce: 2.228416; adversarial accuracy: 0.335; base accuracy: 0.5608; base_ce: 1.5117651; min_base_ce: 1.5117651; since_best_base_ce: 0;
FastEstimator-Train: step: 2500; base_ce: 1.6211283; steps/sec: 107.45;
FastEstimator-Train: step: 3000; base_ce: 1.2883166; steps/sec: 130.24;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time(sec): 8.49;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 72.0;
Eval Progress: 66/100; steps/sec: 99.42;
Eval Progress: 100/100; steps/sec: 102.44;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_ce: 2.2480683; adversarial accuracy: 0.3434; base accuracy: 0.6142; base_ce: 1.4008938; min_base_ce: 1.4008938; since_best_base_ce: 0;
FastEstimator-Train: step: 3500; base_ce: 1.4265198; steps/sec: 107.67;
FastEstimator-Train: step: 4000; base_ce: 1.3448749; steps/sec: 129.96;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time(sec): 8.49;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.2;
Eval Progress: 66/100; steps/sec: 101.76;
Eval Progress: 100/100; steps/sec: 107.84;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_ce: 2.3175824; adversarial accuracy: 0.3538; base accuracy: 0.6424; base_ce: 1.3064942; min_base_ce: 1.3064942; since_best_base_ce: 0;
FastEstimator-Train: step: 4500; base_ce: 1.0399961; steps/sec: 107.38;
FastEstimator-Train: step: 5000; base_ce: 1.3521281; steps/sec: 124.06;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time(sec): 8.7;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 79.81;
Eval Progress: 66/100; steps/sec: 111.25;
Eval Progress: 100/100; steps/sec: 111.08;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_ce: 2.4307284; adversarial accuracy: 0.3348; base accuracy: 0.6528; base_ce: 1.2984786; min_base_ce: 1.2984786; since_best_base_ce: 0;
FastEstimator-Train: step: 5500; base_ce: 1.2796811; steps/sec: 95.37;
FastEstimator-Train: step: 6000; base_ce: 1.217518; steps/sec: 119.79;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time(sec): 9.42;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 81.14;
Eval Progress: 66/100; steps/sec: 103.0;
Eval Progress: 100/100; steps/sec: 98.68;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_ce: 2.3787057; adversarial accuracy: 0.354; base accuracy: 0.674; base_ce: 1.2059352; min_base_ce: 1.2059352; since_best_base_ce: 0;
FastEstimator-Train: step: 6500; base_ce: 1.3439028; steps/sec: 102.48;
FastEstimator-Train: step: 7000; base_ce: 1.0842909; steps/sec: 128.78;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time(sec): 8.76;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 82.98;
Eval Progress: 66/100; steps/sec: 108.91;
Eval Progress: 100/100; steps/sec: 112.25;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_ce: 2.3812196; adversarial accuracy: 0.3724; base accuracy: 0.6778; base_ce: 1.2054927; min_base_ce: 1.2054927; since_best_base_ce: 0;
FastEstimator-Train: step: 7500; base_ce: 1.0006202; steps/sec: 106.09;
FastEstimator-Train: step: 8000; base_ce: 1.028763; steps/sec: 124.46;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time(sec): 8.73;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 80.75;
Eval Progress: 66/100; steps/sec: 109.11;
Eval Progress: 100/100; steps/sec: 111.46;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 8000; epoch: 8; adv_ce: 2.5138474; adversarial accuracy: 0.3722; base accuracy: 0.6846; base_ce: 1.183277; min_base_ce: 1.183277; since_best_base_ce: 0;
FastEstimator-Train: step: 8500; base_ce: 0.7852113; steps/sec: 108.32;
FastEstimator-Train: step: 9000; base_ce: 0.79858696; steps/sec: 128.36;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time(sec): 8.51;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.34;
Eval Progress: 66/100; steps/sec: 108.99;
Eval Progress: 100/100; steps/sec: 112.43;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 9000; epoch: 9; adv_ce: 2.4589732; adversarial accuracy: 0.3836; base accuracy: 0.6956; base_ce: 1.1495961; min_base_ce: 1.1495961; since_best_base_ce: 0;
FastEstimator-Train: step: 9500; base_ce: 0.9919135; steps/sec: 103.85;
FastEstimator-Train: step: 10000; base_ce: 1.4588437; steps/sec: 120.11;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time(sec): 8.98;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 76.94;
Eval Progress: 66/100; steps/sec: 107.46;
Eval Progress: 100/100; steps/sec: 110.06;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 10000; epoch: 10; adv_ce: 2.5377104; adversarial accuracy: 0.3844; base accuracy: 0.692; base_ce: 1.1493763; min_base_ce: 1.1493763; since_best_base_ce: 0;
FastEstimator-Train: step: 10500; base_ce: 0.79470366; steps/sec: 102.21;
FastEstimator-Train: step: 11000; base_ce: 0.7146585; steps/sec: 120.02;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time(sec): 9.05;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 67.19;
Eval Progress: 66/100; steps/sec: 101.21;
Eval Progress: 100/100; steps/sec: 104.08;
FastEstimator-Eval: step: 11000; epoch: 11; adv_ce: 2.5616105; adversarial accuracy: 0.386; base accuracy: 0.6952; base_ce: 1.1582611; min_base_ce: 1.1493763; since_best_base_ce: 1;
FastEstimator-Train: step: 11500; base_ce: 0.66971624; steps/sec: 104.48;
FastEstimator-Train: step: 12000; base_ce: 0.7961953; steps/sec: 126.79;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time(sec): 8.73;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 63.36;
Eval Progress: 66/100; steps/sec: 108.08;
Eval Progress: 100/100; steps/sec: 111.2;
FastEstimator-Eval: step: 12000; epoch: 12; adv_ce: 2.560478; adversarial accuracy: 0.3798; base accuracy: 0.699; base_ce: 1.1520201; min_base_ce: 1.1493763; since_best_base_ce: 2;
FastEstimator-Train: step: 12500; base_ce: 0.9269663; steps/sec: 106.51;
FastEstimator-Train: step: 13000; base_ce: 0.72054017; steps/sec: 133.24;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time(sec): 8.46;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 66.38;
Eval Progress: 66/100; steps/sec: 107.15;
Eval Progress: 100/100; steps/sec: 107.88;
FastEstimator-Eval: step: 13000; epoch: 13; adv_ce: 2.6646338; adversarial accuracy: 0.3782; base accuracy: 0.696; base_ce: 1.1824644; min_base_ce: 1.1493763; since_best_base_ce: 3;
FastEstimator-Train: step: 13500; base_ce: 1.2424265; steps/sec: 107.64;
FastEstimator-Train: step: 14000; base_ce: 0.8177934; steps/sec: 133.66;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time(sec): 8.39;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 65.7;
Eval Progress: 66/100; steps/sec: 109.39;
Eval Progress: 100/100; steps/sec: 110.23;
FastEstimator-Eval: step: 14000; epoch: 14; adv_ce: 2.5994625; adversarial accuracy: 0.3936; base accuracy: 0.7012; base_ce: 1.1612; min_base_ce: 1.1493763; since_best_base_ce: 4;
FastEstimator-Train: step: 14500; base_ce: 0.6988259; steps/sec: 100.73;
FastEstimator-Train: step: 15000; base_ce: 0.70263755; steps/sec: 116.44;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time(sec): 9.25;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 63.26;
Eval Progress: 66/100; steps/sec: 108.64;
Eval Progress: 100/100; steps/sec: 105.48;
FastEstimator-Eval: step: 15000; epoch: 15; adv_ce: 2.6460128; adversarial accuracy: 0.3838; base accuracy: 0.691; base_ce: 1.1852306; min_base_ce: 1.1493763; since_best_base_ce: 5;
FastEstimator-Train: step: 15500; base_ce: 0.8898889; steps/sec: 92.78;
FastEstimator-Train: step: 16000; base_ce: 0.6189864; steps/sec: 118.39;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time(sec): 9.6;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 63.97;
Eval Progress: 66/100; steps/sec: 106.57;
Eval Progress: 100/100; steps/sec: 108.24;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Eval: step: 16000; epoch: 16; adv_ce: 2.5837348; adversarial accuracy: 0.4038; base accuracy: 0.7118; base_ce: 1.1321213; min_base_ce: 1.1321213; since_best_base_ce: 0;
FastEstimator-Train: step: 16500; base_ce: 0.4866921; steps/sec: 100.9;
FastEstimator-Train: step: 17000; base_ce: 0.54811317; steps/sec: 129.03;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time(sec): 8.84;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 60.29;
Eval Progress: 66/100; steps/sec: 96.05;
Eval Progress: 100/100; steps/sec: 97.08;
FastEstimator-Eval: step: 17000; epoch: 17; adv_ce: 2.6166897; adversarial accuracy: 0.3942; base accuracy: 0.7076; base_ce: 1.1557394; min_base_ce: 1.1321213; since_best_base_ce: 1;
FastEstimator-Train: step: 17500; base_ce: 0.4839167; steps/sec: 104.48;
FastEstimator-Train: step: 18000; base_ce: 0.50848377; steps/sec: 131.38;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time(sec): 8.59;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 60.83;
Eval Progress: 66/100; steps/sec: 106.91;
Eval Progress: 100/100; steps/sec: 104.04;
FastEstimator-Eval: step: 18000; epoch: 18; adv_ce: 2.6919034; adversarial accuracy: 0.3834; base accuracy: 0.7032; base_ce: 1.1592705; min_base_ce: 1.1321213; since_best_base_ce: 2;
FastEstimator-Train: step: 18500; base_ce: 0.64096814; steps/sec: 98.43;
FastEstimator-Train: step: 19000; base_ce: 0.62358904; steps/sec: 127.66;
FastEstimator-Train: step: 19000; epoch: 19; epoch_time(sec): 9.02;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 64.04;
Eval Progress: 66/100; steps/sec: 110.82;
Eval Progress: 100/100; steps/sec: 112.18;
FastEstimator-Eval: step: 19000; epoch: 19; adv_ce: 2.62304; adversarial accuracy: 0.4044; base accuracy: 0.7166; base_ce: 1.1378875; min_base_ce: 1.1321213; since_best_base_ce: 3;
FastEstimator-Train: step: 19500; base_ce: 0.52066106; steps/sec: 103.97;
FastEstimator-Train: step: 20000; base_ce: 0.88819975; steps/sec: 125.02;
FastEstimator-Train: step: 20000; epoch: 20; epoch_time(sec): 8.79;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 62.84;
Eval Progress: 66/100; steps/sec: 109.28;
Eval Progress: 100/100; steps/sec: 110.51;
FastEstimator-Eval: step: 20000; epoch: 20; adv_ce: 2.6513033; adversarial accuracy: 0.4058; base accuracy: 0.7106; base_ce: 1.1510154; min_base_ce: 1.1321213; since_best_base_ce: 4;
FastEstimator-Train: step: 20500; base_ce: 0.59055007; steps/sec: 102.23;
FastEstimator-Train: step: 21000; base_ce: 0.34733766; steps/sec: 123.2;
FastEstimator-Train: step: 21000; epoch: 21; epoch_time(sec): 8.94;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 66.78;
Eval Progress: 66/100; steps/sec: 109.78;
Eval Progress: 100/100; steps/sec: 110.84;
FastEstimator-Eval: step: 21000; epoch: 21; adv_ce: 2.6363294; adversarial accuracy: 0.4108; base accuracy: 0.7108; base_ce: 1.1565379; min_base_ce: 1.1321213; since_best_base_ce: 5;
FastEstimator-Train: step: 21500; base_ce: 0.49415398; steps/sec: 105.58;
FastEstimator-Train: step: 22000; base_ce: 0.36337882; steps/sec: 128.41;
FastEstimator-Train: step: 22000; epoch: 22; epoch_time(sec): 8.63;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 65.59;
Eval Progress: 66/100; steps/sec: 105.74;
Eval Progress: 100/100; steps/sec: 107.92;
FastEstimator-Eval: step: 22000; epoch: 22; adv_ce: 2.6627698; adversarial accuracy: 0.409; base accuracy: 0.7078; base_ce: 1.1768879; min_base_ce: 1.1321213; since_best_base_ce: 6;
FastEstimator-Train: step: 22500; base_ce: 0.63375324; steps/sec: 100.74;
FastEstimator-Train: step: 23000; base_ce: 0.505026; steps/sec: 121.12;
FastEstimator-Train: step: 23000; epoch: 23; epoch_time(sec): 9.1;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 67.83;
Eval Progress: 66/100; steps/sec: 103.8;
Eval Progress: 100/100; steps/sec: 101.68;
FastEstimator-Eval: step: 23000; epoch: 23; adv_ce: 2.6889231; adversarial accuracy: 0.4006; base accuracy: 0.7102; base_ce: 1.1820894; min_base_ce: 1.1321213; since_best_base_ce: 7;
FastEstimator-Train: step: 23500; base_ce: 0.35326964; steps/sec: 98.62;
FastEstimator-Train: step: 24000; base_ce: 0.60003966; steps/sec: 121.1;
FastEstimator-Train: step: 24000; epoch: 24; epoch_time(sec): 9.2;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 64.9;
Eval Progress: 66/100; steps/sec: 110.18;
Eval Progress: 100/100; steps/sec: 112.02;
FastEstimator-Eval: step: 24000; epoch: 24; adv_ce: 2.6645036; adversarial accuracy: 0.4112; base accuracy: 0.7088; base_ce: 1.1713258; min_base_ce: 1.1321213; since_best_base_ce: 8;
FastEstimator-Train: step: 24500; base_ce: 0.46151778; steps/sec: 106.62;
FastEstimator-Train: step: 25000; base_ce: 0.37955818; steps/sec: 130.54;
FastEstimator-Train: step: 25000; epoch: 25; epoch_time(sec): 8.54;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 66.79;
Eval Progress: 66/100; steps/sec: 111.22;
Eval Progress: 100/100; steps/sec: 110.93;
FastEstimator-Eval: step: 25000; epoch: 25; adv_ce: 2.6917067; adversarial accuracy: 0.4004; base accuracy: 0.7098; base_ce: 1.1873577; min_base_ce: 1.1321213; since_best_base_ce: 9;
FastEstimator-Train: step: 25500; base_ce: 0.38970187; steps/sec: 101.04;
FastEstimator-Train: step: 26000; base_ce: 0.40019217; steps/sec: 126.5;
FastEstimator-Train: step: 26000; epoch: 26; epoch_time(sec): 8.88;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 63.41;
Eval Progress: 66/100; steps/sec: 111.79;
Eval Progress: 100/100; steps/sec: 113.37;
FastEstimator-EarlyStopping: 'base_ce' triggered an early stop. Its best value was 1.1321213245391846 at epoch 16
FastEstimator-Eval: step: 26000; epoch: 26; adv_ce: 2.822452; adversarial accuracy: 0.3812; base accuracy: 0.7092; base_ce: 1.2155824; min_base_ce: 1.1321213; since_best_base_ce: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/ecc_best_base_ce.h5
FastEstimator-Finish: step: 26000; ecc_lr: 0.001; total_time(sec): 277.54;
FastEstimator-Test: step: 26000; epoch: 26; adv_ce: 2.5278254; adversarial accuracy: 0.4136; base accuracy: 0.7198; base_ce: 1.0935019;

In [13]:

hydra_estimator.fit('Hydra')
hydra_results = hydra_estimator.test()

hydra_estimator.fit('Hydra') hydra_results = hydra_estimator.test()

    ______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; logging_interval: 500; num_device: 1;
FastEstimator-Train: step: 1; base_ce: 2.3081625;
FastEstimator-Train: step: 500; base_ce: 1.7884388; steps/sec: 87.5;
FastEstimator-Train: step: 1000; base_ce: 1.9491867; steps/sec: 94.01;
FastEstimator-Train: step: 1000; epoch: 1; epoch_time(sec): 12.01;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 51.12;
Eval Progress: 66/100; steps/sec: 77.72;
Eval Progress: 100/100; steps/sec: 83.22;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 1000; epoch: 1; adv_ce: 2.1806147; adversarial accuracy: 0.3234; base accuracy: 0.468; base_ce: 1.7376462; min_base_ce: 1.7376462; since_best_base_ce: 0;
FastEstimator-Train: step: 1500; base_ce: 1.5212517; steps/sec: 69.35;
FastEstimator-Train: step: 2000; base_ce: 1.7228339; steps/sec: 89.66;
FastEstimator-Train: step: 2000; epoch: 2; epoch_time(sec): 12.79;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 45.69;
Eval Progress: 66/100; steps/sec: 83.72;
Eval Progress: 100/100; steps/sec: 85.91;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 2000; epoch: 2; adv_ce: 2.0905602; adversarial accuracy: 0.3516; base accuracy: 0.5616; base_ce: 1.5577309; min_base_ce: 1.5577309; since_best_base_ce: 0;
FastEstimator-Train: step: 2500; base_ce: 1.1112486; steps/sec: 74.25;
FastEstimator-Train: step: 3000; base_ce: 1.2904729; steps/sec: 92.34;
FastEstimator-Train: step: 3000; epoch: 3; epoch_time(sec): 12.14;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 47.76;
Eval Progress: 66/100; steps/sec: 84.65;
Eval Progress: 100/100; steps/sec: 86.39;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 3000; epoch: 3; adv_ce: 2.0920057; adversarial accuracy: 0.373; base accuracy: 0.5958; base_ce: 1.4505764; min_base_ce: 1.4505764; since_best_base_ce: 0;
FastEstimator-Train: step: 3500; base_ce: 1.2749859; steps/sec: 72.94;
FastEstimator-Train: step: 4000; base_ce: 1.5297793; steps/sec: 91.22;
FastEstimator-Train: step: 4000; epoch: 4; epoch_time(sec): 12.34;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 47.42;
Eval Progress: 66/100; steps/sec: 83.2;
Eval Progress: 100/100; steps/sec: 85.02;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 4000; epoch: 4; adv_ce: 2.161559; adversarial accuracy: 0.3676; base accuracy: 0.6072; base_ce: 1.3917297; min_base_ce: 1.3917297; since_best_base_ce: 0;
FastEstimator-Train: step: 4500; base_ce: 1.4035791; steps/sec: 71.54;
FastEstimator-Train: step: 5000; base_ce: 0.81152534; steps/sec: 87.02;
FastEstimator-Train: step: 5000; epoch: 5; epoch_time(sec): 12.73;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 45.26;
Eval Progress: 66/100; steps/sec: 85.34;
Eval Progress: 100/100; steps/sec: 92.87;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 5000; epoch: 5; adv_ce: 2.1359746; adversarial accuracy: 0.3994; base accuracy: 0.6376; base_ce: 1.3213928; min_base_ce: 1.3213928; since_best_base_ce: 0;
FastEstimator-Train: step: 5500; base_ce: 1.0143056; steps/sec: 73.22;
FastEstimator-Train: step: 6000; base_ce: 1.5183457; steps/sec: 93.21;
FastEstimator-Train: step: 6000; epoch: 6; epoch_time(sec): 12.21;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 42.1;
Eval Progress: 66/100; steps/sec: 79.6;
Eval Progress: 100/100; steps/sec: 83.86;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 6000; epoch: 6; adv_ce: 2.1291919; adversarial accuracy: 0.398; base accuracy: 0.6568; base_ce: 1.277685; min_base_ce: 1.277685; since_best_base_ce: 0;
FastEstimator-Train: step: 6500; base_ce: 1.2950306; steps/sec: 72.45;
FastEstimator-Train: step: 7000; base_ce: 1.0940616; steps/sec: 92.42;
FastEstimator-Train: step: 7000; epoch: 7; epoch_time(sec): 12.3;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.61;
Eval Progress: 66/100; steps/sec: 84.79;
Eval Progress: 100/100; steps/sec: 89.68;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 7000; epoch: 7; adv_ce: 2.1633997; adversarial accuracy: 0.3944; base accuracy: 0.6646; base_ce: 1.2406228; min_base_ce: 1.2406228; since_best_base_ce: 0;
FastEstimator-Train: step: 7500; base_ce: 1.1472574; steps/sec: 70.23;
FastEstimator-Train: step: 8000; base_ce: 1.2201166; steps/sec: 89.28;
FastEstimator-Train: step: 8000; epoch: 8; epoch_time(sec): 12.72;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 37.31;
Eval Progress: 66/100; steps/sec: 80.14;
Eval Progress: 100/100; steps/sec: 89.35;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 8000; epoch: 8; adv_ce: 2.2015975; adversarial accuracy: 0.3908; base accuracy: 0.6686; base_ce: 1.2382783; min_base_ce: 1.2382783; since_best_base_ce: 0;
FastEstimator-Train: step: 8500; base_ce: 0.9792079; steps/sec: 68.11;
FastEstimator-Train: step: 9000; base_ce: 0.7591236; steps/sec: 88.72;
FastEstimator-Train: step: 9000; epoch: 9; epoch_time(sec): 12.97;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 40.18;
Eval Progress: 66/100; steps/sec: 92.03;
Eval Progress: 100/100; steps/sec: 92.36;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 9000; epoch: 9; adv_ce: 2.2286303; adversarial accuracy: 0.404; base accuracy: 0.6734; base_ce: 1.2206097; min_base_ce: 1.2206097; since_best_base_ce: 0;
FastEstimator-Train: step: 9500; base_ce: 0.90840054; steps/sec: 70.09;
FastEstimator-Train: step: 10000; base_ce: 1.0802205; steps/sec: 90.15;
FastEstimator-Train: step: 10000; epoch: 10; epoch_time(sec): 12.68;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.75;
Eval Progress: 66/100; steps/sec: 89.61;
Eval Progress: 100/100; steps/sec: 87.95;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 10000; epoch: 10; adv_ce: 2.23915; adversarial accuracy: 0.409; base accuracy: 0.6802; base_ce: 1.2094864; min_base_ce: 1.2094864; since_best_base_ce: 0;
FastEstimator-Train: step: 10500; base_ce: 1.0814967; steps/sec: 65.32;
FastEstimator-Train: step: 11000; base_ce: 0.9755617; steps/sec: 89.57;
FastEstimator-Train: step: 11000; epoch: 11; epoch_time(sec): 13.26;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.95;
Eval Progress: 66/100; steps/sec: 86.57;
Eval Progress: 100/100; steps/sec: 82.25;
FastEstimator-Eval: step: 11000; epoch: 11; adv_ce: 2.3192086; adversarial accuracy: 0.3872; base accuracy: 0.6572; base_ce: 1.2798961; min_base_ce: 1.2094864; since_best_base_ce: 1;
FastEstimator-Train: step: 11500; base_ce: 0.94549084; steps/sec: 68.3;
FastEstimator-Train: step: 12000; base_ce: 1.2109189; steps/sec: 89.82;
FastEstimator-Train: step: 12000; epoch: 12; epoch_time(sec): 12.87;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.54;
Eval Progress: 66/100; steps/sec: 92.41;
Eval Progress: 100/100; steps/sec: 92.28;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 12000; epoch: 12; adv_ce: 2.310278; adversarial accuracy: 0.3992; base accuracy: 0.6868; base_ce: 1.1944125; min_base_ce: 1.1944125; since_best_base_ce: 0;
FastEstimator-Train: step: 12500; base_ce: 0.9911569; steps/sec: 67.18;
FastEstimator-Train: step: 13000; base_ce: 1.1323882; steps/sec: 89.61;
FastEstimator-Train: step: 13000; epoch: 13; epoch_time(sec): 13.03;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.16;
Eval Progress: 66/100; steps/sec: 89.55;
Eval Progress: 100/100; steps/sec: 82.19;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 13000; epoch: 13; adv_ce: 2.2913733; adversarial accuracy: 0.4014; base accuracy: 0.6992; base_ce: 1.1622982; min_base_ce: 1.1622982; since_best_base_ce: 0;
FastEstimator-Train: step: 13500; base_ce: 1.1398926; steps/sec: 65.39;
FastEstimator-Train: step: 14000; base_ce: 0.7471756; steps/sec: 90.13;
FastEstimator-Train: step: 14000; epoch: 14; epoch_time(sec): 13.21;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.56;
Eval Progress: 66/100; steps/sec: 90.26;
Eval Progress: 100/100; steps/sec: 92.26;
FastEstimator-Eval: step: 14000; epoch: 14; adv_ce: 2.2787528; adversarial accuracy: 0.4134; base accuracy: 0.6912; base_ce: 1.1762925; min_base_ce: 1.1622982; since_best_base_ce: 1;
FastEstimator-Train: step: 14500; base_ce: 0.8364514; steps/sec: 67.59;
FastEstimator-Train: step: 15000; base_ce: 0.8182548; steps/sec: 88.7;
FastEstimator-Train: step: 15000; epoch: 15; epoch_time(sec): 13.01;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.26;
Eval Progress: 66/100; steps/sec: 85.29;
Eval Progress: 100/100; steps/sec: 86.75;
FastEstimator-Eval: step: 15000; epoch: 15; adv_ce: 2.336492; adversarial accuracy: 0.4; base accuracy: 0.6914; base_ce: 1.1849647; min_base_ce: 1.1622982; since_best_base_ce: 2;
FastEstimator-Train: step: 15500; base_ce: 0.9234868; steps/sec: 71.62;
FastEstimator-Train: step: 16000; base_ce: 0.5794194; steps/sec: 88.95;
FastEstimator-Train: step: 16000; epoch: 16; epoch_time(sec): 12.61;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 34.58;
Eval Progress: 66/100; steps/sec: 83.15;
Eval Progress: 100/100; steps/sec: 82.23;
FastEstimator-Eval: step: 16000; epoch: 16; adv_ce: 2.380893; adversarial accuracy: 0.3974; base accuracy: 0.694; base_ce: 1.1697056; min_base_ce: 1.1622982; since_best_base_ce: 3;
FastEstimator-Train: step: 16500; base_ce: 1.0575782; steps/sec: 65.98;
FastEstimator-Train: step: 17000; base_ce: 0.7640396; steps/sec: 79.39;
FastEstimator-Train: step: 17000; epoch: 17; epoch_time(sec): 13.89;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.14;
Eval Progress: 66/100; steps/sec: 82.73;
Eval Progress: 100/100; steps/sec: 86.26;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 17000; epoch: 17; adv_ce: 2.306381; adversarial accuracy: 0.4082; base accuracy: 0.697; base_ce: 1.1417962; min_base_ce: 1.1417962; since_best_base_ce: 0;
FastEstimator-Train: step: 17500; base_ce: 0.73347974; steps/sec: 66.18;
FastEstimator-Train: step: 18000; base_ce: 0.8569354; steps/sec: 87.86;
FastEstimator-Train: step: 18000; epoch: 18; epoch_time(sec): 13.23;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 37.29;
Eval Progress: 66/100; steps/sec: 81.02;
Eval Progress: 100/100; steps/sec: 85.49;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 18000; epoch: 18; adv_ce: 2.3636973; adversarial accuracy: 0.408; base accuracy: 0.7068; base_ce: 1.1360252; min_base_ce: 1.1360252; since_best_base_ce: 0;
FastEstimator-Train: step: 18500; base_ce: 0.8577887; steps/sec: 63.29;
FastEstimator-Train: step: 19000; base_ce: 0.9215192; steps/sec: 85.35;
FastEstimator-Train: step: 19000; epoch: 19; epoch_time(sec): 13.77;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.23;
Eval Progress: 66/100; steps/sec: 85.16;
Eval Progress: 100/100; steps/sec: 85.0;
FastEstimator-Eval: step: 19000; epoch: 19; adv_ce: 2.4441173; adversarial accuracy: 0.391; base accuracy: 0.7024; base_ce: 1.1542323; min_base_ce: 1.1360252; since_best_base_ce: 1;
FastEstimator-Train: step: 19500; base_ce: 0.67412055; steps/sec: 67.91;
FastEstimator-Train: step: 20000; base_ce: 0.5729052; steps/sec: 89.53;
FastEstimator-Train: step: 20000; epoch: 20; epoch_time(sec): 12.94;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 37.53;
Eval Progress: 66/100; steps/sec: 83.34;
Eval Progress: 100/100; steps/sec: 86.33;
FastEstimator-Eval: step: 20000; epoch: 20; adv_ce: 2.3474467; adversarial accuracy: 0.4114; base accuracy: 0.707; base_ce: 1.137734; min_base_ce: 1.1360252; since_best_base_ce: 2;
FastEstimator-Train: step: 20500; base_ce: 0.63434684; steps/sec: 63.94;
FastEstimator-Train: step: 21000; base_ce: 0.623537; steps/sec: 80.45;
FastEstimator-Train: step: 21000; epoch: 21; epoch_time(sec): 14.03;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.13;
Eval Progress: 66/100; steps/sec: 81.35;
Eval Progress: 100/100; steps/sec: 78.04;
FastEstimator-Eval: step: 21000; epoch: 21; adv_ce: 2.3849573; adversarial accuracy: 0.4032; base accuracy: 0.6994; base_ce: 1.1664996; min_base_ce: 1.1360252; since_best_base_ce: 3;
FastEstimator-Train: step: 21500; base_ce: 0.7532989; steps/sec: 70.14;
FastEstimator-Train: step: 22000; base_ce: 0.6230503; steps/sec: 91.32;
FastEstimator-Train: step: 22000; epoch: 22; epoch_time(sec): 12.6;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.85;
Eval Progress: 66/100; steps/sec: 85.4;
Eval Progress: 100/100; steps/sec: 84.63;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 22000; epoch: 22; adv_ce: 2.4085927; adversarial accuracy: 0.4024; base accuracy: 0.7228; base_ce: 1.1094034; min_base_ce: 1.1094034; since_best_base_ce: 0;
FastEstimator-Train: step: 22500; base_ce: 0.95978475; steps/sec: 69.54;
FastEstimator-Train: step: 23000; base_ce: 0.6422969; steps/sec: 85.62;
FastEstimator-Train: step: 23000; epoch: 23; epoch_time(sec): 13.04;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.17;
Eval Progress: 66/100; steps/sec: 85.78;
Eval Progress: 100/100; steps/sec: 89.68;
FastEstimator-Eval: step: 23000; epoch: 23; adv_ce: 2.429416; adversarial accuracy: 0.3928; base accuracy: 0.71; base_ce: 1.1393782; min_base_ce: 1.1094034; since_best_base_ce: 1;
FastEstimator-Train: step: 23500; base_ce: 0.717039; steps/sec: 66.78;
FastEstimator-Train: step: 24000; base_ce: 0.7696357; steps/sec: 83.61;
FastEstimator-Train: step: 24000; epoch: 24; epoch_time(sec): 13.49;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 39.02;
Eval Progress: 66/100; steps/sec: 88.07;
Eval Progress: 100/100; steps/sec: 88.6;
FastEstimator-Eval: step: 24000; epoch: 24; adv_ce: 2.4276404; adversarial accuracy: 0.4024; base accuracy: 0.7154; base_ce: 1.1275772; min_base_ce: 1.1094034; since_best_base_ce: 2;
FastEstimator-Train: step: 24500; base_ce: 0.4894743; steps/sec: 67.88;
FastEstimator-Train: step: 25000; base_ce: 0.68931097; steps/sec: 88.26;
FastEstimator-Train: step: 25000; epoch: 25; epoch_time(sec): 13.01;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.46;
Eval Progress: 66/100; steps/sec: 87.74;
Eval Progress: 100/100; steps/sec: 87.85;
FastEstimator-BestModelSaver: Saved model to /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Eval: step: 25000; epoch: 25; adv_ce: 2.3925145; adversarial accuracy: 0.4112; base accuracy: 0.7198; base_ce: 1.0948123; min_base_ce: 1.0948123; since_best_base_ce: 0;
FastEstimator-Train: step: 25500; base_ce: 0.94024915; steps/sec: 61.02;
FastEstimator-Train: step: 26000; base_ce: 0.9456716; steps/sec: 73.59;
FastEstimator-Train: step: 26000; epoch: 26; epoch_time(sec): 15.0;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 34.32;
Eval Progress: 66/100; steps/sec: 84.8;
Eval Progress: 100/100; steps/sec: 83.27;
FastEstimator-Eval: step: 26000; epoch: 26; adv_ce: 2.4177082; adversarial accuracy: 0.411; base accuracy: 0.7188; base_ce: 1.0978441; min_base_ce: 1.0948123; since_best_base_ce: 1;
FastEstimator-Train: step: 26500; base_ce: 0.84369594; steps/sec: 63.81;
FastEstimator-Train: step: 27000; base_ce: 0.5412454; steps/sec: 79.4;
FastEstimator-Train: step: 27000; epoch: 27; epoch_time(sec): 14.14;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.56;
Eval Progress: 66/100; steps/sec: 82.42;
Eval Progress: 100/100; steps/sec: 89.81;
FastEstimator-Eval: step: 27000; epoch: 27; adv_ce: 2.352829; adversarial accuracy: 0.4228; base accuracy: 0.7214; base_ce: 1.1148609; min_base_ce: 1.0948123; since_best_base_ce: 2;
FastEstimator-Train: step: 27500; base_ce: 0.53644246; steps/sec: 62.7;
FastEstimator-Train: step: 28000; base_ce: 0.6651532; steps/sec: 75.84;
FastEstimator-Train: step: 28000; epoch: 28; epoch_time(sec): 14.57;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 35.42;
Eval Progress: 66/100; steps/sec: 79.81;
Eval Progress: 100/100; steps/sec: 80.41;
FastEstimator-Eval: step: 28000; epoch: 28; adv_ce: 2.4664576; adversarial accuracy: 0.4022; base accuracy: 0.7206; base_ce: 1.1072303; min_base_ce: 1.0948123; since_best_base_ce: 3;
FastEstimator-Train: step: 28500; base_ce: 0.4171321; steps/sec: 62.68;
FastEstimator-Train: step: 29000; base_ce: 0.6967076; steps/sec: 80.42;
FastEstimator-Train: step: 29000; epoch: 29; epoch_time(sec): 14.21;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 34.83;
Eval Progress: 66/100; steps/sec: 77.09;
Eval Progress: 100/100; steps/sec: 77.51;
FastEstimator-Eval: step: 29000; epoch: 29; adv_ce: 2.4626358; adversarial accuracy: 0.403; base accuracy: 0.7228; base_ce: 1.1151576; min_base_ce: 1.0948123; since_best_base_ce: 4;
FastEstimator-Train: step: 29500; base_ce: 0.58600163; steps/sec: 59.94;
FastEstimator-Train: step: 30000; base_ce: 0.56860876; steps/sec: 76.45;
FastEstimator-Train: step: 30000; epoch: 30; epoch_time(sec): 14.86;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.48;
Eval Progress: 66/100; steps/sec: 74.11;
Eval Progress: 100/100; steps/sec: 79.66;
FastEstimator-Eval: step: 30000; epoch: 30; adv_ce: 2.4656303; adversarial accuracy: 0.3998; base accuracy: 0.7192; base_ce: 1.0986785; min_base_ce: 1.0948123; since_best_base_ce: 5;
FastEstimator-Train: step: 30500; base_ce: 0.65966034; steps/sec: 64.22;
FastEstimator-Train: step: 31000; base_ce: 0.5600899; steps/sec: 77.18;
FastEstimator-Train: step: 31000; epoch: 31; epoch_time(sec): 14.28;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.84;
Eval Progress: 66/100; steps/sec: 87.73;
Eval Progress: 100/100; steps/sec: 83.9;
FastEstimator-Eval: step: 31000; epoch: 31; adv_ce: 2.4026325; adversarial accuracy: 0.4168; base accuracy: 0.7214; base_ce: 1.1182215; min_base_ce: 1.0948123; since_best_base_ce: 6;
FastEstimator-Train: step: 31500; base_ce: 0.6140233; steps/sec: 66.1;
FastEstimator-Train: step: 32000; base_ce: 0.8687953; steps/sec: 82.68;
FastEstimator-Train: step: 32000; epoch: 32; epoch_time(sec): 13.59;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 38.18;
Eval Progress: 66/100; steps/sec: 81.48;
Eval Progress: 100/100; steps/sec: 87.72;
FastEstimator-Eval: step: 32000; epoch: 32; adv_ce: 2.4584744; adversarial accuracy: 0.4134; base accuracy: 0.7192; base_ce: 1.1003925; min_base_ce: 1.0948123; since_best_base_ce: 7;
FastEstimator-Train: step: 32500; base_ce: 0.48308387; steps/sec: 61.36;
FastEstimator-Train: step: 33000; base_ce: 0.8083018; steps/sec: 85.07;
FastEstimator-Train: step: 33000; epoch: 33; epoch_time(sec): 14.08;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.75;
Eval Progress: 66/100; steps/sec: 85.59;
Eval Progress: 100/100; steps/sec: 83.29;
FastEstimator-Eval: step: 33000; epoch: 33; adv_ce: 2.4739573; adversarial accuracy: 0.4078; base accuracy: 0.7206; base_ce: 1.1157078; min_base_ce: 1.0948123; since_best_base_ce: 8;
FastEstimator-Train: step: 33500; base_ce: 0.75273496; steps/sec: 60.92;
FastEstimator-Train: step: 34000; base_ce: 0.7295588; steps/sec: 83.41;
FastEstimator-Train: step: 34000; epoch: 34; epoch_time(sec): 14.14;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 36.19;
Eval Progress: 66/100; steps/sec: 81.28;
Eval Progress: 100/100; steps/sec: 83.99;
FastEstimator-Eval: step: 34000; epoch: 34; adv_ce: 2.4834745; adversarial accuracy: 0.4086; base accuracy: 0.716; base_ce: 1.1190228; min_base_ce: 1.0948123; since_best_base_ce: 9;
FastEstimator-Train: step: 34500; base_ce: 0.7029926; steps/sec: 59.99;
FastEstimator-Train: step: 35000; base_ce: 0.7395061; steps/sec: 80.36;
FastEstimator-Train: step: 35000; epoch: 35; epoch_time(sec): 14.56;
Eval Progress: 1/100;
Eval Progress: 33/100; steps/sec: 32.36;
Eval Progress: 66/100; steps/sec: 74.18;
Eval Progress: 100/100; steps/sec: 74.19;
FastEstimator-EarlyStopping: 'base_ce' triggered an early stop. Its best value was 1.094812273979187 at epoch 25
FastEstimator-Eval: step: 35000; epoch: 35; adv_ce: 2.5064902; adversarial accuracy: 0.4078; base accuracy: 0.7152; base_ce: 1.1371014; min_base_ce: 1.0948123; since_best_base_ce: 10;
FastEstimator-BestModelSaver: Restoring model from /var/folders/lx/drkxftt117gblvgsp1p39rlc0000gn/T/tmpjrnh638m/hydra_ecc_best_base_ce.h5
FastEstimator-Finish: step: 35000; hydra_ecc_lr: 0.001; total_time(sec): 566.2;
FastEstimator-Test: step: 35000; epoch: 35; adv_ce: 2.384631; adversarial accuracy: 0.4168; base accuracy: 0.7282; base_ce: 1.0672554;

Comparing the Results¶

In [14]:

logs = visualize_logs(experiments=[softmax_results, ecc_results, hydra_results], ignore_metrics={'ecc_lr', 'hydra_ecc_lr', 'softmax_lr', 'logging_interval', 'num_device', 'epoch_time', 'min_base_ce', 'adv_ce', 'total_time'})

logs = visualize_logs(experiments=[softmax_results, ecc_results, hydra_results], ignore_metrics={'ecc_lr', 'hydra_ecc_lr', 'softmax_lr', 'logging_interval', 'num_device', 'epoch_time', 'min_base_ce', 'adv_ce', 'total_time'})

As you can see, the conventional network using softmax to convert logits to class probabilities actually gets more and more vulnerable to adversarial attacks as training progresses. It also quickly overfits to the data, reaching an optimal performance around epoch 7. By simply switching the softmax layer for an error-correcting-code, the network is able to train for around 16 epochs before starting to cap out, and even then continuing to train it results in better and better adversarial performance. Creating a multi-headed ecc output layer allows still more training and higher peak performances. If you were to run the experiment out to 160 epochs you would find that the adversarial accuracy can reach between 60-70% with only a slight accuracy degradation on clean samples (performance still above 70%). This is significantly better performance than networks trained specifically to combat this attack, shown in the FGSM notebook. Note also that their is relatively little additional cost to training using ECC as opposed to softmax in terms of steps/sec. This is a big benefit over FGSM, where the training time for each step is doubled. With these benefits in mind, you may want to consider never using softmax again.