[Python] Calculating cross entropy in TensorFlow

I am having a hard time with calculating cross entropy in TensorFlow. In particular, I am using the function:

tf.nn.softmax_cross_entropy_with_logits()


Using what is seemingly simple code, I can only get it to return a zero:

import tensorflow as tf
import numpy as np

sess = tf.InteractiveSession()

a = tf.placeholder(tf.float32, shape =[None, 1])
b = tf.placeholder(tf.float32, shape = [None, 1])
sess.run(tf.global_variables_initializer())
c = tf.nn.softmax_cross_entropy_with_logits(
logits=b, labels=a
).eval(feed_dict={b:np.array([[0.45]]), a:np.array([[0.2]])})
print c


returns

0


My understanding of cross entropy is as follows:

H(p,q) = p(x)*log(q(x))


Where p(x) is the true probability of event x and q(x) is the predicted probability of event x.

There if input any two numbers for p(x) and q(x) are used such that

0<p(x)<1 AND 0<q(x)<1


there should be a nonzero cross entropy. I am expecting that I am using TensorFlow incorrectly.

Continue reading...